Image processing apparatus, imaging apparatus, and image processing method
Patent Information
- Application Number
- JP2021195669
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-12-01
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2041-12-01
AI Technical Summary
Existing image processing methods using polarizing elements suffer from significant influence of incident angle characteristics, particularly when the phase difference imparted by the variable retardation plate is large, leading to reduced accuracy in polarization information.
An image processing apparatus and method that acquires multiple images with different polarization directions, corrects for incident angle characteristics by setting corrected polarization orientation information based on imaging conditions, and generates a composite image using polarization information.
Reduces the influence of incident angle characteristics, enabling accurate acquisition of polarization information and generation of composite images with improved specular and diffuse reflection component ratios.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing technique using polarization information obtained from a captured image.
Background Art
[0002] As an imaging method for acquiring polarization information indicating the polarization direction, intensity, and intensity of the unpolarized component, etc. from a subject, there is a method of rotating a polarizing plate to acquire a plurality of captured images in imaging through a polarizing plate. Further, as disclosed in Patent Document 1, in imaging through a polarization element composed of a retardation plate (λ / 4 plate), a variable retardation plate (liquid crystal), and a polarizing plate, by changing the retardation imparted to the light beam by the variable retardation plate, there is a method of acquiring a plurality of captured images with different polarization states without rotating the polarizing plate.
[0003] Furthermore, as disclosed in Patent Document 2, in imaging through the above polarization element, by setting the optical axis of the variable retardation plate to be parallel to the short side of the imaging element, a method of reducing the influence of the incident angle characteristics of the variable retardation plate on the polarization information is disclosed.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Patent Document 2
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, in the methods of Patent Documents 1 and 2, in a region where the incident angle of the light beam with respect to the variable retardation plate in the captured image is relatively large, or in a region where the retardation imparted to the light beam by the variable retardation plate is large and the incident angle characteristics are relatively reduced, the influence of the incident angle characteristics on the obtained polarization information cannot be ignored.
[0006] The present invention provides an image processing apparatus capable of obtaining polarization information with reduced influence from the incident angle characteristics of a polarizing element, and an imaging apparatus using the same. [Means for solving the problem]
[0007] An image processing apparatus as one aspect of the present invention is characterized by comprising: an image acquisition means for acquiring a first image generated by imaging a subject through a polarizing element that transmits light of multiple different polarization directions; an imaging condition acquisition means for acquiring information on imaging conditions in imaging; a polarization direction setting means for setting information on the polarization direction in the first image according to the information on imaging conditions; and a polarization information acquisition means for acquiring polarization information from the first image using the information on polarization direction. An imaging apparatus having an optical system, a polarizing element, an imaging element, and the above-mentioned image processing apparatus also constitutes another aspect of the present invention.
[0008] Furthermore, an image processing method, as another aspect of the present invention, is characterized by comprising the steps of: acquiring a first image generated by imaging a subject through a polarizing element that transmits light of multiple different polarization directions; acquiring information regarding imaging conditions in imaging; setting information regarding polarization directions in the first image according to the information regarding imaging conditions; and acquiring polarization information from the first image using the information regarding polarization directions. A program that causes a computer to perform processing according to the above image processing method also constitutes another aspect of the present invention. [Effects of the Invention]
[0009] According to the present invention, polarization information can be obtained with reduced influence from the incident angle characteristics of the polarizing element. [Brief explanation of the drawing]
[0010] [Figure 1] A diagram showing the configuration of the image processing apparatus in Example 1. [Figure 2] A diagram showing the relationship between luminance and transmitted polarization direction in Example 1. [Figure 3] A diagram showing an imaging apparatus including the image processing device of Example 1. [Figure 4] A diagram showing the relationship between the axial orientation of the λ / 4 plate and the variable phase difference plate in Example 12. [Figure 5] A diagram showing the usage area of corrected polarization orientation information in Example 1. [Figure 6] A diagram showing the rotation of the usage area in Example 1. [Figure 7] A diagram showing the translation of the usage area in Example 1. [Figure 8] This figure shows the tilt of the optical system relative to the image sensor in Example 1 and the setting of the corrected polarization orientation information at that time. [Figure 9] This figure shows the setting of corrected polarization orientation information to compensate for manufacturing errors in Example 1. [Figure 10] A figure showing an example of digitized corrected polarization orientation information in Example 1. [Figure 11] A flowchart illustrating the process in Example 1. [Figure 12] A diagram showing the configuration of the imaging device in Example 2. [Figure 13] A diagram showing the configuration of the image processing device in Reference Example 1. [Figure 14] A flowchart illustrating the process in Reference Example 1. [Figure 15] A diagram showing the changes in the axial direction and correction coefficient in Reference Example 1. [Figure 16] A figure showing the changes in other correction coefficients in Reference Example 1. [Figure 17] A flowchart illustrating the process in Reference Example 2. [Modes for carrying out the invention]
[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Examples]
[0012] The image processing apparatus of Example 1 has the function of acquiring polarization information from one or more captured images (first image) obtained by imaging a subject through a polarizing element that transmits light of multiple different polarization directions. The image processing apparatus also has the function of correcting the incident angle characteristics of the polarizing element (i.e., the non-uniform in-plane distribution of transmitted polarization directions). Furthermore, the image processing apparatus also has the function of generating a composite image (second image) using the acquired polarization information.
[0013] Figure 1 shows the configuration of the image processing apparatus 100 of Embodiment 1. The image processing apparatus 100 includes an image acquisition unit 101 as an image acquisition means, an imaging condition acquisition unit 102 as an imaging condition acquisition means, a polarization direction setting unit 103 as a polarization direction setting means, a polarization information acquisition unit 104 as a polarization information acquisition means, and an image generation unit 105 as an image generation means.
[0014] The image acquisition unit 101 acquires an image generated by imaging a subject using an imaging device that includes an optical system, a polarizing element, and an image sensor. The image processing device 100 may be a separate device from the imaging device, such as a personal computer, or it may be mounted on the imaging device as described later.
[0015] The polarization information acquisition unit 104 acquires polarization information using the corrected polarization direction information set by the polarization direction setting unit 103. Polarization information and corrected polarization direction information will be described later.
[0016] The image generation unit 105 generates a composite image using the acquired polarization information. Specific examples of composite images will be described later.
[0017] The imaging condition acquisition unit 102 acquires information regarding the imaging conditions at the time of imaging (information indicating or convertible to imaging conditions: hereinafter referred to as imaging condition information) when calculating polarization information. The imaging conditions will be described later.
[0018] The polarization direction setting unit 103 corrects the transmitted polarization direction based on the incident angle characteristics of the polarizing element according to the acquired imaging condition information, thereby setting information regarding the polarization direction in the captured image, i.e., corrected polarization direction information. The corrected polarization direction information is information that indicates or can be converted to indicate the polarization direction for acquiring polarization information, and is set for each pixel or region of the captured image (for example, for each pixel or region corresponding to a position or region with a different light ray incident angle in the polarizing element).
[0019] The following describes in more detail the processes performed by each part of the image processing device 100 described above. The flowchart in Figure 11 shows the flow of the above process (image processing method), and in the following explanation, the corresponding step numbers (S101 to S105) are indicated in parentheses. The image processing device 100, as a computer, performs this process according to the program.
[0020] The image acquisition unit 101 acquires multiple captured images with different transmitted polarization directions, which are generated when an imaging device equipped with a polarizing element captures an image of a subject, as described above (S101).
[0021] The polarizing element in the imaging device is an element whose transmitted polarization direction, which is the polarization direction of the light transmitted through it, is variable. Such a polarizing element can be constructed, for example, using a polarizing plate that can rotate around the optical axis of the optical system, or a variable phase difference plate that can change the phase difference imparted to the transmitted light, as disclosed in Patent Documents 1 and 2. The variable phase difference plate can be constructed from VA liquid crystal or OCB liquid crystal, etc. An imaging device having a polarizing element with a variable transmitted polarization direction generates multiple image images containing polarization information at different transmitted polarization directions by taking multiple images with different transmitted polarization directions of the polarizing element.
[0022] The polarizing element can be placed between the subject and the image sensor. Specifically, it can be placed on the subject side of the optical system, inside the optical system, or between the optical system and the image sensor. However, if a birefringent material or another polarizing element is placed between the subject and the polarizing element, the polarization state of the light from the subject will be disturbed, which is undesirable. For this reason, if the imaging device includes a plastic lens, a variable ND filter using a polarizing plate, and an optical low-pass filter made of a birefringent material, it is preferable to place the polarizing element on the subject side of these optical elements.
[0023] Furthermore, the polarizing element may have multiple transmitted polarization directions. For example, the polarizing element may include multiple polarizers, each with different transmitted polarization directions, arranged for each pixel of the image sensor. That is, a polarizing image sensor equipped with a polarizer for each pixel may be used as the image sensor. In this case, the imaging device generates a single image containing polarization information for multiple transmitted polarization directions in a single image taken by the polarizing image sensor.
[0024] While any image data format is acceptable, so-called RAW images—those that have not undergone compression, correction, or other processing—are preferred.
[0025] The imaging condition acquisition unit 102 acquires imaging condition information at the time of imaging of the captured image acquired by the image acquisition unit 101 (S102). The imaging conditions are various conditions related to imaging, excluding conditions related to the polarizing element, and include the focal length of the optical system, F-number, spectral transmittance, exit pupil position, focus state, subject distance, imaging magnification, and the type and specifications of the optical system. Furthermore, if the optical system is displaceable in the shift or tilt direction relative to the image sensor, the amount of displacement (shift amount or tilt amount) is also included in the imaging conditions. In addition, the imaging conditions include the size of the image sensor, maximum image height, pixel pitch, total number of pixels, and cropped pixel count. The imaging conditions may also include temperature, humidity, atmospheric pressure, and identification information of the imaging device.
[0026] The imaging condition acquisition unit 102 acquires imaging condition information by reading information indicating imaging conditions manually set by the user for the imaging device, or by reading imaging condition information attached as metadata to the header area of the captured image or a specified area in the Exif format (S1). The imaging condition acquisition unit 102 may also acquire imaging condition information by analyzing the captured image using a method such as Deep Learning, or by acquiring imaging condition information from other devices such as a distance measuring device used during imaging or a control device that controls the imaging device.
[0027] The polarization direction setting unit 103 acquires information regarding the transmitted polarization direction of the polarizing element during imaging (for example, information regarding the voltage applied to the variable phase difference plate, which will be described later). Then, it sets the corrected polarization direction information according to the imaging condition information acquired by the imaging condition acquisition unit 102 and the information regarding the transmitted polarization direction of the polarizing element during imaging (S103).
[0028] Ideally, a polarizing element should have a constant transmitted polarization direction and constant transmittance for the incident light ray, regardless of the angle of incidence. However, in reality, the optical path length and the apparent angle of the slow axis of the polarizing element change depending on the angle of incidence of the light ray to the polarizing element, resulting in a distribution in the transmitted polarization direction and transmittance of the polarizing element. In particular, when the range of incident angles is large or when a polarizing element including the variable phase difference plate mentioned above is used, the in-plane transmitted polarization direction and transmittance of the polarizing element become large. Of these, the in-plane distribution of transmitted polarization direction has a significant impact on the polarization information obtained from the captured image. Therefore, in this embodiment, the influence of the incident angle characteristics of the polarizing element on the polarization information obtained from the captured image is reduced by setting corrected polarization direction information according to the imaging conditions (information).
[0029] The corrected polarization direction information is set by storing corrected polarization direction information calculated in advance for multiple imaging conditions and multiple transmission polarization directions of the polarizing element as data, and then obtaining appropriate corrected polarization direction information from this data according to the imaging condition information acquired by the imaging condition acquisition unit 102. The method for calculating the corrected polarization direction information is described below. Assuming that the polarizing element is a polarizer with a transmission polarization direction θ, the Mueller matrix can be defined by the following equation (1).
[0030]
number
[0031] When the Stokes parameters of the light incident on this polarizing element are S0 to S3, and the Stokes parameters of the light emitted from the polarizing element are S0' to S3', then S0' to S3' are expressed by the following equation (2).
[0032]
number
[0033] Therefore, the measured light intensity (luminance) I is expressed by the following equation (3).
[0034]
number
[0035] Furthermore, the Stokes parameters for linearly polarized light with polarization directions of 0°, 45°, 90°, and 135° are expressed by the following equation (4).
[0036]
number
[0037] Therefore, the light intensities I0°, I45°, I90°, and I135° measured when linearly polarized light of 0°, 45°, 90°, and 135° is incident are expressed by the following equations (5) to (8).
[0038]
number
[0039] From equations (5) to (8), the polarization direction θ is expressed by the following equation (9).
[0040]
number
[0041] Equation (9) allows for the calculation of imaging conditions and corrected polarization direction information for the transmitted polarization direction of the polarizing element when imaging four linearly polarized signals (0°, 45°, 90°, 135°) from a subject using an imaging device.
[0042] Furthermore, the method for calculating corrected polarization direction information is not limited to the method described above. Alternatively, the polarization direction of linearly polarized light may be varied, and the polarization direction at which the measured light intensity is maximized may be used as the corrected polarization direction information. In addition, the corrected polarization direction information may be calculated from the theoretical incident angle characteristics of the polarizing element without actually performing imaging.
[0043] In this embodiment, imaging condition information, the transmitted polarization direction of the polarizing element, and corrected polarization direction information associated with the light ray incidence angle of the polarizing element are acquired and converted into data through prior measurement (imaging) or calculation, and this data is used to set the corrected polarization direction information during imaging. This eliminates the need to calculate the corrected polarization direction information each time imaging is performed. Furthermore, by converting the corrected polarization direction information into data associated with the imaging condition information, it becomes possible to set the corrected polarization direction information during imaging with a smaller amount of data than when data is stored for all combinations of optical systems and image sensors.
[0044] For example, when a polarizing element is placed between the subject and the optical system, the angle of incidence of light rays to the polarizing element can be roughly determined by the field of view calculated from the focal length of the optical system and the size of the image sensor. Also, when a polarizing element is placed between the optical system and the image sensor, the angle of incidence of light rays to the polarizing element can be roughly determined from the exit pupil position of the optical system and the size of the image sensor. Therefore, by performing imaging or calculations in advance under multiple imaging conditions with different fields of view or exit pupil positions and image sensor sizes, and digitizing the obtained corrected polarization direction information, it is possible to set appropriate corrected polarization direction information without having to calculate it during imaging.
[0045] To further improve the accuracy of the corrected polarization orientation information, it is preferable to include the optical system's focus state, F-number, imaging magnification, subject distance (subject position), and the amount of shift or tilt of the optical system relative to the image sensor as imaging conditions. The optical system's focus state refers to the position and control amount of the lens group that moves during focusing. In optical systems with high imaging magnification, the angle of incidence of light rays to the polarizing element changes significantly depending on the F-number and focus state, so it is desirable to include the F-number and focus state as imaging conditions. In addition, when using an optical system with a shift function or tilt function relative to the image sensor, it is preferable to consider the amount of shift or tilt.
[0046] The corrected polarization direction information data obtained in advance by measurement or calculation may be stored in a storage unit (storage means) provided within the polarization direction setting unit 103, or in a storage unit provided separately from the polarization direction setting unit 103. The storage unit can use common data storage media such as hard disks, flash memory, optical discs, or cloud storage.
[0047] The polarization information acquisition unit 104 calculates (acquires) polarization information from the captured image using the captured image acquired by the image acquisition unit 101 and the corrected polarization direction information set by the polarization direction setting unit 103 (S104). Specifically, for example, it extracts the brightness value (signal value) of a specific pixel from each of several captured images with different transmitted polarization directions, and plots the brightness value for each polarization direction to obtain the change in brightness in response to a change in polarization direction.
[0048] Figure 2(a) shows an example of plotting the luminance I1 to I4 of a specific pixel obtained from each of four different transmitted polarization orientations θ1 to θ4 for each polarization orientation. Figure 2(b) shows the result of fitting the change in luminance I with respect to polarization orientation θ from the plot results in Figure 2(a) using the following equation (10).
[0049] I(θ) = Acos 2 (θ―φ)+B (10) In equation (10), A represents the polarization component (luminance component that changes depending on the polarization direction) in a specific image, and B represents the unpolarized component (luminance component that is constant regardless of the polarization direction). φ represents the polarization direction at which polarization component A is at its maximum value.
[0050] As shown in Figure 2(a), the brightness changes cosinely with respect to the change in polarization direction. Therefore, by taking images at three or more different transmitted polarization directions of the polarizing element and plotting the brightness, the change in brightness with respect to the change in polarization direction can be obtained. In this embodiment, A, B, and φ in equation (10), which show the change in brightness with respect to such polarization direction (corrected polarization direction information), are calculated (obtained) as polarization information.
[0051] Furthermore, pre-processing may be performed on the captured images as needed before calculating the polarization information. For example, if there are multiple captured images, it is preferable to align them so that the same subject is located at the same relative position across the images. Also, if the captured images have undergone gamma processing, it is preferable to perform inverse gamma processing as pre-processing. Moreover, if the captured images have undergone image processing such as distortion correction, projection method conversion, or brightness correction, it is preferable to perform the reverse processing as pre-processing to eliminate the effects of the image processing.
[0052] The image generation unit 105 uses the polarization information A, B, and φ calculated by the polarization information acquisition unit 104 to create an arbitrary composite image (second image) (S105). Specifically, for example, the polarization component A and a specific polarization direction θ c Polarization component Acos 2 (θ c A composite image is generated by a linear combination of (-φ) and the unpolarized component B. The polarized component contains a large amount of specular reflection, and the unpolarized component contains a large amount of diffuse reflection. Therefore, by generating a composite image using the above linear combination, the ratio of specular reflection to diffuse reflection in the composite image can be arbitrarily adjusted.
[0053] Brightness I of each pixel in the composite image m It is calculated by the addition formula shown in equation (11) below.
[0054] I m =k1Acos 2 {k2(θ c -θ p )}+k3A+k4B (11) In Equation (11), k1 to k4 are arbitrary constants (hereinafter referred to as parameters).
[0055] Figure 3(a) shows an imaging device 200 equipped with the image processing device 100 shown in Figure 1. The imaging device 200 is composed of a polarizing element filter 10, a lens 20, and a camera body 30. The polarizing element filter 10 has a polarizing element, and the lens 20 has an optical system. Also, the camera body 30 includes an imaging element and the image processing device 100. The polarizing element filter 10 is detachable from the lens 20, and the lens 20 is detachable from the camera body 30.
[0056] The polarizing element included in the polarizing element filter 10 is composed of a λ / 4 plate 1, a variable retardation plate 2, a polarizing plate 3, and a λ / 4 plate 4 arranged in order from the subject side to the lens side as shown in Figure 3(a). The slow axes or transmission axes of these elements (1 to 4) are inclined with respect to each other so as to form a 45-degree angle between adjacent elements.
[0057] The variable retardation plate 2 is composed of VA liquid crystal. By changing the voltage applied to the variable retardation plate 2, the retardation imparted to the light beam transmitted through it can be controlled between 0 and a predetermined maximum value. The maximum value of the retardation Δmax is defined by the following equation (12).
[0058] Δmax = |no - ne|d (12) In Equation (11), no is the refractive index of the ordinary ray with respect to the variable retardation plate (liquid crystal layer) 2, ne is the refractive index of the extraordinary ray, and d is the thickness of the liquid crystal layer. Since both the refractive index and the thickness of the liquid crystal layer change depending on the light incident angle, the retardation imparted by the liquid crystal layer to the light beam has an incident angle characteristic, and the greater Δmax is, the greater the change in the retardation becomes.
[0059] Furthermore, the polarizing element filter 10 can variably set the transmitted polarization direction by changing the phase difference that the variable phase difference plate 2 imparts to the light rays. The transmitted polarization direction T (deg) changes according to the following equation (13) depending on the phase difference Δ (nm) imparted by the variable phase difference plate 2.
[0060] T = T0 ± 180·Δ / λ0 (13) In equation (12), T0 is the orientation (deg) of the transmission axis of polarizer 3, and λ0 is the wavelength (nm) of the incident light. Also, the ± in the same equation is + when the angle between the lagging axis of λ / 4 plate 1 and variable phase difference plate 2 is +45 degrees, and - when it is -45 degrees. Note that the angle between the lagging axes is +45 degrees when the lagging axis of variable phase difference plate 2 is located 45 degrees counterclockwise, and -45 degrees when it is located 45 degrees clockwise, with the lagging axis of λ / 4 plate 1 as the reference.
[0061] Figures 4(a) and 4(b) show the case where the slow phase axis of the λ / 4 plate 1 is positioned at 90 degrees to the x-axis. The z-axis is the optical axis of the optical system, and the direction from the subject side to the image sensor side is considered positive. Also, the angle is considered positive in the counterclockwise direction when viewed from the image sensor side to the subject side, with the x-axis being 0 degrees. As shown in Figure 4(a), when the slow phase axis of the variable phase difference plate 2 is at a position of 135 degrees to the x-axis, the angle between the slow phase axes of the λ / 4 plate 1 and the variable phase difference plate 2 is +45 degrees. As shown in Figure 4(b), when the slow phase axis of the variable phase difference plate 2 is at a position of 45 degrees to the x-axis, the angle between the slow phase axes of the λ / 4 plate 1 and the variable phase difference plate 2 is -45 degrees.
[0062] In this embodiment, three or more captured images generated by imaging with the polarizing elements in the polarizing element filter 10 having different transmitted polarization directions are acquired by the image acquisition unit 101.
[0063] Here, the larger Δmax is, the wider the range of transmission polarization direction T that can be set by equation (12). To improve the accuracy of obtaining the difference ΔT between the maximum and minimum values of T, considering the period in equation (10), it is preferable that ΔT be 90° or more with respect to the reference wavelength λ0, and more preferably 120° or more. Furthermore, considering the incident angle characteristics and period of the variable phase difference plate 2, it is preferable that ΔT be less than 180° with respect to λ0, and more preferably less than 150°. The reference wavelength λ0 can be arbitrarily set from within the imaging wavelength range, and can be set according to the purpose, for example, to the intermediate wavelength, minimum wavelength, or maximum wavelength of the imaging wavelength range.
[0064] Furthermore, since the polarizing element filter 10 is located between the subject and the optical system, as described above, the angle of incidence of light rays to the polarizing element filter 10 can be roughly determined by the field of view. Therefore, the imaging condition acquisition unit 102 of this embodiment acquires the focal length of the lens (optical system) 20 and the size of the image sensor in the camera body 30 (hereinafter referred to as the sensor size) from the header information of the captured image.
[0065] The polarization direction setting unit 103 calculates the field of view during imaging from the focal length and sensor size acquired by the imaging condition acquisition unit 102. The polarization direction setting unit 103 then selects corrected polarization direction information corresponding to the field of view during imaging and the voltage applied to the variable phase difference plate 2 from previously measured or calculated corrected polarization direction information data and sets it as the corrected polarization direction information for each captured image.
[0066] If the corrected polarization orientation data does not contain corrected polarization orientation information corresponding to the field of view at the time of imaging, two corrected polarization orientation information values with similar field of view are selected from the data, and the corrected polarization orientation information corresponding to the field of view at the time of imaging is calculated by interpolation of these two corrected polarization orientation information values. For example, if the field of view of the image calculated from the imaging condition information is 30° diagonally, and the field of view in the data is 12°, 23°, 34°, 47°, 63°, and 84°, then the corrected polarization orientation information for 23° and 34° is selected. Then, the corrected polarization orientation information for 30° is calculated by linear interpolation of these two corrected polarization orientation information values.
[0067] Furthermore, the polarization direction setting unit 103 may set a usage area in the captured image where corrected polarization direction information is used to acquire polarization information. In this case, the usage area may be set from the incident angle information to the polarizing element obtained from the imaging condition information.
[0068] Figure 5(a) shows the shaded area in the captured image where corrected polarization direction information is used to acquire polarization information, when the angle of incidence to the polarizing element is large. The angle of incidence to the polarizing element increases from the center to the periphery of the captured image. The polarization direction setting unit 103 calculates the angle of incidence of the light rays to the polarizing element from the imaging condition information and determines the area in which corrected polarization direction information is used.
[0069] Figure 5(b) shows the shaded area of the captured image where corrected polarization orientation information is used to acquire polarization information, when the incident angle to the polarizing element is small. In Figure 5(b), compared to Figure 5(a), corrected polarization orientation information is used in a narrower area near the center of the captured image. In this case, the area of use may be set from the aspect ratio (width-to-height ratio) of the captured image (image sensor) or the cropped area within it. For example, if the aspect ratio of the captured image is 2:3 and the aspect ratio of the cropped area is 1:1, the aspect ratio of the area of use should also be set to 1:1.
[0070] Furthermore, the polarization direction setting unit 103 may acquire the orientation of the optical axis (lagging axis) of the variable phase difference plate 2 and set the usable area based on the information regarding the orientation of that optical axis (hereinafter referred to as axis orientation information). Figure 6 shows the relationship between the orientation of the optical axis of the variable phase difference plate 2 and the usable area of the corrected polarization direction information. The polarization direction setting unit 103 acquires the rotation angle AR of the lagging axis of the variable phase difference plate 2 with respect to the x axis around the optical axis of the optical system (z axis in Figure 4) as axis orientation information, and sets a usable area (shaded area) that is tilted from the horizontal by an angle AR according to this information.
[0071] Figure 7(a) shows the state in which the optical system OS is shifted (translated) relative to the image sensor IP, and Figure 7(b) shows the usable area of the corrected polarization orientation information in that state. In Figure 7(a), OP is the subject plane, LA is the optical axis of the optical system OS, and IA is the normal vector drawn at the center of the image sensor IP (hereinafter referred to as the sensor center normal). The polarization orientation setting unit 103 acquires the shift amount LS of the optical system OS (the amount of displacement of the optical axis LA relative to the sensor center normal IA) as imaging condition information. Then, as shown in Figure 7(b), the usable area (shaded area) of the captured image is set by moving it in the shift direction by the shift amount LS' according to the shift amount LS and shift direction.
[0072] Figure 8(a) shows the state in which the optical system OS is tilted (moved inclined) relative to the image sensor IP, and Figure 8(b) shows the method for setting (calculating) the corrected polarization direction information in that state. In Figure 8(a), LPP is the main plane of the optical system OS. The extensions of the main plane LPP of the optical system OS, the image sensor IP, and the subject plane OP intersect at a single intersection line P according to Scheinproof's law. At this time, the angle of incidence of light rays to the polarizing element changes in the tilt direction according to the amount of tilt AT of the optical system OS relative to the image sensor IP.
[0073] As shown in Figure 8(b), the polarization direction setting unit 103 calculates corrected polarization direction information corresponding to the tilt amount using the corrected polarization direction information set for the captured image when the optical system OS is not tilted, and the amount of change in the transmitted polarization direction within the plane of the polarizing element according to the tilt amount (the shades in the figure indicate different amounts of change). At this time, the polarization direction setting unit 103 may calculate corrected polarization direction information corresponding to the tilt amount by adding or multiplying the corrected polarization direction information and the amount of change in the transmitted polarization direction, or it may obtain new corrected polarization direction information from the correlation between the corrected polarization direction information and the amount of change in the transmitted polarization direction.
[0074] Figure 9 shows how to set corrected polarization orientation information according to the manufacturing error of the polarizing element. Manufacturing errors include, for example, deviations from the assumed values of the thickness d of the liquid crystal layer of the variable phase difference plate 2, refractive index no and ne, and relative angular errors of the lagging axis of the elements (1-4) of the polarizing element. Due to such manufacturing errors, the transmitted polarization orientation of the polarizing element may differ from the assumed value. Therefore, the polarization orientation setting unit 103 calculates corrected polarization orientation information to compensate for the manufacturing error using the assumed value of the transmitted polarization orientation in the plane of the polarizing element shown in Figure 9 and the amount of change in the transmitted polarization orientation according to the manufacturing error in the plane of the polarizing element (the shades in the figure indicate different amounts of change).
[0075] Furthermore, from equation (13), the transmitted polarization direction of the polarizing element changes according to the wavelength of the incident light. For this reason, when the captured image has signal values of multiple wavelengths (or wavelength ranges), such as a color image, the corrected polarization direction information may be set for each wavelength corresponding to the Fraunhofer lines, such as the d line (587.6 nm), g line (435.8 nm), C line (656.3 nm), F line (486.1 nm), and e line (546.1 nm). Alternatively, the corrected polarization direction information for a specific wavelength may be set using a reference wavelength and a coefficient corresponding to the change in wavelength. In addition, the corrected polarization direction information may be set for each color channel corresponding to the image sensor.
[0076] Furthermore, the angle of incidence of light rays onto the polarizing element also changes depending on the pixel position of the image sensor. For this reason, it is preferable to set corrected polarization orientation information for each pixel or for each region containing multiple pixels.
[0077] Figure 10 shows the corrected polarization orientation information (deg) set for each pixel in the R, G, and B channels when a voltage is applied to the variable phase difference plate 2 with a diagonal field of view of 23°. The arrows in the figure indicate the orientation of the optical axis of the variable phase difference plate 2 projected onto the imaging plane of the image sensor (the plane perpendicular to the optical axis of the optical system). As can be seen from Figure 10, the corrected polarization orientation information is set to be distributed symmetrically with respect to the axis of symmetry corresponding to the optical axis of the variable phase difference plate 2 on the captured image.
[0078] The polarization direction setting unit 103 may acquire the orientation of the optical axis of the variable phase difference plate 2 and set corrected polarization direction information according to the orientation of the optical axis. In this case as well, the polarization direction setting unit 103 sets the corrected polarization direction information so as to be symmetric with respect to the axis of symmetry corresponding to the optical axis on the captured image. When the corrected polarization direction information is set symmetrically with respect to the axis of symmetry on the captured image in this way, the corrected polarization direction information can be set for the entire captured image by calculating the corrected polarization direction information for the image region on the axis of symmetry and on one side of the axis of symmetry. For this reason, the amount of calculation can be reduced by approximately half compared to when the corrected polarization direction information for the entire captured image is calculated. In addition, the amount of data for corrected polarization direction information that needs to be digitized in advance for the calculation of corrected polarization direction information can also be reduced by approximately half.
[0079] The polarization information acquisition unit 104 acquires polarization information for each pixel and each color channel from the corrected polarization orientation information set as described above and the captured image. The image generation unit 105 then generates a composite image using the acquired polarization information as described above.
[0080] In addition to the functions described above, the imaging condition acquisition unit 102 may acquire information about the lens 20 (hereinafter referred to as lens information) from the header information of the captured image, and acquire additional imaging condition information according to the acquired lens information. The lens information includes information indicating the lens manufacturer, lens name, and lens characteristics to identify the lens used for imaging. If the lens identified from the lens information is a lens with a high imaging magnification (for example, a macro lens), it is preferable to acquire the focus state and F number as additional imaging condition information and to consider this information when setting the corrected polarization direction information. Furthermore, if the lens identified from the lens information is a lens with a shift function or tilt function, it is preferable to acquire the amount of shift or tilt of the lens relative to the image sensor as additional imaging condition information and to consider this information when setting the corrected polarization direction information.
[0081] Alternatively, optical information such as the optical arrangement of the lens, the exit pupil position, and the imaging magnification may be stored as imaging condition information in a storage device (such as a hard disk in the image processing device 100 or the cloud) in association with the lens information, and this information may be read from the storage device according to the lens information. [Examples]
[0082] The image processing device of Example 2 calculates polarization information from the captured image generated by the imaging device 300 shown in Figure 12 and generates a composite image. The imaging device 300 consists of a polarizing element adapter 11, a lens 21, and a camera body 31. The polarizing element adapter 11 is equipped with a polarizing element whose transmitted polarization direction is variable. The lens 21 and camera body 31 are the same as the lens 20 and camera body 30 of Example 1. Furthermore, the configuration of the image processing device built into the camera body 30 is also the same as the image processing device 100 of Example 1.
[0083] The polarizing element adapter 11 is positioned between the lens 21 and the camera body 31 and is detachable from them. Because the polarizing element adapter 11 is located between the lens (optical system) 21 and the camera body (image sensor) 31, the angle of incidence of light rays to the polarizing element adapter 11 can be roughly determined by the exit pupil position of the optical system and the sensor size. For this reason, in this embodiment as well, the imaging condition acquisition unit 102 of the image processing device acquires information on the exit pupil position and sensor size as imaging condition information from the header information of the captured image. Other processing performed by the image processing device is the same as in Embodiment 1.
[0084] In addition, in the imaging device 300, the polarizing element adapter 11 may be provided integrally with the lens 21, and the polarizing element may be replaceable. (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0085] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. (Reference example 1) Figure 13 shows the configuration of the image processing device 400 in Reference Example 1. The image processing device 400 includes an image acquisition unit 411, a calculation unit 412, and a display unit 413.
[0086] The image acquisition unit 411 acquires polarized images from a storage medium connected to the image processing device 400 or from a folder on a network such as the cloud via the internet. Polarized images are captured images obtained by imaging under multiple polarization states. For example, this includes multiple captured images obtained by imaging with different transmission polarization directions of polarizing plates, or images obtained by imaging using a polarization sensor. Preferably, the acquired image is a so-called RAW image, and if the image has undergone brightness correction such as gamma correction relative to the input brightness, it is preferable to convert it back to the brightness value before correction by inverse conversion.
[0087] The calculation unit 412 calculates the gloss component (specular reflection component) and base component (diffuse reflection) of the subject from the polarized image acquired by the image acquisition unit 411, and creates a corrected gloss component by applying a gradient that changes the brightness in a specific direction to the obtained gloss component. Furthermore, by combining the obtained corrected gloss component and the base component, a composite image different from the acquired polarized image is generated. In this reference example, the gloss component and base component are separated, and then the gradient is applied only to the gloss component. This allows the ratio of the gloss component to the base component to be modulated after imaging, and the appearance of the subject can be adjusted after imaging.
[0088] The display unit 413 displays the composite image generated by the calculation unit 412. The display unit 413 can be any device capable of displaying images, such as a liquid crystal display or an organic EL display.
[0089] Next, the image generation process performed by the calculation unit 412 will be described. Figure 14 shows the basic process. As shown in Figure 13, the calculation unit 412 includes a polarization information acquisition unit (polarization information processing unit) 421, a correction processing unit 422, and an image generation unit (image synthesis unit) 423. The polarization information acquisition unit 421 extracts the luminance component I that changes according to the change in polarization state from the polarization image (S201) acquired by imaging in multiple polarization states. s and the luminance component I, which is independent of the change in polarization state. b The data is acquired for each pixel and each color channel (S202).
[0090] Specifically, I s and I b Calculate.
[0091] The change in brightness I(α) with respect to the transmitted polarization direction α is expressed by the following equation (21).
[0092] I(α)=I s ·cos 2 (α―θ)+I b (twenty one) Also, the maximum brightness value I(α) MAX , minimum brightness value I min Using I(α)=(I MAX -I min )cos 2 (α-θ)+I min (twenty two) It can also be expressed as follows. Here, θ in equations (21) and (22) is the maximum value I(α) MAX This is the polarization direction when that occurs.
[0093] If the polarized image is captured in three or more different polarization states, the luminance component I in each pixel can be calculated using equations (21) or (22). s , I b Ya I MAX ,I min θ can be calculated. Note that if there are three polarization images, one solution is determined, but if there are four or more, a unique solution cannot be determined, so the optimal solution should be calculated using the least squares method or similar.
[0094] On the other hand, if the polarized image is captured in two different polarization states, the luminance difference between the two polarization states is I s , the luminance component that remains unchanged between the two polarization states is I b That's all you need to do.
[0095] In this reference example, the polarization information acquisition unit 421 may appropriately use the two methods described above depending on the type of polarization image and the processing capabilities of the processing device, or it may select the processing method via external input.
[0096] Next, the polarization information acquisition unit 421 uses the two brightness values I s ,I b Using this, gloss component A and base component B are generated (S203). In this reference example, the luminance value I s gloss components A, I b Let this be the base component B.
[0097] Note that the gloss component and base component in this example are not limited to those described above and can be arbitrarily set using the polarization information obtained as described above. For example, the gloss component could be a constant multiple C of the luminance value Is. s I s (C s (where α is an arbitrary constant) or a specific polarization direction α c Polarization component C s 'I s cos 2 (α c ―θ)(C s You may also select any constant (where ' is an arbitrary constant). Additionally, multiple polarization directions α are available. c1 , α c2 ...linear sum of polarization components I s cos 2 (α C1 -θ)+I s cos 2 (α C2 You may also choose -θ)+…. Alternatively, you may choose any combination of these, and you can select as appropriate depending on the application.
[0098] Next, the correction processing unit 422 generates a corrected gloss component with a gradient applied to the gloss component by multiplying the gloss component for each pixel and each color channel by a different correction coefficient depending on the pixel position (S204). The correction coefficient used at this time is set to increase or decrease in any direction of the image (hereinafter referred to as the u-axis direction). It is also set to be a constant value in the direction perpendicular to the u-axis direction (hereinafter referred to as the v-axis direction). In this way, by multiplying the gloss component by a different correction coefficient depending on the pixel position, a corrected gloss component can be generated in which the gloss component increases or decreases in the u-axis direction (hereinafter referred to as gradient-processed).
[0099] Figures 15(a) to 15(c) show the axis directions and correction coefficients for this example. Figure 15(a) shows the axis direction relative to the image, Figure 15(b) shows the correction coefficient k in the u-axis direction, and Figure 15(c) shows the correction coefficient k in the v-axis direction. Note that the horizontal direction of the image is defined as the x-axis direction, any direction indicating the gradient direction as described above is defined as the u-axis direction, and the direction perpendicular to the u-axis is defined as the v-axis direction.
[0100] As shown in Figure 15(c), the correction coefficient k takes a constant value in the v-axis direction. On the other hand, as shown in Figure 15(b), the correction coefficient k decreases in the u-axis direction. Note that the decrease in the u-axis direction means that u c1 c2 Any u that satisfies c1 u c2 In contrast, k uc2 ≤k uc1 (twenty three) The following conditions must be met. On the other hand, although this is the opposite of the example, increasing in the u-axis direction means that u c1 c2 Any u that satisfies c1 u c2 In contrast, k uc2 ≧k uc1 (twenty four) The following conditions must be met.
[0101] In this example, the correction coefficient k can be arbitrarily set within the range that satisfies equation (23) or equation (24), and may be monotonically increasing or monotonically decreasing with respect to u. However, in order to obtain an image with higher texture, it is preferable to have a region (boundary region) in which the amount of change is larger than in other regions.
[0102] In Figure 15(b), u b1 from u b2 Between other regions (u0 to u b1 between or u b2 The coefficient change is larger compared to the range between u1. By having such a boundary region, it becomes possible to perform a sharp gradient processing on the gloss component, and a composite image with high texture can be obtained. In addition, a region in the image where the correction coefficient k is constant may be included.
[0103] Figures 16(a) and (b) show other examples of the correction coefficient k in the u-axis direction. Figure 16(a) shows an example where the boundary region is a single point (without width) on the u-axis, and Figure 16(b) shows an example where the correction coefficient k changes curvilinearly with respect to the u-axis.
[0104] Furthermore, it is preferable that the boundary region can be arbitrarily set, and that it be possible to input an external value in order to generate an image that reflects the user's intent. The external input of the boundary region is, for example, the position B(u) of the boundary region. b ,v b ) and width Δu b =u b2 -u b1 This can be done by specifying it.
[0105] Next, using the generated correction coefficient k, a corrected gloss component A' is generated for each pixel and each color channel from the following equation (5).
[0106] A'=kA (25) Furthermore, the generated corrected gloss component A' is combined with image I' from the base image B (S205). The simplest way to generate it is by the following equation (26).
[0107] I' = A' + B (26) Alternatively, equation (27), which uses arbitrary constants C1 and C2, may be used instead of equation (4).
[0108] I' = C1A' + C2B (27) Finally, the generated image I' is output to the display unit 413 (S206).
[0109] In the above description, the process from acquiring the polarization image to outputting it was explained as a single series of operations, as shown in Figure 14. However, some of these operations may be repeated. For example, after processing S201 to S206, the process from S204 to S206 may be repeated while changing the position and width of the boundary region, the value of the correction coefficient, and the amount of change therein, in order to create an image with higher quality. In addition, the image and coefficient information being created in each step may be output as an image as appropriate.
[0110] Furthermore, the aforementioned gradient application can be used in combination with other image processing techniques (noise reduction, white balance adjustment, contrast enhancement, etc.) as needed. In this case, the order of image processing other than gradient application is not particularly limited. However, performing these techniques before combining the corrected gloss component and the base component allows for independent application of optimal processing to each component. (Reference example 2) Using Figure 17, the processes performed by the image processing apparatus in Reference Example 2 will be explained. In Reference Example 1, corrected gloss components were generated for the entire image, but in Reference Example 2, a portion of the image is selected, and corrected gloss components are generated only within the selected area (S301). In addition, a mask is created to specify the area (S302). The processes and apparatus configuration other than the use of the mask are the same as in Reference Example 1, so the mask part (S301, S302) will be explained below.
[0111] The method for creating a mask in S301 is not particularly limited, but methods such as specifying an arbitrary region in the image using external input, or extracting a specific region using polarization information can be used. Alternatively, a method combining external input and the use of polarization information may be used to specify a region.
[0112] One method using polarization information is to utilize the polarization information A, B, and θ calculated in S202. In this method, a mask can be created by specifying a region where the polarization direction θ falls within a specific range, or by specifying a region where A or B is above a threshold. In this reference example, the polarization degree DOP, defined by the following equation (28), is calculated from the polarization information, and the polarization degree DOP is above the threshold V. th Extract the higher region and use it as a mask.
[0113] DOP=(I max -I min ) / (I max +I min ) =A / (2A+B) (28) Note that threshold V th This can be selected appropriately depending on the image. However, in order to obtain the texture enhancement effect of gradients, the threshold V th It is preferable that this value be above a certain level, and specifically, it is preferable to set the polarization degree of each pixel to a value greater than the average value obtained across the entire image.
[0114] Furthermore, if the image is in color, it is preferable to use a common mask for all colors rather than applying different masks to each color. For example, the results of region extraction for a specific color can be applied to all colors, or region extraction can be performed for each color, and then a region that completely covers all extracted regions can be calculated and applied to all colors.
[0115] Using the mask generated by the method described above, a corrected gloss component is generated from the gloss component (S302). Specifically, the corrected gloss component is generated by multiplying the gloss component within the mask region by a correction coefficient k and applying a gradient to the gloss component within the mask region. Here, the method for creating the correction coefficient k is the same as in Reference Example 1. Note that no gradient is applied to the gloss component outside the mask region. Furthermore, the corrected gloss component may use both the gloss component with a gradient applied within the mask region and the gloss component without a gradient applied outside the mask region, or it may use only the component within the mask region. Finally, an image is generated based on the generated corrected gloss image and the base image (S205). The specific procedure is the same as in S205 of Reference Example 1.
[0116] In this example, by using a mask to apply a gradient only to the glossy components of a specific area, it is possible to create a sharp gradient effect and obtain a composite image with high texture. (Reference example 3) Reference Example 3 is an image processing system that includes the image processing apparatus of the present invention and performs image acquisition and image processing. In addition to the image processing apparatus, the image processing system in this reference example includes an imaging device and an illumination device.
[0117] The imaging device is a device for capturing images in different polarization states, and comprises a lens, a camera (image sensor) with interchangeable lenses, and a polarization means. Here, the polarization means is placed between the subject and the imaging device and consists of a polarizing plate or an element using liquid crystal as disclosed in Patent Document 1. Alternatively, a so-called polarization sensor may be used, in which polarizers are placed in front of each pixel of the image sensor at different axial orientations.
[0118] Ideally, the lighting device should be able to illuminate the subject evenly from all directions, and at least be able to irradiate the subject from two or more directions. By illuminating the entire subject using such a lighting device, the gloss component of the subject can be obtained regardless of its surface normal. Means of achieving the above-mentioned lighting include, for example, using multiple lighting devices to irradiate light from multiple directions, or using a lighting device called a lightbox. There are also methods of using ring lights, strobes, diffusers for diffusing light, etc., individually or in combination.
[0119] An image processing device can, for example, use a monitor and a PC, tablet, or smartphone. The image processing device can perform image processing internally using software or applications installed on it. It may also connect to the internet and perform some or all of the image processing on the cloud, or save images to the cloud. Furthermore, the control unit included in the image processing device may control not only the image processing device itself, but also the imaging device and lighting device. In this case, the control unit, imaging device, and lighting device can be connected via wired or wireless connections. [Explanation of symbols]
[0120] 1 λ / 4 plate 2 Variable phase difference plate 3. Polarizing plate 4 λ / 4 plate 100 Image Processing Devices
Claims
1. a first acquisition unit that acquires a first image generated by capturing an image of a subject via a polarizing element that transmits light having a plurality of different polarization orientations; A second acquisition unit that acquires information regarding an imaging condition in the imaging when the first image was acquired; A setting unit that sets corrected polarization orientation information based on information about the imaging conditions and information about the polarization orientation of the polarizing element in the imaging when the first image is acquired; and a third acquisition unit that acquires polarization information by using the first image and the corrected polarization orientation information.
2. The image processing apparatus according to claim 1 , wherein the corrected polarization orientation information is information about a plurality of polarization orientations set for each pixel or region of the first image.
3. 3. The image processing apparatus according to claim 1, wherein the corrected polarization direction information is information regarding the polarization directions of a plurality of light beams having different wavelengths.
4. 4. The image processing apparatus according to claim 1, wherein the information regarding the imaging conditions includes information regarding an angle of incidence of a light beam onto the polarizing element.
5. 5. The image processing device according to claim 1, wherein, when an optical system and an image sensor are used for the image capture, the information on the image capture conditions includes information on a focal length of the optical system and a size of the image sensor.
6. 5. The image processing device according to claim 1, wherein, when an optical system and an image sensor are used for the image capture, the information on the image capture conditions includes information on an exit pupil position of the optical system and a size of the image sensor.
7. 7. The image processing device according to claim 1, wherein, when an optical system is used for capturing the image, the information on the image capturing conditions includes information on a focus state of the optical system or information on a position of the subject.
8. 8. The image processing device according to claim 1, wherein, when a variable retardation plate and a polarizing plate are used in the polarizing element, the setting unit sets the corrected polarization orientation information using information about an orientation of an optical axis of the variable retardation plate.
9. The image processing device according to claim 8 , wherein the setting unit sets the corrected polarization orientation information so as to be symmetrical with respect to an axis of symmetry extending in the direction of an optical axis of the variable retardation plate.
10. Further, a storage means for storing information regarding the polarization orientation is provided. The image processing device according to claim 1 , wherein the setting unit sets information about the polarization orientation corresponding to information about the image capturing conditions from among the stored corrected polarization orientation information.
11. The image processing device according to claim 1 , wherein the setting unit sets a usage area of the first image in which the polarization information is obtained using the corrected polarization orientation information, in accordance with information regarding the imaging conditions.
12. The imaging device according to claim 11, characterized in that, when the optical system is displaceable in at least one of a shift direction and a tilt direction relative to the imaging element, the setting unit sets the usage area according to an amount of displacement of the optical system relative to the imaging element acquired as information regarding the imaging conditions.
13. The image processing device according to claim 1 , wherein the third acquisition unit acquires the polarization information from a change in a signal value in a plurality of polarization orientations acquired from the first image.
14. 14. The image processing apparatus according to claim 1, further comprising an image generating unit for generating a second image using the polarization information.
15. An image processing device as described in any one of claims 1 to 14, characterized in that the setting unit sets the corrected polarization orientation information based on information regarding the polarization orientation of the polarizing element in the imaging when the first image was acquired and the incident angle characteristics of the polarizing element according to information regarding the imaging conditions.
16. The image processing device described in Claim 15, characterized in that the corrected polarization orientation information is obtained by correcting information regarding the polarization orientation of the polarizing element in the imaging when the first image is acquired.
17. An image processing device according to any one of claims 1 to 16, and an imaging element for performing the imaging.
18. acquiring a first image generated by capturing an image of a subject via a polarizing element that transmits light having a plurality of different polarization orientations; acquiring information about imaging conditions in the imaging when the first image was acquired; setting corrected polarization orientation information based on information about the imaging conditions and information about the polarization orientation of the polarizing element in the imaging when the first image was acquired; and acquiring polarization information using the first image and the corrected polarization orientation information.
19. A program for causing a computer to execute a process according to the image processing method of claim 18.
