Image processing device and image processing method

Abstract

The present disclosure relates to an image processing device and an image processing method that make it possible to heighten the resolution of low-resolution depth images. This image processing device comprises: a guide feature amount conversion unit that sets a guide feature amount using a linear parameter calculated from polarization information pertaining to a subject; and a depth estimation unit that generates, from depth information pertaining to the subject and the guide feature amount set by the guide feature amount conversion unit, high-resolution depth information having a higher resolution than the depth information. The technology according to the present disclosure can be applied to, e.g., an image processing device that heightens the resolution of a depth image.

Description

Image processing device and image processing method

[0001] The present disclosure relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method that are capable of increasing the resolution of a low-resolution depth image.

[0002] Time-of-flight depth sensors, which measure the distance to an object based on the time of flight of light, are becoming increasingly popular. Depth sensors can acquire high-resolution depth values ​​in the distance direction (depth direction) toward the subject, but their resolution in the planar direction (X and Y directions) is lower than that of a regular RGB camera. On the other hand, polarization sensors have high resolution in the planar direction and can measure features that cannot be captured by a regular RGB camera. For example, in a scene with a white box on a white desk, a polarization sensor can acquire more information than an RGB camera.

[0003] Patent Literature 1 discloses a technique for increasing the resolution of a depth image having a low resolution in the planar direction by using information acquired by a polarization sensor. Specifically, the technique discloses a technique for calculating normal information of a subject from polarization information obtained from the polarization sensor, and integrating the low-resolution depth information and the normal information to generate a high-resolution depth image.

[0004] International Publication No. 2018 / 061508

[0005] However, because the signal-to-noise ratio of a polarization sensor's pixel is 1 / √2 of that of a normal RGB camera's pixel, polarization sensors are more susceptible to noise than RGB cameras. The normal information of a subject can be expressed using the zenith angle θ and the azimuth angle φ, but as the degree of polarization decreases, the error and variance of the azimuth angle φ increase, degrading robustness to noise. Therefore, it is desirable to use noise-robust polarization features to enhance the resolution of low-resolution depth images.

[0006] The present disclosure has been made in consideration of such circumstances, and makes it possible to increase the resolution of a low-resolution depth image.

[0007] An image processing device according to one aspect of the present disclosure includes: a guide feature conversion unit that sets a guide feature using linear parameters calculated from polarization information of a subject; and a depth estimation unit that generates high-resolution depth information having a higher resolution than the depth information from the depth information of the subject and the guide feature set by the guide feature conversion unit.

[0008] An image processing method according to one aspect of the present disclosure sets guide features using linear parameters calculated from polarization information of a subject, and generates high-resolution depth information having a higher resolution than the depth information from the depth information of the subject and the set guide features.

[0009] In one aspect of the present disclosure, a guide feature is set using linear parameters calculated from polarization information of a subject, and high-resolution depth information having a higher resolution than the depth information is generated from the depth information of the subject and the set guide feature.

[0010] The image processing device according to one aspect of the present disclosure can be realized by causing a computer to execute a program. The program executed by the computer to realize the image processing device can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0011] The image processing device may be an independent device or an internal block constituting a single device.

[0012] 12. It is a block diagram showing a configuration example of an image processing device according to an embodiment of the present disclosure. It is a diagram explaining a generation operation of a polarization image. It is a diagram explaining the relationship between brightness I and polarization angle ν. It is a diagram explaining the relationship between the degree of polarization ρ and zenith angle θ. It is a block diagram showing a first configuration example of a guide feature quantity conversion unit. It is a diagram showing an example of demosaicing by a demosaicing processing unit. It is a diagram explaining normalization of Stokes parameters. It is a flowchart explaining high-resolution depth image generation processing using the first configuration example. It is a block diagram showing a second configuration example of a guide feature quantity conversion unit. It is a diagram explaining 180-degree ambiguity. It is a flowchart explaining high-resolution depth image generation processing using the second configuration example. It is a block diagram of a first application example to which the image processing device of FIG. 1 is applied. It is a diagram explaining a part of the configuration example of the polarization / depth sensor of FIG. 12. It is a block diagram of a second application example to which the image processing device of FIG. 1 is applied. It is a block diagram showing a configuration example of a computer as the image processing device of FIG. 1. It is a block diagram of a third application example to which the image processing device of FIG. 1 is applied.

[0013] Hereinafter, with reference to the accompanying drawings, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment). Note that in this specification and the drawings, components having substantially the same functional configuration are assigned the same reference numerals, and redundant description will be omitted. The description will be given in the following order: 1. Configuration example of an image processing device 2. Characteristics of polarization images and problems with normal information 3. First configuration example of a guide feature conversion unit 4. High-resolution depth image generation processing using the first configuration example 5. Second configuration example of a guide feature conversion unit 6. High-resolution depth image generation processing using the second configuration example 7. Summary 8. First application example of an image processing device 9. Second application example of an image processing device 10. Third application example of an image processing device

[0014] 1. Configuration Example of Image Processing Apparatus> FIG. 1 is a block diagram showing a configuration example of an image processing apparatus according to an embodiment of the present disclosure.

[0015] The image processing device 1 shown in Figure 1 is an image processing device that acquires polarization information and depth information, increases the resolution of a depth image with low resolution in the planar direction, generates a high-resolution depth image, and outputs it as high-resolution depth information.

[0016] The image processing device 1 includes a polarization information acquisition unit 11, a depth information acquisition unit 12, a guide feature amount conversion unit 13, and a high-precision depth estimation unit 14.

[0017] The polarization information acquisition unit 11 acquires a polarization RAW image, which is polarization information, and supplies it to the guide feature conversion unit 13. The depth information acquisition unit 12 acquires a depth image, which is depth information, and supplies it to the high-precision depth estimation unit 14. The resolution (number of pixels) in the planar direction of the depth image acquired by the depth information acquisition unit 12 as depth information is lower than the resolution (number of pixels) of the polarization RAW image. The polarization RAW image acquired by the polarization information acquisition unit 11 and the depth image acquired by the depth information acquisition unit 12 may be images generated by the same sensor (camera), or may be images generated separately by different sensors. However, the polarization RAW image and the depth image are images captured of the same subject.

[0018] The method by which a sensor (depth sensor) that generates depth information measures the distance to a subject is not particularly limited, and a sensor (camera) using any measurement method can be used. Distance measurement methods include, for example, the Time-of-Flight (ToF) method, which irradiates a subject with active light and measures the time of flight until the light returns; the Structured Light method or stereo camera method, which uses the principle of triangulation; and focusing methods such as white light interferometry and focusing. ToF methods include direct ToF and indirect ToF. The indirect ToF method calculates the distance to an object by detecting the time of flight from the time when irradiated light is emitted to the time when reflected light is received as a phase difference, and can achieve high-precision measurements over relatively short distances. On the other hand, the direct ToF method directly measures the time of flight from the time when irradiated light is emitted to the time when reflected light is received to calculate the distance to an object. Compared to the indirect ToF method, this method is more effective for measuring long distances.

[0019] The guide feature conversion unit 13 sets the guide feature by converting the polarization RAW image supplied as polarization information from the polarization information acquisition unit 11 into a guide feature and supplying it to the high-precision depth estimation unit 14. The guide feature is a feature that serves as a guide for increasing the resolution of the depth image in the high-precision depth estimation unit 14. The guide feature conversion unit 13 calculates linear parameters from the polarization information of the subject, and outputs the guide feature using the linear parameters to the high-precision depth estimation unit 14.

[0020] The high-precision depth estimation unit 14 uses the low-resolution depth image supplied from the depth information acquisition unit 12 and the guide features supplied from the guide feature conversion unit 13 to generate a high-resolution depth image whose resolution in the planar direction is equivalent to that of the polarized RAW image, and outputs it as high-resolution depth information.

[0021] The high-precision depth estimation unit 14 generates depth information of a subject that is more accurate than the acquired depth information from the depth information of the subject and guide features. The high-precision depth estimation unit 14 has a machine learning model that generates a high-resolution depth image, inputs a low-resolution depth image and guide features into the machine learning model, and outputs a high-resolution depth image generated by the machine learning model. In the machine learning model, parameters (model parameters) are generated in advance by supervised learning in which correct labels are provided and learned, and the parameters are stored in the high-precision depth estimation unit 14. The machine learning model is, for example, configured as a CNN (convolutional neural network) and includes a feature extraction unit and a depth estimation unit. The feature extraction unit includes, for example, multiple stages of convolution layers and pooling layers, and the depth estimation unit includes, for example, multiple stages of deconvolution layers and depooling layers. In the learning process for learning the model parameters, a large number of sets of guide features, low-resolution depth images, and high-resolution depth images as correct labels are prepared, and the model parameters are learned using an error backpropagation method or the like. In the backpropagation method, a loss function based on the error between the value of the output layer of the neural network and the correct label is defined, and model parameters are determined using a steepest descent method or the like to minimize the loss function.

[0022] As described above, the image processing device 1 acquires polarization information and depth information, converts the acquired polarization information into guide features, and generates and outputs a high-resolution depth image from the low-resolution depth information and guide features. The polarization information has high resolution in the planar direction and has characteristics that cannot be acquired with a normal RGB camera. The guide features are features different from normal information and are polarization features that are robust to noise. This makes it possible to output a high-resolution depth image as high-resolution depth information, which has high accuracy in the depth direction distance and high resolution in the planar direction compared to when normal information is used as a guide feature.

[0023] 2. Characteristics of Polarized Images and Issues with Normal Vector Information Before describing the guide feature amount employed by the image processing device 1, characteristics of polarized images and issues with normal vector information will be described.

[0024] FIG. 2 is a diagram illustrating the operation of generating a polarization image.

[0025] The camera 21 captures an image of the subject 20 through a polarizing plate 22 disposed in front of it and generates a polarized image. It is known that the brightness of the subject 20 changes in the polarized image generated by the camera 21 as the polarizing plate 22 rotates. Here, the highest brightness when the polarizing plate 22 is rotated is defined as Imax, and the lowest brightness is defined as Imin. Furthermore, if the x-axis and y-axis in a two-dimensional coordinate system are the planar directions of the polarizing plate 22, the angle of the polarization direction on the x-y plane relative to the x-axis when the polarizing plate 22 is rotated is defined as the polarization angle υ. The normal to the object surface of the subject 20 can be expressed by the azimuth angle φ and zenith angle θ in a polar coordinate system, and these azimuth angle φ and zenith angle θ are defined as normal information. The zenith angle θ is the angle between the normal and the optical axis (Z-axis) of the camera 21; the smaller the zenith angle θ, the more directly the object faces the camera 21. The azimuth angle φ represents the angle of the Y-axis relative to the reference axis (X-axis) when the normal is projected onto the image plane of the camera 21.

[0026] When the polarizing plate 22 is rotated 180 degrees, it returns to its original polarization state and has a period of 180 degrees. In addition, in the case of a diffuse reflection model, the polarization angle υ at which the maximum luminance Imax is observed is defined as the azimuth angle φ. With this definition, the luminance I observed when the polarizing plate 22 is rotated can be expressed as in equation (1).

[0027]

[0028] FIG. 3 illustrates the relationship between the observed luminance I and the polarization angle υ. The relationship between the luminance I and the polarization angle υ in FIG. 3 can be modeled using a cosine function. In equation (1), the polarization angle υ of the polarizing plate 22 and the luminance I observed at that time are known, while the maximum luminance Imax, the minimum luminance Imin, and the azimuth angle φ are unknown. Since there are three variables, Imax, Imin, and φ, the luminance of the polarized image in at least three or more polarization directions can be acquired and fitted to the cosine function of equation (1) above, thereby calculating the maximum luminance Imax, the minimum luminance Imin, and the azimuth angle φ at which the maximum luminance Imax is obtained.

[0029] The degree of polarization ρ, which indicates the degree of polarization, can be calculated using the minimum luminance Imin and the maximum luminance Imax according to equation (2), where ρ=1 indicates completely polarized light (linearly polarized light) and ρ=0 indicates non-polarized light.

[0030]

[0031] The relationship between the degree of polarization ρ and the zenith angle θ is known from Fresnel's equation to have the characteristics shown in Fig. 4, for example, and the zenith angle θ can be calculated based on the degree of polarization ρ from the characteristics shown in Fig. 4. Note that the characteristics shown in Fig. 4 are merely examples, and the characteristics change depending on the refractive index of the subject.

[0032] Therefore, by using a polarized image with three or more polarization directions, the azimuth angle φ can be calculated from the polarization angle υ and the brightness I of the polarized image. The degree of polarization ρ can be calculated using the maximum brightness Imax and minimum brightness Imin obtained from the relationship between the brightness I and the polarization angle υ, and the zenith angle θ corresponding to the calculated degree of polarization ρ can be calculated based on a characteristic curve that shows the relationship between the degree of polarization ρ and the zenith angle θ. In this way, the azimuth angle φ and zenith angle θ, which are normal information of the subject, can be obtained for each pixel position based on a polarized image with three or more polarization directions.

[0033] The normal to the object surface (polarization normal) calculated from the polarized image can be expressed by the following equation (3) using the azimuth angle φ and the zenith angle θ: Polarization normal = (sin θ cos φ, sin θ sin φ, cos θ) (3)

[0034] The polarization normal is expressed using the azimuth angle φ and the zenith angle θ as in Equation (3). However, when the degree of polarization ρ is small, the error in the azimuth angle φ tends to increase. For example, when the degree of polarization ρ is small, the amplitude of the cosine function showing the relationship between the luminance I and the polarization angle υ in Figure 3 decreases, which tends to increase the error in the azimuth angle φ due to noise. Furthermore, in the diffuse reflection model, the relationship between the degree of polarization ρ and the zenith angle θ has the characteristics shown in Figure 4. Therefore, when the degree of polarization ρ is small, the zenith angle θ approaches 0, and the variance of the azimuth angle φ increases near zenith angle θ = 0. Therefore, as the degree of polarization ρ decreases, the error and variance of the azimuth angle φ increase, reducing robustness to noise. Furthermore, because the polarization normal is a nonlinear transformation of the information obtained from the polarization image, it is inherently vulnerable to noise.

[0035] If the number of electrons received by a pixel in a normal RGB camera is Q, the amount of noise is √Q, and the S / N ratio is √Q, then the number of electrons in a polarization pixel is Q / 2, the amount of noise is √(Q / 2), and the S / N ratio is √(Q / 2). Therefore, the S / N ratio of the pixel in a polarization sensor is 1 / √2 of that of an RGB camera, and polarization images are more susceptible to noise than RGB images.

[0036] The guide feature conversion unit 13 of the image processing device 1 described below converts normal line information into a guide feature that is robust to noise, unlike normal line information. This allows the image processing device 1 to generate a high-resolution depth image with higher distance accuracy in the depth direction and higher resolution in the planar direction than when normal line information is used as a guide feature.

[0037] 3. First Configuration Example of Guide Feature Transformation Unit FIG. 5 is a block diagram showing a first configuration example of the guide feature transformation unit 13. As shown in FIG.

[0038] The guide feature quantity transforming unit 13 in FIG. 5 includes a demosaic processing unit 41 and a linear parameter calculating unit 42 .

[0039] The demosaic processing unit 41 performs demosaic processing on the polarization RAW image supplied from the polarization information acquisition unit 11, and supplies the polarization image after demosaic processing to the linear parameter calculation unit 42. The polarization RAW image supplied from the polarization information acquisition unit 11 includes, for example, brightness I ０ , I ４５ , I ９０ , I １３５ are images arranged in a mosaic pattern such as a Bayer array or a quad-Bayer array, and the demosaic processing unit 41 generates 4-channel polarization images of 0 degrees, 45 degrees, 90 degrees, and 135 degrees by demosaic processing, and supplies them to the linear parameter calculation unit 42.

[0040] FIG. 6 shows an example of demosaic processing by the demosaic processing unit 41.

[0041] Demosaicing a 1-channel polarized RAW image in which the polarization angles (polarization directions) detected by the pixels are arranged in a Bayer or quad-Bayer array generates 4-channel polarized images of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. A Bayer array is an array in which polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees are arranged in 2 x 2 four-pixel arrays that form repeating units in the row and column directions, and a quad-Bayer array is an array in which the arrangement unit for one polarization direction is 2 x 2 four-pixel arrays, and polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees are arranged in a Bayer array in 2 x 2 four-pixel arrays.

[0042] Details of the demosaic processing are disclosed in, for example, Japanese Patent No. 6750633 and Japanese Patent No. 7070423. The demosaic processing unit 41 can perform the demosaic processing by employing the techniques disclosed in these publications or other known techniques.

[0043] The linear parameter calculation unit 42 calculates linear parameters using the demosaiced polarization image and sets them as guide features in the high-accuracy depth estimation unit 14 ( FIG. 1 ). More specifically, the linear parameter calculation unit 42 calculates the Stokes vector [s0, s1, s2] of equation (4) as the linear parameters, and outputs the parameters (Stokes parameters) s0, s1, and s2 of the Stokes vector as guide features.

[0044]

[0045] The Stokes vector is a vector that indicates the polarization state of light and is composed of four types of parameters (Stokes parameters), s0 to s3. The Stokes parameter s0 represents the light intensity signal of unpolarized light, the Stokes parameter s1 represents the differential signal between the horizontal linear polarization component and the vertical linear polarization component, the Stokes parameter s2 represents the differential signal between the 45-degree linear polarization component and the 135-degree linear polarization component, and the Stokes parameter s3 represents the differential signal between the right-handed circular polarization component and the left-handed circular polarization component.

[0046] In this embodiment, a polarized image is acquired using a linear polarizer as the polarizer, and therefore, equation (4) is expressed using only three Stokes parameters s0 to s2 out of the four Stokes parameters s0 to s3.

[0047] If the range of the brightness I is 0<I<1, the range of the Stokes parameter s0 is 0<s0<2, the range of the Stokes parameter s1 is -1<s1<1, and the range of the Stokes parameter s2 is -1<s2<1. ０ , I ４５ , I ９０ , I １３５ Since the sum of is divided by 2, the distribution of the Stokes parameter s0 is basically the same as the distribution obtained from the RGB image.

[0048] 7, it has been experimentally confirmed that the distribution of the Stokes parameters s1 and s2 is concentrated near 0 within the range from −1 to 1. Since many of the Stokes parameters s1 and s2 take values ​​near 0 and do not take other values ​​in this way, the linear parameter calculation unit 42 may calculate Stokes parameters s1′ and s2′ by normalizing the Stokes parameters s1 and s2 in accordance with the distribution of their values, and set the Stokes parameter s0 and the normalized Stokes parameters s1′ and s2′ as guide feature quantities.

[0049] More specifically, the linear parameter calculation unit 42 approximates the distribution of the Stokes parameters s1 and s2 by a normal distribution, and calculates the mean μ and variance σ ２ Calculate the average μ±σ and the range of average μ±3σ (s min , s max ) and normalize the Stokes parameters s1 and s2 using the following equation (5).

[0050]

[0051] Regarding the Stokes parameters s1 and s2, the signal resolution can be increased by outputting normalized Stokes parameters s1' and s2'.

[0052] 4. High-resolution depth image generation process using the first configuration example Next, a high-resolution depth image generation process performed by the image processing device 1 when the guide feature quantity conversion unit 13 has the first configuration example will be described with reference to the flowchart in Fig. 8. This process is started, for example, when polarization information and depth information are input to the image processing device 1.

[0053] First, in step S11, the polarization information acquisition unit 11 acquires a polarization RAW image, which is polarization information, and the depth information acquisition unit 12 acquires a depth image, which is depth information. The polarization RAW image is supplied to the guide feature conversion unit 13, and the depth image is supplied to the high-precision depth estimation unit 14.

[0054] In step S12, the demosaic processing unit 41 of the guide feature conversion unit 13 performs demosaic processing on the polarization RAW image supplied from the polarization information acquisition unit 11, generates a 4-channel polarization image after demosaic processing, and supplies it to the linear parameter calculation unit 42.

[0055] In step S13, the linear parameter calculation unit 42 of the guide feature conversion unit 13 calculates linear parameters using the polarization image after demosaicing and outputs them as guide features to the high-accuracy depth estimation unit 14. For example, the linear parameter calculation unit 42 calculates the Stokes vector [s0, s1, s2] of equation (4) as the linear parameters, and outputs the Stokes parameters s0, s1, and s2 as guide features. Alternatively, the linear parameter calculation unit 42 may output Stokes parameters s1' and s2' normalized according to the distribution of the values ​​of the Stokes parameters s1 and s2.

[0056] In step S14, the high-precision depth estimation unit 14 performs an integration process on the input depth information and the guide feature converted from the polarization information to generate high-resolution depth information. Specifically, the high-precision depth estimation unit 14 generates a depth image (high-resolution depth image) with higher precision and resolution than the input depth image using the depth image supplied from the depth information acquisition unit 12 and the Stokes parameters s0, s1, s2 or s0, s1', s2', which are the guide feature supplied from the linear parameter calculation unit 42, and outputs the depth image as high-resolution depth information. The high-precision depth estimation unit 14 generates the high-resolution depth image by inputting the depth image and the Stokes parameters s0, s1, s2 or s0, s1', s2' to a machine learning model in which model parameters have been pre-trained.

[0057] The high-resolution depth image generation process is executed as described above.

[0058] The linear parameter calculation unit 42 calculates the intensity I of the polarization image after demosaic processing. ０ , I ４５ , I ９０ , I １３５ However, other linear parameters may be calculated as the guide feature. For example, the luminance I at a predetermined polarization angle υ as shown in the following equation (6) or (7) υ Linear parameters expressed as a linear sum of the above may be calculated as s0, s1, and s2 and used as the guide feature.

[0059]

[0060] According to the image processing device 1 having the first configuration example of the guide feature conversion unit 13, linear parameters s0, s1, s2 (s0, s1', s2'), which are features that can be expressed by linear conversion from the RAW data of the polarization image, are generated as guide features, and by inputting them into a machine learning model, a depth image with higher accuracy and resolution can be obtained.

[0061] 5. Second Configuration Example of Guide Feature Transformation Unit FIG. 9 is a block diagram showing a second configuration example of the guide feature transformation unit 13. As shown in FIG.

[0062] The guide feature quantity conversion unit 13 in FIG. 9 includes a demosaic processing unit 41, a linear parameter calculation unit 42, and a DoLP feature quantity calculation unit 43. Compared to the first configuration example shown in FIG. 5, the guide feature quantity conversion unit 13 in the second configuration example further includes a DoLP feature quantity calculation unit 43 after the linear parameter calculation unit 42. The processing by the demosaic processing unit 41 and the linear parameter calculation unit 42 is the same as in the first configuration example. That is, the demosaic processing unit 41 performs demosaic processing on the polarization RAW image and supplies the demosaic-processed polarization image to the linear parameter calculation unit 42. The linear parameter calculation unit 42 calculates Stokes parameters s0, s1, and s2, which are a type of linear parameter, using the demosaic-processed polarization image.

[0063] The DoLP feature calculation unit 43 uses the Stokes parameters s0, s1, and s2 to calculate parameters a0, a1, and a2, which are polarization feature parameters using the degree of linear polarization (DoLP) and the angle of linear polarization (AoLP), and outputs the Stokes parameter s0 and the parameters a0, a1, and a2 as guide feature parameters.

[0064] The degree of linear polarization DoLP and the linear polarization angle AoLP can be calculated using the Stokes parameters s1 and s2, respectively, using the following equations (8) and (9). The linear polarization angle AoLP is the same as the azimuth angle φ of the normal information described above.

[0065]

[0066] The parameters a0, a1, and a2 are expressed by the following equation (10) using the degree of linear polarization DoLP and the azimuth angle φ (linear polarization angle AoLP).

[0067]

[0068] The parameters a0, a1, and a2 are polarization feature amounts using the doubled polarization angle obtained by doubling the linear polarization angle AoLP and the degree of linear polarization DoLP.

[0069] The azimuth angle φ of the normal information has an indeterminacy of 180 degrees. The indeterminacy of 180 degrees means that the same normal information (azimuth angle φ and zenith angle θ) is output when the polarization direction is 0 degrees and 180 degrees, or when the polarization direction is 45 degrees and 225 degrees.

[0070] For example, as shown in A of Fig. 10, when camera 21 captures an image of hemispherical subject 20 via polarizing plate 22 and calculates normal information, the normal information (indicated by the arrows) is correct such that both the upper half area GA and the lower half area GB of subject 20 are outwardly convex, as shown in C of Fig. 10, but erroneous reverse normal information may be obtained, such as inwardly convex normal information for the lower half area GB, as shown in B of Fig. 10. It is also possible that erroneous reverse normal information is obtained for the upper half area GA, such that the upper half area is inwardly convex.

[0071] The parameters a0, a1, and a2 use a double polarization angle obtained by doubling the linear polarization angle AoLP (azimuth angle φ), thereby eliminating the above-mentioned 180-degree ambiguity.

[0072] 6. High-resolution depth image generation process using the second configuration example Next, a high-resolution depth image generation process performed by the image processing device 1 when the guide feature quantity conversion unit has the second configuration example will be described with reference to the flowchart in Fig. 11. This process is started, for example, when polarization information and depth information are input to the image processing device 1.

[0073] First, in step S31, the polarization information acquisition unit 11 acquires a polarization RAW image, which is polarization information, and the depth information acquisition unit 12 acquires a depth image, which is depth information. The polarization RAW image is supplied to the guide feature conversion unit 13, and the depth image is supplied to the high-precision depth estimation unit 14.

[0074] In step S32, the demosaic processing unit 41 of the guide feature conversion unit 13 performs demosaic processing on the polarization RAW image supplied from the polarization information acquisition unit 11, generates a 4-channel polarization image after demosaic processing, and supplies it to the linear parameter calculation unit 42.

[0075] In step S33, the linear parameter calculation unit 42 of the guide feature conversion unit 13 calculates linear parameters using the demosaiced polarization image and outputs the linear parameters to the DoLP feature calculation unit 43. For example, the linear parameter calculation unit 42 calculates the Stokes vector [s0, s1, s2] of equation (4) as the linear parameters, and supplies the Stokes parameters s0, s1, s2 to the DoLP feature calculation unit 43.

[0076] In step S34, the DoLP feature calculation unit 43 calculates the degree of linear polarization DoLP and the linear polarization angle AoLP using the Stokes parameters s0, s1, and s2, and further calculates parameters a0, a1, and a2 using the degree of linear polarization DoLP and the linear polarization angle AoLP. The DoLP feature calculation unit 43 outputs the parameters s0, a0, a1, and a2 to the high-accuracy depth estimation unit 14 as guide features.

[0077] In step S35, the high-precision depth estimation unit 14 performs an integration process on the input depth information and the guide feature converted from the polarization information to generate high-resolution depth information. Specifically, the high-precision depth estimation unit 14 generates a depth image (high-resolution depth image) with higher precision and resolution than the input depth image using the depth image supplied from the depth information acquisition unit 12 and the parameters s0, a0, a1, and a2, which are the guide feature supplied from the DoLP feature calculation unit 43, and outputs the depth image as high-resolution depth information. The high-precision depth estimation unit 14 generates the high-resolution depth image by inputting the depth image and the parameters s0, a0, a1, and a2 to a machine learning model in which model parameters have been pre-trained.

[0078] The high-resolution depth image generation process is executed as described above.

[0079] According to the image processing device 1 having the second configuration example of the guide feature conversion unit 13, after linearly converting the raw data of the polarization image into Stokes parameters s0, s1, and s2, a polarization feature is generated as a guide feature using a doubled polarization angle (AoLP) and a linear polarization degree DoLP instead of the polarization normal, and input into a machine learning model, thereby making it possible to obtain a depth image with higher accuracy and resolution.

[0080] 7. Summary The image processing device 1 includes a polarization information acquisition unit 11 that acquires polarization information of a subject, and a depth information acquisition unit 12 that acquires depth information of the subject. The image processing device 1 also includes a guide feature conversion unit 13 that calculates linear parameters from the polarization information of the subject and sets a guide feature using the calculated linear parameters, and a high-precision depth estimation unit 14 that generates high-resolution depth information with higher resolution than the depth information from the depth information of the subject and the guide feature set by the guide feature conversion unit 13. By using the guide feature as a polarization feature that is more robust to noise than the polarization normal, which is normal information of the subject, it is possible to increase the resolution of a low-resolution depth image.

[0081] The image processing device 1 may further include a luminance information acquisition unit that acquires a luminance image (RGB image) that is luminance information of the subject, and may be configured to output high-resolution depth information using the luminance information as well. In this case, the high-precision depth estimation unit 14 generates high-resolution depth information with a higher resolution than the acquired depth information using a machine learning model that inputs the luminance information and depth information of the subject and the guide feature. By also using the luminance information, it is possible to reinforce the polarization sensor, which is fundamentally vulnerable to noise, for example, when capturing images in dark places.

[0082] 8. First Application Example of Image Processing Device FIG. 12 shows a first application example to which the image processing device 1 of FIG. 1 is applied.

[0083] 12 shows an example in which the image processing device 1 of FIG. 1 is applied to an image processing device 100 arranged after a polarization / depth sensor 60. The image processing device 100 has a configuration similar to that of the image processing device 1 of FIG. 1, and polarization information and depth information are input to the image processing device 100 from the polarization / depth sensor 60.

[0084] The polarization / depth sensor 60 has a polarization imaging unit 61 and a ToF imaging unit 62. The polarization imaging unit 61 generates polarization information and outputs it to the image processing device 100, and the ToF imaging unit 62 generates depth information and outputs it to the image processing device 100. The polarization information acquisition unit 11 acquires the polarization information output from the polarization imaging unit 61, and the depth information acquisition unit 12 acquires the depth information output from the ToF imaging unit 62.

[0085] FIG. 13 is a diagram illustrating a part of an example of the configuration of the polarization and depth sensor 60. As shown in FIG.

[0086] The polarization imaging section 61 has a pixel area 61A in which polarization angles (polarization directions) of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees are arranged in a Bayer array. Fig. 13 shows 64 pixels, from pixel (0,0) to pixel (7,7), which are part of the pixel area 61A. The polarization imaging section 61 receives four different types of polarized light, accumulates charges according to the received polarization, and outputs the charges pixel by pixel.

[0087] The ToF imaging unit 62 has a pixel region 62A including four pixels a, b, c, and d that receive near-infrared light. The ToF imaging unit 62 receives near-infrared light, accumulates charges according to the amount of near-infrared light received, and outputs the charges pixel by pixel. Hereinafter, the pixels of the polarization imaging unit 61 will be referred to as polarization pixels, and the pixels of the ToF imaging unit 62 will be referred to as ToF pixels to distinguish them from each other.

[0088] The pixel region 61A of the polarization imaging unit 61 and the pixel region 62A of the ToF imaging unit 62 are stacked in the optical axis direction of the incident light to form the pixel region 63A of the polarization / depth sensor 60. For example, as shown in FIG. 13 , the pixel regions 61A and 62A are stacked so that one ToF pixel corresponds to 4 × 4 = 16 polarization pixels. This correspondence is not limited to this, and various correspondences are possible. For example, a configuration in which 2 × 2 = 4 polarization pixels correspond to one ToF pixel may be used, or a configuration in which 8 × 8 = 16 polarization pixels correspond to one ToF pixel may be used. ToF pixels are configured to receive infrared light components and therefore require a large pixel size for noise reduction. In contrast, polarization pixels are configured to receive visible light components and can be configured with a high pixel density similar to that of a conventional camera, while the pixel size of the polarization pixels can be small. The specific stacked configuration of the polarization imaging unit 61 and the ToF imaging unit 62 is disclosed in, for example, the above-mentioned Patent Document 1.

[0089] Such a polarization / depth sensor 60 can simultaneously acquire polarization information and depth information from the same viewpoint (coaxially).

[0090] 9. Second Application Example of Image Processing Device FIG. 14 shows a second application example to which the image processing device 1 of FIG. 1 is applied.

[0091] The second application example shown in Figure 14 shows an example in which a polarization sensor 81 and a depth sensor 82 that output polarization information and depth information are provided separately, and the image processing device 1 of Figure 1 is applied to an image processing device 100 that is arranged downstream of the polarization sensor 81 and the depth sensor 82.

[0092] In the second application example, the polarization sensor 81 and the depth sensor 82 are arranged with a predetermined offset, and therefore the viewpoints of the polarization sensor 81 and the depth sensor 82 are different. Therefore, in the image processing device 100, a viewpoint conversion unit 15 is added between the guide feature amount conversion unit 13 and the high-accuracy depth estimation unit 14.

[0093] The viewpoint conversion unit 15 performs a process of matching the viewpoint positions of the polarization image and the depth image. Specifically, the viewpoint conversion unit 15 performs a process of matching the guide feature amount in pixel units output from the guide feature amount conversion unit 13 with the pixel position of the depth image acquired by the depth information acquisition unit 12. This process requires a prior camera calibration process. For example, feature point matching is performed using two images captured by the polarization sensor 81 and the depth sensor 82, respectively, to calculate calibration parameters (hereinafter referred to as viewpoint conversion parameters) for matching the viewpoint positions of the two images, and the calibration parameters are stored in a memory in the image processing device 100. The viewpoint conversion unit 15 uses the viewpoint conversion parameters stored in this memory to perform a process of matching the pixel positions of the guide feature amount with the pixel positions of the depth image. The viewpoint conversion unit 15 supplies the guide feature amount after matching the pixel positions to the high-precision depth estimation unit 14.

[0094] <Hardware Configuration Example of Image Processing Device> The image processing device 100 in the first and second application examples can be executed by dedicated hardware or by software. When the processing of the image processing device 100 is executed by software, the programs that constitute the software are installed on a computer. Here, the computer includes a microcomputer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0095] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer 200 that executes the above-described series of processes as the image processing apparatus 100 using a program.

[0096] The computer 200 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203. The CPU 201, the ROM 202, and the RAM 203 are interconnected by a bus 204.

[0097] An input / output interface 205 is also connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

[0098] The input unit 206 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 207 includes a display, speaker, output terminal, etc. The storage unit 208 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 209 includes a network interface, etc. The drive 210 drives a removable recording medium 211 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0099] In a computer configured as described above, the CPU 201 performs the above-described series of processes by, for example, loading a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executing the program. The RAM 203 also stores data necessary for the CPU 201 to execute various processes as appropriate.

[0100] The program executed by the computer 200 (CPU 201) can be provided by being recorded on a removable recording medium 211 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. Alternatively, the program can be pre-installed in the ROM 202 or the storage unit 208.

[0101] 10. Third Application Example of Image Processing Device FIG. 16 shows a third application example to which the image processing device 1 of FIG. 1 is applied.

[0102] The third application example shown in FIG. 16 shows an example in which the image processing device 1 of FIG. 1 is applied to a polarization and depth sensor 300.

[0103] The polarization / depth sensor 300 includes at least a pixel array unit 321 having a polarization information acquisition unit 11 and a depth information acquisition unit 12, and a signal processing unit 322 having a guide feature conversion unit 13 and a high-precision depth estimation unit 14. The polarization information acquisition unit 11 of the pixel array unit 321 is configured similarly to the pixel region 61A in FIG. 13 , in which polarization pixels are arranged in a matrix, and the depth information acquisition unit 12 is configured similarly to the pixel region 62A in FIG. 13 , in which ToF pixels are arranged in a matrix. The signal processing unit 322 includes, for example, a logic circuit, which includes a signal processing circuit that processes signals generated by each pixel and an AI processing circuit that performs AI processing based on the signals generated by each pixel. The signal processing unit 322 may also include a memory circuit that stores signals processed by the logic circuit. If the optical axis of the incident light entering the pixel area of ​​the polarization information acquisition unit 11 and the optical axis of the incident light entering the pixel area of ​​the depth information acquisition unit 12 are not coaxial, a viewpoint conversion unit 15 is provided between the guide feature conversion unit 13 and the high-precision depth estimation unit 14 of the signal processing unit 322, as in the second application example of Figure 14.

[0104] In this way, the functions of the image processing device 1 can be realized within the polarization and depth sensor 300 .

[0105] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0106] For example, it is possible to adopt a configuration in which all or part of the above-described embodiments are combined.

[0107] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0108] Furthermore, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. When one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0109] In this specification, the steps described in the flowcharts may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0110] The effects described in this specification are merely examples and are not intended to be limiting, and there may be effects other than those described in this specification.

[0111] The technology disclosed herein may employ the following configurations. (1) An image processing device including: a guide feature conversion unit that sets guide features using linear parameters calculated from polarization information of a subject; and a depth estimation unit that generates high-resolution depth information having a higher resolution than the depth information from depth information of the subject and the guide features set by the guide feature conversion unit. (2) The image processing device described in (1), wherein the guide feature conversion unit includes: a demosaic processing unit that performs demosaic processing on a polarized RAW image, which is the polarization information; and a linear parameter calculation unit that calculates the linear parameters using the polarized image after demosaic processing. (3) The image processing device described in (2), wherein the linear parameter calculation unit calculates Stokes parameters as the linear parameters and sets them as the guide features. (4) The image processing device described in (3), wherein the linear parameter calculation unit sets Stokes parameters indicating the difference between the intensity of unpolarized light and linearly polarized light components as the guide features. (5) The image processing device according to (4), wherein the linear parameter calculation unit sets normalized Stokes parameters as the guide feature quantities for Stokes parameters indicating the difference between the linearly polarized components. (6) The image processing device according to (2), wherein the guide feature conversion unit further includes a feature calculation unit that calculates a polarization feature quantity using the linear parameters and a linear polarization angle. (7) The image processing device according to (6), wherein the feature calculation unit sets a part of the linear parameters and the polarization feature quantity as the guide feature quantities. (8) The image processing device according to (7), wherein the linear parameter calculation unit calculates Stokes parameters as the linear parameters, and the feature calculation unit sets the polarization feature quantity using Stokes parameters indicating the intensity of unpolarized light, a doubled polarization angle obtained by doubling the linear polarization angle, and the linear polarization degree as the guide feature quantities. (9) The image processing device according to any one of (1) to (8), wherein the depth estimation unit generates the high-resolution depth information using a machine learning model that receives the depth information and the guide feature amount as input.(10) The image processing device according to any one of (1) to (9), further comprising: a polarization information acquisition unit that acquires polarization information of the subject; and a depth information acquisition unit that acquires depth information of the subject. (11) The image processing device according to any one of (1) to (10), wherein the guide feature conversion unit and the depth estimation unit are provided in a sensor that acquires the polarization information and the depth information. (12) An image processing method, comprising: setting a guide feature using linear parameters calculated from the polarization information of the subject; and generating high-resolution depth information having a higher resolution than the depth information from the depth information of the subject and the set guide feature.

[0112] DESCRIPTION OF SYMBOLS 1 Image processing device, 11 Polarization information acquisition unit, 12 Depth information acquisition unit, 13 Guide feature conversion unit, 14 High-precision depth estimation unit, 15 Viewpoint conversion unit, 41 Demosaicing processing unit, 42 Linear parameter calculation unit, 43 DoLP feature calculation unit, 60 Depth sensor, 61 Polarization imaging unit, 62 ToF imaging unit, 81 Polarization sensor, 82 Depth sensor, 100 Image processing device, 200 Computer, 202 ROM, 204 Bus, 205 Input / output interface, 206 Input unit, 207 Output unit, 208 Storage unit, 209 Communication unit, 210 Drive, 211 Removable recording medium, 300 Depth sensor, 321 Pixel array unit, 322 Signal processing unit

Claims

1. An image processing device comprising: a guide feature conversion unit that sets a guide feature using linear parameters calculated from polarization information of a subject; and a depth estimation unit that generates high-resolution depth information having a higher resolution than the depth information from the depth information of the subject and the guide feature set by the guide feature conversion unit.

2. The image processing device according to claim 1, wherein the guide feature conversion unit has: a demosaic processing unit that performs demosaic processing on a polarized RAW image, which is the polarization information; and a linear parameter calculation unit that calculates the linear parameters using the polarized image after demosaic processing.

3. The image processing device according to claim 2, wherein the linear parameter calculation unit calculates Stokes parameters as the linear parameters and sets them as the guide feature amounts.

4. The image processing device according to claim 3, wherein the linear parameter calculation unit sets, as the guide feature amount, a Stokes parameter indicating the intensity of unpolarized light and a Stokes parameter indicating the difference between linearly polarized light components.

5. The image processing device according to claim 4, wherein the linear parameter calculation section sets a normalized Stokes parameter representing a difference between the linearly polarized light components as the guide feature amount.

6. The image processing device according to claim 2, wherein the guide feature conversion unit further includes a feature calculation unit that uses the linear parameters to calculate a polarization feature using a degree of linear polarization and a linear polarization angle.

7. The image processing device according to claim 6, wherein the feature calculation section sets a part of the linear parameters and the polarization feature as the guide feature.

8. The image processing device according to claim 7, wherein the linear parameter calculation unit calculates Stokes parameters as the linear parameters, and the feature calculation unit sets the polarization feature using the Stokes parameters indicating the intensity of unpolarized light, a doubled polarization angle obtained by doubling the linear polarization angle, and a degree of linear polarization as the guide feature.

9. The image processing device according to claim 1, wherein the depth estimation unit generates the high-resolution depth information using a machine learning model that receives the depth information and the guide feature as input.

10. The image processing device according to claim 1, further comprising: a polarization information acquisition unit that acquires polarization information of the subject; and a depth information acquisition unit that acquires depth information of the subject.

11. The image processing device according to claim 1, wherein the guide feature conversion unit and the depth estimation unit are provided within a sensor that acquires the polarization information and the depth information.

12. An image processing method, comprising: setting a guide feature using linear parameters calculated from polarization information of a subject; and generating high-resolution depth information having a higher resolution than the depth information from the depth information of the subject and the set guide feature.

Images