Shape estimation device, shape estimation method, and program

The shape estimation device uses a single polarizing camera and polarization analysis to efficiently estimate transparent refractive surfaces' three-dimensional shape with low computational cost and no prior knowledge, addressing inefficiencies in conventional methods.

JP7587191B2Active Publication Date: 2024-11-20NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023526751
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-06-10
Publication Date
2024-11-20
Estimated Expiration
2041-06-10

AI Technical Summary

Technical Problem

Conventional methods for estimating the three-dimensional shape of transparent refractive surfaces require multiple cameras, complex calculations, and prior knowledge of the shape, making them inefficient and costly.

Method used

A shape estimation device and method using a single polarizing camera to capture images with and without a transparent refractive layer, employing feature point tracking and polarization analysis to estimate the refractive surface's normal vectors, formulating the problem as a convex optimization, and generating three-dimensional shape information.

Benefits of technology

Enables efficient estimation of transparent refractive surfaces' three-dimensional shape with low computational cost and no prior learning data, using a single camera and geometric-optical modeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587191000011
    Figure 0007587191000011
  • Figure 0007587191000012
    Figure 0007587191000012
  • Figure 0007587191000013
    Figure 0007587191000013
Patent Text Reader

Abstract

This shape estimation device comprises: a polarization camera 10 for capturing a first image of a subject with no transparent refractive layer interposed between the camera 10 and the subject and a second image of the subject with a refractive layer interposed between the camera 10 and the subject; a feature value acquisition unit 20 for applying a feature point tracking method to the first image and second image and thereby acquiring a distortion vector Δg that is a feature value representing variation in geometric appearance resulting from refraction; a first estimation unit 30 for acquiring, from the second image, brightness value pairs respectively corresponding to at least three different polarization angles and estimating polarization feature values Imax, Imin, Ψ; a second estimation unit 40 for estimating two (ni +, ni -) candidates for normal vectors ni of the refractive layer to be estimated using the polarization feature values; and a refractive surface generation unit 50 for using the distortion vector Δg to select one normal vector candidate from among the normal vector candidates ni +, ni - and generating three-dimensional refractive surface shape information representing the surface shape of the refractive layer.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a shape estimation device, a shape estimation method, and a program for estimating the three-dimensional shape of a transparent refractive layer such as a water surface or air fluctuation from an image. [Background technology]

[0002] Technology for estimating the 3D shape of objects in images is particularly important in fields such as robot vision, augmented reality, and autonomous driving.

[0003] In general, 3D shape estimation involves using multiple cameras to estimate the 3D shape from the difference in appearance due to the difference in the camera's installation position. This conventional method assumes that the estimation target is an opaque and diffuse reflecting surface. Therefore, it is not possible to estimate the shape of a transparent refractive surface such as the surface of water.

[0004] In order to estimate the 3D shape of such a refractive surface, it is necessary to model the propagation of light from the camera to the refractive surface to the object behind the refractive surface. However, the 3D shapes of both the refractive surface and the object behind it are generally unknown. In addition, because the refraction model is complex, it is difficult to formulate it as a clear optimization problem.

[0005] Therefore, Non-Patent Document 1 theoretically and experimentally discloses that at least two cameras are required to estimate the three-dimensional shape of a scene where one refraction occurs. Non-Patent Document 2 discloses that it is possible to estimate the three-dimensional shape of a refracting surface with one camera by considering the difference in how the background appears depending on whether refraction occurs or not. Non-Patent Document 3 discloses a method for estimating the three-dimensional shape of a transparent surface using polarization information. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] [Retrieved June 4, 2021], Internet<URL: https: / / www.cs.toronto.edu / ~kyros / pubs / 08.ijcv.indirect.pdf> [Non-Patent Document 2] [Retrieved June 4, 2021], Internet<URL: http: / / grail.cs.washington.edu / projects / glass-recon / cvpr12_refraction.pdf> [Non-Patent Document 3] [Retrieved June 4, 2021], Internet<URL: http: / / www.info.hiroshima-cu.ac.jp / ~miyazaki / publication / paper / Miyazaki-PAMI2007.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] However, the method of Non-Patent Document 1 requires at least two cameras, and requires alignment and time synchronization between the cameras. In addition, in Non-Patent Document 2, the model formulated as an optimization problem is complex, and shape estimation takes a long time and requires high calculation costs. In addition, in Non-Patent Document 3, it is necessary to know the rough three-dimensional shape of the estimation target in advance.

[0008] As described above, conventional techniques require multiple cameras, high computational costs, and the 3D shape must be known, and there is a problem in that there is no suitable technique for estimating 3D shape.

[0009] The present invention has been made in consideration of this problem, and aims to provide a shape estimation device, a shape estimation method, and a program capable of estimating the three-dimensional shape of a transparent refractive surface using a single camera, with a low computational effort, and without prior learning data. [Means for solving the problem]

[0010] A shape estimation device according to one embodiment of the present invention includes a polarization camera that captures a first image of a subject when a transparent refractive layer is not present and a second image of the subject when the refractive layer is present; a feature acquisition unit that applies a feature point tracking method between the first image and the second image to acquire a distortion vector, which is a feature representing a geometric change in appearance due to refraction; a first estimation unit that acquires sets of brightness values ​​corresponding to at least three different polarization angles from the second image to estimate polarization features; a second estimation unit that estimates two candidates for a normal vector of the refractive layer to be estimated using the polarization features; and a refractive surface generation unit that uses the distortion vector to select one of the candidates for the normal vector and generate three-dimensional shape information of the refractive surface that represents the surface shape of the refractive layer.

[0011] Furthermore, a refraction surface 3D shape estimation method according to one embodiment of the present invention is a refraction surface 3D shape estimation method performed by the refraction surface 3D shape estimation device, in which a polarization camera captures a first image which is an image of a subject when a transparent refraction layer is not present and a second image which is an image of the subject when the refraction layer is present, a feature acquisition unit applies a feature point tracking technique between the first image and the second image to acquire a distortion vector which is a feature representing a geometric change in appearance due to refraction, a first estimation unit acquires at least three different pairs of angles and brightness values ​​from the second image to estimate polarization features, a second estimation unit estimates two candidates for a normal vector of the refraction layer to be estimated using the polarization features, and a refraction surface generation unit selects one of the candidates for the normal vector using the distortion vector, and generates refraction surface 3D shape information representing the surface shape of the refraction layer.

[0012] In addition, a program according to one aspect of the present invention is a program for causing a computer to function as the above-mentioned refracting surface three-dimensional shape estimation device. Effect of the Invention

[0013] Our invention enables estimation of the 3D shape of transparent refractive surfaces with a single camera, low computational complexity, and no prior training data. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of a functional configuration of a shape estimation device according to an embodiment of the present invention. [Diagram 2] 2 is a diagram illustrating a schematic diagram of the relationship between the shape estimation device shown in FIG. 1, a subject, and a transparent refractive layer. [Diagram 3] FIG. 13 is a diagram showing a model of a change in polarization, in which (a) shows a normal vector and light reflected by a refractive layer, and (b) shows the relationship between the zenith angle and the degree of polarization. [Figure 4] FIG. 2 is a diagram illustrating a relationship between an azimuth angle and image brightness. [Diagram 5] FIG. 1 is a schematic diagram showing modeling of geometric changes. [Figure 6] 10 is a diagram showing an example of three-dimensional shape information of a refracting surface. FIG. [Figure 7] 2 is a flowchart showing a processing procedure of a shape estimation method performed by the shape estimation device shown in FIG. 1. [Figure 8] FIG. 1 is a block diagram showing an example of the configuration of a general-purpose computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same reference characters are used to refer to the same parts, and the description will not be repeated.

[0016] Fig. 1 is a block diagram showing an example of the functional configuration of a shape estimation device 100 according to an embodiment of the present invention. The shape estimation device 100 shown in Fig. 1 is a device that estimates the three-dimensional shape of a transparent refractive layer that is interposed between the device and a subject.

[0017] The shape estimation device 100 includes a polarization camera 10, a feature acquisition unit 20, a first estimation unit 30, a second estimation unit 40, and a refraction surface generation unit 50. Each functional component except for the polarization camera 10 can be realized by a computer including, for example, a ROM, a RAM, a CPU, etc. In this case, the processing contents are described by a program.

[0018] The polarization camera 10 is a typical polarization camera. The polarization camera 10 incorporates, for example, polarizers (polarizing filters) with four different polarization angles.

[0019] Polarized camera 10 captures a first image, which is an image of the subject when no transparent refractive layer is present, and a second image, which is an image of the subject when a refractive layer is present. Transparent refractive layers include the surface of water, a layer of fluctuating air, etc.

[0020] The feature amount acquiring unit 20 acquires a distortion vector, which is a feature amount representing a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image captured by the polarization camera 10. The feature point tracking method is, for example, optical flow.

[0021] The first estimation unit 30 estimates a polarization feature by acquiring a set of luminance values ​​corresponding to at least three different polarization angles from the second image. The polarization feature is the degree of polarization. This will be described in detail later.

[0022] The second estimation unit 40 estimates two candidates for the normal vector of the refractive layer to be estimated using the polarization feature amount, that is, narrows down the candidates for the normal vector of the change in polarization to two.

[0023] The refraction surface generating unit 50 uses the distortion vector to select one of the candidates for the normal vector, and generates the refraction surface three-dimensional shape information representing the surface shape of the refraction layer. The normal vector is a vector perpendicular to the tangent of the pixel of the second image. The refraction surface three-dimensional shape information representing the surface shape of the refraction layer can be generated from the normal vector of the pixel of the second image in which the refraction layer is reflected.

[0024] Fig. 2 is a diagram showing a schematic diagram of the relationship between the shape estimation device 100, the subject (background), and a transparent refractive layer (the refractive surface to be estimated). Fig. 2 shows only the image sensor and the polarizing filter that constitute the polarization camera 10.

[0025] The rectangular image sensor shown in Fig. 2 is, for example, a CMOS image sensor with millions of pixels. Pixel i on the image plane receives light that has passed through a refractive layer from the background and through a polarizing filter.

[0026] A polarizing filter is placed in front of the image sensor with four different polarization angles, which is common.

[0027] The shape estimation device 100 aligns the direction of the incident light along the optical axis Z c Then, the normal vector n i The azimuth angle φ and elevation angle θ of the refractive layer are estimated for each pixel i, and the surface shape of the refractive layer is estimated. Hereinafter, each functional component of the shape estimation device 100 will be described.

[0028] Note that if the image plane and the polarizing filter plane are parallel, the normal vector n i Since the azimuth angle φ and the polarization angle have the same geometric meaning, hereinafter, the azimuth angle and the polarization angle will be described as the same variable φ.

[0029] (1st estimation part) The first estimating section 30 obtains sets of luminance values ​​corresponding to at least three different polarization angles from the second image of the subject when a refractive layer is present, and estimates the polarization feature amount.

[0030] The following relationship holds between the polarization angle and the luminance value:

[0031]

number

[0032] Here, the polarization feature is I max ,I min ,Ψ. The polarization feature can be estimated from three or more different pairs of polarization angles and intensity values.

[0033] Polarization feature I max ,I min, Ψ, the Stokes vector s representing the polarization state can be expressed as follows:

[0034]

number

[0035] The change in the Stokes vector s can be expressed by multiplying the Stokes vector before the change by the Mueller matrix M. In other words, the Stokes vector before the change is s in , the Stokes vector after the change is s out Then, s out =M s in It becomes.

[0036]

number

[0037] Here, Ts is the Fresnel transmission coefficient (component horizontal to the plane of incidence) and Tt is the Fresnel transmission coefficient (component perpendicular to the plane of incidence).

[0038] The degree of polarization ρ (Degree of Polarization), which represents the degree of change in polarization, can be expressed by the following equation.

[0039]

number

[0040] Especially s in When the light is unpolarized, it can be expressed as follows:

[0041]

number

[0042] Although equation (7) may seem complicated at first glance, it is a monotonically increasing function and can be formulated as a convex optimization problem. Therefore, the normal vector n i The elevation angle θ can be uniquely estimated from the observed degree of polarization.

[0043]

number

[0044] FIG. 3 is a schematic diagram showing the modeling of the change in polarization. (a) shows the normal vector n i 4(b) shows the relationship between the zenith angle and the degree of polarization. FIG.

[0045] In FIG. 3(a), s in is the light incident on the refractive surface (surface of the refractive layer), s out represents the light captured by the polarization camera 10.

[0046] The horizontal axis of FIG. 3(b) represents the zenith angle, and the vertical axis represents the degree of polarization. As shown in FIG. 3(b), if the elevation angle θ is known, the degree of polarization ρ can be uniquely determined.

[0047] (Second estimation part) The second estimation unit 40 estimates a normal vector n i We estimate two candidates.

[0048] If the image plane and the polarizing filter plane are parallel, the normal vector n i The azimuth angle φ of the polarizing filter coincides with the polarization angle Ψ (the polarization angle at which the brightness value is maximized). Note that when the polarizing filter is rotated once, there are two angles at which the brightness value is maximized, so there is an ambiguity of 180°.

[0049] Fig. 4 is a diagram showing a schematic diagram of the relationship between the azimuth angle φ and the brightness value. The horizontal axis of Fig. 4 shows the azimuth angle φ, and the vertical axis shows the brightness I(φ) of the image. As shown in Fig. 4, the brightness value I(φ) has two maximum values.

[0050] Therefore, the normal vector n i The candidates are narrowed down to the following two:

[0051]

number

[0052] (Feature acquisition unit) A feature point tracking technique is applied between a first image of a subject when no transparent refractive layer is present and a second image of the subject when a refractive layer is present to obtain a distortion vector, which is a feature that represents the geometric change in appearance due to refraction.

[0053] FIG. 5 is a schematic diagram showing the modeling of the geometric change. f represents the light space on the side of the polarization camera 10. μ is the relative refractive index. Also, v r is the direction vector of the refracted light.

[0054] Refracted light direction vector v r can be expressed as follows:

[0055]

number

[0056] The distortion vector Δ is a feature that represents the geometric change in appearance due to refraction. g can be expressed as follows:

[0057]

number

[0058] (Bent surface generation part) The refraction surface generator 50 calculates the distortion vector Δ g Using the normal vector n i Candidate n i + ,n i - One of the above is selected, and three-dimensional shape information of the refractive surface is generated, which represents the surface shape of the refractive layer.

[0059] The refraction surface generating unit 50 selects the normal vector n i Candidate n i + ,n i -Then, the 3D shape information of the refracting surface is generated by solving the optimal solution of the following equation.

[0060]

number

[0061] The calculation to find the optimum solution of equation (12) is performed for all pixels i.

[0062] 6 is a diagram showing an example of the three-dimensional shape information of the refracting surface, which can be generated as shown in FIG.

[0063] (Shape estimation method) FIG. 7 is a flowchart showing the processing steps of the shape estimation method performed by the shape estimation device 100.

[0064] First, polarization camera 10 captures a first image of a subject when no transparent refractive layer is present, and a second image of the subject when a refractive layer is present (step S1).

[0065] Next, the first estimation unit 30 obtains a set of luminance values ​​corresponding to at least three different polarization angles from the second image, and calculates a polarization feature I max ,I min , Ψ is estimated (step S2).

[0066] Next, the second estimation unit 40 estimates the normal vector n i Two candidates for (n i + ,n i - ) is estimated (step S3).

[0067] Next, the feature amount acquiring unit 20 applies a feature point tracking method between the first image and the second image to acquire a distortion vector Δ g (Step S4).

[0068] Next, the refraction surface generator 50 calculates the distortion vector Δg Using the normal vector n i Candidate n i + ,n i - One of the pixels i is selected, and 3D shape information of the refractive surface is generated, which represents the surface shape of the refractive layer (step S5). The processes of steps S2 to S5 are repeated until they are completed for all pixels i (NO in step S6).

[0069] The processes in steps S2 to S5 are easily parallelized because they are performed independently for each pixel, which enables faster estimation of the three-dimensional shape.

[0070] As described above, the shape estimation device 100 according to this embodiment includes a polarization camera 10 that captures a first image of a subject when no transparent refractive layer is present and a second image of the subject when a refractive layer is present, and a feature point tracking method is applied between the first and second images to obtain a distortion vector Δ g and a feature acquisition unit 20 that acquires a set of luminance values ​​corresponding to at least three different polarization angles from the second image to acquire a polarization feature I max ,I min , Ψ, and a first estimation unit 30 that estimates the normal vector n i Two candidates for (n i + ,n i - ) and a second estimation unit 40 for estimating a distortion vector Δ g Using the normal vector n i Candidate n i + ,n i - and a refraction surface generating unit 50 that selects one of the above and generates refraction surface three-dimensional shape information that represents the surface shape of the refraction layer.

[0071] The shape estimation method according to this embodiment is a shape estimation method performed by the shape estimation device 100, in which the polarization camera 10 captures a first image of the subject when no transparent refractive layer is present and a second image of the subject when a refractive layer is present, and the feature acquisition unit 20 applies a feature point tracking method between the first and second images to acquire a distortion vector Δ g The first estimation unit 30 obtains a set of brightness values ​​corresponding to at least three different polarization angles from the second image to obtain a polarization feature quantity I max ,I min , Ψ, and the second estimation unit 40 estimates the normal vector n i Two candidates for (n i + ,n i - ) and the refraction surface generator 50 estimates the distortion vector Δ g Using the normal vector n i Candidate n i + ,n i - One of the above is selected, and three-dimensional shape information of the refractive surface is generated, which represents the surface shape of the refractive layer.

[0072] The shape estimation device 100 can be realized by a general-purpose computer system shown in Fig. 8. For example, in a general-purpose computer system including a CPU 90, a memory 91, a storage 92, a communication unit 93, an input unit 94, and an output unit 95, the CPU 90 executes a predetermined program loaded onto the memory 91, thereby realizing each function of the shape estimation device 100. The predetermined program can be recorded on a computer-readable recording medium such as an HDD, SSD, USB memory, CD-ROM, DVD-ROM, or MO, or can be distributed via a network.

[0073] As described above, the shape estimation device 100 and the shape estimation method according to the present embodiment enable estimation of the three-dimensional shape of a transparent refracting surface with a single camera, low computational complexity, and no prior knowledge of the shape (no prior learning data), which could not be realized in the past. In other words, by constructing a model that takes into account not only the geometric change in the light propagation path due to refraction but also the optical change (polarization) as an approach, it is possible to obtain constraints necessary for the three-dimensional shape of the refracting surface shape from information obtained only with a single camera. In addition, since it is possible to formulate it as a convex optimization problem and the search range for the solution is narrowed, it becomes possible to estimate the three-dimensional shape with a lower computational complexity than the conventional technology.

[0074] The present invention includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention-specifying matters according to the claims that are appropriate from the above description. [Explanation of symbols]

[0075] 10: Polarized camera 20: Feature acquisition unit 30: 1st estimation part 40:Second estimation part 50: Refraction surface generator 100: Shape estimation device

Claims

1. a polarized camera that captures a first image of a subject without a transparent refractive layer and a second image of the subject with the refractive layer; a feature amount acquiring unit that acquires a distortion vector, which is a feature amount that represents a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image; a first estimation unit that estimates a polarization feature by acquiring sets of luminance values ​​corresponding to at least three different polarization angles from the second image; A second estimation unit that estimates two candidates of a normal vector of the refractive layer to be estimated using the feature amount of polarization; a refraction surface generating unit that selects one of the candidates for the normal vector using the distortion vector and generates three-dimensional shape information of the refraction surface that represents a surface shape of the refraction layer; A shape estimation device comprising:

2. A shape estimation method performed by a shape estimation device, comprising: the polarized camera captures a first image of the object without a transparent refractive layer and a second image of the object with the refractive layer; a feature amount acquiring unit that acquires a distortion vector, which is a feature amount representing a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image; a first estimation unit that estimates a polarization feature by acquiring sets of luminance values ​​corresponding to at least three different polarization angles from the second image; The second estimation unit estimates two candidates of a normal vector of the refractive layer to be estimated using the feature amount of polarization; A refraction surface generation unit selects one of the candidates for the normal vector using the distortion vector, and generates refraction surface three-dimensional shape information representing the surface shape of the refraction layer. Shape estimation method.

3. A program for causing a computer to function as the shape estimation device according to claim 1.

Citation Information

Patent Citations

  • Image processor and pseudo-stereoscopic image creation device

    JP2010279044A

  • Image processing system, method, device and image format

    WO2008099589A1