A high-quality transparent object reconstruction method and apparatus

Abstract

The application discloses a high-quality transparent object reconstruction method and device, and inputs multiple RGB images and corresponding environment maps captured by using a mobile phone around a transparent object; and the signed distance field estimation model ImplicitNetwork, the differentiable intersection generation model SampleNetwork and the light field estimation model PhysicalRendering are optimized by using the implicit differentiable rendering mode on the input RGB picture set, so that a high-quality transparent object model is obtained. The high-quality explicit three-dimensional object grid can be extracted from the implicit function by using the isosurface extraction (MarchingCubes) method. The application can realize high-quality transparent object reconstruction based on a mobile phone, greatly reduces the cost of transparent object reconstruction, and improves the effect of transparent object reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for high-quality reconstruction of transparent objects. Background Technology

[0002] Acquiring 3D information from the real world has long been a crucial research area in computer graphics and computer vision. With the increasing demand and interest from researchers and the public in creating 3D models of real objects, driven by various applications such as the digital preservation of historical and cultural heritage and the needs of autonomous driving, research on 3D reconstruction of real objects has become increasingly active. Over the past few decades, researchers have proposed numerous different reconstruction algorithms for diffuse-reflective surfaces, such as fully automated scanning, multi-view stereo vision, and photometric stereo methods. Experiments show that these techniques are effective for non-transparent objects, but cannot be directly applied to the 3D reconstruction of transparent objects. This is because the interaction between transparent objects and light involves complex reflection and refraction relationships; even slight errors in geometry can lead to overall changes in appearance, making traditional 3D reconstruction algorithms ineffective for reconstructing transparent objects. 3D reconstruction of transparent objects, both in terms of data acquisition and 3D reconstruction algorithms, has long been considered a challenging problem. Meanwhile, transparent objects are ubiquitous in daily life, such as glass cups and crystal ornaments, and their reconstruction methods have important applications in augmented reality, industrial manufacturing, and medicine. Therefore, researching how to obtain high-quality transparent object models has significant and practical research value.

[0003] Current methods for transparent object reconstruction mainly fall into two categories: transparent object reconstruction based on traditional methods and transparent object reconstruction based on deep learning methods. Transparent object reconstruction based on traditional methods primarily includes two sub-methods: one is shape recovery from X-rays, including shape recovery from silhouettes, shape recovery from reflections, and shape recovery from infrared radiation. These methods either have low accuracy or require specialized equipment and are costly, making them difficult to apply directly to real-world scenarios. The second is tomography, which involves immersing the object to be reconstructed in solutions of different colors with similar refractive indices. This method is invasive and difficult to apply to the 3D reconstruction of valuable objects or cultural relics. Transparent object reconstruction based on deep learning methods uses multi-view stereo reconstruction methods (such as in the literature [Zhengqin, Li, et al. Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes. IEEE Conference Proceedings, Jan. 2020.]) for transparent object reconstruction. However, this type of method suffers from problems such as complex acquisition equipment, the need for a large number of images as input for optimization, and the difficulty in optimizing explicit point cloud representation methods. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a high-quality transparent object reconstruction method and apparatus.

[0005] The objective of this invention is achieved through the following technical solution: a method for reconstructing high-quality transparent objects, comprising:

[0006] (1) Take a series of RGB image sequences and corresponding environment maps around the transparent object, and obtain the camera pose, camera intrinsic parameters and the mask corresponding to the images;

[0007] (2) The signed distance field of a transparent object is modeled using a neural implicit differentiable function, and then the differentiable intersection of the incident light ray and the surface of the transparent object is obtained.

[0008] (3) The light path of light in a transparent object is modeled by the light field estimation model PhysicalRendering to obtain the light field information of the object; specifically: the incident light of the transparent object undergoes the first refraction and reflection at the differentiable intersection point with the surface of the transparent object, the first refracted light produces a second differentiable intersection point with the surface of the transparent object, and undergoes a second refraction at the second differentiable intersection point to obtain the outgoing light; the relationship between the directions of the incident light, the first refracted light and the second refracted light all follow Snell's law, and the relationship between the radiation of the incident light, the first refracted light, the first reflected light and the second refracted light all follow Fresnel's equation; the radiation is obtained by sampling on the environment map, and then the light field information of the incident light is obtained;

[0009] (4) Optimize the neural implicit differentiable function and light field estimation model on the RGB image sequence, and obtain the transparent object model based on the isosurface extraction method.

[0010] Furthermore, in step (1), the camera pose and camera intrinsic parameters corresponding to the captured image are estimated by the structure-of-motion method.

[0011] Further, in step (1), the RGB image sequence mask is extracted based on the image segmentation algorithm.

[0012] Further, in step (2), the signed distance field estimation model ImplicitNetwork is first used to learn the signed distance field of 3D points in the learning space and generate the intersection of light rays and transparent object surfaces. Then, the differentiable intersection point generation model SampleNetwork is used to represent the intersection points as differentiable functions of implicit geometry and camera parameters.

[0013] Furthermore, in step (3), the differentiable intersection point of the incident ray and the transparent object surface is calculated by the spherical tracing method, and the second differentiable intersection point of the first refracted ray and the transparent object surface is calculated by the ray propagation method.

[0014] Further, in step (4), the signed distance field of the transparent object, the image and mask generated by the light field information, and the original RGB image and the corresponding mask are used to calculate the hybrid loss function, which is used for backpropagation to update the implicit differentiable function parameters, optimize the signed distance field, and extract explicit geometry from the signed distance field as a three-dimensional transparent object model based on the isosurface extraction method.

[0015] Secondly, the present invention also provides a high-quality transparent object reconstruction apparatus, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the high-quality transparent object reconstruction method.

[0016] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for reconstructing a high-quality transparent object.

[0017] The beneficial effects of this invention are:

[0018] 1) The introduction of physical models solves the problem that implicit differentiable renderer methods cannot handle complex optical paths of transparent objects;

[0019] 2) Implicit differentiable renderers use signed distance fields to represent object geometry, which improves the accuracy of 3D model reconstruction. At the same time, implicit differentiable renderers based on physical models can obtain rendering results that are highly consistent with the input image, solving the problem that explicit models do not have enough expressive power.

[0020] 3) Eliminate the acquisition requirements of physical equipment and structured backlighting to achieve high-quality transparent object reconstruction based on ordinary RGB images. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the PhysicalRendering process of the light field estimation model in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of three-dimensional reconstruction of a transparent object according to an embodiment of the present invention.

[0025] Figure 4 This is a structural diagram of a high-quality transparent object reconstruction device according to the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0027] In the field of 3D transparent object reconstruction, traditional image-based methods either require complex acquisition equipment and environmental setups or are limited by the reconstruction capabilities of traditional multi-view geometric methods, making it impossible to reconstruct high-quality 3D transparent object models using portable devices. Implicit differentiable renderers, however, offer several advantages. First, they implicitly convert the explicit 3D model representation into the geometric properties of points in space using implicit functions, typically signed distance fields, thus improving the accuracy of the 3D geometry and offering greater flexibility compared to traditional explicit representation methods. Second, they decouple geometric and appearance information through the renderer to achieve more accurate 3D surface reconstruction. However, implicit differentiable renderers are generally only applicable to the reconstruction of non-transparent objects. Therefore, this invention combines a physical model with an implicit differentiable renderer, simulating the refraction and reflection of light based on Snell's law and Fresnel's equations, and sampling along the deflected light path to achieve high-quality transparent object reconstruction from multiple RGB images captured by a mobile phone.

[0028] The flowchart of the embodiment of the present invention is as follows Figure 1 As shown, the main steps are as follows:

[0029] Step 1: Capture a series of RGB images and corresponding environment images around the transparent object. Based on the matching relationship of feature points between the images, estimate the camera pose and camera intrinsic parameters corresponding to the captured images using the structure-from-motion (SFL) method. This step can be based on any existing SFL algorithm, such as the method in the literature Schonberger, Johannes L., and Jan-Michael Frahm. "Structure-from-motion revisited." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113. 2016.

[0030] Step 2: Extract RGB image sequence masks based on Mask-RCNN. A mask is a binary image indicating whether each pixel p is occupied by the object to be reconstructed. This step can be based on any existing image segmentation algorithm, such as the method described in He, K., Gkioxari, G., Dollár, P., & Girshick, R. "Mask R-CNN." In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969. 2017.

[0031] Step 3: Use neural implicit differentiable functions to directly model the signed distance field of the object, and then obtain the differentiable intersection point of the incident light ray and the transparent object surface. This will be explained in detail below.

[0032] First, the geometry of the transparent object is represented as the zero-level set of the ImplicitNetwork multilayer perceptron. Then, the signed distance field of the 3D points in the ImplicitNetwork learning space is estimated using the signed distance field:

[0033]

[0034] in, S represents the learnable parameters in an Implicit Network. θ Let represent the surface of the object, and f(χ; θ) represent the signed distance value of a 3D point χ. ImplicitNetwork is a multilayer perceptron, including fully connected layers.

[0035] At the same time, p represents a certain pixel, and R represents... p (τ)={C p +tv p / t≥0} represents the ray that intersects with p, where c represents the learnable parameters for camera parameter optimization. p =c p (τ) represents the camera center, v p =v p (τ) represents the direction of the light ray, using Represents the light ray R p and surface S θ The first intersection point. Generated via ray tracing. But the intersection point generated at this time It is not differentiable and cannot be optimized through subsequent steps.

[0036] Therefore, the differentiable intersection point generation model SampleNetwork is used to generate intersection points. Represented as a differentiable function of implicit geometry and camera parameters:

[0037]

[0038] Where x0 = c0 + t0v0 is the intersection point of the ray and the object surface under the current parameters θ0 and τ0. This is the derivative of the signed distance function with current parameters x0 and θ0. Also, The normal vector of surface Sθ at point Sθ can be calculated as follows:

[0039]

[0040] Step 4: Based on the intersection points of the light rays and the surface obtained from ImplicitNetwork and SampleNetwork, the light path of the light rays in the transparent object is directly modeled through the light field estimation model PhysicalRendering to obtain the light field information of the object. This will be explained in detail below.

[0041] like Figure 2 As shown, to simplify the problem and due to GPU memory limitations, this embodiment of the invention considers light paths with at most two collisions, i.e., light rays are reflected once by an object before reaching the environment map, or refracted twice by an object before reaching the environment map. Since light rays with more than two collisions are difficult to observe in a single view, two collisions are not a strict limitation.

[0042] In step 3, the intersection point p1 of the incident ray r1 and the object surface, and the surface S at p1 were obtained through ray tracing, ImplicitNetwork, and SampleNetwork. θ The normal vector is n1 = n1(θ, τ). The incident ray r1 undergoes its first refraction and reflection at the intersection point p1, resulting in the refracted ray r2 and the reflected ray r′2. Using the intersection point p1 as the new camera center and the refracted ray r2 as the new ray direction, a second ray tracing, ImplicitNetwork, and SampleNetwork processing are performed to obtain the second intersection point p2 of the incident ray r1 and the object, and the surface S at p2. θ The normal vector is n2 = n2(θ, τ). Ray r2 undergoes a second refraction and reflection at the intersection point p2, resulting in the final outgoing ray r3. The intersection point of the second refracted ray r3 with the environment diagram is Q, and the intersection point of the first reflected ray r′2 with the environment diagram is P. The relationship between the direction of the incident ray and the direction of the refracted ray follows Snell's law, which can be expressed as:

[0043] r i sinθ i =r j sinθ j , wheni=1, j=2 ori=2, j=3

[0044] Where, θ i For normal n i and light r i The angle between the incident ray, the reflected ray, and the refracted ray. The relationship between the radiation of the incident ray, the reflected ray, and the refracted ray follows Fresnel's equations, which can be expressed as:

[0045]

[0046]

[0047] wheni=1,j=2ori=2,j=3

[0048] Among them, L i L j L j For the incident ray r i Refracted ray r j and reflected light r j The radiation, where η is the refractive index of the current object. Where L... i This can be obtained by sampling on the environment map E, i.e.:

[0049] L i =sampleEnvLight(E, r i )

[0050] The final incident ray r1 presents the following light field color:

[0051] L(θ, τ) = L′2 + L3

[0052] It's important to note that the ray tracing method used to calculate the initial intersection point p1 for the incident ray r1 is the spherical ray tracing method, while the ray tracing method used to calculate the second intersection point p2 for the first refracted ray r2 is the forward ray tracing method. The difference between the two ray tracing methods is as follows:

[0053] Because surface S θ By defining the zero level set of the signed distance field, a spherical tracing method with fast indexing can be directly used to calculate the first intersection point. That is, in each iteration, since the distance from the incident ray to the intersection point x0 is limited by the current signed distance f(c+tv), f(c+tv) can be used as the forward radius to advance t, iterating continuously until the signed distance is less than the threshold of 5e-5 or the maximum number of steps (10) is reached. However, when the starting point is near the internal surface of the object, the signed distance of that point is less than 0, making it unsuitable as a radius, and the absolute value of the signed distance is very small, making it difficult to determine the forward step size and direction. Therefore, this embodiment of the invention replaces the ray tracing method with a ray-forward method, that is, advancing along the ray direction with a fixed step size until the signed distance value is greater than 0 or the maximum number of iterations is reached.

[0054] Step 5: Optimize the signed distance field estimation model ImplicitNetwork, the differentiable intersection generation model SampleNetwork, and the light field estimation model PhysicalRendering on the input RGB image sequence to obtain a high-quality transparent object model, such as... Figure 3 As shown.

[0055] As described in step 3, the signed distance field of the object obtained after processing by ImplicitNetwork, SampleNetwork, and PhysicalRendering, the image and mask generated by the light field information, and the original RGB image and corresponding mask are used together to calculate the mixing loss function, which is then used for backpropagation to update the implicit differentiable function parameters.

[0056] loss(θ, τ) = loss RGB (θ,τ)+ρloss MASK (θ,τ)+λloss E (θ)

[0057] in, For surface-related network parameters, For all camera-related parameters. loss RGB The RGB loss function is expressed as:

[0058]

[0059] Where P represents all pixels, I p ∈[0, 7] 3 Let be the RGB value of each pixel p∈P in the original RGB image, and |·| denote the L1 norm. loss MASK The mask loss function is expressed as:

[0060]

[0061] Among them, O p ∈[0,1] represents the value of the mask corresponding to the original RGB image, S p，α (θ, τ) is the mask of the generated pixel p, CE is the cross entropy, and α is used to constrain the fineness of the shape boundary. E Let the signed distance loss function be used. The signed distance field is optimized by requiring it to approximate a signed distance function with implicit geometric regularization, expressed as:

[0062]

[0063] in, Let represent the first derivative of the signed distance field function at x. This loss function is implemented using Eikonal regularization, where x is distributed on the surface of the object.

[0064] Step 6: Use the isosurface extraction method to extract explicit geometry from the signed distance field as a high-quality 3D transparent object model.

[0065] Corresponding to the aforementioned embodiment of a high-quality transparent object reconstruction method, the present invention also provides an embodiment of a high-quality transparent object reconstruction apparatus.

[0066] See Figure 4 The present invention provides a high-quality transparent object reconstruction device, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a high-quality transparent object reconstruction method in the above embodiment.

[0067] An embodiment of the high-quality transparent object reconstruction device provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is the high-quality transparent object reconstruction device provided by the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0068] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0069] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0070] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a high-quality transparent object reconstruction method as described in the above embodiments.

[0071] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0072] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for high-quality transparent object reconstruction, characterized in that, include: (1) Take a series of RGB image sequences and corresponding environment maps around the transparent object, and obtain the camera pose, camera intrinsic parameters and the mask corresponding to the images; (2) The signed distance field of a transparent object is modeled using a neural implicit differentiable function, and then the differentiable intersection point of the incident light ray and the surface of the transparent object is obtained. Specifically, the signed distance field estimation model ImplicitNetwork is used to learn the signed distance field of 3D points in the space and generate the intersection points of light rays and transparent object surfaces. Then, the differentiable intersection point generation model SampleNetwork is used to represent the intersection points as differentiable functions of implicit geometry and camera parameters. (3) Based on the intersection points of the light rays and the surface obtained from ImplicitNetwork and SampleNetwork, the light path of the light rays in the transparent object is modeled by the light field estimation model PhysicalRendering to obtain the light field information of the object; specifically: the incident light rays of the transparent object undergo the first refraction and reflection at the differentiable intersection point with the surface of the transparent object, the first refracted light rays produce the second differentiable intersection point with the surface of the transparent object, and undergo the second refraction at the second differentiable intersection point to obtain the outgoing light rays; The relationship between the directions of the incident ray, the first refracted ray, and the second refracted ray all follow Snell's law, and the relationship between the radiation of the incident ray, the first refracted ray, the first reflected ray, and the second refracted ray all follow Fresnel's equation; the radiation is obtained by sampling on the environmental map, and thus the light field information of the incident ray is obtained; (4) Optimize the neural implicit differentiable function and light field estimation model on the RGB image sequence, and obtain the transparent object model based on the isosurface extraction method; specifically: calculate the hybrid loss function based on the signed distance field of the transparent object, the image and mask generated by the light field information, the original RGB image and the corresponding mask, and use it for backpropagation to update the implicit differentiable function parameters, optimize the signed distance field, and extract the explicit geometry from the signed distance field as the three-dimensional transparent object model based on the isosurface extraction method.

2. The method for reconstructing a high-quality transparent object according to claim 1, characterized in that, In step (1), the camera pose and camera intrinsic parameters corresponding to the captured image are estimated by the structure-of-motion method.

3. The method for reconstructing a high-quality transparent object according to claim 1, characterized in that, In step (1), the RGB image sequence mask is extracted based on the image segmentation algorithm.

4. The method for reconstructing a high-quality transparent object according to claim 1, characterized in that, In step (3), the differentiable intersection point of the incident ray and the transparent object surface is calculated by the spherical tracing method, and the second differentiable intersection point of the first refracted ray and the transparent object surface is calculated by the ray propagation method.

5. A high-quality transparent object reconstruction apparatus, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a high-quality transparent object reconstruction method as described in any one of claims 1-4.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a high-quality transparent object reconstruction method as described in any one of claims 1-4.

Images