A method for high-quality semi-transparent object reconstruction based on implicit differentiable rendering

Abstract

This invention discloses a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering. First, multiple 2D images are captured around a homogeneous, smooth metal sphere placed in an environment and the target semi-transparent object to be reconstructed. Next, an ambient lighting map is generated using the 2D image of the metal sphere, and the camera pose information corresponding to each 2D image of the target semi-transparent object is extracted. Then, a semi-transparent object reconstruction network, TO-NeuS, is constructed, using the 2D image data captured around the target object as labels and sampling according to the camera pose information to obtain an implicit representation of the geometric information of the target object. Finally, a light field model is constructed, combining the implicit representation and the ambient lighting map to obtain the reflected and transmitted color components of the pixels in the semi-transparent object image, and the pixel colors of the reconstructed image are calculated. This invention overcomes the limitation of semi-transparent object reconstruction being dependent on specific environments, significantly improving reconstruction results while reducing computational load.

Description

Technical Field

[0001] This invention belongs to the fields of industrial modeling, artificial intelligence and computer vision technology, and specifically relates to a method for high-quality semi-transparent object reconstruction based on implicit differentiable rendering. Background Technology

[0002] Multiview Figure 3 3D object reconstruction is a crucial research area in computer graphics and computer vision. Driven by researchers' and the public's pursuit of realism in 3D models of real-world objects, and by the applications of 3D models in various industries such as digital twins, autonomous driving, and large-scale AAA games, research on 3D reconstruction of real objects is increasingly active. While numerous solutions have been proposed for the 3D reconstruction of opaque objects with diffuse reflection properties, such as multi-view stereo vision, photometric stereo methods, and volume rendering-based neural reconstruction, there are fewer technical solutions for the shape reconstruction of semi-transparent objects. Furthermore, due to the complex optical properties of semi-transparent objects, reconstruction methods for opaque objects cannot be directly applied to them. Meanwhile, many semi-transparent objects exist in the world (such as glass, plastic, crystal, and diamonds), making rapid reconstruction of semi-transparent objects crucial for augmented reality and digital twins. Therefore, researching how to obtain high-quality transparent object models has significant and practical research value.

[0003] Reconstruction of translucent materials based on differentiable rendering can be broadly categorized into two types: invasive and non-invasive methods. Invasive methods involve physical interventions on the surface during reconstruction, such as contact, spraying, or immersion. For example, immersing the object in a liquid with a known refractive index to acquire the original image. These methods have several limitations, such as being unable to perform large-scale reconstructions or causing irreversible damage to the object being reconstructed, such as important cultural relics. Non-invasive methods, on the other hand, require image acquisition in specific environments or backgrounds, such as in a darkroom or a room without additional light sources. This makes the reconstruction of translucent objects require complex acquisition equipment and is easily subject to space constraints. Summary of the Invention

[0004] To accurately reconstruct the internal optical properties of translucent materials and reduce the complexity of equipment and procedures during data acquisition, this invention provides a high-quality translucent object reconstruction method based on implicit differentiable rendering. This method utilizes limited viewpoint information to precisely reconstruct the object's shape from an unknown viewpoint. The method includes the following steps:

[0005] S1. Take multiple two-dimensional images around the homogeneous, smooth metal sphere placed in the environment and the target reconstructed translucent object.

[0006] S2. Generate an ambient lighting map using the two-dimensional image data of the metal sphere in S1.

[0007] S3. Extract the camera pose information corresponding to each 2D image of the semi-transparent object reconstructed from S1.

[0008] S4. Using the NeuS neural network as the basic model, construct the semi-transparent object reconstruction network TO-NeuS. Use the two-dimensional image data captured around the target object as labels, sample the spatial environment according to the camera pose information, and obtain the implicit expression of the geometric information of the target object through TO-NeuS: surface density weights, surface feature parameters and internal feature parameters of the spatial sampling points.

[0009] S5. Construct a light field model, combine the implicit representation output by the TO-NeuS network in S4 with the volume rendering method, obtain the reflection color component and transmission color component of the pixels of the semi-transparent object image, and calculate the pixel color of the final reconstructed image to obtain the reconstructed image.

[0010] Furthermore, in step S1, a homogeneous, smooth metal sphere is placed in the center of the environment, requiring that the surface of the sphere completely reflects the surrounding environment. A set of 2D images is collected around the sphere, capturing the same scene from different angles and positions, so that the photos completely cover all positions of the sphere, and the position and size of the sphere appearing in all photos are as consistent as possible. After replacing the metal sphere with the semi-transparent target object to be reconstructed, another set of 2D images is collected in the same manner.

[0011] Further, in step S2, a spherical coordinate system and an ambient lighting map planar coordinate system are established. Sampling points on the spherical coordinate system are mapped to target points in the ambient lighting map, using pixels as the unit, to obtain a complete ambient lighting map. A spherical coordinate system is defined, with the center of the sphere as the origin, azimuth angle θ, and elevation angle Φ. Illumination is defined. Figure 2 The spherical coordinate system has its origin at the top left corner of the image, with the positive direction of the horizontal coordinate axis to the right and the positive direction of the vertical coordinate axis downwards. To map the data from the spherical coordinate system to the ambient lighting map, this invention uses equidistant cylindrical projection to unfold the spherical information into a rectangular two-dimensional image, which is then mapped from the spherical coordinate system to the ambient lighting map's planar coordinate system.

[0012] Furthermore, in step S3, a set of 2D images of the target semi-transparent object are collected and then processed using Colmap software to obtain camera viewpoint parameters, including the camera's position and orientation. These parameters are used to determine the sampling rays emitted from the camera position, passing through each pixel in the image, into the 3D scene.

[0013] Furthermore, in step S4, a TO-NeuS network based on an improved NeuS network is constructed. This is achieved by adding a material network for learning semi-transparency effects and a refraction probability network to the NeuS network to calculate refraction probability features. After calculating the loss function, an optimizer is used to iteratively adjust the network parameters, ultimately enabling the network to implicitly represent the geometry and appearance of objects. Specifically:

[0014] S41. Using camera pose information and a 2D image of the target object, ray tracing and Sphere-Tracing algorithms are employed. With the camera position set as the origin of the ray, a sampled ray is emitted along the direction from the camera to each pixel in the 2D image. Based on the refraction probability characteristics, the propagating and emitting rays are further obtained. This process is repeated for each pixel in the 2D image of the target object, ultimately constructing a complete set of sampled rays. Since total internal reflection and multiple refraction rays have little impact on the rendering effect, only the case of secondary refraction is considered when generating the sampled ray set to reduce the overall network complexity.

[0015] S42. Using the symbolic step algorithm, obtain the incident point P on the i-th sampling ray in step S41. i1 Exit point P i2 And generate sampling points according to the following strategy: Let P i1 and P i2 The magnitude of the vector is I, and... The interval is P i1 Using the midpoint as the reference point, sample N points before and after it, P i2 N points were sampled in the same way before and after.

[0016] S43. Construct a TO-NeuS network. The spatial locations of the sampling points generated in S42 are integrated into a multilayer perceptron via IPE location encoding. The location encoding is then fed into the intermediate layer of the multilayer perceptron via residual connections. The viewpoint direction (θ, φ) information, encoded by a spherical harmonic function, is fed into the last fully connected layer. The surface density weights w of the sampling points are obtained in the penultimate and last layers of the perceptron, respectively. surf and SDF value f G (x); then f G (x) are input into two color networks identical to the NeuS model, respectively, to obtain the object surface feature parameters C. surf and internal characteristic parameter C in .

[0017] S44, P i1 P i2 The two outputs P are obtained by inputting them into the coding layer respectively. i11 and P i22 , will P i11As the input to the first layer of a multilayer perceptron, P i22 As the input to the intermediate layer of the multilayer perceptron, the output of the multilayer perceptron is convolutionally pooled to obtain the refraction probability feature η.

[0018] S45, Obtain the surface density weights w at the sampling points surf Surface characteristic parameter C surf Internal characteristic parameter C in The implicit expression of the geometric information of the target object, consisting of the refraction probability feature η.

[0019] Furthermore, step S5 will specifically include the design of a light field model structure that follows the physical properties of the translucent material. Since the color of the translucent material is formed by the superposition of multiple physical effects, a parameterized light field model needs to be established to model the light path of light rays within the translucent object, thereby calculating the viewpoint color at a specific incident direction at a given spatial point. Specifically:

[0020] S51, The light field model is based on the ambient illumination map and surface density weights w. surf Surface characteristic parameter C surf Internal characteristic parameter C in The reflected color component and the transmitted color component are obtained by calculating the refraction probability characteristic η.

[0021] The reflected color component is used to fit the diffuse reflection and specular reflection phenomena produced by light on the object surface. After sampling the ambient lighting map in S2 using the refraction probability feature η by the ray tracing algorithm, a BRDF model based on microplanes is introduced to describe and express the reflection phenomenon of the object surface.

[0022] Transmitted color components are used to describe the refraction and scattering effects of ambient light in semi-transparent objects. This method introduces a scattering factor and proposes a density scattering model for semi-transparent object reconstruction to describe the effect of the object's own color and ambient color after passing through the semi-transparent object, and calculates the transmitted color components. This density scattering model uses a Laplacian distribution and introduces a scattering factor to simulate the scattering and transmission effects of light within the semi-transparent object. The surface density weights w of sampling points inside the object are calculated using a trainable scattering factor when rendering the transmitted color components. surf Combined with surface feature parameter C surf and internal characteristic parameter C in weighted by surface density w surf The transmitted color components are obtained by summing them.

[0023] S52. Sum the reflected and transmitted color components corresponding to all sampling points on a certain sampling ray according to weights to obtain the final viewpoint color, that is, the color of a certain pixel in the reconstructed image. Obtain the colors of all pixels in the reconstructed image to obtain the reconstructed image.

[0024] The specific beneficial effects of this invention are as follows:

[0025] 1. Overcoming environmental limitations, enhancing reconstruction results, and reducing computational load: This invention overcomes the limitation of semi-transparent object reconstruction being dependent on specific environments. It utilizes deep neural networks to implicitly represent the geometric shape of semi-transparent objects, while introducing ray tracing and global illumination techniques during the reconstruction process to simulate natural light paths. This significantly enhances reconstruction results while reducing computational load.

[0026] 2. Parametric Physical Effects for Effective Simulation of Translucent Material Characteristics: This invention models the refractive index using a deep neural network and introduces a scattering factor to construct a light field model capable of simulating the refraction and scattering of translucent materials. This model can accurately and efficiently handle the dynamic changes in density weights caused by optical effects as light passes through different sampling points. Furthermore, by parameterizing complex reflection and refraction effects, the multilayer perceptron can learn the refractive effects of translucent materials, thereby effectively simulating the physical effects.

[0027] 3. Non-invasive data acquisition, reducing costs and improving reconstruction realism: The original data acquisition of this invention is not limited by the environment and adopts a non-invasive acquisition method. This not only reduces acquisition costs and steps while protecting the target object to be reconstructed, but also, because the acquired data includes environmental information, makes the final reconstruction effect closer to the expression of the real scene. Even when faced with complex semi-transparent objects, detailed reconstruction can be completed. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the high-quality semi-transparent object reconstruction method based on implicit differentiable rendering of the present invention.

[0029] Figure 2 This is a schematic diagram illustrating the data acquisition environment and method of the present invention;

[0030] Figure 3 This is a schematic diagram of the TO-NeuS neural network in this invention;

[0031] Figure 4 This is a schematic diagram of the refraction probability feature neural network in this invention;

[0032] Figure 5 This is a schematic diagram illustrating the refraction of sampled light by a semi-transparent object in this invention. Detailed Implementation

[0033] The following detailed description provides further details on specific implementation methods.

[0034] To minimize the limitations imposed by the complex optical properties inside semi-transparent objects and reduce the dependence of the reconstruction process on the environment, this invention provides a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering.

[0035] The technical solution adopted by this invention to solve its technical problem is: a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering, the process of which is as follows: Figure 1 As shown, the method includes the following steps in sequence:

[0036] (1) Raw data acquisition: Take RGB images around a homogeneous, smooth metal sphere placed in the environment; take RGB images around the semi-transparent target object to be reconstructed; the surrounding shooting angle should cover as many azimuth angles of the object as possible, such as... Figure 2 As shown.

[0037] (2) Image preprocessing: Using the image information acquired in the previous step, calculate the ambient lighting map of the metal sphere photo data, and generate the camera pose information and llff format information corresponding to the target object image.

[0038] (3) Data Training: Using the NeuS neural network as the basic model, a semi-transparent material target reconstruction model TO-NeuS based on NeuS was constructed. Sampling rays were obtained using camera pose information and ray tracing algorithms, and spatial sampling points were generated. The spatial position and viewpoint direction of the sampling points were encoded and fed into the TO-NeuS network for training, predicting and calculating various feature parameters of the reconstructed object: surface density weight w surf SDF value f G (x), surface characteristic parameter C surf and internal characteristic parameter C in .

[0039] (4) Establish a trainable light field model to reconstruct an image of an object from an unknown viewpoint: Calculate the reflection and transmission color components of the sampling points using the parameters obtained in step (3), and calculate the viewpoint color based on the weighting factors in step (3). The two color components are calculated using their respective sampling models. The reflection color component is calculated using a physics-based micro-surface model based on the feature parameters in step (3). The object's own color is predicted by the network in step (3), and the background environment color is obtained by sampling the ambient lighting map based on the sampling light. The transmission color component is obtained by weighting the color of the light passing through the object and the background environment color. Given that translucent objects exhibit complex refraction and scattering physical phenomena during light propagation, a scattering factor and refraction probability feature are introduced, and a density scattering model specifically for the reconstruction of translucent objects is constructed. This model can accurately and efficiently handle the dynamic changes in density weights caused by refraction and scattering when light penetrates different sampling points. The reflection and transmission color components corresponding to all sampling points on a certain sampling light are summed according to the density weights to obtain the final viewpoint color, i.e., the color of a pixel in the reconstructed image.

[0040] Step (1) specifically includes the following steps:

[0041] A homogeneous, smooth metal sphere is placed in the center of the environment, ensuring it completely reflects the surroundings. A series of 2D images are collected around the sphere, capturing the same scene from different angles and positions, ensuring the images completely cover all parts of the sphere, and that the sphere's position and size are as consistent as possible across all images. After replacing the metal sphere with a semi-transparent target object to be reconstructed, another series of 2D images is collected in the same manner.

[0042] Obtain the ambient lighting map. Establish a spherical coordinate system and an ambient lighting map planar coordinate system, and map the sampling points between the two. Define a spherical coordinate system with the center of the sphere as the origin, the azimuth angle as θ (θ∈[0,2π]), and the elevation angle as φ (φ∈[0,π]). Define the lighting... Figure 2 In this method, the ambient lighting map is constructed using an equidistant cylindrical projection to unfold the spherical information into a rectangular two-dimensional image. The following is the mapping formula for converting spherical coordinates to ambient lighting map coordinates:

[0043]

[0044] Collect a set of 2D images that capture the same scene from different angles and positions. These images are all taken of the same 3D scene. Therefore, in this 3D scene, each object will appear in multiple images, only with different viewpoints and positions.

[0045] The system obtains the camera parameters corresponding to each 2D image, including the camera's position and orientation; and uses colmap to obtain the camera's viewpoint parameters. Colmap is a method specifically designed for obtaining camera pose information from images. It retrieves detailed information for each camera image, including (x, y, z, θ, φ), where x, y, and z are the coordinates of the three image directions, and θ and φ are the viewing directions. These parameters are then used to determine the direction of a ray emitted from the camera position, passing through each pixel in the image, and projected into the 3D scene.

[0046] Step (3) specifically includes the following steps:

[0047] like Figure 5 As shown, ray tracing is used to calculate sampled rays. Before training, the SDF values ​​of the sampled points are randomly assigned. Subsequently, the neural network is iteratively trained to make the SDF values ​​approximate the geometry of the object. In this method, the geometric information of the object is implicitly represented by the neural network in the form of SDF. The concept of volume rendering is used to discretize the target object into sampled points in space, and the geometric and material information of the object is expressed through the various parameters of the sampled points.

[0048] The i-th sampled ray L can be obtained from the SDF data using the symbolic stepping algorithm. i The point of incidence P i1 Exit point P i2 The sampled ray is divided into three parts by the incident and exit points: the incident ray L... in Light transmission L trans and emitted light L out Sampling points are generated on the sampling rays according to a fixed strategy. For each sampling ray L... i Let I = |P i1 P i2 |, with The interval is P i1 Using the midpoint as the reference point, sample N points before and after it, P i2 Sample N points in the same way. Because for a semi-transparent object, when f... G When (x)>0, there is no effect on light attenuation. Therefore, sampling is only performed on the front and back of the object surface to improve sampling accuracy while reducing computation. For each pixel on the two-dimensional image in step (2), a ray is first determined, and then a series of spatial points are sampled on the ray in the manner described above.

[0049] Furthermore, this method constructs a TO-NeuS network for implicitly representing the geometry and appearance of objects. The network structure is as follows: Figure 3 As shown, following the NeuS network structure, the three-dimensional spatial coordinates (x, y, z) of the sampling points constructed above are input. After IPE integrated position encoding, the surface density weights w of the sampling points are obtained at different stages through a 9-layer 256-dimensional MLP network. surf SDF value f G (t i In this design, after the reflection density weights are output at the ninth layer of the network, the viewpoint direction (θ, φ) information encoded by a spherical harmonic function is added to enhance the model's learning and reconstruction of illumination and reflection information under different viewpoints. Here, a third-order (quadratic) spherical harmonic function is used to encode the viewpoint direction (θ, φ), that is, 9-dimensional spherical harmonic coefficients are used to replace the sine and cosine functions encoded in NeRF as input to the network, increasing the viewpoint dependence of the model and thus realizing the "Fresnel effect" in real-world scenes. Two more 4-layer, 256-dimensional material networks are then constructed, and f... G (t i The surface feature parameters C are obtained by training the material network. surf and internal characteristic parameter C in Each iteration of training involves a sampling operation based on the latest trained network information.

[0050] Furthermore, this method proposes refraction probability features to replace the constant refractive index of an object, and uses learnable parameters to fit the refraction phenomenon of light within the object. The specific network structure is as follows: Figure 4 As shown, the spatial coordinates P of the sampling point are... i1 The (x,y,z) 3D vector, after undergoing IPE integrated position encoding and dilation operations, is input into an MLP consisting of seven 256-dimensional fully connected layers (FC) to determine the degree of refraction of the sampled point after passing through the object. To enhance the connection between the incident and exit points, a residual structure is added to the fourth layer of the network, along with the input of the exit point P. i2 The three-dimensional coordinates (x, y, z) are used to extract the refraction probability feature η through a convolutional pooling operation, thus narrowing the range of the overall network output data.

[0051] Step (4) specifically includes the following steps:

[0052] A light field model is established that follows the physical properties of semi-transparent materials. The light field model can model the light path of light in a semi-transparent object, thereby calculating the viewpoint color, that is, the color of a certain pixel in the reconstructed image. The viewpoint color value contains two color components: reflected color and transmitted color. It can be calculated by modeling the light path and using the parameters of the sampling points obtained in step (3). The light rays in the modeled light path are called sampling rays. Since total internal reflection and multiple refraction have little impact on the rendering effect, in order to reduce the complexity of the model and simplify the calculation, only the case of secondary refraction is considered in this calculation model.

[0053] Furthermore, in ray tracing, according to P i1 Point location, P i2 The point position and refraction probability characteristic η can be used to calculate the analytical expressions for three types of sampled rays: incident ray, ray transmitted through the object, and outgoing ray.

[0054] L in =O+|OP i1 |·t

[0055] L trans =P i1 +|OP i1 |·η·t

[0056] L out =P i2 +|P i1 P i2 |·η * ·t

[0057] Where O is the location of the camera viewpoint, η * The outgoing deflection probability characteristic is derived from rotating the rotation matrix in the negative direction of the refraction probability characteristic η.

[0058] Furthermore, the background color is calculated based on the three types of sampled rays and the ambient lighting map. First, the spherical coordinate system is converted to a Cartesian coordinate system, considering P... i2 Calculate the point r where the light rays intersect the unit sphere centered at the center. Map r on the unit sphere onto the ambient lighting map using spherical projection to obtain the background color C. background P i1 Points and reflected rays are sampled on the ambient lighting map in the same manner as above to obtain C. env Thus, the transmitted color can be represented as:

[0059] C in =μC background +(1-μ)C in

[0060] Where μ is a hyperparameter that controls the contribution of the background color to the transmitted color in the transmitted color component.

[0061] Reflected color r efelct The calculation is based on a direct lighting model consisting of all rays reflected from the object at first. A micro-surface model is introduced to enhance the physical effects, and the BRDF value of the specular reflection portion is calculated. The specific expression is as follows:

[0062]

[0063] In this formula, the half-range vector is defined. F is the Fresnel term describing the reflection effect based on physical laws, and it relates to the half-range vector h and the outgoing ray ω. o The two-parameter function has the following expression:

[0064] F(c,ω0,ω i )=c+(1-c)(1-(ω o ·ω i )) 5

[0065] In the expression, c represents the speed of light.

[0066] G is the geometric compensation term, used to describe the shadow effect caused by the occlusion of an object. The specific expression is as follows:

[0067]

[0068] D is the normal distribution function, which describes the normal distribution in the microplane. Here, the GGX distribution is used to describe the roughness of the object surface. The specific expression is as follows:

[0069]

[0070] Here, α is the roughness parameter, which is input into the network as a hyperparameter during subsequent training to describe the roughness of the object.

[0071] The diffuse BRDF value is obtained using the Lambertian diffuse reflection model and the ambient color of the reflection environment in step (3):

[0072] The final reflected color components can be expressed as:

[0073] C reflect =(f d +f s )×C env ×(N·ω o )

[0074] Where N is the normal vector of a point on the object's surface, which can be calculated by sampling based on the SDF value. By sampling the SDF values ​​of neighboring spatial points, the normal vectors of the neighboring triangular surfaces are obtained, and the normal vector of the point is calculated according to the area weight.

[0075] To enable the network output to represent the scattering and refraction of light from semi-transparent objects, this invention draws on the work of NuralTO and introduces a scattering factor to establish a density scattering model, which is used to obtain the internal density weights w. in The specific details are as follows:

[0076] Because the NeuS network only optimizes the non-zero weights of sampling points near the surface of opaque objects, the weights inside the object are zero. Furthermore, the scattering ability of light inside a semi-transparent object is not uniform. Therefore, to overcome the limitations of NeuS, this method uses a physical model following the principle of radiative light to simulate the density inside the object, thus fitting the real-world effect of light dissipation in a semi-transparent object. This method borrows the Laplace distribution function to represent the density field inside a semi-transparent object.

[0077]

[0078] Where f G (x) represents the SDF value of the spatial point x predicted by the neural network, σ t β represents the attenuation coefficient of light energy inside an object. The larger the attenuation coefficient, the faster the light energy dissipates inside the object, and the lower the transparency. When β approaches 0, the sampling point density inside the object approaches σ, and the spatial attenuation coefficient outside the object approaches 0. t =t i+1 -t i .

[0079] For physical plausibility, in w in In the calculation, the sampling interval needs to be redefined. The sampling interval is limited to the incident position P. i1 (x,y,z) and the exit position P i2 Let R be the set of sampling points between (x, y, z).

[0080] R = {t} i |f G (t i )·f G (t i+1 )<0)}

[0081] In R, all sampling points and their adjacent ascending sampling points are located inside and outside the surface of the target reconstructed object, respectively. The set Ω of intersection points can be defined using set R:

[0082]

[0083] Where the parameter sampling point t i Derived from the above set R, f G (t i ) represents the spatial point t predicted by the neural network. i The SDF value at the location. Based on the above range, the sampling point t can be determined. i The corresponding weights are calculated using the following formula:

[0084]

[0085] Where t i For the sampling points in the above set Ω, σ t δ represents the sampling point density values ​​needed for neural network training. t For sampling point t i The Euclidean distance to its adjacent descending sampling points This represents the minimum value within the aforementioned sampling area.

[0086] Furthermore, the transmitted color component, which includes the refraction and scattering effects of light within an object, can be calculated using the weighted summation of the sampling points along a certain ray. This calculation is performed using the following expression:

[0087]

[0088] Where C trsns The transmitted color, denoted by C, is a two-parameter function relating the sampling point p(t) and the sampling ray direction vector d, after the sampled ray penetrates the object. trsns The overall color can be divided into surface color and internal color, both of which are adjusted by a hyperparameter γ. The w in the formula... surf C represents the density weights of the sampled points predicted by the TO-NeuS network. surf The surface feature parameters output by the TO-NeuS network; w in The sampling point t calculated for the density scattering model i The corresponding weight, C in These are the internal characteristic parameters of the object after the ambient light dissipates. This represents the maximum value of the above sampling area.

[0089] Ultimately, the pixel colors of the reconstructed image of a semi-transparent object under other unknown viewpoints can be calculated by controlling the reflected and transmitted color components.

[0090] C pixel =(1-κ)κ Creflect +κ Ctrans

[0091] Where κ represents the contribution of environmental reflection to the object's surface. When the object's specular reflection effect is strong, a higher value is needed to enhance the influence of the environment on the object's surface.

[0092] To verify the effectiveness of the TO-NeuS network, a comparative experiment was conducted on the Syn-Trans dataset, which includes four scenes: GummyBear, Stanford Dragon, Yuanbao, and Ancient Dragon. The chamfer distance was used as the evaluation metric. The experimental data are shown in Table 1.

[0093] Table 1

[0094]

[0095] In summary, this invention proposes a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering. It not only acquires object surface information non-invasively, overcoming environmental dependence, but also effectively simulates the surface reflection and internal transmission characteristics of objects after parameterizing physical effects, making the reconstruction results closer to real-world phenomena. Furthermore, the introduction of scattering factors and refraction probability features is another feature that distinguishes this invention from other reconstruction algorithms. It effectively restores the refraction phenomenon of light within semi-transparent objects to a certain extent. Although there are slight deviations in physical details, the error range is controlled within a range that is difficult for the human eye to discern from the real image, thus ultimately yielding excellent reconstruction results.

Claims

1. A method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering, characterized in that, Includes the following steps: S1. Take multiple two-dimensional images around the homogeneous smooth metal sphere placed in the environment and the target reconstructed translucent object respectively; S2. Generate an ambient lighting map using a two-dimensional image of the metal sphere; S3. Extract the camera pose information corresponding to each 2D image of the target reconstructed semi-transparent object; S4. Construct a semi-transparent object reconstruction network TO-NeuS, using 2D image data captured around the target object as labels, sampling the spatial environment based on camera pose information, and obtaining the implicit representation of the geometric information of the target object through TO-NeuS. The TO-NeuS network comprises the following two parts: The first part follows the NeuS network structure, inputting the constructed three-dimensional coordinates of the sampling points. After IPE integrated location encoding, the surface density weights of the sampling points are obtained at different stages through an MLP network. SDF value In this process, after the surface density weights are output in the last layer of the MLP network, the viewpoint direction encoded by a spherical harmonic function is added. Information, here a third-order spherical harmonic function is used to represent the viewpoint direction. Encoding is performed by replacing the sine and cosine functions in NeRF with spherical harmonic coefficients as the encoding input to the MLP network; Two more MLP networks are constructed as material networks, and then... Surface feature parameters are obtained by training the material network. and internal characteristic parameters ; The second part uses refraction probability features to replace the constant refractive index of the object, and employs learnable parameters to fit the refraction phenomenon of light inside the object. Specifically, it uses the spatial coordinates of the sampling points... The 3D vector, after undergoing IPE integrated position encoding and dilation operations, is input into an MLP network to obtain the degree of refraction of the sampling point after passing through the object. A residual structure is added to the intermediate layer of the MLP network, and the exit point is also input. The three-dimensional coordinates are then used to extract the refraction probability features through a convolutional pooling operation. ; S5. Construct a light field model, combine the implicit representation output by the TO-NeuS network with the ambient lighting map to obtain the reflection color component and transmission color component of the pixels in the semi-transparent object image, and calculate the pixel color of the reconstructed image to obtain the reconstructed image.

2. The method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: a homogeneous and smooth metal sphere is placed in the center of the environment, and the surface of the sphere completely reflects the surrounding environment; a set of two-dimensional 2D images are collected around the sphere. These images capture the same scene from different angles and positions, so that the photos completely cover all positions of the sphere, and the position and size of the sphere appearing in all photos are consistent; after replacing the metal sphere with the semi-transparent target object to be reconstructed, another set of 2D images is collected in the same way.

3. The method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering according to claim 2, characterized in that, The specific process of step S2 is as follows: establish a spherical coordinate system and an ambient lighting map planar coordinate system, and map the sampling points on the spherical coordinate system to the target points in the ambient lighting map in units of pixels to obtain a complete ambient lighting map. The establishment of the spherical coordinate system and the ambient lighting plane coordinate system is specifically as follows: Define a spherical coordinate system with the center of the sphere as the origin, the azimuth angle as θ, and the elevation angle as Φ. Define a two-dimensional coordinate system for the lighting map, with the top left corner of the image as the origin, the rightward direction as the positive direction of the horizontal coordinate axis, and the downward direction as the positive direction of the vertical coordinate axis. The specific method of mapping the sampling points on the spherical coordinate system to the target points in the ambient lighting map is as follows: using equidistant cylindrical projection to unfold the spherical information into a rectangular two-dimensional image, and mapping it from the spherical coordinate system to the planar coordinate system of the ambient lighting map.

4. The method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering according to claim 3, characterized in that, Step S3 is specifically implemented as follows: a set of two-dimensional images of the target semi-transparent object are collected and then the camera view parameters, including the position and orientation of the camera, are obtained through Colmap software.

5. The method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering according to claim 4, characterized in that, The specific implementation process of step S4 is as follows: S41. Using the camera pose information and the two-dimensional image of the target object, ray tracing and Sphere-Tracing algorithms are used. The camera position is set as the origin of the ray. For each pixel in the two-dimensional image, a sampling ray is emitted in the direction that the camera points to the pixel. According to the refraction probability characteristics, the transmitted ray and the emitted ray are obtained. By performing this operation on all pixels in the two-dimensional image of the target object one by one, the sampling ray set is finally constructed. S42. Using the symbolic step algorithm, obtain the incident point on the i-th sampling ray in step S41. , exit point ; And generate sampling points according to the following strategy: Let and The magnitude of the vector is ,by The interval is Using the midpoint as the reference point, sample N points before and after it. N points were sampled in the same manner both before and after; S43. Construct a TO-NeuS network. The spatial locations of the sampling points generated in S42 are integrated into a multilayer perceptron via IPE location encoding. The location encoding is then fed into the intermediate layer of the multilayer perceptron via residual connections. The viewpoint direction is encoded using a spherical harmonic function. The information is input to the last fully connected layer, and the surface density weights of the sampling points are obtained in the penultimate and last layers of the perceptron, respectively. and SDF value ; Then The inputs are fed into two color networks identical to the NeuS model to obtain the object surface feature parameters. and internal characteristic parameters ; S44, will , Two outputs are obtained by inputting them into the coding layer respectively. and ,Will As input to the first layer of a multilayer perceptron. As the input to the intermediate layer of this multilayer perceptron, the output of the multilayer perceptron is obtained by convolutional pooling to obtain the refraction probability feature. ; S45, Obtain the surface density weights of the sampling points. Surface characteristic parameters Internal characteristic parameters and refraction probability characteristics An implicit expression of the geometric information of the target object.

6. The method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering according to claim 5, characterized in that, The specific implementation process of step S5 is as follows: S51, The light field model is based on the ambient illumination map and surface density weights. Surface characteristic parameters Internal characteristic parameters and refraction probability characteristics The reflected color component and the transmitted color component are calculated. S52. Sum the reflected and transmitted color components corresponding to all sampling points on a certain sampling ray according to weights to obtain the final viewpoint color, that is, the color of a certain pixel in the reconstructed image. Obtain the colors of all pixels in the reconstructed image to obtain the reconstructed image.

7. The method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering according to claim 6, characterized in that, The specific process for calculating the reflected and transmitted color components is as follows: Reflection color component: Utilizing refraction probability characteristics using ray tracing algorithms. After sampling the ambient lighting map in S2, a BRDF model based on microplanes is introduced to describe and express the reflection phenomenon of the object surface; Transmitted Color Components: A density scattering model for reconstructing translucent objects is introduced, employing a scattering factor to describe the effects of the object's own color and ambient color after passing through the translucent object, and calculating the transmitted color components. This density scattering model uses a Laplacian distribution and introduces a scattering factor to simulate the scattering and transmission effects of light within the translucent object. A trainable scattering factor is used to calculate the surface density weights of sampling points within the object when rendering the transmitted color components. Combined with surface feature parameters and internal characteristic parameters Weighted by surface density The transmitted color components are obtained by summing them.

Images