High-quality semitransparent object reconstruction method based on implicit micro rendering

Abstract

The invention discloses a high-quality semitransparent object reconstruction method based on implicit micro-rendering. The method comprises the following steps: firstly, respectively shooting a plurality of two-dimensional images around a homogeneous smooth metal sphere and a target reconstruction semitransparent object placed in an environment; secondly, generating an environment illumination map by using the two-dimensional images of the metal sphere, and extracting camera pose information corresponding to each two-dimensional image of the target reconstruction semitransparent object; secondly, constructing a translucent object reconstruction network TO-NeuS, taking two-dimensional image data shot around the target object as a label, sampling according to the camera pose information, and obtaining the implicit expression of the geometric information of the target object; and finally, constructing a light field model, obtaining a reflection color component and a transmission color component of a semitransparent object image pixel in combination with the implicit expression and the environment illumination map, and calculating a pixel color of a reconstructed image. According to the method, the dependence limitation of semitransparent object reconstruction on a specific environment is made up, and the reconstruction effect is remarkably enhanced while the calculated amount is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical fields of industrial modeling, artificial intelligence and computer vision, and in particular relates to a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering. Background Art

[0002] Multi-view Figure 3 Three-dimensional reconstruction is an important research direction in computer graphics and computer vision. With the pursuit of realism of three-dimensional models of objects in the real world by researchers and the public, and the application of three-dimensional models in various industries represented by digital twins, autonomous driving, large-scale 3A games, etc., the research on three-dimensional reconstruction of real objects is becoming increasingly active. Although many solutions have been proposed for the three-dimensional reconstruction of opaque objects with diffuse reflection characteristics, such as multi-eye stereo vision, photometric stereo, neural reconstruction based on volume rendering, etc., there are fewer technical solutions for shape reconstruction of translucent objects. And due to the complex optical properties inside translucent objects, the reconstruction scheme for opaque objects cannot be directly applied to translucent objects. At the same time, there are many translucent objects in the world (such as glass, plastic, crystal and diamond), so the rapid reconstruction of translucent objects has important applications in augmented reality, digital twins, etc. Therefore, studying how to obtain high-quality transparent object models has important and practical research significance.

[0003] The reconstruction of translucent materials based on differentiable rendering can be generally divided into two categories: invasive methods and non-invasive methods. Among them, the invasive method involves physical intervention on the surface during the reconstruction process, such as contact, spraying, immersion, etc. For example, the object is immersed in a liquid with a known refractive index to collect the original image. This type of method has many limitations, such as the inability to perform large-scale reconstruction, or it may cause irreversible damage to the reconstructed objects such as important cultural relics. Non-invasive methods require image acquisition in a specific environment or background, such as a dark room or a room without additional light sources. This requires some complex acquisition devices for the reconstruction of translucent material objects and is easily subject to space constraints. Summary of the invention

[0004] In order to restore the optical properties of the inner part of the translucent material as much as possible and reduce the complexity of the equipment and steps during data collection, the present invention provides a high-quality translucent object reconstruction method based on implicit differentiable rendering, which can accurately reconstruct the shape of the object at an unknown viewing angle by using limited viewing angle information. The method includes the following steps:

[0005] S1, take multiple two-dimensional images around a homogeneous smooth metal sphere placed in the environment and a target reconstructed translucent object.

[0006] S2. Generate an ambient lighting map using the two-dimensional image data of the metal sphere in S1.

[0007] S3, extracting the camera pose information corresponding to each two-dimensional image of the target reconstructed semi-transparent object in S1.

[0008] S4. Using the NeuS neural network as the basic model, a semi-transparent object reconstruction network TO-NeuS is constructed. The two-dimensional image data taken around the target object is used as labels. The spatial environment is sampled according to the camera pose information. The implicit expression of the geometric information of the target object is obtained through TO-NeuS: the surface density weight, surface feature parameters and internal feature parameters of the spatial sampling points.

[0009] S5. Construct a light field model, combine the implicit expression and volume rendering method of the TO-NeuS network output in S4, obtain the reflection color component and transmission color component of the image pixel of the semi-transparent object, and calculate the final reconstructed image pixel color to obtain the reconstructed image.

[0010] Furthermore, in step S1, a homogeneous and smooth metal sphere is placed in the center of the environment, and the surface of the sphere is required to completely reflect the surrounding environment. A set of two-dimensional 2D images are collected by shooting around the sphere, and these images capture the same scene from different angles and positions, so that the photographs can completely cover all positions of the sphere, and the position and size of the sphere in all photos are as consistent as possible. After replacing the metal sphere with the translucent target object to be reconstructed, a set of 2D images are collected again in the above manner.

[0011] Furthermore, in step S2, a spherical coordinate system and an ambient lighting map plane coordinate system are established, and the sampling points on the spherical coordinate system are mapped to the target points in the ambient lighting map in units of pixels to obtain a complete ambient lighting map. Define a spherical coordinate system with the center of the sphere as the origin, an azimuth angle of θ, and an elevation angle of Φ. Define lighting Figure 2 The spherical coordinate system is a 2D coordinate system with the upper left corner of the image as the origin, the right direction is the positive direction of the horizontal coordinate axis, and the downward direction is the positive direction of the vertical coordinate axis. In order to map the data in the spherical coordinate system to the ambient lighting map, the present invention uses equidistant cylindrical projection to expand the spherical information into a rectangular two-dimensional image, and maps the spherical coordinate system to the plane coordinate system of the ambient lighting map.

[0012] Furthermore, in step S3, a set of 2D images collected from the target semi-transparent object is used to obtain camera viewing angle parameters, including the position and direction of the camera, through Colmap software. These parameters are used to determine the sampling light emitted from the camera position through each pixel on the image into the 3D scene.

[0013] Furthermore, in step S4, a TO-NeuS network based on the NeuS network is constructed by adding a material network for learning translucent effects and a refraction probability network to the NeuS network for calculating refraction probability features. After calculating the loss function, the optimizer is used to iteratively adjust the network parameters, and finally the network has the ability to implicitly express the geometry and appearance of the object, specifically:

[0014] S41. Using the camera pose information and the two-dimensional image of the target object, using ray tracing and Sphere-Tracing algorithms, the camera position is set as the light origin, and for each pixel in the two-dimensional image, a sampling ray is emitted in the direction of the pixel pointed by the camera. According to the refraction probability characteristics, the transmitted light and the outgoing light are further obtained. This operation is performed one by one through all pixels in the two-dimensional image of the target object, and finally a complete set of sampling rays is constructed. Because total reflection and multiple refraction rays have little effect on the rendering effect, in order to reduce the complexity of the overall network, only the case of secondary refraction is considered when generating the sampling ray set.

[0015] S42, using the sign stepping algorithm, obtain the incident point P on the i-th sampling light in step S41 i1 , the emission point P i2 . And generate sampling points according to the following strategy: Let P i1 and P i2 The modulus of the vector is I, The interval is P i1 is the midpoint, N points are sampled before and after, P i2 N points are sampled in the same way before and after.

[0016] S43, build the TO-NeuS network, integrate the spatial position of the sampling points generated in S42 through the IPE position encoding, input into a multi-layer perceptron, and input the position encoding into the middle layer of the multi-layer perceptron through the residual connection, and input the viewing direction (θ, φ) information encoded by the spherical harmonic function into the last fully connected layer, and obtain the surface density weight w of the sampling point in the penultimate layer and the last layer of the perceptron respectively. surf and SDF value f G (x); then f G (x) are input into two color networks that are the same as the NeuS model to obtain the surface feature parameters C of the object. surf and internal characteristic parameters C in .

[0017] S44, P i1 , P i2 Input the encoding layer separately to get two outputs P i11 and P i22 , P i11As input to the first layer of a multilayer perceptron, P i22 As the input of the middle layer of the multilayer perceptron, the output of the multilayer perceptron is convolutionally pooled to obtain the refraction probability feature η.

[0018] S45. Obtain the sampling point surface density weight w surf , surface characteristic parameters C surf , internal characteristic parameters C in The implicit expression of the geometric information of the target object composed of the refraction probability feature η.

[0019] Furthermore, the content of step S5 will specifically include the design of the light field model structure that follows the physical properties of the translucent material. Since the color of the translucent material is formed by the superposition of multiple physical effects, it is necessary to establish a parameterized light field model to model the light path of the light in the translucent object, so as to calculate the viewpoint color of a certain spatial point in a specified incident direction, specifically:

[0020] S51, light field model according to the ambient lighting map, surface density weight w surf , surface characteristic parameters C surf , internal characteristic parameters C in The reflected color component and the transmitted color component are calculated by the refraction probability characteristic η.

[0021] The reflected color component is used to fit the diffuse reflection and specular reflection phenomena produced by light on the surface of the object. After sampling the refraction probability feature η in the ambient light map in S2 using the ray tracing algorithm, the microplane-based BRDF model is introduced to describe and express the reflection phenomenon on the surface of the object.

[0022] The transmission color component is used to describe the refraction and scattering effects of ambient light in translucent objects. This method introduces a scattering factor and proposes a method for reconstructing translucent objects, which is used to describe the effects of the object's own color and the ambient color after passing through the translucent object, and calculate the transmission color component. The density scattering model adopts Laplace distribution and introduces a scattering factor to simulate the scattering and transmission effects of the light path inside the translucent object. The surface density weight w of the sampling point inside the object when rendering the transmission color component is calculated through the trainable scattering factor. surf , combined with the surface characteristic parameter C surf and internal characteristic parameters C in According to the surface density weight w surf The sum is the transmitted color component.

[0023] S52, summing the reflected color components and the transmitted color components corresponding to all sampling points on a sampling light ray according to the weights to obtain the final viewpoint color, that is, the color of a pixel in the reconstructed image, and obtaining the colors of all pixels in the reconstructed image to obtain the reconstructed image.

[0024] The specific beneficial effects of the present invention are as follows:

[0025] 1. Breaking through environmental limitations, enhancing reconstruction effects and reducing computational complexity: This invention makes up for the dependence of semi-transparent object reconstruction on a specific environment. Deep neural networks are used to implicitly express the geometric shape of semi-transparent objects, while ray tracing and global illumination techniques are introduced in the reconstruction process to simulate the light path in nature. This significantly enhances the reconstruction effect while reducing the computational complexity.

[0026] 2. Parameterize physical effects and effectively simulate the characteristics of translucent materials: The present invention models the refractive index through a deep neural network and introduces a scattering factor to construct a light field model that can simulate the refraction and scattering of translucent materials. The model can accurately and efficiently handle the dynamic changes in density weight caused by optical effects when light penetrates different sampling points. The complex reflection and refraction effects are parameterized so that the multi-layer perceptron can learn the refraction effect of translucent materials, thereby effectively simulating the physical effects.

[0027] 3. Non-invasive acquisition, reducing costs and improving reconstruction authenticity: The acquisition of raw data in the present invention is not restricted by the environment and adopts a non-invasive acquisition method. This not only reduces the acquisition cost and steps while protecting the target reconstructed object, but also makes the final reconstruction effect closer to the real scene expression because the acquired data contains environmental information. Even in the face of complex translucent objects, detailed reconstruction can be completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of a method for reconstructing high-quality semi-transparent objects based on implicit differentiable rendering of the present invention;

[0029] Figure 2 A schematic diagram of the data collection environment and method of the present invention;

[0030] Figure 3 Schematic diagram of the TO-NeuS neural network in the present invention;

[0031] Figure 4 is a schematic diagram of a refraction probability feature neural network in the present invention;

[0032] Figure 5 It is a schematic diagram of the refraction of sampling light by a semi-transparent object in the present invention. DETAILED DESCRIPTION

[0033] The following is further explained in detail through specific implementation methods.

[0034] In order to get rid of the limitations of the complex optical properties inside the translucent object as much as possible and reduce the dependence of the reconstruction process on the environment, the present invention provides a high-quality translucent object reconstruction method based on implicit differentiable rendering.

[0035] The technical solution adopted by the present invention to solve the technical problem is: a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering, the process is as follows Figure 1 As shown, the method comprises the following steps in order:

[0036] (1) Raw data acquisition: RGB images are taken around a homogeneous and smooth metal sphere placed in the environment; RGB images are taken around the translucent target object to be reconstructed; the angle of the surrounding shooting is required to cover the entire azimuth of the object as much as possible, such as Figure 2 shown.

[0037] (2) Image preprocessing: Using the image information collected in the previous step, calculate the metal sphere photo data to generate an ambient lighting map, and generate the camera pose information and llff format information corresponding to the target object image.

[0038] (3) Data training: Using the NeuS neural network as the basic model, a NeuS-based semi-transparent material target reconstruction model TO-NeuS is constructed. The camera pose information and ray tracing algorithm are used to obtain sampling rays and generate spatial sampling points. The spatial position and viewing direction of the sampling points are encoded and put into the TO-NeuS network for training to predict and calculate the characteristic parameters of the reconstructed objects: surface density weight w surf 、SDF value f G (x), surface characteristic parameter C surf and internal characteristic parameters C in .

[0039] (4) Establish a trainable light field model to reconstruct the image of an object at an unknown viewing angle: Calculate the various parameters obtained in step (3) to obtain the reflection color component and transmission color component of the sampling point, and calculate the viewpoint color according to the weight factor in step (3). The two color components are calculated using their respective sampling models. Using the characteristic parameters in step (3), a physically based microsurface model is used to calculate the reflection color component. The object's own color is obtained through network prediction in step (3), and the background environment color is obtained by sampling the sampling light on the environment illumination map. The transmission color component is obtained by weighting the transmission color of the light inside the object and the background environment color. In view of the complex refraction and scattering physical phenomena of translucent objects during the propagation of light, a scattering factor and a refraction probability feature are introduced to construct a density scattering model specifically used for the reconstruction of translucent objects. The model can accurately and efficiently handle the dynamic changes in density weights caused by refraction and scattering when light penetrates different sampling points. The reflection color component and the transmission color component corresponding to all sampling points on a sampling light are summed according to the density weight to obtain the final viewpoint color, that is, the color of a pixel in the reconstructed image.

[0040] The step (1) specifically comprises the following steps:

[0041] Place a homogeneous and smooth metal sphere in the center of the environment, and the sphere is required to completely reflect the surrounding environment. Collect a set of 2D images by shooting around the sphere. These images capture the same scene from different angles and positions, so that the photos taken can completely cover all positions of the sphere, and the position and size of the sphere in all photos are as consistent as possible. After replacing the metal sphere with the semi-transparent target object that needs to be reconstructed, collect another set of 2D images in the above way.

[0042] Get the ambient light map. Establish a spherical coordinate system and an ambient light map plane coordinate system, and map the sampling points of the two. Define a spherical coordinate system with the center of the sphere as the origin, an azimuth angle of θ (θ∈[0,2π]), and an elevation angle of φ (φ∈[0,π]). Define the light Figure 2 dimensional coordinate system, with the upper left corner of the image as the origin, the right as the positive direction of the horizontal coordinate axis, and the downward as the positive direction of the vertical coordinate axis. The horizontal coordinate u∈[0, W], the vertical coordinate v∈[0, H] (W is the image width, H is the image height). In this method, the ambient light map uses equidistant cylindrical projection to expand the spherical information into a rectangular two-dimensional image. The following is the mapping formula for converting spherical coordinates to ambient light map coordinates:

[0043]

[0044] Collect a set of 2D images that capture the same scene from different angles and positions. That is, these images are all shots of the same 3D scene, so each object in this 3D scene will appear in multiple images, just from different perspectives and positions.

[0045] Get the parameters of the camera corresponding to each 2D image, including the camera's position and direction; get the camera's viewing angle parameters through colmap. colmap is a solution method dedicated to obtaining the camera's position information. Use colmap to obtain detailed information about each camera's photo, including (x, y, z, θ, φ), where x, y, z are the three-dimensional coordinates of the image, and θ and φ are the viewing directions. These parameters are used to determine the direction of the ray that starts from the camera's position, passes through each pixel on the image, and shoots into the 3D scene;

[0046] The step (3) specifically comprises the following steps:

[0047] like Figure 5 As shown in the figure, the sampling rays are calculated by ray tracing. Before training, the SDF values ​​of the sampling points are randomly assigned, and then the neural network is iteratively trained to make the SDF values ​​close to the geometric shape of the object. In this method, the geometric information of the object will be implicitly expressed using the neural network in the form of SDF, and the concept of volume rendering will be used to discretize the target object into sampling points in space, and the geometric information and material information of the object will be expressed through various parameters of the sampling points.

[0048] The i-th sampling ray L can be obtained from the SDF data by using the sign-stepping algorithm. i The incident point P i1 , the emission point P i2 The sampling light is divided into three parts by the incident point and the exit point: the incident light L in , transmit light L trans and the outgoing light L out . Generate sampling points on the sampling ray according to a fixed strategy. For each sampling ray L i , let I=|P i1 P i2 |, with The interval is P i1 is the midpoint, N points are sampled before and after, P i2 In the same way, N points are sampled. Because for semi-transparent objects, when f G When (x)>0, it has no effect on light attenuation. Therefore, sampling is only performed on the front and back of the object surface, which improves sampling accuracy and reduces the amount of calculation. For each pixel on the two-dimensional image in step (2), a ray is first determined, and then a series of spatial points are sampled on the ray in the above manner.

[0049] Furthermore, this method constructs a TO-NeuS network to implicitly express the geometry and appearance of objects. Figure 3 As shown in the figure, following the NeuS network structure, the three-dimensional coordinates (x, y, z) of the sampling point constructed above are input. After IPE integrated position encoding, a 9-layer 256-dimensional MLP network is used to obtain the surface density weights w of the sampling points at different stages. surf 、SDF value f G (t i ). After the reflection density weight is outputted in the ninth layer of the network, the viewing direction (θ, φ) information encoded by the spherical harmonic function is added to enhance the model's learning and reconstruction of illumination and reflection information at different viewing angles. Here, the third-order (quadratic) spherical harmonic function is used to encode the viewing direction (θ, φ), that is, the 9-dimensional spherical harmonic coefficients are used to replace the encoding of the sine and cosine functions in NeRF and input into the network, so that the model increases the viewing angle dependence and realizes the "Fresnel effect" in real scenes. Then construct two 4-layer 256-dimensional material networks and convert f G (t i ) are put into the material network for training to obtain the surface feature parameters C surf and internal characteristic parameters C in Each time the training is iterated, a sampling operation is performed based on the latest trained network information.

[0050] Furthermore, this method proposes a refraction probability feature to replace the constant refractive index of the object, and uses learnable parameters to fit the refraction phenomenon of light inside the object. The specific network structure is as follows Figure 4 As shown, the spatial position coordinates P of the sampling point i1 After the IPE integrated position encoding and expansion operation, the (x, y, z) three-dimensional vector is input into the MLP composed of seven layers of 256-dimensional fully connected layers FC, so as to obtain the refraction degree of the sampling point after passing through the object. In order to strengthen the connection between the incident point and the exit point, the network adds a residual structure in the fourth layer and inputs the exit point P i2 The three-dimensional coordinates of (x, y, z) are finally extracted through a convolutional pooling operation to reduce the range of the overall network output data.

[0051] The step (4) specifically comprises the following steps:

[0052] A light field model that follows the physical properties of translucent materials is established. The light field model can model the light path of light in a translucent object, thereby calculating the viewpoint color, that is, the color of a pixel in the reconstructed image. The viewpoint color value contains two color components, the reflection color and the transmission color. After modeling the light path, the light can be calculated using the parameters of the sampling points obtained in step (3). The light in the modeled light path is called the sampling light. Since total reflection and multiple refraction light have little effect on the rendering effect, in order to reduce the complexity of the model and simplify the calculation, only the case of secondary refraction is considered in this calculation model.

[0053] Furthermore, in ray tracing, according to P i1 Point position, P i2 The point position and refraction probability characteristic η can be used to calculate the analytical expressions of three types of sampling rays: incident light, transmitted light in the object, and outgoing light:

[0054] L in =O+|OP i1 |·t

[0055] L trans =P i1 +|OP i1 |·η·t

[0056] L out =P i2 +|P i1 P i2 |·η * ·t

[0057] Where O is the camera viewpoint, η * It is the outgoing deflection probability characteristic derived by rotating the refraction probability characteristic η in the negative direction according to the rotation matrix.

[0058] Furthermore, the background color is calculated based on the three types of sampled light and the ambient light map. First, the spherical coordinate system is converted to the Cartesian coordinate system. Consider P i2 The unit sphere with centered on is used to calculate the intersection point r between the light and the unit sphere. The unit sphere r is projected onto the ambient light map based on the stereographic projection to obtain the background color C. background . i1 The point and the reflected light are sampled on the ambient light map in the same way as above to obtain C env . The transmission color can then be expressed as:

[0059] C in =μC background +(1-μ)C in

[0060] Where μ is a hyperparameter that controls the contribution of the background color to the transmission color component.

[0061] Reflection color efelct It is calculated based on the direct illumination model composed of all the light rays that are reflected from the object once. The microsurface model is introduced to enhance the physical effect, and the BRDF value of the specular reflection part is calculated. The specific expression is as follows:

[0062]

[0063] In this formula, the half-range vector is defined F is the Fresnel term that describes the reflection effect based on physical laws, which is about the half-distance vector h and the outgoing light ω o The double parameter function is as follows:

[0064] F(c,ω0,ω i )=c+(1-c)(1-(ω o ·ω i )) 5

[0065] In this expression, c represents the speed of light.

[0066] G is a geometric compensation term, which is used to describe the shadow effect caused by the occlusion of the object. The specific expression is as follows:

[0067]

[0068] D is the normal distribution function, which describes the normal distribution in the microplane. Here, GGX distribution is used to describe the roughness of the object surface. The specific expression is as follows:

[0069]

[0070] Among them, α is the roughness parameter, which is input into the network as a hyperparameter in the subsequent training to describe the roughness of the object.

[0071] The diffuse BRDF value is obtained using the Lambertian diffuse reflection model and the reflected environment color in step (3):

[0072] The final reflected color component can be expressed as:

[0073] C reflect =(f d +f s )×C env ×(N·ω o )

[0074] Where N is the normal vector of the surface point of the object, which can be sampled and calculated according to the SDF value. Sampling the SDF value of the adjacent spatial point, the normal vector of the adjacent triangle surface is obtained, and the normal vector of the point is calculated according to the area weight.

[0075] In order to make the network output have the ability to express the scattering and refraction of light of translucent objects, this paper draws on the work of NuralTO and introduces a scattering factor to establish a density scattering model to obtain the internal density weight w in , the specific contents are as follows:

[0076] Since the NeuS network only optimizes the non-zero weights of sampling points near the surface of opaque objects, the weight inside the object is zero. At the same time, the scattering ability of the light inside the translucent object is not uniform. Therefore, in order to solve the limitations in NeuS, this method uses a physical model that follows the principle of radiation light to simulate the density inside the object, thereby fitting the effect of light dissipation in translucent objects in reality. This method borrows the Laplace distribution function to represent the density field inside the translucent material:

[0077]

[0078] where f G (x) is the SDF value of the spatial point x predicted by the neural network, σ t Represents the attenuation coefficient of light energy inside the object. The larger the attenuation coefficient, the faster the light energy dissipates inside the object and the lower the transparency. When β approaches 0, the sampling point density inside the object will approach σ, and the spatial attenuation coefficient outside the object will approach 0, δ t =t i+1 -t i .

[0079] For physical plausibility, in w in In the calculation of , the sampling interval needs to be redefined. The sampling interval is limited to the incident position P i1 (x,y,z) and the exit position P i2 The points between (x, y, z) define the sampling point set R:

[0080] R={t i |f G (t i )·f G (t i+1 )<0)}

[0081] All sampling points in R and their adjacent ascending sampling points are located inside and outside the target reconstructed object surface respectively. The set R can be used to define the set of intersection points Ω:

[0082]

[0083] The parameter sampling point t i Derived from the above set R, f G (t i ) is the spatial point t predicted by the neural network i The SDF value at the location. Through the above range, we can get the sampling point t i The corresponding weight is calculated as follows:

[0084]

[0085] where t i is the sampling point in the above set Ω, σ t is the density of sampling points required for neural network training, δ t is the sampling point t i The Euclidean distance to its adjacent descending sampling points, is the minimum value of the above sampling area.

[0086] Furthermore, the transmitted color component can be calculated by the weighted sum of the sampling points on a certain ray, which includes the refraction and scattering effects of the light inside the object, and is calculated by the following expression:

[0087]

[0088] Among them C trsns is the color transmitted after the sampling light penetrates the object, which is a biparametric function of the sampling point p(t) and the sampling light direction vector d. trsns The whole can be divided into surface color and internal color, both of which are adjusted by a hyperparameter γ. surf represents the density weight of the sampling point predicted by the TO-NeuS network, C surf is the surface characteristic parameter output by the TO-NeuS network; w in The sampling point t calculated by the density scattering model i The corresponding weight, C in is the internal characteristic parameter after the background ambient light dissipates inside the object, is the maximum value of the above sampling area.

[0089] Finally, the pixel color of the reconstructed image of the semi-transparent object at other unknown viewing angles can be calculated by controlling the reflected color component and the transmitted color component:

[0090] C pixel =(1-κ)κ Creflect +κ Ctrans

[0091] Among them, κ is the contribution of controlling the environmental reflection on the surface of the object. When the mirror reflection effect of the object is strong, a higher value should be taken to enhance the influence of the environment on the surface of the object.

[0092] In order to verify the effectiveness of the TO-NeuS network, a comparative experiment was conducted on the TO-NeuS algorithm, the NeuS algorithm and the HF-NeuS on the Syn-Trans dataset. The Syn-Trans dataset includes four scenes: GummyBear, Stanford Dragon, Yuanbao and Ancient Dragon. The chamfer distance was used as the evaluation index to obtain the experimental data, as shown in Table 1:

[0093] Table 1

[0094]

[0095] In summary, the present invention proposes a high-quality semi-transparent object reconstruction method based on implicit differentiable rendering, which can not only collect object surface information in a non-invasive way and break through environmental dependence, but also effectively simulate the surface reflection and internal transmission characteristics of the object after parameterizing the physical effect, so that the reconstruction result is closer to the real phenomenon of the physical world; in addition, the introduction of scattering factors and refraction probability characteristics is also a feature of the present invention that is different from other reconstruction algorithms. It effectively restores the refraction phenomenon of light inside the semi-transparent object to a certain extent. Although there are slight deviations in physical details, the error range is controlled within the gap that is difficult for the human eye to distinguish between it and the real image, so a good reconstruction result is finally obtained.

Claims

1. A high-quality semi-transparent object reconstruction method based on implicit differentiable rendering, characterized in that: The following steps are involved: S1, taking multiple two-dimensional images around a homogeneous smooth metal sphere placed in the environment and a target reconstructed translucent object; S2, generating an ambient lighting map using the two-dimensional image of the metal sphere; S3, extracting the camera pose information corresponding to each two-dimensional image of the target reconstructed translucent object; S4. Construct a semi-transparent object reconstruction network TO-NeuS, use the two-dimensional image data shot around the target object as labels, sample the spatial environment according to the camera pose information, and obtain the implicit expression of the geometric information of the target object through TO-NeuS; S5. Construct a light field model, combine the implicit expression output by the TO-NeuS network and the ambient lighting map, obtain the reflection color component and transmission color component of the image pixels of the semi-transparent object, and calculate the pixel color of the reconstructed image to obtain the reconstructed image.

2. The high-quality semi-transparent object reconstruction method based on implicit differentiable rendering according to claim 1, characterized in that: The specific implementation process of step S1 is as follows: a homogeneous and smooth metal sphere is placed in the center of the environment, and the surface of the sphere completely reflects the surrounding environment; a set of two-dimensional 2D images are collected by shooting around the sphere, and these images capture the same scene from different angles and positions, so that the photographs taken completely cover all positions of the sphere, and the position and size of the sphere appearing in all photos are consistent; after replacing the metal sphere with the translucent target object to be reconstructed, a set of 2D images are collected again in the above manner.

3. The high-quality semi-transparent object reconstruction method based on implicit differentiable rendering according to claim 2, characterized in that: The specific process of step S2 is: establishing a spherical coordinate system and a plane coordinate system of the ambient light map, mapping the sampling points on the spherical coordinate system to the target points in the ambient light map in units of pixels, and obtaining a complete ambient light map; The establishment of the spherical coordinate system and the environment illumination map plane coordinate system is specifically as follows: defining a spherical coordinate system, with the center of the sphere as the origin, the azimuth angle as θ, and the elevation angle as Φ; Define the two-dimensional coordinate system of the illumination map, with the upper left corner of the image as the origin, the right as the positive direction of the horizontal coordinate axis, and the downward as the positive direction of the vertical coordinate axis; The method of mapping the sampling points on the spherical coordinate system to the target points in the ambient illumination map is as follows: the spherical information is expanded into a rectangular two-dimensional image by using equidistant cylindrical projection, and the spherical coordinate system is mapped to the plane coordinate system of the ambient illumination map.

4. The high-quality semi-transparent object reconstruction method based on implicit differentiable rendering according to claim 3 is characterized in that: The step S3 is specifically implemented as follows: a set of two-dimensional images collected from the target translucent object is used to obtain camera viewing angle parameters, including the position and direction of the camera, through Colmap software.

5. The high-quality semi-transparent object reconstruction method based on implicit differentiable rendering according to claim 4, characterized in that: The specific implementation process of step S4 is as follows: S41, using the camera posture information and the two-dimensional image of the target object, using ray tracing and Sphere-Tracing algorithms, setting the position of the camera as the origin of the light, for each pixel in the two-dimensional image, emitting a sampling light in the direction of the pixel pointed by the camera; according to the refraction probability characteristics, obtaining the transmitted light and the emitted light; by performing this operation one by one for all pixels in the two-dimensional image of the target object, finally constructing a sampling light set; S42, using the sign stepping algorithm, obtain the incident point P on the i-th sampling light in step S41 i1 , the emission point P i2 ; and generate sampling points according to the following strategy: Let P i1 and P i2 The modulus of the vector is I, The interval is P i1 is the midpoint, N points are sampled before and after, P i2 N points are sampled in the same way before and after; S43, build the TO-NeuS network, integrate the spatial position of the sampling points generated in S42 through the IPE position encoding, input into a multi-layer perceptron, and input the position encoding into the middle layer of the multi-layer perceptron through the residual connection, and input the viewing direction (θ, φ) information encoded by the spherical harmonic function into the last fully connected layer, and obtain the surface density weight w of the sampling point in the penultimate layer and the last layer of the perceptron respectively. surf and SDF value f G (x); then f G (x) are input into two color networks that are the same as the NeuS model to obtain the surface feature parameters C of the object. surf and internal characteristic parameters C in ; S44, P i1 , P i2 Input the encoding layer separately to get two outputs P i11 and P i22 , P i11 As input to the first layer of a multi-layer perceptron, P i22 As the input of the middle layer of the multilayer perceptron, the output of the multilayer perceptron is convolutionally pooled to obtain the refraction probability feature η; S45. Obtain the sampling point surface density weight w surf , surface characteristic parameters C surf , internal characteristic parameters C in The implicit expression of the geometric information of the target object composed of the refraction probability feature η.

6. The high-quality semi-transparent object reconstruction method based on implicit differentiable rendering according to claim 5, characterized in that: The specific implementation process of step S5 is as follows: S51, light field model according to the ambient lighting map, surface density weight w surf , surface characteristic parameters C surf , internal characteristic parameters C in The reflection color component and the transmission color component are calculated by using the refraction probability characteristic η; S52, summing the reflected color components and the transmitted color components corresponding to all sampling points on a sampling light ray according to the weights to obtain the final viewpoint color, that is, the color of a pixel in the reconstructed image, and obtaining the colors of all pixels in the reconstructed image to obtain the reconstructed image.

7. The high-quality semi-transparent object reconstruction method based on implicit differentiable rendering according to claim 6, characterized in that: The specific process of calculating the reflected color component and the transmitted color component is as follows: Reflection color component: After sampling the ambient light map in S2 using the refraction probability feature η using the ray tracing algorithm, a microplane-based BRDF model is introduced to describe and express the reflection phenomenon on the object surface; Transmitted color component: introduces a scattering factor and proposes a density scattering model for semi-transparent object reconstruction, describes the effect of the object's own color and the ambient color after passing through the semi-transparent object, and calculates the transmitted color component; the density scattering model uses Laplace distribution, introduces a scattering factor to simulate the scattering and transmission effects of the light path inside the semi-transparent object, and calculates the surface density weight w of the sampling point inside the object when rendering the transmitted color component through the trainable scattering factor surf , combined with the surface characteristic parameter C surf and internal characteristic parameters C in According to the surface density weight w surf The sum is the transmitted color component.

Images