Scene inverse rendering method and device based on neural implicit representation

Abstract

The application discloses a scene reverse rendering method and device based on neural implicit expression, and the method is based on a signed distance field network, a neural radiation field network and a neural material field network to realize joint implicit expression of geometric information, illumination information and material information of a scene, and a ray tracing rendering algorithm based on Monte Carlo sampling is used to obtain a rendering result, a loss function is constructed according to the rendering result to optimize network parameters, and the signed distance field network, the neural radiation field network and the neural material field network capable of accurately predicting the geometric information, the illumination information and the material information are obtained. Meanwhile, the method also supports re-illumination editing and material editing, and achieves an augmented reality function.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional reconstruction, and particularly relates to a scene reverse rendering method and device based on neural implicit representation. BACKGROUND

[0002] Traditional three-dimensional modeling needs to manually adjust the geometric shape, material parameters and other information of the model, which consumes a large amount of human resources. In addition, the model geometry and appearance of three-dimensional modeling depend on the personal design of the modeler, and it is difficult to perfectly reproduce specific objects and scenes in the real world. Moreover, traditional modeling usually uses a triangular mesh-based model structure, which has the problem of occupying too much space.

[0003] With the development of artificial intelligence and deep learning algorithms, three-dimensional visual reconstruction technology based on neural networks has gradually developed. The idea of three-dimensional reconstruction algorithm based on neural networks is to directly represent the geometry and lighting information of a three-dimensional scene using a neural network. It can directly input multiple pictures of a real-world scene or object, and directly reconstruct the geometry and appearance of the real scene through a deep learning optimization algorithm, such as the face three-dimensional reconstruction method based on a deep neural network disclosed in patent document CN115147543A, and the three-dimensional reconstruction method based on a convolutional neural network disclosed in patent document CN109410321A. Neural three-dimensional reconstruction algorithm is an end-to-end automatic optimization algorithm that does not require much human intervention, which can save the human resources of modelers and automatically reproduce the real reproduction effect that is difficult for humans to manually simulate. In addition, the space occupied by the neural network representing a three-dimensional scene is usually much smaller than the traditional triangular mesh, and has good spatial continuity.

[0004] However, the existing three-dimensional visual reconstruction technology based on neural networks only supports restoring the geometry and lighting features of the scene, and does not support editing and other operations, which has poor flexibility and scalability. SUMMARY

[0005] In view of the above, the purpose of the present application is to provide a scene reverse rendering method and device based on neural implicit representation, which realizes the rendering of geometric information, lighting information and material information.

[0006] To achieve the above-mentioned purpose of the application, the scene reverse rendering method based on neural implicit representation provided by the embodiment comprises the following steps:

[0007] Constructing a symbolic distance field network, a neural material field network and a neural radiation field network based on a neural network;

[0008] inputting the three-dimensional points in the scene into a neural material field network and a symbolic distance field network to perform material prediction and geometry prediction, and performing volume rendering to obtain material information and normal information after the inputting;

[0009] performing importance sampling according to the material information and the normal information, and calculating an outgoing light ray and a probability density function value and a material estimation value corresponding to the outgoing light ray;

[0010] inputting the outgoing light ray into the symbolic distance field network to perform geometry prediction and calculate an intersection point with a geometric surface, inputting the intersection point into a neural radiance field network to perform illumination prediction, and performing volume rendering according to the illumination prediction result to obtain an illumination radiation value corresponding to the outgoing light ray;

[0011] performing Monte Carlo integration according to the probability density function value, the material estimation value, and the illumination radiation value corresponding to the outgoing light ray to obtain a rendering result;

[0012] constructing a first loss function according to the rendering result and a scene picture as a supervision label, and performing parameter optimization on the symbolic distance field network, the neural material field network, and the neural radiance field network by using the first loss function;

[0013] performing geometry prediction, material prediction, and illumination prediction by using the parameter-optimized symbolic distance field network, the neural material field network, and the neural radiance field network, respectively.

[0014] In one embodiment, the method further comprises: constructing a second loss function according to the normal information and a normal map as a supervision label, and performing parameter optimization on the symbolic distance field network, the neural material field network, and the neural radiance field network by using the second loss function and the first loss function.

[0015] In one embodiment, the method further comprises: performing volume rendering to obtain depth information after inputting the three-dimensional points in the scene into the symbolic distance field network to perform geometry prediction; constructing a third loss function according to the depth information and a depth map as a supervision label; and performing parameter optimization on the symbolic distance field network, the neural material field network, and the neural radiance field network by using the third loss function and the first loss function, or the first loss function, the second loss function, and the third loss function.

[0016] In one embodiment, the symbolic distance field network is represented as a symbolic distance function defined as the minimum value of distances of any three-dimensional point in the scene to all surfaces in the scene, and the geometry prediction result is represented as a symbolic distance function value, and the three-dimensional points with the symbolic distance function value of zero are extracted to form a geometric surface.

[0017] In one embodiment, the method further comprises: constructing a density function of a Laplace surface distribution according to the fact that a peak point of the Laplace surface distribution is located at a zero point of the symbolic distance function;

[0018] The body rendering obtains the material information, including: weighting and accumulating the material prediction result and the density function of the Laplace surface distribution to obtain the material information.

[0019] The body rendering obtains the light radiation value corresponding to the exit light, including: weighting and accumulating the light prediction result and the density function of the Laplace surface distribution to obtain the light radiation value.

[0020] The geometry prediction and the body rendering obtain the normal information, including: weighting and accumulating the gradient of the signed distance function and the density function of the Laplace surface distribution to obtain the normal information.

[0021] The geometry prediction and the body rendering obtain the depth information, including: weighting and accumulating the distance from the sampling point to the origin and the density function of the Laplace surface distribution to obtain the depth information.

[0022] In an embodiment, the material is represented as three parameters of the GGX micro-surface material model, including the diffuse albedo, the specular albedo and the roughness.

[0023] The neural material field network performs material prediction according to the input three-dimensional point to output the first RGB value as the diffuse albedo, the second RGB value as the specular albedo, and the floating point number as the roughness.

[0024] In an embodiment, the importance sampling and the calculation of the exit light, the probability density function value corresponding to the exit light and the material estimation value according to the material information and the normal information include:

[0025] On the geometric surface, the importance sampling is performed according to the roughness contained in the material information to obtain the exit light, the probability distribution of the importance sampling follows the GGX micro-surface material model and is determined by the roughness, the incident direction of the sampling point and the surface normal information are input into the GGX micro-surface material model to calculate the direction of the exit light and the probability density function value, the diffuse albedo, the specular albedo and the roughness contained in the material prediction result are input into the GGX micro-surface material model to calculate the material estimation value.

[0026] In an embodiment, the method further includes: providing a material editing function, using the material editing function to perform material editing on the material prediction result output by the parameter-optimized neural material field network, and performing rendering on the material editing result.

[0027] In an embodiment, the method further includes: providing a light editing function, using the light editing function to perform light irradiance editing on the light prediction result output by the parameter-optimized neural radiance field network, and performing rendering on the light irradiance editing result.

[0028] To achieve the above-mentioned object of the application, the embodiment further provides a scene reverse rendering device based on neural implicit representation, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned scene reverse rendering method based on neural implicit representation when executing the computer program.

[0029] Compared with the prior art, the present application has at least the following beneficial effects:

[0030] Based on the symbol distance field network, the neural radiation field network and the neural material field network, the geometric information, the lighting information and the material information of the scene are jointly and implicitly expressed, and a rendering result is obtained by using a ray tracing rendering algorithm based on Monte Carlo sampling. A loss function is constructed according to the rendering result to optimize the network parameters, so as to obtain the symbol distance field network, the neural radiation field network and the neural material field network which can accurately predict the geometric information, the lighting information and the material information. Meanwhile, the relighting editing and the material editing are supported, so as to achieve the augmented reality function. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0032] Figure 1 is a flowchart of the scene reverse rendering method based on neural implicit representation provided by the embodiment;

[0033] Figure 2 is a network training flowchart provided by the embodiment;

[0034] Figure 3 is a flowchart of the reconstruction editing provided by the embodiment. DETAILED DESCRIPTION

[0035] In order to make the object, technical scheme and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0036] Figure 1 is a flowchart of the scene reverse rendering method based on neural implicit representation provided by the embodiment, Figure 2 is a network training flowchart provided by the embodiment. As shown in Figure 1 and Figure 2 The scene reverse rendering method provided by the embodiment comprises the following steps:

[0037] Step 1, based on neural network, symbol distance field network, neural material field network, neural radiance field network is constructed.

[0038] In the embodiment, a neural network such as a multi-layer perceptron is used as the network to construct the symbol distance field network, the neural material field network, and the neural radiance field network. The symbol distance field network is used to express the geometry of a three-dimensional scene, and specifically, a geometric prediction is performed according to the three-dimensional point coordinates to obtain a geometric prediction result. The fitting target of the three-dimensional scene is the symbol distance field, which is defined based on a surface-defined symbol distance function, i.e., the symbol distance field network is represented as a symbol distance function. The symbol distance function defines the minimum distance from any three-dimensional point in the scene to all surfaces of the scene. The geometric prediction result is represented as the value of the symbol distance function. When the value of the symbol distance function is zero, it indicates the zero point of the symbol distance field. When the value of the symbol distance function is greater than zero, it indicates that the three-dimensional point is inside the scene. When the value of the symbol distance function is less than zero, it indicates that the three-dimensional point is outside the scene. The three-dimensional points with a symbol distance function value of zero are extracted to form a geometric surface.

[0039] The neural material field network is used to express the material of the scene material. The fitting target of the scene material is the GGX micro-surface material model, i.e., the material is represented as three parameters of the GGX micro-surface material model, including diffuse albedo, specular albedo, and roughness. Diffuse albedo and specular albedo control the color of the surface when diffuse reflection and specular reflection occur, respectively, and the brightness values of the two determine the proportion of diffuse reflection and specular reflection. Roughness controls the sharpness of specular reflection. The higher the roughness, the rougher the surface, and the more blurred the specular reflection. The GGX micro-surface material model is a physically-based model that can realistically fit the reflection light effect in the real world.

[0040] The neural material field network performs material prediction according to the three-dimensional point coordinates to obtain a material prediction result. Specifically, the input three-dimensional point is used for material prediction to output a first RGB value as the diffuse albedo, a second RGB value as the specular albedo, and a floating-point number between 0 and 1 as the roughness.

[0041] The neural radiance field network is used to express the scene lighting, and specifically, lighting prediction is performed according to the three-dimensional point coordinates to output a third RGB value as the lighting prediction result.

[0042] Step 2, after inputting the three-dimensional points in the scene to the neural material field network and the symbol distance field network for material prediction and geometric prediction, body rendering is performed to obtain material information and normal information.

[0043] In the embodiment, the three-dimensional points in the scene are input into the neural material field network and the symbolic distance field network for material prediction and geometry prediction to obtain material prediction results and geometry prediction results. When performing volume rendering, first, the symbolic distance function and the density function are corresponded using the Laplace surface distribution function, that is, the density function of the Laplace surface distribution is constructed according to the peak point of the Laplace surface distribution located at the zero point of the symbolic distance function. Then, the material prediction results are weighted and accumulated with the density function of the Laplace surface distribution to obtain material information, and the gradient of the symbolic distance function is weighted and accumulated with the density function of the Laplace surface distribution to obtain normal information.

[0044] In the embodiment, the depth information is also obtained by volume rendering, including: weighting and accumulating the distance from the sampling point to the camera position with the density function of the Laplace surface distribution to obtain the depth information.

[0045] Step 3, importance sampling is performed according to the material information and the normal information, and an outgoing light ray and a probability density function value corresponding to the outgoing light ray and a material estimation value are calculated.

[0046] In the embodiment, on the geometric surface, the outgoing light ray is obtained by importance sampling according to the roughness contained in the material information, the probability distribution of the importance sampling follows the GGX micro-surface material model and is determined by the roughness, the incident direction of the sampling point and the surface normal information are input into the GGX micro-surface material model to calculate the direction of the outgoing light ray and the probability density function value, and the diffuse reflectance, the specular reflectance and the roughness contained in the material prediction results are input into the GGX micro-surface material model to calculate the material estimation value.

[0047] Step 4, the outgoing light ray is input into the symbolic distance field network for geometry prediction and calculation of the intersection point with the geometric surface, the intersection point is input into the neural radiation field network for illumination prediction, and volume rendering is performed according to the illumination prediction results to obtain the illumination radiation value corresponding to the outgoing light ray.

[0048] In the embodiment, the corresponding illumination information of the outgoing light ray is calculated by a ray tracing algorithm, specifically, the intersection of the outgoing light ray and the symbolic distance field is calculated in the symbolic distance field, that is, the outgoing light ray is input into the symbolic distance field network for geometry prediction and calculation of the intersection point with the geometric surface, the intersection point is input into the neural radiation field network for illumination prediction, and volume rendering is performed according to the illumination prediction results to obtain the illumination radiation value corresponding to the outgoing light ray.

[0049] Specifically, the illumination radiation value is obtained by volume rendering, including: weighting and accumulating the illumination prediction results with the density function of the Laplace surface distribution to obtain the illumination radiation value.

[0050] Step 5, Monte Carlo integration is performed according to the probability density function value corresponding to the outgoing light ray, the material estimation value and the illumination radiation value to obtain the rendering result.

[0051] In this embodiment, the rendering result, i.e. the color value of the corresponding pixel, is calculated from the probability density function value, material estimate value, and illumination radiation value corresponding to multiple outgoing rays, based on the Monte Carlo integral formula.

[0052] Step 6: Construct a loss function and optimize network parameters based on the rendering results, normal information, and depth information.

[0053] When optimizing network parameters, the training objective is to minimize the difference between the rendered image obtained by the ray tracing rendering algorithm and the real scene image. That is, a first loss function is constructed based on the rendering result and the scene image used as the supervision label. The first loss function can be used to optimize the parameters of the symbolic distance field network, the neural material field network, and the neural radiation field network using the gradient descent algorithm.

[0054] In one possible implementation, another training objective can be added, which is to minimize the difference between the normal information obtained from volume rendering and the provided normal map. That is, a second loss function is constructed based on the normal information and the normal map used as a supervision label. The second loss function and the first loss function are used to optimize the parameters of the symbolic distance field network, the neural material field network, and the neural radiation field network using the gradient descent algorithm.

[0055] In another possible implementation, another training objective can be added, which is to minimize the difference between the depth information obtained from volume rendering and the provided depth map. A third loss function is constructed based on the depth information and the depth map used as a supervision label. The symbolic distance field network, neural material field network, and neural radiation field network are optimized using the third loss function and the first loss function and the gradient descent algorithm. Alternatively, the symbolic distance field network, neural material field network, and neural radiation field network are optimized using the first loss function, the second loss function, and the third loss function and the gradient descent algorithm.

[0056] Step 7: Use the parameter-optimized neural material field network and neural radiation field network to perform material prediction and illumination prediction, respectively.

[0057] In the embodiments, such as Figure 3 As shown, after training the symbolic distance field network, neural material field network, and neural radiation field network, the parameter-optimized symbolic distance field network, neural material field network, and neural radiation field network can be used to perform geometry prediction, material prediction, and lighting prediction, respectively. Based on the prediction results, a ray tracing rendering algorithm is used to render the scene image from any viewpoint.

[0058] The scene reverse rendering method provided by the embodiment also provides a material editing function. After the material prediction result output by the neural material field network with optimized parameters is edited by using the material editing function, the material editing result can be rendered by using a ray tracing rendering algorithm to render a realistic image.

[0059] The scene reverse rendering method provided by the embodiment also provides a light editing function. After the light prediction result output by the neural radiance field network with optimized parameters is edited by using the light editing function, the edited result of the re-lighting is realized. The light editing result is rendered to render a realistic image.

[0060] The scene reverse rendering method based on neural implicit representation provided by the embodiment can reconstruct a three-dimensional model of an indoor scene with high quality and without loss, the three-dimensional model including geometric information, lighting information and material information, and render a realistic scene image, while realizing a realistic editing function.

[0061] Based on the same inventive concept, the embodiment also provides a scene reverse rendering device based on neural implicit representation, including a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned scene reverse rendering method based on neural implicit representation is realized, including the following steps:

[0062] Step 1, constructing a signed distance field network, a neural material field network and a neural radiance field network based on a neural network;

[0063] Step 2, inputting three-dimensional points in the scene to the neural material field network and the signed distance field network for material prediction and geometric prediction, and then performing volume rendering to obtain material information and normal information;

[0064] Step 3, performing importance sampling according to the material information and the normal information, and calculating outgoing light rays and probability density function values and material estimation values corresponding to the outgoing light rays;

[0065] Step 4, inputting the outgoing light rays to the signed distance field network for geometric prediction and calculating the intersection points with the geometric surface, inputting the intersection points to the neural radiance field network for light prediction, and performing volume rendering according to the light prediction result to obtain light radiation values corresponding to the outgoing light rays;

[0066] Step 5, performing Monte Carlo integration according to the probability density function values, the material estimation values and the light radiation values corresponding to the outgoing light rays to obtain a rendering result;

[0067] Step 6, constructing a first loss function according to the rendering result and a scene picture as a supervision label, and using the first loss function to optimize the parameters of the signed distance field network, the neural material field network and the neural radiance field network;

[0068] Step 7, use the parameter-optimized symbolic distance field network, the neural material field network, and the neural radiance field network to respectively perform geometry prediction, material prediction, and illumination prediction.

[0069] Step 8, provide a material editing function, use the material editing function to edit the material prediction result output by the parameter-optimized neural material field network, and then render the material editing result.

[0070] Step 9, provide an illumination editing function, use the illumination editing function to edit the illumination prediction result output by the parameter-optimized neural radiance field network, and then render the illumination editing result.

[0071] In practical applications, the memory can be a volatile memory close to the end, such as RAM, and can also be a non-volatile memory, such as ROM, FLASH, floppy disk, mechanical hard disk, etc., and can also be a remote storage cloud. The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), i.e., the steps of the scene reverse rendering method based on neural implicit representation can be implemented through these processors.

[0072] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A method for scene inverse rendering based on neural implicit representation, characterized in that, The method comprises the following steps: a symbol distance field network, a neural material field network, and a neural radiance field network are constructed based on a neural network, wherein the symbol distance field network is represented as a symbol distance function defined as the minimum distance from any three-dimensional point in the scene to all surfaces in the scene, the geometry prediction result is represented as a symbol distance function value, and a three-dimensional point with a symbol distance function value of zero is extracted to form a geometric surface; three-dimensional points in the scene are input into the neural material field network and the symbol distance field network for material prediction and geometry prediction, and then volume rendering is performed to obtain material information and normal information; wherein the material is represented as three parameters of the GGX micro-surface material model, including diffuse reflectance, specular reflectance, and roughness; the neural material field network outputs a first RGB value as the diffuse reflectance, a second RGB value as the specular reflectance, and a floating-point number as the roughness according to the input three-dimensional points; importance sampling is performed according to the material information and the normal information, and an outgoing light ray and a corresponding probability density function value and material estimation value of the outgoing light ray are calculated, including: on the geometric surface, importance sampling is performed according to the roughness contained in the material information to obtain an outgoing light ray, the probability distribution of the importance sampling follows the GGX micro-surface material model and is determined by the roughness, the incident direction of the sampling point and the surface normal information are input into the GGX micro-surface material model to calculate the direction of the outgoing light ray and the probability density function value, and the diffuse reflectance, the specular reflectance, and the roughness contained in the material prediction result are input into the GGX micro-surface material model to calculate the material estimation value; the outgoing light ray is input into the symbol distance field network for geometry prediction and calculation of the intersection with the geometric surface, the intersection is input into the neural radiance field network for illumination prediction, and volume rendering is performed according to the illumination prediction result to obtain the illumination radiation value corresponding to the outgoing light ray; Monte Carlo integration is performed according to the probability density function value corresponding to the outgoing light ray, the material estimation value, and the illumination radiation value to obtain a rendering result; a first loss function is constructed according to the rendering result and a scene picture as a supervision label, and the symbol distance field network, the neural material field network, and the neural radiance field network are parameter-optimized using the first loss function; geometry prediction, material prediction, and illumination prediction are respectively performed using the parameter-optimized symbol distance field network, neural material field network, and neural radiance field network. A material editing function and an illumination editing function are also provided.

2. The scene inverse rendering method based on neural implicit representation according to claim 1, characterized in that, Further comprising: a second loss function is constructed according to the normal information and a normal map as a supervision label, and the symbol distance field network, the neural material field network, and the neural radiance field network are parameter-optimized using the second loss function and the first loss function.

3. The scene inverse rendering method based on neural implicit representation according to claim 1 or 2, characterized in that, Further comprising: three-dimensional points in the scene are input into the symbol distance field network for geometry prediction, and then volume rendering is performed to obtain depth information; a third loss function is constructed according to the depth information and a depth map as a supervision label; and the symbol distance field network, the neural material field network, and the neural radiance field network are parameter-optimized using the third loss function and the first loss function, or the first loss function, the second loss function, and the third loss function.

4. The scene inverse rendering method based on neural implicit representation according to claim 1, characterized in that, The method further comprises: constructing a density function of the Laplace surface distribution according to a peak point of the Laplace surface distribution being located at a zero point of the signed distance function; The method further comprises: obtaining the material information by volume rendering, including: weighting and accumulating the material prediction result and the density function of the Laplace surface distribution to obtain the material information; The method further comprises: obtaining the light radiation value corresponding to the exit light by volume rendering, including: weighting and accumulating the light prediction result and the density function of the Laplace surface distribution to obtain the light radiation value; The method further comprises: obtaining the normal information by geometry prediction and volume rendering, including: weighting and accumulating the gradient of the signed distance function and the density function of the Laplace surface distribution to obtain the normal information; The method further comprises: obtaining the depth information by geometry prediction and volume rendering, including: weighting and accumulating the distance from the sampling point to the origin and the density function of the Laplace surface distribution to obtain the depth information.

5. The scene inverse rendering method based on neural implicit representation according to claim 1, characterized in that, The method further comprises: The method further comprises: performing material editing on the material prediction result output by the parameter-optimized neural material field network using a material editing function, and then rendering the material editing result.

6. The scene inverse rendering method based on neural implicit representation according to claim 1, characterized in that, The method further comprises: performing light editing on the light prediction result output by the parameter-optimized neural radiation field network using a light editing function, and then rendering the light editing result.

7. A device for scene inverse rendering based on neural implicit representation, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the scene reverse rendering method based on neural implicit representation according to any one of claims 1-6 when executing the computer program.

Images