Efficient three-dimensional scene anti-rendering method and system based on multi-view image, and storage medium

Abstract

The invention discloses an efficient three-dimensional scene anti-rendering method and system based on a multi-view image and a storage medium, and the method comprises the steps: employing a neural symbol distance function network as a geometric model to represent a scene geometric structure of the multi-view image, decomposing a neural radiation field into a diffuse reflection field and a specular reflection field, in combination with the self-shielding probability of the mirror reflection field, the diffuse reflection field, the mirror reflection field and the self-shielding probability of the mirror reflection field, calculating color information of the spatial points and optimizing the geometric model; a material network is adopted to represent albedo and roughness and obtain material information, color information cached in a neural radiation field, an illumination visibility function and direct illumination are introduced, global illumination of each light ray is calculated, importance sampling and illumination Monte Carlo integral are combined to obtain a rendering result, and the material network and a direct illumination network are optimized; and on the basis of the geometric structure, the material information and the illumination information obtained by the decoupled geometric model, the material network and the direct illumination network, a high-quality image is generated by adopting rendering pipeline calculation.

Description

Technical Field

[0001] The present invention belongs to the technical field of the combination of computer vision and computer graphics, and specifically relates to an efficient three-dimensional scene de-rendering method, system and storage medium based on multi-view images. Background Art

[0002] Inverse rendering, as an important research direction in the field of computer vision and computer graphics, aims to deduce the geometric shape, material properties and lighting information of a three-dimensional scene from a two-dimensional image. This technology has broad application value in many fields such as virtual reality (VR), augmented reality (AR), digital content creation, intelligent perception, film and television special effects, digital cultural heritage protection and industrial design. Through inverse rendering, computers can analyze the real world based on the principle of physical consistency and generate highly realistic three-dimensional rendering results. However, due to the limitations of the expressive power and computational efficiency of current computing models, existing inverse rendering methods still face many challenges in terms of accuracy, lighting modeling, material decomposition, etc., making it difficult to meet the needs of high-quality three-dimensional reconstruction and real-time rendering.

[0003] In recent years, the rise of implicit neural representations has provided more powerful modeling capabilities for inverse rendering, making it possible to derive complete 3D information from limited input data. For example, NeRF (Neural Radiance Fields) uses a multi-layer perceptron (MLP) to model the radiance and density of a scene, enabling high-quality free-viewpoint synthesis. However, existing methods still face many challenges in geometry recovery, material decoupling, and lighting calculation, mainly manifested in the following aspects:

[0004] Limited geometric reconstruction accuracy: Existing neural implicit representation-based methods (such as NeRF and TensoRF) are mainly used for volume rendering rather than direct surface modeling, resulting in low reconstruction accuracy in highlight areas, complex geometric structures, and self-occluded areas. Geometric depressions, blurred boundaries, or artifacts may occur, affecting the usability and realism of the model.

[0005] Insufficient material and lighting decomposition: In many inverse rendering methods, direct lighting, indirect lighting, and the inherent material of the object are often difficult to effectively decouple. For example, NeRFactor only considers direct lighting and does not model indirect lighting. This causes shadow and global illumination information to be incorrectly embedded in material maps, resulting in distorted results during re-rendering and affecting lighting consistency.

[0006] Inadequate self-occlusion and indirect lighting modeling: Some methods (such as MII) use spherical Gaussians (SGs) to model indirect lighting, but this method struggles to capture high-frequency details, resulting in overly smooth, unrealistic renderings. Furthermore, many methods inadequately model shadow areas and indirect reflections, making it difficult to achieve physically consistent rendering under different lighting conditions.

[0007] Limited computational efficiency: Most inverse rendering methods based on NeRF and its variants rely on volume rendering integral calculations, and the training process is typically time-consuming. For example, traditional NeRF can take hours to days to train in high-resolution scenes. While optimization-based methods (such as TensoIR) optimize computation through tensor decomposition, they still cannot achieve both high-quality reconstruction and rendering speed in a short time, hindering the feasibility of practical applications. Summary of the Invention

[0008] In view of the above, the purpose of the present invention is to provide an efficient three-dimensional scene inverse rendering method, system and storage medium for multi-view images to address the limitations of existing inverse rendering methods in terms of geometric reconstruction accuracy, material and lighting decomposition, indirect lighting modeling, and computational efficiency.

[0009] To achieve the above-mentioned object of the invention, an embodiment provides an efficient three-dimensional scene de-rendering method for multi-view images, comprising the following steps:

[0010] Collect multi-view images and construct a single-scene dataset. Use a neural signed distance function network as a geometric model to represent the scene geometry of the single-scene dataset. Decompose the neural radiation field corresponding to the geometric model into diffuse reflection field and specular reflection field. Combined with the self-occlusion network, calculate the self-occlusion probability of the specular reflection field. Then, fuse the diffuse reflection field, specular reflection field and their self-occlusion probability, calculate the color information of the spatial point, and optimize the geometric model with volume rendering.

[0011] Based on spatial surface points, the material network is used to represent albedo and roughness to obtain material information. Based on this material information, the spatial point color information cached by the neural radiance field, the lighting visibility function, and the direct lighting predicted by the direct lighting network are introduced. Monte Carlo path tracing is used to calculate the global illumination of each ray. The global illumination of each ray is integrated by combining importance sampling and lighting Monte Carlo integration to obtain the rendering result, while optimizing the material network and direct lighting network.

[0012] Based on the geometric structure, material information and lighting information obtained from the decoupled geometric model, material network and direct lighting network, high-quality images are generated using rendering pipeline calculations.

[0013] Preferably, a neural signed distance function network is used as a geometric model to represent the scene geometry structure of a single scene dataset, including:

[0014] Spatial points are sampled from multi-view images and hash-encoded, then input into the geometric model to output the distance d, normal n, and feature f from the spatial point to the geometric surface, thereby obtaining the scene geometry.

[0015] Preferably, the neural radiation field corresponding to the geometric model is decomposed into a diffuse reflection field and a specular reflection field, including:

[0016] Input the spatial points sampled from the multi-view image into the diffuse reflection network, and input the distance d, normal n and feature f of the geometric model output spatial point to the geometric surface into the diffuse reflection network to obtain the diffuse reflection field L d (x);

[0017] The spatial point x and its ray direction ω sampled from the multi-view image o Input into the specular reflection network, and input the distance d, normal n and feature f of the geometric model output space point to the geometric surface into the specular reflection network to obtain the specular reflection field L s (x,ω o ).

[0018] Preferably, the self-occlusion network calculates the self-occlusion probability of the specular reflection field, including:

[0019] Ray direction ω based on spatial point o And calculate the reflection direction ω in combination with the normal n r , reflection direction ω r Input to the self-occlusion network, and the spatial point x is also input to the self-occlusion network to calculate the self-occlusion probability O(x,ω r ).

[0020] Preferably, the diffuse reflection field, the specular reflection field and their self-occlusion probability are integrated to calculate the color information of the spatial point, and the geometric model is optimized in combination with volume rendering, including:

[0021] L o (x,ω o )=L d (x)+L s (x,ω o )·(1-O(x,ω r ))

[0022] Among them, L d (x) represents the diffuse reflection field of the spatial point x, L s (x,ω o ) represents the specular reflection field, O(x,ω r ) represents the self-occlusion probability, L o(x,ω o ) represents the color information of a spatial point;

[0023] Optimizing the geometric model by combining volume rendering, including: calculating a color value C under volume rendering, and optimizing geometric model parameters based on a difference between the color value C and an image pixel value;

[0024]

[0025] Among them, L o (x i ,ω o ) represents the i-th sampling space point x of the ray i The color contribution, α i Indicates color contribution L o (x i ,ω o ) weight, T i Represents the cumulative transmittance of a spatial point, and N represents the total number of light sampling points.

[0026] Preferably, the material network is used to represent albedo and roughness based on spatial surface points to obtain material information. Based on the material information, the spatial point color information cached by the neural radiance field, the illumination visibility function, and the direct illumination predicted by the direct illumination network are introduced, and the global illumination of each light is calculated using Monte Carlo path tracing, including:

[0027] First, based on the spatial point color information L cached by the neural radiation field o (x ′ ,ω i ) represents indirect lighting, and the joint lighting visibility function V(x,ω i ) and the direct lighting network predicted direct lighting E(ω i ) Calculate global illumination L i (x,ω i ):

[0028] L i (x,ω i )=(1-V(x,ω i ))E(ω i )+V(x,ω i )L o (x ′ ,ω i )

[0029] Among them, ω i represents the i-th incident direction of surface point x;

[0030] Then based on the material information and global illumination L i (x,ω i) Calculate the ray rendering color c(x,ω) by Monte Carlo estimation i ),include:

[0031] c(x,ω i )=L i (x,ω i )f r (x,ω i ,ω o )(ω i ·n)

[0032] Among them, f r (x,ω i ,ω o ) represents the material information obtained by the microsurface BRDF based on the albedo and roughness predicted by the material network, n represents the normal, ω o Indicates the incident direction.

[0033] Preferably, importance sampling and illumination Monte Carlo integration are combined to integrate the global illumination of each ray to obtain a rendering result, including:

[0034] First, based on the pixel point x of the multi-view image c and the corresponding incident direction ω o , obtain the scene geometry through the geometric model, obtain the surface point x of the scene geometry, and use importance sampling to sample the incident direction ω of the light i ;

[0035] Then based on the surface point x and its incident direction ω i Calculate the global illumination L for each light i (x,ω i ) and integrate to get the rendering result c(x,ω o ):

[0036] c(x,ω o )=∫ Ω L i (x,ω i )f r (x,ω i ,ω o )(ω i ·n)dω i

[0037] Where Ω represents the integral space corresponding to the light.

[0038] Preferably, optimize the material network and direct lighting network, including:

[0039] Use the rendering color c(x,ω o ) and the difference between the image pixel value to optimize the material network and direct lighting network parameters.

[0040] To achieve the above-mentioned purpose of the invention, an embodiment also provides an efficient three-dimensional scene de-rendering system based on multi-view images, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned efficient three-dimensional scene de-rendering method based on multi-view images.

[0041] To achieve the above-mentioned object of the invention, an embodiment further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned efficient three-dimensional scene de-rendering method based on multi-view images.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] This paper uses a neural Signed Distance Function (SDF) network based on hash-coded neural implicit representation as a geometric model to construct scene geometry, improving the speed and accuracy of geometric reconstruction. The self-occlusion probability in complex lighting environments is calculated through an occlusion network, improving the quality of geometric reconstruction in highlight areas and addressing the inaccurate geometric restoration in self-occluded areas caused by existing methods.

[0044] Use hash-coded MLP as the material network to represent the spatially varying bidirectional reflectance distribution function (SVBRDF) to accurately estimate albedo and roughness, thereby effectively improving the ability to express material information;

[0045] Global illumination is calculated through Monte Carlo Path Tracing, and the color information of spatial points in the Neural Radiance Field Cache (NeRF Cache) is used to reduce the computational overhead of path tracing and improve computational efficiency. Importance Sampling and Monte Carlo Integration are combined to optimize the accuracy of illumination modeling, improve the physical consistency of global illumination effects, and make rendering results more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 This is a flowchart of an efficient three-dimensional scene de-rendering method based on multi-view images provided by an embodiment;

[0048] Figure 2 is a schematic diagram of geometric reconstruction provided by an embodiment;

[0049] Figure 3 is a schematic diagram of material and lighting reconstruction provided by the embodiment;

[0050] Figure 4 This is an example diagram of the lighting decomposition results of geometric materials in a three-dimensional scene provided by the embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0052] The purpose of the present invention is to provide an efficient three-dimensional scene de-rendering method based on multi-view images to solve the problems existing in the above-mentioned prior art, which can effectively decompose the three-dimensional scene based on multi-view images. Figure 1 As shown, the embodiment provides an efficient three-dimensional scene de-rendering method based on multi-view images, comprising the following steps:

[0053] S1, collects multi-view images and constructs a single scene dataset, uses the neural signed distance function network as the geometric model to represent the scene geometry of the single scene dataset, and decomposes the neural radiation field corresponding to the geometric model into diffuse reflection field and specular reflection field. Combined with the self-occlusion network, the self-occlusion probability of the specular reflection field is calculated, the diffuse reflection field, specular reflection field and their self-occlusion probability are fused, the color information of the spatial point is calculated, and the geometric model is optimized in combination with volume rendering.

[0054] In the embodiment, multi-view RGB images are collected, the pixel points and camera position information of the target object are extracted, and a single scene data set is constructed. The camera array includes N = 16 synchronized cameras, which are evenly distributed around the target object and cover a height of 0° to 45°. At the same time, AprilTag is used for calibration to obtain the internal and external parameter matrices of the camera as camera parameters. All camera parameters are optimized by global bundle adjustment to ensure the geometric consistency of multi-view data. The camera position and camera parameters can be used to obtain the ray position of the pixel, and the incident direction can be obtained for subsequent network parameter optimization.

[0055] In the embodiment, Figure 2As shown in Figure 1, a hash-coded neural signed distance function network (SDF Network) is used as a geometric model to represent the scene geometry structure of a single scene dataset. Using hash-coded SDF to perform 3D modeling on the geometric model can improve the reconstruction accuracy, specifically including:

[0056] The spatial point x is sampled from the multi-view image and hash-encoded, and then input into the geometric model to output the signed distance d, normal n and feature f of the spatial point to the geometric surface, thereby obtaining the scene geometric structure.

[0057] In the embodiment, the neural radiation field corresponding to the geometric model is also decomposed into a diffuse reflection field and a specular reflection field, such as Figure 2 Shown, including:

[0058] Input the spatial points sampled from the multi-view image into the diffuse reflection network (Diffuse Network), and input the distance d, normal n and feature f from the geometric model output spatial point to the geometric surface into the diffuse reflection network to obtain the diffuse reflection field (Diffuse Field) L d (x);

[0059] The spatial point x and its ray direction ω sampled from the multi-view image o Input into the specular reflection network (SpecularNetwork), and input the distance d, normal n and feature f of the geometric model output space point to the geometric surface into the specular reflection network to obtain the specular field (Specular Field) L s (x,ω o ).

[0060] In the embodiment, in order to optimize the reconstruction effect of the highlight area, a self-occlusion network is introduced to calculate the self-occlusion probability of the specular reflection field, including:

[0061] Ray direction ω based on spatial point o And calculate the reflection direction ω in combination with the normal n r , reflection direction ω r Input to the Occlusion Network, and the spatial point x is also input to the Occlusion Network to calculate the self-occlusion probability O(x,ω r ), expressed as:

[0062] O(x,ω r )=S(x,ω r ,n,f)

[0063] Among them, ω r =-2(ω o ·n)n+ω o is the reflection direction, and S is the blocking network.

[0064] In this embodiment, the self-occlusion network is trained using a four-layer MLP, and an additional occlusion compensation strategy is implemented in the specular reflection area. Specifically, an occlusion loss is introduced. During each training session, many points are sampled, each with an SDF distance. Points with an absolute value of the SDF distance less than a threshold are used to trace the reflection path from this point. Occlusion is determined on this ray, with a value of 1 for occlusion and 0 for occlusion. These two values are then regressed against the predicted occlusion probability of the point to prevent random occlusion predictions.

[0065] In the embodiment, the diffuse reflection field L is fused d (x), specular reflection field L s (x,ω o ) and its self-occlusion probability O(x,ω r ), calculate the color information L of the spatial point o (x,ω o ), and combined with volume rendering to optimize the geometric model, specifically including:

[0066] L o (x,ω o )=L d (x)+L s (x,ω o )·(1-O(x,ω r ))

[0067] To improve the quality of geometric reconstruction, this embodiment combines volume rendering technology for optimization. Specifically, the color value C under volume rendering is calculated, and the geometric model parameters are optimized based on the difference between the color value C and the image pixel value.

[0068]

[0069] Among them, L o (x i ,ω o ) represents the i-th sampling space point x of the ray i The color contribution, α i Indicates color contribution L o (x i ,ω o ) weight, T i Represents the cumulative transmittance of a spatial point, and N represents the total number of light sampling points.

[0070] S2, based on the spatial point, uses the material network to represent the albedo and roughness to obtain the material information. Based on the material information, the spatial point color information cached by the neural radiation field, the lighting visibility function, and the direct lighting predicted by the direct lighting network are introduced. The global illumination of each light is calculated using Monte Carlo path tracing. The global illumination of each light is integrated by combining importance sampling and lighting Monte Carlo integration to obtain the rendering result, and the material network and direct lighting network are optimized at the same time.

[0071] In the embodiment, Figure 3 As shown, the material network uses hash-coded MLP (multi-layer perceptron) to represent albedo and roughness, specifically based on the material information f obtained by the microsurface BRDF based on the albedo and roughness predicted by the material network. r (x,ω i ,ω o ), n represents the normal direction, ω o Indicates the viewing angle direction, that is, the incident direction.

[0072] like Figure 3 As shown in the figure, based on the material information, Monte Carlo path tracing is used to calculate the global illumination of each light, including:

[0073] First, based on the spatial point color information L cached by the neural radiation field o (x ′ ,ω i ) represents indirect lighting, and the joint lighting visibility function V(x,ω i ) and the direct illumination E(ω i ) Calculate global illumination L i (x,ω i ):

[0074] L i (x,ω i )=(1-V(x,ω i ))E(ω i )+V(x,ω i )L o (x ′ ,ω i )

[0075] Among them, ω i Represents the i-th incident direction of the spatial surface point x, the illumination visibility function V(x,ω i ) is the incident direction ω of the spatial surface point x i Through sphere tracing, for a ray, if it is direct illumination, then V(x,ω i )=0, if it is indirect lighting, then V(x,ω i)=1, the direct lighting network is based on the incident direction ω of the input spatial point x i The direct illumination E(ω i ), where the direct illumination network uses a fully connected network (MLP). According to the global illumination formula, when direct illumination is used, direct illumination E(ω i ) to calculate global illumination. When using indirect illumination, it is necessary to trace back to the indirect light emission point x ′ , corresponding to the color information L cached in step S1 o (x ′ ,ω i ) to calculate global illumination.

[0076] Then based on the material information and global illumination L i (x,ω i ) Calculate the global rendering color c(x,ω) of each ray by Monte Carlo path tracing i ),include:

[0077] c(x,ω i )=L i (x,ω i )f r (x,ω i ,ω o )(ω i ·n)

[0078] In order to optimize the material information, this embodiment combines Importance Sampling and Monte Carlo Integration to integrate the global rendering color of each light to obtain the rendering result, optimize the lighting sampling strategy, and reduce noise. Figure 3 As shown, specifically including:

[0079] First, based on the pixel point x of the multi-view image c and the corresponding incident direction ω o , obtain the scene geometry through the geometric model, obtain the surface point x of the scene geometry, and use importance sampling to sample the incident direction ω of the light i ;

[0080] Then based on the surface point x and its incident direction ω i Calculate the global illumination L for each light i (x,ω i ) and integrate to get the rendering result c(x,ω o ):

[0081] c(x,ω o )=∫ Ω L i(x,ω i )f r (x,ω i ,ω o )(ω i ·n)dω i

[0082] Where Ω represents the integral space corresponding to the light.

[0083] In the embodiment, based on the rendering result c(x,ω o ) optimizes the material network and direct lighting network by using the rendering color c(x,ω o ) and the difference between the image pixel value to optimize the material network and direct lighting network parameters.

[0084] S3 uses a rendering pipeline to calculate and generate high-quality images based on the geometric structure, material information, and lighting information obtained from the decoupled geometric model, material network, and direct lighting network.

[0085] In the embodiment, after optimizing the geometric model, material network and direct lighting network in steps S1 and S2, the geometric structure, material information and lighting information obtained based on the decoupled geometric model, material network and direct lighting network are used, and the geometric structure, material information and lighting information are input into the rendering pipeline to calculate and generate a high-quality image.

[0086] The final rendering uses a physically consistent rendering pipeline, which ensures the decoupling of the reconstructed geometry, materials, and lighting information. It supports re-rendering, which allows users to render new images from any perspective, and also supports lighting editing, which allows users to modify light sources and recalculate lighting effects.

[0087] After experimental training, the overall performance of the proposed method is good. The results of geometric material lighting reconstruction on the selected dataset are shown in Table 1.

[0088] Table 1 Experimental results of the method of the present invention on the data set

[0089]

[0090] The corresponding three-dimensional scene decomposition results are as follows Figure 4 As shown, combined Figure 4 As shown in Table 1, the model can accurately reconstruct the geometry and material information of the 3D scene and obtain reasonable lighting reconstruction.

[0091] The above method can efficiently reconstruct the geometric structure, material information and lighting information of the three-dimensional scene, improving the inverse rendering quality and computational efficiency.

[0092] The embodiment further provides a computing device including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, the device is used to implement the above-mentioned efficient three-dimensional scene de-rendering method based on multi-view images, which specifically includes the following steps:

[0093] S1: Collect multi-view images and construct a single-scene dataset. Use the neural signed distance function network as the geometric model to represent the scene geometry of the single-scene dataset. Decompose the neural radiation field corresponding to the geometric model into diffuse reflection field and specular reflection field. Combine the self-occlusion network to calculate the self-occlusion probability of the specular reflection field. Fuse the diffuse reflection field, specular reflection field and their self-occlusion probability to calculate the color information of the spatial point. Combine the volume rendering to optimize the geometric model.

[0094] S2, based on spatial points, uses a material network to represent albedo and roughness to obtain material information. Based on this material information, the spatial point color information cached by the neural radiance field, the lighting visibility function, and the direct lighting predicted by the direct lighting network are introduced. Monte Carlo path tracing is used to calculate the global illumination of each ray. The global illumination of each ray is integrated by combining importance sampling and lighting Monte Carlo integration to obtain the rendering result, while optimizing the material network and the direct lighting network.

[0095] S3 uses a rendering pipeline to calculate and generate high-quality images based on the geometric structure, material information, and lighting information obtained from the decoupled geometric model, material network, and direct lighting network.

[0096] The computing device provided in the embodiment, at the hardware level, includes not only a processor and memory, but also hardware required for other services such as an internal bus, a network interface, and memory. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the efficient three-dimensional scene de-rendering method based on multi-view images described in S1-S3 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0097] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for efficiently de-rendering a three-dimensional scene based on multi-view images is implemented, specifically comprising the following steps:

[0098] S1: Collect multi-view images and construct a single-scene dataset. Use the neural signed distance function network as the geometric model to represent the scene geometry of the single-scene dataset. Decompose the neural radiation field corresponding to the geometric model into diffuse reflection field and specular reflection field. Combine the self-occlusion network to calculate the self-occlusion probability of the specular reflection field. Fuse the diffuse reflection field, specular reflection field and their self-occlusion probability to calculate the color information of the spatial point. Combine the volume rendering to optimize the geometric model.

[0099] S2, based on spatial points, uses a material network to represent albedo and roughness to obtain material information. Based on this material information, the spatial point color information cached by the neural radiance field, the lighting visibility function, and the direct lighting predicted by the direct lighting network are introduced. Monte Carlo path tracing is used to calculate the global illumination of each ray. The global illumination of each ray is integrated by combining importance sampling and lighting Monte Carlo integration to obtain the rendering result, while optimizing the material network and the direct lighting network.

[0100] S3 uses a rendering pipeline to calculate and generate high-quality images based on the geometric structure, material information, and lighting information obtained from the decoupled geometric model, material network, and direct lighting network.

[0101] In the embodiment, computer-readable media includes permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0102] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An efficient 3D scene de-rendering method based on multi-view images, characterized in that: The following steps are involved: Collect multi-view images and construct a single-scene dataset. Use a neural signed distance function network as a geometric model to represent the scene geometry of the single-scene dataset. Decompose the neural radiation field corresponding to the geometric model into diffuse reflection field and specular reflection field. Combined with the self-occlusion network, calculate the self-occlusion probability of the specular reflection field. Then, fuse the diffuse reflection field, specular reflection field and their self-occlusion probability, calculate the color information of the spatial point, and optimize the geometric model with volume rendering. Based on spatial surface points, the material network is used to represent albedo and roughness to obtain material information. Based on this material information, the spatial point color information cached by the neural radiance field, the lighting visibility function, and the direct lighting predicted by the direct lighting network are introduced. Monte Carlo path tracing is used to calculate the global illumination of each ray. The global illumination of each ray is integrated by combining importance sampling and lighting Monte Carlo integration to obtain the rendering result, while optimizing the material network and direct lighting network. Based on the geometric structure, material information and lighting information obtained from the decoupled geometric model, material network and direct lighting network, high-quality images are generated using rendering pipeline calculations.

2. The efficient 3D scene de-rendering method based on multi-view images according to claim 1, characterized in that: The neural signed distance function network is used as a geometric model to represent the scene geometry structure of a single scene dataset, including: Spatial points are sampled from multi-view images and hash-encoded, then input into the geometric model to output the distance d, normal n, and feature f from the spatial point to the geometric surface, thereby obtaining the scene geometry.

3. The efficient 3D scene de-rendering method based on multi-view images according to claim 2, characterized in that: Decompose the neural radiation field corresponding to the geometric model into diffuse reflection field and specular reflection field, including: Input the spatial points sampled from the multi-view image into the diffuse reflection network, and input the distance d, normal n and feature f of the geometric model output spatial point to the geometric surface into the diffuse reflection network to obtain the diffuse reflection field L d (x); The spatial point x and its ray direction ω sampled from the multi-view image o Input into the specular reflection network, and input the distance d, normal n and feature f of the geometric model output space point to the geometric surface into the specular reflection network to obtain the specular reflection field L s (x,ω o ).

4. The efficient 3D scene de-rendering method based on multi-view images according to claim 2, characterized in that: The self-occlusion network calculates the self-occlusion probability of the specular reflection field, including: Ray direction ω based on spatial point o And calculate the reflection direction ω in combination with the normal n r , reflection direction ω r Input to the self-occlusion network, and the spatial point x is also input to the self-occlusion network to calculate the self-occlusion probability O(x,ω r ).

5. The efficient 3D scene de-rendering method based on multi-view images according to claim 2, characterized in that: The diffuse reflection field, specular reflection field and their self-occlusion probability are integrated to calculate the color information of the spatial point, and the geometric model is optimized by combining volume rendering, including: L o (x,ω o )=L d (x)+L s (x,ω o )·(1-O(x,ω r )) Among them, L d (x) represents the diffuse reflection field of the spatial point x, L s (x,ω o ) represents the specular reflection field, O(x,ω r ) represents the self-occlusion probability, L o (x,ω o ) represents the color information of a spatial point; Optimizing the geometric model by combining volume rendering, including: calculating a color value C under volume rendering, and optimizing geometric model parameters based on a difference between the color value C and an image pixel value; Among them, L o (x i ,ω o ) represents the i-th sampling space point x on the ray i The color contribution, α i Indicates color contribution L o (x i ,ω o ) weight, T i Represents the cumulative transmittance of a spatial point, and N represents the total number of light sampling points.

6. The efficient 3D scene de-rendering method based on multi-view images according to claim 1, characterized in that: Based on the spatial surface points, the material network is used to represent the albedo and roughness to obtain material information. Based on this material information, the spatial point color information cached by the neural radiation field, the lighting visibility function, and the direct lighting predicted by the direct lighting network are introduced. Monte Carlo path tracing is used to calculate the global illumination of each light, including: First, based on the spatial point color information L cached by the neural radiation field o (x ′ ,ω i ) represents indirect lighting, and the joint lighting visibility function V(x,ω i ) and the direct lighting network predicted direct lighting E(ω i ) Calculate global illumination L i (x,ω i ): L i (x,ω i )=(1-V(x,ω i ))E(ω i )+V(x,ω i )L o (x ′ ,oh i ) Among them, ω i represents the i-th incident direction of the spatial point x; Then based on the material information and global illumination L i (x,ω i ) Calculate the ray rendering color c(x,ω) by Monte Carlo estimation i ),include: c(x,ω i )=L i (x,ω i )f r (x,ω i ,oh o )(oh i ·n) Among them, f r (x,ω i ,ω o ) represents the material information obtained by the microsurface BRDF based on the albedo and roughness predicted by the material network, n represents the normal, ω o Indicates the incident direction.

7. The efficient 3D scene de-rendering method based on multi-view images according to claim 1, characterized in that: Combining importance sampling and illumination Monte Carlo integration to integrate the global illumination of each ray, the rendering results are as follows: First, based on the pixel point x of the multi-view image c and the corresponding incident direction ω o , obtain the scene geometry through the geometric model, obtain the surface point x of the scene geometry, and use importance sampling to sample the incident direction ω of the light i ; Then based on the surface point x and its incident direction ω i Calculate the global illumination L for each light i (x,ω i ) and integrate to get the rendering result c(x,ω o ): c(x,ω o )=∫ Ω L i (x,ω i )f r (x,ω i ,oh o )(oh i ·n)dω i Where Ω represents the integral space corresponding to the light.

8. The efficient 3D scene de-rendering method based on multi-view images according to claim 7, characterized in that: Optimized material networks and direct lighting networks, including: Use the rendering color c(x,ω o ) and the difference between the image pixel value to optimize the material network and direct lighting network parameters.

9. An efficient three-dimensional scene de-rendering system based on multi-view images, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the efficient three-dimensional scene de-rendering method based on multi-view images according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the efficient three-dimensional scene de-rendering method based on multi-view images according to any one of claims 1 to 8 is implemented.

Images