A Shiny-NeRF 3D Reconstruction Method Based on NeRF

Abstract

This invention discloses a Shiny-NeRF 3D reconstruction method based on an improved NeRF, belonging to the field of 3D reconstruction. The method includes: constructing a newly developed Shiny-NeRF 3D reconstruction method based on an improved NeRF; training a model using the Shiny-NeRF 3D reconstruction algorithm and saving the model parameters obtained after training; using the trained object reconstruction model to reconstruct the scene from a new perspective; obtaining the evaluation index of the validation set after the model validation process; and finally comparing the results and displaying the reconstruction effect. The proposed Shiny-NeRF method innovatively improves NeRF by reconstructing the NeRF architecture, incorporating an adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module, a spherical harmonic probe (SH-Probe) module, a spherical harmonic function attention mechanism (SH-Attention) module, and an anisotropic regularization (SH-S3) module. Shiny-NeRF can accurately reconstruct mirror-reflective objects and reconstruct object and scene lighting information with high fidelity by inputting multi-angle photos.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction, and more particularly to a Shiny-NeRF three-dimensional reconstruction method based on NeRF improvement. Background Technology

[0002] With the advancement of digital technology and computing power, 3D reconstruction technology has shown broad application prospects in fields such as virtual reality, medical image processing, cultural heritage preservation, and industrial manufacturing. In the field of 3D reconstruction, Neural Radiation Field (NeRF) has become an important scene modeling method. Unlike traditional explicit representation methods, NeRF implicitly represents 3D scenes through continuous functions. Leveraging its fully differentiable property, it can better optimize scenes, thereby generating photorealistic new perspectives using camera light. The NeRF model takes a set of 3D points and 5D coordinates (3D position and 2D view direction) as input, and generates the final color values ​​of these points through radiation field voxel rendering. This simple input and output allows NeRF to faithfully reconstruct realistic scenes.

[0003] NeRF uses an MLP architecture and surpasses traditional modeling methods through volume rendering and implicit scene representation. However, its static scene modeling assumptions and coupled radiation field output remain core limitations. Traditional NeRF methods typically parameterize radiation values ​​as a function of the view direction. Secondly, NeRF tends to represent specular reflections isotropically rather than anisotropically, resulting in blurred specular rendering in the generated view. To address the issues with NeRF, we propose Shiny-NeRF, a reconstruction algorithm that more closely approximates real-world scenes. This algorithm is an innovative improvement on NeRF, employing a Fresnel effect-based architecture to separate diffuse and specular reflection, enhancing the model's ability to reconstruct anisotropic properties in mirrored scenes. An Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module ensures the model captures indirect lighting during training and reconstruction, while also enhancing its ability to capture ambient lighting by calculating the relationship between indirect lighting and viewpoint, thus improving the reconstruction of mirrored objects. A spherical harmonic probe (SH-Probe) module is proposed, which captures and stores indirect lighting, improving rendering efficiency while accurately describing light field information. A spherical harmonic function attention mechanism (SH-Attention) module is proposed, which learns and extracts diffuse reflection features to distinguish the importance of different probes under different viewpoints. Finally, an anisotropic regularization (SH-S3) module is proposed to avoid the influence of excessive anisotropy on the model. We conducted experiments on the Blender dataset with different module settings to verify the improvement of our model compared to NeRF, and compared the reconstruction results. The results show that our proposed Shiny-NeRF 3D reconstruction method based on NeRF improvement can more accurately represent the real scene. Summary of the Invention

[0004] To address the challenge of reconstructing mirror objects and capturing the relationship between perspective dependence in the NeRF 3D reconstruction algorithm, this invention proposes a Shiny-NeRF 3D reconstruction method based on an improved NeRF algorithm. The objects generated by this method can reconstruct mirror scenes with higher fidelity.

[0005] Specifically, this application proposes a Shiny-NeRF 3D reconstruction method based on NeRF improvement, which includes the following steps:

[0006] S1. Construct a newly developed Shiny-NeRF 3D reconstruction method based on NeRF improvement;

[0007] S2. Use the Shiny-NeRF 3D reconstruction algorithm to train the model and save the model parameters obtained after training. Use the trained object reconstruction model to reconstruct the scene from a new perspective. After the model verification process, the evaluation index of the validation set can be obtained. Finally, the results are compared and the reconstruction effect is displayed.

[0008] Furthermore, step S1 specifically includes the Shiny-NeRF structure design. We propose a diffuse and specular reflection separation architecture based on the Fresnel effect to enhance the model's ability to reconstruct anisotropic properties of specular scenes; an adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module to ensure the model captures indirect lighting during training and reconstruction, while enhancing the model's capture of ambient lighting by calculating the relationship between indirect lighting and viewpoint, thereby improving the ability to reconstruct specular objects; a spherical harmonic probe (SH-Probe) module to improve rendering efficiency and accurately describe light field information by capturing and storing indirect lighting; a spherical harmonic function attention mechanism (SH-Attention) module to distinguish the importance of different probes under different viewpoints by learning and extracting diffuse reflection features; and an anisotropic regularization (SH-S3) module to avoid the influence of excessive anisotropy on the model. Specifically:

[0009] S11. The NeRF rendering architecture was reconstructed. Based on the Fresnel effect, the mirror scene was reconstructed by separating specular reflection and diffuse reflection. We used two distinct Transformer models to learn and reconstruct features. For diffuse reflection, we used a Transformer with a cross-layer residual architecture. By improving the ViT model, we were able to reconstruct the diffuse reflection features in the scene. For specular reflection, we constructed a Transformer with a specular reflection structure. It works together with our proposed adaptive encoding and resampling embedding module, spherical harmonic probe module, and spherical harmonic attention mechanism to reconstruct the specular reflection in the scene by learning the relationship between indirect lighting and viewpoint in the scene.

[0010] S12, Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module: Although spherical harmonic functions are widely used in various types of reconstruction tasks, the NeRF reconstruction algorithm has not incorporated spherical harmonic function embedding and improvement into the coarse-to-fine sampling of NeRF. For the first time, we combine spherical harmonic function embedding and spherical harmonic direction coding to reconstruct the coarse-to-fine training architecture of NeRF. Spherical harmonic functions are embedded in the resampling process of the coarse-to-fine stage. By combining spherical harmonic functions with Transformer, the relationship between viewpoint and color is captured, which makes up for the limitation of traditional low-order spherical harmonic functions in reconstructing high-frequency information and is closer to the real "Fresnel effect".

[0011] S13, Spherical Harmonic Probe (SH-Probe) module. Probes are an indirect lighting technique in computer graphics. However, in NeRF models, there is no method to reconstruct indirect lighting by using Transformer and spherical harmonic embedding. We are the first to combine spherical harmonic embedding with probes to propose the spherical harmonic probe. By using the Transformer architecture to capture and memorize the indirect lighting in the scene, we can significantly improve the rendering efficiency of our model and reconstruct mirror scenes with high fidelity.

[0012] S14, Spherical Harmonic Function Attention Mechanism (SH-Attention) Module: In the process of constructing the Shiny-NeRF method, we discovered the relationship between features in the coarse-to-fine stage of NeRF and innovatively proposed a new attention mechanism architecture. This is the first new module that can be applied to NeRF spherical harmonic functions and probes. This module can use low-frequency features to guide the reconstruction of high-frequency features in the coarse-to-fine stage of NeRF, which greatly improves the model's capture of mirror features and improves the mirror reconstruction effect.

[0013] S15, Anisotropic Regularization (SH-S3) Module: Since we use adaptive spherical harmonic function encoding and sampling embedding, this module introduces anisotropy to improve the expressive power of mirror scenes. However, excessive anisotropy can compromise the robustness of the model. Therefore, we propose the anisotropic regularization SH-S3 module, which consists of spherical harmonic function regularization and the optimized S3IM, and can effectively improve the robustness of the model to the reconstruction of mirror scenes.

[0014] Furthermore, in step S11, two Transformer architectures are used to reconstruct different reflection features, namely diffuse reflection and specular reflection. These two types of features are then stitched together to generate the final reconstructed image from a new perspective. This is described in formula (1):

[0015] RGB=Color_Mapping(Specular feature,Diffuse feature) (1)

[0016] The speacular feature and the diffuse feature are computed by two different Transformer architectures.

[0017] By separating diffuse and specular reflection features, the model can better learn the lighting information at different locations in the scene, improving the model's reconstruction of specular scenes—something most NeRF methods haven't achieved. Secondly, the Transformer-based approach can more effectively capture the intrinsic relationships between different angles in the scene, further enhancing the fidelity of the reconstruction.

[0018] Furthermore, in step S12, the Adaptive Spherical Harmonic Encoding and Resampling Embedding (SH Embedding MLP) module adds adaptive spherical harmonic function encoding to the coarse-to-fine stage of NeRF. This involves encoding the viewing angle using spherical harmonic functions while simultaneously adding spherical harmonic embedding to the resampling stage of NeRF. Note that spherical harmonic orientation encoding and spherical harmonic resampling embedding are two different types of methods. Spherical harmonic orientation encoding aims to establish the relationship between viewing angle-dependent spherical harmonic coefficients in NeRF; while spherical harmonic embedding refers to embedding additional viewing angle-dependent features in the coarse-to-fine resampling stage of NeRF. These features can establish the relationship between object color and viewing angle, and are also more conducive to the viewing angle learning color features, thereby modeling the "Fresnel effect" and generating a more realistic viewing angle.

[0019] Furthermore, in step S13, the spherical harmonic probe (SH-Probe) module addresses the challenge of fully modeling the lighting of the entire scene during rendering, as this involves ray tracing, which is extremely costly. Inspired by probe techniques in computer graphics, we heuristically combine them with spherical harmonic functions to construct the spherical harmonic probe. This model indirect lighting information from different viewpoints within the scene, avoiding expensive ray calculations. Simultaneously, the spherical harmonic probe module allows for better integration with the adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module, linking the lighting representation of the entire scene using spherical harmonic functions. The Transformer architecture is used to capture the relationships between spherical harmonic coefficients, compensating for the shortcomings of traditional spherical harmonic function methods in capturing high-frequency features.

[0020] Furthermore, the spherical harmonic function attention mechanism (SH-Attention) module in step S14 distinguishes the importance of the probes under different viewpoints by weighting the diffuse reflection features. In the modeling process, this enables the model to pay more attention to the feature information at the current angle, thereby achieving weighting of anisotropic features in the scene and enhancing the model's learning of anisotropic mirror features in the scene.

[0021] Furthermore, in step S15, the anisotropic regularization (SH-S3) module, while employing anisotropic spherical harmonic function embedding, significantly enhances the model's ability to reconstruct mirror scenes. However, real-world objects possess both anisotropic and isotropic properties. Excessive reconstruction of anisotropy can make the model's reconstruction results highly unstable, as isotropic information also exists in the scene. Therefore, to enhance the model's robustness, we propose the anisotropic regularization (SH-S3) module. While constraining the viewpoint correlation coefficient of the spherical harmonic function, it effectively balances the model's learning of anisotropy and isotropy through the lighting information constraints of the first two terms in the optimized S3IM, thereby enhancing the model's robustness.

[0022] Furthermore, step S2 specifically includes:

[0023] Input multiple photos from multiple angles, each with its corresponding camera parameter matrix to ensure mapping from the camera coordinate system to the world coordinate system; then feed the data into a neural network for training, and after 30 epochs, obtain the trained object parameter model. Use a validation file to validate the model and obtain evaluation metrics, while also obtaining the new perspective images generated after validation.

[0024] The beneficial effects of this invention are as follows:

[0025] 1. Using NeRF (Neural Radiation Field) as a 3D reconstruction method, complex mirror scenes can be expressed from simple inputs (x,y,z,θ,φ). Compared with traditional 3D reconstruction algorithms, NeRF has a more realistic mirror scene reconstruction effect.

[0026] 2. The improved Shiny-NeRF 3D reconstruction method based on NeRF proposes a diffuse and specular reflection separation architecture centered on the Fresnel effect, an adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module, a spherical harmonic probe (SH-Probe) module, a spherical harmonic function attention mechanism (SH-Attention) module, and an anisotropic regularization (SH-S3) module. This allows the model to learn diffuse and specular reflection information in the scene and the intrinsic relationship between viewpoints. The entire model is constructed using Transformer, making the modeling closer to the representation of real scenes, and it can reconstruct various details of complex objects.

[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0029] Figure 1 Shiny-NeRF network structure diagram;

[0030] Figure 2 Diffuse Transform, Adaptive Spherical Harmonic Coding and Resampling Embedding, Specular Transformer Diagram;

[0031] Figure 3 A diagram of the spherical harmonic function attention mechanism (SH-Attention);

[0032] Figure 4 Comparison of Shiny-NeRF modeling results; Detailed Implementation

[0033] The following detailed description provides further details on specific implementation methods.

[0034] To address three main problems in the NeRF 3D reconstruction algorithm—mirror modeling blur, difficulty in capturing the relationship between viewpoints, and the difficulty of viewpoint-dependent modeling affecting modeling quality—the Shiny-NeRF algorithm is proposed to improve upon these problems, aiming to approximate the restoration and representation of real-world scenes. Its network structure diagram is shown below. Figure 1 As shown, the specific solution of the present invention is as follows:

[0035] The improved Shiny-NeRF 3D reconstruction method based on NeRF includes the following steps:

[0036] S1. Construct a newly developed Shiny-NeRF 3D reconstruction method based on improved NeRF. Specific steps are as follows:

[0037] S11. By reconstructing the NeRF training architecture, it is separated into diffuse reflection feature reconstruction and specular reflection feature reconstruction. Each type of feature is assigned to a different Transformer architecture for reconstruction. The specific model architecture is attached to the instruction manual. Figure 1 As shown, the final RGB color mapping is given by formula (1);

[0038] S12, the Adaptive Spherical Harmonic Encoding and Resampling Embedding (SH Embedding MLP) module, improves upon NeRF. For the first time, we combine spherical harmonic function embedding and spherical harmonic direction encoding, reconstructing the coarse-to-fine training architecture of NeRF. During the resampling process in the coarse-to-fine stage, spherical harmonic functions are embedded. By combining spherical harmonic functions with the Transformer, the model can embed additional spherical harmonic function features during the resampling stage, thereby learning and decoupling color dependencies based on viewpoints. The powerful contextual analysis capabilities of the Transformer enable the model to capture the dependencies between viewpoints, thus faithfully modeling mirror features in the scene. The specific architecture is shown in the appendix of the manual. Figure 2 As shown, we demonstrate the Diffuse Reflection Transformer and the SHEmbedding MLP module, as well as the Specular Reflection Transformer.

[0039] The diffuse Transformer uses a cross-layer residual architecture, where l represents the layer number, and the information of each layer is represented by S. l γ represents the cross-layer residual, as shown in formula (2):

[0040]

[0041] S13, the Spherical Harmonic Probe (SH-Probe) module, is the first to combine spherical harmonic embedding with a probe. It uses Transformer and spherical harmonic embedding to reconstruct indirect lighting. By capturing and learning indirect lighting in the scene through the Transformer architecture, it significantly improves the rendering efficiency of our model. The specific module architecture is shown in the attached manual. Figure 1 As shown:

[0042] S14. Spherical Harmonic Function Attention Mechanism (SH-Attention) Module: In the process of constructing the Shiny-NeRF method, we proposed a new attention mechanism architecture by studying the relationship between features in the coarse-to-fine stage of NeRF. This is the first new module that can be applied to NeRF spherical harmonic functions and probes. This module can capture the relationship between viewpoint and color in the coarse-to-fine stage of NeRF and use low-frequency features to guide the reconstruction of high-frequency features, which greatly improves the model's capture of mirror features and improves the mirror reconstruction effect.

[0043] S15, Anisotropic Regularization (SH-S3) Module: Since we used adaptive spherical harmonic function encoding and sampling embedding, this module introduces anisotropy to improve the expressive power of mirror scenes. However, excessive anisotropy will destroy the robustness of the model. Therefore, we propose the anisotropic regularization SH-S3 module, which consists of spherical harmonic function regularization, Equation (4), and the optimized S3IM, Equation (5). It can effectively improve the robustness of the model to the reconstruction of mirror scenes. The total model loss is shown in Equation (6):

[0044]

[0045] L = L re +λL SH +μL S3IM (6)

[0046] S2. The Shiny-NeRF 3D reconstruction algorithm is used to train the model, and the model parameters obtained from the training of each type of reconstructed object are saved. The trained object reconstruction model is used to reconstruct the scene from a new perspective. After the model validation process, the evaluation index of the validation set can be obtained. Finally, the results are compared and the reconstruction effect is displayed, as shown in the attached figure in the instruction manual. Figure 4 As shown.

[0047] S21. Input multiple photos from multiple angles, and each photo should have its corresponding camera parameter matrix to ensure the mapping from the camera coordinate system to the world coordinate system; then feed the data into the neural network for training, and after 30 epochs, the trained object parameter model can be obtained. Use the validation file to validate the model to obtain the evaluation index, and at the same time, you can also obtain the new perspective image generated after validation.

[0048] In summary, this invention proposes a Shiny-NeRF 3D reconstruction method based on NeRF improvement. It reconstructs the traditional NeRF rendering architecture, representing a novel 3D reconstruction approach. By separating diffuse and specular reflections in the scene and reconstructing information using different Transformers, it significantly improves the model's learning efficiency. Furthermore, each module is not independent of the model but rather establishes connections by leveraging the advantages between modules, thereby improving the overall performance of the model. Our method can approximate the Fresnel effect in real-world scenes, resulting in more faithful generated images.

Claims

1. A Shiny-NeRF 3D reconstruction method based on NeRF improvement, the method comprising: S1. Construct a newly developed Shiny-NeRF 3D reconstruction method based on NeRF improvement; S2. Use the Shiny-NeRF 3D reconstruction algorithm to train the model and save the model parameters obtained after training. Use the trained object reconstruction model to reconstruct the scene from a new perspective. After the model verification process, the evaluation index of the validation set can be obtained. Finally, the results are compared and the reconstruction effect is displayed. The method S1 includes the Shiny-NeRF structure design, which proposes a diffuse reflection and specular reflection separation architecture based on the Fresnel effect to enhance the model's ability to reconstruct the anisotropic properties of specular scenes. An adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module is proposed to ensure the model captures indirect lighting during training and reconstruction. It also enhances the model's capture of ambient lighting by calculating the relationship between indirect lighting and viewpoint, thereby improving the ability to reconstruct mirror objects. A spherical harmonic probe (SH-Probe) module is proposed to improve rendering efficiency and accurately describe light field information by capturing and storing indirect lighting. A spherical harmonic function attention mechanism (SH-Attention) module is proposed to distinguish the importance of different probes under different viewpoints by learning and extracting diffuse reflection features. An anisotropic regularization (SH-S3) module is proposed to avoid the influence of excessive anisotropy on the model. Specifically: S11 reconstructs the NeRF rendering architecture. Based on the Fresnel effect, it reconstructs mirror scenes by separating specular reflection and diffuse reflection, using two different Transformer models to learn and reconstruct features. For diffuse reflection, a Transformer with a cross-layer residual architecture is used, and the ViT model is improved to reconstruct diffuse reflection features in the scene. For specular reflection, a Transformer that constructs a specular reflection structure works together with the proposed adaptive encoding and resampling embedding module, spherical harmonic probe module, and spherical harmonic attention mechanism to reconstruct specular reflection in the scene by learning the relationship between indirect lighting and viewpoint in the scene. The S12 module, Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP), combines spherical harmonic function embedding and spherical harmonic direction coding to reconstruct the NeRF training architecture from coarse to fine. It embeds spherical harmonic functions during the resampling process in the coarse-to-fine stage and uses the combination of spherical harmonic functions and Transformer to capture the relationship between viewpoint and color. This makes up for the limitation of traditional low-order spherical harmonic functions in reconstructing high-frequency information and is closer to the real "Fresnel effect". S13, the SH-Probe module, combines spherical harmonic embedding with probes to propose the spherical harmonic probe. Through the Transformer architecture, it captures and learns indirect lighting in the scene, which greatly improves the rendering efficiency of the model and can reconstruct mirror scenes with high fidelity. S14, Spherical Harmonic Function Attention Mechanism (SH-Attention) Module: A new attention mechanism architecture is proposed, which is a new module that can be applied between NeRF spherical harmonic functions and probes. This module can use low-frequency features to guide the reconstruction of high-frequency features in the coarse-to-fine stage of NeRF, which improves the model's capture of mirror features and improves the mirror reconstruction effect. S15, Anisotropic Regularization (SH-S3) Module: The anisotropic regularization SH-S3 module is proposed. It consists of spherical harmonic function regularization and optimized S3IM, which can effectively improve the robustness of the model to the reconstruction of mirror scenes.

2. The Shiny-NeRF 3D reconstruction method based on NeRF improvement according to claim 1, characterized in that: In step S11, two Transformer architectures are used to reconstruct different reflection features, namely diffuse reflection and specular reflection. The two types of features are then stitched together to generate the final new perspective reconstruction map; as described in formula (1): (1) The speacular feature and the diffuse feature are computed by two different Transformer architectures.

3. The Shiny-NeRF 3D reconstruction method based on NeRF improvement according to claim 1, characterized in that: In step S12, the Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module adds adaptive spherical harmonic function coding to the coarse-to-fine stage of NeRF. By performing spherical harmonic function coding on the viewing angle, spherical harmonic embedding is added to the resampling stage of NeRF. Spherical harmonic direction coding is used to establish the relationship between the viewing angle-dependent spherical harmonic coefficients in NeRF. Spherical harmonic embedding refers to embedding additional viewing angle-dependent features in the coarse-to-fine resampling stage of NeRF. These features can establish the relationship between object color and viewing angle, and at the same time, they can be more conducive to the learning of color features by the viewing angle, thereby modeling the "Fresnel effect" and generating a more realistic viewing angle.

4. The Shiny-NeRF 3D reconstruction method based on NeRF improvement according to claim 1, characterized in that: The spherical harmonic probe (SH-Probe) module in step S13 combines probe technology from computer graphics with spherical harmonic functions to form a spherical harmonic probe. This model indirect lighting information from different viewpoints in the scene, avoiding expensive ray calculations. At the same time, the spherical harmonic probe module can better coordinate with the adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module, linking the lighting representation of the entire scene with spherical harmonic functions. The Transformer architecture is used to capture the relationships between spherical harmonic coefficients, making up for the shortcomings of traditional spherical harmonic function methods in capturing high-frequency features.

5. The Shiny-NeRF 3D reconstruction method based on NeRF improvement according to claim 1, characterized in that: In step S14, the spherical harmonic function attention mechanism (SH-Attention) module weights the spherical harmonic probes under different viewpoints by using diffuse reflection features, thereby distinguishing the importance of the probes under different viewpoints. In modeling, this allows the model to pay more attention to the feature information at the current angle, thereby weighting the anisotropic features in the scene and enhancing the model's learning of the anisotropic mirror features in the scene.

6. The Shiny-NeRF 3D reconstruction method based on NeRF improvement according to claim 1, characterized in that: The anisotropic regularization (SH-S3) module in step S15, while constraining the viewpoint correlation coefficient of the spherical harmonic function, can effectively balance the model's learning between anisotropy and isotropy by constraining the first two terms of illumination information in the optimized S3IM, thereby enhancing the model's robustness. Step S2 is as follows: Input multiple photos from multiple angles, each with its corresponding camera parameter matrix to ensure mapping from the camera coordinate system to the world coordinate system; then feed the data into a neural network for training, and after 30 epochs, obtain the trained object parameter model. Use a validation file to validate the model and obtain evaluation metrics, while also obtaining the new perspective images generated after validation.

Images