Improved Shiny-NeRF three-dimensional reconstruction method based on NeRF

Abstract

The invention discloses an improved Shiny-NeRF three-dimensional reconstruction method based on NeRF, and belongs to the field of three-dimensional reconstruction, and the method comprises the steps: constructing a newly researched and developed Shiny-NeRF three-dimensional reconstruction method based on NeRF; the method comprises the following steps: carrying out model training by using a Shiny-NeRF three-dimensional reconstruction algorithm, storing model parameters obtained by training a reconstructed object, carrying out new view angle reconstruction on a scene by using a trained object reconstruction model, obtaining an evaluation index of a verification set after a model verification process, and finally carrying out result comparison and reconstruction effect display. According to the Shiny-NeRF provided by the invention, the NeRF is innovated and improved through a reconstructed NeRF architecture, a self-adaptive spherical harmonic coding and resampling embedding (SH-Probe) module, a spherical harmonic function attention mechanism (SH-Attention) module and an anisotropic regularization (SH-S3) module; the Shiny-NeRF performs 3D reconstruction on the object in the multi-angle photo by inputting the multi-angle photo, so that a mirror reflection object, a high-fidelity reconstruction object and scene illumination information can be accurately reconstructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction, and particularly relates to a Shiny-NeRF three-dimensional reconstruction method based on NeRF improvement. BACKGROUND

[0002] With the improvement of digital technology and computing power, three-dimensional reconstruction technology has shown a wide application prospect in virtual reality, medical image processing, cultural heritage protection and industrial manufacturing fields. In the field of three-dimensional reconstruction, neural radiance field (NeRF) has become an important scene modeling method. Unlike traditional explicit expression methods, NeRF implicitly expresses three-dimensional scenes through continuous functions, and uses its full-link differentiable characteristics to better optimize the scene, so as to generate photo-level quality of new perspectives using camera rays. The NeRF model takes a set of three-dimensional points and 5D coordinates (3D position and 2D observation direction) as input, and generates the final color value of these points through radiance field voxel rendering. This simple input and output makes NeRF able to faithfully reconstruct real scenes.

[0003] NeRF uses an MLP architecture to surpass traditional modeling methods through volume rendering and implicit scene representation. However, its static scene modeling assumption and coupled radiance field output remain core limitations. Traditional NeRF methods typically simply parameterize radiance values ​​as a function of viewing direction. Furthermore, NeRF tends to represent specular reflections in an isotropic rather than anisotropic manner, resulting in blurry specular renderings across the entire viewing angle. In view of these problems of NeRF, we proposed Shiny-NeRF, a reconstruction algorithm that is closer to real scenes. This algorithm is an innovative improvement to the NeRF algorithm. We proposed a diffuse reflection and specular reflection separation architecture with the Fresnel effect as the core to enhance the model's ability to reconstruct the anisotropic properties of specular scenes; an adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module to ensure that the model captures indirect lighting during training and reconstruction. At the same time, by calculating the relationship between indirect lighting and viewpoint, the model enhances the capture of ambient lighting, thereby improving the ability to reconstruct specular objects; a spherical harmonic probe (SH-Probe) module is proposed to improve rendering efficiency while accurately describing light field information by capturing and memorizing indirect lighting; a spherical harmonic function attention mechanism (SH-Attention) module is proposed to distinguish the importance of different probes under different viewpoints by learning and extracting diffuse reflection features; and an anisotropic regularization (SH-S3) module is proposed to avoid the impact of excessive anisotropy on the model. We conducted experiments with different module settings on the Blender dataset to verify the improvement of our model over NeRF and compared the reconstruction effects. The results show that the Shiny-NeRF 3D reconstruction method we proposed based on NeRF can more accurately represent the situation in the real scene. Summary of the Invention

[0004] In order to solve the relationship problem between the difficulty of reconstructing mirror objects and the difficulty in capturing perspective dependence in the 3D reconstruction algorithm NeRF, the present invention proposes a Shiny-NeRF 3D reconstruction method based on NeRF, and the objects generated by its modeling can reconstruct mirror scenes more faithfully.

[0005] Specifically, this application proposes a NeRF-based improved Shiny-NeRF 3D reconstruction method, which includes the following steps:

[0006] S1. Build the newly developed Shiny-NeRF 3D reconstruction method based on NeRF improvement;

[0007] S2, model training is performed using the Shiny-NeRF three-dimensional reconstruction algorithm, and the model parameters of the reconstructed object obtained after training are saved, a new perspective reconstruction of the scene is performed using the trained object reconstruction model, and after the model verification process, the evaluation index of the verification set can be obtained, and finally the result comparison and reconstruction effect display are performed;

[0008] Further, the content of step S1 will specifically include Shiny-NeRF structure design, we take the Fresnel effect as the core to propose a separation architecture of diffuse reflection and specular reflection to enhance the model's ability to reconstruct the anisotropic properties of the specular scene; an adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module is used to ensure that the model captures indirect lighting during training and reconstruction, and by calculating the relationship between indirect lighting and viewing angle, the model's ability to capture ambient lighting is enhanced, thereby improving the ability to reconstruct specular objects; a spherical harmonic probe (SH-Probe) module is proposed, which captures and stores indirect lighting and improves rendering efficiency while accurately describing light field information; a spherical harmonic function attention mechanism (SH-Attention) module is proposed, which learns and extracts diffuse reflection features to distinguish the importance of different probes under different viewing angles; an anisotropic regularization (SH-S3) module is proposed to avoid the influence of excessive anisotropy on the model, which is specifically:

[0009] S11, the NeRF rendering architecture is reconstructed, and on the basis of the Fresnel effect, the specular reflection and diffuse reflection are separated to reconstruct the specular scene, and two different Transformer models are used to learn and reconstruct the features; for diffuse reflection, a Transformer with a cross-layer residual architecture is used, which is improved based on the ViT model to enable it to reconstruct the diffuse reflection features in the scene; for specular reflection, a Transformer structure for specular reflection is constructed, which works together with the adaptive coding and resampling embedding module, the spherical harmonic probe module and the spherical harmonic attention mechanism to learn the relationship between indirect lighting and viewing angle in the scene and reconstruct the specular reflection in the scene.

[0010] S12, adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module, although spherical harmonics are widely used in various types of reconstruction tasks, they are not embedded and improved in the NeRF sampling from coarse to fine in the NeRF reconstruction algorithm, we first combine spherical harmonic embedding and spherical harmonic direction coding to reconstruct the training architecture of NeRF from coarse to fine, embed spherical harmonics in the resampling process from coarse to fine, use spherical harmonics combined with Transformer to capture the relationship between viewing angle and color, and make up for the limitations of traditional low-order spherical harmonics in reconstructing high-frequency information, which is closer to the real "Fresnel effect".

[0011] S13, SH-Probe module, probe is an indirect lighting technique in computer graphics, but in NeRF model, there is no method to reconstruct indirect lighting by using Transformer and spherical harmonic embedding, we first combine spherical harmonic embedding with probe to propose SH-Probe, which captures and learns the indirect lighting in the scene through the Transformer architecture, greatly improves the rendering efficiency of our model, and can reconstruct the mirror scene with high fidelity.

[0012] S14, SH-Attention module, we found the relationship between the features of the coarse-to-fine stages of NeRF during the construction of the Shiny-NeRF method, and innovatively proposed a new attention mechanism architecture, which is the first time we propose a new module that can be applied to the NeRF spherical harmonic function and probe, this module can use low-frequency features to guide the reconstruction of high-frequency features in the coarse-to-fine stages of NeRF, greatly improving the capture of mirror features by the model and improving the mirror reconstruction effect.

[0013] S15, SH-S3 module, since we use adaptive spherical harmonic function encoding and sampling embedding, this module introduces anisotropy to improve the expression ability of mirror scenes, but excessive anisotropy will destroy the robustness of the model, so we propose the SH-S3 module of anisotropic regularization, which is composed of spherical harmonic function regularization and optimized S3IM, which can effectively improve the robustness of the model for mirror scene reconstruction.

[0014] Further, two different Transformer architectures are used in step S11 to reconstruct different reflection features, namely diffuse reflection and mirror reflection, and then the two types of features are spliced to generate the final new view reconstruction map. The description is as shown in formula (1):

[0015] RGB = Color_Mapping(Speacular feature, Diffuse feature) (1)

[0016] Where Speacular feature and Diffuse feature are calculated by two different Transformer architectures.

[0017] By separating the diffuse and specular reflection characteristics, the model can better learn the lighting information at different positions in the scene, improving the model's reconstruction of specular scenes, which most NeRF methods do not achieve. Second, the Transformer-based method can more effectively capture the intrinsic relationship between different angles in the scene, further improving the reconstruction fidelity.

[0018] Further, the adaptive spherical harmonic encoding and resampling embedding (SH Embedding MLP) module in step S12 adds adaptive spherical harmonic function encoding to the coarse-to-fine stage of NeRF. By encoding the viewing angle with spherical harmonics, the resampling stage of NeRF is added with spherical harmonic embedding. Note that spherical harmonic direction encoding and spherical harmonic resampling embedding are two different types of methods. Spherical harmonic direction encoding is to establish the relationship between the viewing angle-dependent spherical harmonic coefficients in NeRF; while spherical harmonic embedding refers to embedding additional viewing angle-dependent features in the resampling stage of the coarse-to-fine NeRF. This type of feature can establish the relationship between object color and viewing angle, and can more effectively learn the color features from the viewing angle, thereby modeling the "Fresnel effect" and generating more realistic viewing angles.

[0019] Further, the SH-Probe module in step S13, because it is difficult for the model to fully model the lighting of the entire scene in rendering, which involves ray tracing and is very expensive. Therefore, we are inspired by the probe technology in computer graphics, and combine it with spherical harmonics to form a spherical harmonic probe to model the indirect lighting information at different viewing angles in the scene, avoiding expensive ray calculations. At the same time, the use of the SH-Probe module can better work with the adaptive spherical harmonic encoding and resampling embedding (SH Embedding MLP) module to link the lighting of the entire scene with spherical harmonics, and use the Transformer architecture to capture the spherical harmonic coefficient relationship between the information, making up for the lack of high-frequency feature capture ability of traditional spherical harmonic methods.

[0020] Further, the SH-Attention module in step S14, by weighting the diffuse reflection characteristics against the spherical harmonic probe at different viewing angles, the importance of the probe at different viewing angles is distinguished, allowing the model to pay more attention to the feature information at the current angle in modeling, and achieving weighting of the anisotropic features in the scene, enhancing the model's learning of anisotropic specular features in the scene.

[0021] Furthermore, in step S15, the anisotropic regularization (SH-S3) module uses anisotropic spherical harmonic embedding, which significantly increases the model's ability to reconstruct mirror scenes. However, objects in real scenes have both anisotropic and isotropic properties. Excessive reconstruction of anisotropy will make the model reconstruction results very unstable, because isotropic information also exists in the scene. Therefore, in order to enhance the robustness of the model, we propose an anisotropic regularization (SH-S3) module. While constraining the viewing angle correlation coefficient of the spherical harmonic function, it can effectively balance the model's learning between anisotropy and isotropy through the optimized first two illumination information constraints in S3IM, thereby enhancing the robustness of the model.

[0022] Further, step S2 is specifically as follows:

[0023] Input multiple photos from multiple angles. Each photo should have its corresponding camera parameter matrix to ensure the mapping from the camera coordinate system to the world coordinate system. Then send the data to the neural network for training. After waiting for 30 epochs, the trained object parameter model can be obtained. The model is verified using a verification file to obtain evaluation indicators, and the new perspective images generated after verification can also be obtained.

[0024] The beneficial effects of the present invention are:

[0025] 1. Using NeRF (Neural Radiance Field) as a 3D reconstruction method, complex specular scenes can be expressed from simple inputs (x, y, z, θ, φ). Compared with traditional 3D reconstruction algorithms, NeRF has a more realistic specular scene reconstruction effect.

[0026] 2. The Shiny-NeRF 3D reconstruction method improved based on NeRF proposes a diffuse and specular reflection separation architecture with the Fresnel effect as the core, an adaptive spherical harmonic coding and resampling embedding (SH Embedding MLP) module, a spherical harmonic probe (SH-Probe) module, a spherical harmonic function attention mechanism (SH-Attention) module, and an anisotropic regularization (SH-S3) module. The model can learn the diffuse and specular reflection information in the scene, and can learn the intrinsic relationship between perspectives. The entire model is constructed through Transformer, making the modeling closer to the real scene expression, and even when facing complex objects, it can complete the reconstruction of its various details.

[0027] Additional advantages, objects, and features of the application will be apparent to those skilled in the art upon examination of the following specification. It is intended to be covered by the following claims. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to make the purposes, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be combined with the drawings, in which:

[0029] Figure 1 Shiny-NeRF network structure diagram;

[0030] Figure 2 Diffuse reflection Transform, adaptive spherical harmonic coding and resampling embedding, specular reflection Transformer diagram;

[0031] Figure 3 Spherical harmonic function attention mechanism (SH-Attention) diagram;

[0032] Figure 4 Shiny-NeRF modeling effect comparison diagram; DETAILED DESCRIPTION

[0033] The following will be further described in detail through specific embodiments.

[0034] In order to solve the three problems of specular modeling ambiguity in NeRF three-dimensional reconstruction algorithm, difficulty in capturing the relationship between view angles, and view angle dependent modeling difficulty affecting modeling quality, the Shiny-NeRF algorithm is proposed to improve it, aiming to approximate the restoration and expression of the real scene; the network structure diagram is as shown in Figure 1 The specific scheme of the present application is as follows:

[0035] The Shiny-NeRF three-dimensional reconstruction method improved based on NeRF includes the following steps:

[0036] S1, construct the newly developed Shiny-NeRF three-dimensional reconstruction method improved based on NeRF. The specific steps are:

[0037] S11, reconstruct the training architecture of NeRF, separate it into diffuse reflection feature reconstruction and specular reflection feature reconstruction, and give each type of feature to different Transformer architecture for reconstruction. The specific model architecture is shown in the accompanying drawings of the specification Figure 1 The final RGB color mapping is formula (1);

[0038] S12, adaptive spherical harmonic encoding and resampling embedding (SH Embedding MLP) module, improve NeRF, we first combine spherical harmonic function embedding and spherical harmonic direction encoding, and reconstruct the training architecture of NeRF from coarse to fine, embed spherical harmonic function in the resampling process of the coarse-to-fine stage, use spherical harmonic function combined with Transformer, so that the model can embed additional spherical harmonic function features in the resampling stage, thereby learning and decoupling the color dependence on the viewing angle, and the powerful context analysis capability of Transformer can make the model capture the dependence between viewing angles, thereby faithfully modeling the mirror feature information in the scene, the specific architecture is shown in the accompanying drawings of the specification Figure 2 We show the diffuse reflection Transformer (Diffuse Reflection Transformer) and the adaptive spherical harmonic encoding and resampling embedding (SH Embedding MLP) module, and the specular reflection Transformer (Specular Reflection Transformer).

[0039] Among them, the diffuse reflection Transformer uses a cross-layer residual architecture, l represents the number of layers, and the information of each layer is represented as S l , and γ represents the cross-layer residual, as shown in equation (2):

[0040]

[0041] S13, SH-Probe module, we first combine spherical harmonic embedding with probes, use Transformer and spherical harmonic embedding to reconstruct indirect lighting, capture and learn indirect lighting in the scene through the Transformer architecture, greatly improve the rendering efficiency of our model, the specific module architecture is shown in the accompanying drawings of the specification Figure 1

[0042] S14, SH-Attention module, in the process of building the Shiny-NeRF method, we propose a new attention mechanism architecture by studying the relationship between the features of the coarse-to-fine stage of NeRF, which is the first time we propose a new module that can be applied to the relationship between the spherical harmonic function and the probe of NeRF. This module can capture the relationship between the viewing angle and the color in the coarse-to-fine stage of NeRF, use low-frequency features to guide the reconstruction of high-frequency features, greatly improve the capture of mirror features by the model, and improve the mirror reconstruction effect.

[0043] ​S15, anisotropy regularization (SH-S3) module, since we use adaptive spherical harmonic function coding and sampling embedding, this module introduces anisotropy to improve the expression ability of the mirror surface scene, but excessive anisotropy will destroy the robustness of the model, so we propose an anisotropy regularization SH-S3 module, which is composed of spherical harmonic function regularization, formula (4) and optimized S3IM, formula (5), which can effectively improve the robustness of the model for mirror surface scene reconstruction, and the total model loss is shown in formula (6):

[0044]

[0045] L=L re +λL SH +μL S3IM (6)

[0046] S2, the Shiny-NeRF three-dimensional reconstruction algorithm is used for model training, and the model parameters of each type of reconstructed object are saved, the trained object reconstruction model is used for new view reconstruction of the scene, and the evaluation index of the verification set is obtained after the model verification process, and finally the result comparison and reconstruction effect are shown, as shown in the drawings Figure 4 .

[0047] S21, input multiple photos containing multiple angles, and each photo should have its corresponding camera parameter matrix to ensure the mapping from the camera coordinate system to the world coordinate system, then send the data into the neural network for training, and after waiting for 30 epochs, the trained object parameter model can be obtained, the evaluation index of the model is obtained by using the verification file to verify the model, and the new view picture generated after verification can also be obtained.

[0048] In summary, the Shiny-NeRF three-dimensional reconstruction method based on the improvement of NeRF is proposed, which reconstructs the rendering architecture of traditional NeRF, is a novel three-dimensional reconstruction method, separates the diffuse reflection and mirror emission in the scene, and reconstructs the information through different Transformers, greatly improving the learning efficiency of the model, secondly, each module is not independent of the model, but establishes a connection by utilizing the advantages between the modules, which can improve the overall performance of the model, our method can approximate the "Fresnel effect" in the real scene, and the generated photo is more faithful.

Claims

1. A NeRF-based improved Shiny-NeRF 3D reconstruction method, comprising: S1. Build the newly developed Shiny-NeRF 3D reconstruction method based on NeRF improvement; S2. Use the Shiny-NeRF 3D reconstruction algorithm to train the model and save the model parameters obtained after training. Use the trained object reconstruction model to reconstruct the scene from a new perspective. After the model verification process, the evaluation indicators of the verification set can be obtained. Finally, the results are compared and the reconstruction effect is displayed.

2. The NeRF-improved Shiny-NeRF 3D reconstruction method according to claim 1, characterized in that: The content of the method S1 will mainly include the Shiny-NeRF structure design. We propose a diffuse reflection and specular reflection separation architecture based on the Fresnel effect to enhance the model's ability to reconstruct the anisotropic properties of specular scenes; The Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module ensures that the model captures indirect lighting during training and reconstruction. By calculating the relationship between indirect lighting and viewpoint, the model enhances the capture of ambient lighting, thereby improving the ability to reconstruct specular objects. The Spherical Harmonic Probe (SH-Probe) module is proposed to capture and memorize indirect lighting, improving rendering efficiency while accurately describing light field information. The Spherical Harmonic Function Attention Mechanism (SH-Attention) module is proposed to distinguish the importance of different probes under different viewpoints by learning and extracting diffuse reflection features. The Anisotropic Regularization (SH-S3) module is proposed to avoid the impact of excessive anisotropy on the model. Specifically: S11. We reconstructed the NeRF rendering architecture. Based on the Fresnel effect, we reconstructed the specular scene by separating specular reflection from diffuse reflection. We used two completely different Transformer models to learn and reconstruct the features. For diffuse reflection, we used a Transformer with a cross-layer residual architecture. By improving the ViT model, we enabled it to reconstruct the diffuse reflection features in the scene. For specular reflection, we constructed a Transformer with a specular reflection structure, which works together with the adaptive encoding and resampling embedding module, spherical harmonic probe module, and spherical harmonic attention mechanism we proposed. By learning the relationship between indirect lighting and viewpoint in the scene, we can reconstruct the specular reflection in the scene. S12, Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module. Although spherical harmonics are widely used in various types of reconstruction tasks, the NeRF reconstruction algorithm does not embed and improve spherical harmonics during NeRF's coarse-to-fine sampling. For the first time, we combined spherical harmonic embedding with spherical harmonic direction encoding, reconstructed NeRF's coarse-to-fine training architecture, embedded spherical harmonics during the coarse-to-fine resampling process, and combined spherical harmonics with the Transformer to capture the relationship between perspective and color. This overcomes the limitation of traditional low-order spherical harmonics in reconstructing high-frequency information and is closer to the realistic "Fresnel effect." S13, Spherical Harmonic Probe (SH-Probe) module. The probe is an indirect lighting technology in computer graphics. However, in the NeRF model, there is no method that uses Transformer and SH embedding to reconstruct indirect lighting. We are the first to combine SH embedding with the probe and propose the SH-Probe. Through the Transformer architecture, the indirect lighting in the scene is captured and memorized, which greatly improves the rendering efficiency of our model and can reconstruct specular scenes with high fidelity. S14. Spherical harmonics attention mechanism (SH-Attention) module. In the process of building the Shiny-NeRF method, we discovered the relationship between NeRF features from coarse to fine stages and innovatively proposed a new attention mechanism architecture. This is the first time we have proposed a new module that can be applied between NeRF spherical harmonics and probes. This module can use low-frequency features to guide the reconstruction of high-frequency features in the coarse to fine stage of NeRF, greatly improving the model's capture of mirror features and improving the mirror reconstruction effect. S15, anisotropic regularization (SH-S3) module. Since we use adaptive spherical harmonics encoding and sampling embedding, this module introduces anisotropy to improve the expressiveness of mirror scenes. However, excessive anisotropy will destroy the robustness of the model. Therefore, we proposed the anisotropic regularization SH-S3 module, which consists of spherical harmonics regularization and optimized S3IM, which can effectively improve the robustness of the model for mirror scene reconstruction.

3. The NeRF-improved Shiny-NeRF 3D reconstruction method according to claim 2, characterized in that: In step S11, two Transformer architectures are used to reconstruct different reflection features, namely diffuse reflection and specular reflection, and then the two types of features are spliced ​​to generate the final new perspective reconstruction image. The description is shown in formula (1): RGB=Color_Mapping(Specular feature,Diffuse feature) (1) The Speacular feature and Diffuse feature are calculated by two different Transformer architectures. By separating diffuse and specular reflection features, the model can better learn lighting information at different locations in the scene, improving the model's reconstruction of specular scenes, something that most NeRF methods lack. Secondly, methods based on the Transformer architecture can more effectively capture the inherent connections between different angles in the scene, further improving the fidelity of the reconstruction.

4. The NeRF-based improved Shiny-NeRF 3D reconstruction method according to claim 2, characterized in that: The Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module in step S12 adds adaptive spherical harmonic function coding to the coarse-to-fine stage of NeRF. By encoding the observation angle with spherical harmonic functions, spherical harmonic embedding is added to the resampling stage of NeRF. Please note that spherical harmonic direction coding and spherical harmonic resampling embedding are two different types of methods. Spherical harmonic direction coding is used to establish the relationship between the view-dependent spherical harmonic coefficients in NeRF; while spherical harmonic embedding refers to embedding additional view-dependent features in the coarse-to-fine resampling stage of NeRF. Such features can establish the relationship between object color and view angle, and can also better facilitate the learning of view angle color features, thereby modeling the "Fresnel effect" and generating more realistic view angles.

5. The NeRF-based improved Shiny-NeRF 3D reconstruction method according to claim 2, characterized in that: The spherical harmonic probe (SH-Probe) module in step S13 is difficult to fully model the entire scene illumination during rendering, which involves ray tracing and is very expensive. Inspired by the probe technology in computer graphics, we combine it with spherical harmonics to form a spherical harmonic probe, which models indirect illumination information from different perspectives in the scene and avoids expensive ray calculations. Furthermore, the SH-Probe module can better interact with the Adaptive Spherical Harmonic Coding and Resampling Embedding (SH Embedding MLP) module, linking the illumination representation of the entire scene with spherical harmonics and using the Transformer architecture to capture the spherical harmonic coefficient relationships between information, thus making up for the traditional spherical harmonic method's lack of high-frequency feature capture capabilities.

6. The NeRF-based improved Shiny-NeRF 3D reconstruction method according to claim 2, characterized in that: The spherical harmonics attention mechanism (SH-Attention) module in step S14 weights the spherical harmonic probes at different viewing angles by the diffuse reflection features, thereby distinguishing the importance of the probes at different viewing angles. In modeling, this enables the model to pay more attention to the feature information at the current angle, achieves weighting of the anisotropic features in the scene, and enhances the model's learning of the anisotropic specular features in the scene.

7. The NeRF-based improved Shiny-NeRF 3D reconstruction method according to claim 2, characterized in that: In the anisotropic regularization (SH-S3) module in step S15, since we use anisotropic spherical harmonics embedding, this significantly increases the model's ability to reconstruct mirror scenes. However, objects in real scenes have both anisotropic and isotropic properties. Excessive reconstruction of anisotropy will make the model reconstruction results very unstable because isotropic information also exists in the scene. Therefore, in order to enhance the robustness of the model, we propose an anisotropic regularization (SH-S3) module. While constraining the viewing angle correlation coefficient of the spherical harmonics, it can effectively balance the model's learning between anisotropy and isotropy through the optimized lighting information constraints of the first two items in the S3IM, thereby enhancing the robustness of the model. Step S2 is specifically as follows: Input multiple photos from multiple angles. Each photo should have its corresponding camera parameter matrix to ensure the mapping from the camera coordinate system to the world coordinate system. Then send the data to the neural network for training. After waiting for 30 epochs, the trained object parameter model can be obtained. The model is verified using a verification file to obtain evaluation indicators, and the new perspective images generated after verification can also be obtained.

Images