A three-dimensional reconstruction method based on improved NERF

Abstract

This invention relates to the field of 3D reconstruction technology, specifically disclosing a 3D reconstruction method based on an improved NERF (NeRF-based) approach. The method includes: performing multiple independent semi-global matching operations on multiple spacecraft images to obtain depth information; mapping the depth information to a high-dimensional space to obtain an h-dimensional first vector; training the first vector to output a predicted point density; reducing the dimension of the first vector to h / 2 and then training it to output a predicted color value; mapping the direction of sunlight to a high-dimensional space to obtain an h-dimensional second vector; training the first and second vectors to obtain a predicted shadow scalar; reducing the dimension of the second vector to h / 2 and then training it to obtain a predicted environment color cast; and constructing a 3D model of the spacecraft based on the predicted point density, predicted color value, predicted shadow scalar, and predicted environment color cast. This invention makes targeted improvements to traditional NeRF technology to adapt to the 3D reconstruction of spacecraft in a space environment.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and more specifically to a 3D reconstruction method based on an improved NERF. Background Technology

[0002] The observation conditions for spacecraft in the space environment are very limited. The number of observation images is limited and their relative positions to the sun are complex and variable. This corresponds to the problem of spacecraft reconstruction under sparse multi-view images and varying image illumination conditions.

[0003] Multi-view stereo (MVS) algorithms are currently commonly used 3D reconstruction techniques. This technique requires data such as sparse point clouds and camera parameters obtained by structure from motion (SFM) as a basis, and then goes through steps such as dense reconstruction, surface reconstruction, mesh optimization and texture mapping to achieve 3D reconstruction of objects in a dense reconstruction manner.

[0004] Traditional Neural Radiance Fields (NeRF) technology, a new paradigm for 3D reconstruction proposed in recent years, uses a multilayer perceptron (MLP) neural network to implicitly learn a static 3D scene, enabling the synthesis (rendering) of arbitrary new perspectives in complex scenes. It requires multi-view input data for training.

[0005] However, algorithms like MVS rely on sparse point clouds acquired through SFM and camera sensor parameters. In the space environment, due to the inherent instability of cameras and other sensors, and the varying lighting conditions between views, the accuracy and reliability of feature extraction and matching by MVS algorithms are difficult to guarantee. Consequently, the 3D reconstruction accuracy of MVS algorithms is low, and the success rate is also low. Furthermore, MVS recovers the texture information of the spacecraft surface through texture mapping, but the image quality acquired in the space environment is poor, often resulting in low fidelity of the reconstructed 3D model surface.

[0006] Traditional NeRF requires dense multi-view input and cannot cope with the complex and variable lighting conditions in the space environment. As a result, the failure rate of spacecraft 3D reconstruction based on traditional NeRF is relatively high. Summary of the Invention

[0007] To address the aforementioned problems, the purpose of this invention is to provide a three-dimensional reconstruction method based on an improved NERF. This method addresses the challenges of complex observation conditions in the space environment, the limited number of multiple views observed, and the complex and variable sunlight conditions. It makes targeted improvements to the traditional NeRF technology to adapt to the three-dimensional reconstruction of spacecraft in the space environment.

[0008] This invention provides a three-dimensional reconstruction method based on an improved NERF, comprising:

[0009] Step S1: Acquire multiple images of the spacecraft and the direction of sunlight;

[0010] Step S2: Perform multiple independent semi-global matching operations on the multiple spacecraft images to obtain the depth information corresponding to each spacecraft image;

[0011] Step S3: Map the depth information to a high-dimensional space using the SIREN mapping formula to obtain an h-dimensional first vector;

[0012] Step S4: Perform first-depth supervised training on the first vector using an MLP network to output the predicted point density;

[0013] Step S5: Reduce the dimension of the first vector to h / 2 dimensions using an MLP network, then perform second deep supervised training, and output the predicted color value;

[0014] Step S6: Map the direction of sunlight to a high-dimensional space using the SIREN mapping formula to obtain a second vector of h dimensions;

[0015] Step S7: Train the first vector and the second vector using an MLP network to obtain the predicted shadow scalar;

[0016] Step S8: Reduce the dimension of the second vector to h / 2 dimensions using an MLP network, and then perform a third deep supervised training to obtain the predicted environment color cast.

[0017] Step S9: Construct a three-dimensional model of the spacecraft based on the predicted point density, predicted color value, predicted shadow scalar, and predicted environmental color cast.

[0018] In one possible implementation, step S3 includes:

[0019] The SIREN mapping formula γ(x) can be expressed by the following formula:

[0020] γ(x)=(sin(2 0 πx),cos(2 0 πx),sin(2 1 πx),cos(2 1 πx),...,sin(2 L-1 πx),cos(2 L-1 πx))

[0021] In the formula, x is the three-dimensional coordinate of the pixel and L is the frequency.

[0022] In one possible implementation, step S4 includes:

[0023] The loss function for the first deep supervised training can be expressed by the following formula.

[0024]

[0025] In the formula, d pred To predict depth, d GT For true depth, This is the set of ray directions used for the first depth of supervised training.

[0026] In one possible implementation, step S5 includes:

[0027] The loss function L for the second deep supervised training is expressed by the following formula. RGB ((R):

[0028]

[0029] In the formula, c pre (r) represents the predicted color value of the pixel, c GT (r) corresponds to the color value of the real point, r is the position or index of the pixel, and R is the set of pixels.

[0030] In one possible implementation, step S8 includes:

[0031] The loss function for third-level deep supervised training can be expressed by the following formula.

[0032]

[0033] In the formula, N represents the direction of the light ray. SC T represents the number of sampling points along the ray. i For transmittance, α i For opacity, s i Let r be the scalar value for predicting shadows, r be the direction of the ray, and i be the index of the sampling point.

[0034] One possible implementation also includes:

[0035] The loss function L of the 3D reconstruction method can be expressed by the following formula:

[0036]

[0037] In the formula, λ SC and λ DS This refers to the weight definition of supplementary items. For the loss function of the second deep supervised training, For the loss function of the third deep supervised training, The loss function for the first deep supervised training, For the direction of light, For the set of light directions, second-depth supervised training, This is a set of light directions used for first-depth supervised training.

[0038] In one possible implementation, step S9 includes:

[0039] The color C(r) of a spacecraft's 3D model is expressed by the following formula:

[0040]

[0041] In the formula, r(t) represents the viewing direction, σ represents the predicted point density, c represents the final color value, d represents the depth information, and t represents time. n The lower limit of integration is t, i.e., the proximal time or the start time. f The upper limit of the integration is the far-end time or the end time, and T(t) is the transmittance.

[0042] In one possible implementation, step S9 further includes:

[0043] The final color value c(x,ω,t) of a pixel is calculated using the following formula. j ):

[0044] c(x,ω,t j ) = c a (x)·(s(x,ω)+(1-s(x,ω))·a(ω))

[0045] In the formula, ω represents the direction of sunlight, s represents the predicted shadow scalar, and c a (x) represents the predicted color value of pixel x, a represents the ambient color cast, and t represents the predicted color value of pixel x. j For a point in time.

[0046] The present invention provides a 3D reconstruction method based on an improved NERF, which is designed for 3D reconstruction scenarios with sparse views. It obtains a low-resolution 3D point cloud map of the spacecraft object through semi-global matching (SGM), and performs depth supervision on the NeRF 3D reconstruction process based on the low-resolution 3D point cloud map, guiding sample point resampling to reduce dependence on the input view, adapt to 3D reconstruction under sparse views, and significantly improve the success rate of 3D reconstruction. Addressing challenges such as the variable relative position between the sun and the spacecraft in space and complex lighting conditions, the method considers the sun's position ω in the NeRF MLP network, reducing the impact of lighting changes on the model's 3D reconstruction and improving the quality of 3D reconstruction. Attached Figure Description

[0047] Figure 1 A schematic flowchart of a three-dimensional reconstruction method provided for an embodiment of the present invention;

[0048] Figure 2 A flowchart of the improved NeRF algorithm provided for embodiments of the present invention. Detailed Implementation

[0049] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0050] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0051] Figure 1 A schematic flowchart of a three-dimensional reconstruction method provided in an embodiment of the present invention. Figure 2 The improved NeRF algorithm flowchart provided for embodiments of the present invention, combined with Figure 1 and Figure 2 This invention provides a three-dimensional reconstruction method based on an improved NERF, comprising:

[0052] Step S1: Acquire multiple images of the spacecraft and the direction of sunlight;

[0053] Step S2: Perform multiple independent semi-global matching operations on multiple spacecraft images to obtain the depth information corresponding to each spacecraft image;

[0054] Step S3: Map the depth information to a high-dimensional space using the SIREN mapping formula to obtain the first vector of dimension h.

[0055] In one possible implementation, step S3 includes:

[0056] The SIREN mapping formula γ(x) can be expressed by the following formula:

[0057] γ(x)=(sin(2 0 πx),cos(2 0 πx),sin(2 1 πx),cos(2 1 πx),...,sin(2 L-1 πx),cos(2L-1 πx))

[0058] In the formula, x is the three-dimensional coordinate of the pixel, and L is the frequency, which is the spatial dimension after mapping.

[0059] Step S4: Perform first-depth supervised training on the first vector using an MLP network to output the predicted point density;

[0060] In one possible implementation, all layers in the MLP network are fully connected. The first stage of the MLP network, MLP1, processes coordinate information: after the SIREN transformation, the high-dimensional vector γ(x) is input into the MLP network. During training, the depth information d obtained through SGM is used for supervised training, and finally the point density information σ is output.

[0061] The loss function for the first deep supervised training can be expressed by the following formula.

[0062]

[0063] In the formula, d pred To predict depth, d GT For true depth, This is the set of ray directions used for the first depth of supervised training.

[0064] Step S5: Reduce the dimension of the first vector to h / 2 dimensions using an MLP network, then perform second deep supervised training, and output the predicted color value;

[0065] In one possible implementation, the dimension of the h-dimensional intermediate feature vector output by MLP1 in the MLP network is reduced by h / 2 before further training to obtain the diffuse color prediction c. a c a The color of the spacecraft is its inherent color, independent of the direction of observation. For example, in the ideal case of direct sunlight, the spacecraft's color is entirely determined by c. a The decision is made based on the loss function, which is the integral of the difference between the predicted color value and the true color value of a pixel along the direction of ray projection.

[0066] The loss function L for the second deep supervised training is expressed by the following formula. RGB ((R):

[0067]

[0068] In the formula, c pre (r) represents the predicted color value of the pixel, c GT (r) corresponds to the color value of the real point, r is the position or index of the pixel, and R is the set of pixels.

[0069] Step S6: Map the direction of sunlight to a high-dimensional space using the SIREN mapping formula to obtain a second vector of dimension h.

[0070] Step S7: Train the first vector and the second vector using an MLP network to obtain the predicted shadow scalar;

[0071] Step S8: Reduce the dimension of the second vector to h / 2 dimensions using an MLP network, and then perform a third deep supervised training to obtain the predicted environment color cast.

[0072] In one possible implementation, step S7 includes:

[0073] The loss function for third-level deep supervised training can be expressed by the following formula.

[0074]

[0075] In the formula, N represents the direction of the light ray. SC T represents the number of sampling points along the ray. i For transmittance, α i For opacity, s i Let r be the scalar value for predicting shadows, r be the direction of the ray, and i be the index of the sampling point.

[0076] Step S9: Construct a 3D model of the spacecraft based on the predicted point density, predicted color value, predicted shadow scalar, and predicted environmental color cast.

[0077] In one possible implementation, the color C(r) observed by the spacecraft is observed along an arbitrary line-of-sight direction r(t) based on the predicted point density σ and the final color value c.

[0078] The color C(r) of a spacecraft's 3D model is expressed by the following formula:

[0079]

[0080] In the formula, r(t) represents the viewing direction, σ represents the predicted point density, c represents the final color value, d represents the depth information, and t represents time. n The lower limit of integration is t, i.e., the proximal time or the start time. f The upper limit of the integration is the far-end time or the end time, and T(t) is the transmittance.

[0081] After undergoing a SIREN transformation, the direction of sunlight ω is input separately into a neural network for training to obtain the environmental color cast α, which describes the global color cast of shadows in space.

[0082] The final color value c(x,ω,t) of a pixel is calculated using the following formula. j ):

[0083] c(x,ω,t j ) = c a (x)·(s(x,ω)+(1-s(x,ω))·a(ω))

[0084] In the formula, ω represents the direction of sunlight, s represents the predicted shadow scalar, and c a (x) represents the predicted color value of pixel x, a represents the ambient color cast, and t represents the predicted color value of pixel x. j For a point in time.

[0085] In one possible implementation, the loss function L of the 3D reconstruction method is expressed by the following formula:

[0086]

[0087] In the formula, λ SC and λ DS This refers to the weight definition of supplementary items. For the loss function of the second deep supervised training, For the loss function of the third deep supervised training, The loss function for the first deep supervised training, For the direction of light, A set of light directions, used for second-depth supervised training. This is a set of light directions used for first-depth supervised training.

[0088] This invention improves NeRF technology, enabling 3D reconstruction of spacecraft in complex space environments and enhancing reconstruction accuracy. It also further improves the efficiency of 3D reconstruction by learning and storing features in a voxel grid and then inputting them into an MLP for predicting color and density, thereby increasing training and inference speed.

[0089] The 3D reconstruction method based on improved NERF provided by this invention addresses the problems of limited number of spacecraft images and variable lighting in the space environment. On the basis of traditional NeRF, it uses sparse depth information obtained by SGM for depth supervision, reducing the dependence on the number of input multi-views and successfully achieving 3D reconstruction under sparse multi-view conditions. Variables such as the direction of sunlight ω and the color cast of the space environment α are introduced into the MLP network to control their impact on model training, thereby improving the accuracy and realism of 3D reconstruction.

[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method based on an improved NERF, characterized in that, include: Step S1: Acquire multiple images of the spacecraft and the direction of sunlight; Step S2: Perform multiple independent semi-global matching operations on the multiple spacecraft images to obtain the depth information corresponding to each spacecraft image; Step S3: Map the depth information to a high-dimensional space using the SIREN mapping formula to obtain an h-dimensional first vector; Step S4: Perform first-depth supervised training on the first vector using an MLP network to output the predicted point density; Step S5: Reduce the dimension of the first vector to h / 2 dimensions using an MLP network, then perform second deep supervised training, and output the predicted color value; Step S6: Map the direction of sunlight to a high-dimensional space using the SIREN mapping formula to obtain a second vector of h dimensions; Step S7: Train the first vector and the second vector using an MLP network to obtain the predicted shadow scalar; Step S8: Reduce the dimension of the second vector to h / 2 dimensions using an MLP network, and then perform a third deep supervised training to obtain the predicted environment color cast. Step S9: Construct a three-dimensional model of the spacecraft based on the predicted point density, predicted color value, predicted shadow scalar, and predicted environmental color cast.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, Step S3 includes: The SIREN mapping formula γ(x) can be expressed by the following formula: γ(x)=(sin(20πx),cos(20πx),sin(2 1 πx),cos(2 1 πx),...,sin(2 L-1 πx),cos(2 L-1 πx)) In the formula, x is the three-dimensional coordinate of the pixel and L is the frequency.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, Step S4 includes: The loss function for the first deep supervised training can be expressed by the following formula. In the formula, d pred To predict depth, d GT For true depth, Variables related to deep supervision.

4. The three-dimensional reconstruction method according to claim 1, characterized in that, Step S5 includes: The loss function L for the second deep supervised training is expressed by the following formula. RGB (R): In the formula, c pre (r) represents the predicted color value of the pixel, c GT (r) represents the color value of the real point, r is the position of the pixel, and R is the set of pixels.

5. The three-dimensional reconstruction method according to claim 1, characterized in that, Step S8 includes: The loss function for third-level deep supervised training can be expressed by the following formula. In the formula, N represents the direction of the light ray. SC T represents the number of sampling points along the ray. i For transmittance, α i For opacity, s i Let r be the scalar value for predicting shadows, r be the direction of the ray, and i be the index of the sampling point.

6. The three-dimensional reconstruction method according to claim 1, characterized in that, Also includes: The loss function L of the 3D reconstruction method can be expressed by the following formula: In the formula, λ SC and λ DS This refers to the weight definition of supplementary items. For the loss function of the second deep supervised training, For the loss function of the third deep supervised training, The loss function for the first deep supervised training, For the direction of light, A set of light directions, used for second-depth supervised training. This is the set of ray directions used for the first depth of supervised training.

7. The three-dimensional reconstruction method according to claim 1, characterized in that, Step S9 includes: The color C(r) of a spacecraft's 3D model is expressed by the following formula: In the formula, r(t) represents the viewing direction, σ represents the predicted point density, c represents the final color value, d represents the depth information, and t represents time. n The lower bound of integration is the near-term, t. f The upper limit of the integration is the far-end time, and T(t) is the transmittance.

8. The three-dimensional reconstruction method according to claim 7, characterized in that, Step S9 further includes: The final color value c(x,ω,t) of a pixel is calculated using the following formula. j ): c(x,ω,t j )=c a (x)·(s(x,ω)+(1-a(x,ω))·(ω)) In the formula, ω represents the direction of sunlight, s represents the predicted shadow scalar, and c a (x) represents the predicted color value of pixel x, a represents the ambient color cast, and t represents the predicted color value of pixel x. j For a point in time.

Images