Lunar soil test piece in-situ three-dimensional reconstruction method based on improved neural radiation field algorithm

Abstract

The invention belongs to the field of computer vision, image processing and space engineering, and discloses a lunar soil test piece in-situ three-dimensional reconstruction method based on an improved neural radiation field algorithm, and the method comprises the steps: obtaining a multi-view image set of a lunar soil test piece, and carrying out the self-adaptive adjustment of lunar surface illumination; the pose of the high-resolution camera is calculated through multi-view stereoscopic vision, and a depth map is obtained; the method comprises the following steps: generating a neural coding volume based on a Vision Transform model, and extracting a multi-dimensional feature vector; generating a neural radiation field, and predicting rendering features of the volume points; carrying out micro-volume rendering, and carrying out loss feedback optimization by using color rendering, depth and illumination; performing grid extraction by using a moving cube algorithm to obtain a three-dimensional grid model; for feature extraction difficulty and reconstruction mismatching difficulty caused by lunar soil background texture similarity, the problem of low reconstruction precision under a lunar surface non-uniform illumination condition is effectively solved, and high-quality and high-precision in-situ rapid three-dimensional reconstruction of a lunar soil test piece is realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, image processing and space engineering, and in particular to an in-situ three-dimensional reconstruction method of a lunar soil sample based on an improved neural radiation field algorithm. Background Art

[0002] Today, lunar in-situ construction technology has been listed as a key development direction by space powers due to its high efficiency in deep space exploration. Many related technical routes proposed by scholars all involve processing lunar soil into bricks as building materials. However, existing lunar soil bricks are mostly prepared based on simulated lunar soil and tested in a simulated ground environment. Their actual performance in extreme lunar scenarios is still unclear, and it is urgent to carry out relevant physical tests in situ to provide important material and structural data for future base construction.

[0003] In order to conduct effective physical experiments on the lunar surface, it is crucial to use 3D reconstruction technology to quickly and accurately obtain basic parameters such as the 3D size and surface roughness of lunar soil specimens. However, the lighting conditions on the lunar surface are extremely uneven, which significantly affects the image quality, resulting in low accuracy and poor quality of image-based 3D reconstruction. In addition, the texture structure of the lunar soil specimen is highly similar to the lunar background, which is easily confused during feature extraction, causing mismatching problems, introducing noise and erroneous data, and further increasing the difficulty of reconstruction. In recent years, 3D reconstruction technology based on neural radiance fields (NeRF) has attracted much attention due to its superior performance in image generation and reconstruction, but its application to the lunar environment still faces problems such as light sensitivity and long training time, which is not ideal for fast and efficient in-situ 3D reconstruction.

[0004] Therefore, in view of the extreme environment of the lunar surface, there is an urgent need to propose a novel in-situ rapid three-dimensional reconstruction method for lunar soil samples, so as to obtain high-quality image data under complex lighting conditions, effectively extract and match features under similar texture structure conditions, and perform fast and efficient calculations under resource-constrained conditions, so as to obtain high-precision, high-quality reconstruction models of lunar soil samples and provide reliable parameters for subsequent physical experiments. Summary of the invention

[0005] In order to overcome the low reconstruction accuracy caused by uneven illumination changes on the lunar surface and the difficulty in feature extraction and mismatching caused by similar background textures, and to achieve high-quality and high-precision in-situ rapid 3D reconstruction of lunar soil specimens, the present invention proposes an in-situ rapid 3D reconstruction method for lunar soil specimens based on an improved neural radiation field algorithm, which adopts an adaptive adjustment strategy for lunar surface illumination to optimize the quality of acquired images, and introduces the ViT model based on multi-view stereo vision and neural radiation field fusion (MVS-NeRF) to ensure fast and efficient extraction of lunar soil specimen features against the lunar background, thereby achieving in-situ accurate 3D reconstruction of lunar soil specimens under extreme conditions on the lunar surface.

[0006] To achieve the above object, according to one aspect of the present invention, a method for in-situ three-dimensional reconstruction of lunar soil specimens based on an improved neural radiation field algorithm is provided, comprising the following steps:

[0007] Step S1: Acquire a multi-angle image set of the lunar soil sample;

[0008] Step S2: Adaptively adjust the global and local illumination of the lunar surface for the image set obtained in step S1;

[0009] Step S3: Calculate the camera pose and obtain the depth image using multi-view stereo vision for the image adjusted in step S2;

[0010] Step S4: Generate a neural coding volume using the Vision Transformer model based on the depth image obtained in step S3 and extract a multi-dimensional feature vector;

[0011] Step S5: input the multi-dimensional feature vector encoding obtained in step S4 into a multi-layer perceptron to generate a neural radiation field and predict the rendering features of the volume point;

[0012] Step S6: Volume rendering is performed on the volume points obtained in step S5 using differentiable ray marching, and the neural radiation field network model is jointly optimized using three types of losses: color rendering, illumination, and depth, to implicitly represent the three-dimensional scene of the lunar soil specimen;

[0013] Step S7: The implicit representation (ie, the neural network) obtained in step S6 is made explicit using the marching cubes algorithm, thereby obtaining the final three-dimensional mesh model.

[0014] Preferably, the specific steps of step S1 are: use a high-resolution camera carried by a lunar rover / construction robot to perform all-round video recording around the lunar soil sample, and evenly select no less than 50 clear color RGB images from the video as the original image set.

[0015] Preferably, in step S2, in order to cope with the inconsistent image brightness caused by the change of lunar illumination, global illumination adaptive adjustment is first performed, and then local illumination adaptive adjustment is performed, and the brightness of the entire image set is adjusted using a gamma correction method and an adaptive histogram equalization method is used to enhance the local details of each image.

[0016] Preferably, the specific steps of step S2 are:

[0017] S21 gamma correction: First, the brightness histograms of all images are calculated and merged to form a global brightness histogram, which characterizes the brightness distribution of the entire image set and calculates the global brightness mean square error. Dynamically select an initial global gamma value based on the shape of the brightness distribution; iterate over the image set and apply the same gamma correction formula to each image Among them C input , C output denote the input and corrected pixel values ​​respectively, γ is the global gamma value;

[0018] Calculate the global brightness histogram and global brightness mean square error of the corrected image set Compare the two mean square errors and adjust the gamma value, iterating until until;

[0019] The global brightness mean square error calculation formula is:

[0020] Among them, I(x,y) is the brightness value of a pixel, μ is the global average brightness, and n is the number of all pixels;

[0021] S22 adaptive histogram equalization: Divide each image into multiple area blocks and calculate the grayscale histogram of each block; equalize the histogram of each block, that is, map the pixel value of each grayscale level to a new value to enhance the contrast of the block; merge the equalized results of all blocks into a complete image, and use the bilinear interpolation method to smooth the merged area and reduce the boundary effect of the block; traverse each image;

[0022] The equalization formula is: CDF(x)=∑p

[0023] Where CDF(x) is the cumulative distribution function, and p is the probability of each gray level;

[0024] In order to avoid noise amplification caused by over-enhancement, contrast limitation is introduced; that is, a threshold of the number of pixels at the gray level is set to limit the number of pixels at each gray level, and the excess pixels are allocated to other gray levels; after contrast limitation, the cumulative distribution function of each block needs to be recalculated, and the gray value mapping is performed according to the new CDF(x).

[0025] Preferably, in step S3, multi-view stereo vision is used to calculate the camera's internal and external parameters and depth map, and secondary compensation of lunar imaging illumination is performed through multi-view fusion; specifically, the camera pose is calculated based on the motion recovery structure algorithm to obtain the camera's internal and external parameters and sparse point cloud image, and the PatchMatch algorithm is used to find the best depth estimation of each image through random initialization and iterative optimization to serve as input information of the ViT model.

[0026] Preferably, the step S4 involves multi-channel input of known camera pose information, generated depth map and image illumination information, using a deep neural network ViT based on a self-attention mechanism to extract neural features under similar conditions of rock and soil texture structure of lunar test specimens, and completing effective reconstruction of neural coding volumes for neural radiation field generation representation;

[0027] The specific steps of step S4 are:

[0028] S41: Divide the depth map into n small blocks of size P×P, and flatten each small block into a vector to form an input sequence, denoted as patch i ;

[0029] S42: Use a linear layer to convert each small block into an embedding vector and map it to a high-dimensional feature space. The formula is as follows:

[0030] Patch i =W×patch i +b

[0031] Among them, Patch i is the transformed embedding vector, W is the weight matrix, and b is the bias term;

[0032] S43: Add position coding to each small block to retain the position information of each small block in the original image. Sine and cosine functions are usually used to generate position coding. For the k-th dimension position coding of the i-th small block, the formula is as follows:

[0033]

[0034] Among them, PE(pos,2k) and PE(pos,2k+1) correspond to odd-dimensional and even-dimensional position encodings respectively, and pos, k, and d are the encoding position index, dimension index, and embedding dimension of the model respectively;

[0035] S44: Assume that the illumination information includes the features of multiple light sources, and convert them into feature vectors with the same number of small blocks through a linear layer, denoted as Light i ; Similarly, the camera pose information is converted into a feature vector through a linear layer, denoted as Camera i ;

[0036] S45: Concatenate and fuse the small block features of the depth map, the illumination features, and the camera pose features to generate the final input features: Input i =Patch i +Light i +Camera i ;

[0037] S46: The final feature vector is input into the ViT model, and the self-attention mechanism is used to calculate the relationship between blocks. The feature vector of each small block is processed by the following formula:

[0038]

[0039] Among them, Attention(Q,K,V) is the attention mechanism calculation method; Calculate the similarity between the query and the key, and get a scalar by taking the dot product of the query matrix Q and the key matrix K, and normalize it to prevent the calculated value from being too large or too small; the softmax function converts the dot product result into a probability distribution to ensure that the sum of all attention weights is 1. Its output determines the weight coefficient of each value (V). The more relevant the key, the greater the proportion of its corresponding value in the weighted average. Q, K, and V are the matrices of image block query, key, and value, respectively, corresponding to different representations of image blocks. The query represents the feature vector of the current block, the key represents the feature vector of all other blocks, and the value represents the feature information of the block, including the actual semantic content. K T is the transpose of the key matrix, d k Represents the dimension of the key;

[0040] S47: After multiple Transformer encoder layer feature extraction, the final feature representation (x, d, f, z, l) of each small block is output for neural radiation field generation in downstream tasks, where x, d, f, z, l represent the three-dimensional position of the volume point, the camera pose, the linear interpolation neural feature at that location, and the multi-dimensional vector function corresponding to the depth and brightness respectively;

[0041] For the subsequent neural radiation field generation, the color rendering loss is optimized, and the pixel color feature c = (r, g, b) of the original image is supplemented in the final feature, which is regarded as an additional input. That is, the final neural feature vector output is expressed as (x, d, f, c, z, l);

[0042] The ViT model used uses a multi-head attention mechanism, which enables the model to learn information in parallel in different subspaces and speed up processing. After the self-attention layer, a feedforward neural network is used to process the features of each block, effectively fusing features at different levels and avoiding feature loss problems.

[0043] Preferably, step S5 generates a neural radiation field using a multi-layer perceptron neural network based on the multi-dimensional neural feature vector input generated in step S4, and outputs the color, volume density, illumination and depth features of each point in the reconstructed model, which is expressed as:

[0044] F θ (x,d,f,c,z,l)→(c′(r,g,b),σ,z′,l′)

[0045] Among them, F θ(x, d, f, c, z, l) represents the mapping function acting on the multi-dimensional neural feature vector input, c′(r, g, b) represents the color feature of the volume point x in the generated scene in the implicit expression, σ represents the volume density, which represents the accumulated radiation when the light passes through the volume point x, and z′, l′ represent the depth and illumination of the point respectively.

[0046] Preferably, the step S5 completes volume rendering by differentiable ray marching based on a given volume density and color, thereby realizing view synthesis. Specifically, the camera ray color C of any pixel in the tracking image is calculated to calculate the cumulative radiation at the sampling shadow point. The calculation formula is:

[0047]

[0048] Among them, r(x) represents the three-dimensional vector in space at the volume point x, σ(r(x)) is the estimated volume density value of the light at this point, c(r(x), d) corresponds to the estimated color value of the point, and the function T(x) represents the light from x 0 The cumulative transmittance to x is:

[0049]

[0050] By sampling and integrating each light ray, a composite image can be generated, which is the sum of the color contributions of all light rays in the scene: I = ∑αC, where α is the corresponding weight, usually the average weight.

[0051] Preferably, in step S6, three types of loss functions are defined to optimize the model when training the improved neural radiation field, including a rendering loss L that measures the difference between the synthetic image and the real image. render , the depth loss L that compares the difference between the predicted depth and the true depth depth and the illumination loss L which evaluates the difference between the predicted illumination and the real illumination light , are supervised based on the mean square error, and the calculation formula is as follows:

[0052]

[0053] The comprehensive loss function is expressed as:

[0054] L Total =λ 1 L render +λ 2 L depth +λ 3 L light

[0055] Among them, λ i To control the factors, adjust the weight proportion of each loss function to obtain the best scene representation of the neural radiation field;

[0056] The model optimization process uses a back-propagation algorithm to adjust the parameters of the neural network and update image features and network weights. To avoid overfitting, the comprehensive loss function is regularized, including L1 regularization, which adds the absolute value of the weight parameter to the loss function and forces some weights to become zero to achieve feature selection. L2 regularization adds the square sum of the weight parameter to the loss function to suppress the size of the weight and prevent the model from being too complicated.

[0057] Preferably, the step S7 uses a marching cube algorithm to extract mesh features, realizes the explicitness of the implicit expression of the neural radiation field, and thus quickly obtains a high-quality three-dimensional mesh model, retaining clear surface boundaries and texture details of the lunar soil specimen;

[0058] The specific steps of step S7 are:

[0059] S71 scene voxelization: The implicit scene generated by the neural radiation field is discretized into a uniform cubic grid in three-dimensional space. Each grid unit is called a voxel. The volume density corresponding to each corner point in the unit is obtained by mapping the predicted density of the neural radiation field, and is estimated by taking the average value of random sampling within the voxel.

[0060] S72 Isosurface extraction: Set the isovalue threshold and compare it with the density of each corner point of the voxel, use the lookup table to determine the triangular composition of the isosurface in the cube, and then generate a triangular mesh to obtain an explicit expression of the object;

[0061] S73 combined post-processing: Smooth the generated mesh to remove noise and irregularities, use mesh simplification algorithms to remove redundant polygons and improve rendering efficiency, and output the final 3D network model.

[0062] In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:

[0063] 1. The present invention proposes an adaptive adjustment strategy for lunar illumination, which effectively solves the problem of low accuracy of three-dimensional reconstruction of lunar soil specimens caused by poor image quality and inconsistent luminosity under uneven illumination conditions on the lunar surface. The invention uses gamma correction to equalize the luminosity conditions of the entire image set to ensure a good overall reconstruction effect; uses adaptive histogram equalization to adjust the local luminosity of a single image, enhances the reconstruction effect of key areas, and retains local details of three-dimensional reconstruction. In addition, the subsequent use of multi-view stereo vision to calculate depth involves multi-view fusion, and the image can be compensated for illumination again.

[0064] 2. The present invention combines the ViT model based on the self-attention mechanism to extract multi-dimensional feature vectors, and combines it with the MVS-NeRF reconstruction method to effectively deal with the difficulty in feature extraction and reconstruction mismatching caused by the similar texture structure between the lunar soil specimen and the lunar background. ViT uses multi-head self-attention mechanisms at different levels, allowing the model to understand the subtle differences in the image from a global perspective, extract multi-scale features at different levels, greatly enhance the ability to distinguish similar structures, and accurately extract the structural features of the lunar soil specimen under the complex texture background of the lunar surface, reducing noise and erroneous data generation.

[0065] 3. The present invention proposes a method for in-situ rapid three-dimensional reconstruction of lunar soil specimens under extreme conditions on the lunar surface. Existing three-dimensional reconstruction methods have shown poor performance in the extreme environment of the lunar surface, or are sensitive to light, or have long reconstruction times, and cannot quickly and efficiently complete reconstruction to advance subsequent tasks. Moreover, when using lunar rovers or construction robots to carry out related tasks, they have limited computing and energy resources and insufficient endurance. The use of a fast in-situ reconstruction method will significantly reduce computing and energy requirements, improve mission energy efficiency, and facilitate the continued and efficient implementation of subsequent lunar missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of the present invention for rapid in-situ three-dimensional reconstruction of lunar soil specimens under lunar surface conditions.

[0067] Figure 2 It is a schematic diagram of obtaining lunar surface sampling photos in an embodiment of the present invention.

[0068] Figure 3 It is a diagram of the lunar surface adaptive illumination adjustment strategy in an embodiment of the present invention.

[0069] Figure 4 It is a flowchart of calculating a depth map based on multi-view stereo vision in an embodiment of the present invention.

[0070] Figure 5 Schematic diagram of the network structure for reconstructing the neural radiation field in combination with the VIT model in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0072] See also Figure 1The embodiment of the present invention provides a method for rapid in-situ three-dimensional reconstruction of lunar soil specimens based on an improved neural radiation field algorithm, comprising the following steps:

[0073] Step S1: Acquire a multi-angle image set of the lunar soil sample;

[0074] Step S2: Adaptively adjust the global and local illumination of the lunar surface for the image set obtained in step S1;

[0075] Step S3: Calculate the camera pose and obtain the depth image using multi-view stereo vision for the image adjusted in step S2;

[0076] Step S4: Generate a neural coding volume using the Vision Transformer model based on the depth image obtained in step S3 and extract a multi-dimensional feature vector;

[0077] Step S5: input the multi-dimensional feature vector encoding obtained in step S4 into a multi-layer perceptron to generate a neural radiation field, and predict the rendering features of the volume points, the rendering features including color value, volume density, illumination and depth value, etc.;

[0078] Step S6: Volume rendering is performed on the volume points obtained in step S5 using differentiable ray marching, and the neural radiation field network model is jointly optimized using three types of losses: color rendering, illumination, and depth, to implicitly represent the three-dimensional scene of the lunar soil specimen;

[0079] Step S7: The implicit representation (ie, the neural network) obtained in step S6 is made explicit using the marching cubes algorithm, thereby obtaining the final three-dimensional mesh model.

[0080] Among them, the specific steps of step S1 are: use the high-resolution camera carried by the lunar rover / construction robot to record all-round video around the lunar soil sample ( Figure 2 ), evenly select no less than 50 clear color RGB images from the video as the original image set.

[0081] See also Figure 3 In step S2, in order to cope with the image brightness inconsistency caused by the change of lunar illumination, a lunar illumination adaptive adjustment strategy is proposed. First, global illumination adaptive adjustment is performed, and then local illumination adaptive adjustment is performed. The gamma correction method is used to adjust the brightness of the entire image set, and the adaptive histogram equalization (CLAHE) method is used to enhance the local details of each image.

[0082] The specific steps of step S2 are:

[0083] S21 gamma correction: First, the brightness histograms of all images are calculated and merged to form a global brightness histogram, which characterizes the brightness distribution of the entire image set and calculates the global brightness mean square error. Dynamically select an initial global gamma value based on the shape of the brightness distribution; iterate over the image set and apply the same gamma correction formula to each image Among them C input , C output denote the input and corrected pixel values ​​respectively, γ is the global gamma value;

[0084] Calculate the global brightness histogram and global brightness mean square error of the corrected image set Compare the two mean square errors and adjust the gamma value, iterating until until;

[0085] The global brightness mean square error calculation formula is:

[0086] Among them, I(x,y) is the brightness value of a pixel, μ is the global average brightness, and n is the number of all pixels;

[0087] S22 adaptive histogram equalization: Divide each image into multiple area blocks and calculate the grayscale histogram of each block; equalize the histogram of each block, that is, map the pixel value of each grayscale level to a new value to enhance the contrast of the block; merge the equalized results of all blocks into a complete image, and use the bilinear interpolation method to smooth the merged area and reduce the boundary effect of the block; traverse each image;

[0088] The equalization formula is: CDF(x)=∑p

[0089] Where CDF(x) is the cumulative distribution function, and p is the probability of each gray level;

[0090] In order to avoid noise amplification caused by over-enhancement, the adaptive histogram equalization method in step S2 introduces contrast limitation, that is, setting a threshold for the number of pixels at each gray level, limiting the number of pixels at each gray level, and allocating excess pixels to other gray levels; after contrast limitation, it is necessary to recalculate the cumulative distribution function of each block and map the gray values ​​according to the new CDF(x).

[0091] See also Figure 4 In step S3, multi-view stereo vision is used to calculate the camera's internal and external parameters and depth map, and multi-view fusion is used to perform secondary compensation of lunar image illumination. Specifically, the camera pose (rotation matrix and translation vector) is calculated based on the motion recovery structure (SFM) algorithm to obtain the camera's internal and external parameters and sparse point cloud image, and the PatchMatch algorithm is used to find the best depth estimation of each image through random initialization and iterative optimization as the input information of the ViT model.

[0092] Step S4 involves multi-channel input of known camera pose information, generated depth map and image illumination information, and uses a deep neural network ViT (Vision transformer) based on self-attention mechanism to extract neural features and complete the effective reconstruction of neural coding volume.

[0093] See also Figure 5 , the specific steps of step S4 are:

[0094] S41: Divide the depth map into n small blocks of size P×P, and flatten each small block into a vector to form an input sequence, denoted as patch i ;

[0095] S42: Use a linear layer to convert each small block into an embedding vector and map it to a high-dimensional feature space. The formula is as follows:

[0096] Patch i =W×patch i +b

[0097] Among them, Patch i is the transformed embedding vector, W is the weight matrix, and b is the bias term;

[0098] S43: Add position coding to each small block to retain the position information of each small block in the original image. Sine and cosine functions are usually used to generate position coding. For the k-th dimension position coding of the i-th small block, the formula is as follows:

[0099]

[0100] Among them, E(pos,2k) and PE(pos,2k+1) correspond to odd-dimensional and even-dimensional position encodings respectively, and pos, k, and d are the encoding position index, dimension index, and embedding dimension of the model respectively;

[0101] S44: Assuming that the illumination information includes the features of multiple light sources, it can be converted into a feature vector with the same number of small blocks through a linear layer, denoted as Light i ; Similarly, the camera pose information (such as position and orientation) is converted into a feature vector through a linear layer, denoted as Camera i ;

[0102] S45: Concatenate and fuse the small block features of the depth map, the illumination features, and the camera pose features to generate the final input features: i =Patch i +Light i +Camera i ;

[0103] S46: The final feature vector is input into the ViT model, and the self-attention mechanism is used to calculate the relationship between blocks. The feature vector of each small block is processed by the following formula:

[0104]

[0105] Among them, Attention(Q,K,V) is the attention mechanism calculation method; Calculate the similarity between the query and the key, and get a scalar by taking the dot product of the query matrix Q and the key matrix K, and normalize it to prevent the calculated value from being too large or too small; the softmax function converts the dot product result into a probability distribution to ensure that the sum of all attention weights is 1. Its output determines the weight coefficient of each value (V). The more relevant the key, the greater the proportion of its corresponding value in the weighted average. Q, K, and V are the matrices of image block query, key, and value, respectively, corresponding to different representations of image blocks. The query represents the feature vector of the current block, the key represents the feature vector of all other blocks, and the value represents the feature information of the block, including the actual semantic content. K T is the transpose of the key matrix, d k Represents the dimension of the key;

[0106] S47: After multiple Transformer encoder layer feature extraction, the final feature representation (x, d, f, z, l) of each small block is output for downstream tasks (neural radiance field generation), where x, d, f, z, l represent the three-dimensional position of the volume point, the camera pose, the linear interpolation neural feature at that location, and the multidimensional vector function corresponding to the depth and brightness respectively;

[0107] For the subsequent neural radiation field generation, the color rendering loss is optimized, and the pixel color feature c = (r, g, b) of the original image is supplemented in the final feature and regarded as an additional input. That is, the final neural feature vector output is represented as (x, d, f, c, z, l).

[0108] The ViT model used in step S4 uses a multi-head attention mechanism to enable the model to learn information in different subspaces in parallel and speed up processing. After the self-attention layer, a feedforward neural network is used to process the features of each block, effectively fusing features at different levels and avoiding feature loss problems.

[0109] Step S5 is based on the multi-dimensional neural feature vector input generated in step S4, uses a multi-layer perceptron (MPL) neural network to generate a neural radiation field, and outputs the color, volume density, illumination, and depth features of each point in the reconstruction model, which are expressed as:

[0110] F θ (x,d,f,c,z,l)→(c′(r,g,b),σ,z′,l′)

[0111] Among them, F θ (x, d, f, c, z, l) represents the mapping function acting on the multi-dimensional neural feature vector input, c′(r, g, b) represents the color feature of the volume point x in the generated scene in the implicit expression, σ represents the volume density, which represents the accumulated radiation when the light passes through the volume point x, and z′, l′ represent the depth and illumination of the point respectively.

[0112] Step S5 completes volume rendering by differentiable ray marching based on the given volume density and color, thereby realizing view synthesis. Specifically, the camera ray color C of any pixel in the tracking image is calculated to calculate the cumulative radiation at the sampled shadow point. The calculation formula is:

[0113]

[0114] Among them, r(x) represents the three-dimensional vector in space at the volume point x, σ(r(x)) is the estimated volume density value of the light at this point, c(r(x), d) corresponds to the estimated color value of the point, and the function T(x) represents the light from x 0 The cumulative transmittance to x is:

[0115]

[0116] By sampling and integrating each light ray, a composite image (I) can be generated, which is the sum of the color contributions of all light rays in the scene: I = ∑αC, where α is the corresponding weight, usually the average weight.

[0117] Step S6 defines three types of loss functions for model optimization when training the improved neural radiation field, including the rendering loss L that measures the difference between the synthetic image and the real image. render , the depth loss L that compares the difference between the predicted depth and the true depth depth and the illumination loss L which evaluates the difference between the predicted illumination and the real illumination light , are supervised based on the mean square error, and the calculation formula is as follows:

[0118]

[0119] The comprehensive loss function can be expressed as: L Total =λ 1 L render +λ 2 L depth +λ 3 L light , where λ i To control the factors, the weight proportion of each loss function is adjusted to obtain the best scene representation of the neural radiation field.

[0120] The model optimization process uses a back-propagation algorithm to adjust the parameters of the neural network and update image features and network weights.

[0121] To avoid overfitting, the comprehensive loss function can usually be regularized, including L1 regularization, which adds the absolute value of the weight parameter to the loss function and forces some weights to become zero to achieve feature selection; L2 regularization, which adds the sum of the squares of the weight parameters to the loss function to suppress the size of the weight and prevent the model from being too complicated.

[0122] Step S7 uses the marching cube algorithm to extract mesh features and make the implicit expression of the neural radiation field explicit, so as to quickly obtain a high-quality three-dimensional mesh model and retain the clear surface boundaries and texture details of the lunar soil specimen;

[0123] The specific steps of step S7 are:

[0124] S71 scene voxelization: The implicit scene generated by the neural radiation field is discretized into a uniform cubic grid in three-dimensional space. Each grid unit is called a voxel. The volume density corresponding to each corner point in the unit is obtained by mapping the predicted density of the neural radiation field, and is estimated by taking the average value of random sampling within the voxel.

[0125] S72 Isosurface extraction: Set the isovalue threshold and compare it with the density of each corner point of the voxel, use the lookup table to determine the triangular composition of the isosurface in the cube, and then generate a triangular mesh to obtain an explicit expression of the object;

[0126] S73 combined post-processing: Smooth the generated mesh to remove noise and irregularities, use mesh simplification algorithms to remove redundant polygons and improve rendering efficiency, and output the final 3D network model.

[0127] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An in-situ 3D reconstruction method for lunar soil specimens based on an improved neural radiation field algorithm, characterized in that: The following steps are involved: Step S1: Acquire a multi-angle image set of the lunar soil sample; Step S2: Adaptively adjust the global and local illumination of the lunar surface for the image set obtained in step S1; Step S3: Calculate the camera pose and obtain the depth image using multi-view stereo vision for the image adjusted in step S2; Step S4: Generate a neural coding volume using the Vision Transformer model based on the depth image obtained in step S3 and extract a multi-dimensional feature vector; Step S5: input the multi-dimensional feature vector encoding obtained in step S4 into a multi-layer perceptron to generate a neural radiation field and predict the rendering features of the volume point; Step S6: Volume rendering is performed on the volume points obtained in step S5 using differentiable ray marching, and the neural radiation field network model is jointly optimized using three types of losses: color rendering, illumination, and depth, to implicitly represent the three-dimensional scene of the lunar soil specimen; Step S7: The implicit representation obtained in step S6 is made explicit using the marching cubes algorithm, thereby obtaining the final three-dimensional mesh model.

2. The in-situ 3D reconstruction method of lunar soil specimen based on the improved neural radiation field algorithm according to claim 1, characterized in that: The specific steps of step S1 are: use a high-resolution camera carried by the lunar rover / construction robot to perform all-round video recording around the lunar soil sample, and evenly select no less than 50 clear color RGB images from the video as the original image set.

3. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 1, characterized in that: In step S2, in order to cope with the inconsistent image brightness caused by the change of lunar illumination, global illumination adaptive adjustment is first performed, and then local illumination adaptive adjustment is performed, and the brightness of the entire image set is adjusted using the gamma correction method and the adaptive histogram equalization method is used to enhance the local details of each image.

4. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 2, characterized in that: The specific steps of step S2 are: S21 gamma correction: First, the brightness histograms of all images are calculated and merged to form a global brightness histogram, which characterizes the brightness distribution of the entire image set and calculates the global brightness mean square error. Dynamically select an initial global gamma value based on the shape of the brightness distribution; iterate over the image set and apply the same gamma correction formula to each image Among them C input , C output denote the input and corrected pixel values ​​respectively, γ is the global gamma value; Calculate the global brightness histogram and global brightness mean square error of the corrected image set Compare the two mean square errors and adjust the gamma value, iterating until until; The global brightness mean square error calculation formula is: Among them, I(x,y) is the brightness value of a pixel, μ is the global average brightness, and n is the number of all pixels; S22 adaptive histogram equalization: Divide each image into multiple area blocks and calculate the grayscale histogram of each block; equalize the histogram of each block, that is, map the pixel value of each grayscale level to a new value to enhance the contrast of the block; merge the equalized results of all blocks into a complete image, and use the bilinear interpolation method to smooth the merged area and reduce the boundary effect of the block; traverse each image; The equalization formula is: CDF(x)=∑p Where CDF(x) is the cumulative distribution function, and p is the probability of each gray level; In order to avoid noise amplification caused by over-enhancement, contrast limitation is introduced; that is, a threshold of the number of pixels at the gray level is set to limit the number of pixels at each gray level, and the excess pixels are allocated to other gray levels; after contrast limitation, the cumulative distribution function of each block needs to be recalculated, and the gray value mapping is performed according to the new CDF(x).

5. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 1, characterized in that: In step S3, multi-view stereo vision is used to calculate the camera's internal and external parameters and depth map, and multi-view fusion is used to perform secondary compensation for lunar imaging illumination. Specifically, the camera pose is calculated based on the motion recovery structure algorithm to obtain the camera's internal and external parameters and sparse point cloud images, and the PatchMatch algorithm is used to find the best depth estimation of each image through random initialization and iterative optimization to serve as the input information of the ViT model.

6. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 1, characterized in that: The step S4 involves multi-channel input of known camera pose information, generated depth map and image illumination information, using a deep neural network ViT based on a self-attention mechanism to extract neural features under similar rock and soil texture structure conditions of lunar test specimens, and completing effective reconstruction of neural coding volumes for neural radiation field generation representation; The specific steps of step S4 are: S41: Divide the depth map into n small blocks of size P×P, and flatten each small block into a vector to form an input sequence, denoted as patch i ; S42: Use a linear layer to convert each small block into an embedding vector and map it to a high-dimensional feature space. The formula is as follows: Patch i =W×patch i +b Among them, Patch i is the transformed embedding vector, W is the weight matrix, and b is the bias term; S43: Add position coding to each small block to retain the position information of each small block in the original image. Sine and cosine functions are usually used to generate position coding. For the k-th dimension position coding of the i-th small block, the formula is as follows: Among them, PE(pos,2k) and PE(pos,2k+1) correspond to odd-dimensional and even-dimensional position encodings respectively, and pos, k, and d are the encoding position index, dimension index, and embedding dimension of the model respectively; S44: Assume that the illumination information includes the features of multiple light sources, and convert them into feature vectors with the same number of small blocks through a linear layer, denoted as Light i ; Similarly, the camera pose information is converted into a feature vector through a linear layer, denoted as Camera i ; S45: Concatenate and fuse the small block features of the depth map, the illumination features, and the camera pose features to generate the final input features: Input i =Patch i +Light i +Camera i ; S46: The final feature vector is input into the ViT model, and the self-attention mechanism is used to calculate the relationship between blocks. The feature vector of each small block is processed by the following formula: Among them, Attention(Q,K,V) is the attention mechanism calculation method; Calculate the similarity between the query and the key, and get a scalar by taking the dot product of the query matrix Q and the key matrix K, and normalize it to prevent the calculated value from being too large or too small; the softmax function converts the dot product result into a probability distribution to ensure that the sum of all attention weights is 1. Its output determines the weight coefficient of each value (V). The more relevant the key, the greater the proportion of its corresponding value in the weighted average; Q, K, and V are the matrices of image block query, key, and value, respectively, corresponding to different representations of image blocks. The query represents the feature vector of the current block, the key represents the feature vector of all other blocks, and the value represents the feature information of the block, including the actual semantic content. K T is the transpose of the key matrix, d k Represents the dimension of the key; S47: After multiple Transformer encoder layer feature extraction, the final feature representation (x, d, f, z, l) of each small block is output for neural radiation field generation in downstream tasks, where x, d, f, z, l represent the three-dimensional position of the volume point, the camera pose, the linear interpolation neural feature at that location, and the multi-dimensional vector function corresponding to the depth and brightness respectively; For the subsequent neural radiation field generation, the color rendering loss is optimized, and the pixel color feature c = (r, g, b) of the original image is supplemented in the final feature, which is regarded as an additional input. That is, the final neural feature vector output is expressed as (x, d, f, c, z, l); The ViT model used uses a multi-head attention mechanism, which enables the model to learn information in parallel in different subspaces and speed up processing. After the self-attention layer, a feedforward neural network is used to process the features of each block, effectively fusing features at different levels and avoiding feature loss problems.

7. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 1, characterized in that: The step S5 is based on the multi-dimensional neural feature vector input generated in step S4, uses a multi-layer perceptron neural network to generate a neural radiation field, and outputs the color, volume density, illumination and depth features of each point in the reconstruction model. The process is expressed as follows: F θ (x,d,f,c,z,l)→(c ′ (r,g,b),σ,z ′ ,l ′ ) where F θ (x, d, f, c, z, l) represents the mapping function acting on the multi-dimensional neural feature vector input, c ′ (r, g, b) represents the color feature of the volume point x in the generated scene in the implicit expression, σ represents the volume density, which represents the accumulated radiation when the light passes through the volume point x, and z ′ ,l ′ Respectively represent the depth and lighting of the point.

8. The in-situ 3D reconstruction method of lunar soil specimen based on the improved neural radiation field algorithm according to claim 7, characterized in that: In step S5, based on the given volume density and color, volume rendering is performed by differentiable ray marching to achieve view synthesis. Specifically, the camera ray color C of any pixel in the tracking image is calculated to calculate the accumulated radiation at the sampled shadow point. The calculation formula is: Among them, r(x) represents the three-dimensional vector in space at the volume point x, σ(r(x)) is the estimated volume density value of the light at this point, c(r(x), d) corresponds to the estimated color value of the point, and the function T(x) represents the cumulative transmittance of the light from x0 to x, that is: By sampling and integrating each light ray, a composite image can be generated, which is the sum of the color contributions of all light rays in the scene: I = ∑αC, where α is the corresponding weight, usually the average weight.

9. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 1, characterized in that: The step S6 defines three types of loss functions for model optimization when training the improved neural radiation field, including a rendering loss L that measures the difference between the synthetic image and the real image. render , the depth loss L that compares the difference between the predicted depth and the true depth depth and the illumination loss L which evaluates the difference between the predicted illumination and the real illumination light , are supervised based on the mean square error, and the calculation formula is as follows: The comprehensive loss function is expressed as: L Total =λ1L render +λ2L depth +λ3L light Among them, λ i To control the factors, adjust the weight proportion of each loss function to obtain the best scene representation of the neural radiation field; The model optimization process uses a back-propagation algorithm to adjust the parameters of the neural network and update image features and network weights. To avoid overfitting, the comprehensive loss function is regularized, including L1 regularization, which adds the absolute value of the weight parameter to the loss function and forces some weights to become zero to achieve feature selection. L2 regularization adds the square sum of the weight parameter to the loss function to suppress the size of the weight and prevent the model from being too complicated.

10. The in-situ 3D reconstruction method of lunar soil specimen based on improved neural radiation field algorithm according to claim 1, characterized in that: The step S7 uses the marching cube algorithm to extract mesh features and realize the explicitness of the implicit expression of the neural radiation field, so as to quickly obtain a high-quality three-dimensional mesh model and retain the clear surface boundaries and texture details of the lunar soil specimen; The specific steps of step S7 are: S71 scene voxelization: The implicit scene generated by the neural radiation field is discretized into a uniform cubic grid in three-dimensional space. Each grid unit is called a voxel. The volume density corresponding to each corner point in the unit is obtained by mapping the predicted density of the neural radiation field, and is estimated by taking the average value of random sampling within the voxel. S72 Isosurface extraction: Set the isovalue threshold and compare it with the density of each corner point of the voxel, use the lookup table to determine the triangular composition of the isosurface in the cube, and then generate a triangular mesh to obtain an explicit expression of the object; S73 combined post-processing: Smooth the generated mesh to remove noise and irregularities, use mesh simplification algorithms to remove redundant polygons and improve rendering efficiency, and output the final 3D network model.

Images