Method for in-situ three-dimensional reconstruction of lunar soil test piece based on improved neural radiation field algorithm

Abstract

The application belongs to the field of computer vision, image processing and space engineering, and discloses a lunar soil test piece in-situ three-dimensional reconstruction method based on an improved neural radiation field algorithm, which comprises the following steps: acquiring a multi-view image set of the lunar soil test piece and performing lunar surface light self-adaptive adjustment; calculating the high-resolution camera pose through multi-view stereo vision to acquire a depth map; generating a neural encoding roll based on a Vision Transformer model to extract a multi-dimensional feature vector; generating a neural radiation field to predict the rendering features of volume points; performing a differentiable volume rendering and using three types of loss feedback optimization of color rendering, depth and light; using a moving cube algorithm to extract a grid to acquire a three-dimensional grid model; and effectively solving the problem of low reconstruction accuracy under uneven lunar surface light conditions and realizing high-quality and high-precision in-situ rapid three-dimensional reconstruction of the lunar soil test piece by solving the difficulties of feature extraction and reconstruction mismatch caused by the similar lunar soil background texture.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of computer vision, image processing and space engineering, and in particular to a lunar soil sample in-situ three-dimensional reconstruction method based on an improved neural radiation field algorithm. BACKGROUND

[0002] Today's lunar in-situ construction technology is listed as a key development direction by all space powers due to its high deep space exploration efficiency. The multiple related technical routes proposed by scholars all involve processing lunar soil into shaped bricks as building materials. However, the existing lunar soil bricks are mostly prepared based on simulated lunar soil and tested in a simulated lunar environment on the ground, and their real performance in lunar extreme scenarios is not clear, so it is urgent to carry out relevant physical tests in-situ to determine the important material and structural data for future base construction.

[0003] To carry out effective physical tests on the moon, it is crucial to quickly and accurately obtain the three-dimensional size, surface roughness and other basic parameters of the lunar soil sample using three-dimensional reconstruction technology. However, the lighting conditions on the moon are extremely uneven, which significantly affects the image quality, resulting in low precision and poor quality of three-dimensional reconstruction based on images. In addition, the texture structure of the lunar soil sample is highly similar to the lunar background, which can easily cause confusion and mismatching problems during feature extraction, introducing noise and error data and further increasing the reconstruction difficulty. In recent years, three-dimensional reconstruction technology based on neural radiation field (NeRF) has attracted much attention due to its superior performance in image generation and reconstruction, but its application to lunar environment still faces problems such as light sensitivity and long training time, which is not ideal for fast and efficient in-situ three-dimensional reconstruction.

[0004] Therefore, in view of the extreme lunar environment, it is urgently needed to propose a novel lunar soil sample in-situ fast three-dimensional reconstruction method to obtain high-quality image data under complex lighting conditions, effectively extract and match features under similar texture structure conditions, and perform fast and efficient computation under resource-limited conditions, thereby obtaining a high-precision and high-quality reconstruction model of the lunar soil sample and providing reliable parameters for subsequent physical experiments. SUMMARY

[0005] To overcome the problems of low reconstruction accuracy caused by uneven lighting changes on the moon and feature extraction difficulty and mismatching caused by similar background texture, and to achieve high-quality and high-precision in-situ fast three-dimensional reconstruction of lunar soil samples, the present application proposes a lunar soil sample in-situ fast three-dimensional reconstruction method based on an improved neural radiation field algorithm, which uses a lunar lighting self-adaptive adjustment strategy to optimize the quality of collected images, and uses a multi-view stereo vision and neural radiation field fusion (MVS-NeRF) based on a ViT model to ensure fast and efficient extraction of lunar soil sample features under lunar background, thereby achieving in-situ accurate three-dimensional reconstruction of lunar soil samples under lunar extreme conditions.

[0006] To achieve the above object, according to one aspect of the present application, a lunar soil sample in-situ three-dimensional reconstruction method based on improved neural radiance field algorithm is provided, comprising the following steps:

[0007] Step S1: Obtain a multi-angle image set of the lunar soil sample;

[0008] Step S2: Perform global and local light self-adaptive adjustment on the image set obtained in step S1;

[0009] Step S3: Calculate the camera pose and obtain the depth image using multi-view stereo vision on the image adjusted in step S2;

[0010] Step S4: Generate a neural encoding volume using a Vision Transformer model based on the depth image obtained in step S3, and extract a multi-dimensional feature vector;

[0011] Step S5: Encode the multi-dimensional feature vector obtained in step S4 into a multi-layer perception to generate a neural radiance field, and predict the rendering features of the volume points;

[0012] Step S6: Perform volume rendering on the volume points obtained in step S5 using differentiable ray marching, and use color rendering, light, and depth three types of loss to jointly optimize the neural radiance field network model to implicitly represent the three-dimensional scene of the lunar soil sample;

[0013] Step S7: Implement the explicit expression of the implicit representation (i.e. neural network) obtained in step S6 using the marching cubes algorithm, thereby obtaining the final three-dimensional mesh model.

[0014] As a preferred, the step S1 specifically comprises: using a high-resolution camera carried by a lunar rover / construction robot to perform all-around video recording around the lunar soil sample, and uniformly selecting no less than 50 clear color RGB images from the video as the original image set.

[0015] As a preferred, the step S2 is to cope with the inconsistency of image luminosity caused by the change of lunar surface light, and first performs global light self-adaptive adjustment, and then performs local light self-adaptive adjustment, and respectively uses a gamma correction method to adjust the brightness of the entire image set and an adaptive histogram equalization method to enhance the local details of each image.

[0016] As a preferred, the step S2 specifically comprises:

[0017] S21 Gamma correction: first calculate the brightness histogram of all images and combine to form a global brightness histogram representing the brightness distribution of the entire image set, and calculate the global brightness mean square error According to the shape of the brightness distribution, the initial global gamma value is dynamically selected; traverse the image set, and apply the same gamma correction formula to each image wherein C input , C output respectively represent input and corrected pixel values, and γ is a global gamma value;

[0018] Calculate the global brightness histogram and global brightness mean square deviation of the corrected image set Compare the two mean square deviations and adjust the gamma value, iterate until ;

[0019] The formula for calculating the global brightness mean square deviation is:

[0020] wherein I(x, y) is the brightness value of a certain pixel, μ is the global average brightness, and n is the number of all pixel points;

[0021] S22 adaptive histogram equalization: divide each image into multiple regional blocks, calculate the gray level histogram of each block; equalize the histogram of each block, that is, map the pixel value of each gray level to a new value to enhance the contrast of the block; merge the equalized results of all blocks into a complete image, and use a bilinear interpolation method to smooth the merged area to reduce the block boundary effect; traverse each image;

[0022] The equalization formula is: CDF(x) =∑p

[0023] wherein CDF(x) is the cumulative distribution function, and p is the probability of each gray level;

[0024] In order to avoid noise amplification caused by over-enhancement, a contrast limit is introduced; that is, a threshold of the number of pixels of each gray level is set to limit the number of pixels of each gray level, and the excess pixels are allocated to other gray levels; after the contrast limit, the cumulative distribution function of each block needs to be recalculated, and the gray value is mapped according to the new CDF(x).

[0025] As a preferred, in the step S3, the camera internal and external parameters and the depth map are calculated by multi-view stereo vision, and the lunar surface camera lighting is compensated twice by multi-view fusion; specifically, the camera pose is calculated based on the motion recovery structure algorithm, the camera internal and external parameters and the sparse point cloud image are obtained, and the PatchMatch algorithm is used to find the best depth estimation of each image through random initialization and iterative optimization, which is used as the input information of the ViT model.

[0026] As a preferred, the step S4 involves multi-channel input of known camera pose information, generated depth map and image lighting information, and uses a deep neural network ViT based on a self-attention mechanism to extract neural features under the condition of similar lunar surface test piece rock and soil texture structure, completes effective reconstruction of the neural encoding roll, and is used for neural radiation field generation representation;

[0027] The step S4 is specifically as follows:

[0028] S41: Divide the depth map into n small blocks with a size of P x P, and flatten each small block into a vector to form an input sequence, denoted as patch i ;

[0029] S42: Convert each small block into an embedding vector using a linear layer and map it to a high-dimensional feature space, according to the following formula:

[0030] Patch i = W x patch i + b

[0031] where Patch i is the converted embedding vector, W is the weight matrix, and b is the bias term;

[0032] S43: Add position encoding to each small block to preserve the position information of each small block in the original image. Typically, sine and cosine functions are used to generate position encoding. For the k-th dimension position encoding of the i-th small block, the formula is as follows:

[0033]

[0034] where PE(pos, 2k) and PE(pos, 2k+1) correspond to the odd and even dimension position encodings respectively, pos, k, d are the position index, dimension index and embedding dimension of the model respectively;

[0035] S44: Assume that the lighting information includes the characteristics of multiple light sources, and convert it into a feature vector with the same number of small blocks through a linear layer, denoted as Light i ; Similarly, convert the camera pose information into a feature vector through a linear layer, denoted as Camera i ;

[0036] S45: Concatenate and fuse the small block features, lighting features and camera pose features of the depth map to generate the final input feature: Input i = Patch i + Light i + Camera i ;

[0037] S46: Input the final feature vector into the ViT model and use the self-attention mechanism to calculate the relationship between blocks. The feature vector of each small block is processed according to the following formula:

[0038]

[0039] wherein Attention(Q, K, V) is an attention mechanism calculation method; The similarity between the query and the key is calculated, a scalar is obtained by the dot product of the query matrix Q and the key matrix K, and normalization is performed to prevent the calculated value from being too large or too small; the softmax function converts the dot product result into a probability distribution, ensuring that the sum of all attention weights is 1, and the output determines the weighting coefficient of each value (V). The more relevant the key, the greater the proportion of the corresponding value in the weighted average. Q, K, and V are matrices of image block queries, keys, and values, respectively, corresponding to different representations of image blocks. The query represents the feature vector of the current block, the key represents the feature vector of all other blocks, and the value represents the feature information of the block, including the actual semantic content. T k wherein d k represents the dimension of the key;

[0040] S47: After feature extraction by multiple Transformer encoder layers, the final feature representation (x, d, f, z, l) of each small block is outputted for neural radiance field generation in downstream tasks, wherein x, d, f, z, and l represent the three-dimensional position of the volume point, the camera pose, the linear interpolation neural feature at this point, the multi-dimensional vector function corresponding to the depth and brightness, respectively.

[0041] Color rendering loss optimization is performed for subsequent neural radiance field generation, and the pixel color feature c = (r, g, b) of the original picture is supplemented in the final feature, which is regarded as an additional input. The final neural feature vector output is represented as (x, d, f, c, z, l).

[0042] The ViT model used uses a multi-head attention mechanism, which enables the model to learn information in different subspaces in parallel, speeding up processing. After the self-attention layer, a feedforward neural network is used to process the features of each block, effectively fusing features at different levels and avoiding feature loss problems.

[0043] As a preferred embodiment, the step S5 is based on the multi-dimensional neural feature vector input generated in step S4, and a multi-layer perceptron neural network is used to generate a neural radiance field, outputting the color, volume density, illumination, and depth features of each point of the reconstruction model, represented as:

[0044] F θ (x, d, f, c, z, l)→(c'(r, g, b), σ, z', l')

[0045] wherein F θ(x, d, f, c, z, l) represents a mapping function acting on the multi-dimensional neural feature vector input, c'(r, g, b) represents the color feature of the volume point x in the generated scene in the implicit representation, σ represents the volume density, represents the cumulative radiation when the light passes through the volume point x, and z', l' represent the depth and illumination of the point, respectively.

[0046] As preferred, the step S5 completes the volume rendering by differentiable ray marching based on the given volume density and color, and further realizes the view synthesis. Specifically, the camera ray color C of any pixel in the image is tracked to calculate the cumulative radiation at the sampled shadow point, and the calculation formula is as follows:

[0047]

[0048] Wherein, r(x) represents a spatial three-dimensional vector at the volume point x, σ(r(x)) is the estimated volume density value of the light at the point, c(r(x), d) is the estimated color value of the point, and the function T(x) represents the cumulative transmittance of the light from x0 to x, that is:

[0049]

[0050] By sampling and integrating each light, the synthesized image, which is the sum of the color contributions of all lights in the scene, can be generated: I = ∑αC, where α is the corresponding weight, usually taking the average weight.

[0051] As preferred, the step S6 defines three types of loss functions for model optimization when training the improved neural radiance field, including the rendering loss L render measuring the difference between the synthesized image and the real image, depth the depth loss L light comparing the difference between the predicted depth and the real depth, and the illumination loss L Total evaluating the difference between the predicted illumination and the real illumination, all of which are supervised based on the mean square error, and the calculation formula is as follows:

[0052]

[0053] The comprehensive loss function is represented as:

[0054] L Total = λ1L render + λ2L depth + λ3L light

[0055] Wherein, λ i is a control factor for adjusting the weight proportion of each loss function to obtain the best scene representation of the neural radiance field;

[0056] The model optimization process uses a back propagation algorithm to adjust the parameters of the neural network, update the image features and network weights; in order to avoid overfitting, the comprehensive loss function is regularized, including L1 regularization, adding the absolute value of the weight parameter in the loss function and prompting part of the weight to be zero, realizing feature selection;L2 regularization adds the square sum of the weight parameters in the loss function to suppress the size of the weight, preventing the model from being too complex.

[0057] As preferred, the step S7 uses a moving cube algorithm to extract the mesh features, realizes explicitization of the implicit expression of the neural radiation field, thereby quickly obtaining a high-quality three-dimensional mesh model, and retains the clear surface boundary and texture details of the lunar soil sample.

[0058] The specific steps of the step S7 are:

[0059] S71 scene voxelization: the implicit scene generated by the neural radiation field is discretely divided into uniform cubic meshes in three-dimensional space, each mesh unit is called a voxel, and the volume density corresponding to each corner point in the unit is obtained by mapping the neural radiation field prediction density, and the average value is estimated by random sampling in the voxel.

[0060] S72 isosurface extraction: set the isosurface threshold and compare it with the density of each corner point of the voxel, use the lookup table to determine the triangular composition of the isosurface in the cube, thereby generating a triangular mesh and obtaining an explicit expression of the object.

[0061] S73 combination post-processing: smoothing the generated mesh to remove noise and irregularities, using a mesh simplification algorithm to remove redundant polygons and improve rendering efficiency, and outputting the final three-dimensional network model.

[0062] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0063] 1、The present application proposes a lunar surface illumination adaptive adjustment strategy, which effectively solves the problem of low precision of lunar soil sample three-dimensional reconstruction caused by poor image quality and inconsistent luminosity under uneven lunar surface illumination conditions. In the application, gamma correction is used to balance the luminosity conditions of the entire image set, ensuring good overall reconstruction effect;Adaptive histogram equalization is used to adjust the local luminosity of a single image, enhance the reconstruction effect of key areas, and retain local details of three-dimensional reconstruction. In addition, subsequent multi-view stereo vision depth calculation involves multi-view fusion, which can further compensate for the image illumination.

[0064] 2、The application combines the ViT model based on the self-attention mechanism to extract multi-dimensional feature vectors, and combines the MVS-NeRF reconstruction method, which effectively solves the problems of feature extraction difficulty and reconstruction mismatch caused by the similar texture structure of the lunar soil sample and the lunar surface background. ViT uses multi-head self-attention mechanism at different levels, allows the model to understand the subtle differences in the image from a global perspective, extracts multi-scale features at different levels, greatly enhances the ability to distinguish similar structures, and can accurately extract the structural features of the lunar soil sample under the complex texture background of the lunar surface, reducing noise and error data.

[0065] 3、The application provides a method for in-situ rapid three-dimensional reconstruction of lunar soil samples under extreme conditions on the lunar surface. The existing three-dimensional reconstruction method performs poorly in the extreme environment on the lunar surface, is sensitive to light, or takes a long time to reconstruct, and cannot quickly and efficiently complete the reconstruction to promote subsequent tasks. And when using a lunar rover or a construction robot to carry out related tasks, it has limited computing and energy resources, and insufficient endurance. Using a fast in-situ reconstruction method will greatly reduce the computing and energy requirements and improve the energy efficiency of the task, which is conducive to the continuous and efficient development of subsequent lunar surface tasks. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a flowchart of the application for in-situ rapid three-dimensional reconstruction of lunar soil samples under lunar surface conditions.

[0067] Figure 2 is a schematic diagram of obtaining a lunar sampling photo in the embodiment of the application.

[0068] Figure 3 is a lunar surface adaptive lighting adjustment strategy diagram in the embodiment of the application.

[0069] Figure 4 is a flowchart of calculating a depth map based on multi-view stereo vision in the embodiment of the application.

[0070] Figure 5 is a network structure diagram of combining a VIT model for neural radiation field reconstruction in the embodiment of the application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0072] Reference Figure 1The embodiment of the present application provides a lunar soil sample in-situ rapid three-dimensional reconstruction method based on an improved neural radiation field algorithm, comprising the following steps:

[0073] Step S1: obtaining a multi-angle image set of the lunar soil sample;

[0074] Step S2: performing lunar surface global and local light self-adaptive adjustment on the image set obtained in step S1;

[0075] Step S3: using multi-view stereo vision to calculate the camera pose and obtain a depth image for the image adjusted in step S2;

[0076] Step S4: using a Vision Transformer model to generate a neural encoding roll based on the depth image obtained in step S3, and extracting a multi-dimensional feature vector;

[0077] Step S5: inputting the multi-dimensional feature vector obtained in step S4 into a multi-layer perception machine to generate a neural radiation field, and predicting the rendering features of the volume point, wherein the rendering features include color value, volume density, luminosity and depth value, etc.;

[0078] Step S6: using the volume point obtained in step S5 to perform volume rendering by using a differentiable ray marching, using three types of losses of color rendering, light and depth to jointly optimize the neural radiation field network model, and performing implicit representation on the lunar soil sample three-dimensional scene;

[0079] Step S7: using a marching cubes algorithm to realize the explicit of the implicit representation (i.e. neural network) obtained in step S6, so as to obtain a final three-dimensional mesh model.

[0080] In the step S1, a high-resolution camera carried by a lunar rover / building robot is used to perform all-around video recording around the lunar soil sample (video recording is performed in a 360-degree manner around the lunar soil sample), Figure 2 and at least 50 clear color RGB images are uniformly selected from the video as an original image set.

[0081] Referring to Figure 3 , the step S2 is a lunar surface light self-adaptive adjustment strategy for dealing with the inconsistency of image luminosity caused by the change of lunar surface light, which comprises global light self-adaptive adjustment and local light self-adaptive adjustment, and the gamma correction method is used to adjust the brightness of the whole image set, and the CLAHE method is used to enhance the local details of each picture.

[0082] The step S2 comprises the following steps:

[0083] S21 gamma correction: first, the brightness histogram of all images is calculated and combined to form a global brightness histogram, which represents the brightness distribution of the whole image set, and the global brightness mean square error is calculated According to the shape of the luminance distribution, dynamically select an initial global gamma value; traverse the image set, and apply the same gamma correction formula to each image where C input , C output respectively represent the input and corrected pixel values, and γ is a global gamma value;

[0084] Calculate the global brightness histogram and global brightness mean square error of the corrected image set Compare the two mean square errors and adjust the gamma value, and iterate until ;

[0085] The global brightness mean square error calculation formula is:

[0086] where I(x, y) is the luminance value of a certain pixel, μ is the global average luminance, and n is the number of all pixel points;

[0087] S22 adaptive histogram equalization: divide each image into multiple region blocks, calculate the gray level histogram of each block, and perform equalization processing on the histogram of each block, that is, map the pixel value of each gray level to a new value to enhance the contrast of the block. The results of equalizing all blocks are combined into a complete image, and a bilinear interpolation method is used to smooth the merged regions to reduce the block boundary effect; and each image is traversed;

[0088] The equalization formula is: CDF(x) = ∑p

[0089] where CDF(x) is the cumulative distribution function, and p is the probability of each gray level;

[0090] In the adaptive histogram equalization method in step S2, in order to avoid noise amplification caused by excessive enhancement, a contrast limit is introduced, that is, a threshold of the number of pixels of each gray level is set to limit the number of pixels of each gray level, and the excess pixels are distributed to other gray levels. After the contrast limit, the cumulative distribution function of each block needs to be recalculated, and the mapping of the gray value is performed according to the new CDF(x).

[0091] Referring to Figure 4 , in step S3, the camera internal and external parameters and the depth map are calculated by using multi-view stereo vision, and the lunar surface image light is compensated twice by multi-view fusion. Specifically, the camera pose (rotation matrix and translation vector) is calculated based on the structure from motion (SFM) algorithm, the camera internal and external parameters and the sparse point cloud image are obtained, and the PatchMatch algorithm is used to find the best depth estimation of each image through random initialization and iterative optimization, which is used as the input information of the ViT model.

[0092] Step S4 involves the multi-channel input of known camera pose information, generated depth map and image lighting information, neural feature extraction using a deep neural network based on a self-attention mechanism, Vision transformer (ViT), to complete the effective reconstruction of the neural encoding roll

[0093] Referring to Figure 5 , the step S4 includes the following specific steps:

[0094] S41: Divide the depth map into n small blocks, set the size to P x P, flatten each small block into a vector, form an input sequence, and denote it as patch i ;

[0095] S42: Convert each small block into an embedding vector using a linear layer, and map it to a high-dimensional feature space, as follows:

[0096] Patch i = W x patch i + b

[0097] where Patch i is the converted embedding vector, W is the weight matrix, and b is the bias term;

[0098] S43: Add position encoding to each small block to retain the position information of each small block in the original image. Usually, sine and cosine functions are used to generate position encoding. For the k-th dimensional position encoding of the i-th small block, the formula is as follows:

[0099]

[0100] where E(pos, 2k) and PE(pos, 2k+1) correspond to the odd-dimensional and even-dimensional position encodings, respectively, pos, k, and d are the encoding position index, dimension index, and embedding dimension of the model, respectively;

[0101] S44: Assuming that the lighting information includes the characteristics of multiple light sources, it can be converted into a feature vector with the same number of small blocks through a linear layer, denoted as Light i ; Similarly, the camera pose information (such as position and orientation) is converted into a feature vector through a linear layer, denoted as Camera i ;

[0102] S45: Concatenate and fuse the small block features of the depth map, lighting features, and camera pose features to generate the final input feature: Input i = Patch i + Light i + Camera i ;

[0103] S46: input the final feature vector into the ViT model, calculate the relationship between blocks using self-attention mechanism, and process the feature vector of each small block by the following formula:

[0104]

[0105] wherein Attention(Q, K, V) is the attention mechanism calculation method; The similarity between the query and the key is calculated, and a scalar is obtained by the dot product of the query matrix Q and the key matrix K, and normalization is performed to prevent the calculation value from being too large or too small; the softmax function converts the dot product result into a probability distribution, ensuring that the sum of all attention weights is 1, and its output determines the weighting coefficient of each value (V), the more relevant the key, the greater the proportion of the corresponding value in the weighted average. Q, K, V are the matrices of image block query, key and value, respectively, corresponding to different representations of image blocks, the query represents the feature vector of the current block, the key represents the feature vectors of all other blocks, and the value represents the feature information of the block, including the actual semantic content, K T k represents the transpose of the key matrix, d k represents the dimension of the key;

[0106] S47: After feature extraction by multiple Transformer encoder layers, output the final feature representation (x, d, f, z, l) of each small block for downstream tasks (neural radiance field generation), wherein x, d, f, z, l represent the three-dimensional position of the volume point, the camera pose, the linear interpolation neural feature at that point, the multi-dimensional vector function corresponding to the depth and brightness, respectively;

[0107] For color rendering loss optimization for subsequent neural radiance field generation, the pixel color feature c=(r, g, b) of the original picture is supplemented in the final feature, which is regarded as an additional input, and the final neural feature vector output is represented as (x, d, f, c, z, l).

[0108] The ViT model used in step S4 uses a multi-head attention mechanism, which enables the model to learn information in different subspaces in parallel, speeding up the processing; after the self-attention layer, a feedforward neural network is used to process the features of each block, effectively fusing features at different levels and avoiding feature loss problems.

[0109] Step S5 is to input the multi-dimensional neural feature vector generated based on step S4, use a multi-layer perceptron (MPL) neural network to generate a neural radiance field, and output the color, volume density, illumination and depth features of each point of the reconstruction model, represented as:

[0110] F θ (x, d, f, c, z, l)→(c'(r, g, b), σ, z', l')

[0111] Among them, F θ (x, d, f, c, z, l) represents the mapping function acting on the multidimensional neural feature vector input, c′(r, g, b) represents the color feature of the volume point x in the generated scene in the implicit expression, σ represents the volume density, which represents the accumulated radiation when the light passes through the volume point x, and z′, l′ represent the depth and illumination of the point respectively.

[0112] In step S5, based on the given volume density and color, volume rendering is performed by differentiable ray marching to achieve view synthesis. Specifically, the camera ray color C of any pixel in the image is tracked to calculate the accumulated radiance at the sampled shadow point. The calculation formula is:

[0113]

[0114] Among them, r(x) represents the three-dimensional vector in space at the volume point x, σ(r(x)) is the estimated volume density value of the light at that point, c(r(x),d) corresponds to the estimated color value of the point, and the function T(x) represents the cumulative transmittance of the light from x0 to x, that is:

[0115]

[0116] By sampling and integrating each ray, a composite image (I) is generated, which is the sum of the color contributions of all rays in the scene: I = ∑αC, where α is the corresponding weight, usually the average weight.

[0117] Step S6 defines three types of loss functions for model optimization when training the improved neural radiation field, including the rendering loss L that measures the difference between the synthetic image and the real image. render , the depth loss L that compares the difference between the predicted depth and the true depth depth and the illumination loss L which evaluates the difference between the predicted illumination and the real illumination light , are supervised based on the mean square error, and the calculation formula is as follows:

[0118]

[0119] The comprehensive loss function can be expressed as: L Total =λ1L render +λ2L depth +λ3L light , where λ i To control the factors, the weight ratio of each loss function is adjusted to obtain the optimal scene representation of the neural radiation field.

[0120] The model optimization process uses a back-propagation algorithm to adjust the parameters of the neural network and update image features and network weights.

[0121] To avoid overfitting, the integrated loss function can also be regularized, including L1 regularization adding the absolute value of the weight parameter in the loss function and prompting part of the weight to become zero to achieve feature selection; L2 regularization adding the square sum of the weight parameter in the loss function to suppress the size of the weight to prevent the model from being too complex.

[0122] Step S7 extracts the grid features using the moving cube algorithm, realizes the explicit expression of the implicit expression of the neural radiance field, and thus quickly obtains a high-quality three-dimensional grid model, retaining the clear surface boundary and texture details of the lunar soil sample;

[0123] The specific steps of step S7 are:

[0124] S71 scene voxelization: discretely divide the implicit scene generated by the neural radiance field in the three-dimensional space into a uniform cube grid, each grid unit is called a voxel, and the volume density corresponding to each corner point in the unit is obtained by mapping the neural radiance field prediction density, and the average value is estimated by random sampling in the voxel;

[0125] S72 isosurface extraction: set an isosurface threshold and compare it with the density of each corner point of the voxel, use a lookup table to determine the triangular composition of the isosurface in the cube, thereby generating a triangular mesh and obtaining an explicit expression of the object;

[0126] S73 post-processing: smoothing the generated mesh to remove noise and irregularities, using a mesh simplification algorithm to remove redundant polygons and improve rendering efficiency, and outputting the final three-dimensional network model.

[0127] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An in-situ 3D reconstruction method for lunar soil specimens based on an improved neural radiation field algorithm, characterized by: The following steps are involved: Step S1: Acquire a multi-angle image set of the lunar soil sample; Step S2: Adaptively adjust the global and local illumination of the lunar surface for the image set obtained in step S1; Step S3: Using multi-view stereo vision to calculate the camera pose and obtain a depth image for the image adjusted in step S2; Step S4: Generate a neural coding volume using the Vision Transformer model based on the depth image obtained in step S3 and extract a multidimensional feature vector; Step S4 involves multi-channel input of known camera pose information, generated depth map and image illumination information, using a deep neural network ViT based on a self-attention mechanism to extract neural features under similar rock and soil texture structure conditions of lunar test specimens, and completing effective reconstruction of the neural coding volume for neural radiation field generation representation; Step S5: The multi-dimensional feature vector obtained in step S4 is encoded and input into a multi-layer perceptron to generate a neural radiation field and predict the rendering features of the volume point; Rendering characteristics include color values, volume density, illumination, and depth values; Step S6: Volume rendering is performed on the volume points obtained in step S5 using differentiable ray marching. The neural radiation field network model is optimized using three types of losses: color rendering, illumination, and depth, to implicitly represent the three-dimensional scene of the lunar soil specimen. Step S7: The implicit representation obtained in step S6 is made explicit using the marching cubes algorithm, thereby obtaining the final three-dimensional mesh model.

2. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: The specific steps of step S1 are: using a high-resolution camera carried by a lunar rover / construction robot to record a full-scale video around the lunar soil specimen, and evenly selecting no less than 50 clear color RGB images from the video as the original image set.

3. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: Step S2 is to address the image brightness inconsistency caused by changes in lunar illumination. Global illumination adaptive adjustment is first performed, and then local illumination adaptive adjustment is performed. The gamma correction method is used to adjust the brightness of the entire image set, and the adaptive histogram equalization method is used to enhance the local details of each image.

4. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 2, characterized in that: The specific steps of step S2 are: S21 gamma correction: First, the brightness histograms of all images are calculated and merged to form a global brightness histogram to characterize the brightness distribution of the entire image set, and the global brightness mean square error is calculated. ; Dynamically select an initial global gamma value based on the shape of the brightness distribution; iterate over the image set and apply the same gamma correction formula to each image ,in , denote the input and corrected pixel values ​​respectively, γ is the global gamma value; Calculate the global brightness histogram and global brightness mean square error of the rectified image set ; Compare the two mean square errors and adjust the gamma value, iterating until until; The formula for calculating the global brightness mean square error is: in, is the brightness value of a pixel, is the global average brightness, n is the number of all pixels; S22 adaptive histogram equalization: Divide each image into multiple blocks and calculate the grayscale histogram of each block; equalize the histogram of each block, that is, map the pixel value of each grayscale level to a new value to enhance the contrast of the block; merge the equalized results of all blocks into a complete image, and use bilinear interpolation to smooth the merged area and reduce the block boundary effect; traverse each image; The equalization formula is: in, is the cumulative distribution function, p is the probability of each gray level; In order to avoid noise amplification caused by over-enhancement, contrast limitation is introduced; that is, the pixel number threshold of the gray level is set to limit the number of pixels at each gray level, and the excess pixels are allocated to other gray levels; after contrast limitation, the cumulative distribution function of each block needs to be recalculated and the new value is calculated. Mapping of grayscale values.

5. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: In step S3, multi-view stereo vision is used to calculate the camera's internal and external parameters and depth map, and multi-view fusion is used to perform secondary compensation for lunar surface imaging illumination. Specifically, the camera pose is calculated based on the motion recovery structure algorithm to obtain the camera's internal and external parameters and sparse point cloud images, and the PatchMatch algorithm is used to find the best depth estimation for each image through random initialization and iterative optimization to serve as the input information of the ViT model.

6. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: The specific steps of step S4 are: S41: Divide the depth map into n small blocks of size P×P, and flatten each small block into a vector to form an input sequence, denoted as ; S42: Use a linear layer to convert each small block into an embedding vector and map it to a high-dimensional feature space. The formula is as follows: in, is the transformed embedding vector, W is the weight matrix, and b is the bias term; S43: Add position coding to each small block to retain the position information of each small block in the original image. Use sine and cosine functions to generate position coding. For the k-th dimension position coding of the i-th small block, the formula is as follows: in, and They correspond to odd-dimensional and even-dimensional position encodings respectively, pos, k, and d are the encoding position index, dimension index, and embedding dimension of the model respectively; S44: Assuming that the illumination information includes features of multiple light sources, it is converted into a feature vector with the same number of small blocks through a linear layer, denoted as ; Similarly, the camera pose information is converted into a feature vector through a linear layer, recorded as ; S45: Concatenate and fuse the small block features of the depth map, the illumination features, and the camera pose features to generate the final input features: ; S46: The final feature vector is input into the ViT model, and the relationship between blocks is calculated using the self-attention mechanism. The feature vector of each small block is processed using the following formula: in, Calculation method for attention mechanism; Calculate the similarity between the query and the key, obtain a scalar by taking the dot product of the query matrix Q and the key matrix K, and perform normalization to prevent the calculated value from being too large or too small; The function converts the dot product result into a probability distribution, ensuring that the sum of all attention weights is 1. Its output determines the weight coefficient of each value (V). The more relevant the key, the greater the proportion of its corresponding value in the weighted average. Q, K, and V are the matrices of image block query, key, and value, respectively, corresponding to different representations of image blocks. The query represents the feature vector of the current block, the key represents the feature vector of all other blocks, and the value represents the feature information of the block, including the actual semantic content. is the transpose of the key matrix, d k Represents the dimension of the key; S47: After multiple Transformer encoder layer feature extraction, the final feature representation of each small block is output , used for neural radiation field generation in downstream tasks, where Respectively represent the three-dimensional position of the volume point, the camera pose, the linear interpolation neural features at that location, and the multidimensional vector function corresponding to the depth and brightness; Optimize color rendering loss for subsequent neural radiation field generation, and supplement the pixel color features of the original image in the final feature , regarded as an additional input, the final neural feature vector output is expressed as ; The ViT model used uses a multi-head attention mechanism to enable the model to learn information in parallel in different subspaces, speeding up processing; after the self-attention layer, a feedforward neural network is used to process the features of each block, effectively fusing features at different levels and avoiding feature loss problems.

7. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: The step S5 is based on the multi-dimensional neural feature vector input generated in step S4, and uses a multi-layer perceptron neural network to generate a neural radiation field, and outputs the color, volume density, illumination and depth features of each point in the reconstructed model. The process is expressed as follows: in, represents the mapping function acting on the multi-dimensional neural feature vector input, Indicates the color feature of the volume point x in the implicit expression generated scene, σ represents the volume density, which represents the accumulated radiation when the light passes through the volume point x. Represent the depth and illumination of the point respectively.

8. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 7, characterized in that: In step S5, based on the given volume density and color, volume rendering is performed by differentiable ray marching to achieve view synthesis. Specifically, the camera ray color C of any pixel in the image is tracked to calculate the accumulated radiance at the sampled shadow point. The calculation formula is: in, represents the three-dimensional vector in space at the volume point x, is the estimated volume density of the light at that point, Corresponding to the estimated color value of this point, the function T(x) represents the cumulative transmittance of light from x0 to x, that is: By sampling and integrating each ray, a composite image is generated, which is the sum of the color contributions of all rays in the scene: ,in is the corresponding weight.

9. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: The step S6 defines three types of loss functions for model optimization when training the improved neural radiation field, including a rendering loss that measures the difference between the synthetic image and the real image. , depth loss that compares the difference between predicted depth and true depth and the illumination loss that evaluates the difference between the predicted illumination and the real illumination , are supervised based on the mean square error, and the calculation formula is as follows: The comprehensive loss function is expressed as: in, To control the factors, adjust the weight ratio of each loss function to obtain the best scene representation of the neural radiation field; The model optimization process uses a backpropagation algorithm to adjust the parameters of the neural network and update image features and network weights. To avoid overfitting, the comprehensive loss function is regularized, including L1 regularization, which adds the absolute value of the weight parameter to the loss function and forces some weights to zero to achieve feature selection. L2 regularization adds the sum of the squares of the weight parameters to the loss function to suppress the size of the weight and prevent the model from being too complex.

10. The in-situ 3D reconstruction method for lunar soil specimens based on the improved neural radiation field algorithm according to claim 1, characterized in that: Step S7 uses the marching cubes algorithm to extract mesh features and make the implicit expression of the neural radiation field explicit, thereby quickly obtaining a high-quality three-dimensional mesh model that retains clear surface boundaries and texture details of the lunar soil specimen; The specific steps of step S7 are: S71 scene voxelization: The implicit scene generated by the neural radiation field is discretized into a uniform cubic grid in three-dimensional space. Each grid unit is called a voxel. The volume density corresponding to each corner point in the unit is obtained by mapping the predicted density of the neural radiation field and estimating it by taking the average value of random sampling within the voxel. S72 Isosurface Extraction: Set an isovalue threshold and compare it with the density of each corner point of the voxel. Use a lookup table to determine the triangular composition of the isosurface in the cube, thereby generating a triangular mesh and obtaining an explicit representation of the object. S73 combined post-processing: Smooth the generated mesh to remove noise and irregularities, use mesh simplification algorithms to remove redundant polygons and improve rendering efficiency, and output the final 3D network model.

Images