A multi-imaging detection distance fusion small asteroid variable scale three-dimensional reconstruction method

Abstract

The present application relates to a kind of multi-imaging detection distance fusion small asteroid variable scale three-dimensional reconstruction method, comprising the following steps: after the input variable scale image is preprocessed, camera internal and external parameters and sparse point cloud are estimated, 3D Gauss is generated from sparse point cloud;For input image, image segmentation is carried out to target area, and the loss weight of the foreground and background of image under different scales is calculated respectively;For 3D Gauss, the sampling frequency of each 3D Gauss is calculated, variable scale anti-aliasing 3D Gauss filter is introduced, and 3D Gauss is optimized;For training result, using statistical filtering, 3D Gauss significantly outlying is eliminated;For filtering result, introduce surface grid optimization strategy to adjust the attribute of 3D Gauss, and the surface three-dimensional model of asteroid is reconstructed.The present application can realize the three-dimensional reconstruction of small asteroid surface under multi-imaging detection distance, fuse multi-scale image information, alleviate the problem of insufficient imaging under close distance, and improve the precision of three-dimensional reconstruction of small asteroid surface.

Description

Technical Field

[0001] This invention relates to the field of visual 3D reconstruction, and more specifically to a method for variable-scale 3D reconstruction of asteroids using multi-imaging detection range fusion. Background Technology

[0002] Visual 3D reconstruction refers to reconstructing the three-dimensional shape and structure of an object or scene from a two-dimensional image. In asteroid exploration missions, the probe needs to pass through an approach phase and a flyby phase to achieve landing and sampling. As the probe approaches, the scale of the asteroid increases. Asteroid variable-scale 3D reconstruction technology utilizes variable-scale image data of asteroids for 3D reconstruction, which is of great scientific significance for obtaining the shape and structure of asteroids and guiding the probe to land.

[0003] Traditional visual 3D reconstruction methods for asteroids primarily rely on feature point extraction and matching. However, limitations in scale and the number of images can lead to errors in the reconstruction results. Traditional methods typically utilize single-scale image data from a single imaging range for 3D reconstruction, resulting in low data utilization efficiency and wasted exploration data. Considering the enormous costs of deep space exploration missions, asteroid 3D reconstruction technology urgently needs to make fuller use of remote sensing data to improve mapping accuracy.

[0004] There are many types of visual 3D reconstruction techniques, such as multi-view stereo reconstruction (MVS), convolutional neural networks (CNN), generative adversarial networks (GAN), and neural radiation fields (NeRF). These methods can calculate the depth information of each pixel through disparity matching or deep learning, thereby reconstructing the 3D structure of a scene. Despite significant progress in 3D reconstruction technology, challenges remain in some applications, such as inaccuracies with variable-scale data or highly similar images. Therefore, the improvement and optimization of 3D reconstruction techniques remain of great importance.

[0005] The shortcomings of existing technologies for the study of asteroid scale-variable 3D reconstruction under multiple imaging detection distance conditions are mainly reflected in the following aspects: (1) It relies on feature point extraction and matching, and the matching effect is poor when the target scale changes greatly; (2) The 3D Gaussian method has high-frequency artifacts and dilation effect in the synthesis of new views of variable scale images, resulting in low accuracy of the model 3D reconstruction results; (3) The 3D Gaussian method performs poorly in the reconstruction of the object surface, and the point cloud noise is obvious. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a variable-scale three-dimensional reconstruction method for asteroids by fusing multiple imaging detection ranges. This method uses variable-scale image data captured by the probe during the approach and flyby phases to perform three-dimensional reconstruction, so that high reconstruction accuracy can be maintained even when there is less close-range imaging data.

[0007] The technical solution of this invention is a method for variable-scale three-dimensional reconstruction of asteroids by fusing multiple imaging detection ranges, comprising the following steps: (1) After preprocessing the input variable-scale image, the SFM method is used to estimate the camera intrinsic and extrinsic parameters and sparse point cloud, and an initial 3D Gaussian is generated from the sparse point cloud. (2) For the input image in step (1), perform image segmentation on the target region, statistically analyze the pixel ratio inside and outside the mask after morphological dilation, calculate the loss weights of the foreground and background of the image at different scales, and thus construct a multi-scale loss function. (3) For the 3D Gaussian established in step (1), calculate the sampling frequency of each 3D Gaussian, introduce a variable-scale anti-aliasing 3D Gaussian filter, and combine it with the multi-scale loss function calculated in step (2) to optimize the 3D Gaussian during training. (4) For the 3D Gaussian trained and optimized in step (3), perform statistical filtering on the 3D Gaussian to remove the 3D Gaussian with significant outliers; (5) Based on the filtering results of step (4), a surface mesh optimization strategy is introduced to further adjust the properties of 3D Gaussian and obtain the three-dimensional surface model of the asteroid through Poisson reconstruction.

[0008] The advantages of this invention compared to the prior art are: (1) The present invention adopts the variable-scale anti-aliasing 3D Gaussian method and introduces segmentation mask for variable-scale region enhancement learning, realizing the variable-scale three-dimensional reconstruction of asteroids by multi-imaging detection distance fusion. Under the condition of limited close-range single-scale imaging data, the reconstruction accuracy of the three-dimensional model of the asteroid surface is improved.

[0009] (2) The present invention optimizes the reconstructed 3D Gaussian by using statistical filtering to remove outliers of the 3D Gaussian and introduces a surface mesh optimization strategy to improve the continuity and integrity of the surface mesh.

[0010] In summary, the method employed in this invention is simple in principle and can achieve the goal of variable-scale three-dimensional reconstruction of asteroids at multiple imaging detection distances. Attached Figure Description

[0011] Figure 1 This is a flowchart of a variable-scale 3D reconstruction method for asteroids using multi-imaging detection range fusion, as described in this invention. Detailed Implementation

[0012] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0013] like Figure 1 As shown, the specific implementation steps of the multi-imaging detection range fusion method for asteroid variable-scale 3D reconstruction according to the present invention are as follows: Step 1: First, use the SFM method to estimate the camera pose and obtain the camera's intrinsic and extrinsic parameters. Simultaneously, obtain a sparse point cloud model of the scene, using this sparse point cloud as the initial point cloud for 3D Gaussian optimization. 3D Gaussian is an explicit scene representation that uses a large number of 3D anisotropic spheres to represent the scene, each sphere having a mean. Covariance Opacity spherical harmonic coefficient ,in, These are degrees of freedom, used to model colors related to the view. The specific formula for 3D Gaussian is as follows: ; mean Represents the positional information and covariance of a 3D Gaussian. Represents the anisotropy of a 3D Gaussian, and its opacity. Used for volume rendering, spherical harmonic coefficients This is to fit more accurate color information. Specifically, for the first... A 3D Gaussian, represented as: ; in, Indicates the first The position of a 3D Gaussian. Indicates the first The covariance matrix of a 3D Gaussian is used for regularization optimization. Further decomposed into rotation matrices and scaling matrix : ; For this scene representation, view rendering is performed via Gaussian sputtering. First, all 3D Gaussians in the scene are projected onto a 2D image plane, their colors calculated using spherical harmonics. Then, for each of the final image... Pixel blocks are sorted by depth using projected 3D Gaussian shapes that intersect with the pixel blocks. Finally, for each pixel in a pixel block, its color is... It is calculated by synthesizing the opacity and color of all 3D Gaussians covering that pixel: ; in, This indicates the number of 3D Gaussians covering that pixel. This represents the opacity of the 3D Gaussian multiplied by the density of the 2D Gaussian projected at the pixel position. The calculated color is represented by the 3D Gaussian method, which efficiently and accurately represents geometric and appearance attributes by representing objects and scenes as sets of 3D Gaussians.

[0014] Step 2: To improve the robustness and rendering quality of the model under multi-scale conditions, image segmentation techniques are used to output segmentation masks for asteroid images at different scales, which are then used to supervise the effective pixel regions of the asteroid at different scales. First, the original image is segmented using a large visual model (SAM) to output a segmentation mask for the target region. Then, a morphological dilation operator is used to dilate the mask. For loss calculations of foreground and background pixels at different scales, a multi-scale loss function is designed, the proportion of pixels inside and outside the mask is statistically analyzed, and the loss weights inside and outside the target region are calculated. ; ; in, It is the count of deep space background pixels. and These are the width and height of the image, used for the area within the mask. Weighted loss calculation, used for areas outside the mask. Weighted loss is calculated to differentiate the loss in different regions for backpropagation calculations. 3D Gaussian uses a weighted average loss calculation method when calculating the loss. loss: ; in, and These are the pixel grayscale values ​​of the ground truth image and the predicted image, respectively. After introducing a segmentation mask, different weights are used to calculate the loss inside and outside the mask. and These represent the foreground and background pixel areas, respectively. ; Since the input image is a grayscale image, the colors encoded by the spherical harmonic function in 3D Gaussian are converted from RGB encoding to HSV encoding.

[0015] Introducing a segmentation mask improves the rendering quality of the target region. The approach of using a large visual model to output a segmentation mask significantly enhances the model's robustness, enabling it to generalize well even when faced with significant variations in asteroid shape and scale.

[0016] Step 3: In the 3D Gaussian method, to ensure that a single 3D Gaussian spectral density (3D Gaussian) saturates at least one pixel, a fixed-magnification dilation filter is used. This expands the influence range of each 3D Gaussian spectral density (3D Gaussian) saturates the region, but reduces the magnitude of the influence on luminance. This leads to near-field high-frequency artifacts and far-field dilation effects in variable-scale scenes. A 3D smoothing filter is introduced to effectively regulate the maximum frequency of the 3D Gaussian spectral density (3D Gaussian) saturates the region, mitigating the high-frequency artifacts. Simultaneously, based on the imaging mechanism of physical light fields, a 2D limiting filter is used instead of a 2D dilation filter to reduce the excessive near-field illumination stacking, thus resolving aliasing and dilation effects. First, the sampling frequency at the current viewpoint is calculated based on the camera intrinsic and extrinsic parameters of the input image. : ; in, Indicates the sampling period. This indicates the camera's focal length from the current perspective. This represents the distance from the camera's principal point to the center of the 3D Gaussian from the current viewpoint. For any given 3D Gaussian, the number of camera viewpoints observing it from different directions is different. To calculate an exact maximum sampling frequency for each 3D Gaussian, the number of camera viewpoints from different viewpoints is used to determine the maximum sampling frequency. The maximum value of the sampling frequency of a 3D Gaussian is taken as the inherent sampling frequency of that 3D Gaussian. : ; in, It is possible to observe the first The total number of 3D Gaussian cameras, It is a logical function that evaluates the visibility of a 3D Gaussian; if the 3D Gaussian center... Falling in If the image is within the field of view of a camera, then it is true; Next, a variable-scale anti-aliasing Gaussian filter is introduced to mitigate the high-frequency artifacts appearing at close range at large scales and the dilation effect appearing at distant range at small scales in the 3D Gaussian filter. This includes a 3D smoothing filter and a 2D limiting filter. The 3D smoothing filter is used. Processing 3D Gaussian : ; in, To represent the convolution operation, the above formula can be more specifically expressed as: ; in, It is the scaling factor that controls the filter. It is the first The natural frequency of a 3D Gaussian. Representing the identity matrix using a 2D pass-limiting filter Processing 2D Gaussian : ; The above formula can be expressed more specifically as follows: ; ; in, It is the projective transformation matrix of 3D space projected onto a 2D plane. It is the Jacobian matrix of the affine approximation of the projective transformation matrix. The 3D smoothing filter follows the Nyquist sampling theorem and filters the 3D Gaussian at a frequency not exceeding twice the maximum sampling frequency. The 2D limit filter is used to weaken the luminous flux amplitude of a pixel region. Ultimately, this allows the 3D Gaussian to render images and reconstruct 3D models well even when the input image is scaled.

[0017] Step 4: In order to filter out redundant Gaussian patches near the asteroid surface model, a statistical filtering module is designed before extracting the mesh model to correct outlier 3D Gaussians and improve the reconstruction quality of the asteroid surface model. First, define a neighborhood interval for each 3D Gaussian, and then select the nearest neighbor to that 3D Gaussian. Each 3D Gaussian is assigned to a neighborhood interval. Based on the distance matrix and sorting algorithm, all neighborhood points of each 3D Gaussian are found, and the distance from each 3D Gaussian to its neighbors is calculated. The entire model contains a total of A 3D Gaussian, using This means that each 3D Gaussian has A neighbor, using This means that the average distance parameters corresponding to all 3D Gaussians are distributed using a one-dimensional Gaussian distribution. Model it, and calculate the mean of the overall average distance of the model. and standard deviation : ; ; For each 3D Gaussian in the model, we need to find the average distance between it and the other surrounding 3D Gaussians. Iterate through all 3D Gaussians, and remove any whose mean distance is greater than the specified confidence level of the one-dimensional Gaussian distribution. ; Before using the statistical filtering strategy, redundant 3D Gaussians appeared on the asteroid surface, resulting in patch noise around the asteroid mesh model in the reconstruction results. After using statistical filtering, the redundant patches were removed.

[0018] Step 5: 3D Gaussian achieves efficient and accurate representation of the geometric and appearance attributes of a scene by defining a new explicit representation method. However, this differs from traditional representation methods such as point clouds, voxels, and patches, making explicit 3D model extraction difficult. When extracting the mesh model, a surface mesh optimization strategy is adopted, and distance alignment loss functions and orientation alignment loss functions are designed to allow 3D Gaussian to align the surface to better fit the surface shape. Based on this, a density function is introduced. And the symbolic distance function : ; ; in, This represents the opacity of a 3D Gaussian. This represents the mean of a 3D Gaussian. Represents the covariance of a 3D Gaussian. Represents the position of any point in space. This represents the minimum scale coefficient of the 3D Gaussian closest to the surface. Indicate the normal direction corresponding to the smallest scale, and calculate the surface alignment loss function: ; in, Representation space The depth of the back projection point of the point corresponding to the pixel plane in space. Indicates the closest to the surface A 3D Gaussian vector is used to calculate the orientation alignment loss function, where... express gradient direction: ; After supervised optimization, the mean of the 3D Gaussian facets is reconstructed using Poisson to obtain the 3D surface model. Then, the surface model and the 3D Gaussian facets are jointly optimized, refining the 3D Gaussian facets in each triangular facet of the model into... Without changing the position of the 3D Gaussian mesh, the mesh model is fixed, and other properties of the 3D Gaussian mesh are optimized. After a certain number of iterations, the Poisson reconstruction is used again to extract the mesh model. This process is repeated alternately to achieve joint optimization of the surface mesh. Finally, a 3D model of the asteroid surface is reconstructed.

[0019] Therefore, this invention can achieve 3D Gaussian reconstruction of variable-scale images by fusing multi-scale image information under multiple imaging detection distances. When there is limited close-range imaging data of asteroids, it can make full use of variable-scale image data to improve the accuracy of 3D reconstruction of asteroid surfaces.

[0020] Contents not described in detail in this specification are common knowledge to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications will be readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the inventive concept are protected.

Claims

1. A method for variable-scale three-dimensional reconstruction of asteroids using multi-imaging detection range fusion, characterized in that, Includes the following steps: Step (1) After preprocessing the input variable-scale image, the SFM method is used to estimate the camera intrinsic and extrinsic parameters and sparse point cloud, and an initial 3D Gaussian is generated from the sparse point cloud. Step (2) For the variable-scale image input in step (1), perform image segmentation on the target region, statistically analyze the pixel ratio inside and outside the mask after morphological dilation, calculate the loss weights of the foreground and background of the image at different scales, and thus construct a multi-scale loss function. Step (3) For the 3D Gaussian established in step (1), calculate the sampling frequency of each 3D Gaussian, introduce a variable-scale anti-aliasing 3D Gaussian filter, and combine it with the multi-scale loss function calculated in step (2) to optimize the 3D Gaussian during training. Step (4) For the 3D Gaussian trained and optimized in step (3), perform statistical filtering on the 3D Gaussian to remove significant outliers. Step (5) Based on the filtering results of step (4), a surface mesh optimization strategy is introduced to further adjust the properties of the 3D Gaussian and obtain the three-dimensional surface model of the asteroid through Poisson reconstruction. The specific implementation method of step (3) is as follows: First, the sampling frequency at the current viewpoint is calculated based on the camera intrinsic and extrinsic parameters of the input image. : ； in, Indicates the sampling period. This indicates the camera's focal length from the current perspective. This represents the distance from the camera's principal point to the center of the 3D Gaussian plane from the current viewpoint, representing the distance seen from different viewpoints. The maximum value of the sampling frequency of a 3D Gaussian is taken as the inherent sampling frequency of that 3D Gaussian. : ； in, It is possible to observe the first The total number of 3D Gaussian cameras, It is a logical function that evaluates the visibility of a 3D Gaussian; if the 3D Gaussian center... Falling in If the image is within the field of view of a camera, then it is true; Next, a variable-scale anti-aliasing Gaussian filter is introduced, including a 3D smoothing filter and a 2D pass-limiting filter. The 3D smoothing filter is used. Processing 3D Gaussian : ； in, To represent the convolution operation, the above formula can be more specifically expressed as: ； in, It is the scaling factor that controls the filter. It is the first The natural frequency of a 3D Gaussian. Representing the identity matrix using a 2D pass-limiting filter Processing 2D Gaussian : ； The above formula can be expressed more specifically as follows: ； ； in, It is the projective transformation matrix of 3D space projected onto a 2D plane. It is the Jacobian matrix of the affine approximation of the projective transformation matrix. The 3D smoothing filter follows the Nyquist sampling law and filters the 3D Gaussian at no more than twice the maximum sampling frequency. The 2D limit filter is used to reduce the luminous flux amplitude of a pixel region.

2. The asteroid variable-scale three-dimensional reconstruction method according to claim 1, characterized in that: The specific implementation method of step (1) is as follows: First, the camera pose and sparse point cloud are estimated using the SFM method. The sparse point cloud is used as the initial 3D Gaussian cloud, which is an explicit scene representation. The scene is represented using 3D anisotropic spheres, each with a mean. Covariance Opacity spherical harmonic coefficient ,in, These are degrees of freedom used to model colors related to the view. The specific formula for 3D Gaussian is as follows: ； mean Represents the positional information and covariance of a 3D Gaussian. Represents the anisotropy of a 3D Gaussian, and its opacity. Used for volume rendering, spherical harmonic coefficients This is to fit color information; specifically, for the first... A 3D Gaussian, represented as: ； in, Indicates the first The position of a 3D Gaussian. Indicates the first The covariance matrix of a 3D Gaussian is used for regularization optimization. Further decomposed into rotation matrices and scaling matrix : ； For this scene representation, view rendering is performed via Gaussian sputtering; first, all 3D Gaussians in the scene are projected onto a 2D image plane, their colors calculated using spherical harmonics, and then, for each of the final image... Pixel blocks are sorted by depth using projected 3D Gaussian graphs that intersect with the pixel blocks. Finally, for each pixel in a pixel block, its color is... It is calculated by synthesizing the opacity and color of all 3D Gaussians covering that pixel: ； in, This indicates the number of 3D Gaussians covering that pixel. This represents the opacity of the 3D Gaussian multiplied by the density of the 2D Gaussian projected at the pixel position. The color used in the calculation.

3. The asteroid variable-scale three-dimensional reconstruction method according to claim 2, characterized in that: The specific implementation method of step (2) is as follows: First, the original image is segmented using a large visual model (SAM) to output a segmentation mask for the target region. Then, a morphological dilation operator is used to dilate the mask. For loss calculations of foreground and background pixels at different scales, a multi-scale loss function is designed. The proportion of pixels inside and outside the mask is statistically analyzed to calculate the loss weights inside and outside the target region. ； ； in, It is the count of deep space background pixels. and These are the width and height of the image, used for the area within the mask. Weighted loss calculation, used for areas outside the mask. Weighted loss is calculated to differentiate the loss in different regions for backpropagation calculations. 3D Gaussian uses a weighted average loss calculation method when calculating the loss. loss: ； in, and These are the pixel grayscale values ​​of the ground truth image and the predicted image, respectively. After introducing a segmentation mask, different weights are used to calculate the loss inside and outside the mask. and These represent the foreground and background pixel areas, respectively. ； Since the input image is a grayscale image, the colors encoded by the spherical harmonic function in 3D Gaussian are converted from RGB encoding to HSV encoding.

4. The asteroid variable-scale three-dimensional reconstruction method based on multi-imaging detection range fusion according to claim 3, characterized in that: The specific implementation method of step (4) is as follows: First, define a neighborhood interval for each 3D Gaussian, and then select the nearest neighbor to that 3D Gaussian. Each 3D Gaussian is divided into its neighborhood intervals. Based on the distance matrix and sorting algorithm, the distance between each 3D Gaussian is found. Given a neighborhood of points, calculate the distance from each 3D Gaussian to its neighbors. The entire model contains a total of A 3D Gaussian, using This means that each 3D Gaussian has A neighbor, using This means that the average distance parameters corresponding to all 3D Gaussians are distributed using a one-dimensional Gaussian distribution. Model it, and calculate the mean of the overall average distance of the model. and standard deviation : ； ； The average distance between each 3D Gaussian in the calculation model and its neighboring 3D Gaussians. Iterate through all 3D Gaussians, and remove any distribution whose mean distance is greater than the specified confidence level of the one-dimensional Gaussian distribution. 。 5. The asteroid variable-scale three-dimensional reconstruction method according to claim 4, characterized in that: The specific implementation method of step (5) is as follows: When extracting the mesh model, a surface mesh optimization strategy is adopted, and distance alignment loss function and orientation alignment loss function are designed, and a density function is introduced. And the symbolic distance function : ； ； in, This represents the opacity of a 3D Gaussian. This represents the mean of a 3D Gaussian. Represents the covariance of a 3D Gaussian. Represents the position of any point in space. This represents the minimum scale coefficient of the 3D Gaussian closest to the surface. Indicate the normal direction corresponding to the smallest scale, and calculate the surface alignment loss function: ； in, Representation space The depth of the back projection point of the point corresponding to the pixel plane in space. Indicates the closest to the surface A 3D Gaussian vector is used to calculate the orientation alignment loss function, where... express gradient direction: ； After supervised optimization, the mean of the 3D Gaussian surface is reconstructed using Poisson algorithm to obtain the 3D surface model. Joint optimization is then performed on the surface model and the 3D Gaussian surface, refining the 3D Gaussian surface in each triangular facet of the model into... Without changing the position, the properties of the 3D Gaussian are optimized, and the mesh model is repeatedly optimized to achieve alternating joint optimization. Finally, a three-dimensional model of the asteroid surface is reconstructed.

Images