A high-fidelity three-dimensional reconstruction method fusing pose prior and geometric constraint

Abstract

The application belongs to the technical field of simulation modeling, and provides a high-fidelity three-dimensional reconstruction method fusing pose prior and geometric constraint, comprising: acquiring an original image, and determining an initial camera pose based on a laser radar inertial odometer; determining error constraint information and relative pose constraint information, and optimizing the initial camera pose according to the error constraint information and the relative pose constraint information to obtain a fine camera pose; inputting the original image into a pre-trained image inference model to obtain a surface normal map output by the image inference model; constructing a three-dimensional Gaussian distribution model, and determining geometric constraint information, and iteratively optimizing the three-dimensional Gaussian distribution model by using the geometric constraint information according to the surface normal map and the fine camera pose to obtain a three-dimensional scene model. The method improves the overall reconstruction efficiency by using the pose prior and the geometric constraint, and improves the authenticity, accuracy and efficiency of the scene reconstruction process.

Description

Technical Field

[0001] This invention relates to the field of simulation modeling technology, and in particular to a high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints. Background Technology

[0002] With the widespread penetration of robotics technology across various industries, the demand for complex environment reconstruction and interactive tasks continues to rise, posing a serious challenge to existing neural scene representation methods. In particular, during the implementation of high-fidelity digital twin technology and the Real2Sim2Real paradigm, while 3D Gaussian Splatting (3DGS) technology has shown potential in efficient rendering and intuitive editing, it has significant shortcomings in large-scale outdoor scene applications.

[0003] On the one hand, weakly textured regions and repetitive texture structures make visual feature matching difficult, leading to geometric and texture consistency issues that directly affect the training convergence and rendering realism of 3D Gaussian splashing models. On the other hand, while existing technologies such as neural radiation fields and 3D Gaussian splashing perform well in view synthesis, they suffer from metric scale ambiguity due to scale drift, making it difficult to meet the requirements of tasks such as precise collision detection and spatial reasoning in robots. Furthermore, the spherical prior of 3DGS causes inaccuracies in surface roughness and edges, which are particularly noticeable when compared with real LiDAR scanning data, further limiting its practical application value. Traditional 3D scene reconstruction schemes also struggle to address these geometric and texture consistency issues.

[0004] This shows that traditional 3D scene reconstruction methods suffer from low efficiency and insufficient realism and accuracy. Summary of the Invention

[0005] This invention provides a high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints to address the shortcomings of traditional 3D scene reconstruction schemes, which are inefficient and not realistic or accurate enough.

[0006] On the one hand, the present invention provides a high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints, comprising:

[0007] Acquire the original image and determine the initial camera attitude based on the lidar inertial odometry;

[0008] Determine the error constraint information and relative attitude constraint information, and optimize the initial camera attitude based on the error constraint information and relative attitude constraint information to obtain the refined camera attitude.

[0009] The original image is input into a pre-trained image inference model to obtain the surface normal map output by the image inference model;

[0010] A three-dimensional Gaussian distribution model is constructed, and geometric constraint information is determined. Based on the surface normal map and the fine camera pose, the three-dimensional Gaussian distribution model is iteratively optimized using the geometric constraint information to obtain a three-dimensional scene model.

[0011] The high-fidelity 3D reconstruction method merging attitude priors and geometric constraints provided by the present invention determines the initial camera attitude based on lidar inertial odometry, including:

[0012] Target spatial data is acquired based on lidar inertial odometry;

[0013] After extrinsic parameter calibration and time synchronization of the camera and lidar inertial odometry, the initial camera attitude is obtained based on the target space data through position linear interpolation and orientation spherical linear interpolation.

[0014] According to the high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints provided by the present invention, error constraint information is determined, including:

[0015] Determine the current camera pose and calculate the translation and rotation errors between the current camera pose and the initial camera pose;

[0016] Calculate the partial derivatives of the translation error and the rotation error with respect to the incremental rotation parameter, respectively, to obtain the partial derivatives of the translation error and the rotation error;

[0017] The partial derivatives of the translation error and the partial derivatives of the rotation error are used as error constraint information.

[0018] According to the high-fidelity 3D reconstruction method integrating attitude priors and geometric constraints provided by the present invention, relative attitude constraint information is determined, including:

[0019] Transform the camera transformation matrix to Lie algebra space and determine the relative pose error between adjacent cameras;

[0020] The relative attitude error is used as relative attitude constraint information.

[0021] According to the high-fidelity 3D reconstruction method integrating attitude priors and geometric constraints provided by the present invention, the initial camera pose is optimized based on the error constraint information and relative attitude constraint information to obtain a refined camera pose, including:

[0022] The error constraint information and relative attitude constraint information are embedded into the beam adjustment process of the camera localization and mapping algorithm to replace the single image constraint mode.

[0023] Based on the adjusted camera positioning and mapping algorithm, the minimum total error is optimized and iterated to output a fine camera pose.

[0024] According to the high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints provided by the present invention, the geometric constraint information is determined, including:

[0025] The three-dimensional scale parameters of the three-dimensional Gaussian distribution model are determined, and the three-dimensional scale parameters are subjected to minimum scale component regularization, and the scale loss value is determined.

[0026] Based on the minimum scale regularization component, the Gaussian scale entropy value is calculated through normalized axis weights, and the shape loss value is determined.

[0027] The rendering normal image and image rendering loss value of the three-dimensional Gaussian distribution model are determined respectively, and the image gradient loss value between the rendering normal image and the surface normal map is calculated.

[0028] Determine the predicted normal at each pixel in the original image, and determine the local smoothing loss value based on the predicted normal;

[0029] The scale loss value, the shape loss value, the image rendering loss value, the image gradient loss value, and the local smoothing loss value are used as geometric constraint information.

[0030] According to the high-fidelity 3D reconstruction method integrating attitude priors and geometric constraints provided by the present invention, based on the surface normal map and the fine camera attitude, the geometric constraint information is used to iteratively optimize the 3D Gaussian distribution model to reconstruct a 3D scene model, including:

[0031] Based on the geometric constraint information, a total loss function is constructed to guide the optimization of model parameters;

[0032] The total loss function is minimized by gradient descent to iteratively optimize the three-dimensional Gaussian distribution model and obtain a three-dimensional scene model.

[0033] According to the high-fidelity 3D reconstruction method integrating attitude priors and geometric constraints provided by the present invention, a total loss function for guiding model parameter optimization is constructed based on the geometric constraint information, including:

[0034] The scale loss value, shape loss value, image rendering loss value, image gradient loss value, and local smoothing loss value are weighted and summed to obtain the total loss function used to guide the optimization of model parameters.

[0035] According to the high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints provided by the present invention, the method further includes:

[0036] Acquire real 3D data of the same target scene and reconstructed 3D data obtained by reconstructing the 3D scene model;

[0037] Based on the real 3D data and the reconstructed 3D data, determine the values ​​of the multidimensional evaluation index;

[0038] Based on the multidimensional evaluation index values, the reconstruction quality of the three-dimensional scene model is evaluated, and the quality evaluation result is obtained.

[0039] According to the high-fidelity 3D reconstruction method integrating attitude priors and geometric constraints provided by the present invention, multi-dimensional evaluation index values ​​are determined based on the real 3D data and the reconstructed 3D data, including:

[0040] Calculate the peak signal-to-noise ratio between corresponding points in the real 3D data and the reconstructed 3D data;

[0041] For the rendered images in the reconstruction process, the structural similarity and image patch similarity between the rendered images and the original images are calculated respectively.

[0042] The peak signal-to-noise ratio, the structural similarity, and the image patch similarity are used as multidimensional evaluation metrics.

[0043] The high-fidelity 3D reconstruction method fusion of attitude prior and geometric constraints provided by this invention acquires the original image and determines the initial camera attitude based on LiDAR inertial odometry; determines error constraint information and relative attitude constraint information, and optimizes the initial camera attitude based on the error constraint information and relative attitude constraint information to obtain a refined camera attitude; inputs the original image into a pre-trained image inference model to obtain the surface normal map output by the image inference model; constructs a 3D Gaussian distribution model and determines the geometric constraint information; and iteratively optimizes the 3D Gaussian distribution model based on the surface normal map and the refined camera attitude using the geometric constraint information to obtain a 3D scene model. This method determines the initial camera pose using LiDAR inertial odometry, and optimizes it by combining error constraints and relative pose constraints to obtain a refined camera pose, which ensures both metric consistency and visual consistency. By generating a high-fidelity surface normal map and iteratively optimizing it with geometric constraint information, the method ensures that the 3D scene model highly matches the real scene in terms of geometric structure and texture. In the optimization stage of the 3D Gaussian distribution model, the surface normal map is used as supervision and the parameters are optimized in a directional manner through geometric constraints, which improves the overall reconstruction efficiency. At the same time, the multi-stage constraint mechanism reduces the impact of scene complexity on the reconstruction results, and improves the realism, accuracy and efficiency of the scene reconstruction process. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints provided in this embodiment of the invention.

[0046] Figure 2 This is a schematic diagram illustrating the implementation principle of the high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints provided in this embodiment of the invention.

[0047] Figure 3 This is a schematic diagram showing the angle between the normal and the camera's viewpoint;

[0048] Figure 4 This is a comparison diagram of the rendering results. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] The following is combined Figures 1 to 4 This invention describes the detailed scheme of a high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints, as provided in embodiments of the present invention.

[0051] like Figure 1 As shown, the high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints provided in this embodiment of the invention mainly includes the following steps:

[0052] Step 110: Acquire the original image and determine the initial camera pose based on the lidar inertial odometry.

[0053] In this embodiment, the stability of LiDAR Inertial Odometry (LIO) is utilized to provide a precise initial attitude reference for subsequent reconstruction, thereby solving the problem of scale ambiguity in traditional reconstruction schemes.

[0054] Step 120: Determine the error constraint information and relative attitude constraint information, and optimize the initial camera attitude based on the error constraint information and relative attitude constraint information to obtain the refined camera attitude.

[0055] It is understood that this embodiment, based on the initial camera pose, improves the motion recovery structure of the camera localization and mapping algorithm through error constraints and relative pose constraints, thereby enhancing the robustness of pose estimation.

[0056] Step 130: Input the original image into the pre-trained image inference model to obtain the surface normal map output by the image inference model.

[0057] In this embodiment, the surface normal map encodes the local geometric information of the scene, which can be used as a ground reality supervision signal for subsequent optimization, and can guide the surface normal direction of the 3D Gaussian distribution model to be consistent with the real scene.

[0058] Step 140: Construct a 3D Gaussian distribution model and determine the geometric constraint information. Based on the surface normal map and the fine camera pose, use the geometric constraint information to iteratively optimize the 3D Gaussian distribution model to obtain a 3D scene model.

[0059] It is understood that this embodiment can optimize the representation of the three-dimensional Gaussian distribution model through geometric constraints, thereby solving the inherent problems of surface roughness and inaccurate edges in the three-dimensional Gaussian distribution model.

[0060] like Figure 2 As shown, this embodiment first utilizes point cloud data, odometry, and inertial measurement unit (IMU) to obtain the initial camera pose by combining target space data through point cloud correction, IMU fusion, and interpolation operations. Then, through scale-aware visual pose optimization, it obtains information such as fine camera pose and metric-aware point cloud.

[0061] Meanwhile, starting from the original image, the true normals are obtained with the help of image inference models such as Omnidata, and then the surface normal map is obtained. At the same time, information such as metric-aware point cloud and fine camera pose is combined to provide a data foundation for subsequent optimization.

[0062] Subsequently, Gaussian optimization is used to iteratively optimize the 3D Gaussian distribution model through a total loss function that includes various loss values. After rasterization and other operations, high-fidelity 3D reconstruction is finally achieved, resulting in a 3D scene model. It is easy to see that the entire reconstruction process integrates pose priors and geometric constraints to improve the accuracy and realism of the 3D reconstruction.

[0063] In one embodiment, determining the initial camera attitude based on lidar inertial odometry specifically includes:

[0064] First, target spatial data is acquired based on lidar inertial odometry.

[0065] In this embodiment, the depth can be directly measured by the lidar in the lidar inertial odometry to obtain spatial data with a clear metric scale, such as position and orientation. At the same time, the data obtained by the inertial measurement unit can be combined to improve the stability in dynamic scenes.

[0066] Then, after extrinsic parameter calibration and time synchronization of the camera and lidar inertial odometry, the initial camera attitude is obtained based on the target space data through position linear interpolation and orientation spherical linear interpolation.

[0067] In this embodiment, the extrinsic parameters of the camera and the lidar inertial odometry are pre-calibrated. These extrinsic parameters mainly include the relative positions and rotational relationships between the camera and the lidar inertial odometry. The purpose of extrinsic parameter calibration is to ensure that the coordinate systems of the camera and the lidar inertial odometry are consistent, so that subsequent data processing and analysis have a unified spatial reference, avoiding data fusion errors caused by differences in coordinate systems.

[0068] In practical applications, spatiotemporal interpolation can be used to synchronize the data acquisition timestamps of the lidar inertial odometry and the camera. Because the data acquisition times of the camera and lidar inertial odometry may differ during actual data acquisition, failure to synchronize the times will result in attitude deviations when performing attitude calculations and other operations based on data acquired at different times, affecting the accuracy of subsequent reconstruction.

[0069] Furthermore, based on continuous pose data in the target space data, the initial camera pose in the world coordinate system, including translation and rotation parameters, can be derived using position linear interpolation and orientation spherical linear interpolation. This initial camera pose has metric consistency and can be directly aligned with world frames, providing a scale benchmark for subsequent scene reconstruction.

[0070] In one embodiment, determining error constraint information specifically includes:

[0071] First, determine the current camera pose and calculate the translation and rotation errors between the current camera pose and the initial camera pose.

[0072] The specific formulas for calculating translation and rotation errors in this step are as follows:

[0073] (1)

[0074] (2)

[0075] in, Indicates translation error. Indicates rotational error.p 0 represents the translation vector of the initial camera pose. q 0 represents the rotation quaternion of the initial camera pose. This represents the translation vector of the current camera pose. The rotation quaternion represents the current camera pose.

[0076] Then, the partial derivatives of the translation error and rotation error with respect to the incremental rotation parameters are calculated respectively to obtain the partial derivatives of the translation error and rotation error.

[0077] In this embodiment, when calculating the translation error... and rotational error Relative to incremental rotation parameters When the partial derivatives are taken, the partial derivatives of the translation error are taken. It can be directly analyzed as the identity matrix I, while the partial derivatives of the rotation error... Derive the matrix from the quaternion and combine it with I. Specifically:

[0078] (3)

[0079] (4)

[0080] Finally, the partial derivatives of translation error and rotation error are used as error constraint information.

[0081] In one embodiment, determining relative attitude constraint information specifically includes:

[0082] First, the camera transformation matrix is ​​transformed to Lie algebra space, and the relative pose error between adjacent cameras is determined.

[0083] In this step, the relative pose error between adjacent cameras can be defined by transforming the camera transformation matrix using Lie algebra space, as follows:

[0084] (5)

[0085] in, e intra Let Log() represent the relative attitude error, and let Log() represent the logarithmic mapping operator from the Lie group to the Lie algebra. h ij Let represent the relative pose transformation matrix from camera i to camera j. P i and P j Let represent the absolute pose matrices of camera i and camera j, respectively. R i and R j Let these represent the rotation matrices for camera i and camera j, respectively. pi and p j Let i and j represent the translation vectors of camera i and camera j, respectively. R ij This represents the relative rotation matrix from camera i to camera j. p ij Let represent the relative translation vector from camera i to camera j.

[0086] Through the above operations, the transformation matrix is ​​converted to Lie algebra space, enabling precise quantification of the difference between the current and desired poses of the two cameras. Furthermore, the partial derivatives of the relative pose error with respect to the camera pose are solved, as follows:

[0087] (6)

[0088] (7)

[0089] (8)

[0090] (9)

[0091] In this embodiment, the gradient direction of camera pose adjustment can be obtained by calculating the partial derivatives of the relative pose error with respect to the translation vector and the rotation matrix, thereby providing data basis for camera pose optimization.

[0092] Then, the relative attitude error is used as the relative attitude constraint information.

[0093] In one embodiment, the initial camera pose is optimized based on error constraint information and relative pose constraint information to obtain a refined camera pose, specifically including:

[0094] First, error constraint information and relative pose constraint information are embedded into the beam adjustment process of the camera localization and mapping algorithm, replacing the single image constraint mode.

[0095] Then, based on the adjusted camera positioning and mapping algorithm, the minimum total error is optimized and iterated to output a fine camera pose.

[0096] In this embodiment, the aforementioned error constraint information and relative pose constraint information are embedded into the bundle adjustment process of the COLMAP camera localization and mapping algorithm, replacing the traditional COLMAP mode that relies solely on image relative constraints. This improvement enhances the stability of feature matching and triangulation, prevents pose estimation failures caused by weak or repetitive textures, and accelerates the optimization convergence speed.

[0097] After optimization of the above constraints, it is possible to output sparse point clouds with metric perception and fine camera pose. The fine camera pose not only meets the metric scale requirements of LiDAR, but also conforms to the consistency of image visual features, which can provide a precise camera parameter basis for subsequent scene reconstruction.

[0098] In one embodiment, determining geometric constraint information specifically includes:

[0099] On the one hand, the three-dimensional scale parameters of the three-dimensional Gaussian distribution model are determined, and the three-dimensional scale parameters are subjected to minimum scale component regularization processing, and the scale loss value is determined.

[0100] This step aims to transform the three-dimensional Gaussian distribution model into a flat, disk-like shape, where the minor axis (approximately the surface normal direction) is significantly smaller than other shapes. To enhance this planar geometry, this embodiment performs minimum scale component regularization on the three-dimensional scale parameters, where the scale component array can be represented as... The scaling loss value can be specifically expressed as:

[0101] (10)

[0102] in, Indicates the scale loss value. s k1 , s k2 , s k3 These represent the scale parameters of the three axes in the three-dimensional Gaussian distribution model, i.e., the three-dimensional scale parameters.

[0103] This embodiment ensures that the Gaussian shape is a flat disk by minimizing the minimum scale component, with the short axis approximating the surface normal direction and the long axis conforming to the scene surface.

[0104] On the other hand, based on the minimum scale regularization component, the Gaussian scale entropy value is calculated through normalized axis weights, and the shape loss value is determined.

[0105] In this step, in order to suppress degenerate needle-like Gaussians during optimization, an ordered scaling axis constraint is enforced, i.e. Furthermore, their relative magnitudes are regularized using entropy, thereby ensuring that the three-dimensional Gaussian distribution model resembles a planar surface when observed, maintaining numerical stability. Specifically, this embodiment defines entropy using normalized axis weights, and the Gaussian-scale entropy value can be expressed as follows:

[0106] (11)

[0107] in, En k This represents the Gaussian scale entropy value of the k-th three-dimensional Gaussian distribution model.w ki The normalized weights of the k-th 3D Gaussian distribution model in the i-th dimension are as follows:

[0108] (12)

[0109] in, s ki This represents the i-th scale component of the k-th three-dimensional Gaussian distribution model.

[0110] Furthermore, a shape loss value can be defined to constrain the Gaussian shape when... At that time, the Gaussian shape is optimal, and the weighting coefficients are set. In this embodiment, the formula for calculating the shape loss value is as follows:

[0111] (13)

[0112] in, This represents the shape loss value, and N represents the number of 3D Gaussian distribution models. This represents a small constant used for numerical stability.

[0113] Understandably, the constraints based on shape loss values ​​ensure that the relative scaling ratios of the three axes are reasonable, effectively preventing scale imbalance problems.

[0114] On the other hand, the rendering normal image and image rendering loss value of the three-dimensional Gaussian distribution model are determined respectively, and the image gradient loss value between the rendering normal image and the surface normal map is calculated.

[0115] To further align geometric priors with reconstruction fidelity, this embodiment introduces normal loss constraints. Surface normals directly encode local geometric information; forcing their accuracy enhances the recovery of fine details. The minor axis of each Gaussian volume is constrained to align with its estimated surface normal. Applying an angle threshold prevents degenerate linear projections at certain camera angles, such as... Figure 3 As shown, the deviation between the minor axis of the Gaussian body and the true normal must exceed 90 degrees to penalize fundamental elements with incorrect orientation.

[0116] In this step, the rendered normal image can be obtained through the α mixture equation, thus rendering the 3D Gaussian normal into a 2D rendered normal image, similar to RGB rendering, which can be specifically represented as:

[0117] (14)

[0118] in, This indicates the rendering of the normal map. n iThis represents the estimated normal from the i-th viewpoint. This represents the weight coefficient for the i-th viewpoint. T i Let represent the transformation matrix from the i-th viewpoint to the target coordinate system.

[0119] In this embodiment, the predicted normals are supervised using a high-fidelity surface normal map, which provides a strong and smooth geometric prior. Specifically, image gradients can be used, and the loss is minimized to minimize the deviation between the rendered normals and the true normals, thereby obtaining the image gradient loss value, which can be expressed as follows:

[0120] (15)

[0121] in, This represents the image gradient loss value. W p represents a set of pixels, where p represents a single pixel. Represents the gradient of the standard ground truth image. This represents the surface normal map, i.e., the set of real normals. This indicates the rendering normal image, i.e., the set of rendering normals.

[0122] On the other hand, the predicted normal at each pixel in the original image is determined, and the local smoothing loss value is determined based on the predicted normal.

[0123] Understandably, surface normals should vary smoothly between adjacent regions unless disrupted by genuine geometric discontinuities (such as edges or occlusions). To reward local consistency in the geometry, this embodiment applies a smoothing loss value, which quantifies this local continuity by penalizing the difference in normals between two adjacent directions, based on the spatial consistency of the predicted normals through regularization. Specifically, the local smoothing loss value can be expressed as follows:

[0124] (16)

[0125] in, This represents the local smoothing loss value. Indicates pixel position The predicted normal at that location.

[0126] Finally, the scale loss, shape loss, image rendering loss, image gradient loss, and local smoothing loss are used as geometric constraint information.

[0127] In one embodiment, based on surface normal mapping and fine camera pose, the 3D Gaussian distribution model is iteratively optimized using geometric constraint information to reconstruct a 3D scene model, specifically including:

[0128] First, based on the geometric constraint information, a total loss function is constructed to guide the optimization of model parameters.

[0129] In a specific implementation, based on geometric constraint information, a total loss function is constructed to guide the optimization of model parameters, specifically including:

[0130] The scale loss, shape loss, image rendering loss, image gradient loss, and local smoothing loss are weighted and summed to obtain the total loss function used to guide the optimization of model parameters.

[0131] In this embodiment, the total loss function can be specifically expressed as follows:

[0132] (17)

[0133] in, This represents the total loss value. Represents the scaling loss value, and the smoothing loss weight coefficient. .

[0134] Then, the total loss function is minimized by gradient descent to iteratively optimize the 3D Gaussian distribution model and obtain the 3D scene model.

[0135] In practical applications, the total loss function obtained above can be minimized by using gradient descent to iteratively optimize key parameters such as position, scale, transparency, and color of the 3D Gaussian distribution model, ultimately resulting in a geometrically accurate and texture-consistent 3D scene model.

[0136] In one embodiment, the above-described high-fidelity 3D reconstruction method that integrates attitude priors and geometric constraints may further include:

[0137] The first step is to acquire the real 3D data of the same target scene and the reconstructed 3D data obtained by reconstructing the scene using a 3D scene model.

[0138] In this step, high-precision, realistic 3D data of the scene to be reconstructed can be obtained for the same target scene. This data can be acquired using high-precision equipment such as laser scanners and will serve as the accurate values ​​for subsequent evaluation. Simultaneously, a 3D scene model is used to reconstruct the target scene, resulting in reconstructed 3D data.

[0139] Given that the evaluation target is a large-scale real-world outdoor environment, two datasets can be used to ensure diverse geographic and structural coverage. On the one hand, the HKUCampus (HKU) dataset includes synchronized LiDAR point clouds and RGB images, spanning mixed indoor and outdoor scenes. On the other hand, the self-captured QY dataset is a proprietary dataset captured by a four-wheeled robotic platform. The self-captured QY dataset has extended trajectories and complex geometric layouts, enabling it to capture detailed outdoor structures in the environment to address gaps in existing benchmarks.

[0140] The second step is to determine the values ​​of multidimensional evaluation indicators based on the real 3D data and the reconstructed 3D data.

[0141] In a specific implementation, multidimensional evaluation index values ​​are determined based on real 3D data and reconstructed 3D data, including:

[0142] On the one hand, the peak signal-to-noise ratio between the real 3D data and the reconstructed 3D data is calculated.

[0143] In practical applications, the Peak Signal-to-Noise Ratio (PSNR) can be defined using the mean square error. For two m×n monochrome images I and K, if one has approximately the same noise level as the other, then their mean square error can be defined as:

[0144] (18)

[0145] in, MSE Indicates mean square error. m、n These represent the height and width of the image, respectively.

[0146] Furthermore, the peak signal-to-noise ratio can be expressed as:

[0147] (19)

[0148] in, PSNR Indicates peak signal-to-noise ratio. This represents the maximum value of the color at a point in the image. If each sample point is represented by 8 bits, then it is 255. The definition of peak signal-to-noise ratio is similar for color images with three RGB values ​​per point, except that the mean squared error can be obtained by dividing the sum of all variances by the image size and then dividing by 3.

[0149] On the other hand, for the rendered images in the rendering stage of the reconstruction process, the structural similarity and image patch similarity between the rendered images and the original images are calculated respectively.

[0150] It is understandable that structural similarity (SSIM) is a metric for measuring the similarity between two images. In this embodiment, given two images x and y, the structural similarity between the two images can be calculated as follows:

[0151] (20)

[0152] in, SSIM ( x,y ) represents the structural similarity between two images x and y. express The average value, express The average value express variance express variance yes and covariance, , These are all constants used to maintain stability. Indicates the dynamic range of pixel values.

[0153] In this embodiment, the structural similarity value ranges from -1 to 1. When two images are identical, the structural similarity value is equal to 1.

[0154] As is understandable, image patch similarity, also known as Learned Perceptual Image Patch Similarity (LPIPS), is a metric used to measure the perceptual similarity between two images. In this embodiment, image patch similarity is learned through deep learning methods, which can better simulate human visual perception. The calculation formula for image patch similarity is not a simple mathematical formula, but is implemented through a deep neural network. Typically, the image patch similarity prediction model uses two images as input and then outputs the perceptual similarity score between them as the image patch similarity. The specific architecture and parameters of the image patch similarity prediction model are obtained through large-scale training to capture perceptual information between images.

[0155] Finally, peak signal-to-noise ratio, structural similarity, and image patch similarity are used as multidimensional evaluation metrics.

[0156] The third step is to evaluate the reconstruction quality of the 3D scene model based on the multi-dimensional evaluation index values, and obtain the quality evaluation results.

[0157] In practical applications, the calculation results of each indicator in the multidimensional evaluation index can be comprehensively analyzed to determine whether the reconstruction quality meets the requirements. For example, peak signal-to-noise ratio, structural similarity, and image patch similarity can be normalized to ensure that the values ​​of each indicator are within the same range. Then, a weighted summation method can be used to determine the comprehensive evaluation value, and the quality evaluation result can be determined based on the range of the comprehensive evaluation value.

[0158] The reconstruction method provided in this embodiment addresses the scene reconstruction problem caused by inconsistencies in geometric textures from two aspects: pose optimization and scene representation. It leverages the stability and reliability of LiDAR in large-scale scenes to improve the COLMAP optimization framework, providing a reliable initial camera pose for each camera. During COLMAP triangulation, LiDAR inertial odometry is incorporated. This method, similar to the implicit function of random sample consistency, ensures geometric texture consistency and prevents pose estimation failures due to significant mismatches. For scene representation, in addition to using photometric loss to enhance texture consistency, reliable visual normal estimation is used as supervision, specifically by constraining the alignment between the surface normals represented by the Gaussian sphere and the predicted normals. Furthermore, this embodiment introduces an effective rank regularization term to constrain the scale of the three axes of the Gaussian sphere and their relative scaling ratios, thereby avoiding misclassification of planar and needle-like Gaussian distributions together.

[0159] To verify the effectiveness of the reconstruction method provided in this embodiment, experiments were conducted on a single NVIDIA GeForce RTX 4060Ti GPU with 16GB VRAM, and the traditional reconstruction scheme and the reconstruction method provided in this embodiment were evaluated and compared. PGSR (Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction) and 2DGS (2D Gaussian Splatting) are geometric constraint methods based on surface constraints and regularization techniques, respectively. SF (Splat Facto) combines the original 3D Gaussian sputtering, while GOF (Gaussian Opacity Fields) represents a typical scheme specifically designed for unbounded outdoor scenes. Key data obtained through comparison are shown in Table 1, and the rendering results are as follows: Figure 4 As shown.

[0160] Table 1 Key data obtained from the comparison

[0161]

[0162] Combining the above table 1 and Figure 4As can be seen, the reconstruction method provided in this embodiment can more effectively reconstruct scene details, especially in texture-deficient areas such as lawns and roads. Since PGSR relies solely on normal constraints, it possesses sharper edge information. In contrast, this method incorporates smoothing and effective rank regularization terms, thereby enhancing the continuity of rendering and mitigating abrupt pixel-level changes in surrounding areas. Therefore, in weakly textured scenes where edge information dominates, texture-deficient areas exhibit minimal photometric variations, and while the reconstruction quality is high, fine-grained quantitative comparisons remain challenging. However, edge information is captured more directly in quantitative metrics. Therefore, although PGSR achieves stronger numerical performance on the QY dataset, its rendering results are far less realistic than the reconstruction method provided in this embodiment.

[0163] Furthermore, while 2DGS is effective in texture-rich regions, it struggles in texture-scarce areas. Meanwhile, GOF lacks explicit constraints such as normal vectors and scale consistency, making it susceptible to local ambiguity caused by repeating textures. These ambiguities, resulting in surface deformation due to repeating textures within opaque regions, lead to poor performance of both 2DGS and GOF in both quantitative and qualitative evaluations.

[0164] In summary, the reconstruction method provided by the embodiments of the present invention has at least the following advantages:

[0165] First, this method innovatively extends COLMAP by proposing a novel optimization scheme. This scheme utilizes prior pose during triangulation and introduces initial and relative error constraints through bundle adjustment, thereby significantly improving the robustness and computational efficiency of pose optimization. This improvement enables more accurate pose estimation even in complex, repetitive texture environments.

[0166] Second, this method introduces normal and shape constraint mechanisms to further enhance the geometric texture consistency during the 3D reconstruction process. Specifically, normal constraints improve surface consistency and coherence by anisotropically regularizing Gaussian elements; while shape constraints consider all scale parameters, determine the relative scales on the three axes, and ensure that the generated 3D model has a highly accurate geometric representation.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-fidelity 3D reconstruction method integrating attitude priors and geometric constraints, characterized in that, include: Acquire raw images and obtain target spatial data based on lidar inertial odometry; After extrinsic parameter calibration and time synchronization of the camera and lidar inertial odometry, the initial camera attitude is obtained based on the target space data through position linear interpolation and orientation spherical linear interpolation. Determine the error constraint information and relative attitude constraint information, and optimize the initial camera attitude based on the error constraint information and relative attitude constraint information to obtain the refined camera attitude. The original image is input into a pre-trained image inference model to obtain the surface normal map output by the image inference model; A three-dimensional Gaussian distribution model is constructed, and the three-dimensional scale parameters of the model are determined. The three-dimensional scale parameters are then regularized using the minimum scale component, and the scale loss value is determined. Based on the minimum scale regularization component, the Gaussian scale entropy value is calculated using normalized axis weights, and the shape loss value is determined. The rendering normal image and image rendering loss value of the 3D Gaussian distribution model are determined respectively, and the image gradient loss value between the rendering normal image and the surface normal map is calculated; the predicted normal at each pixel in the original image is determined, and the local smoothing loss value is determined based on the predicted normal; the scale loss value, shape loss value, image rendering loss value, image gradient loss value and local smoothing loss value are used as geometric constraint information. The scale loss value, the shape loss value, the image rendering loss value, the image gradient loss value, and the local smoothing loss value are weighted and summed to obtain the total loss function used to guide the optimization of model parameters; The total loss function is minimized by gradient descent to iteratively optimize the 3D Gaussian distribution model and obtain the 3D scene model.

2. The high-fidelity 3D reconstruction method fusing attitude priors and geometric constraints according to claim 1, characterized in that, Determine error constraint information, including: Determine the current camera pose and calculate the translation and rotation errors between the current camera pose and the initial camera pose; Calculate the partial derivatives of the translation error and the rotation error with respect to the incremental rotation parameter, respectively, to obtain the partial derivatives of the translation error and the rotation error; The partial derivatives of the translation error and the partial derivatives of the rotation error are used as error constraint information.

3. The high-fidelity 3D reconstruction method fusing attitude priors and geometric constraints according to claim 1, characterized in that, Determine the relative attitude constraint information, including: Transform the camera transformation matrix to Lie algebra space and determine the relative pose error between adjacent cameras; The relative attitude error is used as relative attitude constraint information.

4. The high-fidelity 3D reconstruction method fusing attitude priors and geometric constraints according to claim 1, characterized in that, Based on the error constraint information and relative pose constraint information, the initial camera pose is optimized to obtain a refined camera pose, including: The error constraint information and relative attitude constraint information are embedded into the beam adjustment process of the camera localization and mapping algorithm to replace the single image constraint mode. Based on the adjusted camera positioning and mapping algorithm, the minimum total error is optimized and iterated to output a fine camera pose.

5. The high-fidelity 3D reconstruction method fusing attitude priors and geometric constraints according to claim 1, characterized in that, The method further includes: Acquire real 3D data of the same target scene and reconstructed 3D data obtained by reconstructing the 3D scene model; Based on the real 3D data and the reconstructed 3D data, determine the values ​​of the multidimensional evaluation index; Based on the multidimensional evaluation index values, the reconstruction quality of the three-dimensional scene model is evaluated, and the quality evaluation result is obtained.

6. The high-fidelity 3D reconstruction method fusing attitude priors and geometric constraints according to claim 5, characterized in that, Based on the actual 3D data and the reconstructed 3D data, multidimensional evaluation index values ​​are determined, including: Calculate the peak signal-to-noise ratio between corresponding points in the real 3D data and the reconstructed 3D data; For the rendered images in the rendering stage of the reconstruction process, the structural similarity and image patch similarity between the rendered images and the original images are calculated respectively; The peak signal-to-noise ratio, the structural similarity, and the image patch similarity are used as multidimensional evaluation metrics.

Images