Multi-view image Gaussian reconstruction method and system guided by LiDAR point cloud

Abstract

The invention discloses a multi-view image Gaussian reconstruction method and system guided by LiDAR point cloud. The method comprises the following steps: step S1, realizing accurate registration of LiDAR point cloud and a multi-view image in the same earth-centered earth-fixed coordinate system through GNSS constrained aerial triangulation, and uniformly carrying out center-of-gravity normalization to ensure space consistency and numerical stability; s2, estimating surface normal and scale parameters based on the local geometric features of the LiDAR point cloud, constructing a Gaussian ellipsoid consistent with the local geometry, and realizing Gaussian ellipsoid initialization of geometric perception; and S3, generating a sub Gaussian ellipsoid according to the Gaussian gradient and the tangent plane direction, and improving the modeling density and geometric continuity of the detail region through local encryption and parameter inheritance. According to the method, the fusion of LiDAR point cloud high-precision geometry and multi-view image texture information is realized, the geometric precision, the structural integrity and the detail expression capability of three-dimensional reconstruction are effectively improved, and the method is particularly suitable for high-precision live-action three-dimensional modeling of unmanned aerial vehicle multi-view images.

Description

Technical Field

[0001] This invention belongs to the field of photogrammetry and remote sensing 3D reconstruction technology, specifically relating to a LiDAR point cloud-guided multi-view image Gaussian reconstruction method and system. Background Technology

[0002] With the rapid development of urban digitalization and smart city construction, 3D urban modeling and digital twin technologies have become important supporting tools in urban planning, infrastructure management, and environmental monitoring. Airborne LiDAR, with its high-precision 3D geometric measurement capabilities, has become an important data source for high-precision urban 3D modeling. However, in the complex urban environment, building walls are usually vertical or near-vertical structures. When laser pulses fly at high altitudes and large incident angles, they are prone to weak or no echoes, resulting in severe loss of point cloud data in the facade area. Especially in areas with dense high-rise buildings and surface materials with strong reflective or absorptive characteristics, such as glass curtain walls and metal facades, the laser beam often cannot effectively receive reflected signals, thus forming obvious "blank bands" that affect the geometric accuracy and structural integrity of the building facade.

[0003] In recent years, 3D reconstruction techniques based on Gaussian splatting have attracted widespread attention due to their ability to continuously represent complex curved surface structures. This type of method achieves efficient geometric reconstruction and rendering by representing spatial elements in a scene as Gaussian ellipsoids or Gaussian polygons. However, the reconstruction effect of Gaussian splatting largely depends on the quality and integrity of the initial point cloud. When using an empty triangulation point cloud as the initial input, the reconstruction result often inherits structural voids and noise distributions from the input data, leading to discontinuities or blurred textures on the model surface. Existing Gaussian densification strategies generally employ isotropic point expansion methods, failing to consider the directional characteristics and spatial morphological constraints of building structure surfaces, thus exhibiting shortcomings in detail recovery and geometric continuity. Therefore, there is an urgent need for an airborne laser point cloud completion and densification method that integrates multi-source information and introduces morphological prior constraints to improve the integrity of point cloud data and the geometric continuity of 3D structures. Summary of the Invention

[0004] This invention proposes a LiDAR point cloud-guided multi-view Gaussian reconstruction method and system, aiming to solve the problems of insufficient reconstruction geometric accuracy, missing building facade areas, and discontinuous Gaussian models caused by the sparseness and incomplete structure of the three-dimensional point cloud in traditional 3D reconstruction.

[0005] This invention achieves deep fusion of multi-view images and LiDAR data through a unified geospatial registration mechanism, a Gaussian ellipsoid initialization method based on LiDAR geometric priors, and a tangent-plane-guided structure-enhanced Gaussian compaction strategy, and constructs a 3D Gaussian representation with high geometric consistency and continuity.

[0006] To achieve the above objectives, the method of the present invention includes the following steps: Step 1: Accurate registration of LiDAR point cloud and multi-view imagery in the same geocentric geofixed coordinate system is achieved through GNSS-constrained aerial triangulation, and unified centroid normalization is performed to ensure spatial consistency and numerical stability. Step 2: Based on the two-dimensional Gaussian sputtering, the geometric characteristics of the LiDAR point cloud are introduced into the Gaussian initialization stage as spatial prior information. The position, color, normal vector, scale and opacity of the Gaussian ellipsoid are constrained and corrected to achieve geometric and photometric consistency of the initial Gaussian set. Step 3 introduces a Gaussian compaction method based on local tangent plane constraints to generate new Gaussian units while maintaining local geometric consistency. These units are then merged with the Gaussian set before compaction to obtain an updated Gaussian set. Subsequently, the merged Gaussian set is re-input into the optimization stage based on differentiable rendering. By iteratively updating the spatial location, scale, rotation, and appearance parameters of the Gaussians, the compacted Gaussian distribution gradually converges to the optimal state that is highly consistent with the actual ground structure.

[0007] Furthermore, step 1 specifically includes the following sub-steps: (11) System configuration and data acquisition The drone platform is equipped with a lidar sensor and a visible light imaging sensor to simultaneously acquire laser point cloud data and multi-view image data. (12) Solving for exterior orientation elements of the image An aerial triangulation method constrained by GNSS is used to obtain the exterior orientation parameters of the images. Specifically, firstly, the acquired multi-view images are preprocessed, including image distortion correction and radiometric consistency processing. Then, image feature extraction and corresponding point matching are used to establish connectivity between images. Based on this, position information provided by the UAV platform is introduced as an external constraint, and bundle adjustment is used to optimize the exterior orientation elements of the images, thereby obtaining the pose parameter set. and the corresponding aerial triangular cloud ; (13) Coordinate unification and spatial datum transformation Three points of cloud in the sky And the camera position is uniformly transformed to the Earth-centered ECEF coordinate system, LiDAR laser point cloud It is also projected onto the ECEF coordinate system to achieve coordinate system unification; (14) Center of gravity normalization treatment After completing the coordinate unification, the centroid normalization process was further performed on the aerial three-point cloud, laser point cloud, and image position parameters:

[0008] In the formula, For the first in the joint point cloud The coordinates of N points are given; N is the total number of points. The coordinates of each point and the camera position parameters are then transformed by translation to obtain the point cloud coordinates in the local coordinate system. : .

[0009] Furthermore, the initialization of the ellipsoid position includes: The centroid-normalized LiDAR point cloud coordinates are directly used as the Gaussian center position, and the point cloud set is set as follows: Then the first i Gaussian center for: .

[0010] Furthermore, color initialization includes: First, the color information corresponding to each 3D point is extracted from the input laser point cloud. The color information comes from the optical image acquired and calibrated synchronously with the lidar system, corresponding to the red color of each point i. ,green ,blue Three color components; normalization and numerical preprocessing of the RGB color vector; The RGB color information is mapped to the DC term, i.e., the 0th order term of the spherical harmonic function, as the initial appearance description; specifically, each point is mapped using a predefined proportional mapping relationship. i Convert the RGB color vector to the corresponding 0th order spherical harmonic coefficient vector:

[0011] in, Represents the 0th order spherical harmonic coefficients on the red, green, and blue channels; Then, construct a size of for each laser point. The vector of spherical harmonic coefficients, where The maximum spherical harmonic order is given, and the three channels correspond to R, G, and B colors respectively; the overall coefficient tensor dimension is given. During initialization, Write the 0th-order spherical harmonic basis for each point on the three channels, and initialize the coefficients of all higher-order bases to 0.

[0012] Furthermore, the initialization of the normal vector includes: For each LiDAR point in the input laser point cloud First, using this point as the center, construct a set of its neighboring points within a preset spatial distance range. Based on the spatial distribution characteristics of points within a neighborhood, statistical analysis is performed on the dispersion of neighborhood points relative to the neighborhood mean to construct a local covariance matrix, which is used to characterize the geometric features around the point. The local covariance matrix is ​​expressed as:

[0013] in For LiDAR points Neighborhood points within a certain distance range, The mean of the neighborhood points. This indicates the number of neighboring points.

[0014] After obtaining the local covariance matrix, eigenvalue decomposition is performed. By comparing the spatial dispersion of directions corresponding to different eigenvalues, the eigenvector corresponding to the smallest eigenvalue is selected as the local surface normal direction of the LiDAR point, denoted as . .

[0015] Furthermore, the initialization of the scale includes: After obtaining the local surface normal of each LiDAR point, the scale parameters of the Gaussian ellipsoid are further initialized according to the point spacing of the laser point cloud to constrain the expansion range of the Gaussian ellipsoid in different spatial directions. Specifically, by statistically analyzing the current point Within the neighborhood, distance The spatial distances from the nearest neighboring points to the point are calculated, and the average of these distances is taken. The resulting value is used as the scale parameter of the Gaussian ellipsoid in the two principal axis directions within the local tangent plane, which is used to characterize the spatial expansion characteristics of the local structure of the ground in the planar direction. During the scale initialization process, the scale parameter corresponding to the normal direction is set to zero or a preset minimum value to constrain the thickness of the Gaussian ellipsoid in the normal direction, so that it presents a thin layer structure spreading along the surface of the ground. Using the above method, a corresponding Gaussian ellipsoidal scale matrix is ​​constructed for each LiDAR point:

[0016] in, and This represents the local tangent plane orientation scale parameter determined based on the average distance of the nearest neighbor points, used to ensure that the Gaussian ellipsoid can completely cover the ground surface in the planar direction; The scale parameter in the normal direction takes a value of zero or a minimum, thus forming a thin Gaussian structure that is closely attached to the surface of the ground.

[0017] Further, the initialization of opacity includes: First, set the initial value of the target opacity in a physical sense for the Gaussian ellipsoid. Furthermore, to ensure the numerical stability of the opacity parameter during continuous optimization, instead of directly optimizing the opacity, a parameterized modeling approach based on inverse activation functions is introduced. This involves converting the initial value of the target opacity into an unconstrained optimization parameter through inverse activation mapping.

[0018] The above transformation maps the opacity from the interval (0,1) to the real number space, thereby avoiding numerical saturation or gradient vanishing problems during the optimization process. In subsequent iterative optimization and rendering calculations, the parameters are adjusted through a forward activation process corresponding to the aforementioned inverse activation function. Remapped to a physical opacity value for use in volume rendering and appearance modeling.

[0019] Furthermore, the Gaussian compaction method based on local tangent plane constraints specifically includes: (31) Selection of the father Gauss During the optimization process, the parent Gaussian unit for compaction is first selected from the current Gaussian set: the gradient magnitude of each Gaussian unit is calculated based on its gradient response strength during iterative optimization. Choose to satisfy And scale The Gaussian unit was selected as the candidate parent Gaussian; (32) Construction of the tangent plane coordinate system After determining the parent Gaussian element, a local tangent plane coordinate system is constructed based on its spatial orientation: Let the center position of the parent Gaussian element be... Its corresponding local surface normal is When no explicit normal information exists, it is calculated inversely from the parent Gaussian rotation quaternion; using this normal as the tangent plane normal constraint, two unit tangent basis vectors are constructed in its orthogonal plane. and Used to establish local tangent planes; and All are orthogonal to the local normal vector, and are mutually orthogonal and normalized, i.e., satisfying:

[0020] From this, we can obtain... For the direction of the law, with and It is a local tangent plane coordinate system with orthogonal tangent bases, used to constrain the distribution direction of sub-Gauss in the tangent plane of the ground surface; (33) Sub-Gaussian generation and parameter inheritance Within the local tangent plane, centered on the parent Gaussian... To generate multiple sub-Gaussian elements for reference, the first... The center position of a Gaussian is represented as:

[0021] in , The offset along the tangent plane direction is determined by the parent Gaussian scale and the preset offset ratio. On the tangent plane, the child Gaussians are symmetrically distributed about the center of the parent Gaussian within the tangent plane, or the child Gaussians are generated by regular grid distribution or random perturbation distribution. (4) Integration and optimization All generated sub-Gaussian units constitute a subset. and the Gaussian set before compaction. The merged set yields the updated Gaussian set:

[0022] Subsequently, the merged Gaussian set is re-input into the optimization stage based on differentiable rendering. By iteratively updating the spatial location, scale, rotation, and appearance parameters of the Gaussians, the densified Gaussian distribution gradually converges to the optimal state that is highly consistent with the actual ground structure.

[0023] The present invention also provides a LiDAR point cloud-guided multi-view image Gaussian reconstruction system, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the LiDAR point cloud-guided multi-view image Gaussian reconstruction method as described in the above technical solution.

[0024] The present invention also provides a computer-readable storage medium, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements the LiDAR point cloud-guided multi-view image Gaussian reconstruction method as described in the above technical solution.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Compared with existing Gaussian reconstruction methods that rely on triangulated point clouds, this invention achieves integrated modeling of LiDAR point clouds and multi-view images at the data fusion level. By unifying the two types of data into a geocentric-fixed coordinate system and normalizing the centroid, the spatial consistency of multi-source data and the numerical stability of the computation process are significantly improved. This results in higher convergence efficiency and robustness for subsequent differentiable rendering and parameter optimization, thereby improving the overall accuracy and reliability of 3D reconstruction.

[0026] 2. This invention utilizes the local geometric features of LiDAR point clouds to perform geometrically perceptual initialization of the Gaussian ellipsoid in terms of orientation, scale, and opacity, enabling the Gaussian representation to possess realistic surface structure features from the initial stage. Simultaneously, through precise back-projection of image color onto the LiDAR point cloud, accurate representation of high-resolution radiometric information in the Gaussian model is achieved. Compared to traditional isotropic or image-optimized Gaussian initialization methods, this invention significantly improves geometric consistency, texture alignment, and structural representation integrity.

[0027] 3. The tangential plane-induced densification strategy proposed in this invention can adaptively generate sub-Gaussians based on local gradients and normal directions, effectively completing the geometric structure at edges, facades, and areas with dense details. This anisotropic densification mechanism overcomes the structural breaks or discontinuities caused by the neglect of surface orientation in traditional Gaussian densification methods, resulting in a final 3D model with excellent performance in surface continuity, geometric detail restoration, and stereo consistency. It is particularly suitable for high-quality reconstruction of building facades, thin-sheet structures, and areas with weak laser echoes. Attached Figure Description

[0028] Figure 1 This is a flowchart of the LiDAR point cloud-guided multi-view image Gaussian reconstruction method provided in this embodiment of the invention; Figure 2 This is a comparison of Gaussian reconstruction results guided by empty three-point clouds and LiDAR point clouds.

[0029] Figure 3 This is a comparison image of LiDAR point cloud (input) and 3D model (output). Detailed Implementation

[0030] To facilitate understanding of the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are merely illustrative of the present invention and not intended to limit the scope of protection of the present invention.

[0031] like Figure 1 As shown in the figure, an embodiment of the present invention provides a LiDAR point cloud-guided multi-view image Gaussian reconstruction method, which includes the following steps: Step 1, Geographic Registration and Center of Gravity Normalization: Accurate registration of LiDAR point cloud and multi-view imagery in the same geocentric geofixed coordinate system is achieved through GNSS-constrained aerial triangulation, and unified center of gravity normalization is performed to ensure spatial consistency and numerical stability.

[0032] This step aims to achieve geometric alignment between airborne laser point cloud and multi-view image data under a unified spatial reference, providing a consistent and numerically stable coordinate basis for subsequent Gaussian ellipsoid initialization and 2DGS-based tangent-plane-guided Gaussian compaction. The specific implementation process is as follows.

[0033] (1) System configuration and data acquisition In this embodiment, the drone platform is equipped with a lidar sensor and a visible light imaging sensor (such as the DJI Zenmuse L1) to simultaneously acquire lidar point cloud data and multi-view image data. The lidar and camera acquire data through the same flight platform, and their time synchronization and spatial installation relationship are determined during the system calibration phase, thereby ensuring an initial geometric correlation between the two types of sensor data.

[0034] (2) Solving for exterior orientation elements of the image To obtain precise position and attitude information of the images in space, a GNSS-constrained aerial triangulation method is employed to acquire the image exterior orientation parameters. Specifically, the acquired multi-view images are first preprocessed, including image distortion correction and necessary radiometric consistency processing. Then, connectivity between images is established through image feature extraction and corresponding point matching. Based on this, position information provided by the UAV platform is introduced as an external constraint, and bundle adjustment is used to optimize the image exterior orientation elements, thereby obtaining a pose parameter set with high geometric accuracy and stability. and the corresponding aerial triangular cloud .

[0035] (3) Coordinate unification and spatial reference transformation To ensure that LiDAR and image data are processed under the same reference system, the aerial triangulation cloud will be used. And the camera positions were uniformly transformed to the Earth-centered Earth-fixed (ECEF) coordinate system. LiDAR laser point cloud It is also projected onto the ECEF system to achieve coordinate system unification.

[0036] (4) Center of gravity normalization treatment Because the ECEF coordinate values ​​are of a large magnitude, directly using them for subsequent Gaussian parameter modeling and optimization calculations may cause numerical instability. In this embodiment, after coordinate unification, centroid normalization is further performed on the aerial tri-point cloud, laser point cloud, and image position parameters:

[0037] In the formula, The first point in the joint point cloud (including aerial triangulation point cloud, laser point cloud and image position parameters) The coordinates of the points The total number of points is given; the coordinates of each point and the camera position parameters are translated and transformed to obtain the point cloud coordinates in the local coordinate system. :

[0038] By performing the above-mentioned centroid normalization process, the absolute magnitude of the point cloud and camera position coordinate values ​​can be effectively reduced, so that the joint data exhibits a zero-mean distribution in coordinate representation. This reduces the magnitude of subsequent numerical calculations and improves the numerical stability and convergence performance of parameter differentiation and iterative optimization processes.

[0039] Step 2, LiDAR-guided Gaussian ellipsoid initialization based on 2DGS: Based on the local geometric features of the LiDAR point cloud, the surface normal and scale parameters are estimated, and a Gaussian ellipsoid consistent with the local geometry is constructed to achieve geometrically aware Gaussian initialization; Based on the 2D Gaussian Splatting (2DGS) algorithm framework, the geometric characteristics of LiDAR point clouds are introduced as spatial prior information into the Gaussian initialization stage. The position, color, normal vector, scale, and opacity of the Gaussian ellipsoid are constrained and corrected, thereby achieving geometric and photometric consistency of the initial Gaussian set and providing stable initialization conditions for subsequent differentiable rendering and parameter optimization.

[0040] (1) Initialization of ellipsoid position The centroid-normalized LiDAR point cloud coordinates are directly used as the Gaussian center position, and the point cloud set is set as follows: Then the first i Gaussian center for:

[0041] If both LiDAR and aerial triangulation data exist at a certain location, the LiDAR points will be prioritized to ensure that the Gaussian center corresponds to the actual ground surface.

[0042] (2) Color initialization First, the color information corresponding to each 3D point is extracted from the input laser point cloud. This color information originates from optical images acquired and calibrated synchronously with the lidar system, corresponding to each point. i Red ( ),green( ),blue( The RGB color vector consists of three color components. It is then normalized and preprocessed to meet the numerical stability requirements of the subsequent spherical harmonic representation.

[0043] Since the zeroth-order term (DC term) of the spherical harmonic function represents an isotropic component independent of the viewing direction, this invention maps RGB color information to this DC term as the initial appearance description. Specifically, through a predefined proportional mapping relationship, each point... i Convert the RGB color vector to the corresponding 0th order spherical harmonic coefficient vector:

[0044] in, This represents the 0th-order spherical harmonic coefficients in the red, green, and blue channels. This process realizes the conversion of color information from pixel space to spherical harmonic function parameter space, providing a stable initial color constraint for three-dimensional Gaussian representation.

[0045] Then, construct a size of for each laser point. The vector of spherical harmonic coefficients, where The maximum spherical harmonic order is represented by the three channels, corresponding to R, G, and B colors respectively. The overall coefficient tensor has a dimension of [missing value]. During initialization, Write the 0th-order spherical harmonic basis for each point on the three channels, and initialize the coefficients of all higher-order bases to 0.

[0046] Therefore, the initial Gaussian ellipsoid contains only isotropic color components that are independent of the viewpoint, while the appearance details related to higher-order viewpoints are gradually updated in subsequent iterations.

[0047] (3) Initialization of normal vector To make the Gaussian ellipsoid spatially similar to the representation of a 2D Gaussian sputtering pattern spreading along the object's surface, with its principal axis perpendicular to the local surface of the ground feature, this implementation initializes the geometric pose of the Gaussian ellipsoid based on information from the neighborhood normal of the laser point cloud, thereby achieving geometric alignment between the Gaussian ellipsoid and the real ground feature surface. Specifically, for each LiDAR point in the input laser point cloud... First, using this point as the center, construct a set of its neighboring points within a preset spatial distance range. Based on the spatial distribution characteristics of points within a neighborhood, statistical analysis is performed on the dispersion of neighborhood points relative to the neighborhood mean to construct a local covariance matrix, which is used to characterize the geometric features around the point. The local covariance matrix can be expressed as:

[0048] in For LiDAR points Neighborhood points within a certain distance range, The mean of the neighborhood points. This indicates the number of neighboring points.

[0049] After obtaining the local covariance matrix, eigenvalue decomposition is performed. By comparing the spatial dispersion of directions corresponding to different eigenvalues, the eigenvector corresponding to the smallest eigenvalue is selected as the local surface normal direction of the LiDAR point, denoted as . Since point clouds have the least dispersion in the normal direction in planar or near-planar structures, this normal direction can accurately reflect the local geometric orientation of building walls, roofs, and other structures.

[0050] (4) Scale initialization After obtaining the local surface normal of each LiDAR point, the scale parameters of the Gaussian ellipsoid are further initialized based on the point spacing of the laser point cloud to constrain the expansion range of the Gaussian ellipsoid in different spatial directions.

[0051] Specifically, by statistically analyzing the current point Within the neighborhood, distance The spatial distances from the nearest several neighboring points to the given point are calculated, and the average of these distances is taken. The resulting value is used as the scale parameter of the Gaussian ellipsoid along the two principal axes within the local tangent plane, characterizing the spatial expansion characteristics of the local ground structure in the planar direction. Preferably, the number of neighboring points is... (In this embodiment, it is set to 3).

[0052] Meanwhile, to prevent the Gaussian ellipsoid from expanding unreasonably along the local normal direction, causing the ellipsoid to detach from the actual ground surface, the scale parameter corresponding to the normal direction is set to zero or a preset minimum value during the scale initialization process. This constrains the thickness of the Gaussian ellipsoid in the normal direction, making it present a thin layer structure that spreads along the ground surface.

[0053] Using the above method, a corresponding Gaussian ellipsoidal scaling matrix can be constructed for each LiDAR point:

[0054] in, and This represents the local tangent plane orientation scale parameter determined based on the average distance of the nearest neighbor points, used to ensure that the Gaussian ellipsoid can completely cover the ground surface in the planar direction; The scale parameter in the normal direction takes a value of zero or a minimum, thus forming a thin Gaussian structure that is closely attached to the surface of the ground.

[0055] (5) Opacity initialization To enhance the constraint of the laser point cloud on the reconstruction results in the initial optimization stage, the opacity parameter of the Gaussian ellipsoid is reasonably initialized. Considering that airborne or UAV laser point clouds usually have high geometric reliability and spatial confidence, in order to ensure that the reconstruction results in the initial stage can fully inherit the structural information of the laser point cloud, this embodiment initializes the opacity corresponding to the Gaussian ellipsoid to a relatively high level.

[0056] Specifically, firstly, an initial value for the physical opacity of the target is set for the Gaussian ellipsoid. Its value is located in the higher range, preferably taking the value of . This setting ensures that the Gaussian ellipsoid corresponding to the laser point has high visibility in the initial rendering stage, which is beneficial for stabilizing the constrained geometry during the early optimization process.

[0057] Furthermore, to ensure the numerical stability of the opacity parameter during continuous optimization, this implementation does not directly optimize the opacity. Instead, it introduces a parameterized modeling approach based on inverse activation functions. Specifically, by performing an inverse activation mapping on the initial value of the target opacity, it is transformed into an unconstrained optimization parameter.

[0058] The above transformation maps the opacity from the interval (0,1) to the real number space, thereby avoiding numerical saturation or gradient vanishing problems during the optimization process.

[0059] In subsequent iterative optimization and rendering calculations, the parameters are optimized through a forward activation process corresponding to the aforementioned inverse activation function. The opacity is remapped to a physically defined value for use in volume rendering and appearance modeling. This initialization strategy allows the laser point cloud to exert a stronger constraint on geometric reconstruction in the early stages of optimization, while ensuring numerical stability. This provides stable input conditions for subsequently introducing image information and gradually relaxing the constraints to achieve denser reconstruction.

[0060] Step 3: 2DGS-based tangential plane-guided Gaussian compaction: To improve the resolution of 3D reconstruction in local high curvature and geometric detail regions, a Gaussian compaction method based on local tangential plane constraints is introduced on the basis of the 2D Gaussian Splatting surface spreading idea. This method generates new Gaussian units while maintaining local geometric consistency, thereby enhancing the structural continuity and geometric fineness of the reconstructed model.

[0061] (1) Selection of the father Gauss During the optimization process, a parent Gaussian unit for compaction is first selected from the current Gaussian set. Preferably, the gradient magnitude of each Gaussian unit is calculated based on its gradient response strength during iterative optimization. Choose the option that satisfies the requirements. And scale The Gaussian units are selected as candidate parent Gaussians. To limit the growth rate of the number of Gaussians, a proportional constraint is imposed on the number of newly added Gaussians in each round. When the number of candidates exceeds a preset upper limit, parent Gaussians with larger gradient responses are preferentially retained to participate in the compaction process.

[0062] (2) Construction of the tangent plane coordinate system After determining the parent Gaussian element, a local tangent plane coordinate system is constructed based on its spatial orientation. Let the center position of the parent Gaussian element be... Its corresponding local surface normal is When explicit normal information is unavailable, it can be calculated inversely from the parent Gaussian rotation quaternion. Using this normal as a constraint on the tangent plane normal, two unit tangent basis vectors are constructed within its orthogonal plane. and This is used to establish a local tangent plane. Preferably, and All are orthogonal to the local normal vector, and are mutually orthogonal and normalized, i.e., satisfying:

[0063] From this, we can obtain... For the direction of the law, with and It is a local tangent plane coordinate system with orthogonal tangent basis, used to constrain the distribution direction of sub-Gauss in the tangent plane of the ground surface.

[0064] (3) Sub-Gaussian generation and parameter inheritance Within the local tangent plane, centered on the parent Gaussian... To generate multiple sub-Gaussian elements for reference, the first... The center position of a Gaussian can be represented as:

[0065] in , The offset along the tangent plane is determined by the parent Gaussian scale and a preset offset ratio. Preferably, the child Gaussians are symmetrically distributed about the center of the parent Gaussian within the tangent plane; in other embodiments, a regular grid distribution or a random perturbation distribution can also be used to generate the child Gaussians.

[0066] (4) Integration and optimization All generated sub-Gaussian units constitute a subset. and the Gaussian set before compaction. The merged set yields the updated Gaussian set:

[0067] Subsequently, the merged Gaussian set is re-input into the optimization stage based on differentiable rendering. By iteratively updating the spatial location, scale, rotation, and appearance parameters of the Gaussians, the densified Gaussian distribution gradually converges to the optimal state that is highly consistent with the actual ground structure.

[0068] like Figure 2 The image shows a comparison of 3D reconstruction results from Gaussian sputtering under different guidance methods. Figure 2 (a) in the figure represents the Gaussian sputtering reconstruction result guided by aerial triangulation point cloud. As can be seen from the figure, there are large areas of surface defects in the vertical structural areas such as building walls and columns, and continuous geometric surfaces cannot be formed in some areas, affecting the integrity of the overall 3D model. Figure 2 (b) in the diagram represents the Gaussian sputtering reconstruction result guided by LiDAR point clouds. By introducing the high-precision spatial constraint information provided by LiDAR point clouds, the geometric structure of areas such as walls and columns is effectively restored, missing areas are significantly reduced, structural boundaries are clearer, and the surface continuity of the model is improved. In summary, using LiDAR point cloud guidance helps improve the reconstruction quality of building facades and detailed structural areas, and enhances the structural integrity and geometric consistency of the 3D model.

[0069] like Figure 3 The image shows a comparison diagram between LiDAR point cloud data and the 3D reconstruction results generated by the method of this invention. Wherein, Figure 3 (a) shows the original LiDAR point cloud distribution. Due to factors such as laser echo obstruction, incident angle and material reflection characteristics, there are sparse and missing point clouds on the building facade and in some areas. Figure 3 (b) shows the 3D reconstruction result generated based on the method of this invention. By combining multi-view aerial imagery information to complete the LiDAR point cloud, it is possible to effectively fill in missing areas based on the original LiDAR geometric constraints, making the point cloud distribution in the building facade area more uniform and the model surface more complete. For areas that are not observable in aerial images, such as under the eaves, there may still be a small number of unrecovered areas. The image-assisted LiDAR point cloud completion method proposed in this invention effectively reduces the missing LiDAR point cloud in areas such as building facades while maintaining geometric consistency, improves point cloud density and model integrity, and thus enhances the overall quality of the 3D reconstruction results.

[0070] The accuracy of the 3D model reconstruction in this invention was determined using laser point clouds with higher density and more complete structure, collected through cross-strip flight data, as reference data. Precision was selected as the accuracy. P ), recall R ) and comprehensive indicators ( F 1) As an evaluation index, it is used to comprehensively assess the geometric accuracy and completeness of the reconstructed model. Its calculation formula is as follows:

[0071] As shown in Table 1, two typical building structures, flat roofs and gable roofs, were selected as experimental subjects. The Gaussian sputtering reconstruction method guided by three-point clouds was used as a comparison scheme, while the Gaussian sputtering reconstruction method guided by LiDAR point clouds was the implementation scheme of this invention.

[0072] Table 1. Comparison of accuracy between Aerial Triangulation Point Cloud and LiDAR Point Cloud-Guided Gaussian Splash Reconstruction Results (Unit: %)

[0073] In the scenario of a bungalow roof, the method of this invention achieves improvements in accuracy, recall, and... F The 1-values ​​are all higher than those of the comparison method. F The accuracy rate increased from 81.1% in the comparison scheme to 87.8%, indicating that the completeness of the reconstruction results was effectively improved while maintaining high accuracy. In scenarios with relatively complex structures such as gable roofs, the method of this invention also achieved superior results. F The I value increased from 66.3% to 71.5%, indicating that the present invention can still stably improve the overall performance of 3D reconstruction under complex structural conditions.

[0074] In summary, the experimental results in Table 1 show that using LiDAR point clouds as guiding information for Gaussian sputtering reconstruction can simultaneously improve the accuracy and recall of the reconstruction model under different building structure types, thereby obtaining a better comprehensive evaluation index and verifying the effectiveness of the method of the present invention in terms of geometric integrity and robustness.

[0075] Secondly, embodiments of the present invention also provide a LiDAR point cloud-guided multi-view image Gaussian reconstruction system, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the LiDAR point cloud-guided multi-view image Gaussian reconstruction method as described in the above technical solution.

[0076] Thirdly, embodiments of the present invention also provide a computer-readable storage medium, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements the LiDAR point cloud-guided multi-view image Gaussian reconstruction method as described in the above technical solution.

[0077] The specific embodiments described in this invention are merely illustrative examples to illustrate the spirit of the invention. Those skilled in the art can make various modifications or additions to the described specific embodiments or use similar methods to replace them, but without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A LiDAR point cloud guided multi-view image Gaussian reconstruction method, characterized in that, The method comprises the following steps: Step 1: precise registration of LiDAR point cloud and multi-view image in the same geocentric coordinate system by GNSS-constrained aerial triangulation, and unified barycentric normalization is performed to ensure spatial consistency and numerical stability; Step 2: on the basis of two-dimensional Gaussian sputtering, the geometric characteristics of the LiDAR point cloud are introduced as spatial prior information into the Gaussian initialization stage, and the position, color, normal vector, scale and opacity of the Gaussian ellipsoid are constrained and corrected, so that the geometric consistency and photometric consistency of the initial Gaussian ellipsoid set are realized; Step 3: introduce a Gaussian densification method based on local tangent plane constraint, which is used to generate new Gaussian units while maintaining local geometric consistency, and then merge the Gaussian set before densification to obtain an updated Gaussian set; subsequently, the merged Gaussian set is re-input into the optimization stage based on differentiable rendering, and the spatial position, scale, rotation and appearance parameter information of the Gaussian are iteratively updated, so that the Gaussian distribution after densification gradually converges to an optimal state highly consistent with the real object structure.

2. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: Step 1 specifically comprises the following sub-steps: (11) System configuration and data acquisition The unmanned aerial vehicle platform is equipped with a laser radar sensor and a visible light imaging sensor, and laser point cloud data and multi-view image data are synchronously acquired; (12) Image exterior orientation element solving The GNSS-constrained aerial triangulation method is used to obtain image exterior orientation parameters; specifically, first, image preprocessing is performed on the obtained multi-view images, including image distortion correction and radiation consistency processing; subsequently, a connection relationship between the images is established through image feature extraction and same-name point matching; on this basis, the position information provided by the unmanned aerial vehicle platform is introduced as an external constraint condition, and the bundle adjustment is used to optimize and solve the image exterior orientation elements, so as to obtain a set of pose parameters and corresponding aerial triangulation point clouds ​ (13) Coordinate unification and space reference conversion The empty three-point cloud and camera position are converted to the Earth-Centered, Earth-Fixed (ECEF) coordinate system, and the LiDAR laser point cloud is also projected to the ECEF coordinate system to achieve the unification of coordinate systems; (14) Barycentric normalization processing After completing the coordinate unification, the barycentric normalization processing is further performed on the aerial triangulation point cloud, the laser point cloud and the image position parameters: In the formula, For the first in the joint point cloud The coordinates of N points are given; N is the total number of points. The coordinates of each point and the camera position parameters are then transformed by translation to obtain the point cloud coordinates in the local coordinate system. : 。 3. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: The initialization of the ellipsoid position comprises: The coordinates of the gravity normalized LiDAR point cloud are directly taken as the Gaussian center positions, and the point cloud set is denoted as The first i Gaussian center is: 。 4. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: The initialization of the color comprises: Firstly, color information corresponding to each 3D point is extracted from the input laser point cloud. The color information is derived from the synchronized and calibrated optical images acquired with the LiDAR system, corresponding to the red , green , and blue color components of each point i. The RGB color vector is then normalized and numerically pre-processed. RGB color information is mapped to the DC term, i.e. the 0th order term of the spherical harmonics, as the initial appearance description; in particular, the RGB color vector of each point i is converted to the corresponding 0th order spherical harmonic coefficient vector by a predefined scaling mapping: wherein denotes the spherical harmonic 0thorder coefficients on the red, green, blue three channels; Then, for each laser point, a spherical harmonic coefficient vector of size is constructed, where is the maximum spherical harmonic order and the three channels correspond to R, G, B colors, respectively; the total dimension of the coefficient tensor is ; at initialization, the zeroth order spherical harmonic basis on three channels for each point is written and all higher order bases are initialized to 0.

5. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: The initialization of the normal vector comprises: For each LiDAR point in the input laser point cloud , first, a neighborhood point set is constructed within a preset spatial distance range centered on the point ; based on the spatial distribution characteristics of the neighborhood points, the dispersion of the neighborhood points relative to the neighborhood mean is statistically analyzed, thereby constructing a local covariance matrix for depicting the geometric morphological features around the point; the local covariance matrix is represented as: wherein LiDAR points neighbor points within a certain distance range, neighbor points mean, denotes the number of neighbor points; After obtaining the local covariance matrix, it is processed by eigenvalue decomposition, and by comparing the spatial dispersion degree of the direction corresponding to different eigenvalues, the eigenvalue corresponding to the minimum eigenvalue is selected as the local surface normal direction of the LiDAR point, denoted as .

6. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: The initialization of the scale comprises: After obtaining the local surface normal of each LiDAR point, the scale parameter of the Gaussian ellipsoid is initialized according to the point spacing of the laser point cloud to constrain the expansion range of the Gaussian ellipsoid in different spatial directions; Specifically, by statistically analyzing the current point Within the neighborhood, distance The spatial distances from the nearest neighboring points to the point are calculated, and the average of these distances is taken. The resulting value is used as the scale parameter of the Gaussian ellipsoid in the two principal axis directions within the local tangent plane, which is used to characterize the spatial expansion characteristics of the local structure of the ground in the planar direction. During the scale initialization process, the scale parameter corresponding to the normal direction is set to zero or a preset minimum value to constrain the thickness of the Gaussian ellipsoid in the normal direction, so that it presents a thin layer structure spreading along the object surface; In this way, a corresponding Gaussian ellipsoid scale matrix is constructed for each LiDAR point: wherein, with denotes the local tangent plane direction scale parameter determined based on the average distance of the nearest points in the neighborhood, which is used to ensure that the Gaussian ellipsoid can completely cover the surface of the object in the plane direction; denotes the scale parameter in the normal direction, which takes a value of zero or a very small value, thereby forming a thin layer of Gaussian structure that closely fits the surface of the object.

7. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: The initialization of the opacity comprises: Firstly, the initial value of the target opacity of the Gaussian ellipsoid in the physical sense is set . The Gaussian ellipsoid corresponding to the laser point cloud is set to a large value. Further, in order to keep the numerical stability of the opacity parameter in the continuous optimization process, the opacity is not directly optimized, but a parameterized modeling method based on the inverse activation function is introduced, that is, the target opacity initial value is converted into an unconstrained optimization parameter by inverse activation mapping. Through the above transformation, the opacity is mapped from the interval (0, 1) to the real number space, so as to avoid the problem of numerical saturation or gradient disappearance in the optimization process; In the subsequent iteration optimization and rendering calculation process, through the forward activation process corresponding to the above inverse activation function, the parameters are remapped to physically meaningful opacity values for participating in volume rendering and appearance modeling.

8. The LiDAR point cloud guided multi-view image Gaussian reconstruction method of claim 1, wherein: The Gaussian densification method based on local tangent plane constraint specifically comprises: (31) Selection of parent Gaussian In the optimization process, first, select the parent Gaussian unit for densification from the current Gaussian set: according to the gradient response intensity of each Gaussian unit in the iterative optimization, calculate its gradient amplitude , select the Gaussian unit satisfying and the scale of the Gaussian unit as the candidate parent Gaussian (32) Construction of tangent plane coordinate system After the parent Gaussian cell is determined, a local tangent plane coordinate system is constructed according to its spatial pose: let the center position of the parent Gaussian be , and its corresponding local surface normal be , which is obtained by back calculation from the parent Gaussian rotation quaternion when there is no explicit normal information; with the normal as the tangent plane normal constraint, two unit tangent basis vectors and are constructed in its orthogonal plane to establish the local tangent plane; and are both orthogonal to the local normal vector, and are orthogonal to each other and unitized, that is, satisfy: Thus, the local tangent plane coordinate system with as the normal direction, with and as the orthogonal tangent basis is obtained, which is used to constrain the distribution direction of the sub-Gaussians in the tangent plane of the surface of the object; (33) Generation of child Gaussian and parameter inheritance In the local tangent plane, the center of the parent Gaussian is taken as the reference To generate a plurality of sub-Gaussian units with respect to the reference, the center position of the first sub-Gaussian is represented as: wherein , is the offset in the tangent plane direction, which is determined by the parent Gaussian scale and a preset offset ratio. On the tangent plane, the child Gaussians are symmetrically distributed about the parent Gaussian center in the tangent plane, or are generated in a regular grid distribution or random perturbation distribution manner; (4) Fusion and optimization All generated sub-Gaussian units form a sub-set and merged with the pre-densification Gaussian set resulting in an updated Gaussian set: Subsequently, the merged Gaussian set is re-input into the optimization stage based on differentiable rendering, and the spatial position, scale, rotation and appearance parameter information of the Gaussian are iteratively updated, so that the Gaussian distribution after densification gradually converges to an optimal state highly consistent with the real object structure.

9. A LiDAR point cloud guided multi-view image Gaussian reconstruction system, characterized by: A computer program product comprising a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the LiDAR point cloud guided multi-view image Gaussian reconstruction method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, A computer program product comprising a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the LiDAR point cloud guided multi-view image Gaussian reconstruction method according to any one of claims 1-8.

Images