Lidar point cloud and three-dimensional gaussian field coupling completion modeling method and device

Abstract

The application provides a laser radar point cloud and three-dimensional Gaussian field coupling completion modeling method and device, relates to the technical field of three-dimensional reconstruction, and comprises the following steps: acquiring laser radar point cloud data and multi-view image data of a target building area; constructing a semantic enhanced three-dimensional Gaussian splash radiation field based on the multi-view image data, and constructing a spatial density field based on the laser radar point cloud data, wherein each Gaussian elementary unit in the three-dimensional Gaussian splash radiation field is endowed with a semantic category probability attribute; calculating a difference confidence map by combining the feature indexes of the three-dimensional Gaussian splash radiation field and the spatial density field, and positioning a geometric missing area according to the difference confidence map; fitting a virtual completion plane for the laser radar point cloud on the edge of the geometric missing area, and performing parameter correction and morphological remodeling on the three-dimensional Gaussian elementary unit; and fusing the repaired Gaussian point cloud and the laser radar point cloud data to reconstruct a watertight triangular mesh model. The application can realize three-dimensional modeling of a non-Lambertian region.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method and apparatus for coupling and completing modeling of lidar point clouds and 3D Gaussian fields. Background Technology

[0002] With the rapid development of digital twin cities, distributed photovoltaic potential assessment, and digital preservation of ancient buildings, extremely high requirements have been placed on the accuracy of 3D modeling of urban building roofs. Currently, the main data sources for 3D roof reconstruction rely on airborne LiDAR and oblique photogrammetry.

[0003] However, existing technologies have significant limitations and shortcomings when dealing with roof areas made of non-Lambertian materials. Modern buildings extensively use glass skylights (such as shopping mall atriums and train station roofs) or black, highly reflective photovoltaic panels. Near-infrared lasers emitted by lidar are transmitted through glass; the low reflectivity, specular reflection, and transmission collectively result in insufficient or missing effective echoes, causing these areas to appear as large areas of voids (geometric gaps) in point cloud data. Traditional photogrammetry methods based on multi-view solid geometry (MVS) rely on feature point matching, which is based on the assumption that the object surface is Lambertian (diffuse reflection). However, glass and smooth photovoltaic panels have strong specular reflection and transmission characteristics, leading to feature matching failures and generating models that often contain a large amount of noise, artifacts, or erroneous indentations. Summary of the Invention

[0004] In view of this, this application proposes a method and apparatus for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields.

[0005] Firstly, this application provides a method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields, including: Acquire lidar point cloud data and multi-view image data of the target building area; A semantically enhanced three-dimensional Gaussian splash radiation field is constructed based on the multi-view image data, and a spatial density field is constructed based on the lidar point cloud data. Each Gaussian element in the three-dimensional Gaussian splash radiation field is assigned a semantic category probability attribute. The difference confidence map is calculated by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field, and the geometric missing region corresponding to the planar non-Lambertian region is located based on the difference confidence map; The lidar point cloud at the edge of the geometrically missing region is extracted as a geometric prior. A virtual completion plane is fitted based on the geometric prior, and the parameters of the three-dimensional Gaussian primitives in the region are corrected and the shape is reshaped to generate the repaired Gaussian point cloud. The repaired Gaussian point cloud is fused with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area.

[0006] In one embodiment, constructing a semantically enhanced three-dimensional Gaussian splash radiation field based on the multi-view image data includes: The multi-view image data is input into a pre-trained semantic segmentation model to perform pixel-by-pixel classification of the multi-view image data, define the target material category, and output the corresponding two-dimensional semantic soft probability map of the target material. Configure geometric and color parameters for the Gaussian elements of the initial three-dimensional Gaussian splash radiation field, and add a semantic log probability parameter; A visual differentiable rendering pipeline and a semantic differentiable rendering pipeline are constructed. An end-to-end iterative training is carried out on the initial three-dimensional Gaussian splash radiation field using a target loss function. During training, adaptive density control of Gaussian units and optimization and update of all parameters are completed simultaneously, and the semantic log probability parameter is activated and transformed into semantic category probability. The target loss function includes a visual loss function and a semantic loss function. After training convergence, the semantically enhanced three-dimensional Gaussian splash radiation field is obtained.

[0007] In one embodiment, constructing the spatial density field based on the lidar point cloud data includes: The lidar point cloud data is preprocessed, and the three-dimensional space of the roof is determined based on the preprocessed lidar point cloud data. The three-dimensional space of the roof is modeled in voxel form to construct a voxel mesh adapted to the roof features; The number of LiDAR point clouds within each voxel is counted and normalized to generate spatial density values; The normalized density values ​​are bound to the voxel mesh to generate the spatial density field.

[0008] In one embodiment, the calculation of the difference confidence map by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field includes: Based on the rendering pipeline of the three-dimensional Gaussian splash radiation field, the cumulative opacity value corresponding to the center of each voxel is calculated. Determine the set of three-dimensional Gaussian elements corresponding to each voxel in the spatial density field, and calculate the average semantic probability of the Gaussian elements in each voxel belonging to the target material category. Based on the density distribution of the spatial density field, the cumulative opacity value, and the average semantic probability, a difference score for each voxel is calculated, and a difference confidence map is generated.

[0009] In one embodiment, locating the geometrically missing region caused by material characteristics based on the difference confidence map includes: Based on the difference confidence map, voxels with difference scores greater than a preset score threshold are marked as geometric missing candidate regions; The geometrically missing candidate regions are filtered based on the semantic properties of the target material to determine the geometrically missing regions.

[0010] In one embodiment, extracting the lidar point cloud of the edge of the geometrically missing region as a geometric prior, and fitting a virtual completion plane based on the geometric prior, includes: Extract the lidar point cloud within a preset neighborhood range outside the boundary of the geometrically missing region as a reference point set; The reference point set is fitted with a random sampling consensus algorithm to construct a candidate fitting plane. Calculate the root mean square error from the reference point set to the candidate fitting plane, and when the root mean square error is less than a preset error threshold, determine the candidate fitting plane as the virtual completion plane.

[0011] In one embodiment, the step of performing parameter correction and morphological reshaping on the three-dimensional Gaussian primitives within the region to generate a repaired Gaussian point cloud includes: An anisotropic projection constraint mechanism is established based on the virtual completion plane; Based on the anisotropic projection constraint mechanism, the position of the three-dimensional Gaussian primitives in the geometrically missing region is corrected and the shape is reshaped to generate the repaired Gaussian point cloud.

[0012] In one embodiment, the step of correcting the position and reshaping the shape of the three-dimensional Gaussian primitives in the geometrically missing region according to the anisotropic projection constraint mechanism to generate a repaired Gaussian point cloud includes: Project the center coordinates of each 3D Gaussian element within the geometrically missing region onto the virtual completion plane along the plane normal vector of the virtual completion plane, and update the center position of each 3D Gaussian element within the geometrically missing region; The rotation matrix of each three-dimensional Gaussian element within the geometrically missing region is forcibly adjusted so that the shortest axis direction of each three-dimensional Gaussian element is parallel to the plane normal vector; The scaling factor of each three-dimensional Gaussian element in the shortest axis direction within the geometrically missing region is set to a preset minimum value, so that each three-dimensional Gaussian element degenerates into a two-dimensional Gaussian sheet that fits the virtual completion plane.

[0013] In one embodiment, fusing the repaired Gaussian point cloud with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area includes: Filter out outlier Gaussian elements with opacity below the confidence threshold in the three-dimensional Gaussian splash radiation field; The repaired Gaussian metacenter coordinates were converted into point cloud format, while preserving color attributes; The converted point cloud is filled into the holes in the lidar point cloud, and the watertight triangular mesh model is generated using the Poisson surface reconstruction algorithm or the restricted Deloni triangulation algorithm.

[0014] Secondly, this application provides a device for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields, comprising: The acquisition module is used to acquire lidar point cloud data and multi-view image data of the target building area; The construction module is used to construct a semantically enhanced three-dimensional Gaussian splash radiation field based on the multi-view image data, and to construct a spatial density field based on the lidar point cloud data, wherein each Gaussian element in the three-dimensional Gaussian splash radiation field is assigned a semantic category probability attribute. The joint analysis module is used to calculate the difference confidence map by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field, and to locate the geometric missing region of the corresponding planar non-Lambertian region based on the difference confidence map; The correction module is used to extract the lidar point cloud of the edge of the geometrically missing region as a geometric prior, fit a virtual completion plane based on the geometric prior, and perform parameter correction and morphological reshaping on the three-dimensional Gaussian primitives in the region to generate the repaired Gaussian point cloud. The reconstruction module is used to fuse the repaired Gaussian point cloud with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area.

[0015] The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields proposed in this application has the following advantages over related technologies: 1. The lidar point cloud and 3D Gaussian field coupled completion modeling method of this application first simultaneously acquires lidar point cloud and multi-view image dual-source data. It then constructs a semantically enhanced 3D Gaussian splash radiation field based on the multi-view image. Leveraging the characteristic of 3D Gaussian splashing technology that does not rely on the Lambertian diffuse reflection assumption, it effectively adapts to the specular reflection and transmission characteristics of glass and photovoltaic panels, avoiding noise and artifacts caused by feature matching failures in traditional MVS. Simultaneously, it achieves accurate identification of target materials through the semantic category probability attributes of Gaussian units, and quantifies the point cloud distribution density based on a spatial density field constructed from the lidar point cloud. Finally, by jointly calculating the difference confidence map using dual-field feature indices, it accurately locates geometrically missing regions caused by material transmission and absorption, overcoming the limitation of single data in accurately identifying geometrically missing regions.

[0016] 2. Subsequently, using the laser point cloud at the edge of the geometrically missing region as the geometric prior, a virtual completion plane is fitted. Through Gaussian meta-parameter correction and morphological reshaping, a precisely repaired Gaussian point cloud is generated, achieving targeted completion of large-area point cloud holes and avoiding model distortion caused by blind completion. Finally, the repaired Gaussian point cloud is fused with the original lidar point cloud to reconstruct a watertight triangular mesh model, which not only makes up for the geometric missing parts of the lidar point cloud but also avoids the modeling defects of traditional photogrammetry, ultimately achieving complete, accurate, and artifact-free 3D modeling of non-Lambertian regions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for coupling and completing modeling of lidar point clouds and three-dimensional Gaussian fields in one embodiment of this application. Figure 2 This is a schematic diagram illustrating the principle of constructing a light-geometric difference confidence map and locating special material regions in one embodiment of this application; Figure 3 This is a schematic diagram of the anisotropic projection constraint principle of Gaussian elements based on a virtual completion plane in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of a lidar point cloud and three-dimensional Gaussian field coupling and completion modeling device in one embodiment of this application. Detailed Implementation

[0019] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] In some embodiments, such as Figure 1 As shown, this application provides a method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields, including the following steps S101 to S105.

[0021] S101: Acquire lidar point cloud data and multi-view image data of the target building area.

[0022] The lidar point cloud data can be acquired by using a drone equipped with a lidar (LiDAR) and a visible light camera to collect data on the target building. The lidar point cloud data can include three-dimensional coordinates (X, Y, Z) and intensity information. The image data can include multi-view RGB images and corresponding camera pose parameters (POS).

[0023] It should be noted that after acquiring the original LiDAR point cloud data and multi-view image data of the target building area, aerial triangulation can be performed based on the camera intrinsic and extrinsic parameters or pose parameters corresponding to the multi-view images to determine the world coordinate system of the three-dimensional Gaussian splash radiation field. Based on the world coordinate system, the LiDAR point cloud data and multi-view image data are uniformly registered using the extrinsic calibration parameters of the airborne LiDAR and camera, or using control points, feature correspondence points, ICP registration, or similar transformation, so that the three-dimensional Gaussian splash radiation field, LiDAR point cloud data, and subsequent voxel mesh are in the same spatial reference system.

[0024] S102: Construct a semantically enhanced 3D Gaussian splash radiation field based on multi-view image data, and construct a spatial density field based on lidar point cloud data. In the 3D Gaussian splash radiation field, each Gaussian element is assigned a semantic category probability attribute.

[0025] It is understandable that, based on the rich visual texture and spatial pose information contained in multi-view images, a semantically enhanced 3D Gaussian splash radiation field can be constructed. Leveraging the characteristic of 3D Gaussian splashing technology that does not rely on the Lambertian diffuse reflection assumption, it can effectively adapt to the specular reflection and transmission characteristics of non-Lambertian materials such as glass and photovoltaic panels. Simultaneously, by assigning semantic category probability attributes to each Gaussian element in the radiation field, each Gaussian element not only carries geometric shape and color features but also accurately represents the probability of belonging to the target material (such as glass or photovoltaic panels) or the background material, thus achieving semantic enhancement of the radiation field and accurately distinguishing target material areas from other areas. Meanwhile, a spatial density field is constructed based on LiDAR point cloud data. This is centered on the 3D spatial location information of the LiDAR point cloud. Through discretization of the 3D space and quantitative statistics of the point cloud distribution density, a spatial density field reflecting the density of LiDAR point clouds in different spatial regions is formed. The simultaneous construction of these two fields leverages both the visual semantic advantages of multi-view images and the spatial geometric advantages of LiDAR point clouds.

[0026] S103: Calculate the difference confidence map by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field, and locate the geometric missing region of the corresponding planar non-Lambertian region based on the difference confidence map.

[0027] Among them, the characteristic index of the three-dimensional Gaussian splash radiation field can be the cumulative opacity value reflecting the visual presence and the average semantic probability of Gaussian units characterizing the material category, and the characteristic index of the spatial density field can be the normalized spatial density value that quantifies the density of the laser point cloud distribution.

[0028] In the application, the characteristic indicators of the three-dimensional Gaussian splash radiation field and the spatial density field are coupled and calculated according to a preset logic to obtain the difference score of each spatial unit. This allows the construction of a difference confidence map covering the entire domain, which presents the degree of mismatch between visual semantic features and geometric density features in a quantitative form. By setting a reasonable difference score threshold, spatial units with high cumulative opacity values ​​(visually present effective areas), high semantic probability of the target material (confirmed as non-Lambertian materials such as glass and photovoltaic panels), but low normalized spatial density values ​​(sparse or missing laser point clouds) are selected. This accurately locates geometrically missing areas where LiDAR detection fails due to material transmission and absorption characteristics. This effectively avoids the problem that a single data dimension cannot distinguish between missing material characteristics and other factors causing the missing data, providing accurate regional positioning basis for subsequent targeted data completion.

[0029] S104: Extract the lidar point cloud from the edge of the geometrically missing region as a geometric prior, fit a virtual completion plane based on the geometric prior, and perform parameter correction and morphological reshaping on the three-dimensional Gaussian primitives in the region to generate the repaired Gaussian point cloud.

[0030] It is understandable that the lidar point cloud at the edge of the geometrically missing region is unaffected by the transmission and absorption characteristics of non-Lambertian materials, preserving the true geometric information of the roof structure and possessing reliable spatial reference value. Based on this, extracting the lidar point cloud at the edge of the geometrically missing region as a geometric prior, and fitting a virtual completion plane based on the geometric prior, can construct a geometric benchmark that is continuous and consistent with the surrounding structure of the missing region, providing precise spatial constraints for subsequent completion. Furthermore, for the three-dimensional Gaussian elements within the geometrically missing region, parameter correction and morphological reshaping can be performed using the virtual completion plane as constraints. This involves both positional correction to ensure the Gaussian elements fit the spatial position of the completion plane and morphological reshaping to guarantee their consistency with the overall roof structure. Ultimately, this generates a repaired Gaussian point cloud that accurately fills the gaps in the lidar point cloud, effectively avoiding artifacts or structural deviations that are prone to occur in traditional completion methods.

[0031] S105: The repaired Gaussian point cloud is fused with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area.

[0032] In the application, the repaired Gaussian point cloud and the original LiDAR point cloud are first precisely aligned in spatial coordinates to ensure the consistency of their positions in the same coordinate system. Then, through deduplication and feature complementarity integration strategies, the high-precision geometric structure information of the LiDAR point cloud in non-missing regions is preserved. At the same time, the repaired Gaussian point cloud is used to fill geometric holes caused by non-Lambertian materials, forming a complete point cloud dataset with full coverage and no structural contradictions. Based on this, a 3D mesh reconstruction algorithm is used to perform topology construction and surface generation on the integrated point cloud, ultimately forming a watertight triangular mesh model.

[0033] The aforementioned method for coupled completion modeling of LiDAR point clouds and 3D Gaussian fields first simultaneously acquires dual-source data: LiDAR point clouds and multi-view images. It then constructs a semantically enhanced 3D Gaussian splash radiation field based on the multi-view images. Leveraging the characteristic of 3D Gaussian splashing technology that does not rely on the Lambertian diffuse reflection assumption, it effectively adapts to the specular reflection and transmission characteristics of glass and photovoltaic panels, avoiding noise and artifacts caused by feature matching failures in traditional MVS. Simultaneously, it achieves accurate identification of target materials through the semantic category probability attributes of Gaussian units, and quantifies the point cloud distribution density based on a spatial density field constructed from the LiDAR point clouds. Finally, by jointly calculating the difference confidence map using dual-field feature indices, it accurately locates geometrically missing regions caused by material transmission and absorption, overcoming the limitation of accurately identifying geometrically missing regions using single data. Subsequently, using the laser point cloud at the edge of the geometrically missing region as the geometric prior, a virtual completion plane was fitted. Through Gaussian meta-parameter correction and morphological reshaping, a precisely repaired Gaussian point cloud was generated, achieving targeted completion of large-area point cloud holes and avoiding model distortion caused by blind completion. Finally, the repaired Gaussian point cloud was fused with the original lidar point cloud to reconstruct a watertight triangular mesh model, which not only made up for the geometric missing parts of the lidar point cloud but also avoided the modeling defects of traditional photogrammetry, ultimately achieving complete, accurate, and artifact-free 3D modeling of non-Lambertian regions.

[0034] In some embodiments, constructing a semantically enhanced 3D Gaussian splash radiation field based on multi-view image data includes: inputting multi-view image data into a pre-trained semantic segmentation model, performing pixel-by-pixel classification on the multi-view image data, defining target material categories and outputting a corresponding 2D semantic soft probability map of the target material; configuring geometric and color parameters for the Gaussian primitives of the initial 3D Gaussian splash radiation field, and adding a semantic log-probability parameter; constructing a visual differentiable rendering pipeline and a semantic differentiable rendering pipeline, performing end-to-end iterative training of the initial 3D Gaussian splash radiation field using a target loss function, simultaneously completing adaptive density control of the Gaussian primitives and optimization updates of all parameters during training, and converting the semantic log-probability parameter activation into semantic category probability; and obtaining the semantically enhanced 3D Gaussian splash radiation field after training convergence. The target loss function includes a visual loss function and a semantic loss function.

[0035] The construction of the initial 3D Gaussian splash radiation field for semantic enhancement includes image semantic segmentation and extraction of special material probability fields. This aims to accurately identify special material regions that are difficult for LiDAR to detect from a 2D visual perspective, providing a data foundation for subsequent 3D semantic injection. This process may include the following detailed steps: semantic segmentation model selection, target category definition strategy, and semantic probability map output.

[0036] For semantic segmentation models, a Transformer-based semantic segmentation network (such as Mask2Former or SegFormer) can be used as the backbone network. Compared to traditional CNN networks, this type of network has stronger global context awareness capabilities and can effectively handle the problem of continuous recognition of large-scale roof textures (such as continuous photovoltaic panels) from a drone's top-down perspective, as well as the fine segmentation task of glass skylight edges. During the inference stage, multi-view RGB raw images are input into the pre-trained and fine-tuned semantic segmentation model, and each pixel in the image is classified pixel-wise.

[0037] For the target category definition strategy (i.e., the strategy for defining the target material category), in order to cooperate with the subsequent "geometric hole repair" logic, this embodiment adopts a "positive and negative sample asymmetric definition" strategy to construct the target category set. Positive samples (targets requiring repair): Key definitions ={"Photovoltaic panels", "Glass skylights"}. These two types of materials correspond to the "specular reflection loss" and "transmission penetration loss" characteristics of LiDAR, respectively, belonging to the non-Lambertian feature regions that are "visible but geometrically missing". Negative samples (background / interference): ={"walls","vegetation","ground","shadow"} are all classified as background.

[0038] For the semantic probabilistic map output, unlike the traditional hard classification (Hard Argmax) output, this embodiment outputs a two-dimensional semantic soft probabilistic map that includes confidence information. For any pixel in the image... Calculate its belonging to Joint probability value of any class This probability value This will serve as a supervisory signal for subsequent training of the 3DGS radiation field, so that each 3D Gaussian primitive generated in the end not only carries color (SH coefficient) and geometric properties (position, rotation), but also explicitly carries "material semantic properties", thereby achieving a precise mapping from 2D semantics to 3D space.

[0039] In the process of training 3DGS, in addition to optimizing color (spherical harmonic coefficients) and geometric parameters, a new set of "Semantic Logits" parameters is added.

[0040] Among them, color (spherical harmonics, SH) is not just a single RGB color value, but a set of mathematical coefficients that can express "anisotropic color". It can simulate the changes in light and shadow on the surface of an object as the viewing angle changes.

[0041] Geometric parameters define the physical state of each Gaussian primitive (i.e., the tiny "clouds" that make up the scene) in three-dimensional space. The main geometric parameters include: position, ): Gao Siyuan in Coordinates in a coordinate system; Covariance (Scale & Rotation): controls the shape of the Gaussian unit (round or elongated) and its rotation angle; Opacity. ): Controls whether the primitive is transparent or solid.

[0042] For example, suppose there is a point on a "black photovoltaic panel" on the roof of the target building area. The SH coefficient would record: "Deep black when viewed from directly above, but with a bright white reflection when viewed at a 45-degree angle from the side." The geometric parameters would define: "This is a point located at coordinates..." The flattened ellipsoid, with its rotation angle parallel to the roof slope. Semantic Logits parameter: used to give the 3D scene "cognitive ability," enabling it to distinguish materials. It is an additional data dimension attached to each 3D Gaussian unit. During training, it does not participate in the color calculation of the rendered image, but is projected back to 2D through the "semantic rendering pipeline," and is used in conjunction with the previously generated "2D semantic segmentation map (generated by the previous semantic segmentation model)" to calculate the cross-entropy loss. Logits: It is an unnormalized probability value, which, after being processed by the Softmax activation function, is transformed into the fact that the Gaussian unit belongs to a specific category (e.g., ...). Normalized probability of photovoltaic or glass in .

[0043] Continuing with the example of the points on the "black photovoltaic panel," in addition to color and location, a set of semantic parameters (Semantic Logits) is assigned to them: [PV panel: 5.2, glass: -1.0, vegetation: -3.5, others: -2.0]. The meaning after conversion is (…). The system determines that this point has a 98% probability of being a "solar panel". Comparison example (tree canopy): If it's a point on the edge of the canopy: Semantic Logits will show the highest value for "vegetation". Ultimately, The value is close to 0. Therefore, although the LiDAR did not detect any points here (there are holes), the system will automatically ignore it and not perform plane fitting repair because the semantics determine that it is not the target material.

[0044] Next, end-to-end iterative training needs to be carried out on the initial three-dimensional Gaussian splash radiation field. Through the differentiable rendering pipeline, the semantic parameters of the 3D Gaussian points are projected back to the 2D image, and the cross-entropy loss is calculated with the previously generated 2D semantic segmentation map.

[0045] First, the image is rendered using all currently differentiable Gaussian functions. During each training iteration's forward pass, the rasterizer executes two rendering tasks in parallel: visual rendering and semantic rendering. The visual rendering pass: based on the geometric parameters (position, rotation, scaling) and color parameters (spherical harmonics SH, opacity) of the current Gaussian elements... The current RGB rendered image is generated through Alpha Blending, following the standard process. Semantic rendering pass: Using the same geometric projection and opacity blending mechanism, the semantic logits carried by each Gaussian unit are projected into screen space. The result is a 2D rendered semantic feature map consistent with the viewport resolution, where the value of each pixel represents the weighted contribution of all Gaussian units traversed by the ray to the semantic category of that pixel.

[0046] Then, the loss is calculated based on the difference between the rendered image and the real image (and the semantic mask), with the target loss function being... It includes two parts: visual loss and semantic loss, which respectively constrain the consistency between visual and semantic features. For example, if the semantic segmentation model identifies a pixel as a "photovoltaic panel", but the rendered semantic features show it as "vegetation", the cross-entropy loss will increase.

[0047] Visual Loss: Calculates the difference between an “RGB rendered image” and a “real RGB image taken by a drone” (usually using L1 Loss or D-SSIM Loss), forcing Gaussian elements to visually fit the roof appearance.

[0048] Semantic Loss: Calculates the cross-entropy loss between the "2D rendered semantic feature map" and the "2D semantic segmentation map".

[0049] After calculating the loss, the parameters of each Gaussian image are adjusted based on the loss. The automatic differentiation mechanism of frameworks like PyTorch can be used to calculate the total loss. Backpropagation is performed on all parameters. In practice, the gradient descent algorithm updates the positions of Gaussian elements. Rotation Scaling The spherical harmonic coefficients SH are used to move them to the correct position and color, thus adjusting the traditional parameters. The gradient flow updates the semantic logits of each Gaussian unit, adjusting the semantic parameters. After thousands of iterations, the logit value of the Gaussian unit located in the rooftop photovoltaic region for the "photovoltaic panel" category continuously increases, eventually normalizing the probability. The Logit value approaches 1; however, for primitives located in tree areas, the Logit value of the "vegetation" category increases.

[0050] It is also necessary to perform adaptive density control on the current relevant Gaussian images as needed. During training, the system dynamically manages the number and distribution of Gaussian units (splitting, cloning, or deleting), a mechanism that is also indirectly affected by semantic information. The management of Gaussian units includes densification and pruning.

[0051] Densification involves automatically splitting and generating more Gaussian spheres at the edges of photovoltaic panels or glass skylight frames due to complex geometry and large gradient changes. These newly generated spheres inherit the semantic Logits of their parent nodes and are further optimized in subsequent iterations.

[0052] Pruning is defined as: for opacity... Deleting primitives with extremely low (almost transparent) levels.

[0053] After training, each 3D Gaussian element Gi carries not only color and position, but also a normalized semantic category probability pi∈[0,1] (i.e., the probability that the point is photovoltaic or glass). Semantic category probability This will serve as the core weight parameter for subsequent 'optical-geometric difference diagnosis'. It enables discrete Gaussian point clouds to possess semantic awareness, providing a data foundation for subsequent steps to convert semantic information from a Lagrange (point) perspective to an Eulerian (voxel) perspective.

[0054] In some embodiments, constructing a spatial density field based on lidar point cloud data includes: preprocessing the lidar point cloud data and determining the three-dimensional space of the roof based on the preprocessed lidar point cloud data; performing voxel modeling on the three-dimensional space of the roof to construct a voxel mesh adapted to the roof features; counting the number of lidar point clouds inside each voxel and normalizing them to generate spatial density values; and binding the normalized density values ​​to the voxel mesh to generate a spatial density field.

[0055] In applications, preprocessing can include denoising and ground point filtering of the LiDAR point cloud data. Then, based on the preprocessed LiDAR point cloud data, a roof point cloud set can be obtained through height layering, building segmentation, or roof extraction rules. .

[0056] It is understandable that after obtaining the roof point cloud set, the 3D bounding box containing the roof can be divided into sections with a resolution of [resolution value missing]. A voxel mesh (e.g., 10cm) is used to voxelize and model the 3D space of the roof, constructing a voxel mesh that adapts to the roof's geometry and spatial features, providing a unified spatial quantization unit for subsequent point cloud statistics and density calculations. Then, the number of laser point clouds within each voxel is counted and normalized to generate spatial density values. If there are no laser points within the voxel (such as in the glass area), then Standardized spatial density values ​​are generated through normalization, achieving a unified quantitative representation of the density of point cloud distribution in each spatial unit. Finally, the obtained normalized spatial density values ​​are spatially bound to the corresponding voxel mesh to form a spatial density field that can be used for subsequent joint feature calculation and locating geometrically missing regions.

[0057] In some embodiments, the difference confidence map is calculated by combining the feature indices of the three-dimensional Gaussian splatter radiation field and the spatial density field, including: calculating the cumulative opacity value corresponding to the center of each voxel based on the rendering pipeline of the three-dimensional Gaussian splatter radiation field; determining the set of three-dimensional Gaussian primitives corresponding to each voxel in the spatial density field, and calculating the average semantic probability of the Gaussian primitives in each voxel belonging to the target material category; and calculating the difference score of each voxel based on the density distribution, cumulative opacity value and average semantic probability of the spatial density field to generate the difference confidence map.

[0058] To avoid misidentifying gaps in the tree canopy or shaded areas as roof defects, semantic constraints need to be introduced.

[0059] To quantify the differences, we first perform voxelization aggregation of semantic attributes: traversing each voxel in the spatial density field. Retrieve all three-dimensional Gaussian units that fall within the voxel. Read the specific material probability obtained for each primitive in the set in step S2. Calculate the mean value as the region semantic probability of the voxel. :

[0060] in, This represents the number of Gaussian units within the voxel.

[0061] Define semantic enhancement difference score The calculation formula is as follows: Or it can be simplified to multiplication logic: in, This represents the voxel index or spatial coordinate position in three-dimensional space. The present invention divides the three-dimensional space to be reconstructed into a voxel mesh of uniform size (e.g., 10cm side length), and all calculations are performed in units of voxels. (Difference score / Score) indicates the position The confidence level for "the presence of a visual entity but lack of geometric data, and the target is made of a special material" is as follows: the higher the value, the more likely the area is to be a blind spot for lidar detection due to glass or photovoltaic panels. (Visual Accumulated Opacity) represents the visual appearance of this location. The probability of an object's existence is determined based on the trained 3DGS radiation field, using the volume rendering formula. The cumulative opacity corresponding to the voxel center can be calculated. A value close to 1 indicates that the area is visually opaque, while a value close to 0 indicates that the area is visually air. Normalized Geometric Density characterizes the LiDAR at this location. Did you detect a physical echo? Count the number of original laser point clouds contained within the voxel. And perform normalization processing (e.g., divide by a preset density threshold). ), , This indicates that the lidar echo is dense and the geometry is complete. This indicates that the lidar did not receive an echo (i.e., a void area). The geometric vacancy rate is used for inversion logic. When the laser point cloud is missing, this rate approaches 1, thereby activating the repair mechanism. When the laser point cloud is sufficient, this rate approaches 0, suppressing false repairs. (Regional Semantic Probability) represents the location. It belongs to "special materials that need to be repaired" (i.e.) The probability of a photovoltaic panel or glass is the probability of that voxel. All discrete Gaussian elements inside The probability of the material it carries Statistical aggregate values ​​(e.g., average or maximum values); ,like Figure 2 As shown, This indicates that the area has been confirmed as containing photovoltaic panels or glass. This indicates that the area is confirmed to be a background of vegetation, walls, etc. The Missing Sensitivity Factor represents the first moderating factor used to control sensitivity to the degree of geometric missingness. (Semantic Weight Coefficient) represents the second adjustment factor, used to control the weight of semantic information on the final score.

[0062] In this embodiment, the rendering pipeline based on the three-dimensional Gaussian splash radiation field fully utilizes the geometric shape and optical characteristics of Gaussian elements in the radiation field. Through pixel-by-pixel ray projection and energy accumulation calculations, the cumulative opacity value corresponding to the center of each voxel is obtained. This value accurately characterizes the visual presence and filling degree of the three-dimensional Gaussian splash radiation field at the corresponding spatial location. Simultaneously, using the voxel mesh constructed from the spatial density field as a unified spatial reference, the spatial positional correspondence between Gaussian elements and voxels is retrieved to determine the set of three-dimensional Gaussian elements contained within each voxel. Then, combined with the semantic category probability attribute assigned to each Gaussian element, the average semantic probability of each voxel's Gaussian element belonging to the target material category is statistically obtained, achieving precise quantification of the voxel's semantic attributes. Based on this, the density distribution of the spatial density field, the cumulative opacity value of the three-dimensional Gaussian splash radiation field, and the average semantic probability of each voxel are comprehensively called. The degree of difference of each voxel is quantified according to the preset coupled calculation logic to obtain the difference score that can reflect the inconsistency between "visual presence-semantic matching-geometric absence". Then, the difference scores of all voxels are bound to the spatial position of the voxel grid to generate a difference confidence map with full coverage and clear quantification.

[0063] In some embodiments, locating the geometrically missing region corresponding to a planar non-Lambertian region based on the difference confidence map includes: marking voxels with difference scores greater than a preset score threshold as geometrically missing candidate regions based on the difference confidence map; and filtering the geometrically missing candidate regions based on the semantic attributes of the target material to determine the geometrically missing region.

[0064] Based on the above semantic enhancement difference score The calculation formula is defined in this embodiment, which constructs a "three-in-one" logical AND gate: only when "visually visible (...)"... "High" and "cannot be detected by radar" "High" and "believed to be glass" When all three conditions (high) are met, the score is [high]. Only when the threshold is exceeded will subsequent geometric repair steps be triggered. This mathematically guarantees the accuracy of the repair and avoids incorrect filling of natural voids (such as gaps between leaves).

[0065] For example, such as Figure 2 As shown, in case A (glass skylight): there is something visible ( )×Laser without point ( The semantic meaning of )× is glass ( = Score 1 (determined as an area requiring repair).

[0066] Situation B (inside the tree canopy): There is something visible ( )×Laser without point ( The semantic meaning of )× is trees ( The score is 0 (determined to require no repair, to avoid accidental errors). This stems from the low-probability response of Gaussian primitives to non-target materials during the S2 training phase, i.e. The result of aggregation ).

[0067] In this embodiment, the difference confidence map is used as the core basis. The quantified "visual-geometric-semantic" difference scores of each voxel in the map are used to mark voxels with scores greater than a preset score threshold as candidate regions for geometric missing areas. This preset score threshold is set based on the characteristics of non-Lambertian material modeling scenarios and is used to initially distinguish normal areas from geometrically missing areas suspected to be caused by material characteristics, thus initially delineating the range of missing areas. Then, the semantic attributes of the target material (such as glass or photovoltaic panels) are used for a second screening of the above-mentioned candidate regions for geometric missing areas. By judging whether the semantic category probability of Gaussian units within the candidate regions matches the semantic characteristics of the target material, false candidate regions caused by non-material characteristic factors such as tree occlusion, measurement noise, and temporary obstacles are eliminated. Finally, the geometrically missing areas caused solely by the transmission and absorption characteristics of non-Lambertian materials are accurately determined.

[0068] In some embodiments, extracting the lidar point cloud at the edge of the geometrically missing region as a geometric prior, and fitting a virtual completion plane based on the geometric prior, includes: extracting lidar point clouds within a preset neighborhood range outside the boundary of the geometrically missing region as a reference point set; performing plane fitting on the reference point set using a random sampling consensus algorithm to construct a candidate fitting plane; calculating the root mean square error from the reference point set to the candidate fitting plane, and determining the candidate fitting plane as the virtual completion plane when the root mean square error is less than a preset error threshold.

[0069] In the application, the LiDAR point cloud within a preset neighborhood outside the boundary of the geometrically missing region is first extracted as a reference point set. This point cloud can realistically reflect the roof geometry around the missing region, providing a stable and reliable geometric prior for plane fitting. Then, based on the extracted geometric prior, the virtual completion plane is fitted using the RANSAC algorithm. Its mathematical expression follows the point-normal form:

[0070] in, A Virtual Completion Plane (PCP) is an ideal, infinitely large plane inferred from existing geometric data to fill geometric gaps. Physically, this plane represents the geometric surface where a missing skylight or photovoltaic panel would be located. (Plane normal vector) is perpendicular to the plane. The unit direction vector represents the orientation of the roof plane (e.g., tilted 30 degrees to the south). And satisfy The edge point cloud is calculated by principal component analysis (PCA) or least squares fitting using the RANSAC algorithm. (Arbitrary Point) indicates a point located on a plane. Any point in three-dimensional space on the graph, with coordinates denoted as In the subsequent Gaussian transformation step, this variable corresponds to the center point of the Gaussian element that is forcibly projected onto the plane. . (Reference Point on Plane) is a known fixed point on the plane, usually taken as the geometric centroid of all "inliers" selected by the RANSAC algorithm, i.e.:

[0071] in, The laser point cloud data is the edge of the plane that is determined to belong to this plane.

[0072] For ease of engineering implementation, the above formula can be expanded into scalar form (general form): in, Corresponding normal vector The three components ; ,in For reference point The coordinates.

[0073] Regarding the search radius, for example, it can be 0.5m. Setting this threshold ensures that the point cloud used in the fitting is adjacent to the missing area (i.e., the tile at the edge of the window frame or photovoltaic panel), thus guaranteeing that the fitted plane is coplanar with the actual missing surface. If the input data range is too small (e.g., <0.1m), insufficient points will lead to unstable fitting; if the range is too large (e.g., >2m), it may introduce interference points from other roof surfaces.

[0074] After determining the candidate fitting plane, the root mean square error from the reference point set to the candidate fitting plane is calculated to quantitatively evaluate the degree of fit between the candidate fitting plane and the real roof structure. When the root mean square error is less than the preset error threshold, it indicates that the candidate fitting plane meets the geometric accuracy requirements, and it is then determined as a virtual completion plane, providing a standardized geometric constraint benchmark for subsequent parameter correction and morphological reshaping of three-dimensional Gaussian elements.

[0075] In some embodiments, parameter correction and morphological reshaping are performed on the three-dimensional Gaussian primitives in the region to generate a repaired Gaussian point cloud, including: establishing an anisotropic projection constraint mechanism based on a virtual completion plane; and performing position correction and morphological reshaping on the three-dimensional Gaussian primitives in the geometrically missing region according to the anisotropic projection constraint mechanism to generate a repaired Gaussian point cloud.

[0076] In this embodiment, a virtual completion plane is used as the core geometric reference to construct an anisotropic projection constraint mechanism that adapts to the structural features of the roof plan and can orient and standardize 3D Gaussian primitives. This provides a unified geometric constraint basis for the subsequent correction and reshaping of the Gaussian point cloud. Based on the established anisotropic projection constraint mechanism, the 3D Gaussian primitives in the geometrically missing region are individually corrected in position and reshaped in form. This ensures that the spatial distribution and morphological features of the Gaussian primitives conform to the structural rules of the virtual completion plane and maintain continuity with the geometric features of the surrounding normal region. Ultimately, this completes the repair and optimization of the Gaussian point cloud in the geometrically missing region, generating a geometrically accurate and structurally regular repaired Gaussian point cloud.

[0077] In some embodiments, such as Figure 3As shown, the method involves correcting the position and reshaping the shape of three-dimensional Gaussian primitives within the geometrically missing region based on the anisotropic projection constraint mechanism, generating a repaired Gaussian point cloud. This includes: projecting the center coordinates of each three-dimensional Gaussian primitive within the geometrically missing region onto the virtual completion plane along the plane normal vector, updating the center position of each three-dimensional Gaussian primitive within the geometrically missing region; forcibly adjusting the rotation matrix of each three-dimensional Gaussian primitive within the geometrically missing region so that the shortest axis direction of each three-dimensional Gaussian primitive is parallel to the plane normal vector; and setting the scaling factor of each three-dimensional Gaussian primitive within the geometrically missing region in the shortest axis direction to a preset minimum value, causing each three-dimensional Gaussian primitive to degenerate into a two-dimensional Gaussian thin sheet conforming to the virtual completion plane.

[0078] Among them, the anisotropic projection constraint is used to filter out all objects located on the mask. Internal three-dimensional Gaussian elements This involves forcibly correcting its parameters, including position correction, rotation correction, and scale correction.

[0079] Position correction: centering the Gaussian unit Along the normal vector Projected onto a plane Go up and get a new position . .

[0080] Rotation Correction: Rotation Matrix of Gaussian Elements Adjustments are made to make the Z-axis (shortest axis) of its local coordinate system align with the plane normal vector. parallel.

[0081] Scale Correction: Scaling factor of the Gaussian elements in the Z-axis direction to force compression ,make (Minimum value), while keeping the scaling factors of the X and Y axes (in-plane directions) unchanged.

[0082] In this embodiment, by constraining the position, orientation and scale as described above, the Gaussian elements within the mask area are made to have a uniform planar distribution near the virtual completion plane, thereby improving the flatness and structural consistency of the completion area.

[0083] In some embodiments, the repaired Gaussian point cloud is fused with lidar point cloud data to reconstruct a complete three-dimensional roof model, including: filtering out outlier Gaussian elements with opacity below a confidence threshold in the three-dimensional Gaussian splash radiation field; converting the center coordinates of the repaired Gaussian elements into point cloud format while preserving color attributes; filling the holes in the lidar point cloud with the converted point cloud; and generating a watertight triangular mesh model using a Poisson surface reconstruction algorithm or a restricted Deloni triangulation algorithm.

[0084] In the application, the repaired Gaussian elements are converted into point cloud format, and the coordinates of each point are... The color is determined by its spherical harmonic coefficient. The solution value determined from the vertical perspective is the normal vector. The generated Gaussian point cloud is then filled into the original laser point cloud. Within the voids, Poisson Surface Reconstruction or Restricted Delaunay Triangulation algorithms are used to generate the final closed 3D mesh model.

[0085] In this embodiment, outlier Gaussian primitives with opacity below the confidence threshold in the 3D Gaussian splash radiation field are filtered out. This removes invalid and noisy Gaussian primitives from the radiation field, preventing such outliers from interfering with the accuracy of subsequent point cloud transformation and fusion, and ensuring that the Gaussian data used for completion possesses reliable geometric and visual features. Subsequently, the center coordinates of the repaired Gaussian primitives are converted into point cloud format, while retaining their color attributes. This achieves both the conversion of the 3D Gaussian splash radiation field data to a LiDAR point cloud-compatible format and the complete preservation of the visual texture information carried by the Gaussian primitives, ensuring consistency between the completed point cloud and the original LiDAR point cloud in terms of data format and feature dimensions. The converted point cloud is then precisely filled into the voids in the lidar point cloud caused by the non-Lambertian material properties, achieving spatial complementarity and integration of the two types of point clouds. Finally, the Poisson surface reconstruction algorithm or the restricted Deloni triangulation algorithm is used to reconstruct the topology of the integrated complete point cloud. The algorithm advantages are used to generate a topologically continuous, geometrically flaw-free watertight triangular mesh model. This model not only makes up for the geometric deficiencies of the original lidar point cloud, but also has the core topological feature of watertightness required for engineering applications. It can be directly adapted to the needs of actual scenarios such as architectural 3D analysis and lightweight modeling.

[0086] In some embodiments, please refer to Figure 4 The present application provides a lidar point cloud and three-dimensional Gaussian field coupled completion modeling device 40, which includes: an acquisition module 41, a construction module 42, a joint analysis module 43, a correction module 44 and a reconstruction module 45.

[0087] The acquisition module 41 is used to acquire lidar point cloud data and multi-view image data of the target building area.

[0088] The construction module 42 is used to construct a semantically enhanced three-dimensional Gaussian splash radiation field based on multi-view image data and to construct a spatial density field based on lidar point cloud data. Each Gaussian element in the three-dimensional Gaussian splash radiation field is assigned a semantic category probability attribute.

[0089] The joint analysis module 43 is used to calculate the difference confidence map by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field, and to locate the geometric missing region of the corresponding planar non-Lambertian region based on the difference confidence map.

[0090] The correction module 44 is used to extract the lidar point cloud of the edge of the geometrically missing region as a geometric prior, fit a virtual completion plane based on the geometric prior, and perform parameter correction and morphological reshaping on the three-dimensional Gaussian primitives in the region to generate the repaired Gaussian point cloud.

[0091] The reconstruction module 45 is used to fuse the repaired Gaussian point cloud with the lidar point cloud data to reconstruct and generate a watertight triangular mesh model of the corresponding target building area.

[0092] It should be noted that the lidar point cloud and three-dimensional Gaussian field coupling completion modeling device 40 provided in this application embodiment and the lidar point cloud and three-dimensional Gaussian field coupling completion modeling method provided in this application embodiment are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned lidar point cloud and three-dimensional Gaussian field coupling completion modeling method, and the repeated parts will not be described again.

[0093] In some embodiments, an electronic device provided in this application includes a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the above-described method for coupling and completing the modeling of lidar point clouds and three-dimensional Gaussian fields.

[0094] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0095] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0096] This application also provides a non-transitory computer storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0097] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0099] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields, characterized in that, include: Acquire lidar point cloud data and multi-view image data of the target building area; A semantically enhanced three-dimensional Gaussian splash radiation field is constructed based on the multi-view image data, and a spatial density field is constructed based on the lidar point cloud data. Each Gaussian element in the three-dimensional Gaussian splash radiation field is assigned a semantic category probability attribute. The difference confidence map is calculated by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field, and the geometric missing region corresponding to the planar non-Lambertian region is located based on the difference confidence map; The lidar point cloud at the edge of the geometrically missing region is extracted as a geometric prior. A virtual completion plane is fitted based on the geometric prior, and the parameters of the three-dimensional Gaussian primitives in the region are corrected and the shape is reshaped to generate the repaired Gaussian point cloud. The repaired Gaussian point cloud is fused with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area.

2. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 1, characterized in that, The construction of a semantically enhanced three-dimensional Gaussian splash radiation field based on the multi-view image data includes: The multi-view image data is input into a pre-trained semantic segmentation model to perform pixel-by-pixel classification of the multi-view image data, define the target material category, and output the corresponding two-dimensional semantic soft probability map of the target material. Configure geometric and color parameters for the Gaussian elements of the initial three-dimensional Gaussian splash radiation field, and add a semantic log probability parameter; A visual differentiable rendering pipeline and a semantic differentiable rendering pipeline are constructed. An end-to-end iterative training is carried out on the initial three-dimensional Gaussian splash radiation field using a target loss function. During training, adaptive density control of Gaussian units and optimization and update of all parameters are completed simultaneously, and the semantic log probability parameter is activated and transformed into semantic category probability. The target loss function includes a visual loss function and a semantic loss function. After training convergence, the semantically enhanced three-dimensional Gaussian splash radiation field is obtained.

3. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 1, characterized in that, The construction of the spatial density field based on the lidar point cloud data includes: The lidar point cloud data is preprocessed, and the three-dimensional space of the roof is determined based on the preprocessed lidar point cloud data. The three-dimensional space of the roof is modeled in voxel form to construct a voxel mesh adapted to the roof features; The number of LiDAR point clouds within each voxel is counted and normalized to generate spatial density values; The normalized density values ​​are bound to the voxel mesh to generate the spatial density field.

4. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 1, characterized in that, The calculation of the difference confidence map by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field includes: Based on the rendering pipeline of the three-dimensional Gaussian splash radiation field, the cumulative opacity value corresponding to the center of each voxel is calculated. Determine the set of three-dimensional Gaussian elements corresponding to each voxel in the spatial density field, and calculate the average semantic probability of the Gaussian elements in each voxel belonging to the target material category. Based on the density distribution of the spatial density field, the cumulative opacity value, and the average semantic probability, a difference score for each voxel is calculated, and a difference confidence map is generated.

5. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 4, characterized in that, The step of locating the geometrically missing region corresponding to the planar non-Lambertian region based on the difference confidence map includes: Based on the difference confidence map, voxels with difference scores greater than a preset score threshold are marked as geometric missing candidate regions; The geometrically missing candidate regions are filtered based on the semantic properties of the target material to determine the geometrically missing regions.

6. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 1, characterized in that, The step of extracting the lidar point cloud of the geometrically missing region's edge as a geometric prior, and fitting a virtual completion plane based on the geometric prior, includes: Extract the lidar point cloud within a preset neighborhood range outside the boundary of the geometrically missing region as a reference point set; The reference point set is fitted with a random sampling consensus algorithm to construct a candidate fitting plane. Calculate the root mean square error from the reference point set to the candidate fitting plane, and when the root mean square error is less than a preset error threshold, determine the candidate fitting plane as the virtual completion plane.

7. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 1, characterized in that, The process of performing parameter correction and morphological reshaping on the three-dimensional Gaussian primitives within the region to generate a repaired Gaussian point cloud includes: An anisotropic projection constraint mechanism is established based on the virtual completion plane; Based on the anisotropic projection constraint mechanism, the position of the three-dimensional Gaussian primitives in the geometrically missing region is corrected and the shape is reshaped to generate the repaired Gaussian point cloud.

8. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 7, characterized in that, The step of correcting the position and reshaping the shape of the three-dimensional Gaussian elements in the geometrically missing region according to the anisotropic projection constraint mechanism to generate a repaired Gaussian point cloud includes: Project the center coordinates of each 3D Gaussian element within the geometrically missing region onto the virtual completion plane along the plane normal vector of the virtual completion plane, and update the center position of each 3D Gaussian element within the geometrically missing region; The rotation matrix of each three-dimensional Gaussian element within the geometrically missing region is forcibly adjusted so that the shortest axis direction of each three-dimensional Gaussian element is parallel to the plane normal vector; The scaling factor of each three-dimensional Gaussian element in the shortest axis direction within the geometrically missing region is set to a preset minimum value, so that each three-dimensional Gaussian element degenerates into a two-dimensional Gaussian sheet that fits the virtual completion plane.

9. The method for coupled completion modeling of lidar point clouds and three-dimensional Gaussian fields as described in claim 1, characterized in that, The process of fusing the repaired Gaussian point cloud with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area includes: Filter out outlier Gaussian elements with opacity below the confidence threshold in the three-dimensional Gaussian splash radiation field; The repaired Gaussian metacenter coordinates were converted into point cloud format, while preserving color attributes; The converted point cloud is filled into the holes in the lidar point cloud, and the watertight triangular mesh model is generated using the Poisson surface reconstruction algorithm or the restricted Deloni triangulation algorithm.

10. A device for coupled and complete modeling of lidar point clouds and three-dimensional Gaussian fields, characterized in that, include: The acquisition module is used to acquire lidar point cloud data and multi-view image data of the target building area; The construction module is used to construct a semantically enhanced three-dimensional Gaussian splash radiation field based on the multi-view image data, and to construct a spatial density field based on the lidar point cloud data, wherein each Gaussian element in the three-dimensional Gaussian splash radiation field is assigned a semantic category probability attribute. The joint analysis module is used to calculate the difference confidence map by combining the characteristic indices of the three-dimensional Gaussian splash radiation field and the spatial density field, and to locate the geometric missing region of the corresponding planar non-Lambertian region based on the difference confidence map; The correction module is used to extract the lidar point cloud of the edge of the geometrically missing region as a geometric prior, fit a virtual completion plane based on the geometric prior, and perform parameter correction and morphological reshaping on the three-dimensional Gaussian primitives in the region to generate the repaired Gaussian point cloud. The reconstruction module is used to fuse the repaired Gaussian point cloud with the lidar point cloud data to reconstruct a watertight triangular mesh model of the corresponding target building area.

Images