A method for 3D mapping of buildings based on multi-source data fusion

Abstract

This invention relates to the field of 3D spatial mapping technology and discloses a method for 3D mapping of buildings based on multi-source data fusion. The method acquires visible light, 3D laser point cloud, and infrared thermal imaging data. Based on the visible light and infrared data, it identifies highly reflective / transparent materials to generate material masks. It then constructs 3D semantic skeletons containing semantic intersections for the image and laser point cloud, and performs initial registration using the same semantic intersections. The material mask is mapped onto the registered point cloud to remove high-reflectivity abnormal noise points, forming holes. The point clouds of adjacent solid structures are extracted and fitted using building topology rules to generate complete geometric surfaces, which are then merged with the point cloud. Finally, global fine registration is performed to construct a mesh model, and texture repair and mapping are combined with the material mask to output a 3D real-world model. This invention solves the problems of modeling holes and texture distortion caused by materials such as glass curtain walls, significantly improving the geometric and visual accuracy of the mapping model.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional spatial mapping technology, and in particular to a method for three-dimensional mapping of buildings based on multi-source data fusion. Background Technology

[0002] This invention relates to the field of three-dimensional spatial mapping technology. In recent years, the fusion of multi-source data from UAV oblique photogrammetry and 3D laser scanning (LiDAR) has been widely applied to 3D real-world mapping of buildings. However, existing technologies have the following significant drawbacks when dealing with complex urban buildings:

[0003] First, modern buildings widely use highly reflective or transparent materials such as glass curtain walls. When the 3D laser scanning beam encounters such materials, it is very easy to penetrate or undergo multiple specular reflections, resulting in serious data loss or multipath artifact noise in the collected point cloud. At the same time, visible light oblique photography will also produce highlight overexposure or mapping distortion in these areas.

[0004] Secondly, existing heterogeneous point cloud fusion solutions typically apply pure geometric registration algorithms (such as the ICP algorithm). When faced with multimodal data and large areas of missing features (such as holes in glass curtain walls) or repetitive textures on building facades, they are prone to getting stuck in local optima, leading to spatial misalignment.

[0005] Finally, for point cloud holes, traditional surveying software often uses purely mathematical surface interpolation repair algorithms. However, for large areas of missing building planar structures, pure mathematical interpolation is prone to topological distortion, failing to restore the true flatness of the building. This ultimately leads to geometric cracks and texture stretching in the generated 3D model, making it difficult to meet the application requirements of high-precision digital twins. Summary of the Invention

[0006] This invention provides a method for 3D mapping of buildings based on multi-source data fusion, which overcomes the problems of point cloud distortion and multimodal registration misalignment caused by highly reflective / transparent materials, and realizes high-precision, automated, and seamless reconstruction of building reality models in terms of geometric topology and visual texture.

[0007] This invention provides a method for three-dimensional mapping of buildings based on multi-source data fusion, comprising:

[0008] S1. Acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data of the target building. Based on the visible light image data and infrared thermal imaging data, identify the highly reflective areas and / or transparent material areas on the surface of the target building. Define the pixel range of the highly reflective areas and / or transparent material areas on the two-dimensional image as a material mask.

[0009] S2. Based on the visible light image data, perform three-dimensional reconstruction to obtain a three-dimensional point cloud of the image and construct a first three-dimensional semantic skeleton, and construct a second three-dimensional semantic skeleton based on the three-dimensional laser scan point cloud data; wherein, both the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton contain semantic intersections for representing the structural features of the target building;

[0010] S3. Extract the semantic intersections representing the same building structure features from the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton, and define them as homonymous semantic intersections. Perform initial registration of the image three-dimensional point cloud and the three-dimensional laser scan point cloud data based on the initial transformation matrix calculated by the homonymous semantic intersections to obtain coarse registration point cloud data.

[0011] S4. Project the spatial position of the material mask onto the coarse registration point cloud data, determine the three-dimensional region in the coarse registration point cloud data that coincides with the space of the material mask as a high reflection anomaly region, remove the noise in the high reflection anomaly region, so as to form a point cloud hole region in the coarse registration point cloud data.

[0012] S5. Extract the point cloud that is physically adjacent to the hole region of the point cloud from the coarse registration point cloud data and define it as a solid structure point cloud. Based on the preset building topology rules, perform plane equation fitting on the solid structure point cloud to generate a complete geometric surface. Then merge the complete geometric surface with the coarse registration point cloud data after removing noise to obtain the repaired point cloud data.

[0013] S6. Perform global fine registration using the repaired point cloud data, and construct a three-dimensional mesh model of the target building based on the fine registration result. Based on the visible light image data and material mask, determine the texture mapping and texture repair of the three-dimensional mesh model, and output the three-dimensional real scene model of the target building.

[0014] Furthermore, S1 specifically includes:

[0015] S101. Acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data at the same spatial location, and perform spatial registration of the visible light image data and infrared thermal imaging data using the camera intrinsic parameter matrix and relative extrinsic parameter matrix of the visible light camera and the infrared thermal imager.

[0016] S102. Extract the luminance value and high reflectance of the RGB color space from the spatially registered visible light image data as visual features, and extract the absolute temperature value and local temperature gradient distribution of each pixel from the spatially registered infrared thermal imaging data as temperature features. Then, stitch and fuse the visual features and temperature features in the channel dimension to construct a vector-form temperature-visual joint feature.

[0017] S103. Set a brightness threshold and a temperature gradient threshold. When the brightness value in the visual feature is greater than the brightness threshold, and the temperature jump value represented by the local temperature gradient distribution in the temperature feature is greater than the temperature gradient threshold, determine that the corresponding pixel belongs to the highly reflective area and / or the transparent material area; or,

[0018] The temperature-visual joint features are input into a pre-trained multimodal semantic segmentation network. When the classification probability of the corresponding pixel output by the multimodal semantic segmentation network is greater than a preset probability threshold, the corresponding pixel is determined to belong to the high reflectivity area and / or transparent material area.

[0019] S104. The pixels that are determined to belong to the highly reflective area and / or transparent material area are binarized and morphologically smoothed to output a two-dimensional image with smooth boundaries, which is defined as the material mask.

[0020] Furthermore, S2 specifically includes:

[0021] S201. Perform feature calculation and dense matching on the visible light image data to generate a dense three-dimensional point cloud with color attributes as the image three-dimensional point cloud, and obtain the spatial projection matrix corresponding to each visible light image data.

[0022] S202. Extract the two-dimensional semantic contour and two-dimensional feature points from the visible light image data, and reproject them onto the spatial location of the three-dimensional point cloud of the image using the spatial projection matrix to construct the first three-dimensional semantic skeleton.

[0023] S203. Perform plane extraction and equation fitting on the three-dimensional laser scanning point cloud data, calculate the three-dimensional structure edge lines and their intersection points in three-dimensional space, and construct the second three-dimensional semantic skeleton.

[0024] S204. Classify the intersection points in the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton according to the structural attributes, and define the three-dimensional coordinate points with building structure attribute labels as semantic intersection points.

[0025] Furthermore, S3 specifically includes:

[0026] S301. Construct the first three-dimensional semantic skeleton into a first topological graph, construct the second three-dimensional semantic skeleton into a second topological graph, traverse the nodes in the first and second topological graphs to perform structural attribute matching and geometric consistency verification, remove mismatched point pairs, and define the remaining high-confidence matching point pairs as a set of semantic intersection points with the same name.

[0027] S302. Divide the set of semantic intersection points with the same name into a source point set and a target point set and construct a spatial transformation objective function. By decentralizing the point set and calculating the covariance matrix, the singular value decomposition algorithm is used to solve the three-dimensional rotation matrix and translation vector, and then construct the initial transformation matrix.

[0028] S303. Set the coordinate system of the three-dimensional laser scanning point cloud data as the global reference coordinate system, use the initial transformation matrix to perform spatial mapping transformation on the image three-dimensional point cloud, and superimpose the transformed image three-dimensional point cloud with the three-dimensional laser scanning point cloud data to obtain the coarse registration point cloud data.

[0029] S304. Calculate the registration error of the set of semantic intersection points with the same name after spatial mapping transformation. If the registration error meets the preset convergence condition, output the coarse registration point cloud data. If the convergence condition is not met, adjust the interior point distance threshold and return to step S301 for iterative calculation.

[0030] Furthermore, S302 specifically includes:

[0031] Let P be the source point set and Q be the target point set, and construct the spatial transformation objective function based on the least squares method. Where N is the number of logarithms of the intersection points in the set of semantic intersection points with the same name, R is the three-dimensional rotation matrix, and T is the translation vector;

[0032] Calculate the centroid coordinates of the source point set P and the target point set Q respectively, and then perform a decentering process on the two sets of points.

[0033] Calculate the covariance matrix of the decentralized point set, perform singular value decomposition on the covariance matrix, solve for the optimal three-dimensional rotation matrix R based on the orthogonal matrix obtained from the decomposition, and calculate the translation vector T in combination with the centroid coordinates. The homogeneous transformation matrix formed by the three-dimensional rotation matrix R and the translation vector T is used as the initial transformation matrix.

[0034] Furthermore, S4 specifically includes:

[0035] S401. Obtain the camera intrinsic parameter matrix and camera extrinsic parameter matrix when acquiring the visible light image data, and calculate the projection coordinates of each three-dimensional coordinate point in the coarse alignment point cloud data on the two-dimensional image plane based on the camera frustum model.

[0036] S402. Using the projection coordinates as an index, query the binarized value of the corresponding pixel position in the material mask. When the binarized value of the projection coordinates in the material mask is a preset foreground value representing a highly reflective area and / or a transparent material area, and the depth value of the corresponding three-dimensional coordinate point is within a preset imaging frustum depth range, it is determined that the coincidence condition is met. All three-dimensional coordinate points in the coarse registration point cloud data that meet the coincidence condition are marked and delineated as the high reflectivity anomaly area.

[0037] S403. Remove all point cloud data within the high-reflectivity anomaly region to eliminate multipath artifacts and spatial noise caused by the physical properties of the material.

[0038] S404. Using a point cloud edge extraction algorithm, scan the surface topology of the remaining coarsely registered point cloud data after noise removal, and define the blank space inside the extracted non-closed three-dimensional edge lines as the point cloud cavity region.

[0039] Furthermore, in step S401, the projected coordinates are calculated using the inverse projection mathematical mapping formula, which is as follows:

[0040]

[0041] in, The spatial coordinates of the three-dimensional coordinate points in the coarse registration point cloud data. The projected coordinates of the three-dimensional coordinate point on the two-dimensional image plane are... Let K be the depth value of the 3D coordinate point in the camera coordinate system, and K be the camera intrinsic parameter matrix. Let be the camera extrinsic matrix.

[0042] Furthermore, S5 specifically includes:

[0043] S501. Obtain the unclosed three-dimensional edge line of the point cloud hole region, construct a spatial index in the coarse registration point cloud data after removing noise, and perform a nearest neighbor search based on the three-dimensional edge line to extract the high-density point cloud within the preset structure buffer radius and define it as the entity structure point cloud.

[0044] S502. Based on the building topology rule that the physical structures physically adjacent to the high-reflectivity areas are coplanar or parallel, a random sampling consensus algorithm or the overall least squares method is used to perform plane equation fitting on the point cloud of the physical structures, and the dominant plane equation representing the real spatial geometry of the missing high-reflectivity areas is calculated and extracted. Where A, B, and C are the components of the normal vector of the fitted dominant plane in the three-dimensional rectangular coordinate system, and D is the intercept parameter of the dominant plane.

[0045] S503. Project the unclosed three-dimensional edge line onto a two-dimensional plane according to the normal vector of the dominant plane equation to generate a closed projection contour. Generate two-dimensional regular grid points inside the closed projection contour. Use the dominant plane equation to perform inverse coordinate calculation to elevate the two-dimensional regular grid points to three-dimensional space, generate a patch point cloud and define it as the completed geometric surface.

[0046] S504. Spatially merge the completed geometric surface with the coarse registration point cloud data after noise removal, and apply a filter to the stitching area of ​​the three-dimensional edge line of the merged point cloud for local density equalization processing, and output the repaired point cloud data with complete structure.

[0047] Furthermore, S6 specifically includes:

[0048] S601. Based on the repaired point cloud data after completing the geometric surface, the image 3D point cloud and the 3D laser scan point cloud data are globally finely registered using the iterative nearest point algorithm or the normal distribution transformation algorithm.

[0049] S602. Extract the point cloud normal vector features after global fine registration, and use the surface reconstruction algorithm to encapsulate the discrete point cloud into a continuous triangular patch network to construct a three-dimensional mesh model of the target building.

[0050] S603. Using the material mask as the Alpha guiding channel, an image restoration algorithm is used in conjunction with the surrounding texture context information to reconstruct the highly reflective pixels in the visible light image data to generate the restored texture, and the original visible light image not covered by the material mask is retained as the visible light texture.

[0051] S604. Perform UV coordinate unpacking and parametric mapping on the three-dimensional mesh model, perform partition control based on the material mask, and map the repair texture and the visible light texture to the spatial regions corresponding to the three-dimensional mesh model respectively, and output the three-dimensional real scene model of the target building.

[0052] The present invention also provides a three-dimensional mapping device for buildings based on multi-source data fusion, and based on the three-dimensional mapping method for buildings based on multi-source data fusion as described above, the device includes:

[0053] The acquisition module is used to acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data of the target building, and to identify highly reflective areas and / or transparent material areas on the surface of the target building based on the visible light image data and infrared thermal imaging data, and to define the pixel range of the highly reflective areas and / or transparent material areas on the two-dimensional image as a material mask.

[0054] The construction module is used to perform three-dimensional reconstruction based on the visible light image data to obtain a three-dimensional point cloud of the image and construct a first three-dimensional semantic skeleton, and to construct a second three-dimensional semantic skeleton based on the three-dimensional laser scan point cloud data; wherein, both the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton contain semantic intersections for representing the structural features of the target building.

[0055] The registration module is used to extract semantic intersections representing the same building structure features in the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton, define them as homonymous semantic intersections, and perform initial registration of the image three-dimensional point cloud and the three-dimensional laser scan point cloud data based on the initial transformation matrix calculated by the homonymous semantic intersections to obtain coarse registration point cloud data.

[0056] The projection module is used to project and map the spatial position of the material mask onto the coarse registration point cloud data, identify the three-dimensional region in the coarse registration point cloud data that coincides with the space of the material mask as a high-reflection anomalous region, and remove noise points in the high-reflection anomalous region to form a point cloud hole region in the coarse registration point cloud data.

[0057] The merging module is used to extract the point clouds that are physically adjacent to the hole regions of the point cloud in the coarse registration point cloud data and define them as entity structure point clouds. Based on the preset building topology rules, the entity structure point cloud is fitted with a plane equation to generate a complete geometric surface. The complete geometric surface is then merged with the coarse registration point cloud data after noise removal to obtain the repaired point cloud data.

[0058] The repair module is used to perform global fine registration using the repaired point cloud data, construct a three-dimensional mesh model of the target building based on the fine registration result, determine texture mapping and texture repair of the three-dimensional mesh model based on the visible light image data and material mask, and output a three-dimensional real-scene model of the target building.

[0059] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0060] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0061] The beneficial effects of this invention are as follows:

[0062] This invention overcomes the limitations of traditional multi-source data fusion in the modeling of complex facades of modern buildings. By introducing combined infrared and visible light features to accurately identify and generate material masks, it actively eliminates noise in high-reflectivity abnormal areas and completely eliminates multipath interference caused by material physical properties. It uses the first and second three-dimensional semantic skeletons extracted across modalities and the intersection points of the same semantic names as registration anchors to achieve high-precision dimensionality reduction and initial registration of heterogeneous point clouds, effectively avoiding the misalignment problem caused by traditional geometric blind stitching. At the same time, combined with building topology rules, it uses the point clouds of physically adjacent solid structures with point cloud holes to perform planar fitting to generate complete geometric surfaces. With the help of partitioned texture repair and mapping technology based on material masks, it not only achieves high-precision flat repair of large-area holes, but also eliminates visual specular distortion, improves the automation level of urban-level real-scene 3D mapping and the 3D spatial fidelity of the final output model. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a method flow according to an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the device structure according to an embodiment of the present invention.

[0065] Figure 3 This is a schematic diagram of the internal structure of a computer device according to an embodiment of the present invention.

[0066] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0067] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0068] like Figure 1 As shown, this invention provides a method for three-dimensional mapping of buildings based on multi-source data fusion, including:

[0069] S1. Acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data of the target building. Based on the visible light image data and infrared thermal imaging data, identify highly reflective areas and / or transparent material areas on the surface of the target building. Define the pixel range of the highly reflective areas and / or transparent material areas on the two-dimensional image as a material mask.

[0070] In a preferred embodiment of the present invention, step S1 specifically includes the following sub-steps:

[0071] S101 employs a drone equipped with a dual-light pod (containing a visible light camera and an infrared thermal imager) and an airborne lidar scanner to simultaneously collect data from the target building. During the data collection process, a hardware clock synchronization mechanism records timestamps to obtain visible light image data, infrared thermal image data, and 3D laser scan point cloud data from the same spatial location.

[0072] Because of the differences in field of view (FOV) and resolution between visible light cameras and infrared thermal imagers, it is necessary to obtain the intrinsic parameter matrices of both cameras in advance. Using visible light image data as the reference image and infrared thermal image data as the image to be registered, based on the calibrated intrinsic parameter matrix and relative extrinsic parameter matrix, an affine transformation is performed through perspective transformation or homography matrix to achieve pixel-by-pixel precise alignment between the visible light image and the infrared thermal image in the pixel coordinate system.

[0073] S102. For the registered and aligned 2D image, extract the visual features and temperature features of the target building surface. Specifically, extract the brightness value and high reflectance of the RGB color space from the visible light image data as visual features; extract the absolute temperature value and local temperature gradient distribution of each pixel from the infrared thermal imaging data as temperature features. Concatenate the visual features and temperature features along the channel dimension to construct a temperature-visual joint feature vector.

[0074] S103. Based on the extracted temperature-visual joint features, classify the material of the target building surface. In actual physical scenarios, highly reflective / transparent materials such as glass curtain walls usually appear as locally overexposed and exhibit strong specular highlights under visible light; at the same time, because the specific heat capacity of glass is different from that of the surrounding cement walls, and glass usually reflects the cold / hot radiation from the sky or the sun, it shows a significant temperature jump (temperature gradient distortion) compared to the surrounding solid walls in infrared thermal imaging.

[0075] In practice, a dual determination can be made by setting a brightness threshold and a temperature gradient threshold, i.e., pre-setting a brightness threshold and a temperature gradient threshold. For each pixel after spatial registration, the data in its temperature-visual joint feature vector is read. When the brightness value of the visual feature in the vector is greater than the brightness threshold (indicating that the area has strong reflection or highlights under visible light), and the temperature jump value represented by the local temperature gradient distribution of the temperature feature in the vector is greater than the temperature gradient threshold (indicating that the thermal radiation characteristics of the area have a significant physical abrupt change from the surrounding conventional walls, consistent with the thermal properties of transparent materials such as glass), an AND logic judgment is performed to determine that the corresponding pixel belongs to the highly reflective area and / or transparent material area of ​​the target building surface.

[0076] Alternatively, the constructed temperature-visual joint feature vector can be input into a pre-trained multimodal semantic segmentation neural network (e.g., an improved network based on the U-Net architecture with an attention fusion mechanism). This network model extracts deep fusion features through multi-layer convolution and pooling operations, ultimately outputting a probability heatmap with the same resolution as the original image. The heatmap is read pixel by pixel. When the classification probability of a corresponding pixel output by the multimodal semantic segmentation network is greater than a preset probability threshold (e.g., 0.85), the corresponding pixel is determined to belong to a highly reflective area and / or a transparent material area.

[0077] S104. Pixels identified in step S103 belonging to highly reflective areas and / or transparent material areas are assigned a value of 1, while pixels in other solid wall and background areas are assigned a value of 0, generating an initial two-dimensional binary mask. Further, mathematical morphological filtering (such as opening and closing operations) is applied to the initial binary mask to eliminate discrete noise and fill in tiny holes within the regions, ultimately outputting a material mask with smooth boundaries. The pixel coordinate range of this material mask precisely defines the "abnormal target areas" that need to be removed or repaired in subsequent point cloud processing.

[0078] S2. Based on the visible light image data, perform three-dimensional reconstruction to obtain a three-dimensional point cloud of the image and construct a first three-dimensional semantic skeleton, and construct a second three-dimensional semantic skeleton based on the three-dimensional laser scan point cloud data; wherein, both the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton contain semantic intersections for representing the structural features of the target building.

[0079] In a preferred embodiment of the present invention, step S2 specifically includes the following sub-steps:

[0080] S201. The acquired multi-view visible light image data is input into the Structure from Motion (SfM) algorithm model. SIFT or SURF feature points between each image are extracted for feature matching to calculate the camera's interior and exterior orientation elements and sparse point cloud. Subsequently, a multi-view stereo (MVS) algorithm is used for dense matching to generate a dense 3D point cloud with RGB color attributes, which is defined as the image 3D point cloud. At the same time, the spatial projection matrix corresponding to each visible light image calculated by the SfM algorithm is saved.

[0081] S202. Using a pre-trained two-dimensional instance segmentation neural network (e.g., Mask R-CNN network), the visible light image data is processed frame by frame to extract two-dimensional semantic contours representing the edges of the target building structure (specifically including roof edge lines, corner lines, door and window outer frame lines, etc.); further, the intersection points of the two-dimensional semantic contours are extracted as two-dimensional feature points.

[0082] Using the spatial projection matrix saved in step S201, and based on the collinearity equation, the extracted two-dimensional semantic contours and two-dimensional feature points are reprojected along the ray direction onto the corresponding spatial positions of the image's three-dimensional point cloud. This forms a network structure in the image's three-dimensional point cloud, consisting of three-dimensional line segments with semantic labels and their intersections, which is defined as the first three-dimensional semantic skeleton.

[0083] S203. For the three-dimensional laser scanning point cloud data, since it has high-precision three-dimensional spatial geometric coordinates, a three-dimensional point cloud semantic segmentation network (such as PointNet++ or RandLA-Net) or a region growing algorithm is used to directly segment a set of planar point clouds representing the facade and roof of the target building in three-dimensional space.

[0084] The RANSAC (Random Sample Consensus) algorithm is used to fit planar equations to adjacent planar point cloud sets, and the intersection lines of adjacent planes are calculated as the edge lines of the 3D structure. The intersection points of multiple 3D structure edge lines in 3D space are extracted. The spatial topology formed by these 3D structure edge lines and their intersection points is defined as the second 3D semantic skeleton.

[0085] S204. After the first and second 3D semantic skeletons are constructed, the intersection points are assigned attribute weights and normalized. Specifically, based on the semantic edge type of the intersection point (such as the intersection of two window frame lines, or the intersection of the roof line and the corner line), a structural attribute label is assigned to each intersection point. These 3D coordinate points with clear architectural physical structural meaning (such as the upper left corner of a window, or the corner of the eaves) are uniformly defined as the semantic intersection points to provide high-confidence anchor points for subsequent cross-modal initial registration.

[0086] S3. Extract the semantic intersections representing the same building structure features from the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton, and define them as semantic intersections with the same name. Perform initial registration of the image three-dimensional point cloud and the three-dimensional laser scanning point cloud data based on the initial transformation matrix calculated by the semantic intersections with the same name to obtain coarse registration point cloud data.

[0087] In a preferred embodiment of the present invention, step S3 specifically includes the following sub-steps:

[0088] S301. Since the first three-dimensional semantic skeleton (derived from visible light images) and the second three-dimensional semantic skeleton (derived from laser point clouds) are in different initial coordinate systems, the semantic intersection points with structural attribute labels are extracted first.

[0089] In practice, the first 3D semantic skeleton is constructed as the first topological graph, and the second 3D semantic skeleton is constructed as the second topological graph. Nodes represent semantic intersections, and edges represent the Euclidean distance or spatial angle between intersections. The nodes in both topological graphs are traversed. First, hard attribute matching is performed (e.g., only matching between "window top left corner point" and "window top left corner point" is allowed). Based on attribute matching, a subgraph matching algorithm or a random sampling consensus algorithm (RANSAC) is further used for geometric consistency verification. That is, it checks whether the matched point pairs satisfy the distance invariant property of rigid body transformation, eliminates mismatched point pairs caused by occlusion or segmentation errors, and finally retains high-confidence matching point pairs, which are defined as the set of semantic intersections with the same name.

[0090] S302. After obtaining the set of semantic intersection points with the same name, divide it into two point sets: the source point set P (intersection coordinates from the first 3D semantic skeleton) and the target point set Q (intersection coordinates from the second 3D semantic skeleton), where... and This is a pair of semantic intersections with the same name.

[0091] Construct a space transformation objective function based on the least squares method:

[0092]

[0093] Where N is the logarithm of the semantic intersections with the same name, R is the three-dimensional rotation matrix, and T is the three-dimensional translation vector.

[0094] Subsequently, the centroid coordinates of point sets P and Q are calculated separately, and the two point sets are then decentered. The covariance matrix of the decentered point sets is calculated, and singular value decomposition (SVD) is performed on the covariance matrix. The optimal rotation matrix R is calculated based on the orthogonal matrix obtained from the SVD decomposition, and the translation vector T is calculated in combination with the centroid coordinates. The homogeneous transformation matrix formed by R and T is the initial transformation matrix.

[0095] S303. Set the coordinate system of the 3D laser scanning point cloud data as the global reference coordinate system. Using the initial transformation matrix calculated in step S302, perform multiplication matrix operations (i.e., rotation and translation transformations) on each 3D coordinate point in the image 3D point cloud. Through this coordinate system transformation, the image 3D point cloud is dragged and mapped to the global reference coordinate system of the 3D laser scanning point cloud. At this point, the two heterogeneous point clouds achieve consistency in macroscopic spatial posture. The image 3D point cloud processed by the initial transformation matrix is ​​then superimposed with the 3D laser scanning point cloud data to output coarsely registered point cloud data.

[0096] S304. After generating the coarse registration point cloud data, calculate the root mean square error (RMSE) of the corresponding semantic intersection points after the initial transformation matrix mapping. If the RMSE value is less than the preset coarse registration error threshold, the coarse registration point cloud data is directly output to step S4; if it is greater than the error threshold, it indicates that the matching point set still contains outlier noise. The interior distance threshold of the RANSAC algorithm is reduced, and the process returns to step S301 to re-perform the corresponding semantic intersection point matching and matrix calculation until the error converges.

[0097] S4. Project the spatial position of the material mask onto the coarse registration point cloud data, determine the three-dimensional region in the coarse registration point cloud data that coincides with the space of the material mask as a high reflection anomaly region, and remove the noise in the high reflection anomaly region to form a point cloud hole region in the coarse registration point cloud data.

[0098] In a preferred embodiment of the present invention, step S4 specifically includes the following sub-steps:

[0099] S401. Extract the material mask generated in step S1 (which is essentially a two-dimensional binary image), and obtain the camera intrinsic parameter matrix K and the corresponding camera extrinsic parameter matrix when acquiring the corresponding visible light image. (The camera extrinsic parameters here can be obtained by combining the spatial projection matrix in step S201 with the initial transformation matrix in step S302).

[0100] Traverse each three-dimensional coordinate point in the coarse-calibrated point cloud data Based on the pinhole camera imaging model, calculate the projected coordinates of the three-dimensional point on the two-dimensional image plane. Its inverse projection mathematical mapping model satisfies the following formula:

[0101]

[0102] in, This represents the depth value of the 3D point in the camera coordinate system.

[0103] S402, after obtaining the two-dimensional projected coordinates corresponding to each three-dimensional point. Then, using these coordinates as an index, the binarized value of the corresponding pixel position in the material mask matrix is ​​retrieved. If the coordinates in the material mask are... The pixel value at that location is the preset foreground value (i.e., the mask value is 1, representing a highly reflective / transparent material area), and the depth value of that 3D point... If the point is located within a reasonable imaging cone depth range, then the three-dimensional coordinates are determined. It falls within the spatial ray cone containing the highly reflective / transparent material. All three-dimensional coordinate points in the coarse registration point cloud data that satisfy the projection mapping coincidence condition are uniformly marked and delineated as the high-reflectivity anomalous region.

[0104] S403. In physical reality, due to the specular reflection (resulting in no echo) or multipath refraction (passing through the glass and hitting indoor objects before returning) of the laser pulses emitted by the 3D laser scanner (LiDAR) by highly reflective materials such as glass curtain walls, the point clouds collected by LiDAR in these areas often appear as "multipath artifact points" or indoor structural points with severely distorted depth information.

[0105] All point cloud data marked as belonging to the high-reflectivity anomalous region are forcibly removed and excluded from subsequent geometric surface reconstruction. By stripping away these erroneous topological data caused by the physical properties of the materials, interference sources in the subsequent fusion process are significantly eliminated.

[0106] S404. After spatial noise removal, the coarse registration point cloud data forms physically missing blank areas in regions that originally belonged to highly reflective / transparent materials. Using point cloud edge extraction algorithms (such as those based on the angle between normal vectors or the AlphaShape algorithm), the surface topology of the current coarse registration point cloud data is scanned. The blank spaces inside the unclosed 3D edge lines generated by the removal operation are clearly defined as the point cloud void regions, providing a precise target boundary range for the next step of adaptive geometric completion.

[0107] S5. Points physically adjacent to the hole regions in the coarse registration point cloud data are defined as entity structure point clouds. Plane equation fitting is performed on the entity structure point clouds based on preset building topology rules to generate complete geometric surfaces. The complete geometric surfaces are then merged with the coarse registration point cloud data after noise removal to obtain repaired point cloud data.

[0108] In a preferred embodiment of the present invention, step S5 specifically includes the following sub-steps:

[0109] S501. Obtain the point cloud cavity region and its unclosed three-dimensional edge line defined in step S4, and construct a KD-Tree (K-dimensional tree) or Octtree spatial index in the coarsely registered point cloud data after noise removal.

[0110] Centered on each point on the three-dimensional edge line, and within a preset structural buffer radius (e.g., 0.5 meters to 1.0 meter), a nearest neighbor search is performed using the spatial index. The area covered by this buffer radius is typically a robust physical structure (such as a window frame, curtain wall metal frame, or adjacent solid wall) supporting highly reflective materials (such as glass). The high-density point cloud within this search range that retains real physical echoes is uniformly extracted and defined as the solid structure point cloud.

[0111] S502. In the topological rules of modern architecture, highly reflective / transparent materials (such as glass windows and glass curtain walls) are usually exhibited as rigid planar geometries, and the physical structures (window frames or curtain wall frames) physically adjacent to their edges are coplanar or have a parallel relationship with a fixed depth.

[0112] Based on the aforementioned architectural topology prior rules, instead of using mathematical surface interpolation, robust planar estimation is performed on the extracted entity structure point cloud. Specifically, the RANSAC (Random Sample Consensus) algorithm or the total least squares (TLS) method is used to calculate and fit the dominant plane equation for the local region:

[0113]

[0114] Where A, B, and C are the normal vector components of the fitted plane, and D is the intercept parameter of the plane. By excluding outliers, this equation accurately characterizes the true three-dimensional geometric plane where the missing highly reflective area should have been located.

[0115] S503. After obtaining the equation of the dominant plane, the unclosed three-dimensional edge lines extracted in step S4 are projected onto the fitted two-dimensional plane along the normal vector direction of the dominant plane to form a closed two-dimensional projection profile.

[0116] Within the two-dimensional projected contour, uniformly distributed two-dimensional regular grid points are generated according to a point spacing resolution matching the original solid structure point cloud. Subsequently, the dominant plane equation is used... Inverse coordinate calculation is performed to re-enlarge the coordinates of these two-dimensional regular grid points to three-dimensional space, thereby generating a uniform and flat patch of point cloud "out of thin air" at the original locations of the point cloud voids. This part of the three-dimensional data generated by the algorithm (procedural generation) is defined as the completed geometric surface.

[0117] S504. Spatially merge the completed geometric surface with the noise-removed coarse registration point cloud data. To eliminate potential overlap or density inconsistencies at the interface, a voxel grid downsampling filter is applied to the stitching region of the 3D edge line of the merged point cloud. This filter redistributes the point cloud in the stitching region while maintaining the macroscopic architectural geometry, ensuring a smooth transition in spatial point cloud density between the procedurally generated completed region and the actual physical scanned region. The final output is a repaired point cloud data with a complete structure and no topological breaks.

[0118] S6. Perform global fine registration using the repaired point cloud data, and construct a three-dimensional mesh model of the target building based on the fine registration result. Based on the visible light image data and material mask, determine the texture mapping and texture repair of the three-dimensional mesh model, and output the three-dimensional real scene model of the target building.

[0119] In a preferred embodiment of the present invention, step S6 specifically includes the following sub-steps:

[0120] S601. The point cloud data repaired in step S5 (i.e., the fused point cloud with complete geometric surfaces) now possesses a complete building physical topology. At this point, the iterative nearest point algorithm (ICP) or the normal distribution transformation algorithm (NDT) is used to perform global high-precision registration between the image 3D point cloud and the 3D laser scan point cloud.

[0121] Since high-reflectivity multipath artifacts have been removed in steps S4 and S5, and smooth geometric surfaces have been filled in the holes, the global fine registration at this time effectively overcomes the defect of the traditional ICP algorithm in the glass curtain wall area that gets stuck in a local optimum due to missing or distorted point cloud, and the convergence speed and registration accuracy are improved by orders of magnitude.

[0122] S602. After completing global fine registration, extract the normal vector features of the point cloud, and use the Poisson Surface Reconstruction algorithm or the Delaunay Triangulation algorithm to encapsulate the discrete repair point cloud data into a continuous, closed triangular patch network to construct the three-dimensional mesh model of the target building. At this time, the output three-dimensional mesh model is a "white model" without color attributes.

[0123] S603. Before performing texture mapping, an image restoration mechanism is introduced to address visual distortion issues such as overexposure of highlights and specular reflection (such as reflecting the sky or opposite buildings) in visible light image data on materials such as glass curtain walls.

[0124] Specifically, the material mask generated in step S1 is retrieved and used as the alpha guide channel of the image inpainting algorithm (i.e., marking the contaminated areas that need to be repaired). An exemplar-based inpainting algorithm or a pre-trained deep generative adversarial network (GAN) is employed, combined with the texture context information of normal window frames or non-reflective walls surrounding the contaminated areas, to perform content prediction and pixel reconstruction on the highly reflective pixel blocks in the visible light image data. This generates a repaired texture with strong specular reflection removed and a smooth, uniform visual appearance, while the original visible light image not covered by the material mask is preserved as the visible light texture.

[0125] S604. After acquiring the 3D mesh model and two textures, UV coordinate unwrapping and parametric mapping are performed. By calculating the spatial perspective relationship between each triangular facet on the 3D mesh model and the camera viewpoint, the 2D texture coordinates are mapped to 3D space.

[0126] During the mapping process, “zoning control” is performed based on the material mask: for the area in the three-dimensional mesh model whose spatial location corresponds to the area covered by the material mask (i.e. the physical area of ​​the repaired glass curtain wall), the repair texture generated in S603 is extracted and mapped in that area; for the remaining conventional wall structures in the three-dimensional mesh model, the visible light texture containing rich realistic details is mapped.

[0127] Through the multi-channel partitioned texture mapping described above, the geometric and visual gaps caused by the fusion of multi-source data are finally eliminated, and the target building 3D real scene model (such as OSGB, OBJ or 3DTiles format file) with high-precision geometric topology and high-fidelity visual texture is output.

[0128] like Figure 2 As shown, this invention provides a three-dimensional mapping device for buildings based on multi-source data fusion, and based on the three-dimensional mapping method for buildings based on multi-source data fusion as described above. The device includes:

[0129] The acquisition module 1 is used to acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data of the target building, and to identify highly reflective areas and / or transparent material areas on the surface of the target building based on the visible light image data and infrared thermal imaging data, and to define the pixel range of the highly reflective areas and / or transparent material areas on the two-dimensional image as a material mask.

[0130] Construction module 2 is used to perform three-dimensional reconstruction based on the visible light image data to obtain a three-dimensional point cloud of the image and construct a first three-dimensional semantic skeleton, and to construct a second three-dimensional semantic skeleton based on the three-dimensional laser scan point cloud data; wherein, both the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton contain semantic intersections for representing the structural features of the target building;

[0131] The registration module 3 is used to extract semantic intersections representing the same building structure features in the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton, define them as homonymous semantic intersections, and perform initial registration of the image three-dimensional point cloud and the three-dimensional laser scan point cloud data based on the initial transformation matrix calculated by the homonymous semantic intersections to obtain coarse registration point cloud data.

[0132] Projection module 4 is used to project and map the spatial position of the material mask onto the coarse registration point cloud data, determine the three-dimensional region in the coarse registration point cloud data that coincides with the space of the material mask as a high reflection anomaly region, and remove noise in the high reflection anomaly region to form a point cloud hole region in the coarse registration point cloud data.

[0133] The merging module 5 is used to extract the point cloud that is physically adjacent to the hole area of ​​the point cloud in the coarse registration point cloud data and define it as a solid structure point cloud. Based on the preset building topology rules, the solid structure point cloud is fitted with a plane equation to generate a complete geometric surface. The complete geometric surface is then merged with the coarse registration point cloud data after noise removal to obtain the repaired point cloud data.

[0134] Repair module 6 is used to perform global fine registration using the repaired point cloud data, construct a three-dimensional mesh model of the target building based on the fine registration result, determine texture mapping and texture repair of the three-dimensional mesh model based on the visible light image data and material mask, and output a three-dimensional real scene model of the target building.

[0135] Each of the above modules is used to perform the respective steps in the above-mentioned three-dimensional surveying method for buildings based on multi-source data fusion. The specific implementation method is as described in the above-mentioned method embodiment, and will not be repeated here.

[0136] like Figure 3 As shown, the present invention also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores all the data required for the process of a multi-source data fusion-based 3D building mapping method. The network interface is used for communication with external terminals via a network connection. The computer program is executed by the processor to implement the multi-source data fusion-based 3D building mapping method.

[0137] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0138] An embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described methods for three-dimensional mapping of buildings based on multi-source data fusion.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. Any references to memory, storage, databases, or other media provided in this application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), such as dynamic RAM (used as main storage) or static RAM (commonly used as cache memory). By way of illustration and not limitation, RAM has various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and Rambus DRAM (RDRAM).

[0140] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0141] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for three-dimensional mapping of buildings based on multi-source data fusion, characterized in that, include: S1. Acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data of the target building. Based on the visible light image data and infrared thermal imaging data, identify the highly reflective areas and / or transparent material areas on the surface of the target building. Define the pixel range of the highly reflective areas and / or transparent material areas on the two-dimensional image as a material mask. S2. Based on the visible light image data, perform three-dimensional reconstruction to obtain a three-dimensional point cloud of the image and construct a first three-dimensional semantic skeleton, and construct a second three-dimensional semantic skeleton based on the three-dimensional laser scan point cloud data; wherein, both the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton contain semantic intersections for representing the structural features of the target building; S3. Extract the semantic intersections representing the same building structure features from the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton, and define them as homonymous semantic intersections. Perform initial registration of the image three-dimensional point cloud and the three-dimensional laser scan point cloud data based on the initial transformation matrix calculated by the homonymous semantic intersections to obtain coarse registration point cloud data. S4. Project the spatial position of the material mask onto the coarse registration point cloud data, determine the three-dimensional region in the coarse registration point cloud data that coincides with the space of the material mask as a high reflection anomaly region, remove the noise in the high reflection anomaly region, so as to form a point cloud hole region in the coarse registration point cloud data. S5. Extract the point cloud that is physically adjacent to the hole region of the point cloud from the coarse registration point cloud data and define it as a solid structure point cloud. Based on the preset building topology rules, perform plane equation fitting on the solid structure point cloud to generate a complete geometric surface. Then merge the complete geometric surface with the coarse registration point cloud data after removing noise to obtain the repaired point cloud data. S6. Perform global fine registration using the repaired point cloud data, and construct a three-dimensional mesh model of the target building based on the fine registration result. Based on the visible light image data and material mask, determine the texture mapping and texture repair of the three-dimensional mesh model, and output the three-dimensional real scene model of the target building.

2. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 1, characterized in that, S1 specifically includes: S101. Acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data at the same spatial location, and perform spatial registration of the visible light image data and infrared thermal imaging data using the camera intrinsic parameter matrix and relative extrinsic parameter matrix of the visible light camera and the infrared thermal imager. S102. Extract the luminance value and high reflectance of the RGB color space from the spatially registered visible light image data as visual features, and extract the absolute temperature value and local temperature gradient distribution of each pixel from the spatially registered infrared thermal imaging data as temperature features. Then, stitch and fuse the visual features and temperature features in the channel dimension to construct a vector-form temperature-visual joint feature. S103. Set a brightness threshold and a temperature gradient threshold. When the brightness value in the visual feature is greater than the brightness threshold, and the temperature jump value represented by the local temperature gradient distribution in the temperature feature is greater than the temperature gradient threshold, determine that the corresponding pixel belongs to the highly reflective area and / or the transparent material area; or, The temperature-visual joint features are input into a pre-trained multimodal semantic segmentation network. When the classification probability of the corresponding pixel output by the multimodal semantic segmentation network is greater than a preset probability threshold, the corresponding pixel is determined to belong to the high reflectivity area and / or transparent material area. S104. The pixels that are determined to belong to the highly reflective area and / or transparent material area are binarized and morphologically smoothed to output a two-dimensional image with smooth boundaries, which is defined as the material mask.

3. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 1, characterized in that, S2 specifically includes: S201. Perform feature calculation and dense matching on the visible light image data to generate a dense three-dimensional point cloud with color attributes as the image three-dimensional point cloud, and obtain the spatial projection matrix corresponding to each visible light image data. S202. Extract the two-dimensional semantic contour and two-dimensional feature points from the visible light image data, and reproject them onto the spatial location of the three-dimensional point cloud of the image using the spatial projection matrix to construct the first three-dimensional semantic skeleton. S203. Perform plane extraction and equation fitting on the three-dimensional laser scanning point cloud data, calculate the three-dimensional structure edge lines and their intersection points in three-dimensional space, and construct the second three-dimensional semantic skeleton. S204. Classify the intersection points in the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton according to the structural attributes, and define the three-dimensional coordinate points with building structure attribute labels as semantic intersection points.

4. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 1, characterized in that, S3 specifically includes: S301. Construct the first three-dimensional semantic skeleton into a first topological graph, construct the second three-dimensional semantic skeleton into a second topological graph, traverse the nodes in the first and second topological graphs to perform structural attribute matching and geometric consistency verification, remove mismatched point pairs, and define the remaining high-confidence matching point pairs as a set of semantic intersection points with the same name. S302. Divide the set of semantic intersection points with the same name into a source point set and a target point set and construct a spatial transformation objective function. By decentralizing the point set and calculating the covariance matrix, the singular value decomposition algorithm is used to solve the three-dimensional rotation matrix and translation vector, and then construct the initial transformation matrix. S303. Set the coordinate system of the three-dimensional laser scanning point cloud data as the global reference coordinate system, use the initial transformation matrix to perform spatial mapping transformation on the image three-dimensional point cloud, and superimpose the transformed image three-dimensional point cloud with the three-dimensional laser scanning point cloud data to obtain the coarse registration point cloud data. S304. Calculate the registration error of the set of semantic intersection points with the same name after spatial mapping transformation. If the registration error meets the preset convergence condition, output the coarse registration point cloud data. If the convergence condition is not met, adjust the interior point distance threshold and return to step S301 for iterative calculation.

5. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 4, characterized in that, S302 specifically includes: Let P be the source point set and Q be the target point set, and construct the spatial transformation objective function based on the least squares method. Where N is the number of logarithms of the intersection points in the set of semantic intersection points with the same name, R is the three-dimensional rotation matrix, and T is the translation vector; Calculate the centroid coordinates of the source point set P and the target point set Q respectively, and then perform a decentering process on the two sets of points. Calculate the covariance matrix of the decentralized point set, perform singular value decomposition on the covariance matrix, solve for the optimal three-dimensional rotation matrix R based on the orthogonal matrix obtained from the decomposition, and calculate the translation vector T in combination with the centroid coordinates. The homogeneous transformation matrix formed by the three-dimensional rotation matrix R and the translation vector T is used as the initial transformation matrix.

6. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 1, characterized in that, S4 specifically includes: S401. Obtain the camera intrinsic parameter matrix and camera extrinsic parameter matrix when acquiring the visible light image data, and calculate the projection coordinates of each three-dimensional coordinate point in the coarse alignment point cloud data on the two-dimensional image plane based on the camera frustum model. S402. Using the projection coordinates as an index, query the binarized value of the corresponding pixel position in the material mask. When the binarized value of the projection coordinates in the material mask is a preset foreground value representing a highly reflective area and / or a transparent material area, and the depth value of the corresponding three-dimensional coordinate point is within a preset imaging frustum depth range, it is determined that the coincidence condition is met. All three-dimensional coordinate points in the coarse registration point cloud data that meet the coincidence condition are marked and delineated as the high reflectivity anomaly area. S403. Remove all point cloud data within the high-reflectivity anomaly region to eliminate multipath artifacts and spatial noise caused by the physical properties of the material. S404. Using a point cloud edge extraction algorithm, scan the surface topology of the remaining coarsely registered point cloud data after noise removal, and define the blank space inside the extracted non-closed three-dimensional edge lines as the point cloud cavity region.

7. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 6, characterized in that, In step S401, the projected coordinates are calculated using the inverse projection mathematical mapping formula, which is as follows: in, The spatial coordinates of the three-dimensional coordinate points in the coarse registration point cloud data. The projected coordinates of the three-dimensional coordinate point on the two-dimensional image plane are... Let K be the depth value of the 3D coordinate point in the camera coordinate system, and K be the camera intrinsic parameter matrix. Let be the camera extrinsic matrix.

8. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 1, characterized in that, S5 specifically includes: S501. Obtain the unclosed three-dimensional edge line of the point cloud hole region, construct a spatial index in the coarse registration point cloud data after removing noise, and perform a nearest neighbor search based on the three-dimensional edge line to extract the high-density point cloud within the preset structure buffer radius and define it as the entity structure point cloud. S502. Based on the building topology rule that the physical structures physically adjacent to the high-reflectivity areas are coplanar or parallel, a random sampling consensus algorithm or the overall least squares method is used to perform plane equation fitting on the point cloud of the physical structures, and the dominant plane equation representing the real spatial geometry of the missing high-reflectivity areas is calculated and extracted. Where A, B, and C are the components of the normal vector of the fitted dominant plane in the three-dimensional rectangular coordinate system, and D is the intercept parameter of the dominant plane. S503. Project the unclosed three-dimensional edge line onto a two-dimensional plane according to the normal vector of the dominant plane equation to generate a closed projection contour. Generate two-dimensional regular grid points inside the closed projection contour. Use the dominant plane equation to perform inverse coordinate calculation to elevate the two-dimensional regular grid points to three-dimensional space, generate a patch point cloud and define it as the completed geometric surface. S504. Spatially merge the completed geometric surface with the coarse registration point cloud data after noise removal, and apply a filter to the stitching area of ​​the three-dimensional edge line of the merged point cloud for local density equalization processing, and output the repaired point cloud data with complete structure.

9. The method for three-dimensional mapping of buildings based on multi-source data fusion according to claim 1, characterized in that, S6 specifically includes: S601. Based on the repaired point cloud data after completing the geometric surface, the image 3D point cloud and the 3D laser scan point cloud data are globally finely registered using the iterative nearest point algorithm or the normal distribution transformation algorithm. S602. Extract the point cloud normal vector features after global fine registration, and use the surface reconstruction algorithm to encapsulate the discrete point cloud into a continuous triangular patch network to construct a three-dimensional mesh model of the target building. S603. Using the material mask as the Alpha guiding channel, an image restoration algorithm is used in conjunction with the surrounding texture context information to reconstruct the highly reflective pixels in the visible light image data to generate the restored texture, and the original visible light image not covered by the material mask is retained as the visible light texture. S604. Perform UV coordinate unpacking and parametric mapping on the three-dimensional mesh model, perform partition control based on the material mask, and map the repair texture and the visible light texture to the spatial regions corresponding to the three-dimensional mesh model respectively, and output the three-dimensional real scene model of the target building.

10. A three-dimensional mapping device for buildings based on multi-source data fusion, based on the three-dimensional mapping method for buildings based on multi-source data fusion as described in any one of claims 1 to 9, characterized in that, The device includes: The acquisition module is used to acquire visible light image data, three-dimensional laser scanning point cloud data and infrared thermal imaging data of the target building, and to identify highly reflective areas and / or transparent material areas on the surface of the target building based on the visible light image data and infrared thermal imaging data, and to define the pixel range of the highly reflective areas and / or transparent material areas on the two-dimensional image as a material mask. The construction module is used to perform three-dimensional reconstruction based on the visible light image data to obtain a three-dimensional point cloud of the image and construct a first three-dimensional semantic skeleton, and to construct a second three-dimensional semantic skeleton based on the three-dimensional laser scan point cloud data; wherein, both the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton contain semantic intersections for representing the structural features of the target building. The registration module is used to extract semantic intersections representing the same building structure features in the first three-dimensional semantic skeleton and the second three-dimensional semantic skeleton, define them as homonymous semantic intersections, and perform initial registration of the image three-dimensional point cloud and the three-dimensional laser scan point cloud data based on the initial transformation matrix calculated by the homonymous semantic intersections to obtain coarse registration point cloud data. The projection module is used to project and map the spatial position of the material mask onto the coarse registration point cloud data, identify the three-dimensional region in the coarse registration point cloud data that coincides with the space of the material mask as a high-reflection anomalous region, and remove noise points in the high-reflection anomalous region to form a point cloud hole region in the coarse registration point cloud data. The merging module is used to extract the point clouds that are physically adjacent to the hole regions of the point cloud in the coarse registration point cloud data and define them as entity structure point clouds. Based on the preset building topology rules, the entity structure point cloud is fitted with a plane equation to generate a complete geometric surface. The complete geometric surface is then merged with the coarse registration point cloud data after noise removal to obtain the repaired point cloud data. The repair module is used to perform global fine registration using the repaired point cloud data, construct a three-dimensional mesh model of the target building based on the fine registration result, determine texture mapping and texture repair of the three-dimensional mesh model based on the visible light image data and material mask, and output a three-dimensional real-scene model of the target building.

Images