Building high-precision three-dimensional reconstruction method and system based on multi-source data fusion

Abstract

The application provides a kind of building high-precision three-dimensional reconstruction method and system based on multi-source data fusion, system includes data acquisition module, data preprocessing module, reflective target auxiliary multi-sensor dynamic calibration module, transmembrane state data space-time reference uniform module, multi-source data fusion module, fusion generative adversarial network and physical feature constraint dynamic interference repair module and three-dimensional model reconstruction module.The method breaks through the space-time reference difference restriction of cross-modal data by proposing reflective target auxiliary multi-sensor dynamic calibration technology, uses fusion generative adversarial network and physical feature constraint dynamic interference repair algorithm, solves the pollution problem of open scene modeling, constructs "end-edge-cloud" collaborative computing architecture, realizes the seamless connection of large-scale building-level reconstruction and lightweight output, so as to realize the high-precision three-dimensional reconstruction of building.

Description

Technical Field

[0001] This invention relates to the fields of digital surveying and computer vision technology, specifically to a method and system for high-precision 3D reconstruction of buildings based on multi-source data fusion. Background Technology

[0002] With the rapid development of urban construction and the increasing demand for digital urban management, high-precision 3D reconstruction of buildings is of great significance for urban planning, cultural heritage protection, and virtual scene construction. However, using traditional surveying techniques for 3D building reconstruction is greatly limited by spatial terrain, making surveying difficult when facing complex terrain or building structures. Furthermore, for irregular buildings with intricate components, the surveying process is not only lengthy but also results in low accuracy. In addition, traditional surveying methods are costly and inefficient, failing to meet the needs of large-scale, high-precision 3D building reconstruction. With the development of 3D reconstruction technology, methods based on LiDAR data, aerial imagery data, or satellite imagery data have emerged. While LiDAR-based reconstruction methods can acquire high-precision point cloud data, data acquisition costs are high, and there are limitations in acquiring details of the building's internal structure and texture. Aerial and satellite imagery data, although covering a wide range, are insufficient for accurate measurement of building geometry and high-resolution reconstruction of texture details. Meanwhile, single data sources are easily affected by factors such as weather and obstruction, making it difficult for the accuracy and completeness of the reconstruction results to meet the needs of practical applications. For example, in fields such as urban planning, ancient building preservation, and virtual scene construction, more accurate and detailed 3D building models are required. Therefore, the need for multi-source data fusion is becoming increasingly prominent: as various fields continue to increase the requirements for the accuracy and detail of 3D building models, there is an urgent need for a method and system that can integrate the advantages of multiple measurement technologies. By fusing multi-source data, the strengths of different data sources can be comprehensively utilized, complementing and verifying each other, thereby improving the accuracy, completeness, and reliability of 3D reconstruction models and meeting the needs of different application scenarios for high-precision 3D building models. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method and system for high-precision 3D reconstruction of buildings based on multi-source data fusion, which can effectively solve the aforementioned problems.

[0004] The technical solution adopted in this invention is as follows:

[0005] This invention provides a high-precision 3D reconstruction method for buildings based on multi-source data fusion, comprising the following steps:

[0006] Step S1: Arrange multiple reflective targets around the target building;

[0007] Step S2: The lidar and multiple imaging devices are moved relative to the target building to simultaneously observe and scan the target building from different angles; at the same scanning moment, the lidar simultaneously obtains the target point cloud data of a certain reflective target and the target building point cloud data, and each imaging device simultaneously acquires the target image data and the target building image data.

[0008] Step S3: The target point cloud data, the target building point cloud data, the target image data, and the target building image data corresponding to the same scanning time are preprocessed by the data preprocessing module and then transmitted to the data processing layer.

[0009] Step S4: The reflective target-assisted multi-sensor dynamic calibration module of the data processing layer analyzes the target point cloud data and each target image data to establish the relative position and attitude transformation parameters of the lidar and each imaging device at the current scanning time.

[0010] Step S5: Based on the relative position and attitude transformation parameters, dynamically calibrate the target building point cloud data and each target building image data to obtain calibrated target building point cloud data and calibrated target building image data that are unified to the same spatiotemporal reference.

[0011] Step S6: Using a multi-source data fusion module, the calibrated point cloud data of the target building and the calibrated image data of each target building, which are unified to the same spatiotemporal reference, are fused to obtain fused data.

[0012] Step S7: The dynamic interference repair module that integrates generative adversarial network and physical feature constraints is used to perform dynamic interference repair on the fused data to obtain the repaired fused data.

[0013] Step S8: Using the 3D model reconstruction module, the repaired fused data from different scanning times is reconstructed into a 3D model of the target building.

[0014] Preferably, the imaging device includes a drone equipped with a camera and a ground-based mobile device; the drone equipped with the camera acquires multi-angle image data of the target building using oblique photography; the ground-based mobile device acquires close-up image data of the target building.

[0015] Preferably, at the same scanning moment, the lidar maintains its position and orientation to obtain target point cloud data of a certain reflective target and target building point cloud data; at the same scanning moment, each imaging device maintains its position and orientation, only adjusts its focal length, and acquires target image data and target building image data.

[0016] Preferably, the data preprocessing module is specifically used to: denoise and convert the target point cloud data, the target building point cloud data, the target image data, and the target building image data.

[0017] Preferably, the reflective target-assisted multi-sensor dynamic calibration module is specifically used for:

[0018] Step S4.1: Perform feature matching of the same target feature points on the target point cloud data and each target image data. Let the number of target feature point pairs identified be n pairs, and the i-th target feature point pair includes target point cloud data point i and target image data point i.

[0019] Step S4.2, the three-dimensional coordinates of target point cloud data point i in the lidar coordinate system. Calculated using the following formula:

[0020]

[0021] Where: r i θ represents the distance from target point i to the lidar, corresponding to target point i in the target point cloud data obtained by the lidar measurement; i φ is the horizontal angle φ of target point i corresponding to target point i in the target point cloud data obtained by lidar measurement. i The vertical angle of target point i corresponding to target point i obtained from the laser radar point cloud data point measurement;

[0022] Step S4.3, establish the imaging model of the imaging device as follows:

[0023]

[0024] Where: (u i ,v i ) represents the coordinates of target point i on the camera's imaging plane, corresponding to target image data point i; K is the camera's intrinsic parameter matrix. f x and f y Let (u0, v0) be the focal lengths of the camera in the x and y directions, respectively; (u0, v0) be the coordinates of the image center; [R|t] is the camera extrinsic parameter matrix, where R is the rotation matrix, t is the translation vector, and s is the scale factor. i ,Y i Z i () represents the coordinates of target point i in the world coordinate system corresponding to target image data point i;

[0025] Step S4.4: Using the obtained target feature point pairs, solve the following optimization problem using the least squares method to obtain the rotation matrix R and translation vector t:

[0026]

[0027] Wherein: when using the least squares method to solve for the rotation matrix R and the translation vector t, the error function E(R,t) is constructed:

[0028]

[0029] in, Let i be the three-dimensional coordinates of target point cloud data point i in the lidar coordinate system; Let i be the coordinates of target point i on the camera imaging plane, corresponding to target image data point i.

[0030] By taking the partial derivatives of E(R,t) with respect to R and t, and setting the partial derivatives to 0, a set of equations can be obtained. Solving the set of equations yields the optimal rotation matrix R and translation vector t, thereby obtaining the relative position and attitude transformation parameters of the lidar and each of the imaging devices at the current scanning moment.

[0031] Preferably, step S6 specifically includes:

[0032] The point cloud data of the target building, after being calibrated to the same spatiotemporal reference, is represented as D. cloud The calibrated target building image data, unified to the same spatiotemporal reference, is represented as D. image H is the parameter to be estimated, i.e., the coordinates of the fused spatial points, then the posterior probability P(H|D) is... cloud D image )for:

[0033]

[0034] Where: P(H) is the prior probability, P(D) is the prior probability. cloud D image |H) is the likelihood function, P(D) cloud D image ) is the evidence factor; the value of the fused data is determined according to the maximum a posteriori probability criterion.

[0035] Preferably, step S7 specifically includes:

[0036] The dynamic interference repair module, which integrates generative adversarial networks and physical feature constraints, includes a generator and a discriminator.

[0037] For the discriminator, its loss function L D for:

[0038]

[0039] E represents the expectation, Preal(y) is the distribution of the real data y, and Pdata(x) is the distribution of the input fused data x; θD For discriminator parameters; D(y; θ) D θ represents the discriminant's judgment result on the real data y; g For generator parameters; G(x; θ) G ) represents the samples generated by the generator based on the fused data x; This refers to the discriminator's judgment result on the samples generated by the generator; These are the discriminator hyperparameters;

[0040] For the generator, its loss function L G The aim is to minimize the probability that generated data is identified as fake data, while considering physical feature constraints. Assume the loss due to physical feature constraints is L. phys Then the generator's loss function L G for:

[0041]

[0042] Where: λ is the weighting factor of the physical feature constraint.

[0043] Preferably, step S8 specifically includes:

[0044] A preliminary 3D model of the target building is constructed using the Delaunay triangulation algorithm. Then, a vertex clustering algorithm is used to further simplify the preliminary 3D model of the target building and reduce the amount of data in the model. The specific steps are as follows:

[0045] Step S8.1, Determine Cluster Centers: Based on the vertex distribution of the preliminary 3D model of the target building, determine the cluster centers. Specifically, randomly select k vertices as the initial cluster centers C = {c1, c2, ..., c...} k The following formula is used to calculate each vertex p. i to each cluster center c j Euclidean distance d(p) i ,c j ), assign the vertex to the cluster containing the nearest cluster center:

[0046]

[0047] Where: vertex p i The three-dimensional coordinates are (x i ,y i ,z i Cluster center c j The three-dimensional coordinates are Step S8.2, Vertex Merging: For each cluster, calculate the centroid of the vertices within the cluster. n jIt represents the number of vertices in the j-th cluster. By merging all vertices within the cluster into a single vertex at the centroid position, the number of vertices in the model can be reduced.

[0048] This invention also provides a high-precision 3D reconstruction system for buildings based on multi-source data fusion, comprising:

[0049] The data acquisition module is used to deploy multiple reflective targets around the target building; it uses a lidar and various imaging devices to move relative to the target building and simultaneously scan the target building from different angles; at the same scanning moment, the lidar simultaneously acquires the target point cloud data of a certain reflective target and the point cloud data of the target building, and each imaging device simultaneously acquires the target image data and the target building image data;

[0050] The data preprocessing module is used to preprocess the target point cloud data, the target building point cloud data, the target image data, and the target building image data corresponding to the same scanning time, and then transmit them to the data processing layer.

[0051] The data processing layer includes a reflective target-assisted multi-sensor dynamic calibration module, a cross-membrane data spatiotemporal benchmark unification module, a multi-source data fusion module, a dynamic interference repair module that fuses generative adversarial networks and physical feature constraints, and a three-dimensional model reconstruction module.

[0052] The reflective target-assisted multi-sensor dynamic calibration module is used to analyze the target point cloud data and each target image data to establish the relative position and attitude transformation parameters of the lidar and each imaging device at the current scanning time.

[0053] The cross-membrane data spatiotemporal reference unification module is used to dynamically calibrate the target building point cloud data and each target building image data based on the relative position and attitude transformation parameters, so as to obtain the calibrated target building point cloud data and each target building image data unified to the same spatiotemporal reference.

[0054] The multi-source data fusion module is used to fuse the calibrated point cloud data of the target building, which is unified to the same spatiotemporal reference, and the calibrated image data of each target building to obtain fused data.

[0055] The dynamic interference repair module that integrates generative adversarial networks and physical feature constraints is used to perform dynamic interference repair on the fused data to obtain repaired fused data.

[0056] The three-dimensional model reconstruction module is used to reconstruct the repaired fused data from different scanning times into a three-dimensional model of the target building.

[0057] The high-precision 3D reconstruction method and system for buildings based on multi-source data fusion provided by this invention has the following advantages:

[0058] This method overcomes the limitations of spatiotemporal benchmark differences in cross-modal data by proposing a multi-sensor dynamic calibration technology assisted by a reflective target. It also solves the problem of pollution in open scene modeling by using a dynamic interference repair algorithm that integrates generative adversarial networks and physical feature constraints. Furthermore, it constructs an "edge-cloud" collaborative computing architecture to achieve seamless integration between large-scale building-level reconstruction and lightweight output, thereby realizing high-precision 3D reconstruction of buildings. Attached Figure Description

[0059] Figure 1 This is a schematic diagram illustrating the principle of the high-precision 3D reconstruction method for buildings based on multi-source data fusion provided by the present invention. Detailed Implementation

[0060] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0061] This invention provides a method and system for high-precision 3D reconstruction of buildings based on multi-source data fusion. This method overcomes the limitations of spatiotemporal reference differences in cross-modal data by proposing a multi-sensor dynamic calibration technology assisted by a reflective target. It also solves the problem of modeling pollution in open scenes by employing a dynamic interference repair algorithm that fuses generative adversarial networks and physical feature constraints. Furthermore, it constructs an "edge-cloud" collaborative computing architecture to achieve seamless integration of large-scale building-level reconstruction and lightweight output, thereby realizing high-precision 3D reconstruction of buildings and meeting the high-precision and high-efficiency demands of modern society for building digitization. The specific steps of the method are as follows:

[0062] Step 1: Dynamic calibration of multiple sensors assisted by a reflective target

[0063] Reflective targets are deployed at suitable locations around the building, and multiple sensors are used to observe the targets simultaneously. By extracting features from the reflective targets in the data from different sensors, a correspondence between the data from different sensors is established. Based on these correspondences, the relative position and attitude transformation parameters between the sensors are calculated, enabling dynamic calibration of the multiple sensors. This overcomes the limitations of spatiotemporal reference differences in cross-modal data, achieving a unified spatiotemporal reference for data collected by different sensors.

[0064] Step 2: Multi-source data fusion

[0065] The preprocessed and calibrated data are fused. For point cloud data and image data, a feature matching algorithm is used to establish a correspondence between feature points in the point cloud data and features in the image data; a spatial coordinate transformation method is used to unify different types of data into the same coordinate system; and a Bayesian estimation-based fusion method is used to fuse multi-source data to form fused data containing complete geometric structure and rich texture information.

[0066] Step 3: Dynamic interference repair by fusing generative adversarial networks and physical feature constraints.

[0067] A fusion generative adversarial network (GAN) is constructed, where the generator is responsible for generating repaired images or data, and the discriminator is used to distinguish between generated and real data. Simultaneously, physical feature constraints are introduced to guide the generator in producing data that more closely reflects reality. The fused data is then input into this model to repair contamination issues caused by factors such as lighting variations, occlusion, and noise during open scene modeling, thereby improving the quality and accuracy of the data.

[0068] Step 4: Build a collaborative computing architecture and data processing system for the "device-edge-cloud" platform.

[0069] At the data acquisition end, various sensor devices perform preliminary data processing to reduce data transmission volume. Edge devices receive data from the acquisition end, perform real-time analysis and processing, alleviating the computing pressure on the cloud. The cloud, on the other hand, utilizes powerful computing resources to perform in-depth data processing and analysis. Through collaborative work between the "end-edge-cloud," efficient data processing and transmission are achieved, providing support for large-scale building-level reconstruction.

[0070] Step 5: Large-scale building reconstruction and lightweight output

[0071] Based on fused and repaired data, a 3D mesh model of the building is constructed using the Delaunay triangulation algorithm to accurately reproduce the building's geometry. Texture information from the processed image data is mapped onto the 3D mesh model to generate a 3D model with realistic textures. During texture mapping, issues such as texture resolution, stretching, and deformation are considered to ensure accurate matching between the texture and the model.

[0072] Then, a model lightweighting algorithm was used to further simplify the optimized 3D model and reduce its data volume. The lightweight model was then formatted to adapt to different application scenarios and platforms, achieving a seamless connection between large-scale architectural reconstruction and lightweight output, ultimately completing a high-precision 3D reconstruction of the building.

[0073] The detailed steps of this invention are as follows:

[0074] Step S1: Arrange multiple reflective targets around the target building;

[0075] Step S2: The lidar and multiple imaging devices are moved relative to the target building to simultaneously observe and scan the target building from different angles; at the same scanning moment, the lidar simultaneously obtains the target point cloud data of a certain reflective target and the target building point cloud data, and each imaging device simultaneously acquires the target image data and the target building image data.

[0076] The imaging equipment includes a drone equipped with a camera and a ground-based mobile device; the drone equipped with the camera acquires multi-angle image data of the target building using oblique photography; the ground-based mobile device acquires close-up image data of the target building.

[0077] At the same scanning moment, the lidar maintains its position and orientation, and obtains target point cloud data of a certain reflective target and target building point cloud data; at the same scanning moment, each imaging device maintains its position and orientation, and only adjusts its focal length to acquire target image data and target building image data.

[0078] Step S3: The target point cloud data, the target building point cloud data, the target image data, and the target building image data corresponding to the same scanning time are preprocessed by the data preprocessing module and then transmitted to the data processing layer.

[0079] The data preprocessing module is specifically used to: denoise and convert the target point cloud data, the target building point cloud data, the target image data, and the target building image data.

[0080] Step S4: The reflective target-assisted multi-sensor dynamic calibration module of the data processing layer analyzes the target point cloud data and each target image data to establish the relative position and attitude transformation parameters of the lidar and each imaging device at the current scanning time.

[0081] Specifically, multiple sensors, including lidar and cameras, are used to simultaneously observe the deployed reflective target, thereby establishing connections between data from different sensors and completing dynamic calibration. This step is as follows:

[0082] Step S4.1: Perform feature matching of the same target feature points on the target point cloud data and each target image data. Let the number of target feature point pairs identified be n pairs, and the i-th target feature point pair includes target point cloud data point i and target image data point i.

[0083] Step S4.2: The lidar acquires the target's distance information by emitting a laser beam and receiving the reflected light, generating point cloud data. The three-dimensional coordinates of target point cloud data point i in the lidar coordinate system are shown below. Calculated using the following formula:

[0084]

[0085] Where: r i θ represents the distance from target point i to the lidar, corresponding to target point i in the target point cloud data obtained by the lidar measurement; i φ is the horizontal angle φ of target point i corresponding to target point i in the target point cloud data obtained by lidar measurement. i The vertical angle of target point i corresponding to target point i obtained from the laser radar point cloud data point measurement;

[0086] Step S4.3: Multiple cameras capture images of the reflective target from different angles. The camera imaging process can be described using a pinhole camera model. Therefore, based on the pinhole camera model, the imaging model of the imaging device is established as follows:

[0087]

[0088] Where: (u i ,v i ) represents the coordinates of target point i on the camera's imaging plane, corresponding to target image data point i; K is the camera's intrinsic parameter matrix. f x and f y Let (u0, v0) be the focal lengths of the camera in the x and y directions, respectively; (u0, v0) be the coordinates of the image center; [R|t] is the camera extrinsic parameter matrix, where R is the rotation matrix, t is the translation vector, and s is the scale factor. i ,Y i Z i () represents the coordinates of target point i in the world coordinate system corresponding to target image data point i;

[0089] Step S4.4: Establish the correspondence between the reflective target features in the lidar point cloud data and the camera image data, and use the obtained corresponding point pairs to calculate the transformation parameters between the lidar coordinate system and the camera coordinate system.

[0090] Specifically, using the acquired target feature point pairs, the following optimization problem is solved using the least squares method to obtain the rotation matrix R and translation vector t, thereby achieving the unification of the lidar coordinate system and the camera coordinate system.

[0091]

[0092] Wherein: when using the least squares method to solve for the rotation matrix R and the translation vector t, the error function E(R,t) is constructed:

[0093]

[0094] in, Let i be the three-dimensional coordinates of target point cloud data point i in the lidar coordinate system; Let i be the coordinates of target point i on the camera imaging plane, corresponding to target image data point i.

[0095] By taking the partial derivatives of E(R,t) with respect to R and t, and setting the partial derivatives to 0, a set of equations can be obtained. Solving the set of equations yields the optimal rotation matrix R and translation vector t, thereby obtaining the relative position and attitude transformation parameters of the lidar and each of the imaging devices at the current scanning moment.

[0096] Step S5: Based on the relative position and attitude transformation parameters, dynamically calibrate the target building point cloud data and each target building image data to obtain calibrated target building point cloud data and calibrated target building image data that are unified to the same spatiotemporal reference.

[0097] Step S6: Using a multi-source data fusion module, the calibrated point cloud data of the target building and the calibrated image data of each target building, which are unified to the same spatiotemporal reference, are fused to obtain fused data.

[0098] Step S6 specifically involves fusing point cloud data and image data using a Bayesian estimation-based fusion method after feature matching and coordinate transformation.

[0099] The point cloud data of the target building, after being calibrated to the same spatiotemporal reference, is represented as D. cloud The calibrated target building image data, unified to the same spatiotemporal reference, is represented as D. image H is the parameter to be estimated, i.e., the coordinates of the fused spatial points. According to Bayes' theorem, the posterior probability P(H|D) is... cloud D image )for:

[0100]

[0101] Where: P(H) is the prior probability, P(D) is the prior probability. cloud D image |H) is the likelihood function, P(D) cloud D image These are evidence factors; in practical applications, these probability values ​​are estimated and the values ​​of the fused data are determined according to the maximum a posteriori probability criterion.

[0102] Step S7: The dynamic interference repair module that integrates generative adversarial network and physical feature constraints is used to perform dynamic interference repair on the fused data to obtain the repaired fused data.

[0103] Step S7 is as follows:

[0104] To train the generator and discriminator, a suitable loss function needs to be defined. The dynamic interference repair module, which integrates generative adversarial networks with physical feature constraints, includes a generator and a discriminator.

[0105] For the discriminator, its loss function L D for:

[0106]

[0107] E represents the expectation, Preal(y) is the distribution of the real data y, and Pdata(x) is the distribution of the input fused data x; θ D For discriminator parameters; D(y; θ) D θ represents the discriminant's judgment result on the real data y; G For generator parameters; G(x; θ) G ) represents the samples generated by the generator based on the fused data x; This refers to the discriminator's judgment result on the samples generated by the generator; These are the discriminator hyperparameters;

[0108] For the generator, its loss function L G The aim is to minimize the probability that generated data is identified as fake data, while considering physical feature constraints. Assume the loss due to physical feature constraints is L. phys Then the generator's loss function L G for:

[0109]

[0110] Where: λ is the weighting factor of the physical feature constraint.

[0111] Step S8: Using the 3D model reconstruction module, the repaired fused data from different scanning times is reconstructed into a 3D model of the target building.

[0112] Step S8 is as follows:

[0113] To achieve large-scale building reconstruction and lightweight output, the Delaunay triangulation algorithm is used to construct a preliminary 3D model of the target building. Then, a vertex clustering algorithm is used to further simplify the preliminary 3D model of the target building and reduce the amount of data in the model. The specific steps are as follows:

[0114] Step S8.1, determine the cluster centers:

[0115] Based on the vertex distribution of the preliminary 3D model of the target building, cluster centers are determined, specifically by randomly selecting k vertices as initial cluster centers C = {c1, c2, ..., c3}. k The following formula is used to calculate each vertex p. i to each cluster center c j Euclidean distance d(p) i ,c j ), assign the vertex to the cluster containing the nearest cluster center:

[0116]

[0117] Where: vertex p i The three-dimensional coordinates are (x i ,y i ,z i Cluster center c j The three-dimensional coordinates are Step S8.2, Vertex Merging: For each cluster, calculate the centroid of the vertices within the cluster. n j It represents the number of vertices in the j-th cluster. By merging all vertices within the cluster into a single vertex at the centroid position, the number of vertices in the model can be reduced.

[0118] This invention also provides a high-precision 3D reconstruction system for buildings based on multi-source data fusion, comprising:

[0119] The data acquisition module is used to deploy multiple reflective targets around the target building; it uses a lidar and various imaging devices to move relative to the target building and simultaneously scan the target building from different angles; at the same scanning moment, the lidar simultaneously acquires the target point cloud data of a certain reflective target and the point cloud data of the target building, and each imaging device simultaneously acquires the target image data and the target building image data;

[0120] The data preprocessing module is used to preprocess the target point cloud data, the target building point cloud data, the target image data, and the target building image data corresponding to the same scanning time, and then transmit them to the data processing layer.

[0121] The data processing layer includes a reflective target-assisted multi-sensor dynamic calibration module, a cross-membrane data spatiotemporal benchmark unification module, a multi-source data fusion module, a dynamic interference repair module that fuses generative adversarial networks and physical feature constraints, and a three-dimensional model reconstruction module.

[0122] The reflective target-assisted multi-sensor dynamic calibration module is used to analyze the target point cloud data and each target image data to establish the relative position and attitude transformation parameters of the lidar and each imaging device at the current scanning time.

[0123] The cross-membrane data spatiotemporal reference unification module is used to dynamically calibrate the target building point cloud data and each target building image data based on the relative position and attitude transformation parameters, so as to obtain the calibrated target building point cloud data and each target building image data unified to the same spatiotemporal reference.

[0124] The multi-source data fusion module is used to fuse the calibrated point cloud data of the target building, which is unified to the same spatiotemporal reference, and the calibrated image data of each target building to obtain fused data.

[0125] The dynamic interference repair module that integrates generative adversarial networks and physical feature constraints is used to perform dynamic interference repair on the fused data to obtain repaired fused data.

[0126] The three-dimensional model reconstruction module is used to reconstruct the repaired fused data from different scanning times into a three-dimensional model of the target building.

[0127] The following is an example:

[0128] Appendix Figure 1 The flowchart for 3D reconstruction of a building consists of three parts: data acquisition layer, data processing layer, and model building layer.

[0129] In the data acquisition layer, LiDAR is used to collect point cloud data of the building to obtain precise geometric structure information. Then, multi-angle image data is collected through oblique photography using drones / cameras to provide rich texture information. Finally, ground-based mobile devices collect close-up image data of the building to supplement more details. All collected data is transmitted to the data preprocessing module.

[0130] In the data processing layer, firstly, the raw data is pre-processed in the data preprocessing module, including denoising and format conversion. Secondly, since reflective targets are placed around the building, the characteristics of the reflective targets in different sensors are analyzed to establish correspondences, calculate the relative position and attitude transformation parameters between the sensors, and achieve dynamic calibration. Point cloud data is generated by using LiDAR to acquire distance information of the targets; its three-dimensional coordinates in the LiDAR coordinate system can be calculated using a specific formula. The camera is described using a pinhole imaging model, and the corresponding parameters are calculated using a pinhole camera model. The least squares method is used to solve the optimization problem to complete the calibration. The calibrated data from different sensors are unified to the same spatiotemporal reference, eliminating temporal and spatial differences caused by different data sources. Then, feature matching algorithms and spatial coordinate transformation methods are used to fuse the point cloud data and image data into a unified dataset. Then, the data fusion effect is further optimized using a Bayesian estimation method to form fused data containing complete geometric structure and rich texture information. Next, a GAN model is constructed and physical feature constraints are introduced to fix the pollution problem generated during the open scene modeling process, improve data quality and accuracy, and utilize the collaborative computing capabilities of end devices, edge devices and the cloud to process data efficiently. Finally, a 3D mesh model is constructed based on the fused data using an appropriate algorithm, the texture of the image data is mapped onto the model, and the model is then optimized by smoothing, simplification and other operations to reduce the amount of data, making it easier to store and transmit, and thus outputting a high-precision 3D reconstruction model.

[0131] The high-precision 3D reconstruction method and system for buildings based on multi-source data fusion provided by this invention has the following advantages:

[0132] (1) This invention utilizes multi-sensor dynamic calibration technology assisted by reflective targets to overcome the limitations of spatiotemporal benchmark differences in cross-modal data, achieve accurate fusion of multi-source data, and provide a reliable data foundation for high-precision three-dimensional reconstruction.

[0133] (2) By using a dynamic interference repair algorithm that integrates generative adversarial networks and physical feature constraints, the pollution problem in open scene modeling can be solved, effectively improving the data quality and accuracy of the reconstruction model.

[0134] (3) Construct a “end-edge-cloud” collaborative computing architecture to achieve seamless connection between large-scale building-level reconstruction and lightweight output, improve data processing efficiency and meet the application needs of different scenarios.

[0135] (4) Unlike traditional 3D reconstruction methods, it can integrate multi-source data to complete a high-precision 3D reconstruction of a building in all directions at once, fully presenting the building's geometric structure and texture details, and is widely applicable to multiple fields.

[0136] This invention not only overcomes the limitations of traditional surveying and mapping technology, but also improves the accuracy, completeness and reliability of three-dimensional building reconstruction, meeting the needs of modern society for building digitization.

[0137] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for high-precision 3D reconstruction of buildings based on multi-source data fusion, characterized in that, Includes the following steps: Step S1: Arrange multiple reflective targets around the target building; Step S2: The lidar and multiple imaging devices are moved relative to the target building to simultaneously observe and scan the target building from different angles; at the same scanning moment, the lidar simultaneously obtains the target point cloud data of a certain reflective target and the target building point cloud data, and each imaging device simultaneously acquires the target image data and the target building image data. Step S3: The target point cloud data, the target building point cloud data, the target image data, and the target building image data corresponding to the same scanning time are preprocessed by the data preprocessing module and then transmitted to the data processing layer. Step S4: The reflective target-assisted multi-sensor dynamic calibration module of the data processing layer analyzes the target point cloud data and each target image data to establish the relative position and attitude transformation parameters of the lidar and each imaging device at the current scanning time. Step S5: Based on the relative position and attitude transformation parameters, dynamically calibrate the target building point cloud data and each target building image data to obtain calibrated target building point cloud data and calibrated target building image data that are unified to the same spatiotemporal reference. Step S6: Using a multi-source data fusion module, the calibrated point cloud data of the target building and the calibrated image data of each target building, which are unified to the same spatiotemporal reference, are fused to obtain fused data. Step S7: The dynamic interference repair module that integrates generative adversarial network and physical feature constraints is used to perform dynamic interference repair on the fused data to obtain the repaired fused data. Step S8: Using the 3D model reconstruction module, the repaired fused data from different scanning times are reconstructed into a 3D model of the target building. The reflective target-assisted multi-sensor dynamic calibration module is specifically used for: Step S4.1: Perform feature matching on the target point cloud data and each target image data for the same target feature points. Let the number of target feature point pairs identified be... Yes, the first Each target feature point pair includes target point cloud data point i and target image data point i; Step S4.2, the three-dimensional coordinates of target point cloud data point i in the lidar coordinate system. Calculated using the following formula: （1） in: The distance from the target point i to the lidar is the distance corresponding to the target point cloud data point i obtained by the lidar measurement. Let i be the horizontal angle corresponding to target point i in the target point cloud data obtained from lidar measurement. The vertical angle of target point i corresponding to target point i obtained from the laser radar point cloud data point measurement; Step S4.3, establish the imaging model of the imaging device as follows: （2） in: Let i be the coordinates of target point i on the camera imaging plane, corresponding to target image data point i. This is the intrinsic parameter matrix of the camera. , and These are the focal lengths of the camera in the x and y directions, respectively. The coordinates of the image center; For the camera extrinsic matrix, For rotation matrix, It is a translation vector. As a scale factor, Let i be the coordinates of target point i in the world coordinate system. Step S4.4: Using the acquired target feature point pairs, solve the following optimization problem using the least squares method to obtain the rotation matrix. Translation vector : （3） Among them: when solving the rotation matrix using the least squares method Translation vector At that time, construct the error function : （4） in, Let i be the three-dimensional coordinates of target point cloud data point i in the lidar coordinate system; Let i be the coordinates of target point i on the camera imaging plane, corresponding to target image data point i. Through the about and By taking the partial derivatives and setting them to zero, we obtain a set of equations. Solving these equations yields the optimal rotation matrix. Translation vector Thus, the relative position and attitude transformation parameters of the lidar and each of the imaging devices at the current scanning moment are obtained; Step S6 is as follows: The calibrated point cloud data of the target building, unified to the same spatiotemporal reference, is represented as follows: The calibrated target building image data, unified to the same spatiotemporal reference, is represented as: ; If the parameter to be estimated is the coordinates of the fused spatial points, then the posterior probability is... for: （5） in: It is the prior probability. It is the likelihood function. It is an evidence factor; the value of the fused data is determined according to the maximum a posteriori probability criterion. Step S7 is as follows: The dynamic interference repair module, which integrates generative adversarial networks and physical feature constraints, includes a generator and a discriminator. For the discriminator, its loss function for: （6） Expressing expectations, It is real data. The distribution It is the input fused data Distribution; These are the parameters for the discriminator; For the discriminator to analyze real data The judgment result; These are the generator parameters; For generators based on fused data The generated samples; This refers to the discriminator's judgment result on the samples generated by the generator; These are the discriminator hyperparameters; For the generator, its loss function The aim is to minimize the probability of generated data being identified as fake data, while considering physical feature constraints. Assume the loss due to physical feature constraints is... Then the generator's loss function for: （7） in: The weighting factor for physical feature constraints; Step S8 is as follows: A preliminary 3D model of the target building is constructed using the Delaunay triangulation algorithm. Then, a vertex clustering algorithm is used to further simplify the preliminary 3D model of the target building and reduce the amount of data in the model. The specific steps are as follows: Step S8.1, Determine cluster centers: Based on the vertex distribution of the preliminary 3D model of the target building, determine the cluster centers, specifically by randomly selecting k vertices as the initial cluster centers. The following formula is used to calculate each vertex. To each cluster center Euclidean distance Assign vertices to the cluster containing the nearest cluster center: （8） Where: Vertex The three-dimensional coordinates are ( Cluster Center The three-dimensional coordinates are ( ); Step S8.2, Vertex Merging: For each cluster, calculate the centroid of the vertices within the cluster. , It is the first The number of vertices in each cluster is reduced by merging all vertices within a cluster into a single vertex at the centroid position, thus reducing the number of vertices in the model.

2. The method for high-precision three-dimensional reconstruction of buildings based on multi-source data fusion according to claim 1, characterized in that, The imaging equipment includes a drone equipped with a camera and a ground-based mobile device; the drone equipped with the camera acquires multi-angle image data of the target building using oblique photography; the ground-based mobile device acquires close-up image data of the target building.

3. The method for high-precision three-dimensional reconstruction of buildings based on multi-source data fusion according to claim 1, characterized in that, At the same scanning moment, the lidar maintains its position and orientation, and obtains target point cloud data of a certain reflective target and target building point cloud data; at the same scanning moment, each imaging device maintains its position and orientation, and only adjusts its focal length to acquire target image data and target building image data.

4. The method for high-precision three-dimensional reconstruction of buildings based on multi-source data fusion according to claim 1, characterized in that, The data preprocessing module is specifically used to: denoise and convert the target point cloud data, the target building point cloud data, the target image data, and the target building image data.

5. A high-precision three-dimensional reconstruction system for buildings based on multi-source data fusion, characterized in that, A method for high-precision 3D reconstruction of buildings based on multi-source data fusion as described in any one of claims 1-4 includes: The data acquisition module is used to deploy multiple reflective targets around the target building; it uses a lidar and various imaging devices to move relative to the target building and simultaneously scan the target building from different angles; at the same scanning moment, the lidar simultaneously acquires the target point cloud data of a certain reflective target and the point cloud data of the target building, and each imaging device simultaneously acquires the target image data and the target building image data; The data preprocessing module is used to preprocess the target point cloud data, the target building point cloud data, the target image data, and the target building image data corresponding to the same scanning time, and then transmit them to the data processing layer. The data processing layer includes a reflective target-assisted multi-sensor dynamic calibration module, a cross-modal data spatiotemporal reference unification module, a multi-source data fusion module, a dynamic interference repair module that fuses generative adversarial networks and physical feature constraints, and a three-dimensional model reconstruction module. The reflective target-assisted multi-sensor dynamic calibration module is used to analyze the target point cloud data and each target image data to establish the relative position and attitude transformation parameters of the lidar and each imaging device at the current scanning time. The cross-modal data spatiotemporal reference unification module is used to dynamically calibrate the target building point cloud data and each target building image data based on the relative position and attitude transformation parameters, so as to obtain the calibrated target building point cloud data and each target building image data unified to the same spatiotemporal reference. The multi-source data fusion module is used to fuse the calibrated point cloud data of the target building, which is unified to the same spatiotemporal reference, and the calibrated image data of each target building to obtain fused data. The dynamic interference repair module that integrates generative adversarial networks and physical feature constraints is used to perform dynamic interference repair on the fused data to obtain repaired fused data. The three-dimensional model reconstruction module is used to reconstruct the repaired fused data from different scanning times into a three-dimensional model of the target building.

Images