A refined 3D modeling method and system based on air-ground multi-source data fusion

Abstract

The present invention provides a refined 3D modeling method and system based on air-ground multi-source data fusion. Firstly, data is collected from three data sources, namely, unmanned aerial vehicle oblique photography, scanning point cloud data and panoramic video data, and pre-processed. Then, the multi-source data is roughly aligned, and an improved ICP algorithm is used for fine alignment to achieve multi-source data fusion. Finally, an improved Poisson surface reconstruction algorithm is used for modeling. The present invention comprehensively utilizes the advantages of different data sources to achieve refined and fully covered 3D reconstruction of the target environment, and solves the problems of missing details and incomplete coverage caused by a single data source in the modeling process of complex buildings. The multi-source data fusion technology in the present invention can fully capture the geometric forms and detailed features of buildings in complex scenes, is suitable for complex scenes such as urban villages and old buildings, and realizes all-round and detailed 3D modeling.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and more specifically, to a refined three-dimensional modeling method and system based on air-ground multi-source data fusion. Background Art

[0002] In the current 3D modeling field, detailed modeling of architectural environments has become a core demand in fields such as urban planning, architectural design, and cultural relics protection. However, in complex architectural environments (such as urban villages, old urban areas, and high-density building areas), traditional 3D modeling technology often fails to achieve ideal results due to complex building structures, narrow spaces, many obstructions, and irregular shapes. In particular, the spatial layout of buildings in urban villages is highly irregular, with dense buildings, diverse levels, and a large number of detailed structures (such as awnings, various pipelines, etc.), which puts higher demands on existing 3D modeling technology.

[0003] Traditional 3D building modeling technology usually relies on two main methods: one is image modeling technology based on drone oblique photography, and the other is point cloud modeling technology based on ground Lidar scanning.

[0004] Drone oblique photography technology uses the multi-lens system on the drone to collect images of buildings from multiple angles and heights. This method can quickly cover a large range of building areas and is particularly suitable for wide-area monitoring and preliminary modeling, and the model has a good texture mapping effect. However, due to the limitations of image resolution and viewing angle, drone oblique photography often has difficulty accurately capturing the complex details of buildings, especially local features such as the bottom, edges, corners, and three-dimensional structures of buildings.

[0005] In contrast, ground-based Lidar scanning technology can provide high-precision point cloud data, which is very suitable for capturing detailed geometric shapes on building surfaces. However, this method is usually only suitable for obtaining local data and is difficult to fully cover a large range of buildings. At the same time, in terms of texture mapping, ground-based Lidar scanning mainly relies on the density and accuracy of the point cloud data itself to reconstruct textures, lacking real color and texture information, and needs to be supplemented by other means (such as panoramic photography).

[0006] Due to the limitations of these two methods, it is difficult to take into account both the overall coverage of the building and the accuracy of local details in a complex building environment. Therefore, there is an urgent need for a modeling technology that can combine the advantages of wide-area coverage and local high-precision to solve the problem of refined 3D modeling in complex building environments. Summary of the invention

[0007] In order to overcome the defect of the above-mentioned prior art that it is difficult to take into account both the coverage of modeling and the accuracy of local details when facing a complex environment, the present invention provides a refined 3D modeling method and system based on air-ground multi-source data fusion. By fusing drone oblique photography, handheld Lidar scanning and panoramic video data, efficient fusion of multi-source data and high-precision 3D model generation are achieved, which can meet the needs of urban planning, architectural design, cultural relics protection and other fields for refined 3D modeling in complex architectural environments.

[0008] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0009] A refined 3D modeling method based on air-ground multi-source data fusion includes the following steps:

[0010] S1: Collect the UAV oblique photography data, scanning point cloud data and panoramic video data of the target environment, and pre-process them respectively;

[0011] S2: Preliminarily align the preprocessed UAV oblique photography data and the scanned point cloud data to obtain the preliminarily aligned UAV oblique photography data and the scanned point cloud data;

[0012] S3: Using the improved ICP algorithm, the texture information of the pre-processed panoramic video data is introduced to perform fine registration on the preliminarily aligned UAV oblique photography data and scanning point cloud data to obtain the multi-source data registration result;

[0013] S4: An improved Poisson surface reconstruction algorithm is used to introduce normal vector smoothing and outlier removal mechanisms. Then, based on a multi-scale reconstruction strategy, multi-source data registration results are reconstructed at multiple scales to generate a high-precision three-dimensional model of the target environment and complete the modeling.

[0014] Preferably, before step S1, the step further includes:

[0015] A number of image control points are evenly arranged in the target environment, and the three-dimensional coordinates of all the image control points are measured to obtain image control point data; the image control points serve as a geometric reference for subsequent multi-source data fusion.

[0016] Preferably, in step S1, collecting the drone oblique photography data, scanning point cloud data and panoramic video data of the target environment includes:

[0017] Using a drone equipped with a camera, perform multi-angle and multi-altitude aerial oblique photography of the target environment according to a predetermined route to obtain drone oblique photography data;

[0018] Use a handheld Lidar scanner to scan the ground area of ​​the target environment and obtain scanning point cloud data;

[0019] The panoramic video of the target environment is captured by a panoramic camera to record the texture information and detailed appearance of the target environment and obtain panoramic video data.

[0020] Preferably, in the process of acquiring the oblique photography data of the UAV, the A-Star search algorithm is used to plan the flight route of the UAV in the target environment in real time, and it is used as the predetermined route in real time to realize intelligent obstacle avoidance of the UAV.

[0021] Preferably, in step S1, the preprocessing includes:

[0022] The collected UAV oblique photography data, scanning point cloud data and panoramic video data are adjusted to the same size and resolution; the panoramic video data with the adjusted size and resolution is used as the pre-processed panoramic video data;

[0023] For the drone oblique photography data and scanned point cloud data whose size and resolution have been adjusted, spatial synchronization is performed based on the three-dimensional coordinates of all image control points, and time synchronization is performed based on the timestamp when the data was collected, to obtain the synchronized drone oblique photography data and scanned point cloud data;

[0024] Perform image denoising and enhancement processing on the synchronized UAV oblique photography data, and use the image control point data for geometric correction to obtain the pre-processed UAV oblique photography data;

[0025] Gaussian filtering is used to filter the synchronized scanned point cloud data, and the pre-processed panoramic video data is used to color the scanned point cloud data after point cloud filtering. The point cloud coloring is expressed as:

[0026]

[0027] in, is the i-th point in the scanned point cloud data after point cloud filtering; For point The color value of is the mapping function; is the jth frame in the preprocessed panoramic video data;

[0028] After the point cloud is colored, the normal vector of each point in the scanned point cloud data is calculated and saved to obtain the preprocessed scanned point cloud data.

[0029] Preferably, in step S2, the pre-processed UAV oblique photography data and the scanning point cloud data are preliminarily aligned using a feature point matching algorithm and uniformly aligned to the same coordinate system.

[0030] Preferably, step S3 comprises:

[0031] S31: Select N points from the UAV oblique photography data after preliminary alignment , N is a positive integer; and the texture features of each pixel in the preprocessed panoramic video data are collected at the same time;

[0032] The process of collecting texture features is as follows: first, extract the image frames of the preprocessed panoramic video and extract the image data of each frame from the video; second, use the scale-invariant feature transform (SIFT) algorithm to extract texture features and extract recognizable key points from the image frames of the panoramic video; finally, around each extracted key point, generate a feature vector based on its local neighborhood to describe the texture features of the area ;

[0033] S32: For each point , find the closest point in the scanned point cloud data after preliminary alignment , and form N point pairs ;

[0034] S33: Use the singular value decomposition algorithm to solve the following optimization problem and calculate the rigid body transformation matrix of the UAV oblique photography data and the scanned point cloud data after preliminary alignment. , so that the Euclidean distance between the two is minimized, and at the same time, the texture information difference between the two and the preprocessed panoramic video data is minimized:

[0035]

[0036] in, is the rotation matrix, is the translation vector; is the balance coefficient; is the midpoint of the preprocessed panoramic video data Texture features of corresponding pixels; is the midpoint of the preprocessed panoramic video data Texture features of corresponding pixels; represents the square of the norm;

[0037] S34: Using the rigid body transformation matrix Update the position of the drone oblique photography data after preliminary alignment;

[0038] S35: repeating steps S31 to S34 until the preset error function converges, completing the refined registration, and obtaining the multi-source data registration result;

[0039] The preset error function is:

[0040]

[0041]

[0042] in, is the value of the error function; For point pair The weight of is a parameter used to control the influence of texture features on weights.

[0043] Preferably, the step S4 comprises:

[0044] S41: performing downsampling and voxel grid filtering processing on the multi-source data registration result to obtain a low-resolution sparse point cloud; selecting a number of key areas from the multi-source data registration result, and saving the point clouds of the key areas together as a high-resolution dense point cloud;

[0045] S42: performing normal vector smoothing and outlier removal on the sparse point cloud and the dense point cloud respectively to obtain denoised sparse point cloud and dense point cloud;

[0046] S43: Obtain the normal vector field of the corresponding point cloud based on the normal vectors of each point in the denoised sparse point cloud and dense point cloud and , construct and solve the following Poisson equation in turn to obtain the optimal surface function corresponding to the sparse point cloud and the dense point cloud and :

[0047]

[0048]

[0049] in, and express and The Laplace operator of ; Indicates the horizontal coordinate of the point;

[0050] According to the optimal surface function and generating a first and a second three-dimensional model, wherein the first three-dimensional model is used to represent the overall structure of the target environment, and the second three-dimensional model is used to represent the detailed features of the key area;

[0051] S44: performing trilinear interpolation and weighted fusion on the first and second three-dimensional models to generate the high-precision three-dimensional model of the target environment.

[0052] Preferably, in step S42, normal vector smoothing is performed on each point in the sparse point cloud and the dense point cloud according to the following formula:

[0053]

[0054] in, is the i-th point in the point cloud; For point Smoothed normal vector; For point The neighborhood point set of ; For point and Point The weighting coefficient of

[0055] For each point in the sparse point cloud and the dense point cloud after normal vector smoothing , respectively calculate the average normal vector of its neighborhood point set The deviation between , the deviation is expressed as: ,in represents the Euclidean distance;

[0056] The deviation Greater than the preset rejection threshold point Remove them as outliers and complete denoising.

[0057] The present invention also provides a refined three-dimensional modeling system based on air-ground multi-source data fusion, based on the above-mentioned refined three-dimensional modeling method based on air-ground multi-source data fusion, comprising:

[0058] Preprocessing unit: used to collect UAV oblique photography data, scanning point cloud data and panoramic video data of the target environment, and preprocess them respectively;

[0059] Rough registration unit: used to perform preliminary alignment on the pre-processed UAV oblique photography data and the scanned point cloud data, and obtain the pre-aligned UAV oblique photography data and the scanned point cloud data;

[0060] Fine registration unit: used to adopt the improved ICP algorithm, introduce the texture information of the pre-processed panoramic video data, perform fine registration on the preliminarily aligned UAV oblique photography data and scanning point cloud data, and obtain the multi-source data registration result;

[0061] 3D modeling unit: It is used to adopt the improved Poisson surface reconstruction algorithm, introduce the normal vector smoothing and outlier removal mechanism, and then based on the multi-scale reconstruction strategy, perform multi-scale reconstruction on the multi-source data registration results to generate a high-precision 3D model of the target environment and complete the modeling.

[0062] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0063] The present invention provides a refined three-dimensional modeling method and system based on air-ground multi-source data fusion. First, unmanned aerial vehicle oblique photography data, scanning point cloud data and panoramic video data of a target environment are collected and preprocessed respectively; then, the preprocessed unmanned aerial vehicle oblique photography data and scanning point cloud data are preliminarily aligned to obtain the preliminarily aligned unmanned aerial vehicle oblique photography data and scanning point cloud data; then, an improved ICP algorithm is adopted to introduce texture information of the preprocessed panoramic video data, and the preliminarily aligned unmanned aerial vehicle oblique photography data and scanning point cloud data are finely registered to obtain a multi-source data registration result; finally, an improved Poisson surface reconstruction algorithm is adopted to introduce a normal vector smoothing and abnormal point elimination mechanism, and then, based on a multi-scale reconstruction strategy, the multi-source data registration result is multi-scale reconstructed to generate a high-precision three-dimensional model of the target environment, and the modeling is completed;

[0064] The present invention innovatively combines three data sources: drone oblique photography, scanning point cloud data and panoramic video data. By fusing multi-source data, it solves the problem of missing details and incomplete coverage caused by a single data source in the modeling process of complex buildings. Multi-source data fusion technology can fully capture the geometric form and detailed features of buildings in complex scenes. It is suitable for complex scenes such as urban villages and old buildings, and realizes all-round and detailed three-dimensional modeling.

[0065] Secondly, the present invention improves the traditional ICP (Iterative Closest Point) algorithm, adopts a registration strategy that combines geometric information with panoramic video texture information, and optimizes the geometric registration process of point clouds by introducing texture constraints; the improved ICP algorithm not only reduces the influence of noise points and abnormal points through a dynamic weighting mechanism, but also achieves high-precision registration of UAV oblique images and scanned point cloud data by minimizing geometric errors and texture differences; the algorithm improves the accuracy and robustness of registration, and is particularly suitable for application in scenes with complex architectural environments and rich details;

[0066] In addition, in the process of reconstructing the three-dimensional model, the present invention proposes a multi-scale reconstruction strategy based on the Poisson surface reconstruction algorithm, which ensures that the overall outline of the building is generated on the sparse point cloud and the complex local details are finely reconstructed on the dense point cloud; at the same time, the influence of noise is effectively reduced through the normal vector smoothing and outlier removal mechanism, and the smoothness and details of the generated three-dimensional model are further improved; the improved Poisson algorithm can generate a three-dimensional model that is both highly accurate and easy to operate in a complex building environment, thereby meeting the needs of urban planning, architectural design, cultural relics protection and other fields for fine three-dimensional modeling in complex building environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1This is a flow chart of a refined three-dimensional modeling method based on multi-source data fusion of air and ground provided in Example 1.

[0068] Figure 2 This is a specific implementation flow chart of a refined three-dimensional modeling method based on multi-source data fusion of air and ground provided in Example 2.

[0069] Figure 3 This is a flowchart of the intelligent planning of drone routes provided in Example 2.

[0070] Figure 4 These are the global and local images of the 3D reconstruction effect of a densely built-up area in a certain region provided in Example 2.

[0071] Figure 5 This is a comparison diagram of the original oblique photography model provided in Example 2 and the model effect after the air-ground integration.

[0072] Figure 6 This is a structural diagram of a refined three-dimensional modeling system based on air-ground multi-source data fusion provided in Example 3. DETAILED DESCRIPTION

[0073] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0074] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0075] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0076] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0077] Example 1

[0078] like Figure 1 As shown, this embodiment provides a refined 3D modeling method based on air-ground multi-source data fusion, comprising the following steps:

[0079] S1: Collect the UAV oblique photography data, scanning point cloud data and panoramic video data of the target environment, and pre-process them respectively;

[0080] S2: Preliminarily align the preprocessed UAV oblique photography data and the scanned point cloud data to obtain the preliminarily aligned UAV oblique photography data and the scanned point cloud data;

[0081] S3: Using the improved ICP algorithm, the texture information of the pre-processed panoramic video data is introduced to perform fine registration on the preliminarily aligned UAV oblique photography data and scanning point cloud data to obtain the multi-source data registration result;

[0082] S4: An improved Poisson surface reconstruction algorithm is used to introduce normal vector smoothing and outlier removal mechanisms. Then, based on a multi-scale reconstruction strategy, multi-source data registration results are reconstructed at multiple scales to generate a high-precision three-dimensional model of the target environment and complete the modeling.

[0083] In the specific implementation process, with the rapid development of smart cities, digital modeling and virtual reality (VR) technology, the market demand for fine 3D models of buildings continues to grow; especially in the fields of urban renewal, building safety inspection, post-disaster reconstruction and cultural relics protection, the demand for fine 3D modeling of complex buildings has become more and more prominent; the current market needs a technology that can cope with the needs of building modeling in high-density and complex environments, which can not only quickly generate large-scale 3D models, but also accurately capture the detailed features of buildings; in addition, with the advancement of drone technology and handheld Lidar scanning technology, the research on fusion of multi-source data for 3D modeling has gradually become a hot direction; the combination of multi-source data can play their respective advantages in different scenarios and achieve more comprehensive and accurate modeling results; therefore, the development of a modeling technology based on the fusion of drone oblique photography and handheld Lidar point cloud has become an important trend in solving the problem of 3D modeling in complex building environments;

[0084] In view of the demand for refined 3D modeling of complex building environments and the limitations of traditional modeling technologies, this embodiment proposes an innovative air-ground integrated 3D modeling method, which aims to achieve efficient fusion of multi-source data and high-precision 3D model generation by integrating drone oblique photography, handheld Lidar scanning and panoramic video data, so as to meet the demand for refined 3D modeling in complex building environments in the fields of urban planning, architectural design, and cultural relics protection.

[0085] First, the drone oblique photography data, scanning point cloud data and panoramic video data of the target environment are collected and preprocessed respectively. The preprocessing mainly aligns the data collected from multiple data sources in time and space to ensure the time and space consistency of data collection, and performs preliminary processing on multi-source data, which provides a solid foundation for subsequent multi-source data fusion and precise modeling.

[0086] Next, multi-source data registration is performed. Registration is a key step to unify the drone image data and Lidar point cloud data into the same coordinate system. This method uses a two-stage registration method, namely, preliminary coarse registration and improved ICP (iterative closest point) algorithm fine registration. In the fine registration stage, not only the geometric distance is considered, but also the texture information of the panoramic video is combined for multi-dimensional constraints to ensure high-precision point cloud fusion and obtain multi-source data registration results.

[0087] The traditional ICP algorithm performs registration by minimizing the Euclidean distance between the source point cloud and the target point cloud, but it is prone to local optimality and excessive initial error when facing complex data sets. Therefore, the improved ICP introduces the texture information of panoramic video, adds texture constraints and error optimization mechanisms to improve the registration accuracy and robustness.

[0088] Finally, an improved Poisson surface reconstruction algorithm is used to introduce normal vector smoothing and outlier removal mechanisms. Then, based on a multi-scale reconstruction strategy, multi-source data registration results are reconstructed at multiple scales to generate a high-precision three-dimensional model of the target environment and complete the modeling.

[0089] The Poisson surface reconstruction algorithm is a 3D reconstruction method based on the point cloud normal vector field, which is widely used in the surface generation of point cloud data. It reconstructs a continuous and smooth 3D surface by converting the point cloud normal vector field into a surface gradient field using the Poisson equation. However, the traditional Poisson algorithm is prone to blurring and detail loss when processing large-scale point clouds, especially in complex building structures, where the detail expression is insufficient. Therefore, this method optimizes the Poisson surface reconstruction, including multi-scale reconstruction strategy and denoising processing, thereby improving the precision and efficiency of 3D reconstruction.

[0090] The final generated 3D model can be exported to multiple formats (OSGB, OBJ, etc.) to support a wide range of application scenarios, such as BIM systems, urban planning, digital twins, etc.

[0091] This method solves the problem that traditional single modeling methods cannot take into account both wide-area coverage and local precision in complex building environments by fusing drone oblique photography, handheld Lidar scanning and panoramic video data. It ensures the generation of high-precision 3D models on a global scale and effectively enriches the surface details of the models. This method is particularly suitable for complex building scenes such as urban villages and old urban areas. It can quickly generate 3D models with wide coverage and rich details, significantly improving modeling efficiency. The high-precision 3D model finally generated supports output in multiple formats and is suitable for urban planning, architectural design, post-disaster reconstruction, smart city construction and other fields.

[0092] Example 2

[0093] like Figure 2 As shown, this embodiment provides a refined 3D modeling method based on air-ground multi-source data fusion, comprising the following steps:

[0094] S1: Collect the UAV oblique photography data, scanning point cloud data and panoramic video data of the target environment, and pre-process them respectively;

[0095] S2: Preliminarily align the preprocessed UAV oblique photography data and the scanned point cloud data to obtain the preliminarily aligned UAV oblique photography data and the scanned point cloud data;

[0096] S3: Using the improved ICP algorithm, the texture information of the pre-processed panoramic video data is introduced to perform fine registration on the preliminarily aligned UAV oblique photography data and scanning point cloud data to obtain the multi-source data registration result;

[0097] S4: An improved Poisson surface reconstruction algorithm is used to introduce normal vector smoothing and outlier removal mechanisms. Then, based on a multi-scale reconstruction strategy, multi-source data registration results are reconstructed at multiple scales to generate a high-precision 3D model of the target environment and complete the modeling.

[0098] Before step S1, the following steps are also included:

[0099] Evenly arrange a number of image control points in the target environment, measure the three-dimensional coordinates of all image control points, and obtain image control point data; the image control points serve as a geometric reference for subsequent multi-source data fusion;

[0100] In step S1, collecting the drone oblique photography data, scanning point cloud data and panoramic video data of the target environment includes:

[0101] Using a drone equipped with a camera device, the target environment is photographed from multiple angles and multiple heights in the air according to a predetermined route to obtain drone oblique photography data; in this process, the flight route of the drone in the target environment is planned in real time using the A-Star search algorithm, and is used as the predetermined route in real time to achieve intelligent obstacle avoidance of the drone;

[0102] Use a handheld Lidar scanner to scan the ground area of ​​the target environment and obtain scanning point cloud data;

[0103] The panoramic video of the target environment is captured by a panoramic camera to record the texture information and detailed appearance of the target environment and obtain panoramic video data;

[0104] In step S1, the preprocessing includes:

[0105] The collected UAV oblique photography data, scanning point cloud data and panoramic video data are adjusted to the same size and resolution; the panoramic video data with the adjusted size and resolution is used as the pre-processed panoramic video data;

[0106] For the drone oblique photography data and scanned point cloud data whose size and resolution have been adjusted, spatial synchronization is performed based on the three-dimensional coordinates of all image control points, and time synchronization is performed based on the timestamp when the data was collected, to obtain the synchronized drone oblique photography data and scanned point cloud data;

[0107] Perform image denoising and enhancement processing on the synchronized UAV oblique photography data, and use the image control point data for geometric correction to obtain the pre-processed UAV oblique photography data;

[0108] Gaussian filtering is used to filter the synchronized scanned point cloud data, and the pre-processed panoramic video data is used to color the scanned point cloud data after point cloud filtering. The point cloud coloring is expressed as:

[0109]

[0110] in, is the i-th point in the scanned point cloud data after point cloud filtering; For point The color value of is the mapping function; is the jth frame in the preprocessed panoramic video data;

[0111] Calculate and save the normal vector of each point in the scanned point cloud data after coloring the point cloud, and obtain the pre-processed scanned point cloud data;

[0112] In step S2, the pre-processed UAV oblique photography data and the scanning point cloud data are preliminarily aligned using a feature point matching algorithm and uniformly aligned to the same coordinate system;

[0113] The step S3 comprises:

[0114] S31: Select N points from the UAV oblique photography data after preliminary alignment , N is a positive integer; and the texture features of each pixel in the preprocessed panoramic video data are collected at the same time;

[0115] S32: For each point , find the closest point in the scanned point cloud data after preliminary alignment , and form N point pairs ;

[0116] S33: Use the singular value decomposition algorithm to solve the following optimization problem and calculate the rigid body transformation matrix of the UAV oblique photography data and the scanned point cloud data after preliminary alignment. , so that the Euclidean distance between the two is minimized, and at the same time, the texture information difference between the two and the preprocessed panoramic video data is minimized:

[0117]

[0118] in, is the rotation matrix, is the translation vector; is the balance coefficient; is the midpoint of the preprocessed panoramic video data Texture features of corresponding pixels; is the midpoint of the preprocessed panoramic video data Texture features of corresponding pixels; represents the square of the norm;

[0119] S34: Using the rigid body transformation matrix Update the position of the drone oblique photography data after preliminary alignment;

[0120] S35: repeating steps S31 to S34 until the preset error function converges, completing the refined registration, and obtaining the multi-source data registration result;

[0121] The preset error function is:

[0122]

[0123]

[0124] in, is the value of the error function; For point pair The weight of is a parameter used to control the effect of texture features on weight;

[0125] The step S4 comprises:

[0126] S41: performing downsampling and voxel grid filtering processing on the multi-source data registration result to obtain a low-resolution sparse point cloud; selecting a number of key areas from the multi-source data registration result, and saving the point clouds of the key areas together as a high-resolution dense point cloud;

[0127] S42: performing normal vector smoothing and outlier removal on the sparse point cloud and the dense point cloud respectively to obtain denoised sparse point cloud and dense point cloud;

[0128] S43: Obtain the normal vector field of the corresponding point cloud based on the normal vectors of each point in the denoised sparse point cloud and dense point cloud and , construct and solve the following Poisson equation in turn to obtain the optimal surface function corresponding to the sparse point cloud and the dense point cloud and :

[0129]

[0130]

[0131] in, and express and The Laplace operator of ; Indicates the horizontal coordinate of the point;

[0132] According to the optimal surface function and generating a first and a second three-dimensional model, wherein the first three-dimensional model is used to represent the overall structure of the target environment, and the second three-dimensional model is used to represent the detailed features of the key area;

[0133] S44: performing trilinear interpolation and weighted fusion on the first and second three-dimensional models to generate the high-precision three-dimensional model of the target environment;

[0134] In step S42, normal vector smoothing is performed on each point in the sparse point cloud and the dense point cloud according to the following formula:

[0135]

[0136] in, is the i-th point in the point cloud; For point Smoothed normal vector; For point The neighborhood point set of ; For point and Point The weighting coefficient of

[0137] For each point in the sparse point cloud and the dense point cloud after normal vector smoothing , respectively calculate the average normal vector of its neighborhood point set The deviation between , the deviation is expressed as: ,in represents the Euclidean distance;

[0138] The deviation Greater than the preset rejection threshold point Remove them as outliers and complete denoising.

[0139] In the specific implementation process, this embodiment provides an air-ground integrated 3D modeling method for complex building environments, which realizes the generation of high-precision 3D models of complex buildings by integrating drone oblique photography, handheld Lidar scanning and panoramic video data, combined with an improved ICP algorithm and Poisson surface reconstruction algorithm; the following takes the 3D modeling of a building in a village in a certain city as an example to explain the specific implementation process of this method in detail;

[0140] First, multi-source data collection is carried out. In the same time period, drone oblique photography, handheld Lidar scanning and panoramic video data are synchronously acquired to ensure the temporal and spatial consistency of data collection, thus providing guarantee for subsequent multi-source data fusion and accurate modeling.

[0141] Before data collection, several image control points are evenly arranged around the target urban village buildings, and the three-dimensional coordinate information of these image control points is measured using high-precision GPS equipment (such as RTK). The image control points serve as the geometric reference for subsequent data fusion. These points are set at key locations of the buildings, such as building corners, surrounding roads and other reference structures, and ensure that the layout locations are open and unobstructed with good communication signals. Through these image control points, the subsequent multi-source data (UAV images and Lidar point clouds) are accurately aligned.

[0142] Then, we used the DJI M300 drone equipped with a Ruibo five-lens oblique photography camera to take oblique photography of the building from multiple angles and heights according to the predetermined route;

[0143] In complex environments such as urban villages, where building heights vary and building spacing is relatively narrow, the traditional fixed route planning method (based on a fixed flight altitude) is unable to meet the needs of fine measurement of key objects and there is a risk of collision with obstacles.

[0144] Therefore, when planning the route, this embodiment comprehensively considers the complexity of the building and incorporates the A-Star intelligent obstacle avoidance algorithm to ensure that the drone can safely avoid obstacles during flight; Figure 3 As shown in the figure, the specific implementation process is as follows: First, a drone equipped with an infrared sensor is used to collect comprehensive terrain data of the target area, and the data type obtained is fine three-dimensional point cloud data; then, these three-dimensional point cloud data are converted into usable terrain and obstacle information, and an accurate elevation matrix is ​​constructed based on this; on this basis, the obstacle avoidance factor is ignored and the initial coverage path of the target area is calculated; then, the path that may have collisions is replanned, and the A-Star algorithm is used to search for the optimal flight path to ensure that the drone can automatically bypass obstacles;

[0145] The intelligent obstacle avoidance algorithm performs trajectory replanning based on the A-Star algorithm. The algorithm comprehensively considers the actual cost and estimated cost of the trajectory point, calculates the global total cost G of the current node, and selects the child node with the smallest cost value for expansion until the target point is searched, thereby achieving the optimal path planning; the calculation formula of the global total cost G is:

[0146]

[0147] In the formula, For the current node The global total cost of is the actual cost function, that is, the current node The actual flight cost is taken as the Euclidean distance; is the heuristic function, that is, the current node The estimated flight cost is Manhattan distance. During the route planning process, the drone will detect obstacles within the flight range in real time. Once an obstacle is found, the re-planning starting point and end point will be determined, and the A-Star algorithm will be called for trajectory re-planning to automatically bypass obstacles (such as poles, trees, building edges, etc.). The introduction of intelligent obstacle avoidance algorithms not only improves the integrity of data collection, but also reduces the risk of drones flying in complex environments.

[0148] The images collected by the drone should fully cover the facade, top and surrounding environment of the building, forming a set of image data sets with geographic coordinates; during the actual flight process, the drone flies along the intelligently planned route, outputs a shooting point every preset sampling distance, and the resolution of the image is 3 cm, ensuring that the drone image can capture the overall structure and environmental information of the building;

[0149] A handheld Lidar scanner is used to perform supplementary scanning of the bottom details of the building, especially for detailed point cloud collection of areas of the building that are difficult to be covered by drone images (such as the facade of the first floor of the building, shadow areas, etc.); in this embodiment, the Beijing Digital Green Earth LiGrip H120 handheld Lidar scanner is used for supplementary scanning on the first floor of the building, shadow areas and complex geometric areas; during the scanning process, the Lidar point cloud data collection density is 300,000 points per second to ensure the complete recording of complex details and avoid data loss due to occlusion of drone images;

[0150] Use a panoramic camera to shoot panoramic video of the building's surrounding environment with a video resolution of 5760×2880@30fps to capture the building's texture details and visual information of the surrounding environment; the panoramic video data will provide an important basis for the subsequent 3D model texture coloring;

[0151] Then the collected multi-source data is preprocessed;

[0152] Firstly, the collected UAV oblique photography data, scanning point cloud data and panoramic video data are adjusted to the same size and resolution; the panoramic video data with adjusted size and resolution is directly used as the pre-processed panoramic video data;

[0153] For the drone oblique photography data and scanned point cloud data whose size and resolution have been adjusted, spatial synchronization is performed based on the three-dimensional coordinates of all image control points, and time synchronization is performed based on the timestamp when the data was collected, to obtain the synchronized drone oblique photography data and scanned point cloud data;

[0154] In this embodiment, the process of spatial synchronization based on image control point data includes: first, finding the corresponding image control point positions in the drone oblique photography data and the scanned point cloud data; then, using the known three-dimensional coordinates of the image control points and their corresponding coordinates in each data set, calculating the spatial transformation matrix, including the rotation matrix and the translation vector; finally, using the calculated spatial transformation matrix, unifying all point cloud data in the drone oblique photography data and the ground Lidar scanning data into the same spatial coordinate system based on the image control points, thereby achieving data synchronization in space; since the panoramic video data already contains spatial information, it can be directly used in the subsequent texture mapping and registration process after the size and resolution are adjusted;

[0155] Perform image denoising and enhancement processing on the synchronized UAV oblique photography data to eliminate noise and distortion and ensure that the image data is clear and stable; perform geometric correction using image control point data to ensure that the image is consistent with the actual geographic coordinate system and obtain the pre-processed UAV oblique photography data;

[0156] Use Gaussian filtering to filter the synchronized scanned point cloud data to ensure the quality and accuracy of the point cloud data;

[0157] The image information in the panoramic video is used to finely color the point cloud data and provide a basis for the texture mapping of the subsequent model. In this embodiment, multiple video frames are selected from the pre-processed panoramic video data, and the scanned point cloud data after point cloud filtering is colorized to improve the realism and visual effect of the subsequent three-dimensional model. The point cloud coloring is expressed as:

[0158]

[0159] in, is the i-th point in the scanned point cloud data after point cloud filtering; For point The color value of is a mapping function used to map the color information of the panoramic video data to each point of the point cloud data; is the jth frame in the preprocessed panoramic video data;

[0160] After calculating the point cloud coloring, the normal vector of each point in the scanned point cloud data is saved. The normal vector of each point is estimated based on the geometric relationship of the neighboring points to ensure that the data has high-precision normal vector information, providing support for subsequent 3D model generation and registration; thus obtaining the pre-processed scanned point cloud data;

[0161] Then, multi-source data registration is performed;

[0162] Firstly, the UAV image and the handheld Lidar point cloud data are roughly aligned according to the GPS coordinates. The feature points in the image are matched with the feature points in the Lidar point cloud. The initial rigid body transformation matrix T is estimated by the feature point matching algorithm, so that the image and the point cloud data are in the same coordinate system.

[0163] After the initial alignment, the improved ICP (Iterative Closest Point) algorithm is used to further improve the registration accuracy of multi-source data. The ICP algorithm combines the texture information in the panoramic video on the basis of traditional geometric error to minimize both geometric distance and texture difference.

[0164] The traditional ICP algorithm performs registration by minimizing the Euclidean distance between the source point cloud and the target point cloud, but it is prone to local optimality and excessive initial error when facing complex data sets. To this end, the improved ICP introduces the texture information of panoramic videos, adds texture constraints and error optimization mechanisms to improve registration accuracy and robustness;

[0165] The steps of the traditional ICP algorithm mainly include:

[0166] 1) Select a point set from the source point cloud And select point sets from the target point cloud , these point sets represent the 3D points in the two datasets;

[0167] 2) For each point ∈P, find the nearest point in the target point cloud Q ∈Q, and form a corresponding point pair set ( , );

[0168] 3) Calculate the rigid body transformation matrix T = (R, t) of the source point cloud and the target point cloud so that the Euclidean distance between the two point clouds is minimized:

[0169]

[0170] Where R is the rotation matrix and t is the translation vector. This optimization problem can usually be solved by SVD (singular value decomposition);

[0171] Apply the solution of the rigid body transformation matrix to the source point cloud, update the point cloud position, and continue iterating until convergence (that is, the error change is less than the threshold);

[0172] The improved ICP algorithm has made the following improvements based on the above steps:

[0173] 1) Introducing panoramic video texture information for multimodal matching;

[0174] In addition to the geometric distance constraints, the improved ICP algorithm incorporates the texture features in the panoramic video during the registration process, which enhances the accuracy of point cloud registration;

[0175] The process of collecting texture features is as follows: first, extract the image frames of the preprocessed panoramic video and extract the image data of each frame from the video; second, use the SIFT algorithm to extract texture features and extract recognizable key points from the image frames of the panoramic video; finally, around each extracted key point, generate a feature vector based on its local neighborhood to describe the texture features of the area. ;

[0176] Assume that each point The texture feature of the corresponding pixel point in the panoramic video is , registration not only considers the geometric distance, but also minimizes the difference in texture features between the point cloud and the image:

[0177]

[0178] in, and are the rotation matrix and translation vector respectively, is a point in the Lidar point cloud, is a point in the drone image, and is its texture feature value, To control the weight parameter of texture constraint;

[0179] 2) Dynamic weighted closest point matching;

[0180] In each iteration, in order to enhance the robustness to noise and outliers, the improved ICP introduces weighted nearest point matching; the specific approach is: for each matching point pair ( ) Calculate a weight , which is adjusted based on geometric distance and texture difference:

[0181]

[0182] in, It is a parameter used to control the influence of texture features on weights. The introduction of weights makes the point pairs far away from the correct matching pairs (i.e., point pairs with large distance or large texture difference) have less influence in the optimization process, thereby reducing their interference with the registration results.

[0183] 3) Error optimization and convergence

[0184] In order to improve the accuracy of registration, the improved ICP algorithm also introduces a residual-based global error minimization strategy; this strategy minimizes the error between the source point cloud and the target point cloud by optimizing the residual function:

[0185]

[0186] In each iteration, the algorithm updates the transformation matrix , until the error function Convergence (i.e. the residual change is less than the preset threshold); at the same time, the weights are dynamically adjusted according to the distance and texture differences Ensure high-quality registration results;

[0187] Then, the three-dimensional model is reconstructed. On the registered point cloud data, the optimized Poisson surface reconstruction algorithm is used to generate the three-dimensional model of the building. This method uses the optimized Poisson surface reconstruction algorithm to generate the three-dimensional model. Compared with the traditional algorithm, this method makes many improvements to the Poisson algorithm to ensure the high accuracy and detail retention of the model.

[0188] Poisson surface reconstruction is a 3D reconstruction method based on point cloud normal vector field, which is widely used in surface generation of point cloud data. It reconstructs a continuous and smooth 3D surface by converting the point cloud normal vector field into a surface gradient field using the Poisson equation. However, the traditional Poisson algorithm is prone to blurring and detail loss when processing large-scale point clouds, especially in complex building structures, where the detail expression is insufficient. Therefore, the present invention optimizes Poisson surface reconstruction, including multi-scale reconstruction strategy and noise processing, to improve the precision and efficiency of 3D reconstruction.

[0189] The basic principle of Poisson surface reconstruction is to convert the normal vector field N(p) of the point cloud into a surface gradient field, and then construct a three-dimensional surface by solving the Poisson equation; specifically, the mathematical form of the Poisson equation is:

[0190]

[0191] in, is the Laplace operator of the target three-dimensional surface to be solved; is the normal vector field calculated from the point cloud data; represents the divergence of the normal vector field, defined as the geometry of the input point cloud; this equation generates a smooth 3D surface by globally minimizing the divergence of the point cloud data; the process of solving this equation can be expressed as the following optimization problem:

[0192]

[0193] That is, find the surface function , so that its gradient field ∇f is closest to the normal vector field N(p); the Poisson reconstruction algorithm generates a surface by interpolating the divergence of the normal vector field, and finally obtains the reconstructed surface by solving the optimization problem;

[0194] On this basis, the optimized Poisson surface reconstruction algorithm has made the following improvements: normal vector smoothing and outlier removal mechanisms are introduced, followed by multi-scale reconstruction;

[0195] Specifically, the multi-source data registration result (original point cloud) is first downsampled and voxel grid filtered (i.e., the point cloud space is divided into cubic grids, and only one representative point is retained in each grid) to reduce the number of points and obtain a low-resolution sparse point cloud; a number of key areas (such as architectural details, edges, etc.) are selected from the multi-source data registration result, and more sampling points are retained (all are retained in this embodiment) to ensure fineness, and all point clouds in each key area are saved together as a high-resolution dense point cloud;

[0196] In order to deal with the noise and outliers in the point cloud data, the optimized Poisson algorithm introduces a noise processing and outlier removal mechanism, and performs normal vector smoothing and outlier removal on the sparse point cloud and dense point cloud respectively to obtain the denoised sparse point cloud and dense point cloud, as follows:

[0197] 1) Normal vector smoothing: The normal vector is smoothed using the local weighted average method to reduce the normal vector anomaly caused by noise points. The normal vector is smoothed for each point in the sparse point cloud and the dense point cloud according to the following formula:

[0198]

[0199] in, is the i-th point in the point cloud; For point Smoothed normal vector; For point The neighborhood point set of ; For point and Point The weighting coefficient of

[0200] 2) Outlier removal: For each point in the sparse point cloud and dense point cloud after normal vector smoothing , respectively calculate the average normal vector of its neighborhood point set The deviation between , the deviation is expressed as: ,in represents the Euclidean distance;

[0201] The deviation Greater than the preset rejection threshold point Remove them as outliers and complete denoising;

[0202] It is worth mentioning that the outlier removal uses the smoothed normal vector. This is because the normal vector can more accurately reflect the local geometric features of the data point after smoothing, thereby improving the accuracy of outlier identification. By comparing the deviation between the smoothed normal vector and the average normal vector of its neighborhood point set, it is possible to more reliably determine which points are outliers and remove them, thereby improving the denoising effect.

[0203] In order to ensure that the detailed areas can be reconstructed finely while maintaining the global structure, the optimized Poisson algorithm introduces a multi-scale reconstruction strategy; specifically, the reconstruction process is divided into multiple levels:

[0204] 1) Perform a rough surface reconstruction on the sparse point cloud to capture the overall outline of the building;

[0205] 2) Further fine reconstruction is performed on the dense point cloud, focusing on the detailed areas of the building;

[0206] That is, a rough reconstruction is first performed on the low-resolution point cloud to quickly obtain the overall structure and reduce the computational burden; then, the model details are further optimized and refined on the high-resolution point cloud to retain the complex details and texture features of the building, thereby achieving the goal of balancing computational efficiency and model accuracy;

[0207] Get the normal vector field of the corresponding point cloud according to the normal vector of each point in the denoised sparse point cloud and dense point cloud and , construct and solve the following Poisson equation in turn to obtain the optimal surface function corresponding to the sparse point cloud and the dense point cloud and :

[0208]

[0209]

[0210] in, and express and The Laplace operator of ; Indicates the horizontal coordinate of the point;

[0211] According to the optimal surface function and Generate the first and second 3D models, the first 3D model is used to represent the overall structure of the target environment, and the second 3D model is used to represent the detailed features of the key areas; the reconstruction results of different scales (first and second 3D models) are combined through the commonly used trilinear interpolation and weighted fusion to finally obtain a complete high-precision model, ensuring the smoothness of the overall surface and the retention of local details;

[0212] The final generated 3D model can be exported to multiple formats (OSGB, OBJ, etc.) to support a wide range of application scenarios, such as BIM systems, urban planning, digital twins, etc.

[0213] like Figure 4 The figure shows the 3D reconstruction effect of a densely built area in a certain area obtained by using this method; it can be seen that this method can fully capture the geometric form and detailed features of buildings in complex scenes, and the 3D modeling is comprehensive and detailed;

[0214] like Figure 5 The figure shows the comparison between the modeling effect using this method and the modeling effect using only the oblique photography of the UAV. It can be seen that the details and smoothness of this method are better;

[0215] This method innovatively combines three data sources: drone oblique photography, scanning point cloud data, and panoramic video data. By fusing multi-source data, it solves the problem of missing details and incomplete coverage caused by a single data source in the modeling process of complex buildings. Multi-source data fusion technology can fully capture the geometric form and detailed features of buildings in complex scenes. It is suitable for complex scenes such as urban villages and old buildings, and realizes all-round and detailed 3D modeling.

[0216] Secondly, this method improves the traditional ICP (Iterative Closest Point) algorithm, adopts a registration strategy that combines geometric information with panoramic video texture information, and optimizes the geometric registration process of point clouds by introducing texture constraints; the improved ICP algorithm not only reduces the influence of noise points and outliers through a dynamic weighting mechanism, but also achieves high-precision registration of UAV oblique images and scanned point cloud data by minimizing geometric errors and texture differences; the algorithm improves the accuracy and robustness of registration, and is particularly suitable for application in scenes with complex architectural environments and rich details;

[0217] In addition, during the reconstruction of the three-dimensional model, this method proposes a multi-scale reconstruction strategy based on the Poisson surface reconstruction algorithm, which ensures that the overall outline of the building is generated on the sparse point cloud and the complex local details are finely reconstructed on the dense point cloud; at the same time, the influence of noise is effectively reduced through normal vector smoothing and outlier removal mechanisms, further improving the smoothness and details of the generated three-dimensional model; the improved Poisson algorithm can generate a three-dimensional model that is both highly accurate and easy to operate in a complex building environment, thereby meeting the needs of urban planning, architectural design, cultural relics protection and other fields for fine three-dimensional modeling in complex building environments.

[0218] Example 3

[0219] like Figure 6 As shown, this embodiment provides a refined 3D modeling system based on air-ground multi-source data fusion, based on the above-mentioned refined 3D modeling method based on air-ground multi-source data fusion, including:

[0220] Preprocessing unit 301: used to collect drone oblique photography data, scanning point cloud data and panoramic video data of the target environment, and perform preprocessing respectively;

[0221] A coarse registration unit 302 is used to perform preliminary alignment on the pre-processed UAV oblique photography data and the scanned point cloud data, and obtain the pre-aligned UAV oblique photography data and the scanned point cloud data;

[0222] The fine registration unit 303 is used to adopt the improved ICP algorithm, introduce the texture information of the pre-processed panoramic video data, perform fine registration on the drone oblique photography data and the scanning point cloud data after preliminary alignment, and obtain the multi-source data registration result;

[0223] The three-dimensional modeling unit 304 is used to adopt an improved Poisson surface reconstruction algorithm, introduce a normal vector smoothing and outlier removal mechanism, and then based on a multi-scale reconstruction strategy, perform multi-scale reconstruction on the multi-source data registration result to generate a high-precision three-dimensional model of the target environment to complete the modeling.

[0224] In the specific implementation process, the pre-processing unit 301 uses a drone to perform oblique photography in a target complex building environment (such as a village in the city) to obtain oblique photography data of the target environment; uses a handheld Lidar scanner to perform additional scanning on the bottom of the building and complex areas to obtain point cloud data; uses a panoramic camera to capture the texture details of the building and visual information of the surrounding environment; then, the drone oblique photography data is denoised and corrected to ensure that the image is clear and consistent with the actual geographic coordinate system; the handheld Lidar scanning point cloud data is filtered and denoised to optimize the data structure; key frames are selected from the panoramic video, the point cloud data is colored, and basic preparations for texture mapping are performed; the normal vector of the point cloud data is calculated to provide support for subsequent three-dimensional reconstruction;

[0225] Then, the coarse registration unit 302 performs preliminary coarse registration based on the GPS coordinates so that the drone image and the handheld Lidar point cloud data are in the same coordinate system;

[0226] Then the fine registration unit 303 uses the improved ICP algorithm to perform fine registration, combines the texture information of the panoramic video to perform multi-dimensional constraints, and improves the registration accuracy;

[0227] Afterwards, the 3D modeling unit 304 generates a 3D model based on the registered point cloud data using an optimized Poisson surface reconstruction algorithm; introduces a normal vector smoothing and outlier point removal mechanism to reduce the impact of noise and maintain the smoothness and details of the model; and then uses a multi-scale reconstruction strategy to ensure fine reconstruction of the global structure and detail areas;

[0228] Finally, the generated 3D model is exported in a standard format and applied to architectural surveying and mapping, virtual reality, urban planning and other fields to meet the needs of fine 3D modeling in complex architectural environments.

[0229] The same or similar reference numerals correspond to the same or similar components;

[0230] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0231] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A refined 3D modeling method based on multi-source data fusion of air and ground, characterized in that: The following steps are involved: S1: Collect the UAV oblique photography data, scanning point cloud data and panoramic video data of the target environment, and pre-process them respectively; S2: Preliminarily align the preprocessed UAV oblique photography data and the scanned point cloud data to obtain the preliminarily aligned UAV oblique photography data and the scanned point cloud data; S3: Using the improved ICP algorithm, the texture information of the pre-processed panoramic video data is introduced to perform fine registration on the preliminarily aligned UAV oblique photography data and scanning point cloud data to obtain the multi-source data registration results, including: S31: Select N points from the UAV oblique photography data after preliminary alignment , N is a positive integer; and the texture features of each pixel in the preprocessed panoramic video data are collected at the same time; S32: For each point , find the closest point in the scanned point cloud data after preliminary alignment , and form N point pairs ; S33: Use the singular value decomposition algorithm to solve the following optimization problem and calculate the rigid body transformation matrix of the UAV oblique photography data and the scanned point cloud data after preliminary alignment. , so that the Euclidean distance between the two is minimized, and at the same time, the texture information difference between the two and the preprocessed panoramic video data is minimized: in, is the rotation matrix, is the translation vector; is the balance coefficient; is the midpoint of the preprocessed panoramic video data Texture features of corresponding pixels; is the midpoint of the preprocessed panoramic video data Texture features of corresponding pixels; represents the square of the norm; S34: Using the rigid body transformation matrix Update the position of the drone oblique photography data after preliminary alignment; S35: repeating steps S31 to S34 until the preset error function converges, completing the refined registration, and obtaining the multi-source data registration result; The preset error function is: in, is the value of the error function; For point pair The weight of is a parameter used to control the effect of texture features on weight; S4: An improved Poisson surface reconstruction algorithm is used to introduce normal vector smoothing and outlier removal mechanisms. Then, based on a multi-scale reconstruction strategy, multi-source data registration results are reconstructed at multiple scales to generate a high-precision three-dimensional model of the target environment and complete the modeling.

2. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 1 is characterized in that: Before step S1, the following steps are also included: A number of image control points are evenly arranged in the target environment, and the three-dimensional coordinates of all the image control points are measured to obtain image control point data.

3. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 2 is characterized in that: In step S1, collecting the drone oblique photography data, scanning point cloud data and panoramic video data of the target environment includes: Using a drone equipped with a camera, perform multi-angle and multi-altitude aerial oblique photography of the target environment according to a predetermined route to obtain drone oblique photography data; Use a handheld Lidar scanner to scan the ground area of ​​the target environment and obtain scanning point cloud data; The panoramic video of the target environment is captured by a panoramic camera to record the texture information and detailed appearance of the target environment and obtain panoramic video data.

4. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 3 is characterized in that: In the process of acquiring the oblique photography data of the UAV, the A-Star search algorithm is used to plan the flight route of the UAV in the target environment in real time, and it is used as the predetermined route in real time to realize intelligent obstacle avoidance of the UAV.

5. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 3 or 4, characterized in that: In step S1, the preprocessing includes: The collected UAV oblique photography data, scanning point cloud data and panoramic video data are adjusted to the same size and resolution; the panoramic video data with the adjusted size and resolution is used as the pre-processed panoramic video data; For the drone oblique photography data and scanned point cloud data whose size and resolution have been adjusted, spatial synchronization is performed based on the three-dimensional coordinates of all image control points, and time synchronization is performed based on the timestamp when the data was collected, to obtain the synchronized drone oblique photography data and scanned point cloud data; Perform image denoising and enhancement processing on the synchronized UAV oblique photography data, and use the image control point data for geometric correction to obtain the pre-processed UAV oblique photography data; Gaussian filtering is used to filter the synchronized scanned point cloud data, and the pre-processed panoramic video data is used to color the scanned point cloud data after point cloud filtering. The point cloud coloring is expressed as: in, is the i-th point in the scanned point cloud data after point cloud filtering; For point The color value of is the mapping function; is the jth frame in the preprocessed panoramic video data; After the point cloud is colored, the normal vector of each point in the scanned point cloud data is calculated and saved to obtain the preprocessed scanned point cloud data.

6. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 5 is characterized in that: In step S2, the pre-processed UAV oblique photography data and the scanning point cloud data are preliminarily aligned using a feature point matching algorithm and uniformly aligned to the same coordinate system.

7. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 5 is characterized in that: The step S4 comprises: S41: performing downsampling and voxel grid filtering processing on the multi-source data registration result to obtain a low-resolution sparse point cloud; selecting a number of key areas from the multi-source data registration result, and saving the point clouds of the key areas together as a high-resolution dense point cloud; S42: performing normal vector smoothing and outlier removal on the sparse point cloud and the dense point cloud respectively to obtain denoised sparse point cloud and dense point cloud; S43: Obtain the normal vector field of the corresponding point cloud based on the normal vectors of each point in the denoised sparse point cloud and dense point cloud and , construct and solve the following Poisson equation in turn to obtain the optimal surface function corresponding to the sparse point cloud and the dense point cloud and : in, and express and The Laplace operator of The horizontal coordinate of the point; According to the optimal surface function and generating a first and a second three-dimensional model, wherein the first three-dimensional model is used to represent the overall structure of the target environment, and the second three-dimensional model is used to represent the detailed features of the key area; S44: performing trilinear interpolation and weighted fusion on the first and second three-dimensional models to generate the high-precision three-dimensional model of the target environment.

8. The refined 3D modeling method based on multi-source data fusion of space and ground according to claim 7 is characterized in that: In step S42, normal vector smoothing is performed on each point in the sparse point cloud and the dense point cloud according to the following formula: in, is the i-th point in the point cloud; For point Smoothed normal vector; For point The neighborhood point set of ; For point and Point The weighting coefficient of For each point in the sparse point cloud and the dense point cloud after normal vector smoothing , respectively calculate the average normal vector of its neighborhood point set The deviation between , the deviation is expressed as: ,in represents the Euclidean distance; The deviation Greater than the preset rejection threshold point Remove them as outliers and complete denoising.

9. A refined 3D modeling system based on multi-source data fusion of space and ground, based on a refined 3D modeling method based on multi-source data fusion of space and ground as described in any one of claims 1 to 8, characterized in that: include: Preprocessing unit: used to collect UAV oblique photography data, scanning point cloud data and panoramic video data of the target environment, and preprocess them respectively; Rough registration unit: used to perform preliminary alignment on the pre-processed UAV oblique photography data and the scanned point cloud data, and obtain the pre-aligned UAV oblique photography data and the scanned point cloud data; Fine registration unit: used to adopt the improved ICP algorithm, introduce the texture information of the pre-processed panoramic video data, perform fine registration on the preliminarily aligned UAV oblique photography data and scanning point cloud data, and obtain the multi-source data registration result; 3D modeling unit: It is used to adopt the improved Poisson surface reconstruction algorithm, introduce the normal vector smoothing and outlier removal mechanism, and then based on the multi-scale reconstruction strategy, perform multi-scale reconstruction on the multi-source data registration results to generate a high-precision 3D model of the target environment and complete the modeling.

Images