Method and system for aligning aerial three-dimensional reconstruction data and ground three-dimensional reconstruction data

By calculating the offset of the horizontal and height projection images of the three-dimensional reconstruction data in the air and ground, the alignment of the air and ground data is achieved, solving the problems of data perspective differences and coverage differences in the prior art, and improving the accuracy of the three-dimensional map and Robustness.

CN118918248APending Publication Date: 2024-11-08SHICHEN INFORMATION TECH SHANGHAI CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410941194.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively align the aerial three-dimensional reconstruction data with the ground three-dimensional reconstruction data, resulting in large differences in perspective angles and large coverage, hindering the subsequent development of fusion algorithms and failing to meet the data matching requirements required by visual maps.

Method used

By obtaining target scene pictures from the air and ground for 3D reconstruction, using horizontal and height projection profiles and height maps, the data is calculated and adjusted to achieve the alignment of six degrees of freedom, including horizontal and height offsets, ensuring the data is ensured by using horizontal and height offsets. Align under the same coordinate system.

Benefits of technology

High-precision alignment of air three-dimensional reconstruction data and ground three-dimensional reconstruction data is achieved, and the problem that point cloud accuracy does not meet the requirements of visual map reconstruction is solved, and the system robustness and data matching accuracy are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918248A_ABST
    Figure CN118918248A_ABST
Patent Text Reader

Abstract

The invention provides a method and system for aligning aerial three-dimensional reconstruction data and ground three-dimensional reconstruction data, and the method comprises the steps: carrying out the projection of the aerial three-dimensional point cloud data Cs and the ground three-dimensional point cloud data Cg to a horizontal plane, so as to obtain a contour image ICs and a contour image ICg; calculating a horizontal offset between the contour images and adjusting the ground three-dimensional point cloud data Cg to obtain ground three-dimensional point cloud data Cg1 after horizontal offset; projecting in the horizontal direction based on the aerial three-dimensional point cloud data Cs and the horizontally shifted ground three-dimensional point cloud data Cg1 to obtain a height image Ihs and a height image Ihg1; and calculating the height offset between the height images, and adjusting the ground three-dimensional point cloud data Cg1 after horizontal offset to obtain ground three-dimensional point cloud data Cg2 after height offset. According to the invention, effective alignment matching of the aerial image and the ground image can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system and computer-readable storage medium for aligning aerial three-dimensional reconstruction data with ground three-dimensional reconstruction data. Background Art

[0002] In the process of visual map construction in the existing technology, the shooting and reconstruction of the target scene involves the alignment of aerial images and ground images, as well as the reconstructed data of aerial images and ground images. Generally speaking, the alignment of aerial 3D reconstruction data and ground 3D reconstruction data is a complex but critical process, which ensures that the data obtained from different perspectives and scales can be accurately matched in space.

[0003] Using drones or satellites to photograph target scenes from the air and reconstructing them based on aerial images is a relatively mature technology that can achieve high global accuracy because there is less interference from GPS signals in the air and a single photo covers a wide range. Although 3D reconstruction using photos taken by humans or unmanned vehicles using cameras or mobile phones has been developed for decades, its global accuracy is still much lower than that of 3D reconstruction based on aerial data.

[0004] The ground GPS signal is even worse or even completely absent due to occlusion, and the coverage of a single photo is narrow, so it needs to be supplemented by aerial 3D reconstruction data. Fusion of the ground 3D reconstruction results with the aerial 3D reconstruction results can obtain a more complete reconstruction result with higher global and local accuracy, so combining the two has increasingly important practical significance.

[0005] However, the premise of fusing aerial 3D reconstruction data with ground 3D reconstruction data is that the coordinate systems of the two are the same, that is, the two are aligned in 3D. However, due to the huge difference in the perspectives of the two, the little overlap of the reconstructed objects, and the huge difference in the quality of the reconstructed point clouds, it is difficult to align the two, which hinders the development of subsequent fusion algorithms.

[0006] In general, the aerial images and ground images provided by the existing technology have large changes in viewing angles and large differences in coverage, and do not solve the problem of matching the aerial images and ground images required for visual maps. Summary of the invention

[0007] The technical problem solved by the technical solution of the present invention is: how to achieve mutual matching between aerial images and ground images required for visual maps.

[0008] In order to solve the above technical problems, the technical solution of the present invention provides a method for aligning aerial 3D reconstruction data with ground 3D reconstruction data, comprising:

[0009] Get the target scene image P from the air i s , i can be 1, 2, 3, ..., N, where N is a natural number greater than 1;

[0010] Get ground image P j g , j can be 1, 2, 3, ..., M, where M is a natural number greater than 1;

[0011] For the target scene image P i s And ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g ;

[0012] Based on the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Projecting onto a horizontal plane, respectively counting the number of projected aerial 3D point cloud data and projected ground 3D point cloud data falling in a first preset horizontal grid, so as to obtain a contour image I of the projected aerial 3D point cloud data C s And the contour image I of the projected ground 3D point cloud data C g ;

[0013] Contour image based on projected aerial 3D point cloud data I C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjust to obtain the ground three-dimensional point cloud data C after horizontal offset g1 ;

[0014] Based on the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Project in the horizontal direction and count the projected aerial 3D point cloud data and the ground 3D point cloud data after horizontal offset C g1 The number of points falling in the second preset horizontal grid is used to obtain the height image I of the projected aerial three-dimensional point cloud data. h sAnd the ground three-dimensional point cloud data C after horizontal offset g1 Height image I h g1 ;

[0015] Based on the aerial three-dimensional point cloud data C s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offset between the two, and the ground three-dimensional point cloud data C after the horizontal offset according to the height offset g1 Adjust to obtain the ground 3D point cloud data C after height offset g2 ; The ground three-dimensional point cloud data C after the height offset g2 With the aerial 3D point cloud data C s Alignment.

[0016] Optionally, the target scene image P i s And ground image P j g The first GPS information and the second GPS information are respectively provided, and the three-dimensional point cloud data C in the air is based on the three-dimensional point cloud data C in the air. s And ground 3D point cloud data C g Projection onto the horizontal plane includes:

[0017] According to the first GPS information and the second GPS information, the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g respectively aligned to the GPS so that the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Have the same GPS information at the same location;

[0018] Output the aerial 3D point cloud data C aligned with GPS s0 And ground 3D point cloud data C g0 ;

[0019] The aerial three-dimensional point cloud data C s0 And ground 3D point cloud data C g0 Projection onto the horizontal plane;

[0020] The projected aerial 3D point cloud data and the projected ground 3D point cloud data are respectively based on the aerial 3D point cloud data C s0And ground 3D point cloud data C g0 Projection obtained.

[0021] Optionally, the respectively counting of the number of the projected aerial three-dimensional point cloud data and the projected ground three-dimensional point cloud data falling in the first preset horizontal grid includes:

[0022] Assume that the point cloud coordinate range of the projected aerial 3D point cloud data is in, They represent the minimum values ​​of the x-coordinate, y-coordinate, and z-coordinate of each point cloud in the aerial 3D point cloud data. Respectively represent the maximum value of each point cloud in the three-dimensional aerial point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0023] Assume that the point cloud coordinate range of the projected ground 3D point cloud data is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0024] Assume that the horizontal plane range covered by the projected aerial 3D point cloud data and the projected ground 3D point cloud data is in

[0025] The horizontal plane range covered by the projection is set to the preset step size α Performing division to generate the first preset horizontal grid;

[0026] Let the point cloud coordinates of the currently projected aerial 3D point cloud data or the projected ground 3D point cloud data be (x a ,y a ,z a ), the grid coordinates of the first preset horizontal grid are (u a ,v a ), then the following formula is used to count whether the point cloud coordinates fall into the grid (u a ,v a ):

[0027] like Then the point cloud coordinates (x a ,y a ,z a ) falls into the grid, floor() is a function that rounds down the floating point numbers therein;

[0028] The number of point clouds that the projected aerial three-dimensional point cloud data falls into the first preset horizontal grid is used as the contour image I Cs The pixel value corresponding to the coordinate;

[0029] The number of point clouds that the projected ground three-dimensional point cloud data falls into the first preset horizontal grid is used as the contour image I C g The pixel value corresponding to the coordinate;

[0030] Assume that the ground three-dimensional point cloud data C g1 The coordinate range is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0031] The projected aerial three-dimensional point cloud data and the ground three-dimensional point cloud data C after horizontal offset are respectively counted. g1 The quantities that fall into the second preset horizontal grid include:

[0032] Recalculate the projected aerial 3D point cloud data and the horizontally offset ground 3D point cloud data C g1 Horizontal range covered by projection in,

[0033] The horizontal plane range covered by the projection is set to the preset step size b. Performing division to generate the second preset horizontal grid;

[0034] Let the point cloud coordinates of the currently projected aerial 3D point cloud data or ground 3D point cloud data Cg1 be (x b ,y b ,z b ), the grid coordinates of the second preset horizontal grid are (u b ,v b ), then the following formula is used to count the number of cells that fall into the grid (u b ,v b )’s point cloud number:

[0035] like Then the point cloud coordinates (x b ,y b ,z b ) falls into this grid (u b ,v b );

[0036] The projected aerial 3D point cloud data falls into the grid (u b ,v b ) The average value of the z coordinates of the point cloud is taken as the grid (ub ,v b ) and outputs the height image I of the projected aerial three-dimensional point cloud data at this height. h s The pixel value corresponding to the coordinate;

[0037] The ground 3D point cloud data C g1 falls into the grid (u b ,v b ) is taken as the average value of the z coordinates of the point cloud of the grid (u b ,v b ) and output the ground 3D point cloud data C at this height g1 Height image I h g1 The pixel value corresponding to the coordinate.

[0038] Optionally, the contour image I based on the projected aerial three-dimensional point cloud data C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjustments include:

[0039] For the contour image I C s and contour image I C g Perform template matching to locate the contour image I C s and contour image I C g The displacement pixels (d u ,d v ), assuming that the step size of the first preset horizontal grid is α, then the contour image I C s and contour image I C g The horizontal offset between u ×a,d v ×a);

[0040] According to the horizontal offset (d u ×a,d v ×a) the ground three-dimensional point cloud data C g Adjust and recalculate the coordinate range of the ground point cloud, and output the ground 3D point cloud data C after horizontal offsetg1 .

[0041] Optionally, the three-dimensional point cloud data C based on the air s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offsets between include:

[0042] Assume that the height image I h g1 The horizontal coordinates of the image are The corresponding grid coordinates are Take the grid coordinates (u i ,v i )

[0043] Similarly, take the height image I h s At the corresponding grid coordinates (u i ,v i )

[0044] Get height image I h g1 And height image I h s The offset of all pixel values ​​at corresponding grid coordinates;

[0045] Take the average value of all pixel value offsets as the height image I h s And height image I h g1 The height offset between .

[0046] Optionally, the three-dimensional point cloud data C of the ground after horizontal offset is processed according to the height offset. g1 Adjustments include:

[0047] The ground three-dimensional point cloud data C g1 The height offset is offset on the z coordinate to obtain the ground three-dimensional point cloud data C after the height offset g2 .

[0048] In order to solve the above technical problems, the technical solution of the present invention further provides a system for aligning aerial 3D reconstruction data with ground 3D reconstruction data, comprising:

[0049] The first acquisition unit is adapted to acquire a target scene image P from the air. i s , i can be 1, 2, 3, ..., N, where N is a natural number greater than 1;

[0050] The second acquisition unit is adapted to acquire the ground image P j g , j can be 1, 2, 3, ..., M, where M is a natural number greater than 1;

[0051] A three-dimensional reconstruction unit, adapted to reconstruct the target scene image P i s And ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g ;

[0052] The first projection statistics unit is adapted to be based on the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Projecting onto a horizontal plane, respectively counting the number of projected aerial 3D point cloud data and projected ground 3D point cloud data falling in a first preset horizontal grid, so as to obtain a contour image I of the projected aerial 3D point cloud data C s And the contour image I of the projected ground 3D point cloud data C g ;

[0053] The first offset adjustment unit is adapted to adjust the contour image I based on the projected aerial three-dimensional point cloud data C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjust to obtain the ground three-dimensional point cloud data C after horizontal offset g1 ;

[0054] The second projection statistics unit is adapted to be based on the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Project in the horizontal direction and count the projected aerial 3D point cloud data and the ground 3D point cloud data after horizontal offset C g1The number of points falling in the second preset horizontal grid is used to obtain the height image I of the projected aerial three-dimensional point cloud data. h s And the ground three-dimensional point cloud data C after horizontal offset g1 Height image I h g1 ;

[0055] The second offset adjustment unit is adapted to adjust the airborne three-dimensional point cloud data C based on the airborne three-dimensional point cloud data C s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offset between the two, and the ground three-dimensional point cloud data C after the horizontal offset according to the height offset g1 Adjust to obtain the ground 3D point cloud data C after height offset g2 ; The ground three-dimensional point cloud data C after the height offset g2 With the aerial 3D point cloud data C s Alignment.

[0056] Optionally, the target scene image P i s And ground image P j g The system comprises first GPS information and second GPS information respectively, and further comprises:

[0057] The GPS alignment unit is adapted to align the aerial three-dimensional point cloud data C according to the first GPS information and the second GPS information. s And ground 3D point cloud data C g respectively aligned to the GPS so that the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Have the same GPS information at the same location;

[0058] Output unit, suitable for outputting aerial 3D point cloud data C aligned with GPS s0 And ground 3D point cloud data C g0 ;

[0059] Based on the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Projecting onto a horizontal plane includes: projecting the aerial three-dimensional point cloud data C s0And ground 3D point cloud data C g0 Projection onto the horizontal plane;

[0060] Based on the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Projecting in the horizontal direction includes: projecting the aerial three-dimensional point cloud data C s0 And the ground three-dimensional point cloud data C after horizontal offset g1 Projection in horizontal direction;

[0061] The projected aerial 3D point cloud data and the projected ground 3D point cloud data are respectively based on the aerial 3D point cloud data C s0 And ground 3D point cloud data C g0 Projection obtained.

[0062] Optionally, the point cloud coordinate range of the projected aerial 3D point cloud data is in, They represent the minimum values ​​of the x-coordinate, y-coordinate, and z-coordinate of each point cloud in the aerial 3D point cloud data. Respectively represent the maximum value of each point cloud in the three-dimensional aerial point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0063] Assume that the point cloud coordinate range of the projected ground 3D point cloud data is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0064] The first projection statistics unit comprises:

[0065] The first horizontal plane setting unit is adapted to set the horizontal plane range covered by the projected aerial three-dimensional point cloud data and the projected ground three-dimensional point cloud data as in

[0066] The first grid division subunit is adapted to divide the horizontal plane range covered by the projection into Performing division to generate the first preset horizontal grid;

[0067] The first statistical subunit is adapted to make the point cloud coordinates of the currently projected aerial 3D point cloud data or the projected ground 3D point cloud data be (x a ,y a ,z a ), the grid coordinates of the first preset horizontal grid are (u a ,v a), then the following formula is used to count whether the point cloud coordinates fall into the grid (u a ,v a ):

[0068] like Then the point cloud coordinates (x a ,y a ,z a ) falls into the grid, floor() is a function that rounds down the floating point numbers therein;

[0069] The first contour image generating subunit is adapted to use the number of point clouds of the projected aerial three-dimensional point cloud data falling into the first preset horizontal grid as the contour image I C s The pixel value corresponding to the coordinate;

[0070] The second contour image generating subunit is adapted to use the number of point clouds of the projected ground three-dimensional point cloud data falling into the first preset horizontal grid as the contour image I C g The pixel value corresponding to the coordinate;

[0071] Assume that the ground three-dimensional point cloud data C g1 The coordinate range is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0072] The second projection statistics unit comprises:

[0073] The second horizontal plane setting subunit is adapted to recalculate the projected aerial 3D point cloud data and the horizontally offset ground 3D point cloud data C g1 Horizontal range covered by projection in,

[0074] The second grid division subunit is adapted to divide the horizontal plane range covered by the projection into two parts with a preset step size b. Performing division to generate the second preset horizontal grid;

[0075] The second statistical subunit is adapted to calculate the currently projected aerial three-dimensional point cloud data or ground three-dimensional point cloud data C g1 The point cloud coordinates are (x b ,y b ,z b ), the grid coordinates of the second preset horizontal grid are (u b ,v b), then the following formula is used to count the number of cells that fall into the grid (u b ,v b )’s point cloud number:

[0076] like Then the point cloud coordinates (x b ,y b ,z b ) falls into this grid (u b ,v b );

[0077] The first height image generating subunit is adapted to place the projected aerial three-dimensional point cloud data into a grid (u b ,v b ) The average value of the z coordinates of the point cloud is taken as the grid (u b ,v b ) and outputs the height image I of the projected aerial three-dimensional point cloud data at this height. h s The pixel value corresponding to the coordinate;

[0078] The second height image generation subunit is adapted to convert the ground three-dimensional point cloud data C g1 falls into the grid (u b ,v b ) is taken as the average value of the z coordinates of the point cloud of the grid (u b ,v b ) and output the ground 3D point cloud data C at this height g1 Height image I h g1 The pixel value corresponding to the coordinate.

[0079] In order to solve the above technical problems, the technical solution of the present invention also provides a system for aligning aerial three-dimensional reconstruction data with ground three-dimensional reconstruction data, including: a processor and a memory; the memory stores a computer program, and when the processor executes the computer program, it performs the method steps of aligning aerial three-dimensional reconstruction data with ground three-dimensional reconstruction data as described above.

[0080] In order to solve the above technical problems, the technical solution of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method steps of aligning the aerial three-dimensional reconstruction data and the ground three-dimensional reconstruction data as described above.

[0081] The beneficial effects of the technical solution of the present invention include at least:

[0082] The technical solution of the present invention can decompose and align aerial 3D reconstruction data and ground 3D reconstruction data according to six degrees of freedom, and uses the horizontal projection contour map and horizontal projection height map of the two sets of data to calculate the horizontal deviation and height deviation of the two sets of data. This can solve the shortcoming of the prior art that the point cloud accuracy does not meet the requirements of visual map reconstruction.

[0083] The technical solution of the present invention is based on aerial 3D reconstruction data and ground 3D reconstruction data, and the two sets of data are aligned by GPS. By adopting the horizontal projection contour map and the horizontal projection height map of the two sets of data, the horizontal deviation and the height difference between the aerial 3D reconstruction data and the ground 3D reconstruction data can be calculated; according to the angular deviation, horizontal deviation and height deviation of the aerial 3D data and the ground 3D data, the problems of large viewing angle changes and large coverage differences between aerial images and ground images are solved, thereby realizing the alignment and matching of aerial images and ground images.

[0084] The technical solution of the present invention further sets the coverage range of the horizontal projection of the two sets of data through the aerial three-dimensional reconstruction data and the ground three-dimensional reconstruction data, and based on different step sizes, obtains the first preset grid and the second preset grid that can better reflect the horizontal and height deviations of the aerial three-dimensional reconstruction data and the ground three-dimensional reconstruction data, so as to respectively calculate the deviation in the horizontal direction and the deviation in height, and improves the robustness of the technical solution system of the present invention through statistical characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0086] Figure 1 A schematic flow chart of a method for aligning aerial 3D reconstruction data with ground 3D reconstruction data provided by the technical solution of the present invention;

[0087] Figure 2 A schematic diagram of a system structure for aligning aerial 3D reconstruction data with ground 3D reconstruction data provided by the technical solution of the present invention;

[0088] Figure 3 A schematic diagram of another system structure for aligning aerial 3D reconstruction data with ground 3D reconstruction data provided by the technical solution of the present invention;

[0089] Figure 4 A schematic diagram of a unit structure for outputting a contour image based on aerial 3D reconstruction data and terrestrial 3D reconstruction data based on projection statistics of aerial 3D reconstruction data and terrestrial 3D reconstruction data provided by the technical solution of the present invention;

[0090] Figure 5A schematic diagram of a unit structure for outputting a height image based on aerial 3D reconstruction data and ground 3D reconstruction data based on projection statistics of aerial 3D reconstruction data and ground 3D reconstruction data provided by the technical solution of the present invention;

[0091] Figure 6 A schematic flow chart of another method step for aligning aerial 3D reconstruction data with ground 3D reconstruction data provided by the technical solution of the present invention. DETAILED DESCRIPTION

[0092] In order to better illustrate the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings.

[0093] In the technical solution of the present invention, by aligning the aerial 3D reconstruction data and the ground 3D reconstruction data, an accurate and precise 3D model can be constructed for the target scene map. In aerial 3D reconstruction, such as drone oblique photogrammetry, it is possible to efficiently cover a large area and obtain the top and side information of the object, but the structure and texture information may be lost or insufficient in the near-ground and occluded areas. Ground 3D reconstruction, such as through ground laser scanning or close-range photography, can provide high-resolution near-ground structural data to supplement the shortcomings of the aerial perspective. Therefore, the method of multi-source data fusion can correct the problems of local texture distortion, ground object holes, etc. that may be caused by modeling with a single data source, thereby significantly improving the geometric accuracy and quality of the model. For problems such as ground object occlusion, tree crown occlusion, eaves occlusion, glass transmittance, etc. in aerial urban scenes, a single perspective is difficult to solve. Ground data can provide a detailed view of the occluded area and improve the integrity of the model. Ground close-range photography images have a short acquisition distance, high resolution, and rich information, and can provide more angle information, providing the possibility of building a real 3D model. By aligning the aerial 3D reconstruction data with the ground 3D reconstruction data, a more complete 3D model can be constructed, which can be better used for spatial analysis and data visualization, and provide more accurate geographic information for decision makers. The technical solution of the present invention can fully utilize the advantages of the aerial 3D reconstruction data and the ground 3D reconstruction data by aligning them, and make up for their respective deficiencies, thereby obtaining a high-precision, high-quality 3D model of the target scene. The technical solution of the present invention can not only improve the accuracy and application value of map data, but also open up new possibilities for future map positioning and visual map applications.

[0094] Aligning aerial 3D reconstruction data with ground 3D reconstruction data is a relatively complex process, including: obtaining high-resolution images and laser radar (LiDAR) aerial images from drones (or other aircraft), and using ground 3D scanning equipment such as laser scanners, photogrammetry or mobile mapping systems to collect ground images; preprocessing the original collected aerial images and ground images, including denoising, radiation correction, geometric correction, etc.; further reconstructing the aerial images and ground images in 3D, and using specialized software to reconstruct the aerial and ground data separately; 3D reconstruction involves extracting feature points from aerial or ground images , establish a dense point cloud, and generate meshes and textures; convert the formed aerial 3D reconstruction data and ground 3D reconstruction data into the same coordinate system; identify the common feature points or feature objects (such as corners of buildings, unique terrain features, etc.) of the aerial 3D reconstruction data and the ground 3D reconstruction data, and use automatic or manual methods to perform feature matching; perform an initial rough alignment based on the matched feature points or feature objects, such as using a technique such as an ICP (iterative closest point) algorithm, and fine-tune the rough alignment results, and finally match the aerial 3D reconstruction data and the ground 3D reconstruction data into a unified model.

[0095] However, the aerial 3D reconstruction data and the ground 3D reconstruction data have huge differences in image perspectives and little overlap in reconstructed objects, resulting in a large difference in the quality of the reconstructed point clouds, making it difficult to align the two.

[0096] The technical solution of the present invention constructs a contour map and a height map of the aerial 3D reconstruction data and the ground 3D reconstruction data projected onto the horizontal plane, and can calculate the horizontal deviation and height deviation of the two sets of data based on the contour map and the height map of the horizontal projection, thereby achieving effective alignment of the aerial image and the ground image.

[0097] refer to Figure 1 In an embodiment of the technical solution of the present invention, the method for aligning aerial 3D reconstruction data with ground 3D reconstruction data comprises the following steps:

[0098] Step S100, obtaining a target scene image P from the air i s , i can be 1, 2, 3, ..., N, where N is a natural number greater than 1;

[0099] Step S101, obtain ground image P j g , j can be 1, 2, 3, …, M, where M is a natural number greater than 1.

[0100] In steps S100 and S101, an aerial picture P of the target scene is taken. i s And the ground picture Pj g , P i s It is a picture of the target scene taken by a drone or satellite. i s = {P1 s , P2 s , P3 s ,…,P N s}, N is a natural number greater than 1, and i can be 1, 2, 3, ..., N. j g It is a ground image taken by a camera held by a person or mounted on a robot or vehicle. j g = {P1 g , P2 g , P3 g ,…,P M g}, M is a natural number greater than 1, j can be 1, 2, 3, ..., M. Generally, aerial images have relatively high-precision GPS information, and ground images also need GPS information. Therefore, in this embodiment, the target scene image P i s And ground image P j g The first GPS information and the second GPS information are respectively provided.

[0101] Furthermore, the camera can be mounted on a gimbal, and the gimbal can be mounted on a drone to obtain the above-mentioned aerial picture P i s . Aerial Photo P i sIt further includes GPS information, posture information and gimbal information, wherein: GPS information includes longitude, latitude and altitude, which can be detected by the GPS module on the drone; posture information includes pitch angle, roll angle and yaw angle, which can be detected by the IMU (inertial measurement unit) on the drone; gimbal information includes rotation, pitch and zoom information of the gimbal, representing the all-round (left / right / up / down) movement of the gimbal and the zoom and zoom control of the lens. In a specific application example, the drone projection device can communicate with the drone flight control system through wireless transmission (such as Bluetooth, WiFi, 4G, 5G, etc.), and receive positioning information, posture information, gimbal information, focal length information of the drone camera and video stream shot by the drone camera. After receiving the video stream sent back by the drone, the video stream is decoded by soft decoding or hard decoding, and the video frame is obtained. Based on the positioning information, posture information and gimbal information of the drone, the initial position and posture of the drone camera in the world coordinate system are determined. The positioning information, attitude information and gimbal information of the drone can be combined as the position and attitude of the drone camera in the world coordinate system (i.e., the initial position and attitude), and the aerial image information can be obtained based on this. The method of obtaining ground image information is similar and will not be repeated here.

[0102] Continue to refer Figure 1 The method for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data also includes:

[0103] Step S102: i s And ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g .

[0104] Commercial or open source software can be used to P i s And ground image P j g Point cloud reconstruction is performed to obtain the pose and point cloud of the sky data and the ground data respectively. i s The aerial point cloud data is C s , ground image P j g The ground point cloud data is C g .

[0105] In step S102, the point cloud reconstruction involved is a three-dimensional data processing technology that can extract three-dimensional information from a two-dimensional image, which can be achieved through a variety of commercial and open source software in the prior art. For example, commercial software such as Agisoft Metashape (Professional Edition), Pix4D, ContextCapture, etc. can be used. Open source software such as OpenDroneMap, Meshroom, COLMAP, etc. can also be used. These commercial and open source software automatically detect the feature points of each image and match them between images. Through image matching, the software calculates the position and direction of the shooting device and starts to build a sparse point cloud. On the basis of the sparse point cloud, more points are added to form a complete object surface. The texture of the original photo is applied to the reconstructed model to make the model more realistic and detailed. Use software tools to remove redundant points, fill in missing parts, smooth the model surface, etc. Export the model to different formats as required, such as .obj, .ply, .dae, etc. Output the reconstructed point cloud.

[0106] Continue to refer Figure 1 The method for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data also includes:

[0107] Step S200: convert the aerial three-dimensional point cloud data C into s And ground 3D point cloud data C g respectively aligned to the GPS so that the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Have the same GPS information at the same location.

[0108] Step S201, output the aerial 3D point cloud data C aligned with GPS s0 And ground 3D point cloud data C g0 .

[0109] In steps S200 and S201, the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g The two sets of data are aligned to GPS respectively, and the two sets of data are consistent in the direction of gravity and the horizontal direction (both are aligned to the North Pole), and the horizontal position error is several meters to more than ten meters. Assume that the coordinate system is defined as: the x-axis points to the east, the y-axis points to the north, and the z-axis points to the sky. In this embodiment, the aerial three-dimensional point cloud data after alignment with GPS is recorded as C s0 , the ground three-dimensional point cloud data is C g0 .

[0110] More specifically, the aerial 3D point cloud data C can be transformed intos And ground 3D point cloud data C g Align to GPS: Align the aerial 3D point cloud data C s And ground 3D point cloud data C g Import into GIS software such as ArcGIS, QGIS or other professional software. Check the initial display of the data to confirm whether there is any obvious misalignment or geographical deviation. If the data is not in the correct coordinate system, use the georeferencing tool to convert it to WGS84. For each dataset, use ground control points (GCPs) to make corrections to ensure that the same locations in each dataset have the same GPS coordinates. Adjust the offset, rotation and scale in the dataset until the corresponding features of the two sets of data are completely overlapped, and output the aerial 3D point cloud data aligned with GPS. s0 And ground 3D point cloud data C g0 .

[0111] Continue to refer Figure 1 The method for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data also includes:

[0112] Step S103: convert the aerial three-dimensional point cloud data C s0 And ground 3D point cloud data C g0 Project to the horizontal plane and count the projected aerial 3D point cloud data C s0 And the projected ground three-dimensional point cloud data C g0 The number of points falling in the first preset horizontal grid is used to obtain the projected aerial three-dimensional point cloud data C s0 The contour image I C s And the projected ground three-dimensional point cloud data C g0 The contour image I C g .

[0113] In step S103:

[0114] Assume that the point cloud coordinate range of the projected aerial 3D point cloud data is in, They represent the minimum values ​​of the x-coordinate, y-coordinate, and z-coordinate of each point cloud in the aerial 3D point cloud data. Respectively represent the maximum value of each point cloud in the three-dimensional aerial point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0115] Assume that the point cloud coordinate range of the projected ground 3D point cloud data is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate; then,

[0116] The method of respectively counting the number of projected aerial three-dimensional point cloud data and projected ground three-dimensional point cloud data falling in the first preset horizontal grid comprises:

[0117] Assume that the horizontal plane range covered by the projected aerial 3D point cloud data and the projected ground 3D point cloud data is in

[0118] The horizontal plane range covered by the projection is set to the preset step size α Performing division to generate the first preset horizontal grid;

[0119] Let the point cloud coordinates of the currently projected aerial 3D point cloud data or the projected ground 3D point cloud data be (x a ,y a ,z a ), the grid coordinates of the first preset horizontal grid are (u a ,v a ), then the following formula is used to count whether the point cloud coordinates fall into the grid (u a ,v a ):

[0120] like Then the point cloud coordinates (x a ,y a ,z a ) falls into the grid, floor() is a function that rounds down the floating point numbers therein;

[0121] The number of point clouds that the projected aerial three-dimensional point cloud data falls into the first preset horizontal grid is used as the contour image I C s The pixel value corresponding to the coordinate;

[0122] The number of point clouds that the projected ground three-dimensional point cloud data falls into the first preset horizontal grid is used as the contour image I C g The pixel value corresponding to the coordinate.

[0123] In the technical solution of the present invention, a first preset horizontal grid is constructed based on the calculated horizontal plane range covered by the aerial 3D point cloud data and the projected ground 3D point cloud data and the preset step length α to count the number of aerial 3D point cloud data or ground 3D point cloud data falling into each grid coordinate, and the number of aerial 3D point cloud data falling into each grid coordinate is used as the contour image I C sThe pixel value corresponding to the grid coordinates and the number of ground 3D point cloud data falling into each grid coordinate are used as the contour images I C g The pixel values ​​corresponding to the grid coordinates are used to form the contour image I of the aerial 3D point cloud data. C s , and the contour image I of the ground three-dimensional point cloud data is formed accordingly C g .

[0124] In the prior art, obtaining the contour map of the aerial image of the visual map is mainly achieved through computer vision and image processing technology. This process involves detecting key information such as shape, features, and edges from the image, and converting this information into a contour map: First, it is necessary to obtain an aerial image, which is usually done through a camera mounted on a drone or aircraft. The acquired image needs to have a high enough resolution for subsequent processing; an edge detection algorithm is used to identify the edges in the image. Commonly used algorithms such as Canny edge detection can effectively identify edge information in the image; before finding the contour, it is usually necessary to threshold the image or convert it into a binary image using the Canny algorithm, which is the basis for contour extraction; the cv2.findContours() function in OpenCV is used to extract the contour in the binary image after edge detection. This function can find and return the contour information in the image; the detected contour is drawn on the original image using the cv2.drawContours() function, which can intuitively display the result of contour extraction; contour optimization: further processing of the contour is performed as needed, such as simplifying the number of contour points (contour approximation), calculating the bounding rectangle or circumscribed circle of the contour, etc. The above steps of this embodiment customize the method of obtaining the aerial image contour map, and use the contour map to calculate the horizontal deviation of the aerial three-dimensional point cloud data and the ground three-dimensional point cloud data, so as to more accurately match the aerial image data and the ground image data.

[0125] Continue to refer Figure 1 The method for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data also includes:

[0126] Step S104: based on the projected aerial three-dimensional point cloud data C s0 The contour image I C s And the projected ground three-dimensional point cloud data C g0 The contour image I C g Calculate the contour image I C s and contour image I C gThe horizontal offset between the ground three-dimensional point cloud data C g Adjust to obtain the ground three-dimensional point cloud data C after horizontal offset g1 .

[0127] In step S104, the contour image I based on the projected aerial three-dimensional point cloud data C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjustments include:

[0128] For the contour image I C s and contour image I C g Perform template matching to locate the contour image I C s and contour image I C g The displacement pixels (d u ,d v ), it is known that the step size of the first preset horizontal grid is α, then the contour image I C s and contour image I C g The horizontal offset between u ×a,d v ×a);

[0129] According to the horizontal offset (d u ×a,d v ×a) the ground three-dimensional point cloud data C g Adjust and recalculate the coordinate range of the ground 3D point cloud, and output the ground 3D point cloud data C after horizontal offset g1 . Can record ground 3D point cloud data C g1 The coordinate range is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. They respectively represent the maximum values ​​of the x-coordinate, y-coordinate, and z-coordinate of each point cloud in the ground three-dimensional point cloud data.

[0130] In this embodiment, the image template matching technology is used to obtain the contour image IC s and contour image I C g The displacement pixels (d u ,d v ).

[0131] In image template matching, an input image is usually used as a query template, and then a similar or identical area is searched in another input image. This process is mainly implemented through the sliding window technology, that is, moving the template on the image and calculating the degree of matching at each position. The calculation of the degree of matching can be based on a variety of algorithms, such as gray value difference, feature point matching or more complex image feature descriptors. Different from the prior art, the image template matching technology is used to locate the position of the template image in the input image to find the most similar part of the template in the input image. The technical solution of the present invention further uses the image template matching technology to obtain the contour image I C s and contour image I C g The displacement pixels (d u ,d v ), so as to obtain the horizontal deviation of the aerial 3D point cloud data and the ground 3D point cloud data in the horizontal direction.

[0132] Continue to refer Figure 1 The method for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data also includes:

[0133] Step S105: convert the aerial three-dimensional point cloud data C s0 And the ground three-dimensional point cloud data C after horizontal offset g1 Project in the horizontal direction and count the projected aerial 3D point cloud data C s0 And the ground three-dimensional point cloud data C after horizontal offset g1 The number of points falling in the second preset horizontal grid is used to obtain the projected aerial three-dimensional point cloud data C s0 Height image I h s And the ground three-dimensional point cloud data C after horizontal offset g1 Height image I h g1 .

[0134] Assume that the ground three-dimensional point cloud data C g1 The coordinate range is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate;

[0135] In step S105, the projected aerial three-dimensional point cloud data C are counted respectively. s0 And the ground three-dimensional point cloud data C after horizontal offset g1 The quantities that fall into the second preset horizontal grid include:

[0136] Recalculate the projected aerial 3D point cloud data C s0 And the ground three-dimensional point cloud data C after horizontal offset g1 Horizontal range covered by projection in,

[0137] The horizontal plane range covered by the projection is set to the preset step size b. Performing division to generate the second preset horizontal grid;

[0138] Let the currently projected aerial 3D point cloud data C s0 Or ground 3D point cloud data C g1 The point cloud coordinates are (x b ,y b ,z b ), the grid coordinates of the second preset horizontal grid are (u b ,v b ), then the following formula is used to count the number of cells that fall into the grid (u b ,v b )’s point cloud number:

[0139] like Then the point cloud coordinates (x b ,y b ,z b ) falls into this grid (u b ,v b );

[0140] The projected aerial 3D point cloud data C s0 falls into the grid (u b ,v b ) The average value of the z coordinates of the point cloud is taken as the grid (u b ,v b ) and output the projected aerial 3D point cloud data C at this height s0 Height image I h s The pixel value corresponding to the coordinate;

[0141] The ground 3D point cloud data C g1 falls into the grid (u b ,v b ) is taken as the average value of the z coordinates of the point cloud of the grid (u b,v b ) and output the ground 3D point cloud data C at this height g1 Height image I h g1 The pixel value corresponding to the coordinate.

[0142] Similar to step S103, in step S105, a second horizontal grid in the projection direction is set to obtain a height map of the corresponding three-dimensional data point cloud, and the average Z value of the point cloud coordinates falling into the grid is used as the pixel value of the corresponding grid coordinates of the point cloud corresponding to the height map, thereby generating a point cloud height map to evaluate the offset of the two sets of data in the z direction, that is, the height direction.

[0143] In traditional technology, obtaining the height map of aerial maps is a process involving multiple steps, which mainly relies on remote sensing technology and geographic information system (GIS) technology. It is necessary to obtain raw data through appropriate remote sensing equipment and collect ground elevation data. Through the ground height data, the acquired raw data is processed (data cleaning, format conversion, etc.), and the processed elevation data is interpolated to generate a gridded DEM to create the corresponding height map. This can be done in GIS software, such as ArcGIS, QGIS, etc., to convert DEM data into a visual height map. The technical solution of the present invention calculates the three-dimensional point cloud data C in the air. s0 And the ground three-dimensional point cloud data C after horizontal offset g1 The horizontal plane range covered by the reprojection is divided into the second preset horizontal grid by the step size b. b ,v b ) of aerial 3D point cloud data C s0 The number is used as the grid coordinate (u b ,v b ) to output the aerial 3D point cloud data C s0 Height image I h s . The statistics fall into the grid (u b ,v b ) of the ground three-dimensional point cloud data C g1 The number is used as the grid coordinate (u b ,v b ) to output the ground three-dimensional point cloud data C g1 Height image I h g1 .

[0144] Continue to refer Figure 1 The method for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data also includes:

[0145] Step S106: based on the aerial three-dimensional point cloud data C s0 Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offset between the two, and the ground three-dimensional point cloud data C after the horizontal offset according to the height offset g1 Adjust to obtain the ground 3D point cloud data C after height offset g2 ; The ground three-dimensional point cloud data C after the height offset g2 With the aerial 3D point cloud data C s Alignment.

[0146] In step S106, based on the aerial three-dimensional point cloud data C s0 Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offsets between include:

[0147] Assume that the height image I h g1 The horizontal coordinates of the image are The corresponding grid coordinates are Take the grid coordinates (u i ,v i )

[0148] Similarly, take the height image I h s At the corresponding grid coordinates (u i ,v i )

[0149] Get height image I h g1 And height image I h s The offset of all pixel values ​​at corresponding grid coordinates;

[0150] Take the average value of all pixel value offsets as the height image I hs And height image I h g1 The height offset between .

[0151] In the above embodiment, for ground data, the image is collected by a camera held by a person or mounted on a robot. Therefore, the area below the image is generally the ground or other feasible areas. For an image in the ground data, assuming that its horizontal coordinates are Its The corresponding pixel coordinates in are Its pixel value is recorded as Its The corresponding pixel coordinates in are also (u i ,v i ), whose pixel value is recorded as remember The same process is performed on all ground images to obtain a set of difference values ​​D = {d0, d1, ..., d M}, M is the number of ground images. The median of D is d m , then the z-direction offset between the aerial point cloud and the ground point cloud is d m .

[0152] The three-dimensional point cloud data C of the ground after horizontal offset is processed according to the height offset g1 The adjustment includes: converting the ground three-dimensional point cloud data C g1 The height offset is offset on the z coordinate to obtain the ground three-dimensional point cloud data C after the height offset g2 . C g2 That is, the sky point cloud C s0 Fully aligned ground point cloud

[0153] In the above technical scheme: GPS alignment is performed on the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data through GPS information of the aerial image and the ground image; a first horizontal grid plane is performed based on the projection horizontal plane range of the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data to count the contour maps of the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data, and the horizontal offset between the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data is obtained by template matching of the contour map, so as to align the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data in the horizontal direction; the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data aligned in the horizontal direction are projected onto the horizontal plane again and a second horizontal grid plane is performed to count the height maps of the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data, and the pixel values ​​corresponding to the z coordinate values ​​in the height maps are counted to obtain the offset between the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data in the z direction (that is, the height direction), so as to align the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data in the vertical direction.

[0154] The technical solution of the present invention can utilize the projection images of the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data, and statistically analyze the contour map and height map of the aerial 3D reconstructed point cloud data and the ground 3D reconstructed point cloud data, so as to obtain the horizontal deviation and height deviation of the two sets of data, and realize effective data alignment and matching.

[0155] Corresponding to the above embodiment, Figure 2 As shown, this embodiment also provides a system for aligning aerial 3D reconstruction data with ground 3D reconstruction data, including: a first acquisition unit, a second acquisition unit, a 3D reconstruction unit, a first projection statistics unit, a first offset adjustment unit, a second projection statistics unit, and a second offset adjustment unit. The first acquisition unit, the second acquisition unit, the 3D reconstruction unit, the first projection statistics unit, the first offset adjustment unit, the second projection statistics unit, and the second offset adjustment unit are respectively used to perform the above steps S100 to S106 (except steps S200 to S201).

[0156] Continue to refer Figure 3 ,exist Figure 2 Based on the system shown, the system for aligning aerial 3D reconstruction data with ground 3D reconstruction data may further include: a GPS alignment unit and an output unit. The GPS alignment unit and the output unit are used to execute the above steps S200 to S201 respectively.

[0157] Combination Figure 4 , the first projection statistics unit further comprises:

[0158] The first horizontal plane setting unit is adapted to set the horizontal plane range covered by the projected aerial three-dimensional point cloud data and the projected ground three-dimensional point cloud data as in

[0159] The first grid division subunit is adapted to divide the horizontal plane range covered by the projection into Performing division to generate the first preset horizontal grid;

[0160] The first statistical subunit is adapted to make the point cloud coordinates of the currently projected aerial 3D point cloud data or the projected ground 3D point cloud data be (x a ,y a ,z a ), the grid coordinates of the first preset horizontal grid are (u a ,v a ), then the following formula is used to count whether the point cloud coordinates fall into the grid (u a ,v a ):

[0161] like Then the point cloud coordinates (x a ,y a ,z a ) falls into the grid, floor() is a function that rounds down the floating point numbers therein;

[0162] The first contour image generating subunit is adapted to use the number of point clouds of the projected aerial three-dimensional point cloud data falling into the first preset horizontal grid as the contour image I C s The pixel value corresponding to the coordinate;

[0163] The second contour image generating subunit is adapted to use the number of point clouds of the projected ground three-dimensional point cloud data falling into the first preset horizontal grid as the contour image I C g The pixel value corresponding to the coordinate.

[0164] Combination Figure 5 , the second projection statistics unit further comprises:

[0165] The second horizontal plane setting subunit is adapted to recalculate the projected aerial 3D point cloud data and the horizontally offset ground 3D point cloud data C g1 Horizontal range covered by projection in,

[0166] The second grid division subunit is adapted to divide the horizontal plane range covered by the projection into two parts with a preset step size b. Performing division to generate the second preset horizontal grid;

[0167] The second statistical subunit is adapted to calculate the currently projected aerial three-dimensional point cloud data or ground three-dimensional point cloud data C g1 The point cloud coordinates are (x b ,y b ,z b ), the grid coordinates of the second preset horizontal grid are (u b ,v b ), then the following formula is used to count the number of cells that fall into the grid (u b ,v b )’s point cloud number:

[0168] like Then the point cloud coordinates (x b ,y b ,z b ) falls into this grid (u b ,v b );

[0169] The first height image generating subunit is adapted to place the projected aerial three-dimensional point cloud data into a grid (u b ,v b ) The average value of the z coordinates of the point cloud is taken as the grid (u b ,v b ) and outputs the height image I of the projected aerial three-dimensional point cloud data at this height. h s The pixel value corresponding to the coordinate;

[0170] The second height image generation subunit is adapted to convert the ground three-dimensional point cloud data C g1 falls into the grid (u b ,v b ) is taken as the average value of the z coordinates of the point cloud of the grid (u b ,v b ) and output the ground 3D point cloud data C at this height g1 Height image I h g1 The pixel value corresponding to the coordinate.

[0171] Based on the above embodiments, the technical solution of the present invention also provides a system for aligning aerial three-dimensional reconstruction data and ground three-dimensional reconstruction data, including: a processor and a memory; a computer program is stored in the memory, and when the processor executes the computer program, the method steps for aligning aerial three-dimensional reconstruction data and ground three-dimensional reconstruction data provided in this embodiment are executed.

[0172] Based on the above embodiments, the technical solution of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method steps provided in this embodiment for aligning aerial three-dimensional reconstruction data and ground three-dimensional reconstruction data.

[0173] In the above embodiment, based on the aerial image P i s and ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g Afterwards, the GPS alignment method of step S200 and step S201 is used to improve the spatiotemporal accuracy and consistency of the two sets of data. In other embodiments of the technical solution of the present invention, other methods can be used to replace the GPS alignment method to improve the spatiotemporal accuracy of the data itself, or when collecting aerial images P i s and ground image P j g If the devices have system consistency, the GPS alignment of step S200 and step S201 can also be omitted.

[0174] More specifically, the aerial three-dimensional point cloud data C can also be realized in the following ways s And ground 3D point cloud data C g Positioning calibration: An augmented reality (AR)-based system can be used to achieve positioning calibration of two sets of data using the positioning service provided by the camera and IMU (inertial measurement unit) data; the AR system can be used to identify preset landmarks or use SLAM (simultaneous localization and mapping) technology as an alternative to GPS alignment; positioning calibration can also be performed based on the device that captures the image through feature matching and image processing technology; UWB technology can be used to further provide high-precision short-range wireless positioning of the target scene, and data calibration can be achieved by measuring the distance by sending and receiving extremely short pulses; Wi-Fi positioning means based on the shooting device can be used to determine the image position by utilizing the signal strength of surrounding Wi-Fi hotspots to achieve positioning calibration; the AI ​​model of the IDPS is used to obtain sensor-related data of the shooting device in real time, and these sensor data are used to perform positioning calibration on the relevant point cloud data.

[0175] Therefore, reference Figure 6 , and also provides a method for aligning aerial 3D reconstruction data with ground 3D reconstruction data, comprising the following steps:

[0176] Step S100, obtaining a target scene image P from the air i s, i can be 1, 2, 3, ..., N, where N is a natural number greater than 1;

[0177] Step S101, obtain ground image P j g , j can be 1, 2, 3, ..., M, where M is a natural number greater than 1;

[0178] Step S102', the target scene image P i s And ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g ;

[0179] Step S103', convert the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Project to the horizontal plane and count the projected aerial 3D point cloud data C s And the projected ground three-dimensional point cloud data C g The number of points falling in the first preset horizontal grid is used to obtain the projected aerial three-dimensional point cloud data C s The contour image I C s And the projected ground three-dimensional point cloud data C g The contour image I C g ;

[0180] Step S104', based on the projected aerial three-dimensional point cloud data C s The contour image I C s And the projected ground three-dimensional point cloud data C g The contour image I C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjust to obtain the ground three-dimensional point cloud data C after horizontal offset g1 ;

[0181] Step S105', convert the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Project in the horizontal direction and count the projected aerial 3D point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1The number of points falling in the second preset horizontal grid is used to obtain the projected aerial three-dimensional point cloud data C s Height image I h s And the ground three-dimensional point cloud data C after horizontal offset g1 Height image I h g1 ;

[0182] Step S106', based on the aerial three-dimensional point cloud data C s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offset between the two, and the ground three-dimensional point cloud data C after the horizontal offset according to the height offset g1 Adjust to obtain the ground 3D point cloud data C after height offset g2 ; The ground three-dimensional point cloud data C after the height offset g2 With the aerial 3D point cloud data C s Alignment.

[0183] exist Figure 6 In the process steps shown, different from Figure 1 The process steps shown in the figure use the aerial 3D point cloud data C generated by the system s And ground 3D point cloud data C g , directly execute relevant projection, generate contour map, re-projection, and generate height map and other process steps, which can be output after calibration through the data positioning calibration scheme of the existing technology, or can be directly taken under the premise of system consistency, Figure 6 The process steps shown in the figure do not apply to the aerial 3D point cloud data C s And ground 3D point cloud data C g The consistency calibration method is defined.

[0184] Figure 6 For other method processes of steps S100 to S106' shown in FIG. Figure 1 The method process of steps S100 to S106 is not repeated in this embodiment.

[0185] Corresponding to the above Figure 6 The steps of the process can be used independently as Figure 2The system for aligning the aerial 3D reconstruction data with the ground 3D reconstruction data is implemented as shown, wherein the first acquisition unit, the second acquisition unit, the third dimensional reconstruction unit, the first projection statistics unit, the first offset adjustment unit, the second projection statistics unit and the second offset adjustment unit of the system are respectively used to perform Figure 6 The first projection statistics unit and the second projection statistics unit can respectively adopt the following steps S100 to S106': Figure 4 and Figure 5 The example shown is to execute step S103 ′ and step S105 ′.

[0186] Corresponding to the above Figure 6 The technical solution of the present invention also provides a system for aligning aerial 3D reconstruction data with ground 3D reconstruction data, including: a processor and a memory; the memory stores a computer program, and the processor executes the computer program when executing the computer program Figure 6 The method steps for aligning aerial 3D reconstruction data and ground 3D reconstruction data are shown.

[0187] Corresponding to the above Figure 6 The technical solution of the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, Figure 6 The method steps for aligning aerial 3D reconstruction data and ground 3D reconstruction data are shown.

[0188] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for aligning aerial 3D reconstruction data with ground 3D reconstruction data, characterized in that: include: Get the target scene image P from the air i s , i can be 1, 2, 3, ..., N, where N is a natural number greater than 1; Get ground image P j g , j can be 1, 2, 3, ..., M, where M is a natural number greater than 1; For the target scene image P i s And ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g ; Based on the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Projecting onto a horizontal plane, respectively counting the number of projected aerial 3D point cloud data and projected ground 3D point cloud data falling in a first preset horizontal grid, so as to obtain a contour image I of the projected aerial 3D point cloud data C s And the contour image I of the projected ground 3D point cloud data C g ; Contour image based on projected aerial 3D point cloud data I C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjust to obtain the ground three-dimensional point cloud data C after horizontal offset g1 ; Based on the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Project in the horizontal direction and count the projected aerial 3D point cloud data and the ground 3D point cloud data after horizontal offset C g1 The number of points falling in the second preset horizontal grid is used to obtain the height image I of the projected aerial three-dimensional point cloud data. h s And the ground three-dimensional point cloud data C after horizontal offset g1 Height image I h g1 ; Based on the aerial three-dimensional point cloud data C s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offset between the two, and the ground three-dimensional point cloud data C after the horizontal offset according to the height offset g1 Adjust to obtain the ground 3D point cloud data C after height offset g2 ; The ground three-dimensional point cloud data C after the height offset g2 With the aerial 3D point cloud data C s Alignment.

2. The method for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 1, characterized in that: The target scene picture P i s And ground image P j g The first GPS information and the second GPS information are respectively provided, and the three-dimensional point cloud data C in the air is based on the three-dimensional point cloud data C in the air. s And ground 3D point cloud data C g Projection onto the horizontal plane includes: According to the first GPS information and the second GPS information, the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g respectively aligned to the GPS so that the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Have the same GPS information at the same location; Output the aerial 3D point cloud data C aligned with GPS s0 And ground 3D point cloud data C g0 ; The aerial three-dimensional point cloud data C s0 And ground 3D point cloud data C g0 Projection onto the horizontal plane; The projected aerial 3D point cloud data and the projected ground 3D point cloud data are respectively based on the aerial 3D point cloud data C s0 And ground 3D point cloud data C g0 Projection obtained.

3. The method for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 1 or 2, characterized in that: The method of respectively counting the number of projected aerial three-dimensional point cloud data and projected ground three-dimensional point cloud data falling in the first preset horizontal grid comprises: Assume that the point cloud coordinate range of the projected aerial 3D point cloud data is in, They represent the minimum values ​​of the x-coordinate, y-coordinate, and z-coordinate of each point cloud in the aerial 3D point cloud data. Respectively represent the maximum value of each point cloud in the three-dimensional aerial point cloud data at the x-coordinate, y-coordinate, and z-coordinate; Assume that the point cloud coordinate range of the projected ground 3D point cloud data is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate; Assume that the horizontal plane range covered by the projected aerial 3D point cloud data and the projected ground 3D point cloud data is in The horizontal plane range covered by the projection is set to the preset step size α Performing division to generate the first preset horizontal grid; Let the point cloud coordinates of the currently projected aerial 3D point cloud data or the projected ground 3D point cloud data be (x a ,y a ,z a ), the grid coordinates of the first preset horizontal grid are (u a ,v a ), then the following formula is used to count whether the point cloud coordinates fall into the grid (u a ,v a ): like Then the point cloud coordinates (x a ,y a ,z a ) falls into the grid, floor() is a function that rounds down the floating point numbers therein; The number of point clouds that the projected aerial three-dimensional point cloud data falls into the first preset horizontal grid is used as the contour image I C s The pixel value corresponding to the coordinate; The number of point clouds that the projected ground three-dimensional point cloud data falls into the first preset horizontal grid is used as the contour image I C g The pixel value corresponding to the coordinate; Assume that the ground three-dimensional point cloud data C g1 The coordinate range is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate; The projected aerial three-dimensional point cloud data and the ground three-dimensional point cloud data C after horizontal offset are respectively counted. g1 The quantities that fall into the second preset horizontal grid include: Recalculate the projected aerial 3D point cloud data and the horizontally offset ground 3D point cloud data C g1 Horizontal range covered by projection in, The horizontal plane range covered by the projection is set to the preset step size b. Performing division to generate the second preset horizontal grid; Let the point cloud coordinates of the currently projected aerial 3D point cloud data or ground 3D point cloud data Cg1 be (x b ,y b ,z b ), the grid coordinates of the second preset horizontal grid are (u b ,v b ), then the following formula is used to count the number of cells that fall into the grid (u b ,v b )’s point cloud number: like Then the point cloud coordinates (x b ,y b ,z b ) falls into this grid (u b ,v b ); The projected aerial 3D point cloud data falls into the grid (u b ,v b ) The average value of the z coordinates of the point cloud is taken as the grid (u b ,v b ) and outputs the height image I of the projected aerial three-dimensional point cloud data at this height. h s The pixel value corresponding to the coordinate; The ground 3D point cloud data C g1 falls into the grid (u b ,v b ) is taken as the average value of the z coordinates of the point cloud of the grid (u b ,v b ) and output the ground 3D point cloud data C at this height g1 Height image I h g1 The pixel value corresponding to the coordinate.

4. The method for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 1, characterized in that: The contour image I based on the projected aerial three-dimensional point cloud data C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjustments include: For the contour image I C s and contour image I C g Perform template matching to locate the contour image I C s and contour image I C g The displacement pixels (d u ,d v ), assuming that the step size of the first preset horizontal grid is α, then the contour image I C s and contour image I C g The horizontal offset between u ×a,d v ×a); According to the horizontal offset (d u ×a,d v ×a) the ground three-dimensional point cloud data C g Adjust and recalculate the coordinate range of the ground point cloud, and output the ground 3D point cloud data C after horizontal offset g1 .

5. The method for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 1, characterized in that: Based on the aerial three-dimensional point cloud data C s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offsets between include: Assume that the height image I h g1 The horizontal coordinates of the image are The corresponding grid coordinates are Take the grid coordinates (u i ,v i ) Similarly, take the height image I h s At the corresponding grid coordinates (u i ,v i ) Get height image I h g1 And height image I h s The offset of all pixel values ​​at corresponding grid coordinates; Take the average value of all pixel value offsets as the height image I h s And height image I h g1 The height offset between .

6. The method for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 1, characterized in that: The three-dimensional point cloud data C of the ground after horizontal offset is processed according to the height offset g1 Adjustments include: The ground three-dimensional point cloud data C g1 The height offset is offset on the z coordinate to obtain the ground three-dimensional point cloud data C after the height offset g2 .

7. A system for aligning aerial 3D reconstruction data with ground 3D reconstruction data, characterized in that: include: The first acquisition unit is adapted to acquire a target scene image P from the air. i s , i can be 1, 2, 3, ..., N, where N is a natural number greater than 1; The second acquisition unit is adapted to acquire the ground image P j g , j can be 1, 2, 3, ..., M, where M is a natural number greater than 1; A three-dimensional reconstruction unit, adapted to reconstruct the target scene image P i s And ground image P j g Perform 3D reconstruction to obtain aerial 3D point cloud data C s And ground 3D point cloud data C g ; The first projection statistics unit is adapted to be based on the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Projecting onto a horizontal plane, respectively counting the number of projected aerial 3D point cloud data and projected ground 3D point cloud data falling in a first preset horizontal grid, so as to obtain a contour image I of the projected aerial 3D point cloud data C s And the contour image I of the projected ground 3D point cloud data C g ; The first offset adjustment unit is adapted to adjust the contour image I based on the projected aerial three-dimensional point cloud data C s And the contour image I of the projected ground 3D point cloud data C g Calculate the contour image I C s and contour image I C g The horizontal offset between the ground three-dimensional point cloud data C g Adjust to obtain the ground three-dimensional point cloud data C after horizontal offset g1 ; The second projection statistics unit is adapted to be based on the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Project in the horizontal direction and count the projected aerial 3D point cloud data and the ground 3D point cloud data after horizontal offset C g1 The number of points falling in the second preset horizontal grid is used to obtain the height image I of the projected aerial three-dimensional point cloud data. h s And the ground three-dimensional point cloud data C after horizontal offset g1 Height image I h g1 ; The second offset adjustment unit is adapted to adjust the airborne three-dimensional point cloud data C based on the airborne three-dimensional point cloud data C s Height image I h s And the ground 3D point cloud data C after migration g1 Height image I h g1 Calculate the height image I h s And height image I h g1 The height offset between the two, and the ground three-dimensional point cloud data C after the horizontal offset according to the height offset g1 Adjust to obtain the ground 3D point cloud data C after height offset g2 ; The ground three-dimensional point cloud data C after the height offset g2 With the aerial 3D point cloud data C s Alignment.

8. The system for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 7, characterized in that: The target scene picture P i s And ground image P j g The system comprises first GPS information and second GPS information respectively, and further comprises: The GPS alignment unit is adapted to align the aerial three-dimensional point cloud data C according to the first GPS information and the second GPS information. s And ground 3D point cloud data C g respectively aligned to the GPS so that the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Have the same GPS information at the same location; Output unit, suitable for outputting aerial 3D point cloud data C aligned with GPS s0 And ground 3D point cloud data C g0 ; Based on the aerial three-dimensional point cloud data C s And ground 3D point cloud data C g Projecting onto a horizontal plane includes: projecting the aerial three-dimensional point cloud data C s0 And ground 3D point cloud data C g0 Projection onto the horizontal plane; Based on the aerial three-dimensional point cloud data C s And the ground three-dimensional point cloud data C after horizontal offset g1 Projecting in the horizontal direction includes: projecting the aerial three-dimensional point cloud data C s0 And the ground three-dimensional point cloud data C after horizontal offset g1 Projection in horizontal direction; The projected aerial 3D point cloud data and the projected ground 3D point cloud data are respectively based on the aerial 3D point cloud data C s0 And ground 3D point cloud data C g0 Projection obtained.

9. The method for aligning aerial 3D reconstruction data with ground 3D reconstruction data according to claim 7 or 8, characterized in that: Assume that the point cloud coordinate range of the projected aerial 3D point cloud data is in, They represent the minimum values ​​of the x-coordinate, y-coordinate, and z-coordinate of each point cloud in the aerial 3D point cloud data. Respectively represent the maximum value of each point cloud in the three-dimensional aerial point cloud data at the x-coordinate, y-coordinate, and z-coordinate; Assume that the point cloud coordinate range of the projected ground 3D point cloud data is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate; The first projection statistics unit comprises: The first horizontal plane setting unit is adapted to set the horizontal plane range covered by the projected aerial three-dimensional point cloud data and the projected ground three-dimensional point cloud data as in The first grid division subunit is adapted to divide the horizontal plane range covered by the projection into Performing division to generate the first preset horizontal grid; The first statistical subunit is adapted to make the point cloud coordinates of the currently projected aerial 3D point cloud data or the projected ground 3D point cloud data be (x a ,y a ,z a ), the grid coordinates of the first preset horizontal grid are (u a ,v a ), then the following formula is used to count whether the point cloud coordinates fall into the grid (u a ,v a ): like Then the point cloud coordinates (x a ,y a ,z a ) falls into the grid, floor() is a function that rounds down the floating point numbers therein; The first contour image generating subunit is adapted to use the number of point clouds of the projected aerial three-dimensional point cloud data falling into the first preset horizontal grid as the contour image I C s The pixel value corresponding to the coordinate; The second contour image generating subunit is adapted to use the number of point clouds of the projected ground three-dimensional point cloud data falling into the first preset horizontal grid as the contour image I C g The pixel value corresponding to the coordinate; Assume that the ground three-dimensional point cloud data C g1 The coordinate range is in, Respectively represent the minimum value of each point cloud in the ground three-dimensional point cloud data in the x-coordinate, y-coordinate, and z-coordinate. Respectively represent the maximum value of each point cloud in the ground three-dimensional point cloud data at the x-coordinate, y-coordinate, and z-coordinate; The second projection statistics unit comprises: The second horizontal plane setting subunit is adapted to recalculate the projected aerial 3D point cloud data and the horizontally offset ground 3D point cloud data C g1 Horizontal range covered by projection in, The second grid division subunit is adapted to divide the horizontal plane range covered by the projection into two parts with a preset step size b. Performing division to generate the second preset horizontal grid; The second statistical subunit is adapted to calculate the currently projected aerial three-dimensional point cloud data or ground three-dimensional point cloud data C g1 The point cloud coordinates are (x b ,y b ,z b ), the grid coordinates of the second preset horizontal grid are (u b ,v b ), then the following formula is used to count the number of cells that fall into the grid (u b ,v b )’s point cloud number: like Then the point cloud coordinates (x b ,y b ,z b ) falls into this grid (u b ,v b ); The first height image generating subunit is adapted to place the projected aerial three-dimensional point cloud data into a grid (u b ,v b ) The average value of the z coordinates of the point cloud is taken as the grid (u b ,v b ) and outputs the height image I of the projected aerial 3D point cloud data at this height. h s The pixel value corresponding to the coordinate; The second height image generation subunit is adapted to convert the ground three-dimensional point cloud data C g1 falls into the grid (u b ,v b ) is taken as the average value of the z coordinates of the point cloud of the grid (u b ,v b ) and output the ground 3D point cloud data C at this height g1 Height image I h g1 The pixel value corresponding to the coordinate.

10. A system for aligning aerial 3D reconstruction data with ground 3D reconstruction data, characterized in that: include: Processor and memory; The memory stores a computer program, and when the processor executes the computer program, the processor performs the method steps of aligning aerial three-dimensional reconstruction data with ground three-dimensional reconstruction data as claimed in any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps for aligning aerial three-dimensional reconstruction data with ground three-dimensional reconstruction data as described in any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Air-ground bridging image generation method and device, equipment, medium and product

    CN120997354A