Air-ground data fusion method and three-dimensional reconstruction device

Abstract

This application relates to the field of spatial 3D reconstruction technology, specifically to an air-ground data fusion method, a 3D reconstruction device, and a computer storage medium. The air-ground data fusion method uses 3D point image matching to find cross-view matching points based on 3D neighborhoods and descriptors. It merges a large number of 2D features into robust 3D feature representations and performs neighborhood matching in an aligned 3D space, replacing direct image matching. This bypasses the difficulties of 2D matching of air-ground images, solves the problem of cross-view image mismatch, and significantly improves the robustness of air-ground image matching.

Description

Technical Field

[0001] This application relates to the field of spatial three-dimensional reconstruction technology, specifically to an air-ground data fusion method, a three-dimensional reconstruction device, and a computer storage medium. Background Technology

[0002] Traditional 3D reconstruction typically relies on a single platform's data source, such as UAV oblique photogrammetry or ground-based mobile scanning systems. However, in scenarios such as urban canyons, complex industrial facilities, and forest occlusion, there are significant parallax and occlusion differences between the UAV's top-down view and the ground's horizontal view, making it difficult to obtain a complete and consistent 3D structure through single-view reconstruction.

[0003] The inventors of this application discovered in their research that air-ground fusion reconstruction can significantly improve the completeness of the model. However, existing air-ground fusion reconstruction technologies heavily rely on high-precision satellite navigation and positioning technology, which makes it impossible to perform accurate 3D reconstruction in scenarios where navigation and positioning cannot be used. Summary of the Invention

[0004] In view of the above problems, this application provides an air-ground data fusion method, a three-dimensional reconstruction device, and a computer storage medium to solve the above-mentioned technical problems existing in the prior art.

[0005] According to one aspect of the embodiments of this application, an air-to-ground data fusion method is proposed, the method comprising:

[0006] Acquire aerial and ground image data of the area to be scanned;

[0007] Based on the aerial image data and the ground image data, corresponding two-dimensional image feature points and three-dimensional points corresponding to the two-dimensional image feature points are generated respectively, wherein each three-dimensional point corresponds to multiple two-dimensional image feature points;

[0008] The descriptor of the three-dimensional point is determined based on the descriptors of the multiple two-dimensional image feature points corresponding to the three-dimensional point;

[0009] Search for 3D points in the ground image data within the neighborhood of 3D points in the aerial image data;

[0010] If the distance between the descriptor of the three-dimensional point in the aerial image data and the descriptor of the three-dimensional point in the searched ground image data is less than a preset threshold, then the three-dimensional point in the aerial image data and the three-dimensional point in the ground image data are determined as matching three-dimensional points.

[0011] Data fusion is performed on the aerial image data and the ground image data based on the matched 3D points.

[0012] Preferably, in some embodiments, generating corresponding two-dimensional image feature points and corresponding three-dimensional points based on the aerial image data and the ground image data respectively includes:

[0013] Two-dimensional image feature points are extracted from the aerial image data and the ground image data respectively using a preset scale-invariant feature transformation algorithm;

[0014] Perform feature matching on each of the two-dimensional image feature points to determine the corresponding two-dimensional image matching feature points for the same scene in different aerial image data and ground image data;

[0015] Based on the feature points matched in the two-dimensional image, corresponding three-dimensional points are generated using a preset motion recovery structure algorithm.

[0016] Preferably, in some embodiments, multiple image control points are pre-set in the area to be scanned;

[0017] The method further includes:

[0018] Acquire multiple aerial image data and multiple ground image data of multiple control points;

[0019] Based on multiple aerial image data and multiple ground image data of the multiple image control points, respectively construct the distance matrix of the ground image control point set and the distance matrix of the aerial image control point set;

[0020] The air-to-ground coordinate transformation matrix is ​​determined based on the ground control point set distance matrix and the air control point set distance matrix.

[0021] The aerial image data and the ground image data are aligned according to the air-to-ground coordinate transformation matrix.

[0022] Preferably, in some embodiments, determining the air-to-ground coordinate transformation matrix based on the distance matrix of the ground control point set and the distance matrix of the air control point set includes:

[0023] Determine the matrix similarity between the distance matrix of the ground control point set and the distance matrix of the air control point set;

[0024] Based on the matrix similarity, matching point pairs in the distance matrix of the ground control point set and the distance matrix of the air control point set are determined;

[0025] The rotation matrix and translation vector are calculated based on the matching point pairs using a preset Kabsch algorithm.

[0026] The air-ground coordinate transformation matrix is ​​determined based on the rotation matrix and translation vector.

[0027] Preferably, in some embodiments, aligning the aerial image data and the ground image data according to the air-to-ground coordinate transformation matrix further includes:

[0028] Based on the aligned aerial image data and ground image data, feature matching is performed on adjacent aerial image data and ground image data.

[0029] The corresponding three-dimensional points are determined based on the two-dimensional image feature points of the aerial image data and the ground image data after feature matching;

[0030] The air-ground coordinate transformation matrix is ​​optimized based on the three-dimensional points using an iterative nearest-point algorithm;

[0031] The aerial image data and the ground image data are aligned according to the optimized air-to-ground coordinate transformation matrix.

[0032] Preferably, in some embodiments, determining the descriptor of the three-dimensional point based on the descriptors of the plurality of two-dimensional image feature points corresponding to the three-dimensional point includes:

[0033] Calculate the mean descriptor based on the descriptors of the multiple two-dimensional image feature points corresponding to the three-dimensional point;

[0034] Determine the dot product between the descriptors of multiple two-dimensional image feature points and the mean descriptor, and use the descriptor with the smallest dot product as the descriptor of the three-dimensional point.

[0035] Preferably, in some embodiments, the step of searching for three-dimensional points of ground image data within the neighborhood based on three-dimensional points of aerial image data includes:

[0036] Construct a binary tree structure based on the three-dimensional points of the aerial image data and the three-dimensional points of the ground image data;

[0037] The binary tree structure is queried and compared with its neighboring points to determine the similarity of the three-dimensional points;

[0038] Based on the similarity of the three-dimensional points, neighborhood points in the ground image data that match the three-dimensional points in the aerial image data are determined.

[0039] Preferably, in some embodiments, the step of fusing the aerial image data and the ground image data based on the matched three-dimensional points includes:

[0040] A global bundle adjustment map is constructed based on the camera pose and 3D points corresponding to the aerial image data, the camera pose and 3D points corresponding to the ground image data, the distance matrix of the ground control point set, the matching point pairs in the distance matrix of the aerial control point set, and the matching 3D points.

[0041] The global bundle adjustment map is iteratively optimized based on the preset minimization of reprojection error, minimization of 3D point distance constraints, and minimization of transformation priors, thereby achieving data fusion of the aerial image data and the ground image data.

[0042] According to another aspect of the embodiments of this application, a three-dimensional reconstruction device is proposed, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0043] The memory is used to store at least one program that causes the processor to execute the air-to-ground data fusion method described in the above embodiments.

[0044] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one program is stored in the storage medium, and when the program is run on a 3D reconstruction device, the 3D reconstruction device performs the air-ground data fusion method described in the above embodiments.

[0045] In summary, the air-to-ground data fusion method proposed in this application acquires two-dimensional image feature points from both aerial and ground images. It then determines the descriptors of corresponding three-dimensional points based on the descriptors of these two-dimensional feature points. Next, it searches for three-dimensional points in neighboring ground images based on the three-dimensional points in the aerial image. Finally, it determines matching three-dimensional points in the aerial and ground images based on the distance between the descriptors of these three-dimensional points. After matching the three-dimensional points in the aerial and ground images, it establishes a matching relationship between the feature points in the aerial and ground images based on these matching three-dimensional points, and completes data fusion based on this matching relationship. This method of image matching using three-dimensional points finds cross-view matching points based on 3D neighborhood and descriptors, merging a large number of 2D features into robust 3D feature representations. Neighborhood matching is performed in an aligned 3D space, replacing direct image matching. This bypasses the difficulty of 2D matching between aerial and ground images, solves the problem of cross-view image mismatch, and significantly improves the robustness of aerial-to-ground image matching.

[0046] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0047] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0048] Figure 1 This is a flowchart illustrating an air-to-ground data fusion method proposed in an embodiment of this application.

[0049] Figure 2 This is a schematic diagram of the control point annotation proposed in an embodiment of this application;

[0050] Figure 3 This is a flowchart of the three-dimensional point generation method proposed in the embodiments of this application;

[0051] Figure 4 This is a flowchart of the preliminary alignment method for air-ground image data proposed in an embodiment of this application;

[0052] Figure 5 This is a flowchart of the method for determining the air-to-ground coordinate transformation matrix proposed in an embodiment of this application;

[0053] Figure 6 This is a flowchart of the air-to-ground coordinate transformation matrix optimization method proposed in the embodiments of this application;

[0054] Figure 7 This is a flowchart of the three-dimensional point neighborhood matching process proposed in the embodiments of this application;

[0055] Figure 8 This is a schematic diagram of three-dimensional point matching proposed in an embodiment of this application;

[0056] Figure 9 This is a schematic diagram of the structure of the three-dimensional reconstruction device proposed in the embodiments of this application. Detailed Implementation

[0057] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0058] In obstructed scenarios such as urban canyons, complex industrial facilities, and forest areas, air-to-ground fusion reconstruction can significantly improve model integrity, but this requires achieving high-precision positioning and joint optimization of aerial and ground-scanned images in the absence of RTK (Real-Time Kinematic) and external geographic references. In these complex scenarios, extrinsic parameters are difficult to obtain, and the perspectives of aerial and ground images differ greatly (e.g., top-down and horizontal views), making traditional feature matching almost impossible and hindering the establishment of initial alignment in existing air-to-ground fusion methods. Furthermore, these scenarios typically lack Global Navigation Satellite System (GNSS) signals, requiring measured coordinates for ground control points. Existing technologies rely on RTK and cannot be used in GNSS-free environments. Simultaneously, cross-view matching between aerial and ground-scanned images presents challenges. Due to scale variations, lighting differences, and severe tilt, corresponding features are scarce in both aerial and ground images, making it difficult for existing air-to-ground fusion techniques to construct optimizable global BA (Bundle Adjustment) constraints.

[0059] To address the aforementioned issues, the applicant proposes an air-to-ground data fusion method. This method acquires 2D image feature points from both aerial and ground images, determines the descriptors of corresponding 3D points based on the descriptors of these 2D feature points, searches for 3D points in neighboring ground images based on the 3D points in the aerial image, and determines matching 3D points in the aerial and ground images based on the distance between their descriptors. After matching the 3D points in the aerial and ground images, a matching relationship is established between the feature points in the aerial and ground images based on the matching 3D points, and data fusion is completed based on this matching relationship. This image matching method using 3D points finds cross-view matching points based on 3D neighborhood and descriptors, merging a large number of 2D features into robust 3D feature representations. Neighborhood matching is performed in an aligned 3D space, replacing direct image matching. This bypasses the difficulties of 2D matching between aerial and ground images, solves the problem of cross-view image mismatch, and significantly improves the robustness of aerial-to-ground image matching. Furthermore, this embodiment of the application also achieves automatic alignment of air-to-ground data without initial extrinsic parameters by setting image control points within the area to be scanned, realizing visual triangulation through multi-angle images of the image control points, and establishing preliminary matching relationships through a distance matrix using the visual 3D reconstruction results of the image control points. This achieves air-to-ground coordinate transformation and alignment without relying on RTK, GPS, or LiDAR coordinate systems. Further, this embodiment of the application also constructs a global BA optimization map jointly for air and ground, introducing cross-view constraints generated by 3D matching to achieve global BA constraints. This solves the problem of difficulty in constructing globally optimizable constraints for air-to-ground images due to scale variations, illumination differences, and severe tilt.

[0060] The air-to-ground data fusion method proposed in this application can be applied to various scenarios and various 3D reconstruction devices, such as drones, handheld radars, robots, and other 3D reconstruction devices. It is not limited to these devices in this application.

[0061] like Figure 1 The diagram illustrates a flowchart of an air-to-ground data fusion method proposed in an embodiment of this application, including:

[0062] Step S100: Acquire aerial image data and ground image data of the area to be scanned;

[0063] The area to be scanned is the region to be reconstructed in 3D, which can be an urban canyon, a factory, etc. The aerial image data can be obtained by using a drone equipped with a LiDAR or camera to take aerial photos of the urban canyon, factory, or other areas to be scanned, and can include camera pose data, angle, etc.

[0064] The ground image data can be obtained by capturing continuous image sequences, inertial data, and lidar data in the area to be scanned, either manually or robotically, using multi-eye fisheye cameras, IMU devices, or lidar devices. When capturing or scanning, if lidar is used, the aerial and ground image data can be provided with initial values ​​for each frame using an LIO lidar-inertial odometry system (e.g., a LIVO lidar-inertial-visual odometry system); if a camera or other imaging device is used, VIO (e.g., VINS-Fusion, MSCKF) can be used to estimate the initial pose.

[0065] In some embodiments, to eliminate reliance on RTK and achieve alignment of aerial image data with ground image data in GNSS-free environments, preferably, in this application embodiment, several image control points can be pre-arranged within the area to be scanned, such as... Figure 2 As shown, the control points (GCPs) are distributed as evenly as possible around the perimeter. The ground-based scanning device scans the area to be scanned, circling the GCPs as it passes them, thus acquiring multiple ground image data points. The aerial drone takes off from the GCP and captures aerial images above the area to be scanned, allowing its camera to capture multiple aerial image data points. This multiple GCP image data ensures the stability of visual triangulation, resulting in high-precision 3D coordinates for the GCPs. The GCPs may have special shapes or colors to facilitate identification from aerial or ground images.

[0066] Step S200: Generate corresponding two-dimensional image feature points and three-dimensional points corresponding to the two-dimensional image feature points based on the aerial image data and the ground image data, respectively, wherein each three-dimensional point corresponds to multiple two-dimensional image feature points;

[0067] After acquiring the aerial and ground image data, it is necessary to extract image feature points from these image data. These image feature points include two-dimensional feature points and three-dimensional feature points. Figure 3 The flowchart of the image feature point extraction method is shown, including:

[0068] Step S210: Extract corresponding two-dimensional image feature points from the aerial image data and ground image data respectively using a preset scale-invariant feature transformation algorithm;

[0069] First, two-dimensional feature points are extracted from each of the aerial and ground image data using a preset Scale-Invariant Feature Transform (SIFT) algorithm. These two-dimensional feature points are unique and repeatable points in the image, such as corners and edges.

[0070] The Scale-Variable Feature Transformation (SIFT) algorithm is a feature detection and descriptor extraction algorithm used in image processing and computer vision. The main feature of the SIFT algorithm is its invariance to scale and rotation changes in the image, as well as its robustness to changes in illumination, viewpoint, and noise.

[0071] Step S220: Perform feature matching on each of the two-dimensional image feature points to determine the two-dimensional image matching feature points corresponding to the same scene in different aerial image data and ground image data;

[0072] Feature matching is performed on 2D image feature points from different aerial and ground image data. Nearest neighbor and ratio tests are used for feature matching to determine corresponding points in different images of the same scene. This can be achieved by calculating the similarity between the descriptors of each 2D image feature point, thus realizing feature matching between 2D image feature points in aerial and ground image data. Furthermore, IMU-guided window constraints can be used to improve matching stability.

[0073] Step S230: Generate corresponding three-dimensional points based on the feature points matched in the two-dimensional image using a preset motion recovery structure algorithm;

[0074] A pre-defined Structure from Motion (SFM) algorithm is applied to estimate the camera's motion and the 3D structure of the scene. The SFM algorithm analyzes the positions and correspondences of feature points in different images to deduce the camera's motion trajectory and the 3D coordinates of points in the scene, constructing a sparse point cloud in both ground and aerial coordinate systems. Specifically, the SFM algorithm is a technique for recovering 3D structure and camera motion from a series of 2D images. It estimates the camera's pose and the 3D coordinates of points in the scene by matching feature points in different images.

[0075] On the ground, we obtain the pose data Pose_g(i) of the ground image, the point cloud data PointCloud_g, and the 3D coordinates of the control points. All of this pose data, point cloud data, and control point 3D coordinates are based on the ground coordinate system. On the aerial side, we obtain the pose data Pose_u(i) of the aerial image, the point cloud data PointCloud_u, and the 3D coordinates of the control points. All of this pose data, point cloud data, and control point 3D coordinates are based on the ground coordinate system.

[0076] Furthermore, in this embodiment, to reduce reliance on RTK technology and achieve air-to-ground image alignment in the absence of GNSS, after acquiring the image data of the control points in step S100, this embodiment can also achieve preliminary alignment of the air image data and the ground image data through visual triangulation based on the control points, and obtain the coordinate transformation matrix between the coordinate systems through pure visual triangulation. Specifically, as shown... Figure 4 As shown, it includes:

[0077] Step S240: Acquire multiple aerial image data and multiple ground image data of the multiple image control points;

[0078] For ground image data: When scanning, the handheld ground device needs to scan around a control point, taking an image of the control point to determine its three-dimensional coordinates in the ground coordinate system: { , , …}.

[0079] For aerial imagery data: When using a drone for aerial photography, flying low to capture images of the control points (GCPs) is beneficial. Taking off and landing near the GCPs can also improve image quality. Determine the three-dimensional coordinates of the GCPs in the aerial coordinate system: { , ,…}.

[0080] Step S250: Construct a distance matrix for the ground control point set and a distance matrix for the air control point set based on multiple aerial image data and multiple ground image data of the multiple control points, respectively;

[0081] Construct the distance matrix DG(i,j) for the ground phased array point set and the distance matrix DU(i,j) for the airborne phased array point set respectively:

[0082] DG(i,j) = || – ||;

[0083] DU(i,j) = || – ||.

[0084] Step S260: Determine the air-to-ground coordinate transformation matrix based on the distance matrix of the ground control point set and the distance matrix of the air control point set;

[0085] Since the distance matrix remains unchanged under rigid body transformations (rotation + translation) and does not require the same coordinate system, corresponding point pairs can be identified through the matrix structure characteristics (relative distance invariance).

[0086] Specifically, the process of determining the air-to-ground coordinate transformation matrix is ​​as follows: Figure 5 As shown, it includes:

[0087] Step S2601: Determine the matrix similarity between the distance matrix of the ground control point set and the distance matrix of the air control point set;

[0088] First, the distance matrices of the ground control point set and the aerial control point set are normalized to ensure they have the same scale and distribution. Then, the matrix similarity between the normalized ground control point set distance matrix and the aerial control point set distance matrix is ​​calculated based on Euclidean distance or cosine similarity.

[0089] Step S2602: Determine matching point pairs in the distance matrix of the ground control point set and the distance matrix of the air control point set based on the matrix similarity;

[0090] Based on the matrix similarity, find similar rows (columns) in the distance matrix of the ground image data and the aerial image data. The two-dimensional image feature points corresponding to these rows (columns) are the matching point pairs, such as: {(G1,U1),(G2,U2),…}.

[0091] Step S2603: Calculate the rotation matrix and translation vector based on the matching point pair using the preset Kabsch algorithm;

[0092] The transformation matrix of the air-ground coordinates is solved based on Singular Value Decomposition (SVD), and the rotation matrix R and translation vector t are obtained by using the Kabsch algorithm with matching point pairs.

[0093] Step S2604: Determine the air-ground coordinate transformation matrix based on the rotation matrix and translation vector;

[0094] The relationship between three-dimensional points in aerial image data and three-dimensional points in ground image data is determined by the following formula:

[0095] U≈R*G+t;

[0096] The air-ground coordinate transformation matrix Tg_u is determined based on the above relationship.

[0097] The above embodiments identify corresponding matching point pairs through matrix structure features, namely relative distance invariance, avoiding problems such as large changes in 2D feature scale, differences in image perspective, and image matching failure caused by weak texture. At the same time, through SVD decomposition, the problem of automatic coarse alignment of air and ground data under the condition of no initial extrinsic parameters can be effectively solved, providing a foundation for subsequent high-precision positioning and 3D reconstruction.

[0098] Step S270: Align the aerial image data and the ground image data according to the air-to-ground coordinate transformation matrix;

[0099] By using a coarse alignment transformation matrix obtained based on distance matrix matching and the Kabsch algorithm, preliminary alignment of air-ground data can be achieved without external geographic references such as RTK.

[0100] While the above method achieves preliminary alignment of the aerial image data and the ground image data, its alignment accuracy is affected by various factors, such as errors in feature matching and distance matrix calculation. Another embodiment of this application proposes a method to further optimize the aerial-ground coordinate transformation matrix based on the above-mentioned aerial-ground image data alignment, making the aerial-ground transformation more accurate.

[0101] Specifically, such as Figure 6 As shown, it includes:

[0102] Step S2701: Based on the aligned aerial image data and ground image data, perform feature matching on neighboring aerial image data and ground image data;

[0103] After initially aligning the aerial and ground image data through the rotation and translation operations in the above steps, feature matching is performed on the adjacent aerial and ground image data to find corresponding feature points in the aerial and ground images. Feature matching algorithms such as SIFT, SURF, or ORB can be used for feature matching.

[0104] Step S2702: Determine the corresponding three-dimensional points based on the two-dimensional image feature points of the aerial image data and the ground image data after feature matching;

[0105] After feature matching, the corresponding two-dimensional image feature points in the two images are converted into three-dimensional points using camera parameters.

[0106] Step S2703: Optimize the air-ground coordinate transformation matrix based on the three-dimensional points using the iterative nearest point algorithm;

[0107] The Iterative Closest Point (ICP) algorithm is a point cloud registration method that uses an iterative approach to continuously find the best match between three-dimensional points, thereby solving for a more accurate air-ground coordinate transformation matrix.

[0108] Step S2704: Align the aerial image data and the ground image data according to the optimized air-to-ground coordinate transformation matrix;

[0109] By aligning the aerial image data and the ground image data using the optimized air-to-ground coordinate transformation matrix, a precise transformation relationship between the air-to-ground coordinate systems is solved, making the air-to-ground transformation more accurate.

[0110] After determining the optimized air-ground coordinate transformation matrix in step 200, although alignment between the air-ground coordinate systems is achieved, significant distortion occurs between air-ground images due to the shooting angle, leading to numerous matching failures. Therefore, in this embodiment, feature matching between air-ground images is further performed in step S300. In this embodiment, instead of directly performing feature matching between images, image matching is performed using three-dimensional point descriptors.

[0111] Step S300: Determine the descriptor of the three-dimensional point based on the descriptors of the multiple two-dimensional image feature points corresponding to the three-dimensional point;

[0112] The descriptor, in computer vision, is a mathematical representation used to describe local features of an image. It is typically a vector or matrix containing information about a local region (such as keypoints or points of interest) in the image, including texture, shape, and color. The purpose of descriptors is to perform feature matching between different images, thereby enabling tasks such as image registration, object recognition, and 3D reconstruction.

[0113] In order to accurately determine the descriptor of the three-dimensional point, the median method is used in this embodiment of the application to determine the descriptor of the three-dimensional point.

[0114] First, the mean descriptor is calculated based on the descriptors of the multiple two-dimensional image feature points corresponding to the three-dimensional point.

[0115] Each 3D point corresponds to n feature points, and each feature point descriptor is composed of a multi-dimensional vector (SIFT feature points are 128-dimensional vectors). p: .

[0116] Feature point descriptor matrix d:

[0117] ;

[0118] The mean d_mean of all feature points in the two-dimensional image is:

[0119] ;

[0120] Then, the dot product of the descriptors of the multiple two-dimensional image feature points and the mean descriptor is determined. :

[0121] ;

[0122] Finally, the descriptor with the smallest dot product is taken as the unique descriptor of the three-dimensional point.

[0123] By following the steps above, a robust and representative 3D point descriptor can be obtained, thereby eliminating the effects of perspective differences and lighting changes, resolving the impact of fisheye camera distortion, and improving matching accuracy.

[0124] Step S400: Search for 3D points in nearby ground image data based on the 3D points in the aerial image data;

[0125] After initial alignment of ground and aerial image data, image feature matching is performed using neighborhood matching of 3D points. Specifically, for example... Figure 7 As shown, it includes:

[0126] Step S410: Construct a binary tree structure based on the three-dimensional points of the aerial image data and the three-dimensional points of the ground image data;

[0127] Collect and organize the corresponding 3D points from ground image data and aerial image data, ensuring that the data format is consistent, and preprocess the 3D points to remove noise and perform smoothing.

[0128] Using 3D points as data input, a KD-tree is constructed. The KD-tree is a binary tree structure used for efficient storage and retrieval of points in multidimensional space. Optionally, other structures can also be used for the binary tree structure.

[0129] Step S420: Query the binary tree structure and compare it with its neighboring points to determine the similarity of the three-dimensional points;

[0130] The three-dimensional points in the aerial image data are used as query points. The query is performed in the binary tree. The nearest neighbor search is performed using a KD-tree to find the surrounding neighboring points. The descriptors of the query point and its neighboring points are calculated and compared. By comparing the similarity between the descriptors, the neighboring points that match the query point are determined.

[0131] Step S430: Determine the neighborhood points in the ground image data that match the three-dimensional points in the aerial image data based on the similarity of the three-dimensional points.

[0132] By comparing the similarity between descriptors, neighboring points that match the query point are determined. Preferably, the matching results can be post-processed, such as removing false matches, optimizing using algorithms like RANSAC (Random Sample Consensus), and visually verifying the matching results to ensure accuracy and reliability. Multiple iterative optimizations can be performed as needed to improve matching accuracy.

[0133] The above embodiments enable neighborhood matching of 3D points based on binary trees, thereby replacing traditional 2D image matching methods and improving matching accuracy.

[0134] Step S500: If the distance between the descriptor of the three-dimensional point of the aerial image data and the descriptor of the three-dimensional point of the searched ground image data is less than a preset threshold, then the three-dimensional point of the aerial image data and the three-dimensional point of the ground image data are determined as matching three-dimensional points;

[0135] like Figure 8 As shown, the neighborhood matching process is illustrated. The three-dimensional points within the circles in the figure are the three-dimensional points within the search-determined neighborhood. The three-dimensional point of the ground image data that is closest to the three-dimensional point of the aerial image data is determined as the matching three-dimensional point.

[0136] Step S600: Perform data fusion on the aerial image data and the ground image data based on the matched three-dimensional points.

[0137] After determining the matching three-dimensional points between aerial image data and ground image data, data fusion between aerial image data and ground image data can be performed based on the matching three-dimensional points.

[0138] Furthermore, in this embodiment of the application, in order to further improve the effect of air-to-ground data fusion and realize cross-platform data fusion, a cross-platform global BA constraint is further introduced to introduce joint air-to-ground BA optimization, so that the final fusion result is better than the traditional SFM fusion.

[0139] Global bundle adjustment (BA) is a commonly used optimization method in 3D reconstruction and visual localization to improve the accuracy of reconstruction and localization. In this embodiment, firstly, a global bundle adjustment map is constructed based on the camera pose and 3D points corresponding to the aerial image data, the camera pose and 3D points corresponding to the ground image data, the matching point pairs in the distance matrix of the ground control point set and the distance matrix of the aerial control point set, and the matching 3D points. Then, minimizing the reprojection error, minimizing the 3D point distance constraint, and minimizing the transformation prior are introduced. Minimizing the reprojection error ensures that the projection of 3D points in different images is as consistent as possible with the actually observed feature points. Minimizing the 3D point distance constraint ensures that the relative distance between 3D points remains constant during the optimization process. Minimizing the transformation prior maintains the smoothness of the camera pose and 3D point coordinates.

[0140] After introducing the above constraints, the global bundle adjustment map is iteratively optimized according to minimizing the reprojection error, minimizing the three-dimensional point distance constraint, and minimizing the transformation prior, thereby realizing the data fusion of the aerial image data and the ground image data.

[0141] Preferably, during iterative optimization, nonlinear optimization algorithms such as the Levenberg-Marquardt (LM) Gauss-Newton method can be used. In each iteration, the camera pose and 3D point coordinates are adjusted to minimize the objective function value, thereby obtaining a high-precision unified coordinate system, unified point cloud structure, and image pose after air-to-ground fusion.

[0142] The above embodiments, through air-ground joint BA optimization, combine ground and aerial image data, which can effectively improve the accuracy and consistency of reconstruction results, and are especially suitable for 3D reconstruction in complex scenes.

[0143] In summary, the embodiments provided in this application find cross-view matching points based on 3D neighborhoods and descriptors, merging a large number of 2D features into robust 3D feature representations. Neighborhood matching is performed in the aligned 3D space, replacing direct image matching, thus bypassing the difficulties of 2D matching of air-ground images and solving the problem of cross-view image mismatch, significantly improving the robustness of air-ground image matching. Furthermore, by setting image control points within the area to be scanned, visual triangulation is achieved through multi-angle images of the image control points. Using the visual 3D reconstruction results of the image control points, a preliminary matching relationship is established through a distance matrix, enabling automatic coarse alignment of air-ground data without initial extrinsic parameters. This achieves air-ground coordinate transformation and alignment without relying on RTK, GPS, or LiDAR coordinate systems. Additionally, by constructing a global BA optimization graph jointly for air and ground, and introducing cross-view constraints generated by 3D matching, global BA constraints are realized, solving the problem of difficulty in constructing globally optimizable constraints for air-ground images due to scale changes, illumination differences, and severe tilt.

[0144] In other embodiments, such as Figure 9 As shown in the embodiments of this application, a three-dimensional reconstruction device is also proposed. The three-dimensional reconstruction device is used to run the air-ground data fusion method proposed in the above embodiments. The three-dimensional reconstruction device may include: a processor 402, a memory 406, a communication interface 404, and a communication bus 408.

[0145] The processor 402, memory 406, and communication interface 404 communicate with each other via communication bus 408. The memory 406 stores at least one program 410, which causes the processor 402 to execute steps related to the air-to-ground data fusion method proposed in this application embodiment.

[0146] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0147] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the 3D reconstruction device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0148] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0149] Specifically, program 410 can be called by processor 402 to cause the three-dimensional reconstruction device to execute an embodiment of the air-ground data fusion method proposed in this application, which will not be described in detail here.

[0150] This application also provides a computer-readable storage medium storing executable instructions. When the executable instructions are run on a 3D reconstruction device, the 3D reconstruction device performs the air-ground data fusion method provided in any of the above embodiments.

[0151] This application also provides an air-to-ground data fusion program, which is used to execute the air-to-ground data fusion method provided in the above embodiments.

[0152] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0153] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0154] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments of this application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0155] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying abstract and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying abstract and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0156] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can design alternative embodiments without departing from the scope. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

Claims

1. A method for air-to-ground data fusion, characterized in that, The method includes: Acquire aerial and ground image data of the area to be scanned; Based on the aerial image data and the ground image data, corresponding two-dimensional image feature points and three-dimensional points corresponding to the two-dimensional image feature points are generated respectively, wherein each three-dimensional point corresponds to multiple two-dimensional image feature points; Calculate the mean descriptor based on the descriptors of the multiple two-dimensional image feature points corresponding to the three-dimensional point; Determine the dot product of the descriptors of multiple two-dimensional image feature points with the mean descriptor, and take the descriptor with the smallest dot product as the descriptor of the three-dimensional point; Search for 3D points in the ground image data within the neighborhood of 3D points in the aerial image data; If the distance between the descriptor of the three-dimensional point in the aerial image data and the descriptor of the three-dimensional point in the searched ground image data is less than a preset threshold, then the three-dimensional point in the aerial image data and the three-dimensional point in the ground image data are determined as matching three-dimensional points. Data fusion is performed on the aerial image data and the ground image data based on the matched 3D points.

2. The method according to claim 1, characterized in that, The step of generating corresponding two-dimensional image feature points and corresponding three-dimensional points based on the aerial image data and the ground image data includes: Two-dimensional image feature points are extracted from the aerial image data and the ground image data respectively using a preset scale-invariant feature transformation algorithm; Perform feature matching on each of the two-dimensional image feature points to determine the corresponding two-dimensional image matching feature points for the same scene in different aerial image data and ground image data; Based on the feature points matched in the two-dimensional image, corresponding three-dimensional points are generated using a preset motion recovery structure algorithm.

3. The method according to claim 1 or 2, characterized in that, Multiple image control points are pre-set within the area to be scanned; The method further includes: Acquire multiple aerial image data and multiple ground image data of multiple control points; Based on multiple aerial image data and multiple ground image data of the multiple image control points, respectively construct the distance matrix of the ground image control point set and the distance matrix of the aerial image control point set; The air-to-ground coordinate transformation matrix is ​​determined based on the ground control point set distance matrix and the air control point set distance matrix. The aerial image data and the ground image data are aligned according to the air-to-ground coordinate transformation matrix.

4. The method according to claim 3, characterized in that, The step of determining the air-to-ground coordinate transformation matrix based on the distance matrix of the ground control point set and the distance matrix of the air control point set includes: Determine the matrix similarity between the distance matrix of the ground control point set and the distance matrix of the air control point set; Based on the matrix similarity, matching point pairs in the distance matrix of the ground control point set and the distance matrix of the air control point set are determined; The rotation matrix and translation vector are calculated based on the matching point pairs using a preset Kabsch algorithm. The air-ground coordinate transformation matrix is ​​determined based on the rotation matrix and translation vector.

5. The method according to claim 4, characterized in that, The step of aligning the aerial image data and the ground image data according to the air-to-ground coordinate transformation matrix further includes: Based on the aligned aerial image data and ground image data, feature matching is performed on adjacent aerial image data and ground image data. The corresponding three-dimensional points are determined based on the two-dimensional image feature points of the aerial image data and the ground image data after feature matching; The air-to-ground coordinate transformation matrix is ​​optimized based on the three-dimensional points using an iterative nearest-point algorithm; The aerial image data and the ground image data are aligned according to the optimized air-to-ground coordinate transformation matrix.

6. The method according to claim 1, characterized in that, The step of searching for 3D points in the ground image data within the neighborhood of 3D points in the aerial image data includes: Construct a binary tree structure based on the three-dimensional points of the aerial image data and the three-dimensional points of the ground image data; The binary tree structure is queried and compared with its neighboring points to determine the similarity of the three-dimensional points; Based on the similarity of the three-dimensional points, neighborhood points in the ground image data that match the three-dimensional points in the aerial image data are determined.

7. The method according to claim 4, characterized in that, The data fusion of the aerial image data and the ground image data based on the matched 3D points includes: A global bundle adjustment map is constructed based on the camera pose and 3D points corresponding to the aerial image data, the camera pose and 3D points corresponding to the ground image data, the distance matrix of the ground control point set, the matching point pairs in the distance matrix of the aerial control point set, and the matching 3D points. The global bundle adjustment map is iteratively optimized based on the preset minimization of reprojection error, minimization of 3D point distance constraints, and minimization of transformation priors, thereby achieving data fusion of the aerial image data and the ground image data.

8. A three-dimensional reconstruction device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one program that causes the processor to execute the air-to-ground data fusion method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which, when run on a 3D reconstruction device, causes the 3D reconstruction device to perform the air-to-ground data fusion method as described in any one of claims 1-7.

Images