Three-dimensional reconstruction method and terminal equipment

Abstract

The invention relates to the technical field of three-dimensional reconstruction, and discloses a three-dimensional reconstruction method and terminal equipment, and the method comprises the steps: carrying out the unmanned plane route planning and ground end scanning planning of a scanning region; acquiring an aerial image and a ground image; performing air model reconstruction according to the air image to obtain an air sparse point cloud and first pose information, and performing ground model reconstruction according to the ground image to obtain a ground sparse point cloud and second pose information; matching a target aerial image and a target ground image with an overlapping region in the aerial image and the ground image to obtain relative pose information between the target aerial image and the target ground image; and according to the first pose information, the second pose information and the relative pose information, utilizing global binding with control point constraint to adjust and optimize pose information and sparse point positions of each image in the target aerial image and the target ground image with the overlapping region, and obtaining a fusion model. According to the invention, the space-ground coordinate alignment precision of three-dimensional reconstruction is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of three-dimensional reconstruction technology, and specifically to a three-dimensional reconstruction method and terminal device. Background Art

[0002] 3D reconstruction technology involves extracting geometric information from single or multiple 2D images or point cloud data to restore the 3D model and spatial structure of an object or scene. This technology is a core research area in computer vision, digital photogrammetry, and graphics, and has been widely applied in a variety of fields, including robotic navigation, autonomous driving, virtual reality, cultural relic digitization, industrial inspection, and medical imaging.

[0003] The three-dimensional reconstruction technology used in surveying, mapping, planning and other fields is mainly divided into aerial photogrammetry, ground photogrammetry and the fusion of aerial photogrammetry and ground photogrammetry.

[0004] Aerial photogrammetry typically uses a drone (single or multiple cameras) equipped with a gimbal and a real-time kinematic (RTK) device to ensure stability and capture large-scale bird's-eye views. The advantages of aerial photogrammetry include wide coverage, high image acquisition efficiency, and relatively stable reconstruction. However, the long shooting distance can result in limited accuracy in details such as building facades. RTK avoids the large errors associated with traditional GPS positioning, providing higher absolute accuracy for aerial models.

[0005] Terrestrial photogrammetry typically uses laser equipment equipped with panoramic cameras, handheld cameras, or vehicle-mounted cameras to capture street scenes or close-up images. While capable of capturing highly accurate facade details, this approach is limited in coverage and can be cumbersome. Lasers provide highly accurate 3D geometric information (point clouds), while cameras provide realistic texture and color information, complementing each other.

[0006] The fusion of aerial and terrestrial photogrammetry is difficult because the ground and aerial data use different coordinate systems. This leads to insufficient alignment accuracy and misalignment of the model. Furthermore, when fusion of ground and aerial data is performed directly on the entire image, geometric distortion and misalignment are likely to occur in the fused area.

[0007] Therefore, a 3D reconstruction method that can improve the coordinate alignment accuracy of the fusion of aerial photogrammetry and terrestrial photogrammetry and realize the precise fusion of aerial and terrestrial images is needed to obtain a high-precision and large-scale 3D model at the same time. Summary of the Invention

[0008] In view of the above problems, the embodiments of the present application provide a three-dimensional reconstruction method and terminal device to solve the problems of insufficient coordinate system alignment accuracy and easy geometric distortion and misalignment in the fusion area in the prior art.

[0009] According to one aspect of an embodiment of the present application, a three-dimensional reconstruction method is provided, the method comprising: Performing drone route planning and ground-end scanning planning on the scanning area to obtain a drone shooting route and a ground-end scanning path, wherein the drone shooting route includes the drone take-off and landing points and the drone shooting path, and the ground-end scanning path includes the scanning path of the ground scanning device, and the scanning path of the ground scanning device passes through the drone take-off and landing points; Acquire a plurality of aerial images from different perspectives obtained by continuous aerial photography by the drone along the drone photography route, and a plurality of ground images obtained by ground photography by the ground scanning device along the ground-end scanning path, wherein the coordinate system of the ground images is the same as the coordinate system of the aerial images; Reconstructing an aerial model based on the plurality of aerial images to obtain an aerial sparse point cloud and first pose information of each aerial image, and reconstructing a ground model based on the plurality of ground images to obtain a ground sparse point cloud and second pose information of each ground image; Based on the aerial sparse point cloud and the ground sparse point cloud, matching target aerial images and target ground images in overlapping areas among the plurality of aerial images and the plurality of ground images to obtain relative pose information between the target aerial image and the target ground image; According to the first pose information, the second pose information and the relative pose information, global bundling adjustment with control point constraints is used to optimize the pose information and sparse point positions of each image in the target aerial image and the target ground image in the overlapping area to obtain a fusion model.

[0010] Optionally, reconstructing the aerial model based on the plurality of aerial images to obtain an aerial sparse point cloud and first position information of each aerial image includes: Get the position and posture POS data of each aerial image; Determining feature points and feature descriptors of the feature points in each aerial image, wherein the feature points include corner points, edge intersection points, and centers of high-contrast areas; Determining a plurality of aerial image pairs from the plurality of aerial images according to the POS data, wherein each pair of the aerial image pairs includes two aerial images, and the two aerial images have an overlapping area; For each pair of aerial images, determining feature points with the same name in the two aerial images, and obtaining multiple pairs of matching feature points consisting of feature points with the same name; The relative pose information between the two aerial images to which the matching feature points belong is determined based on the matching feature points, and the initial first pose information and the initial aerial sparse point cloud of each of the aerial images are determined based on the relative pose information between the two aerial images. The initial first pose information and the initial aerial sparse point cloud are optimized through global bundling adjustment to obtain the first pose information and the aerial sparse point cloud.

[0011] Optionally, the ground scanning device is provided with a laser radar and a camera, and the ground image is an image taken by the camera; Reconstructing a ground model based on the plurality of ground images to obtain a ground sparse point cloud and second pose information of each ground image includes: Determining a laser trajectory based on the point cloud data collected by the laser radar, wherein the laser trajectory includes a pose sequence of the laser radar; Transferring the laser trajectory to an image using pre-marked extrinsic parameters between the laser radar and the camera to obtain an image trajectory, wherein the image trajectory includes an initial estimated pose of each of the ground images; Based on the image trajectory, the pose information of the ground image is globally bundled, adjusted and optimized through feature point extraction and matching to obtain the second pose information and the ground sparse point cloud.

[0012] Optionally, based on the aerial sparse point cloud and the ground sparse point cloud, matching target aerial images and target ground images having overlapping areas in the plurality of aerial images and the plurality of ground images to obtain relative pose information between the target aerial image and the target ground image includes: Determining target aerial images and target ground images having overlapping areas based on the first pose information of the plurality of target aerial images and the second pose information of the plurality of target ground images; Based on the aerial sparse point cloud and the ground sparse point cloud, determining sparse points with the same name in the target aerial image and the target ground image located in the overlapping area, and obtaining a plurality of sparse point pairs consisting of the sparse points with the same name, wherein each sparse point pair includes an aerial sparse point and a ground sparse point; For each sparse point pair, determining a target aerial image corresponding to the aerial sparse point in the sparse point pair from the plurality of aerial images to obtain a plurality of target aerial images, and determining a target ground image corresponding to the ground sparse point in the sparse point pair from the plurality of ground images to obtain a plurality of target ground images; Combining the plurality of target aerial images and the plurality of target ground images in pairs to obtain a plurality of air-ground image pairs, wherein each pair of the air-ground image pairs includes one target aerial image and one target ground image; For each pair of the air-ground image pairs, the target aerial image is scaled at multiple scales to obtain multiple aerial scaled images of different scales, and the target ground image is scaled at multiple scales to obtain multiple ground scaled images of different scales; Determining an aerial zoomed image and a ground zoomed image having the same scale from the plurality of aerial zoomed images and the plurality of ground zoomed images; Feature point extraction and matching based on perspective weight are performed on the aerial zoomed image and the ground zoomed image of the same scale to obtain relative pose information between the target aerial image and the target ground image, wherein the perspective weight of the feature points extracted from the aerial zoomed image with a lower shooting altitude is greater than the perspective weight of the feature points extracted from the aerial zoomed image with a higher shooting altitude.

[0013] Optionally, determining the sparse points with the same name in the target aerial image and the target ground image located in the overlapping area based on the aerial sparse point cloud and the ground sparse point cloud includes: Based on the aerial sparse point cloud, obtaining three-dimensional coordinates of aerial sparse points located in the target aerial image in the overlapping area, and based on the ground sparse point cloud, obtaining three-dimensional coordinates of ground sparse points located in the target ground image in the overlapping area; The aerial sparse points and the ground sparse points with similar three-dimensional coordinates are determined as the sparse points with the same name.

[0014] Optionally, the optimizing the pose information and sparse point positions of each of the target aerial image and the target ground image having the overlapping area using global bundling adjustment with control point constraints based on the first pose information, the second pose information, and the relative pose information to obtain a fusion model includes: Determine control points from all sparse points included in the aerial sparse point cloud and the ground sparse point cloud; According to the first pose information, the second pose information and the relative pose information, with the position of the control point as a constraint, global bundling adjustment is used to optimize the pose information and sparse point positions of each image in the target aerial image and the target ground image with overlapping areas to obtain a fusion model, wherein the position of the control point in the fusion model remains unchanged from the position of the control point before optimization.

[0015] Optionally, determining control points from all sparse points included in the aerial sparse point cloud and the ground sparse point cloud includes: Obtaining the number of images corresponding to all sparse points included in the aerial sparse point cloud and the ground sparse point cloud; The sparse points, the number of which corresponding to the control points is greater than a first preset value and the number of which corresponding to the control points does not include the overlapping area is less than or equal to a second preset value.

[0016] Optionally, the fusion model includes multiple optimized sparse points and their positions, and optimized pose information of each image.

[0017] Optionally, performing drone route planning and ground-side scanning planning on the scanning area to obtain the drone shooting route and the ground-side scanning path includes: Dividing the scanning area into a plurality of grids; Determine the vertex of each grid as the take-off and landing point of the UAV; Determining a starting shooting altitude and an ending shooting altitude of the drone to obtain the drone shooting path, wherein the drone shooting path is a path along a vertical direction, wherein the starting point is a position vertically distanced from the drone take-off and landing point by the starting shooting altitude, and the ending point is a position vertically distanced from the drone take-off and landing point by the ending shooting altitude, and the difference between the starting shooting altitude and the shooting altitude of the ground scanning device is within a preset range, so that the plurality of aerial images and the plurality of ground images have overlapping areas of the target aerial image and the target ground image; The scanning path of the ground scanning device is determined based on the take-off and landing point of the UAV, so that the scanning path of the ground scanning device passes through the take-off and landing point of the UAV.

[0018] According to another aspect of an embodiment of the present application, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the three-dimensional reconstruction method as described above.

[0019] The embodiment of the present application performs drone route planning and ground-end scanning planning on the scanning area, so that the scanning path of the ground scanning device passes through the drone take-off and landing point, so that there is an overlapping area between the take-off and landing part of the aerial image taken by the drone and the ground image taken by the ground scanning device. Feature points of the overlapping area can be extracted and matched between the air-ground images. The images of the overlapping area are added to the global bundling adjustment optimization process, which improves the alignment accuracy of the coordinate system during air-ground fusion and improves the consistency of the air-ground posture. Compared with the simple alignment of the two scenes of the aerial image and the ground image, it has higher accuracy. In addition, the control point constraints are used during the global bundling adjustment optimization to improve the reconstruction stability. Images without overlapping areas on the ground will not be optimized incorrectly. Similarly, images without overlapping areas in the air will not be incorrectly optimized, which reduces the geometric distortion and misalignment problems in the fusion area and further improves the accuracy.

[0020] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings: Figure 1 A schematic diagram of an application scenario of an embodiment of the present application is shown; Figure 2 A flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application is shown; Figure 3 shows a grid diagram of the divided scanning area; Figure 4 A schematic diagram of the drone's shooting path is shown; Figure 5 A schematic structural diagram of a terminal device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0023] The present application provides a three-dimensional reconstruction method for performing three-dimensional reconstruction based on space-ground fusion. Figure 1A schematic diagram of an application scenario of an embodiment of the present application is shown. As shown in the figure, in a three-dimensional scene, a plurality of aerial images of different perspectives are obtained by continuous aerial shooting in the air by a drone 1 according to the drone shooting route, and a plurality of ground images are obtained by ground shooting by a ground scanning device 2 according to the ground end scanning path. A laser radar and a camera can be set on the ground scanning device 2, wherein the laser radar collects laser point cloud data of the three-dimensional scene, and the camera obtains the ground image of the three-dimensional scene. The ground scanning device 2 can be a handheld device or a non-handheld device. If the ground scanning device 2 is a handheld device, the user holds the ground scanning device 2 and moves it in the three-dimensional scene to collect the laser point cloud data and the ground image of the three-dimensional scene. If the ground scanning device 2 is a non-handheld device, the ground scanning device 2 can be used to move autonomously in the three-dimensional scene to collect the laser point cloud data and the ground image of the three-dimensional scene.

[0024] Terminal device 3 is connected to drone 1 and ground scanning device 2, respectively, to receive aerial images captured by drone 1 and laser point cloud data and ground images scanned by ground scanning device 2. It then performs 3D reconstruction based on this data to produce a fused air-ground 3D reconstruction result (fusion model). Terminal device 3 can directly display this 3D reconstruction result or send it to another display device for display. Terminal device 3 can be a mobile phone, tablet computer, laptop computer, desktop computer, server, or other device. Terminal device 3 and ground scanning device 2 can be connected via wired or wireless communication.

[0025] Drone 1 can be equipped with an oblique camera, such as a five-, three-, or two-lens camera, equipped with RTK and capable of recording position and orientation system (POS) information. Ground scanning device 2 can also be equipped with RTK to record image pose information. Drone oblique photography refers to an aerial photography technique that uses a drone equipped with an oblique camera to capture images of surface objects from vertical and multiple oblique angles (typically, front, back, left, and right).

[0026] In other scenarios, the lidar and camera may be separate devices and not integrated into the ground scanning device 2. They send the data collected by each of them to the terminal device 3, which performs data fusion processing to obtain the three-dimensional reconstruction result.

[0027] The three-dimensional scene in the embodiment of the present application mainly refers to a large-scale outdoor scene.

[0028] Figure 2 The flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application is shown, and the method is executed by a terminal device. The terminal device can be Figure 1Terminal device 3 in Figure 2 As shown, the method includes the following steps: S110 , performing drone route planning and ground-side scanning planning on the scanning area to obtain a drone shooting route and a ground-side scanning path.

[0029] Among them, the drone shooting route includes the drone take-off and landing point and the drone shooting path, and the ground scanning path includes the scanning path of the ground scanning equipment, and the scanning path of the ground scanning equipment passes through the drone take-off and landing point.

[0030] S110 may specifically include the following steps: S111, dividing the scanning area into multiple grids; Figure 3 The grid diagram of the divided scanning area is shown in FIG. Figure 3 As shown, each grid can be a square grid with the same side length, for example, 30 meters. Setting a larger grid side length can cover a certain number of sparse points in the captured image, which can improve the matching accuracy in the subsequent feature point matching process, thereby improving the alignment accuracy of the air-ground data.

[0031] S112, determining the vertex of each grid as the take-off and landing point of the UAV; like Figure 3 As shown, each grid vertex shown by the black circle in the figure is determined as the drone take-off and landing point P. There are 36 drone take-off and landing points P in the figure.

[0032] S113: Determine the starting and ending shooting altitudes of the drone to obtain a drone shooting path R, where the drone shooting path R is a vertical path with a starting point H1 being the vertical distance from the drone's take-off and landing point P to the starting shooting altitude, and an ending point H2 being the vertical distance from the drone's take-off and landing point P to the ending shooting altitude. The difference between the starting shooting altitude and the shooting altitude of the ground scanning device is within a preset range, so that the multiple aerial images captured by the drone and the multiple ground images captured by the ground scanning device have overlapping areas. The preset range can be set based on the actual application scenario, for example, to [-10cm, 10cm] or another range. For example, the starting shooting altitude can be set to 1.5m. Based on the height of a person holding a ground scanning device of 1.7m, the shooting altitude of the ground scanning device is approximately 1.5m. Setting the starting shooting altitude of the drone to 1.5m ensures that the field of view of the aerial images initially captured by the drone and the ground images captured by the ground scanning device are the same, with overlapping areas between the two, which can be used for subsequent fusion. If the drone starts shooting immediately after takeoff, or if the starting shooting altitude is set too low, the initial images captured by the drone will be mostly ground images rather than building images, which will be of little use in 3D reconstruction and result in wasted resources. The "overlapping area" mentioned above refers to areas in the same physical space that appear in both an aerial image and a ground image. Such areas are considered overlapping areas.

[0033] The ending shooting height can be determined according to the usual oblique photography height, for example, 80-120 meters in urban areas, and can be increased to 150-200 meters in rural areas or open areas.

[0034] Figure 4 A schematic diagram of the drone shooting path is shown. Figure 4 As shown, the arrow in the figure indicates the drone's flight direction, which is vertically ascending. For example, at each drone takeoff and landing point P, the drone takes off from point P, begins filming upon reaching starting point H1, then continues its vertical ascent and continues filming, ceasing filming upon reaching end point H2, completing the filming process for that particular takeoff and landing point P. The thick line R indicates the drone's filming path.

[0035] S114: Determine a scanning path of the ground scanning device based on the take-off and landing point of the UAV, so that the scanning path of the ground scanning device passes through the take-off and landing point of the UAV.

[0036] The scanning path of the ground scanning device passes through the take-off and landing point of the UAV, which means that the points passed by the scanning path of the ground scanning device include points whose coordinates are the same as the coordinates of the take-off and landing point of the UAV, or points whose difference does not exceed a preset threshold (the difference is not large), for example, the difference is only within tens of centimeters.

[0037] During the above process, traditional oblique photography route planning methods can be used to automatically design the scanned area, ensuring an overlap ratio greater than 80%. The overlap ratio for oblique photography routes refers to the proportion of the same area between adjacent photos. This ensures complete and comprehensive coverage of the captured area and provides sufficient perspective information for subsequent 3D model reconstruction. Overlap ratio is typically measured in two ways: heading overlap and lateral overlap. An overlap ratio greater than 80% refers to heading overlap.

[0038] If there is no overlapping area between the aerial image and the ground image, it will be difficult to match the aerial image from a pure aerial perspective and the ground image from a ground perspective due to the difference in perspective. This application performs the above-mentioned drone route planning and ground-end scanning planning on the scanning area, so that there is an overlapping area between the take-off and landing part images in the aerial image taken by the drone and the ground image taken by the ground scanning equipment. This is equivalent to creating a common area in space. This area is covered by both the ground laser point cloud / image and the drone image, and can establish a connection between the two images, perform feature point extraction and matching between the air-ground images, and provide a reliable control basis for high-precision matching of the aerial model and the ground model, thereby improving the alignment accuracy of the fusion of aerial and ground images.

[0039] S120, acquiring a plurality of aerial images from different perspectives obtained by continuous aerial photography by the drone according to the drone photography route, and a plurality of ground images obtained by ground photography by the ground scanning device according to the ground end scanning path.

[0040] After planning a drone flight path with the S110, you can manually control the drone to take off at each takeoff and landing point, and then control the drone to shoot along the planned flight path. The images captured by the drone include multiple images from different perspectives. Different takeoff and landing points and shooting altitudes result in different perspectives.

[0041] On the ground side, an operator can scan along the ground side scanning path with a handheld ground scanning device, or send the ground side scanning path to a ground scanning device with autonomous movement function, which will scan along the ground side scanning path.

[0042] After the shooting work of the drone and the ground scanning equipment is completed, the terminal device obtains multiple aerial images from different perspectives obtained by the drone taking continuous aerial photos along the drone's shooting route, and obtains multiple ground images from the ground scanning equipment obtained by the ground scanning equipment taking ground photos along the ground end scanning path.

[0043] The coordinate system of the ground imagery is the same as that of the aerial imagery. For example, if the drone captures data using the WGS84 coordinate system, the ground scanning device should also use the WGS84 coordinate system; if the drone captures data using the CGCS2000 coordinate system, the ground scanning device should also use the CGCS2000 coordinate system. In short, the coordinate system of the ground imagery should be consistent with that of the aerial imagery. By unifying the coordinate system and the spatial reference from the source of data collection, we avoid the complex and often inaccurate coordinate system conversion work later on and ensure the consistency of the spatial reference between the aerial and ground models.

[0044] S130, reconstructing an aerial model based on the multiple aerial images to obtain an aerial sparse point cloud and first pose information of each aerial image, and reconstructing a ground model based on the multiple ground images to obtain a ground sparse point cloud and second pose information of each ground image.

[0045] This step performs aerial oblique photography reconstruction and ground image reconstruction respectively.

[0046] Among them, the aerial model is reconstructed based on multiple aerial images to obtain the aerial sparse point cloud and the first pose information of each aerial image, including: Step a1, obtaining the position and posture POS data of each aerial image; The RTK system onboard the drone provides initial positioning information, which can be used to determine the initial position (3D coordinates) and attitude, also known as POS data. POS data provides the initial values ​​for subsequent feature matching, significantly reducing computational effort, avoiding matching errors, and improving the absolute accuracy of the entire model.

[0047] Step a2, determining feature points and feature descriptors of the feature points in each aerial image, wherein the feature points include corner points, edge intersection points, centers of high contrast areas, etc.; This step involves feature extraction. Unique, stable, and reproducible feature points can be extracted from each image using algorithms such as the scale-invariant feature transform (SIFT), SURF, or a faster feature extraction algorithm that combines FAST keypoint detection with the Oriented FAST and Rotated BRIEF (ORB) descriptor. The entire image is scanned for features such as corners, edge intersections, and the centers of high-contrast areas. For each feature point found, a feature descriptor is calculated, a digital vector that encodes unique information about the pixel region surrounding that point. This process yields a set of feature points and their corresponding descriptors for each image.

[0048] Step a3, determining a plurality of aerial image pairs from the plurality of aerial images according to the POS data; Each pair of aerial images includes two aerial images, and there is an overlapping area between the two aerial images; the "overlapping area" here refers to an area in the same physical space that appears in two or more aerial images at the same time, and such an area is the overlapping area here.

[0049] This step identifies pairs of aerial images with overlapping regions based on the image pose information (POS data). These overlapping images typically contain feature points that belong to the same physical point in the 3D scene.

[0050] Step a4: for each pair of aerial images, determining feature points with the same name in the two aerial images, and obtaining multiple pairs of matching feature points consisting of feature points with the same name; Feature points with the same name refer to pixels that appear in different images and represent the same physical point in the real world.

[0051] Steps a3 and a4 are the feature matching process, aimed at identifying the same physical point captured in different images. Due to differences in shooting angle, lighting, scale, and occlusion, the same point may appear very different in two images. By leveraging POS data (initial position and pose) to intelligently select image pairs most likely to have overlapping areas for comparison, the computational effort is reduced. Then, for aerial image pairs with overlapping areas (for example, Image A and Image B), the descriptor of a feature point in Image A is extracted. The most similar descriptor is then found among all descriptors in Image B (this can be achieved by calculating the Euclidean distance or Hamming distance, typically the nearest neighbor). The most similar point in Image B is then identified as a feature point with the same name as the feature point in Image A. Initial matching may contain errors (false matches). Robustness estimation algorithms, such as the Random Sample Consensus (RANSAC) method, can be used to randomly select a small number of matching point pairs and calculate a hypothetical model of the fundamental matrix or homography. This model is then used to test all other matching point pairs to see how many points fit the model (i.e., "inliers"). This process is repeated multiple times, ultimately adopting the model with the most "inliers" and eliminating all "outliers" (false matches) that do not fit the model. The result is a purified, accurate set of matching feature points.

[0052] Step a5: Determine the relative pose information between the two aerial images to which the matching feature points belong based on the matching feature points, determine the initial first pose information and the initial aerial sparse point cloud of each aerial image based on the relative pose information between the two aerial images, and optimize the initial first pose information and the initial aerial sparse point cloud through global bundling adjustment to obtain the first pose information and the aerial sparse point cloud.

[0053] This step involves incremental reconstruction (SfM), determining the initial rotation and translation, and performing global bundle adjustment. Global bundle adjustment (BA), a core technology in computer vision, photogrammetry, and robotics (particularly in the fields of SLAM and SfM), is a nonlinear optimization process that simultaneously optimizes all camera parameters (pose - position and orientation) and the coordinates of all 3D points in the scene. BA optimizes by minimizing the difference (called reprojection error) between the observed data (typically 2D feature points in the image) and the projected points predicted based on the current 3D structure and camera parameters. Ultimately, the precise camera position and orientation (pose information) and the precise coordinates of the 3D points are determined for each image captured.

[0054] This step gradually recovers the precise pose information and sparse point cloud (sparse 3D point cloud) for each image from the 2D image matching points. First, two images with the highest overlap, a large number of evenly distributed matching feature points, and a high number of matching feature points are selected as the starting point for the calculation. Their POS data is used as the initial values. Based on the matching feature points in this pair of images, the initial relative rotation and translation between them is accurately calculated. For example, the essential matrix can be solved using algorithms such as the five-point method. Knowing the precise relative position of the two images, the 3D coordinates (X, Y, Z) of each pair of matching feature points can be calculated using triangulation principles, thereby obtaining an initial batch of sparse points. Next, a third image is searched for 2D matching points (i.e., the feature points obtained in step a2) that have a sufficient number (e.g., 20 or more) of known 3D points (i.e., the initial batch of sparse points) with the current model. Using algorithms such as PnP (Perspective-n-Point) and similar algorithms, the precise camera position and pose of the third image are calculated using 3D-2D correspondences. The same process as for the third image is used to match additional new images to the existing model. New 3D points are triangulated and added to the sparse point cloud. This process is repeated, registering new images to the growing model and expanding the sparse 3D point cloud until all eligible images have been added. Finally, a global bundle adjustment is performed to simultaneously optimize the pose information of all images and the coordinates of all sparse points to minimize the reprojection error, resulting in the optimized first pose information and the aerial sparse point cloud.

[0055] The above is the reconstruction process of aerial oblique photography. The final aerial model includes the first pose information and the aerial sparse point cloud. The reconstruction process utilizes the matching information of feature points between each aerial image.

[0056] Regarding the reconstruction of the ground model, the high-precision, low-drift trajectory generated by laser SLAM can be used as a powerful constraint for visual SLAM or visual SfM (Structure from Motion), thereby generating a model with both laser-level geometric accuracy and rich visual texture and feature information. Specifically, the ground image is an image captured by the camera of the ground scanning device. The ground model is reconstructed based on multiple ground images to obtain a sparse ground point cloud and the second pose information of each ground image, including: Step b1, determining a laser trajectory based on the point cloud data collected by the laser radar, wherein the laser trajectory includes a position sequence of the laser radar; This step processes the raw point cloud sequence captured by the lidar, extracting feature points from each frame. By calculating the matching relationships between feature points between consecutive frames (using algorithms such as ICP and NDT), the relative motion (pose transformation) of the lidar itself is estimated. The point clouds of these frames are then registered with a global map, and loop closure detection and graph optimization may be performed to correct for accumulated errors. The result is a high-precision lidar pose sequence T_li (i=1,2,...,N), or laser trajectory. This trajectory is highly accurate and has minimal drift in the local coordinate system.

[0057] Step b2: Using the pre-marked extrinsic parameters between the lidar and the camera, the laser trajectory is transferred to the image to obtain an image trajectory, where the image trajectory includes the initial estimated pose of each ground image; This step uses the pre-calibrated external parameters between the lidar and the camera (which can be a 4x4 rigid body transformation matrix, representing the rotation and translation relationship between the lidar coordinate system and the camera coordinate system). For each laser pose T_li, its corresponding camera pose T_ci can be obtained through a simple coordinate transformation: T_ci = T_li * T_lc; Wherein, T_lc is the external parameter between the above-mentioned laser radar and camera.

[0058] Finally, an initial, high-precision pose estimate is computed for each ground-level image.

[0059] In step b3, based on the image trajectory, the pose information of the ground image is globally bundled and optimized by extracting and matching feature points to obtain second pose information and a ground sparse point cloud.

[0060] This step performs visual SfM optimization based on the image trajectory obtained in the previous step. First, feature points are extracted (e.g., SIFT, ORB, etc.) from all ground-level images and matching relationships are established. A global bundle adjustment optimization is then performed to optimize the precise pose (T_ci) (i.e., camera pose) of all images and the spatial positions of all 3D feature points (sparse points). The optimization process satisfies the reprojection error constraint: the optimized camera pose and 3D points should minimize the error between the reprojected 3D points and the initially extracted feature point positions. This step also utilizes a priori constraints on the laser trajectory: during the optimization process, the camera pose cannot deviate too far from the initial pose in the image trajectory. This constraint can be added as a penalty term to the optimization objective function, taking the form: Σ || T_ci_optimized - T_ci_initial ||², where T_ci_initial is the initial pose in the image trajectory and T_ci_optimized is the optimized camera pose. After global optimization, the optimal camera pose (secondary pose information) and a high-precision 3D point cloud (sparse ground-level point cloud) are obtained.

[0061] The above is the ground image reconstruction process. The final ground model includes the second pose information and the ground sparse point cloud. The reconstruction process utilizes the matching information of feature points in the ground image.

[0062] The posture information (first posture information, second posture information, relative posture information and other posture information, etc.) in the embodiments of the present application refers to external parameters, including the rotation matrix R and the translation vector T, which describe the position (determined by translation) and posture (determined by rotation) of the image in three-dimensional space.

[0063] S140 , based on the aerial sparse point cloud and the ground sparse point cloud, matching the target aerial image and the target ground image in overlapping areas among the multiple aerial images and the multiple ground images to obtain relative pose information between the target aerial image and the target ground image.

[0064] As mentioned above, it's difficult to match images from a purely aerial perspective with images from a ground perspective due to differences in perspective. Therefore, the present embodiment utilizes drone images of takeoff and landing locations and nearby ground images to extract and match feature points, improving the matching success rate and thus the accuracy of the aerial and ground coordinate system alignment. This step only matches images in overlapping areas. For example, if there are 1,000 aerial images and 500 ground images, of which 300 have overlapping areas, this step matches the aerial and ground images within these 300 images.

[0065] This step uses a scale- and viewpoint-weighted matching method to perform sparse matching in the overlapping area between air and ground. Because aerial and ground images are shot at different distances and resolutions, the scale (size) of the same object can vary significantly between them. Therefore, this step unifies the image scale and extracts and matches feature points from the rescaled images.

[0066] S140 may specifically include the following steps: S141, determining target aerial images and target ground images having overlapping areas based on first pose information of the plurality of target aerial images and second pose information of the plurality of target ground images; Among them, aerial images and ground images with the same or similar positions and the same or similar postures can be determined as target aerial images and target ground images with overlapping areas.

[0067] S142, based on the aerial sparse point cloud and the ground sparse point cloud, determining sparse points with the same name in the target aerial image and the target ground image located in an overlapping area, and obtaining a plurality of sparse point pairs consisting of the sparse points with the same name, wherein each sparse point pair includes an aerial sparse point and a ground sparse point; In this step, the 3D coordinates of the aerial sparse points in the target aerial image in the overlapping region can be obtained based on the aerial sparse point cloud, and the 3D coordinates of the ground sparse points in the target ground image in the overlapping region can be obtained based on the ground sparse point cloud. The aerial sparse points and ground sparse points with similar 3D coordinates are then identified as sparse points with the same name. Sparse points with the same name are similar to feature points with the same name, except that feature points with the same name are 2D points, while sparse points with the same name are 3D points. Sparse points with the same name represent the same physical point in the real world.

[0068] S143, for each sparse point pair, determining a target aerial image corresponding to the aerial sparse point in the sparse point pair from the plurality of aerial images to obtain a plurality of target aerial images, and determining a target ground image corresponding to the ground sparse point in the sparse point pair from the plurality of ground images to obtain a plurality of target ground images; In this step, the image corresponding to the sparse point refers to the image in which the sparse point can be observed. The number of images may be one or more. The more images corresponding to the sparse point, the more stable the sparse point is.

[0069] S144, combining the plurality of target aerial images and the plurality of target ground images in pairs to obtain a plurality of air-ground image pairs, wherein each air-ground image pair includes one target aerial image and one target ground image; For example, for a pair of sparse points, there are 2 determined target aerial images and 3 determined target ground images. In this step, 6 pairs of air-ground image pairs can be obtained by combining the air-ground images in pairs.

[0070] S145, for each pair of air-ground images, scaling the target air image therein at multiple scales to obtain multiple aerial scaled images of different scales, and scaling the target ground image therein at multiple scales to obtain multiple ground scaled images of different scales; Multi-scale scaling can be performed using a pyramid-like approach, creating multiple scales of images. This can be scaled according to a variety of preset ratios. By setting a reasonable number of scales, it is possible to ensure that both the aerial and ground images of the target are scaled to the same scale.

[0071] S146, determining an aerial zoomed image and a ground zoomed image having the same scale from the plurality of aerial zoomed images and the plurality of ground zoomed images; S147, extracting and matching feature points based on the view weights of the aerial zoomed image and the ground zoomed image of the same scale to obtain relative pose information between the target aerial image and the target ground image; The perspective weights of feature points extracted from aerial zoom images taken at lower altitudes are greater than those of feature points extracted from aerial zoom images taken at higher altitudes. Images taken at lower altitudes have a flatter perspective and are more useful as references, so feature points in such images are given higher weights.

[0072] S150, based on the first pose information, the second pose information and the relative pose information, the pose information and sparse point positions of each image in the target aerial image and the target ground image in the overlapping area are optimized using global bundling adjustment with control point constraints to obtain a fusion model.

[0073] This step only optimizes the image information in the overlapping area. For example, if there are 1,000 aerial images and 500 ground images, and 300 of them have overlapping areas, this step optimizes the aerial and ground images in these 300 images.

[0074] In the air and ground fusion process of the present embodiment, the image in the overlapping area needs to be optimized. Therefore, the sparse points outside the overlapping area can be fixed as control points, and control point constraints can be added during the optimization process to improve the stability of the optimization.

[0075] S150 may specifically include the following steps: S151, determining control points from all sparse points included in the aerial sparse point cloud and the ground sparse point cloud; In this step, the number of images corresponding to all sparse points included in the aerial sparse point cloud and the ground sparse point cloud is obtained. Then, sparse points whose number of corresponding images is greater than a first preset value and whose number of images without overlapping areas is less than or equal to a second preset value are determined as control points. The first preset value can be 4, and the second preset value can be 2.

[0076] S152, based on the first pose information, the second pose information and the relative pose information, with the position of the control point as a constraint, the pose information and sparse point position of each image in the target aerial image and the target ground image with overlapping areas are optimized by global bundling adjustment to obtain a fusion model.

[0077] Since the control points are outside the overlapping area, or although they are in the overlapping area, the corresponding images are relatively large and stable, the control point constraints are used during the global bundle adjustment optimization to ensure that the positions of the control points do not change, thereby improving the reconstruction stability. Images with no overlapping areas on the ground will not be optimized incorrectly. Similarly, images with no overlapping areas in the air will not be optimized incorrectly.

[0078] The fusion model includes the optimized sparse points and their positions, as well as the optimized pose information of each image. The positions of the control points in the fusion model remain unchanged from the positions of the control points before optimization.

[0079] This application performs drone route planning and ground-end scanning planning on the scanning area, so that the scanning path of the ground scanning device passes through the drone's take-off and landing points, resulting in overlapping areas between the take-off and landing images in the aerial images taken by the drone and the ground images taken by the ground scanning device. Feature points in the overlapping areas can be extracted and matched between the air-ground images. The images in the overlapping areas are added to the global bundling adjustment optimization process, which improves the alignment accuracy of the coordinate system during air-ground fusion and the consistency of the air-ground posture. Compared with the simple alignment of the aerial image and the ground image alone, it has higher accuracy. In addition, the control point constraints are used during the global bundling adjustment optimization to improve the reconstruction stability. Images without overlapping areas on the ground will not be optimized incorrectly. Similarly, images without overlapping areas in the air will not be incorrectly optimized, which reduces the geometric distortion and misalignment problems in the fusion area and further improves the accuracy.

[0080] After obtaining a fusion model containing sparse points and image poses through this application, subsequent operations such as dense reconstruction and texture mapping can be further performed. Since the sparse point positions and image poses in the fusion model obtained in the embodiment of the present application are highly accurate, that is, the accuracy of sparse reconstruction is high, and subsequent dense reconstruction and texture mapping depend on the accuracy of sparse reconstruction, the accuracy of subsequent dense reconstruction, 3dgs, etc. can be improved through this application. The fused sparse three-dimensional model has both large-scale coverage and high detail accuracy, and is suitable for subsequent urban modeling, digital twins, smart cities and other fields.

[0081] Figure 5 A schematic diagram of the structure of the terminal device provided in an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the terminal device.

[0082] like Figure 5 As shown, the terminal device 3 may include: a processor (processor) 302 and a memory (memory) 304.

[0083] The memory 304 is used to store a computer program 306. The memory 304 may include a high-speed RAM memory, or may also include a non-volatile memory, such as at least one disk memory. The computer program 306 may include computer-executable instructions.

[0084] The processor 302 is configured to execute the computer program 306 to implement the above-mentioned 3D reconstruction method embodiment.

[0085] The processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the terminal device 3 may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0086] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned three-dimensional reconstruction method embodiment is implemented.

[0087] An embodiment of the present application provides a computer program that can be executed by a processor to implement the above-mentioned three-dimensional reconstruction method embodiment.

[0088] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned three-dimensional reconstruction method embodiment is implemented.

[0089] In the several embodiments provided in this application, if any function is implemented in the form of a software function module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or other terminal device) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store computer program code.

[0090] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the present application.

[0091] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims that list several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

[0092] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Performing drone route planning and ground-end scanning planning on the scanning area to obtain a drone shooting route and a ground-end scanning path, wherein the drone shooting route includes the drone take-off and landing points and the drone shooting path, and the ground-end scanning path includes the scanning path of the ground scanning device, and the scanning path of the ground scanning device passes through the drone take-off and landing points; Acquire a plurality of aerial images from different perspectives obtained by continuous aerial photography by the drone along the drone photography route, and a plurality of ground images obtained by ground photography by the ground scanning device along the ground-end scanning path, wherein the coordinate system of the ground images is the same as the coordinate system of the aerial images; Reconstructing an aerial model based on the plurality of aerial images to obtain an aerial sparse point cloud and first pose information of each aerial image, and reconstructing a ground model based on the plurality of ground images to obtain a ground sparse point cloud and second pose information of each ground image; Based on the aerial sparse point cloud and the ground sparse point cloud, matching target aerial images and target ground images in overlapping areas among the plurality of aerial images and the plurality of ground images to obtain relative pose information between the target aerial image and the target ground image; According to the first pose information, the second pose information and the relative pose information, global bundling adjustment with control point constraints is used to optimize the pose information and sparse point positions of each image in the target aerial image and the target ground image in the overlapping area to obtain a fusion model.

2. The method according to claim 1, characterized in that The aerial model reconstruction is performed based on the plurality of aerial images to obtain an aerial sparse point cloud and first position information of each aerial image, including: Get the position and posture POS data of each aerial image; Determining feature points and feature descriptors of the feature points in each aerial image, wherein the feature points include corner points, edge intersection points, and centers of high-contrast areas; Determining a plurality of aerial image pairs from the plurality of aerial images according to the POS data, wherein each pair of the aerial image pairs includes two aerial images, and the two aerial images have an overlapping area; For each pair of aerial images, determining feature points with the same name in the two aerial images, and obtaining multiple pairs of matching feature points consisting of feature points with the same name; The relative pose information between the two aerial images to which the matching feature points belong is determined based on the matching feature points, and the initial first pose information and the initial aerial sparse point cloud of each of the aerial images are determined based on the relative pose information between the two aerial images. The initial first pose information and the initial aerial sparse point cloud are optimized through global bundling adjustment to obtain the first pose information and the aerial sparse point cloud.

3. The method according to claim 1, characterized in that The ground scanning device is provided with a laser radar and a camera, and the ground image is an image taken by the camera; Reconstructing a ground model based on the plurality of ground images to obtain a ground sparse point cloud and second pose information of each ground image includes: Determining a laser trajectory based on the point cloud data collected by the laser radar, wherein the laser trajectory includes a pose sequence of the laser radar; Transferring the laser trajectory to an image using pre-marked extrinsic parameters between the laser radar and the camera to obtain an image trajectory, wherein the image trajectory includes an initial estimated pose of each of the ground images; Based on the image trajectory, the pose information of the ground image is globally bundled, adjusted and optimized through feature point extraction and matching to obtain the second pose information and the ground sparse point cloud.

4. The method according to claim 1, wherein Based on the aerial sparse point cloud and the ground sparse point cloud, matching target aerial images and target ground images in overlapping areas among the plurality of aerial images and the plurality of ground images to obtain relative pose information between the target aerial images and the target ground images includes: Determining target aerial images and target ground images having overlapping areas based on the first pose information of the plurality of target aerial images and the second pose information of the plurality of target ground images; Based on the aerial sparse point cloud and the ground sparse point cloud, determining sparse points with the same name in the target aerial image and the target ground image located in the overlapping area, and obtaining a plurality of sparse point pairs consisting of the sparse points with the same name, wherein each sparse point pair includes an aerial sparse point and a ground sparse point; For each sparse point pair, determining a target aerial image corresponding to the aerial sparse point in the sparse point pair from the plurality of aerial images to obtain a plurality of target aerial images, and determining a target ground image corresponding to the ground sparse point in the sparse point pair from the plurality of ground images to obtain a plurality of target ground images; Combining the plurality of target aerial images and the plurality of target ground images in pairs to obtain a plurality of air-ground image pairs, wherein each pair of the air-ground image pairs includes one target aerial image and one target ground image; For each pair of the air-ground image pairs, the target aerial image is scaled at multiple scales to obtain multiple aerial scaled images of different scales, and the target ground image is scaled at multiple scales to obtain multiple ground scaled images of different scales; Determining an aerial zoomed image and a ground zoomed image having the same scale from the plurality of aerial zoomed images and the plurality of ground zoomed images; Feature point extraction and matching based on perspective weight are performed on the aerial zoomed image and the ground zoomed image of the same scale to obtain relative pose information between the target aerial image and the target ground image, wherein the perspective weight of the feature points extracted from the aerial zoomed image with a lower shooting altitude is greater than the perspective weight of the feature points extracted from the aerial zoomed image with a higher shooting altitude.

5. The method according to claim 4, characterized in that The determining, based on the aerial sparse point cloud and the ground sparse point cloud, sparse points with the same name in the target aerial image and the target ground image located in the overlapping area includes: Based on the aerial sparse point cloud, obtaining three-dimensional coordinates of aerial sparse points located in the target aerial image in the overlapping area, and based on the ground sparse point cloud, obtaining three-dimensional coordinates of ground sparse points located in the target ground image in the overlapping area; The aerial sparse points and the ground sparse points with similar three-dimensional coordinates are determined as the sparse points with the same name.

6. The method according to claim 1, characterized in that The method further comprises: optimizing the pose information and sparse point positions of each of the target aerial image and the target ground image in the overlapping area by using a global bundle adjustment with control point constraints based on the first pose information, the second pose information, and the relative pose information to obtain a fusion model, including: Determine control points from all sparse points included in the aerial sparse point cloud and the ground sparse point cloud; According to the first pose information, the second pose information and the relative pose information, with the position of the control point as a constraint, global bundling adjustment is used to optimize the pose information and sparse point positions of each image in the target aerial image and the target ground image with overlapping areas to obtain a fusion model, wherein the position of the control point in the fusion model remains unchanged from the position of the control point before optimization.

7. The method according to claim 6, characterized in that The determining of control points from all sparse points included in the aerial sparse point cloud and the ground sparse point cloud comprises: Obtaining the number of images corresponding to all sparse points included in the aerial sparse point cloud and the ground sparse point cloud; The sparse points, the number of which corresponding to the control points is greater than a first preset value and the number of which corresponding to the control points does not include the overlapping area is less than or equal to a second preset value.

8. The method according to claim 6, characterized in that The fusion model includes multiple optimized sparse points and their positions, and optimized pose information of each image.

9. The method according to claim 1, characterized in that The UAV route planning and ground-side scanning planning for the scanning area are performed to obtain the UAV shooting route and the ground-side scanning path, including: Dividing the scanning area into a plurality of grids; Determine the vertex of each grid as the take-off and landing point of the UAV; Determining a starting shooting altitude and an ending shooting altitude of the drone to obtain the drone shooting path, wherein the drone shooting path is a path along a vertical direction, wherein the starting point is a position vertically distanced from the drone take-off and landing point by the starting shooting altitude, and the ending point is a position vertically distanced from the drone take-off and landing point by the ending shooting altitude, and the difference between the starting shooting altitude and the shooting altitude of the ground scanning device is within a preset range, so that the plurality of aerial images and the plurality of ground images have overlapping areas of the target aerial image and the target ground image; The scanning path of the ground scanning device is determined based on the take-off and landing point of the UAV, so that the scanning path of the ground scanning device passes through the take-off and landing point of the UAV.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the three-dimensional reconstruction method according to any one of claims 1 to 9.

Images