A multi-platform combined bundle adjustment method based on object space structure information

Abstract

The present invention relates to a method for multi-platform combined bundle adjustment based on object space structural information. The present invention optimizes the position and direction of object facet elements through the variational facet optimization method to make it closer to the actual surface situation of the object, obtains cross-platform connection points and object space structural information based on object facet element matching, and aligns the two platforms through the cross-platform connection points to eliminate differences, thereby improving the stability during the bundle adjustment process. More importantly, different weights are assigned to image points and object points according to object space structural information and connection point degrees, effectively improving the accuracy of multi-platform combined bundle adjustment. The present invention combines aerial oblique images with ground images for bundle adjustment, solving the problems of unstable solutions and low accuracy that are prone to occur in traditional methods when facing data from different platforms. The accuracy of three-dimensional modeling is improved, providing an effective solution for high-fidelity and high-precision comprehensive modeling in urban environments.

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional reconstruction, and in particular relates to a multi-platform combined bundle adjustment method based on object space structure information. Background Art

[0002] In urban areas, oblique photogrammetry based on drones often faces many challenges, including occlusion, narrow flight areas, and large camera tilt angles, which lead to incomplete information acquisition, resulting in holes, blur or distortion in the generated 3D model. While ground images can provide complementary perspectives and provide complete building outline information, relying solely on ground images can easily lead to the loss of information on the top of buildings in the generated 3D model. Therefore, through the fusion of multiple platforms, combining drone-based images with ground platform images can help enrich the detailed information of the 3D model and further realize the fine 3D modeling of urban areas. This multi-platform fusion method can not only overcome the limitations of each data collection process, but also improve the accuracy and integrity of the overall modeling.

[0003] Aerial triangulation is a key step in obtaining the three-dimensional coordinates of the target in photogrammetry. It obtains connection points by matching images and determines the position of the target and the pose of the image based on the intersection calculation of the light rays with the same name. Aerial triangulation has gone through two stages: simulation and analytical methods, and is now in the full digital automation stage. At present, the mainstream methods tend to use the beam method in analytical photogrammetry because its calculation method is the most rigorous, theoretically the most accurate method, and can achieve higher accuracy in practical applications, which can meet the application needs of engineering in different situations.

[0004] However, the traditional bundle adjustment is only applicable to a single photogrammetric platform and the case of downward-looking images. It ignores the object space structure information of the connection points, such as the ground sampling distance of the local object surface and the distance to the camera, which seriously affects the stability and accuracy of the bundle adjustment. In addition, it assumes that the accuracy of all observations is the same and applies the same and symmetrical weights to the x and y coordinates of image points observed from different directions. However, for the case of multiple platforms combined aerial triangulation, this assumption of the traditional bundle adjustment is invalid. Therefore, it is necessary to propose a multi-platform combined bundle adjustment method. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a combined bundle adjustment method suitable for multiple platforms based on object space structural information. The method mainly includes the following two contents: (1) Matching based on object space elements: Create surface elements based on sparse point clouds obtained separately from aerial images and ground images, and make the surface elements closer to the surface of real objects through variational surface element optimization, and then obtain cross-platform connection points based on surface element matching. (2) Reweighting based on object space structural information: Calculate the footprint shape of square pixels projected onto the surface of the building based on the variational surface element optimization method, and determine the prior weight of the image point observation value according to its ground sampling distance in the x and y directions and the image point depth value, so that the image point observation value with coarse resolution can be penalized; cross-platform connection points are connected with each platform to strengthen the consistency between different platforms. Although they account for a small proportion of all connection points, they are more important. The proposed process divides all connection points into intra-platform connection points and cross-platform connection points, and assigns different weights to the cross-platform connection points according to the degree and the ratio of all connection points.

[0006] Specifically, the method of the present invention comprises the following steps:

[0007] (1) Using available auxiliary data, such as global navigation satellite systems and ground control points, obtain the initial exterior orientation elements of the aerial and ground platforms, perform the structure-from-motion process for each platform separately, and generate sparse point clouds for each platform separately;

[0008] (2) Constructing facets from sparse point clouds and optimizing them using the variational method. Then, these optimized facets are matched with object facets to obtain cross-platform connection points. Based on the optimized facets, the ground sampling distance and depth value of the projection of the image point on the real surface of the building are calculated.

[0009] (3) The sparse point cloud of the ground platform is aligned with the point cloud of the aerial platform through the cross-platform connection points to obtain a set of rigid transformation parameters; this set of transformation parameters is applied to the initial exterior orientation elements of the ground image to obtain the corrected exterior orientation elements, thereby eliminating the differences in the exterior orientation elements between the two platforms;

[0010] (4) The initial exterior orientation elements of the aerial image and the corrected exterior orientation elements of the ground image are used as initial values, and are reweighted according to the cross-platform connection points and object structure information, and a joint bundle adjustment is performed.

[0011] The object element established includes the center point and the normal vector, both of which are quantities that need to be optimized. In order to improve the optimization efficiency, the geometric displacement of the object element is constrained according to the light between the object point and the photography center, and the change in the coordinates of the center point of the object element is constrained to the displacement in the direction of the light. The expression of the object element is as follows: a(XX c )+b(YY c)+c(ZZ c )=0

[0012] (a, b, c) is the normal vector;

[0013] (X c ,Y c ,Z c ) is the center point of the object element.

[0014] The present invention optimizes the position and direction of the object facet element through the variational facet optimization method to make it closer to the actual situation of the object surface, and obtains the cross-platform connection points and object structure information (i.e., ground sampling distance and depth value) based on the object facet element matching, and aligns the two platforms through the cross-platform connection points to eliminate the differences, thereby improving the stability during the bundle adjustment process. More importantly, different weights are assigned to image points and object points according to the object structure information and the degree of the connection points, which effectively improves the accuracy of multiple platforms in the joint bundle adjustment. The present invention combines aerial oblique images with ground images for bundle adjustment, which solves the problems of unstable solution and low accuracy that are prone to occur in traditional methods when facing data from different platforms. The accuracy of three-dimensional modeling is improved, providing an effective solution for high-fidelity and high-precision comprehensive modeling in urban environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the object element, including the center point and normal vector.

[0016] Figure 2 Schematic diagram of matching based on object aspect elements.

[0017] Figure 3 Schematic diagram of the principle for calculating ground sampling distance.

[0018] Figure 4 Schematic diagram of ground sampling distance for different platforms.

[0019] Figure 5 Schematic diagram of pixel resolution on different platforms.

[0020] Figure 6 This is the flowchart of multi-platform joint bundle adjustment based on object space structure information.

[0021] Figure 7 Renderings of sparse point clouds generated for aerial and ground platforms respectively.

[0022] Figure 8 This is a comparison chart of the matching results with the traditional algorithm ASIFT. DETAILED DESCRIPTION

[0023] The present invention is a multi-platform combined bundle adjustment method based on object space structure information. The technical process is as follows: Figure 6 The sparse point clouds of the two platforms are obtained by performing motion recovery on the images of the aerial and ground platforms respectively. Then, the surface elements are created based on the sparse point clouds and the object space structure information is calculated using the variational surface element optimization method. Finally, the multi-platform joint bundle adjustment is performed based on the object space structure information and the cross-platform connection points.

[0024] Based on the above principles, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0025] 1. For each platform, use the available auxiliary data such as global navigation satellite system and ground control points to perform motion recovery structure process, obtain sparse point clouds respectively, and construct surface elements based on the sparse point clouds. The surface element mainly includes the center point position and normal vector information, such as Figure 1 shown.

[0026] 2. Optimize the facets by using the variational facet optimization method, and then match the facets by object facets (such as Figure 2 The cross-platform connection points are obtained by aligning the sparse point cloud of the ground platform with the point cloud of the aerial platform according to the cross-platform connection points to obtain a set of rigid transformation parameters. This set of transformation parameters is applied to the initial exterior orientation elements of the ground image to obtain the corrected exterior orientation elements, thereby eliminating the differences in the exterior orientation elements between the two platforms.

[0027] 3. According to the optimized facet position and normal vector, calculate the ground sampling distance and depth value of the projection of the image point on the real surface of the building. The formula for calculating the ground sampling distance is as follows, and the schematic diagram is as follows: Figure 3 As shown:

[0028]

[0029]

[0030] Where D is the distance from the projection center O to the face element, and f is the focal length (mm). The optimized face element direction n is obtained by rotating the initial direction n' around the x-axis by τ and around the y-axis by an angle of υ. The schematic diagrams of the ground sampling distance and the image point resolution are shown in Figure 2. Figure 4 and Figure 5 shown.

[0031] 4. In the process of bundle adjustment, the image points and object points are re-weighted. The specific weighting formula is as follows:

[0032]

[0033]

[0034]

[0035] Where Px and P y Represents the weight of the image point in the horizontal and vertical directions respectively. dep0 represents the average depth of the image where the image point is located, dep represents the depth value of the image point, P objPt represents the weight of the object point, N0 represents the total number of connection points, N represents the number of cross-platform connection points, C1 and C2 represent the number of images of the two platforms where the connection point is visible, and G represents the degree of cross-platform connection points (G = C1 + C2).

[0036] Finally, the initial exterior orientation elements of the aerial image and the corrected exterior orientation elements of the ground image are used as the initial values ​​of the unknowns. The coordinates of the cross-platform connection points and image points are re-weighted using the above weighting formula according to the degree of the cross-platform connection points and the object space structure information of each image point, and the multi-platform joint bundle adjustment is performed.

[0037] The multi-platform combined bundle adjustment method proposed in the present invention mainly includes two parts: (1) matching based on facets: facets are created based on sparse point clouds obtained from aerial images and ground images respectively, and facets are optimized by variational facet optimization method to make them closer to the real surface of the object to reduce the influence of perspective deformation and scale difference on matching, and then facet-based matching is performed to obtain cross-platform connection points; (2) re-weighting based on object-space structural information: object-space structural information is calculated based on the result of variational facet optimization. The ground sampling distance of the pixel can be calculated by the direction of the optimized facet, and the depth value can be calculated by the position of the optimized facet. The above factors are combined to re-weight the image points. This process divides all connection points into two groups: single-platform connection points (whose visible image points belong to the image obtained from the same platform) and cross-platform connection points (whose visible image points belong to the image obtained from different platforms). Among them, cross-platform connection points establish connections between each platform. Although they only account for a small part of all connection points, they are of vital importance. This method assigns different weights according to the degree of cross-platform connection points and the ratio of the total number of connection points.

[0038] The following is an example of the method:

[0039] This embodiment is verified using the Center Hall data of Dortmund, Germany, a public data set of the International Society for Photogrammetry and Remote Sensing (ISPRS). The drone image is acquired by the DIJ S800 multi-rotor platform equipped with a Sony Nex-7, and the image GSD range is 1 to 3 cm. The camera model of the ground image is the same as that of the drone, but the GSD accuracy reaches the sub-centimeter level, and the image size is 6000×4000. This embodiment uses a total of 350 images (178 drone images and 172 ground images) and 40 checkpoints (4 of which are used as control points).

[0040] The specific implementation process of this embodiment is:

[0041] Step 1: Obtain sparse point clouds for each platform

[0042] The motion recovery structure process is performed on multiple platforms to generate sparse point clouds, which provide a data basis for surface element-based matching. The sparse point clouds generated by the air and ground platforms are as follows: Figure 7 shown.

[0043] Step 2: Perform facet-based matching

[0044] The proposed facet-based matching method is compared with the traditional algorithm Affine-SIFT through examples. Five pairs of images with large perspective deformation are selected. The matching results are shown in the figure below. Figure 8 As shown in Table 1, the percentage of correct matching points obtained by several methods are shown in Table 1.

[0045] Table 1: Comparison of matching results

[0046]

[0047] Step 3: Eliminate differences between platforms and obtain object structure information

[0048] The cross-platform connection points obtained by facet-based matching are used to align the ground platform and the aerial platform to eliminate the differences between the platforms. The object space structure information is calculated based on the facet optimization results to quantify the weights of image points and object points during the multi-platform joint bundle adjustment.

[0049] Step 4: Multi-platform combined bundle adjustment

[0050] According to the calculated object space structure information and weighting formula, the image points and object points are re-weighted, and the multi-platform joint bundle adjustment is performed and the results are compared with the traditional bundle adjustment method, as shown in Table 2.

[0051] Table 2: Multi-platform combined bundle adjustment results

[0052]

Claims

1. A multi-platform combined bundle adjustment method based on object space structure information, the characteristics of which include Follow these steps: (1) Using available auxiliary data, the initial exterior orientation elements of the aerial and ground platforms are obtained, and the structure-from-motion process is performed on each platform to generate sparse point clouds for each platform. (2) Constructing facets from sparse point clouds and optimizing them using the variational method. Then, these optimized facets are matched with object facets to obtain cross-platform connection points. Based on the optimized facets, the ground sampling distance and depth value of the projection of the image point on the real surface of the building are calculated. (3) The sparse point cloud of the ground platform is aligned with the point cloud of the aerial platform through the cross-platform connection points to obtain a set of rigid transformation parameters; this set of transformation parameters is applied to the initial exterior orientation elements of the ground image to obtain the corrected exterior orientation elements, thereby eliminating the differences in the exterior orientation elements between the two platforms; (4) Using the initial exterior orientation elements of the aerial image and the corrected exterior orientation elements of the ground image as initial values, reweighting them according to the cross-platform tie points and object structure information and performing a joint bundle adjustment; The ground sampling distance and depth value of the image point are calculated through the result of object aspect element matching. The formula for calculating the ground sampling distance is as follows: Where D is the distance from the projection center O to the face element, f is the focal length, in mm, and the direction of the face element after optimization is obtained after the initial face element is rotated around the x-axis by an angle τ and around the y-axis by an angle υ; When performing multi-platform combined bundle adjustment, different image points and object points are assigned different weights according to the object space structure information and the degree of cross-platform connection points. The weight calculation formula is as follows: Where P x and P y Represent the weights of the image point in the horizontal and vertical directions respectively, dep0 represents the average depth of the image where the image point is located, dep represents the depth value of the image point, P objPt represents the weight of the object point, N0 represents the total number of connection points, N represents the number of cross-platform connection points, C1 and C2 represent the number of images of the two platforms where the connection points are visible, G represents the degree of cross-platform connection points, G=C1+C2.

2. The method according to claim 1, characterized in that: The available assistance data include global navigation satellite system and ground control points.

3. The method according to claim 1 or 2, characterized in that: The object element established includes the center point and the normal vector. According to the light constraint between the object point and the photography center, the geometric displacement of the object element is constrained, and the change in the coordinates of the center point of the object element is constrained to be the displacement in the direction of the light. The expression of the object element is as follows: a(XX c )+b(YY c )+c(ZZ c )=0 (a, b, c) is the normal vector; (X c ,Y c ,Z c ) is the center point of the object element.

Images