An Unmanned Aerial Vehicle Image Positioning and Attitude Determination Method Based on Adaptive Fusion of Point and Line Features

Abstract

A method for UAV image positioning and attitude determination based on adaptive fusion of point and line features, which relates to the field of position and attitude estimation at the moment of UAV image shooting. The method for UAV image positioning and attitude determination based on adaptive fusion of point and line features in the present invention starts from the observation accuracies of point features and line features, takes whether the mean square error of unit weight of point features and line features is equal as the evaluation criterion, iteratively calculates and adjusts the weights of the two, and finally ensures that the weights of point and line features are determined under a unified standard. Since the reprojection error of point and line features is used as the evaluation criterion for the accuracy of observation values, and only relies on the reprojection error of point and line features to determine the weights of the two types of features, it avoids the interference of human experience on the calculation of weights, making the process of determining the weights of point features and line features more reasonable. Therefore, the present invention can reasonably evaluate the weights of point and line features in different application scenarios, etc., and is suitable for large-scale promotion and application.

Description

Technical Field

[0001] The present invention relates to the field of position and attitude estimation at the moment of UAV image shooting, and particularly to a UAV image positioning and attitude determination method based on adaptive fusion of point and line features, specifically to a method for calculating the weight of two types of features according to the real-time determination of the observation accuracy of point features and line features and fusing point and line features to calculate the camera pose. Background Art

[0002] It is known that image positioning and attitude determination is the process of determining the spatial position and attitude at the moment of camera shooting, that is, determining the exterior orientation elements of the camera. Accurately solving the exterior orientation elements of the camera is the basic content and core link of orthophoto generation, 3D modeling, and visual positioning.

[0003] Generally speaking, the main technical process of current image positioning and attitude determination is to set out image control points in the field, conduct aerial photography, and perform aerial triangulation calculation in the office. Since the workload of setting out image control points in the field is relatively large, in professional surveying and mapping UAVs, high-precision and expensive GNSS or IMU sensors are often equipped. Using high-precision pose observation values as prior information for pose solution can not only ensure the accuracy of pose solution but also greatly reduce the number of image control points. However, ordinary surveying and mapping UAVs or consumer UAVs do not use GNSS or IMU instruments, or the instrument accuracy is very poor, and the strategy of using GNSS or IMU measurement values as pose prior information will no longer be feasible. Therefore, a common practice is to add sensor data or geometric constraint relationships to the bundle adjustment model. For example, introducing lidar data and DTM information into the block adjustment process of the bundle method can reduce the influence of image point measurement errors on the pose solution result. However, the method of introducing sensor data constraints requires obtaining auxiliary data of the shooting area in advance. Whether querying existing data or obtaining data on-site, it reduces the usage efficiency of the UAV. And using the geometric constraint relationships existing between observation data to solve the camera pose can avoid the above problems while improving the pose solution accuracy. Therefore, this method is widely used in the field of UAV image pose estimation. Specifically, the geometric constraints between observation data mainly include geometric information such as coplanar points, straight lines, and right angles. Among them, the straight line feature can not only express the structural information of the scene and reflect the geometric topological relationship of the scene, but also the corresponding straight line of the object-side straight line on the image does not require complete visibility, and the endpoints of the straight lines corresponding to the object side and the image side do not require to be homologous points. In some complex scenes prone to occlusion, it can play an advantage that cannot be compared with point features. Since there are rich and available straight line features in urban and other artificial structure scenes. Therefore, fusing point features and straight line features to solve the camera pose becomes a feasible solution for high-precision positioning and attitude determination of UAV images for urban scenes.

[0004] However, in the existing technical solutions for calculating the camera pose by fusing point and line features, either treating point and line features equally or determining the weight value during their fusion based on experience level fails to reasonably evaluate the contribution degree of point and line features in calculating the camera pose. Due to differences in distribution and quantity, point features and line features often exhibit significant accuracy differences in different scenarios. Treating point and line features equally cannot reasonably reflect the accuracy of the observed values in the result of pose calculation, while relying on experience to set the weight value for fusing point and line features cannot be applied to multiple application scenarios, and the calculation process depends on manual interaction, which is not conducive to the automated operation of the algorithm. Therefore, in the process of pose calculation, how to adaptively determine the weight values of point and line features according to the characteristics of the application scenario and fuse point and line features to obtain high-precision pose parameters has become an urgent problem to be solved in the field of UAV image positioning and orientation, etc. Summary of the Invention

[0005] To overcome the deficiencies in the background technology, the present invention provides a UAV image positioning and orientation method based on adaptive fusion of point and line features. The present invention avoids the interference of human experience in the calculation of weight values, making the process of determining the weight values of point features and line features more reasonable. Therefore, the present invention can reasonably evaluate the weight values of point and line features in different application scenarios, etc.

[0006] To achieve the above-mentioned invention purpose, the present invention adopts the following technical solutions:

[0007] A UAV image positioning and orientation method based on adaptive fusion of point and line features, the UAV image positioning and orientation method specifically includes the following steps:

[0008] The first step, obtaining target area data:

[0009] Using a UAV to obtain the image data, POS information and ground control point data of the target area, and then evaluating the shooting quality of the image, checking whether the data format, metadata are correct, and whether the image overlap degree meets the requirements;

[0010] The second step, performing feature matching:

[0011] Extracting and matching point feature and line feature information respectively, performing free network bundle adjustment for point features, and eliminating mis-matched lines;

[0012] The third step, performing pose optimization:

[0013] Based on the successfully matched features, calculating the initial pose value using point features, and adaptively determining the weight value during fusion according to the respective observation accuracies of point and line features to calculate the pose parameters of the camera;

[0014] The fourth step, outputting the final pose result:

[0015] The pose result is transformed to the world coordinate system based on the measured control points, and thus the process of UAV positioning and pose determination is completed.

[0016] For the UAV image positioning and pose determination method based on adaptive fusion of point and line features, when obtaining the image data, POS information, and ground control point data of the target area by the UAV in the first step, the working area of the UAV in the field should be determined according to specific task requirements. Control points should be arranged in the area and their three-dimensional coordinates in the world coordinate system should be measured. When the UAV collects data, the UAV flight path should be planned according to the task requirements, and the corresponding overlap degree and flight height parameters should be set to obtain the UAV image data within the working area and the GNSS coordinates at the moment of image capture.

[0017] For the UAV image positioning and pose determination method based on adaptive fusion of point and line features, when evaluating the shooting quality of the image, checking the data format, whether the metadata is correct, and whether the image overlap degree meets the requirements in the first step, the integrity of the control points and POS data should also be checked, and whether the control points are clearly and completely imaged on the image.

[0018] For the UAV image positioning and pose determination method based on adaptive fusion of point and line features, when extracting the point features and line features respectively in the second step, the SIFT (Scale-invariant feature transform) algorithm is used for point feature information extraction, the brute-force matching method is used for matching, and the RANSAC (Random Sample Consensus) method is used for rejecting mismatches. The EDLines algorithm is used for straight line feature information extraction, the LP (Line-Points, invariant line-point distance ratio) algorithm is used for matching, and the geometric relationship of the trifocal tensor is used for rejecting mismatches.

[0019] For the UAV image positioning and pose determination method based on adaptive fusion of point and line features, when calculating the initial pose value using the point features based on the successfully matched features in the third step, using the calculated initial pose value, the weights of the point features and straight line features are both set to 1, the residuals between the observed values and predicted values of the point and line features are calculated, and the Helmert variance component estimation method is used to calculate the updated weights of the point and line features. Iterate the above process. When the unit weight variance components of the point features and straight line features are equal or nearly equal (when the difference between the unit weight variance components of the point features and straight line features is less than 1E-6, this method considers them to be nearly equal), it is considered that the weights of the point and line features have reached a reasonably determined degree at this time.

[0020] For the UAV image positioning and attitude determination method based on adaptive fusion of point and line features, when calculating the pose parameters of the camera by adaptively determining the weights during fusion according to the respective observation accuracies of points and lines in the third step, an iterative solution method is adopted to improve the accuracy of pose calculation. The accuracy of the initial value directly affects the accuracy of the pose calculation result. Using the successfully matched point features, select the two images with the largest number of matching points as the initial image pair, calculate the relative pose parameters between the two images, and then based on this, use the local three-dimensional points generated by the matching pairs to calculate the pose of the next image. Repeat the above process to complete the pose calculation of all images. To avoid the problem of large deviation in the pose calculation result caused by error accumulation, after calculating the poses of 3 to 4 images, perform a bundle adjustment once to reasonably distribute the accumulated errors.

[0021] For the UAV image positioning and attitude determination method based on adaptive fusion of point and line features, the weights of points and lines in the third step are the ratio of the unit weight variance to the observation value variance, and the calculation methods of the unit weight variances of points and lines are shown in the following formula:

[0022]

[0023] In the formula

[0024]

[0025]

[0026]

[0027] where \(n_1\) and \(n_2\) are the numbers of two types of observations, \(B_1\) and \(B_2\) are the coefficient matrices of the observation equations, \(P_1\) and \(P_2\) are the weight matrices of the observations, \(N_1\) and \(N_2\) are the normal equation matrices of the two types of observations, \(tr()\) is the sum of the main diagonal elements of the matrix, is the estimated value of the unit weight variance of the two types of observations, and \(V_1\) and \(V_2\) are the corrections of the observations;

[0028] The update method of the weights of points and lines is to estimate the variance components of points and lines according to the above formula, and obtain the estimated value of the unit weight variance of point and line observations Then update the weights according to the following formula:

[0029]

[0030] In the formula: \(c\) is an arbitrary constant, generally selected as one of the values in; \(k\) is the number of iterations.

[0031] For the UAV image positioning and attitude determination method based on adaptive fusion of point and line features, when calculating the pose parameters of the camera by adaptively determining the weights during fusion according to the respective observation accuracies of the points and lines in the third step, after obtaining the weights of the point features and the line features, the residual errors are reasonably assigned to the pose increments according to the magnitudes of the weights, and the pose calculation result is the sum of the initial pose value and the pose increment value.

[0032] Adopting the above technical solution, the present invention has the following advantages:

[0033] The UAV image positioning and attitude determination method based on adaptive fusion of point and line features of the present invention starts from the observation accuracies of the point features and the line features, uses whether the mean error of unit weight of the point features and the line features is equal as the evaluation criterion, and iteratively calculates and adjusts the weights of the two to finally ensure that the weights of the point and line features are determined under a unified standard. Since the reprojection error of the point and line features is used as the evaluation criterion for the accuracy of the observed values, and only the reprojection error of the point and line features is relied on to determine the weights of the two types of features, it avoids the interference of human experience on the calculation of the weights, making the process of determining the weights of the point features and the line features more reasonable. Therefore, the present invention can reasonably evaluate the weights of the point and line features in different application scenarios, etc., and is suitable for wide promotion and application. Description of the Drawings

[0034] Figure 1 is the flowchart of the embodiment of the present invention. Detailed Embodiment

[0035] The present invention can be more detailedly explained through the following embodiments, and the present invention is not limited to the following embodiments;

[0036] Combined with the attached Figure 1 For a UAV image positioning and attitude determination method based on adaptive fusion of point and line features, the UAV image positioning and attitude determination method specifically includes the following steps:

[0037] First step, obtaining target area data:

[0038] Using a UAV to obtain the image data, POS information and ground control point data of the target area, and then evaluating the shooting quality of the images, checking whether the data format, metadata are correct, and whether the image overlap meets the requirements;

[0039] During implementation, when using the UAV to obtain the image data, POS information and ground control point data of the target area, the working area of the UAV in the field should be determined according to specific task requirements, control points should be arranged in the area and their three-dimensional coordinates in the world coordinate system should be measured. When the UAV collects data, the UAV flight path should be planned according to the task requirements, and the corresponding overlap and flight height parameters should be set to obtain the UAV image data within the working area and the GNSS coordinates at the moment of image shooting;

[0040] Further, when evaluating the shooting quality of the evaluation image, the inspection data format, whether the metadata is correct, and whether the image overlap meets the requirements, the integrity of the control points and POS data can also be checked, and whether the control points are clearly and completely imaged on the image;

[0041] Step 2: Perform feature matching:

[0042] Extract and match the point feature and line feature information respectively, perform free network bundle adjustment on the point feature, and eliminate the mismatched straight lines;

[0043] During implementation, when extracting the point feature and line feature respectively, the SIFT (Scale-invariant feature transform) algorithm is used for point feature information extraction, the brute-force matching method is used for matching, the RANSAC (Random Sample Consensus) method is used for eliminating mismatches, the EDLines algorithm is used for line feature information extraction, the LP (Line-Points, invariant line-point distance ratio) algorithm is used for matching, and the geometric relationship of the trifocal tensor is used for eliminating mismatches;

[0044] Step 3: Perform pose optimization:

[0045] Based on the successfully matched features, calculate the initial pose value using the point feature, and the point and line features adaptively determine the weights during fusion according to their respective observation accuracies to solve the pose parameters of the camera;

[0046] During implementation, when calculating the initial pose value using the point feature based on the successfully matched features, using the calculated initial pose value, set the weights of the point feature and the line feature to 1, calculate the residuals between the observed values and predicted values of the point and line features, and use the method of Helmert variance component estimation to calculate the updated weights of the point and line features. Iterate the above process. When the unit weight variance components of the point feature and the line feature are equal or close to equal (when the difference between the unit weight variance components of the point feature and the line feature is less than 1E-6, this method considers them to be close to equal), it is considered that the weights of the point and line features have reached a reasonably determined level;

[0047] Further, when the point and line features adaptively determine the weights during fusion to calculate the pose parameters of the camera according to their respective observation accuracies, an iterative solution method is adopted to improve the accuracy of pose calculation. The accuracy of the initial value directly affects the accuracy of the pose calculation result. Using the successfully matched point features, the two images with the largest number of matching points are selected as the initial image pair, and the relative pose parameters between the two images are calculated. Then, based on this, the pose of the next image is calculated using the local three-dimensional points generated by the matching pairs, and the above process is repeated to complete the pose calculation of all images. To avoid the problem of large deviations in the pose calculation results caused by error accumulation, after calculating the poses of 3 to 4 images, a bundle adjustment is performed once to reasonably distribute the accumulated errors.

[0048] Further, the weights of the point and line features are the ratio of the unit weight variance to the observation value variance, and the calculation method of the unit weight variance of the point and line features is shown in the following formula:

[0049]

[0050] In the formula

[0051]

[0052]

[0053]

[0054] where n1 and n2 are the numbers of two types of observations, B1 and B2 are the coefficient matrices of the observation equations, P1 and P2 are the weight matrices of the observations, N1 and N2 are the normal equation matrices of the two types of observations, tr() is the sum of the main diagonal elements of the matrix, is the estimated value of the unit weight variance of the two types of observations, and V1 and V2 are the corrections of the observations;

[0055] The update method of the weights of the point and line features is to estimate the variance components of the point and line features according to the above formula, and obtain the estimated value of the unit weight variance of the point and line observations Then, the weights are updated according to the following formula:

[0056]

[0057] In the formula: c is an arbitrary constant, generally selected as one of the values in; k is the number of iterations.

[0058] Further, when the point and line features adaptively determine the weights during fusion to calculate the pose parameters of the camera according to their respective observation accuracies, after obtaining the weights of the point features and the line features, the residual values are reasonably assigned to the pose increments according to the magnitudes of the weights, and the pose calculation result is the sum of the pose initial value and the pose increment value;

[0059] When the weight distribution of point features and line features is reasonable, the iteration termination conditions include two categories: one is that the mean square error of unit weight of point features and line features is equal, and the other is that the iterative change value of reprojection / pose increment is extremely small. At this time, the weights of point-line features reflect the accuracy levels of the two types of features in the current scene, and the output pose solution result is the result of the adaptive fusion processing of point-line features;

[0060] Conversion of pose solution result: The camera pose solved above is in the local coordinate system. In order to apply the pose parameters in the real scale space, it is necessary to calculate the coordinate transformation parameters using the coordinates of the ground control points measured in advance to convert the camera pose to the real scale space;

[0061] Step 4: Output the final pose result:

[0062] Based on the measured control points, the pose result is converted to the world coordinate system, and thus the process of UAV positioning and orientation is completed.

[0063] The technical problems to be solved by the present invention are as follows:

[0064] The UAV image positioning and orientation method based on the adaptive fusion of point-line features starts from the observation accuracies of point features and line features, uses whether the mean square error of unit weight of point features and line features is equal as the evaluation criterion, and iteratively calculates and adjusts their weights, finally ensuring that the weights of point-line features are determined under a unified standard. Since the reprojection error of point-line features is used as the evaluation criterion for the accuracy of observation values, and only relies on the reprojection error of point-line features to determine the weights of the two types of features, it thus avoids the interference of human experience on the weight calculation, making the process of determining the weights of point features and line features more reasonable. Therefore, the present invention can reasonably evaluate the weights of point-line features in different application scenarios.

[0065] The reasonable determination of the weights of point-line features provides a technical basis for high-precision pose solution. According to their weights, the accuracy of the observation values can be reflected in the solution result in a more reasonable way, thus meeting the requirements of high-precision pose solution for UAV images. And the method of adaptive fusion of point-line features can also solve the problem of sensor pose estimation with multi-feature fusion in the fields of computer vision, robotics, etc.

[0066] The advantages of the present invention are as follows:

[0067] The present invention uses the simulation experiment data, and takes the mean square error of the difference between the pose solution result and the true value as the reference quantity to evaluate the correctness and effectiveness of the proposed method.

[0068] Generation method of simulated data: Set the size of the experimental area, randomly generate the three-dimensional coordinates of object points within the experimental area, and set the true pose of the camera. Generate simulated image point coordinate data according to the strict collinearity equation, and add Gaussian noise with different levels to the image point coordinates to simulate the observed values of image point coordinates in the real scene. Among them, the Gaussian noise is expressed as zero mean and different standard deviations.

[0069] The simulation experiment generates 9 UAV images in different poses, with a total of 50 ground feature points and 150 linear features imaged on the images. Since the linear feature extraction adopts the method of edge fitting, the mutual constraint between pixels makes the extraction accuracy of linear features higher than that of point features. Therefore, in order to objectively evaluate the pose solution accuracy of the adaptive fusion of the two types of features, the combinations of the standard deviations of Gaussian noise for point features and linear features are Experiment 1 (1, 0.5), Experiment 2 (1, 0.2), and Experiment 3 (1, 0.1) in turn. In each group of experiments, the camera pose is solved using point features, the camera pose is solved with equal weights of 1 for point and line features (hereinafter referred to as point-line 1_1), and the camera pose is solved with adaptive weight determination for point and line features.

[0070] Results of the simulation experiment:

[0071] (1) Determination of point-line feature weights

[0072] The comparison of the fusion weights of point-line features only involves two pose solution strategies: equal weights of 1 for point-line features and adaptive weight determination for point-line features.

[0073] Table 1 Comparison of the effects of fixed weights for point-line

[0074]

[0075] It can be seen from the table that as the Gaussian noise of the linear feature coordinate observations is gradually reduced, the weight of the linear feature gradually increases, proving that the weight determination strategy adopted in this scheme can reflect the accuracy difference of the point-line feature observations.

[0076] (2) Comparison of pose solution results

[0077] The point-line features adaptively determine weights during the iterative solution process, and then participate in the pose solution according to the determined weights. The root mean square error is calculated using the solved camera pose and the true pose. The following table shows the solution of the camera pose under different noise conditions.

[0078] Table 2 Mean square error of pose solution results

[0079]

[0080] As can be seen from the table, in different test scenarios, compared with the solution strategy where the weight values of point and line features are both set to 1, the method of adaptive fusion of point and line features can significantly improve the accuracy of pose solution, demonstrating the potential of this technical solution in practical applications.

[0081] Based on the above test results, this method can not only reflect the respective observation accuracies of point and line features, but also adaptively fuse point and line features to improve the accuracy of pose solution results, showing great potential in practical applications.

[0082] The parts not detailed in this invention are prior art.

[0083] The embodiments selected herein for disclosing the object of the present invention are considered to be suitable at present. However, it should be understood that the present invention is intended to cover all variations and improvements of all embodiments falling within the scope of this concept and invention.

Claims

1. A method for UAV image positioning and attitude determination based on adaptive fusion of point and line features, characterized in that: The method for UAV image positioning and attitude determination specifically includes the following steps: The first step: Obtain target area data: Use a UAV to obtain image data, POS information, and ground control point data of the target area, and then evaluate the shooting quality of the images, check the data format, whether the metadata is correct, and whether the image overlap meets the requirements; The second step: Perform feature matching: Extract and match point feature and line feature information respectively. Perform free network bundle adjustment on point features and eliminate mis-matched straight lines; Among them, when extracting point features and line features respectively, the SIFT algorithm is used for point feature information extraction, the brute-force matching method is used for matching, the RANSAC method is used for eliminating mis-matches, the EDLines algorithm is used for straight line feature information extraction, the LP algorithm is used for matching, and the geometric relationship of trifocal tensors is used for eliminating mis-matches; The third step: Perform attitude optimization: Based on the successfully matched features, calculate the initial attitude value using point features, and the point and line features adaptively determine the weights during fusion according to their respective observation accuracies to solve the attitude parameters of the camera; Among them, the weights of point and line features are the ratio of the unit weight variance to the observation value variance. The calculation methods of the unit weight variances of point and line features are shown in the following formula: In the formula N = N1 + N2 where \(n_1\) and \(n_2\) are the numbers of two types of observed values, \(B_1\) and \(B_2\) are the coefficient matrices of the observation equations, \(P_1\) and \(P_2\) are the weight matrices of the observed values, \(N_1\) and \(N_2\) are the normal equation matrices of the two types of observed values, \(tr()\) is the sum of the main diagonal elements of the matrix, is the estimated variance of unit weight for the two types of observed values, and \(V_1\) and \(V_2\) are the corrections of the observed values; The update method of the point and line feature weights is to estimate the variance components of the point and line features according to the above formula, and obtain the estimation of the unit weight variance of the point and line observations. Then update the weights according to the following formula: where: c is an arbitrary constant, generally selected as one of the values in , and k is the number of iterations; The fourth step: Output the final attitude result: Based on the measured control points, convert the attitude result to the world coordinate system, and thus complete the process of UAV positioning and attitude determination.

2. The method for positioning and attitude determination of UAV images based on adaptive fusion of point and line features according to claim 1, wherein: When using a UAV to obtain image data, POS information, and ground control point data in the first step, the working area of the UAV in the field should be determined according to specific task requirements. Control points should be arranged in the area and their three-dimensional coordinates in the world coordinate system should be measured. When the UAV collects data, plan the UAV flight path according to the task requirements, set the corresponding overlap and flight height parameters, and obtain the UAV image data within the working area and the GNSS coordinates at the moment of image shooting.

3. The method for UAV image positioning and attitude determination based on adaptive fusion of point-line features according to claim 1, wherein: When evaluating the shooting quality of the images, checking the data format, whether the metadata is correct, and whether the image overlap meets the requirements in the first step, the integrity of the control points and POS data should also be checked, and whether the control points are clearly and completely imaged on the images.

4. The method for UAV image positioning and attitude determination based on adaptive fusion of point-line features according to claim 1, characterized in that: In the third step, when calculating the initial attitude value using point features based on the successfully matched features, use the calculated initial attitude value to set the weights of point features and line features to 1, calculate the residuals between the observed values and predicted values of point and line features, and use the method of Helmert variance component estimation to calculate the updated weights of point and line features. Iterate the above process. When the unit weight variance components of point and line features are equal or nearly equal, it is considered that the weights of point and line features have reached a reasonably determined degree. When the difference between the unit weight variance components of point and line features is less than 1E-6, this method considers the two to be nearly equal.

5. The method for positioning and attitude determination of UAV images based on adaptive fusion of point-line features according to claim 1, characterized in that: When calculating the pose parameters of the camera by adaptively determining the weights during fusion according to the respective observation accuracies of the point and line features in the third step, an iterative solution method is used to improve the accuracy of pose calculation. The accuracy of the initial value directly affects the accuracy of the pose calculation result. Using the successfully matched point features, the two images with the largest number of matching points are selected as the initial image pair, and the relative pose parameters between the two images are calculated. Then, based on this, the pose of the next image is calculated using the local three-dimensional points generated by the matching pairs, and the above process is repeated to complete the pose calculation of all images. After calculating the poses of 3 to 4 images, a bundle adjustment is performed once to reasonably distribute the accumulated errors.

6. The method for positioning and attitude determination of UAV images based on adaptive fusion of point and line features according to claim 1, characterized in that: When calculating the pose parameters of the camera by adaptively determining the weights during fusion according to the respective observation accuracies of the point and line features in the third step, after obtaining the weights of the point feature and the line feature, the residual values are reasonably assigned to the pose increments according to the magnitudes of the weights, and the pose calculation result is the sum of the pose initial value and the pose increment value.

Images