An unmanned aerial vehicle downward-looking camera installation error online correction method

Abstract

This application relates to the field of image processing technology and discloses an online correction method for the installation error of a UAV's downward-looking camera. The method includes: acquiring a first image frame and a second image frame; obtaining a third set of matching points and a fourth set of matching points based on the first image frame, the second image frame, and a satellite map; obtaining the intersection of the third set of matching points and the fourth set of matching points, and performing uniformization processing on the intersection to obtain a fifth set of matching points; generating an elevation map; obtaining the three-dimensional coordinates of each feature point in the fifth set of matching points based on the pixel coordinates matched in the satellite map, and the corresponding satellite map and elevation map; and obtaining the attitude error by solving a least-squares problem, and compensating for the current attitude of the UAV using the attitude error. This application can calculate the installation error of the downward-looking camera in real time, thereby ensuring the positioning accuracy of the UAV during visual navigation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an online correction method for installation errors of a drone's downward-looking camera. Background Technology

[0002] In recent years, drone technology has developed rapidly and has been widely used in civilian and commercial fields. With the increase in the number and types of drones, higher requirements are being placed on their autonomous navigation and obstacle avoidance capabilities.

[0003] Computer vision technology has made significant progress in image processing, target recognition, and tracking. By using computer vision algorithms, drones can extract useful information, such as ground features, obstacles, and target objects, from images or videos acquired by airborne cameras. The airborne camera acts as the drone's eyes, perceiving and analyzing its surroundings to enable autonomous navigation and other functions.

[0004] Airborne cameras are commonly mounted in two ways: gimbal mounting and rigid body mounting. Gimbal mounting can improve image acquisition quality, but it will significantly increase system costs. The mechanical and electronic components of the gimbal will also reduce system stability and make it more prone to failure during use.

[0005] Rigid-body camera mounting requires no additional mechanical or electronic components, thus exhibiting extremely high system stability, but it places high demands on the camera's mounting precision. Summary of the Invention

[0006] In view of this, this application provides an online correction method for the installation error of a UAV downward-looking camera.

[0007] This application discloses an online correction method for the installation error of a UAV's downward-looking camera, which includes:

[0008] Step 1: Acquire an initial image frame captured by the camera carried by the drone during its flight and use it as the first image frame. After a preset time interval, acquire multiple image frames. Select the second image frame from the multiple image frames based on the pixel translation amount between each image frame and the first image frame.

[0009] Step 2: Perform feature matching on the first image frame and the second image frame to obtain the first set of matching points. Perform image feature matching on the second image frame and the satellite map to obtain the second set of matching points. Perform mismatch removal operation on the first set of matching points and the second set of matching points to obtain the third set of matching points and the fourth set of matching points.

[0010] Step 3: Obtain the intersection between the third and fourth matching point sets, and perform uniformization on the intersection to obtain the fifth matching point set;

[0011] Step 4: Based on the latitude and longitude of the corner points of the satellite map, obtain the number of tiles and their index numbers. Combine the number of tiles with the satellite map to generate an elevation map. Based on the pixel coordinates of each feature point in the fifth matching point set matched in the satellite map, as well as the corresponding satellite map and elevation map, obtain the three-dimensional coordinates of each feature point in the fifth matching point set. The three-dimensional coordinates include longitude, latitude, and altitude.

[0012] Step 5: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the three-axis attitude error, and use the three-axis attitude error as the quantity to be optimized to construct a least squares problem. By solving the least squares problem, obtain the attitude error, and use the attitude error to compensate for the current attitude of the UAV.

[0013] Further, step 1 includes:

[0014] Step 11: Using the feature matching method, the first image frame is matched with each of the multiple image frames to obtain the set of first pixel coordinates of all feature points in the first image frame, and the set of second pixel coordinates of all feature points in each image frame.

[0015] Step 12: Based on the first image frame captured by the camera on the UAV and the first heading angle and second heading angle corresponding to each image frame, rotate the pixel coordinates of all feature points in the first pixel coordinate set and the second pixel coordinate set to the same direction to obtain the third pixel coordinate set and the fourth pixel coordinate set respectively;

[0016] Step 13: Based on the third pixel coordinate set and the fourth pixel coordinate set, obtain the pixel translation amount of each image frame relative to the first image frame;

[0017] Step 14: Select a second image frame from multiple image frames based on the pixel translation amount of each image frame relative to the first image frame.

[0018] Further, step 12 includes:

[0019] The first heading angle corresponding to the first image frame and each of the multiple image frames captured by the camera on the drone. Second heading angle The pixel coordinates of all feature points in the first and second pixel coordinate sets are rotated to a uniform direction using the following formulas to obtain the third and fourth pixel coordinate sets respectively:

[0020] (1)

[0021] in, Let i be the coordinate of the i-th pixel in the third set of pixel coordinates. Let i be the coordinate of the i-th pixel in the fourth set of pixel coordinates. Let be the pixel coordinates of the i-th feature point in the first pixel coordinate set within the first image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively. Let i be the pixel coordinates of the i-th feature point in the second set of pixel coordinates in each image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively;

[0022] Step 13 includes:

[0023] according to and The pixel shift of each of the multiple image frames relative to the first image frame can be obtained using the following formula:

[0024] (2)

[0025] in, The pixel shift for each image frame relative to the first image frame. The number of feature points that were successfully matched;

[0026] Step 14 includes:

[0027] Each of the multiple image frames is shifted relative to the first image frame by a pixel shift amount and a preset threshold. Compare and select pixel shift amount Greater than the preset threshold The image frame is used as the second keyframe.

[0028] Further, step 3 includes:

[0029] Obtain a binary index map of the same size as the second image frame, where the initial value of all pixels in the binary index map is the first value;

[0030] A pixel coordinate group is established for each feature point. The pixel coordinate group includes the pixel coordinates of the feature point in the first image frame, the pixel coordinates in the second image frame, and the pixel coordinates in the satellite map.

[0031] Iterate through each pixel coordinate group in the intersection of the third and fourth matching point sets. If the pixel value of the binary indexed image at the pixel coordinate of the second image frame is the first value, then construct a corresponding square region centered on the pixel coordinate of the second image frame with sides of a preset width, and set all pixel values ​​in the square region of the binary indexed image to the second value. If the pixel value of the binary indexed image at the pixel coordinate of the second image frame is the second value, then delete the pixel coordinate group corresponding to the pixel coordinate of the second image frame in the intersection.

[0032] After traversing all pixel coordinate sets in the intersection of the third and fourth matching point sets, the fifth matching point set is obtained.

[0033] Further, step 4 includes:

[0034] Step 41: Based on the latitude and longitude of the upper left and lower right corners of the satellite map, obtain the tile index number and the number of tiles. Based on the tile index number and the number of tiles, stitch the tiles onto the satellite map to obtain the elevation map.

[0035] Step 42: Based on the pixel coordinates of each feature point in the satellite map in the fifth matching point set, obtain the longitude and latitude coordinates of each feature point in the fifth matching point set;

[0036] Step 43: Based on the longitude and latitude coordinates of each feature point in the fifth matching point set, obtain the altitude coordinates of each feature point in the fifth matching point set.

[0037] Further, step 41 includes:

[0038] Based on the latitude and longitude of the top left corner of the satellite map Latitude and longitude of the bottom right corner point The tile index number is calculated using the following formula:

[0039] (6)

[0040] in, The longitude tile index number of the top left corner point on the satellite map. This is the longitude tile index number of the point in the lower right corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This is the floor operator. The longitude of the point in the upper left corner of the satellite map. The latitude of the top left corner of the satellite map. The longitude of the point in the lower right corner of the satellite map. The latitude of the point in the lower right corner of the satellite map. For elevation map tile levels;

[0041] In the x-axis direction to y-axis direction to The number of tiles within the area formed is By piecing together the tiles corresponding to the specified number, an elevation map can be obtained. ;

[0042] Step 42 includes:

[0043] Based on the pixel coordinates of the kth feature point in the fifth matching point set in the satellite map The longitude and latitude of the k-th feature point in the fifth matching point set on the satellite map are obtained using the following formula:

[0044] (7)

[0045] in, Let k be the longitude of the k-th feature point in the fifth matching point set on the satellite map. Let k be the latitude of the k-th feature point in the fifth set of matching points on the satellite map. For satellite map resolution, The current latitude of the drone;

[0046] Based on the pixel coordinates of the kth feature point in the satellite map, the longitude and latitude of the kth feature point in the fifth matching point set are obtained using the following formula:

[0047] (8)

[0048] (9)

[0049] in, Rotate the satellite map around the center The pixel coordinates x after the angle Rotate the satellite map around the center The pixel coordinates y after the angle, The u-axis pixel coordinates of the feature point matched in the satellite map for the k-th feature point in the fifth matching point set. Let v-axis pixel coordinates be the feature point in the satellite map that matches the k-th feature point in the fifth matching point set. This refers to the number of pixels in the width direction of the satellite map. This represents the number of pixels in the height direction of the satellite map. Here are the longitude coordinates of the k-th feature point in the fifth matching point set. Let k be the latitude coordinates of the k-th feature point in the fifth matching point set. The current longitude of the drone. This is the scaling factor;

[0050] Step 43 includes:

[0051] according to , and The height of the k-th feature point in the fifth matching point set can be obtained using the following formula:

[0052] (10) (11)

[0053] in, The longitude tile index number of the k-th feature point. The latitude tile index number of the k-th feature point. For high-resolution elevation maps, The height of the k-th feature point in the fifth matching point set.

[0054] Further, step 5 includes:

[0055] Step 51: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the pixel coordinates of each feature point in the first image frame and the pixel coordinates in the second image frame.

[0056] Step 52: Reproject the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame to the first image frame and the second image frame to obtain the first projected pixel coordinates and the second projected pixel coordinates.

[0057] Step 53: Based on the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame, as well as the first projected pixel coordinates and the second projected pixel coordinates, obtain the three-axis pose error.

[0058] Step 54: With the goal of minimizing the three-axis attitude error, construct a least squares problem, use the LM algorithm to solve for the attitude angle to be optimized, and obtain the optimized attitude angle. Based on the attitude angle before optimization and the optimized attitude angle, obtain the camera mounting angle error.

[0059] Further, step 51 includes:

[0060] Step 511: Convert the three-dimensional coordinates of each feature point in the fifth matching point set to geocentric rectangular coordinates;

[0061] Step 512: Based on the three-dimensional coordinates of the feature points in the first and second image frames of the fifth matching point set, obtain the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system;

[0062] Step 513: Based on the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system, and the geocentric rectangular coordinates, obtain the coordinates of each feature point in the fifth matching point set in the camera coordinate system.

[0063] Further, step 511 includes:

[0064] The latitude, longitude, and altitude coordinates of the kth feature point in the fifth matching point set. Convert to geocentric rectangular coordinates ;

[0065] (12)

[0066] in, The semi-major axis of the Earth's ellipse. Let N be the eccentricity of the Earth's ellipse, and N be the radius of curvature of the prime and trochanter.

[0067] Step 512 includes:

[0068] Based on the latitude, longitude, and altitude coordinates of the feature points in the fifth matching point set in the first and second image frames, the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system is obtained; the relative transformation relationship includes the attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when the first image frame was captured. The attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the second image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. ;

[0069] Step 513 includes:

[0070] The coordinates of each feature point in the fifth matching point set in the camera coordinate system, in the first and second image frames, are obtained using the following formula:

[0071] (13)

[0072] in, Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system used to capture the first image frame. Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system of the second image frame. This represents the total number of feature points in the fifth matching point set.

[0073] Further, step 52 includes:

[0074] The coordinates of the first and second projected pixels are obtained using the following formula:

[0075] (14)

[0076] in, and Let the k-th feature point in the fifth matching point set be the first projected pixel coordinates and the second projected pixel coordinates on the first image frame and the second image frame, respectively. This is the camera intrinsic parameter matrix;

[0077] Step 53 includes:

[0078] The three-axis attitude error can be obtained using the following formula:

[0079] (15)

[0080] in, For three-axis attitude error, Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the first image frame. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the second image frame. Symbols represent operations for finding the L2 norm;

[0081] Step 54 includes:

[0082] To minimize three-axis attitude error To achieve the desired outcome, a least-squares problem is constructed, and the LM algorithm is used to solve for the attitude angles to be optimized, yielding the optimized attitude angles. The attitude angles include the yaw, pitch, and roll angles. The LM algorithm is used to calculate the required Jacobian matrix. , , , , , ;

[0083] The LM algorithm terminates its optimization after reaching the expected termination condition. (Previous attitude angles are shown.) The difference between the optimized attitude angle and the actual angle is the camera mounting angle error.

[0084] (2)

[0085] in, and These are the heading angles of the first image frame before and after optimization, respectively. and These are the heading angles of the second image frames before and after optimization, respectively. and These are the pitch angles of the first image frame before and after optimization, respectively. and These are the pitch angles of the second image frame before and after optimization, respectively. and These are the roll angles of the first image frame before and after optimization. and These are the roll angles of the second image frames before and after optimization. For three-axis attitude error, , , These are the heading angle error, pitch angle error, and roll angle error, respectively.

[0086] Due to the adoption of the above technical solution, this application has the following advantages: This application can select image feature points with strong stability online during flight, and combine them with information such as the UAV's pose, images, satellite maps, and elevation maps to calculate and correct the installation error of the downward-looking camera in real time, thereby ensuring the positioning accuracy of the UAV during visual navigation. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0088] Figure 1 This is a flowchart illustrating an online correction method for the installation error of a UAV downward-looking camera according to an embodiment of this application.

[0089] Figure 2 This is a schematic diagram of a visual navigation estimation trajectory without installation error correction according to an embodiment of this application.

[0090] Figure 3 This is a schematic diagram of the visual navigation estimation trajectory for installation error correction according to an embodiment of this application. Detailed Implementation

[0091] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0092] See Figure 1This application provides an embodiment of an online correction method for the installation error of a UAV downward-looking camera, which includes:

[0093] Step 1: Acquire an initial image frame captured by the camera carried by the drone during its flight and use it as the first image frame. After a preset time interval, acquire multiple image frames. Select the second image frame from the multiple image frames based on the pixel translation amount between each image frame and the first image frame.

[0094] Step 2: Perform feature matching on the first image frame and the second image frame to obtain the first set of matching points. Perform image feature matching on the second image frame and the satellite map to obtain the second set of matching points. Perform mismatch removal operation on the first set of matching points and the second set of matching points to obtain the third set of matching points and the fourth set of matching points.

[0095] Step 3: Obtain the intersection between the third and fourth matching point sets, and perform uniformization on the intersection to obtain the fifth matching point set;

[0096] Step 4: Based on the latitude and longitude of the corner points of the satellite map, obtain the number of tiles and their index numbers. Combine the number of tiles with the satellite map to generate an elevation map. Based on the pixel coordinates of each feature point in the fifth matching point set matched in the satellite map, as well as the corresponding satellite map and elevation map, obtain the three-dimensional coordinates of each feature point in the fifth matching point set. The three-dimensional coordinates include longitude, latitude, and altitude.

[0097] Step 5: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the three-axis attitude error, and use the three-axis attitude error as the quantity to be optimized to construct a least squares problem. By solving the least squares problem, obtain the attitude error, and use the attitude error to compensate for the current attitude of the UAV.

[0098] In this embodiment of the application, step 1 includes:

[0099] Step 11: Using the feature matching method, the first image frame is matched with each of the multiple image frames to obtain the set of first pixel coordinates of all feature points in the first image frame, and the set of second pixel coordinates of all feature points in each image frame.

[0100] Step 12: Based on the first image frame captured by the camera on the UAV and the first heading angle and second heading angle corresponding to each image frame, rotate the pixel coordinates of all feature points in the first pixel coordinate set and the second pixel coordinate set to the same direction to obtain the third pixel coordinate set and the fourth pixel coordinate set respectively;

[0101] Step 13: Based on the third pixel coordinate set and the fourth pixel coordinate set, obtain the pixel translation amount of each image frame relative to the first image frame;

[0102] Step 14: Select a second image frame from multiple image frames based on the pixel translation amount of each image frame relative to the first image frame.

[0103] Optionally, in step 11, since the first image frame needs to be matched with multiple image frames multiple times, it is necessary to ensure that the matched feature points have a high accuracy. Therefore, the SURF (Speed ​​Up Robust Feature) feature matching method is used. The SURF feature matching method can simultaneously take into account both computational speed and accuracy.

[0104] In this embodiment of the application, step 12 includes:

[0105] The first heading angle corresponding to the first image frame and each of the multiple image frames captured by the camera on the drone. Second heading angle The pixel coordinates of all feature points in the first and second pixel coordinate sets are rotated to a uniform direction using the following formulas to obtain the third and fourth pixel coordinate sets respectively:

[0106] (1)

[0107] in, Let i be the coordinate of the i-th pixel in the third set of pixel coordinates. Let i be the coordinate of the i-th pixel in the fourth set of pixel coordinates. Let be the pixel coordinates of the i-th feature point in the first pixel coordinate set within the first image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively. Let i be the pixel coordinates of the i-th feature point in the second set of pixel coordinates in each image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively;

[0108] Step 13 includes:

[0109] according to and The pixel shift of each of the multiple image frames relative to the first image frame can be obtained using the following formula:

[0110] (2)

[0111] in, The pixel shift for each image frame relative to the first image frame. The number of feature points that were successfully matched;

[0112] Step 14 includes:

[0113] Each of the multiple image frames is shifted relative to the first image frame by a pixel shift amount and a preset threshold. Compare and select pixel shift amount Greater than the preset threshold The image frame is used as the second keyframe.

[0114] Alternatively, based on engineering experience, The value is one-quarter of the image's narrow side resolution, for example. Image resolution, value .

[0115] Optionally, in step 2, the process described in the adaptive satellite image generation method for UAV visual positioning published in CN114201633A can be used to stitch satellite map tiles of the adjacent area in real time according to the current GNSS latitude and longitude position and heading angle of the UAV. After cropping, scaling and rotating, an adaptive satellite map similar to the actual image captured by the downward-looking camera is generated. The satellite map generated based on this method can significantly improve the number and accuracy of feature matching.

[0116] Optionally, in step 2, unlike the coarse matching in step 1, the formal image feature matching needs to ensure that the number of successfully matched feature points is as large as possible and that the accuracy is high. Image feature matching is performed between the second camera frame selected in step 1 and the adaptive satellite map generated in step 2 to obtain a second set of matching points. Since the camera-captured image and the satellite map are heterogeneous images, they often differ significantly in color, lighting, and scenery. Traditional feature matching methods struggle to obtain a large number of stable and reliable matching points. SuperPoint feature extraction and SuperGlue feature matching based on deep learning methods can maximize the number of feature points and the accuracy, and also exhibits extremely high generalization ability across different flight scenarios.

[0117] The first and second image frames are from the same source, and feature matching between them is relatively easy. Therefore, the traditional KNN (k-nearest neighbor) feature matching method can be used to speed up the operation. However, in order to ensure the consistency of feature points, the first image frame still needs to use SuperPoint for feature extraction.

[0118] After feature matching is completed, to further improve the accuracy of feature matching, a mismatch removal operation is also required. Based on the feature mismatch removal method for aerial images of UAV systems and the process described in the system (publication number CN114998773A), the matching points are filtered. Let the final filtered matching point sets, namely the third matching point set and the fourth matching point set, be:

[0119] (3)

[0120] Among them, superscript and These represent the first set of matching points and the second set of matching points, respectively. For the first set of matching points, the first The pixel coordinates of each feature point in the first image frame For the first set of matching points, the first The pixel coordinates of the feature points in the second image frame For the second set of matching points, the first The pixel coordinates of the feature points in the second image frame For the second set of matching points, the first The pixel coordinates of each feature point in the satellite map and These represent the number of feature points in the first set of matching points and the second set of matching points, respectively.

[0121] Feature points in the second image frame that are simultaneously matched by the first image frame and the satellite map, i.e., satisfying The feature points are common feature points. We take these common feature points to form a new set, which is the intersection of the third and fourth matching point sets:

[0122] (3)

[0123] In some flight scenarios, feature points may be concentrated in a local area of ​​the image, which can lead to incorrect estimations of the drone's attitude by the algorithm. Therefore, it is necessary to homogenize the matching point set 2 in step 3. The specific method is as follows:

[0124] Step 3 includes:

[0125] Obtain a binary index map of the same size as the second image frame, where the initial value of all pixels in the binary index map is the first value;

[0126] A pixel coordinate group is established for each feature point. The pixel coordinate group includes the pixel coordinates of the feature point in the first image frame, the pixel coordinates in the second image frame, and the pixel coordinates in the satellite map.

[0127] Iterate through each pixel coordinate group in the intersection of the third and fourth matching point sets. If the pixel value of the binary indexed image at the pixel coordinate of the second image frame is the first value, then construct a corresponding square region centered on the pixel coordinate of the second image frame with sides of a preset width, and set all pixel values ​​in the square region of the binary indexed image to the second value. If the pixel value of the binary indexed image at the pixel coordinate of the second image frame is the second value, then delete the pixel coordinate group corresponding to the pixel coordinate of the second image frame in the intersection.

[0128] After traversing all pixel coordinate sets in the intersection of the third and fourth matching point sets, the fifth matching point set is obtained.

[0129] For example, a binary indexed map of the same size as the second image frame is created. Index diagram The initial values ​​are all 0.

[0130] Traversing the set of common points If a binary indexed graph In coordinates If the value is 0, then the coordinates will be... Centered on a square area with a width of 15 pixels, all pixel values ​​are set to 255. (This is from a binary indexed image.) In coordinates If the value is 255, then from the common point set Delete the feature point.

[0131] The set of matching points after homogenization is called the stable set of matching points (the fifth set of matching points), denoted as:

[0132] (5)

[0133] If a stable matching point set Number of feature points Minimum threshold not met If so, return to step 1 to reselect a camera frame. Threshold The selection should be based on the image resolution, according to engineering experience. Image resolution values .

[0134] The intersection operation of matching points can extract feature points that appear simultaneously in satellite map matching and camera frame matching. These feature points have high stability and are more likely to represent actual terrain features rather than general pixel grayscale features. Matching point homogenization can effectively improve the accuracy and reliability of subsequent pose calculation.

[0135] In this embodiment of the application, step 4 includes:

[0136] Step 41: Based on the latitude and longitude of the upper left and lower right corners of the satellite map, obtain the tile index number and the number of tiles. Based on the tile index number and the number of tiles, stitch the tiles onto the satellite map to obtain the elevation map.

[0137] Step 42: Based on the pixel coordinates of each feature point in the satellite map in the fifth matching point set, obtain the longitude and latitude coordinates of each feature point in the fifth matching point set;

[0138] Step 43: Based on the longitude and latitude coordinates of each feature point in the fifth matching point set, obtain the altitude coordinates of each feature point in the fifth matching point set.

[0139] In this embodiment of the application, step 41 includes:

[0140] Based on the latitude and longitude of the top left corner of the satellite map Latitude and longitude of the bottom right corner point The tile index number is calculated using the following formula:

[0141] (6)

[0142] in, The longitude tile index number of the top left corner point on the satellite map. This is the longitude tile index number of the point in the lower right corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This is the floor operator. The longitude of the point in the upper left corner of the satellite map. The latitude of the top left corner of the satellite map. The longitude of the point in the lower right corner of the satellite map. The latitude of the point in the lower right corner of the satellite map. For elevation map tile levels;

[0143] In the x-axis direction to y-axis direction to The number of tiles within the area formed is By piecing together the tiles corresponding to the specified number, an elevation map can be obtained. For example, the tiles of an elevation map can be... A TIFF image of a specific size;

[0144] Step 42 includes:

[0145] Based on the pixel coordinates of the kth feature point in the fifth matching point set in the satellite map The longitude and latitude of the k-th feature point in the fifth matching point set on the satellite map are obtained using the following formula:

[0146] (7)

[0147] in, Let k be the longitude of the k-th feature point in the fifth matching point set on the satellite map. Let k be the latitude of the k-th feature point in the fifth set of matching points on the satellite map. For satellite map resolution, This is the current latitude of the drone; when the satellite map layer is fixed, the satellite map resolution is a fixed value, which can be obtained by looking up a table;

[0148] Based on the pixel coordinates of the kth feature point in the satellite map, the longitude and latitude of the kth feature point in the fifth matching point set are obtained using the following formula:

[0149] (8)

[0150] (9)

[0151] in, Rotate the satellite map around the center The pixel coordinates x after the angle Rotate the satellite map around the center The pixel coordinates y after the angle, The u-axis pixel coordinates of the feature point matched in the satellite map for the k-th feature point in the fifth matching point set. Let v-axis pixel coordinates be the feature point in the satellite map that matches the k-th feature point in the fifth matching point set. This refers to the number of pixels in the width direction of the satellite map. This represents the number of pixels in the height direction of the satellite map. Here are the longitude coordinates of the k-th feature point in the fifth matching point set. Let k be the latitude coordinates of the k-th feature point in the fifth matching point set. The current longitude of the drone. This is the scaling factor;

[0152] Step 43 includes:

[0153] according to , and The height of the k-th feature point in the fifth matching point set can be obtained using the following formula:

[0154] (10) (11)

[0155] in, The longitude tile index number of the k-th feature point. The latitude tile index number of the k-th feature point. For high-resolution elevation maps, The height of the k-th feature point in the fifth matching point set.

[0156] In this embodiment of the application, step 5 includes:

[0157] Step 51: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the pixel coordinates of each feature point in the first image frame and the pixel coordinates in the second image frame.

[0158] Step 52: Reproject the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame to the first image frame and the second image frame to obtain the first projected pixel coordinates and the second projected pixel coordinates.

[0159] Step 53: Based on the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame, as well as the first projected pixel coordinates and the second projected pixel coordinates, obtain the three-axis pose error.

[0160] Step 54: With the goal of minimizing the three-axis attitude error, construct a least squares problem, use the LM algorithm to solve for the attitude angle to be optimized, and obtain the optimized attitude angle. Based on the attitude angle before optimization and the optimized attitude angle, obtain the camera mounting angle error.

[0161] In this embodiment of the application, step 51 includes:

[0162] Step 511: Convert the three-dimensional coordinates of each feature point in the fifth matching point set to geocentric rectangular coordinates;

[0163] Step 512: Based on the three-dimensional coordinates of the feature points in the first and second image frames of the fifth matching point set, obtain the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system;

[0164] Step 513: Based on the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system, and the geocentric rectangular coordinates, obtain the coordinates of each feature point in the fifth matching point set in the camera coordinate system.

[0165] In this embodiment of the application, step 511 includes:

[0166] The latitude, longitude, and altitude coordinates of the kth feature point in the fifth matching point set. Convert to geocentric rectangular coordinates ;

[0167] (12)

[0168] in, The semi-major axis of the Earth's ellipse. Let N be the eccentricity of the Earth's ellipse, and N be the radius of curvature of the prime mover; for example... , ;

[0169] Step 512 includes:

[0170] Based on the latitude, longitude, and altitude coordinates of the feature points in the fifth matching point set in the first and second image frames, the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system is obtained; the relative transformation relationship includes the attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when the first image frame was captured. The attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the second image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. ;

[0171] Step 513 includes:

[0172] The coordinates of each feature point in the fifth matching point set in the camera coordinate system, in the first and second image frames, are obtained using the following formula:

[0173] (13)

[0174] in, Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system used to capture the first image frame. Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system of the second image frame. This represents the total number of feature points in the fifth matching point set. (Superscript) , These represent the camera coordinate systems corresponding to the first and second image frames, respectively.

[0175] In this embodiment of the application, step 52 includes:

[0176] The coordinates of the first and second projected pixels are obtained using the following formula:

[0177] (14)

[0178] in, and Let the kth feature point in the fifth matching point set be the first projected pixel coordinates and the second projected pixel coordinates on the first image frame and the second image frame, respectively. This is the camera intrinsic parameter matrix, which can be obtained through prior calibration;

[0179] Step 53 includes:

[0180] The three-axis attitude error can be obtained using the following formula:

[0181] (15)

[0182] in, For three-axis attitude error, Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the first image frame. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the second image frame. Symbols represent operations for finding the L2 norm;

[0183] Step 54 includes:

[0184] To minimize three-axis attitude error To achieve the desired outcome, a least-squares problem is constructed, and the LM algorithm is used to solve for the attitude angles to be optimized, yielding the optimized attitude angles. The attitude angles include the yaw, pitch, and roll angles. The LM algorithm is used to calculate the required Jacobian matrix. , , , , , ;

[0185] The LM algorithm terminates its optimization after reaching the expected termination condition. (Previous attitude angles are shown.) The difference between the optimized attitude angle and the actual angle is the camera mounting angle error.

[0186] (3)

[0187] in, and These are the heading angles of the first image frame before and after optimization, respectively. and These are the heading angles of the second image frames before and after optimization, respectively. and These are the pitch angles of the first image frame before and after optimization, respectively. and These are the pitch angles of the second image frame before and after optimization, respectively. and These are the roll angles of the first image frame before and after optimization. and These are the roll angles of the second image frames before and after optimization. For three-axis attitude error, , , These are the heading angle error, pitch angle error, and roll angle error, respectively.

[0188] Figure 2 The green line represents the relatively straight GNSS trajectory, while the red line represents the visual navigation estimated trajectory. It is evident that the visual navigation estimated position without installation error correction exhibits a significant deviation (as indicated by the arrows in the diagram). With installation error correction, the visual navigation estimated position largely coincides with the actual GNSS trajectory. Figure 3 As shown, this demonstrates that the method described in this application is effective.

[0189] This application ensures the positioning accuracy of the UAV during visual navigation by online screening of image feature points with strong stability during flight and combining information such as the UAV's GNSS position, attitude, satellite map, and elevation map to calculate the installation error of the downward-looking camera in real time.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A method for online correction of installation error of a UAV downward-looking camera, characterized in that, include: Step 1: Acquire an initial image frame captured by the camera carried by the drone during its flight and use it as the first image frame. After a preset time interval, acquire multiple image frames. Select the second image frame from the multiple image frames based on the pixel translation amount between each image frame and the first image frame. Step 2: Perform feature matching on the first image frame and the second image frame to obtain the first set of matching points. Perform image feature matching on the second image frame and the satellite map to obtain the second set of matching points. Perform mismatch removal operation on the first set of matching points and the second set of matching points to obtain the third set of matching points and the fourth set of matching points. Step 3: Obtain the intersection between the third and fourth matching point sets, and perform uniformization on the intersection to obtain the fifth matching point set; Step 4: Based on the latitude and longitude of the corner points of the satellite map, obtain the number of tiles and their index numbers. Combine the number of tiles with the satellite map to generate an elevation map. Based on the pixel coordinates of each feature point in the fifth matching point set matched in the satellite map, as well as the corresponding satellite map and elevation map, obtain the three-dimensional coordinates of each feature point in the fifth matching point set. The three-dimensional coordinates include longitude, latitude, and altitude. Step 5: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the three-axis attitude error, and use the three-axis attitude error as the quantity to be optimized to construct a least squares problem. By solving the least squares problem, obtain the attitude error, and use the attitude error to compensate for the current attitude of the UAV. Step 1 includes: Step 11: Using the feature matching method, the first image frame is matched with each of the multiple image frames to obtain the set of first pixel coordinates of all feature points in the first image frame, and the set of second pixel coordinates of all feature points in each image frame. Step 12: Based on the first image frame captured by the camera on the UAV and the first heading angle and second heading angle corresponding to each image frame, rotate the pixel coordinates of all feature points in the first pixel coordinate set and the second pixel coordinate set to the same direction to obtain the third pixel coordinate set and the fourth pixel coordinate set respectively. Step 13: Based on the third pixel coordinate set and the fourth pixel coordinate set, obtain the pixel translation amount of each image frame relative to the first image frame; Step 14: Select a second image frame from multiple image frames based on the pixel translation amount of each image frame relative to the first image frame; Step 12 includes: The first heading angle corresponding to the first image frame and each of the multiple image frames captured by the camera on the drone. Second heading angle The pixel coordinates of all feature points in the first and second pixel coordinate sets are rotated to a uniform direction using the following formulas to obtain the third and fourth pixel coordinate sets respectively: (1) in, Let i be the coordinate of the i-th pixel in the third set of pixel coordinates. Let i be the coordinate of the i-th pixel in the fourth set of pixel coordinates. Let be the pixel coordinates of the i-th feature point in the first pixel coordinate set within the first image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively. Let i be the pixel coordinates of the i-th feature point in the second set of pixel coordinates in each image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively; Step 13 includes: according to and The pixel shift of each of the multiple image frames relative to the first image frame can be obtained using the following formula: (2) in, The pixel shift for each image frame relative to the first image frame. The number of feature points that were successfully matched; Step 14 includes: Each of the multiple image frames is shifted relative to the first image frame by a pixel shift amount and a preset threshold. Compare and select pixel shift amount Greater than the preset threshold The image frame is used as the second keyframe; Step 3 includes: Obtain a binary index map of the same size as the second image frame, where the initial value of all pixels in the binary index map is the first value; A pixel coordinate group is established for each feature point. The pixel coordinate group includes the pixel coordinates of the feature point in the first image frame, the pixel coordinates in the second image frame, and the pixel coordinates in the satellite map. Iterate through each pixel coordinate group in the intersection of the third and fourth matching point sets. If the pixel value of the binary indexed image at the pixel coordinate of the second image frame is the first value, then construct a corresponding square region with the pixel coordinate of the second image frame as the center and a side of a preset width as the square, and set all pixel values ​​in the square region of the binary indexed image to the second value; if the pixel value of the binary indexed image at the pixel coordinate of the second image frame is the second value, then delete the pixel coordinate group corresponding to the pixel coordinate of the second image frame in the intersection. After traversing all pixel coordinate sets in the intersection of the third and fourth matching point sets, the fifth matching point set is obtained. Step 4 includes: Step 41: Based on the latitude and longitude of the upper left and lower right corners of the satellite map, obtain the tile index number and the number of tiles. Based on the tile index number and the number of tiles, stitch the tiles onto the satellite map to obtain the elevation map. Step 42: Based on the pixel coordinates of each feature point in the satellite map in the fifth matching point set, obtain the longitude and latitude coordinates of each feature point in the fifth matching point set; Step 43: Based on the longitude and latitude coordinates of each feature point in the fifth matching point set, obtain the altitude coordinates of each feature point in the fifth matching point set; Step 41 includes: Based on the latitude and longitude of the top left corner of the satellite map Latitude and longitude of the bottom right corner point The tile index number is calculated using the following formula: (6) in, The longitude tile index number of the top left corner point on the satellite map. This is the longitude tile index number of the point in the lower right corner of the satellite map. This is the latitude tile index number of the point in the upper left corner of the satellite map. This is the latitude tile index number of the point in the upper left corner of the satellite map. This is the floor operator. The longitude of the point in the upper left corner of the satellite map. The latitude of the top left corner of the satellite map. The longitude of the point in the lower right corner of the satellite map. The latitude of the point in the lower right corner of the satellite map. For elevation map tile levels; In the x-axis direction to y-axis direction to The number of tiles within the area formed is By piecing together the tiles corresponding to the specified number, an elevation map can be obtained. ; Step 42 includes: Based on the pixel coordinates of the kth feature point in the fifth matching point set in the satellite map The longitude and latitude of the k-th feature point in the fifth matching point set on the satellite map are obtained using the following formula: (7) in, Let k be the longitude of the k-th feature point in the fifth matching point set on the satellite map. Let k be the latitude of the k-th feature point in the fifth set of matching points on the satellite map. For satellite map resolution, The current latitude of the drone; Based on the pixel coordinates of the kth feature point in the satellite map, the longitude and latitude of the kth feature point in the fifth matching point set are obtained using the following formula: (8) (9) in, Rotate the satellite map around the center The pixel coordinates x after the angle Rotate the satellite map around the center The pixel coordinates y after the angle, The u-axis pixel coordinates of the feature point matched in the satellite map for the k-th feature point in the fifth matching point set. Let v-axis pixel coordinates be the feature point in the satellite map that matches the k-th feature point in the fifth matching point set. This refers to the number of pixels in the width direction of the satellite map. This represents the number of pixels in the height direction of the satellite map. Here are the longitude coordinates of the k-th feature point in the fifth matching point set. Let k be the latitude coordinates of the k-th feature point in the fifth matching point set. The current longitude of the drone. This is the scaling factor; Step 43 includes: according to , and The height of the k-th feature point in the fifth matching point set can be obtained using the following formula: (10) (11) in, The longitude tile index number of the k-th feature point. The latitude tile index number of the k-th feature point. For high-resolution elevation maps, The height of the kth feature point in the fifth matching point set; Step 5 includes: Step 51: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the pixel coordinates of each feature point in the first image frame and the pixel coordinates in the second image frame. Step 52: Reproject the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame to the first image frame and the second image frame to obtain the first projected pixel coordinates and the second projected pixel coordinates. Step 53: Based on the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame, as well as the first projected pixel coordinates and the second projected pixel coordinates, obtain the three-axis pose error. Step 54: With the goal of minimizing the three-axis attitude error, construct a least squares problem, use the LM algorithm to solve for the attitude angle to be optimized, and obtain the optimized attitude angle. Based on the attitude angle before optimization and the optimized attitude angle, obtain the camera mounting angle error. Step 52 includes: The coordinates of the first and second projected pixels are obtained using the following formula: (14) in, and Let the k-th feature point in the fifth matching point set be the first projected pixel coordinates and the second projected pixel coordinates on the first image frame and the second image frame, respectively. This is the camera intrinsic parameter matrix; Step 53 includes: The three-axis attitude error can be obtained using the following formula: (15) in, This refers to the three-axis attitude error. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the first image frame. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the second image frame. Symbols represent operations for finding the 2-norm; Step 54 includes: To minimize three-axis attitude error To achieve the desired outcome, a least-squares problem is constructed, and the LM algorithm is used to solve for the attitude angles to be optimized, yielding the optimized attitude angles. The attitude angles include the yaw, pitch, and roll angles. The LM algorithm is used to calculate the required Jacobian matrix. , , , , , ; The LM algorithm terminates its optimization after reaching the expected termination condition. (Previous attitude angles are shown.) The difference between the optimized attitude angle and the actual angle is the camera mounting angle error. (16) in, and These are the heading angles of the first image frame before and after optimization, respectively. and These are the heading angles of the second image frames before and after optimization, respectively. and These are the pitch angles of the first image frame before and after optimization, respectively. and These are the pitch angles of the second image frame before and after optimization, respectively. and These are the roll angles of the first image frame before and after optimization. and These are the roll angles of the second image frames before and after optimization. This refers to the three-axis attitude error. , , These are the heading angle error, pitch angle error, and roll angle error, respectively.

2. The method according to claim 1, characterized in that, Step 51 includes: Step 511: Convert the three-dimensional coordinates of each feature point in the fifth matching point set to geocentric rectangular coordinates; Step 512: Based on the three-dimensional coordinates of the feature points in the first and second image frames of the fifth matching point set, obtain the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system; Step 513: Based on the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system, and the geocentric rectangular coordinates, obtain the coordinates of each feature point in the fifth matching point set in the camera coordinate system.

3. The method according to claim 2, characterized in that, Step 511 includes: The latitude, longitude, and altitude coordinates of the kth feature point in the fifth matching point set. Convert to geocentric rectangular coordinates ; (12) in, The semi-major axis of the Earth's ellipse. Let N be the eccentricity of the Earth's ellipse, and N be the radius of curvature of the prime and trochanter. Step 512 includes: Based on the latitude, longitude, and altitude coordinates of the feature points in the fifth matching point set in the first and second image frames, the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system is obtained; the relative transformation relationship includes the attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when the first image frame was captured. The attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the second image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. ; Step 513 includes: The coordinates of each feature point in the fifth matching point set in the camera coordinate system, in the first and second image frames, are obtained using the following formula: (13) in, Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system used to capture the first image frame. Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system of the second image frame. This represents the total number of feature points in the fifth matching point set.

Images