Online correction method for installation error of downward-looking camera of unmanned aerial vehicle

Abstract

The invention relates to the technical field of image processing, and discloses an unmanned aerial vehicle downward-looking camera installation error online correction method comprising the following steps: obtaining a first image frame and a second image frame; obtaining a third matching point set and a fourth matching point set according to the first image frame, the second image frame and a satellite map; acquiring an intersection between the third matching point set and the fourth matching point set, and homogenizing the intersection to obtain a fifth matching point set; generating an elevation map, and obtaining the three-dimensional coordinates of each feature point in the fifth matching point set according to the matched pixel coordinates of each feature point in the fifth matching point set in the satellite map, the corresponding satellite map and the elevation map; and solving a least square problem to obtain an attitude error, and compensating the current attitude of the unmanned aerial vehicle through the attitude error. According to the invention, the installation error of the down-looking camera can be calculated in real time, so that the positioning precision of the unmanned aerial vehicle during visual navigation is guaranteed.

Description

An Online Correction Method for Installation Errors of UAV Downward-Looking Cameras Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an online correction method for installation errors of a drone's downward-looking camera. Background Technology

[0002] In recent years, drone technology has developed rapidly and has been widely used in civilian and commercial fields. With the increase in the number and types of drones, higher requirements are being placed on their autonomous navigation and obstacle avoidance capabilities.

[0003] Computer vision technology has made significant progress in image processing, target recognition, and tracking. By using computer vision algorithms, drones can extract useful information, such as ground features, obstacles, and target objects, from images or videos acquired by airborne cameras. The airborne camera acts as the drone's eyes, perceiving and analyzing its surroundings to enable autonomous navigation and other functions.

[0004] Airborne cameras are commonly mounted in two ways: gimbal mounting and rigid body mounting. Gimbal mounting can improve image acquisition quality, but it will significantly increase system costs. The mechanical and electronic components of the gimbal will also reduce system stability and make it more prone to failure during use.

[0005] Rigid-body camera mounting requires no additional mechanical or electronic components, thus exhibiting extremely high system stability, but it places high demands on the camera's mounting precision. Summary of the Invention

[0006] In view of this, this application provides an online correction method for the installation error of a UAV downward-looking camera.

[0007] This application discloses an online correction method for the installation error of a UAV's downward-looking camera, comprising: Step 1: acquiring an initial image frame captured by the camera carried by the UAV during flight and using it as the first image frame; after a preset time interval, acquiring multiple image frames; and selecting a second image frame from the multiple image frames based on the pixel translation amount between each image frame and the first image frame; Step 2: performing feature matching on the first and second image frames to obtain a first set of matching points; performing image feature matching on the second image frame and a satellite map to obtain a second set of matching points; and performing a mismatch removal operation on the first and second set of matching points to obtain a third and a fourth set of matching points; Step 3: acquiring the third and fourth set of matching points... Step 4: Obtain the number of tiles and their index numbers based on the latitude and longitude of the corner points of the satellite map. Then, stitch the tile count with the satellite map to generate an elevation map. Based on the pixel coordinates of each feature point in the fifth matching point set matched in the satellite map, and the corresponding satellite map and elevation map, obtain the three-dimensional coordinates of each feature point in the fifth matching point set. The three-dimensional coordinates include longitude, latitude, and altitude. Step 5: Obtain the three-axis attitude error based on the three-dimensional coordinates of each feature point in the fifth matching point set. Use the three-axis attitude error as the optimization parameter to construct a least-squares problem. Solve the least-squares problem to obtain the attitude error, and use the attitude error to compensate for the current attitude of the UAV.

[0008] Further, step 1 includes: Step 11: Using a feature matching method, the first image frame is matched with each of the multiple image frames to obtain a first pixel coordinate set of all feature points in the first image frame and a second pixel coordinate set of all feature points in each image frame; Step 12: Based on the first heading angle and the second heading angle corresponding to the first image frame and each image frame captured by the camera mounted on the UAV, the pixel coordinates of all feature points in the first pixel coordinate set and the second pixel coordinate set are rotated to the same direction to obtain a third pixel coordinate set and a fourth pixel coordinate set; Step 13: Based on the third pixel coordinate set and the fourth pixel coordinate set, the pixel translation amount of each image frame relative to the first image frame is obtained; Step 14: Based on the pixel translation amount of each image frame relative to the first image frame, a second image frame is selected from the multiple image frames.

[0009] Furthermore, step 12 includes: the first heading angle corresponding to each of the first image frame and multiple image frames captured by the drone's onboard camera. Second heading angle The pixel coordinates of all feature points in the first and second pixel coordinate sets are rotated to a uniform direction using the following formulas to obtain the third and fourth pixel coordinate sets respectively: (1) Among them, Let i be the coordinate of the i-th pixel in the third set of pixel coordinates. Let i be the coordinate of the i-th pixel in the fourth set of pixel coordinates. Let be the pixel coordinates of the i-th feature point in the first pixel coordinate set within the first image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively. Let i be the pixel coordinates of the i-th feature point in the second set of pixel coordinates in each image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively; step 13 includes: according to... and The pixel shift of each of the multiple image frames relative to the first image frame can be obtained using the following formula: (2) Among them, The pixel shift for each image frame relative to the first image frame. The number of successfully matched feature points; step 14 includes: shifting the pixel distance of each image frame in the multiple image frames relative to the first image frame by a preset threshold. Compare and select pixel shift amount Greater than the preset threshold The image frame is used as the second keyframe.

[0010] Further, step 3 includes: obtaining a binary index map of the same size as the second image frame, wherein the initial value of all pixels in the binary index map is a first value; establishing a pixel coordinate group for each feature point, the pixel coordinate group including the pixel coordinates of the feature point in the first image frame, the pixel coordinates in the second image frame, and the pixel coordinates in the satellite map; traversing each pixel coordinate group in the intersection of the third matching point set and the fourth matching point set; if the pixel value of the binary index map at the pixel coordinates of the second image frame is the first value, then a corresponding square region is constructed with the pixel coordinates of the second image frame as the center and a side of a preset width as a square, and all pixel values ​​in the square region of the binary index map are set to the second value; if the pixel value of the binary index map at the pixel coordinates of the second image frame is the second value, then the pixel coordinate group corresponding to the pixel coordinates of the second image frame in the intersection is deleted; after traversing all pixel coordinate groups in the intersection of the third matching point set and the fourth matching point set, a fifth matching point set is obtained.

[0011] Further, step 4 includes: Step 41: Obtain the tile index number and tile quantity based on the latitude and longitude of the upper left corner and lower right corner of the satellite map; stitch the tiles onto the satellite map based on the tile index number and tile quantity to obtain an elevation map; Step 42: Obtain the longitude and latitude coordinates of each feature point in the fifth matching point set based on the pixel coordinates of each feature point in the satellite map; Step 43: Obtain the altitude coordinates of each feature point in the fifth matching point set based on the longitude and latitude coordinates of each feature point in the fifth matching point set.

[0012] Furthermore, step 41 includes: based on the latitude and longitude of the upper left corner point of the satellite map. Latitude and longitude of the bottom right corner point The tile index number is calculated using the following formula: (6) Among them, The longitude tile index number of the top left corner point on the satellite map. This is the longitude tile index number of the point in the lower right corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This is the floor operator. The longitude of the point in the upper left corner of the satellite map. The latitude of the top left corner of the satellite map. The longitude of the point in the lower right corner of the satellite map. The latitude of the point in the lower right corner of the satellite map. For elevation map tile levels; in the x-axis direction to y-axis direction to The number of tiles within the area formed is By piecing together the tiles corresponding to the specified number, an elevation map can be obtained. Step 42 includes: determining the pixel coordinates of the k-th feature point in the satellite map based on the fifth matching point set. The longitude and latitude of the k-th feature point in the fifth matching point set on the satellite map are obtained using the following formula: (7) Among them, Let k be the longitude of the k-th feature point in the fifth matching point set on the satellite map. Let k be the latitude of the k-th feature point in the fifth set of matching points on the satellite map. For satellite map resolution, Given the current latitude of the drone; based on the pixel coordinates of the kth feature point in the satellite map, the longitude and latitude of the kth feature point in the fifth matching point set are obtained using the following formula: (8) (9) Among them, Rotate the satellite map around the center The pixel coordinates x after the angle Rotate the satellite map around the center The pixel coordinates y after the angle, The u-axis pixel coordinates of the feature point matched in the satellite map for the k-th feature point in the fifth matching point set. Let v-axis pixel coordinates be the feature point in the satellite map that matches the k-th feature point in the fifth matching point set. This refers to the number of pixels in the width direction of the satellite map. This represents the number of pixels in the height direction of the satellite map. Here are the longitude coordinates of the k-th feature point in the fifth matching point set. Let k be the latitude coordinates of the k-th feature point in the fifth matching point set. The current longitude of the drone. The scaling factor is used; step 43 includes: according to , and The height of the k-th feature point in the fifth matching point set can be obtained using the following formula: (10) (11) Among them, The longitude tile index number of the k-th feature point. The latitude tile index number of the k-th feature point. For high-resolution elevation maps, The height of the k-th feature point in the fifth matching point set.

[0013] Further, step 5 includes: Step 51: Obtaining the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame based on the three-dimensional coordinates of each feature point in the fifth matching point set; Step 52: Reprojecting the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame onto the first image frame and the second image frame to obtain the first projected pixel coordinates and the second projected pixel coordinates; Step 53: Obtaining the three-axis attitude error based on the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame, as well as the first projected pixel coordinates and the second projected pixel coordinates; Step 54: Constructing a least squares problem with the goal of minimizing the three-axis attitude error, using the LM algorithm to solve for the attitude angle to be optimized, obtaining the optimized attitude angle, and obtaining the camera mounting angle error based on the attitude angle before optimization and the optimized attitude angle.

[0014] Further, step 51 includes: step 511: converting the three-dimensional coordinates of each feature point in the fifth matching point set into geocentric rectangular coordinates; step 512: obtaining the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system based on the three-dimensional coordinates of the feature points in the first and second image frames of the fifth matching point set; step 513: obtaining the coordinates of each feature point in the fifth matching point set in the camera coordinate system based on the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system, and the geocentric rectangular coordinates.

[0015] Further, step 511 includes: setting the latitude, longitude, and altitude coordinates of the kth feature point in the fifth matching point set. Convert to geocentric rectangular coordinates ; (12) Among them, The semi-major axis of the Earth's ellipse. Let N be the eccentricity of the Earth's ellipse and N be the radius of curvature of the primordial circle. Step 512 includes: obtaining the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system based on the latitude, longitude, and altitude coordinates of the feature points in the fifth matching point set in the first and second image frames; the relative transformation relationship includes the attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when the first image frame was captured. The attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the second image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. Step 513 includes: obtaining the coordinates of each feature point in the fifth matching point set in the camera coordinate system in the first image frame and the second image frame using the following formula: (13) Among them, Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system used to capture the first image frame. Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system of the second image frame. This represents the total number of feature points in the fifth matching point set.

[0016] Further, step 52 includes: obtaining the first projected pixel coordinates and the second projected pixel coordinates using the following formula: (14) Among them, and Let the k-th feature point in the fifth matching point set be the first projected pixel coordinates and the second projected pixel coordinates on the first image frame and the second image frame, respectively. The camera intrinsic parameter matrix is ​​used; step 53 includes obtaining the three-axis attitude error using the following formula: (15) Among them, For three-axis attitude error, Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the first image frame. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the second image frame. The symbol represents the calculation of the L2 norm; step 54 includes: minimizing the three-axis attitude error. To achieve the desired outcome, a least-squares problem is constructed, and the LM algorithm is used to solve for the attitude angles to be optimized, yielding the optimized attitude angles. The attitude angles include the yaw, pitch, and roll angles. The LM algorithm is used to calculate the required Jacobian matrix. , , , , , The LM algorithm terminates optimization after iterating to the expected termination condition. The attitude angle before optimization is... The difference between the attitude angle before and after optimization is the camera mounting angle error.

[0017] (16) Among them, and These are the heading angles of the first image frame before and after optimization, respectively. and These are the heading angles of the second image frames before and after optimization, respectively. and These represent the pitch angles of the first image frame before and after optimization. and These are the pitch angles of the second image frame before and after optimization, respectively. and These are the roll angles of the first image frame before and after optimization. and These are the roll angles of the second image frames before and after optimization. For three-axis attitude error, , , These are the heading angle error, pitch angle error, and roll angle error, respectively.

[0018] Due to the adoption of the above technical solution, this application has the following advantages: This application can select image feature points with strong stability online during flight, and combine them with information such as the UAV's pose, images, satellite maps, and elevation maps to calculate and correct the installation error of the downward-looking camera in real time, thereby ensuring the positioning accuracy of the UAV during visual navigation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 is a flowchart illustrating an online correction method for the installation error of a UAV downward-looking camera according to an embodiment of this application; Figure 2 is a schematic diagram illustrating the visual navigation estimation trajectory without installation error correction according to an embodiment of this application; Figure 3 is a schematic diagram illustrating the visual navigation estimation trajectory with installation error correction according to an embodiment of this application. Detailed Implementation

[0021] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0022] Referring to Figure 1, this application provides an embodiment of an online correction method for the installation error of a UAV's downward-looking camera, which includes: Step 1: Acquiring an initial image frame captured by the camera carried by the UAV during flight and using it as the first image frame; after a preset time interval, acquiring multiple image frames; selecting a second image frame from the multiple image frames based on the pixel translation amount between each image frame and the first image frame; Step 2: Performing feature matching on the first image frame and the second image frame to obtain a first set of matching points; performing image feature matching on the second image frame and a satellite map to obtain a second set of matching points; performing a mismatch removal operation on the first set of matching points and the second set of matching points to obtain a third set of matching points and a fourth set of matching points; Step 3: Obtaining the third set of matching points. Step 4: Obtain the number of tiles and index number based on the latitude and longitude of the corner points of the satellite map. Stitch the number of tiles with the satellite map to generate an elevation map. Based on the pixel coordinates of each feature point in the fifth matching point set matched in the satellite map, as well as the corresponding satellite map and elevation map, obtain the three-dimensional coordinates of each feature point in the fifth matching point set. The three-dimensional coordinates include longitude, latitude, and altitude. Step 5: Obtain the three-axis attitude error based on the three-dimensional coordinates of each feature point in the fifth matching point set. Use the three-axis attitude error as the quantity to be optimized and construct a least squares problem. By solving the least squares problem, obtain the attitude error and use the attitude error to compensate for the current attitude of the UAV.

[0023] In this embodiment, step 1 includes: Step 11: Using a feature matching method, the first image frame is matched with each of the multiple image frames to obtain a first pixel coordinate set of all feature points in the first image frame and a second pixel coordinate set of all feature points in each image frame; Step 12: Based on the first heading angle and the second heading angle corresponding to the first image frame and each image frame captured by the camera mounted on the UAV, the pixel coordinates of all feature points in the first pixel coordinate set and the second pixel coordinate set are rotated to the same direction to obtain a third pixel coordinate set and a fourth pixel coordinate set; Step 13: Based on the third pixel coordinate set and the fourth pixel coordinate set, the pixel translation amount of each image frame relative to the first image frame is obtained; Step 14: Based on the pixel translation amount of each image frame relative to the first image frame, a second image frame is selected from the multiple image frames.

[0024] Optionally, in step 11, since the first image frame needs to be matched with multiple image frames multiple times, it is necessary to ensure that the matched feature points have a high accuracy. Therefore, the SURF (Speed ​​Up Robust Feature) feature matching method is used. The SURF feature matching method can simultaneously take into account both computational speed and accuracy.

[0025] In this embodiment of the application, step 12 includes: the first heading angle corresponding to the first image frame and each of the multiple image frames captured by the drone's onboard camera. Second heading angle The pixel coordinates of all feature points in the first and second pixel coordinate sets are rotated to a uniform direction using the following formulas to obtain the third and fourth pixel coordinate sets respectively: (1) Among them, Let i be the coordinate of the i-th pixel in the third set of pixel coordinates. Let i be the coordinate of the i-th pixel in the fourth set of pixel coordinates. Let be the pixel coordinates of the i-th feature point in the first pixel coordinate set within the first image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively. Let i be the pixel coordinates of the i-th feature point in the second set of pixel coordinates in each image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively; Step 13 includes: according to... and The pixel shift of each of the multiple image frames relative to the first image frame can be obtained using the following formula: (2) Among them, The pixel shift for each image frame relative to the first image frame. The number of successfully matched feature points; Step 14 includes: shifting the pixel amount of each image frame in the multiple image frames relative to the first image frame by a preset threshold. Compare and select pixel shift amount Greater than the preset threshold The image frame is used as the second keyframe.

[0026] Alternatively, based on engineering experience, The value is one-quarter of the image's narrow side resolution, for example. Image resolution, value .

[0027] Optionally, in step 2, the process described in the adaptive satellite image generation method for UAV visual positioning published in CN114201633A can be used to stitch satellite map tiles of the adjacent area in real time according to the current GNSS latitude and longitude position and heading angle of the UAV. After cropping, scaling and rotating, an adaptive satellite map similar to the actual image captured by the downward-looking camera is generated. The satellite map generated based on this method can significantly improve the number and accuracy of feature matching.

[0028] Optionally, in step 2, unlike the coarse matching in step 1, the formal image feature matching needs to ensure that the number of successfully matched feature points is as large as possible and that the accuracy is high. Image feature matching is performed between the second camera frame selected in step 1 and the adaptive satellite map generated in step 2 to obtain a second set of matching points. Since the camera-captured image and the satellite map are heterogeneous images, they often differ significantly in color, lighting, and scenery. Traditional feature matching methods struggle to obtain a large number of stable and reliable matching points. SuperPoint feature extraction and SuperGlue feature matching based on deep learning methods can maximize the number of feature points and the accuracy, and also exhibits extremely high generalization ability across different flight scenarios.

[0029] The first and second image frames are from the same source, and feature matching between them is relatively easy. Therefore, the traditional KNN (k-nearest neighbor) feature matching method can be used to speed up the operation. However, in order to ensure the consistency of feature points, the first image frame still needs to use SuperPoint for feature extraction.

[0030] After feature matching is completed, to further improve the accuracy of feature matching, a mismatch removal operation is also required. Based on the feature mismatch removal method for aerial images of UAV systems and the process described in the system (publication number CN114998773A), the matching points are filtered. Let the final filtered matching point sets, namely the third matching point set and the fourth matching point set, be: (3) Wherein, superscript and These represent the first set of matching points and the second set of matching points, respectively. For the first set of matching points, the first The pixel coordinates of each feature point in the first image frame For the first set of matching points, the first The pixel coordinates of the feature points in the second image frame For the second set of matching points, the first The pixel coordinates of the feature points in the second image frame For the second set of matching points, the first The pixel coordinates of each feature point in the satellite map and These represent the number of feature points in the first set of matching points and the second set of matching points, respectively.

[0031] Feature points in the second image frame that are simultaneously matched by the first image frame and the satellite map, i.e., satisfying The feature points are common feature points. We take these common feature points to form a new set, which is the intersection of the third and fourth matching point sets: (4) In some flight scenarios, feature points may be concentrated in a local area of ​​the image, which will cause the algorithm to make incorrect estimates of the attitude of the UAV. Therefore, it is necessary to homogenize the matching point set 2 in step 3. The specific method is as follows: Step 3 includes: obtaining a binary index map of the same size as the second image frame, where the initial value of all pixels in the binary index map is the first value; establishing a pixel coordinate group for each feature point, which includes the pixel coordinates of the feature point in the first image frame, the pixel coordinates in the second image frame, and the pixel coordinates in the satellite map; traversing each pixel coordinate group in the intersection of the third and fourth matching point sets, if the pixel value of the binary index map at the pixel coordinates in the second image frame is the first value, then a corresponding square region is constructed with the pixel coordinates of the second image frame as the center and a square with a preset width as the side, and all pixel values ​​in the square region of the binary index map are set to the second value; if the pixel value of the binary index map at the pixel coordinates in the second image frame is the second value, then the pixel coordinate group corresponding to the pixel coordinates of the second image frame in the intersection is deleted; after traversing all pixel coordinate groups in the intersection of the third and fourth matching point sets, the fifth matching point set is obtained.

[0032] For example, a binary indexed map of the same size as the second image frame is created. Index diagram The initial values ​​are all 0.

[0033] Traversing the set of common points If a binary indexed graph In coordinates If the value is 0, then the coordinates will be... Centered on a square area with a width of 15 pixels, all pixel values ​​are set to 255. (This is from a binary indexed image.) In coordinates If the value is 255, then from the common point set Delete the feature point.

[0034] The set of matching points after homogenization is called the stable set of matching points (the fifth set of matching points), denoted as: (5) If the set of stable matching points Number of feature points Minimum threshold not met If so, return to step 1 to reselect a camera frame. Threshold The selection should be based on the image resolution, according to engineering experience. Image resolution values .

[0035] The intersection operation of matching points can extract feature points that appear simultaneously in satellite map matching and camera frame matching. These feature points have high stability and are more likely to represent actual terrain features rather than general pixel grayscale features. Matching point homogenization can effectively improve the accuracy and reliability of subsequent pose calculation.

[0036] In this embodiment of the application, step 4 includes: Step 41: Based on the latitude and longitude of the upper left corner and the lower right corner of the satellite map, obtain the tile index number and the number of tiles; based on the tile index number and the number of tiles, stitch the tiles onto the satellite map to obtain an elevation map; Step 42: Based on the pixel coordinates of each feature point in the fifth matching point set in the satellite map, obtain the longitude and latitude coordinates of each feature point in the fifth matching point set; Step 43: Based on the longitude and latitude coordinates of each feature point in the fifth matching point set, obtain the altitude coordinates of each feature point in the fifth matching point set.

[0037] In this embodiment of the application, step 41 includes: based on the latitude and longitude of the upper left corner point of the satellite map. Latitude and longitude of the bottom right corner point The tile index number is calculated using the following formula: (6) Among them, The longitude tile index number of the top left corner point on the satellite map. This is the longitude tile index number of the point in the lower right corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This is the floor operator. The longitude of the point in the upper left corner of the satellite map. The latitude of the top left corner of the satellite map. The longitude of the point in the lower right corner of the satellite map. The latitude of the point in the lower right corner of the satellite map. For elevation map tile levels; in the x-axis direction to y-axis direction to The number of tiles within the area formed is By piecing together the tiles corresponding to the specified number, an elevation map can be obtained. For example, the tiles of an elevation map can be... A TIFF image of the specified size; Step 42 includes: determining the pixel coordinates of the k-th feature point in the satellite map based on the size of the fifth matching point set. The longitude and latitude of the k-th feature point in the fifth matching point set on the satellite map are obtained using the following formula: (7) Among them, Let k be the longitude of the k-th feature point in the fifth matching point set on the satellite map. Let k be the latitude of the k-th feature point in the fifth set of matching points on the satellite map. For satellite map resolution, The latitude of the drone is given. When the satellite map layer is fixed, the satellite map resolution is a fixed value and can be obtained by looking up a table. Based on the pixel coordinates of the kth feature point in the satellite map within the fifth matching point set, the longitude and latitude of the kth feature point in the fifth matching point set are obtained using the following formula: (8) (9) Among them, Rotate the satellite map around the center The pixel coordinates x after the angle Rotate the satellite map around the center The pixel coordinates y after the angle, The u-axis pixel coordinates of the feature point matched in the satellite map for the k-th feature point in the fifth matching point set. Let v-axis pixel coordinates be the feature point in the satellite map that matches the k-th feature point in the fifth matching point set. This refers to the number of pixels in the width direction of the satellite map. This represents the number of pixels in the height direction of the satellite map. Here are the longitude coordinates of the k-th feature point in the fifth matching point set. Let k be the latitude coordinates of the k-th feature point in the fifth matching point set. The current longitude of the drone. For scaling factor; step 43 includes: according to , and The height of the k-th feature point in the fifth matching point set can be obtained using the following formula: (10) (11) Among them, The longitude tile index number of the k-th feature point. The latitude tile index number of the k-th feature point. For high-resolution elevation maps, The height of the k-th feature point in the fifth matching point set.

[0038] In this embodiment, step 5 includes: Step 51: Obtaining the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame based on the three-dimensional coordinates of each feature point in the fifth matching point set; Step 52: Reprojecting the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame onto the first image frame and the second image frame to obtain the first projected pixel coordinates and the second projected pixel coordinates; Step 53: Obtaining the three-axis attitude error based on the pixel coordinates of each feature point in the fifth matching point set in the first image frame and the pixel coordinates in the second image frame, as well as the first projected pixel coordinates and the second projected pixel coordinates; Step 54: Constructing a least squares problem with the goal of minimizing the three-axis attitude error, using the LM algorithm to solve for the attitude angle to be optimized, obtaining the optimized attitude angle, and obtaining the camera mounting angle error based on the attitude angle before optimization and the optimized attitude angle.

[0039] In this embodiment of the application, step 51 includes: step 511: converting the three-dimensional coordinates of each feature point in the fifth matching point set into geocentric rectangular coordinates; step 512: obtaining the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system based on the three-dimensional coordinates of the feature points in the first and second image frames of the fifth matching point set; step 513: obtaining the coordinates of each feature point in the fifth matching point set in the camera coordinate system based on the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system, and the geocentric rectangular coordinates.

[0040] In this embodiment of the application, step 511 includes: setting the latitude, longitude, and altitude coordinates of the kth feature point in the fifth matching point set. Convert to geocentric rectangular coordinates ; (12) Among them, The semi-major axis of the Earth's ellipse. Let N be the eccentricity of the Earth's ellipse, and N be the radius of curvature of the prime mover; for example... , Step 512 includes: obtaining the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system based on the latitude, longitude, and altitude coordinates of the feature points in the fifth matching point set in the first and second image frames; the relative transformation relationship includes the attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when the first image frame was captured. The attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the second image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. Step 513 includes: obtaining the coordinates of each feature point in the fifth matching point set in the camera coordinate system in the first and second image frames using the following formula: (13) Among them, Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system used to capture the first image frame. Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system of the second image frame. This represents the total number of feature points in the fifth matching point set. (Superscript) , These represent the camera coordinate systems corresponding to the first and second image frames, respectively.

[0041] In this embodiment of the application, step 52 includes: obtaining the first projected pixel coordinates and the second projected pixel coordinates using the following formula: (14) Among them, and Let the kth feature point in the fifth matching point set be the first projected pixel coordinates and the second projected pixel coordinates on the first image frame and the second image frame, respectively. The camera intrinsic parameter matrix can be obtained through prior calibration; step 53 includes: obtaining the three-axis attitude error using the following formula: (15) Among them, For three-axis attitude error, Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the first image frame. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the second image frame. The symbol represents the calculation of the L2 norm; step 54 includes: minimizing the three-axis attitude error. To achieve the desired outcome, a least-squares problem is constructed, and the LM algorithm is used to solve for the attitude angles to be optimized, yielding the optimized attitude angles. The attitude angles include the yaw, pitch, and roll angles. The LM algorithm is used to calculate the required Jacobian matrix. , , , , , The LM algorithm terminates optimization after iterating to the expected termination condition. The attitude angle before optimization is... The difference between the attitude angle before and after optimization is the camera mounting angle error.

[0042] (16) Among them, and These are the heading angles of the first image frame before and after optimization, respectively. and These are the heading angles of the second image frames before and after optimization, respectively. and These represent the pitch angles of the first image frame before and after optimization. and These are the pitch angles of the second image frame before and after optimization, respectively. and These are the roll angles of the first image frame before and after optimization. and These are the roll angles of the second image frames before and after optimization. For three-axis attitude error, , , These are the heading angle error, pitch angle error, and roll angle error, respectively.

[0043] In Figure 2, the green line represents the relatively straight GNSS trajectory, while the red line represents the visual navigation estimated trajectory. It is evident that the visual navigation estimated position without installation error correction exhibits a significant deviation (indicated by the arrows in the figure). The visual navigation estimated position after installation error correction essentially coincides with the actual GNSS trajectory, as shown in Figure 3. This demonstrates the effectiveness of the method described in this application.

[0044] This application ensures the positioning accuracy of the UAV during visual navigation by online screening of image feature points with strong stability during flight and combining information such as the UAV's GNSS position, attitude, satellite map, and elevation map to calculate the installation error of the downward-looking camera in real time.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A method for online correction of installation error of a UAV downward-looking camera, characterized in that, include: Step 1: Acquire an initial image frame captured by the camera carried by the drone during its flight and use it as the first image frame. After a preset time interval, acquire multiple image frames. Select the second image frame from the multiple image frames based on the pixel translation amount between each image frame and the first image frame. Step 2: Perform feature matching on the first and second image frames to obtain the first set of matching points. Perform image feature matching on the second image frame and the satellite map to obtain the second set of matching points. Perform mismatch removal on the first and second sets of matching points to obtain the third and fourth sets of matching points. Step 3: Obtain the intersection between the third and fourth sets of matching points and perform uniformization on the intersection to obtain the fifth set of matching points. Step 4: Obtain the number of tiles and their index numbers based on the latitude and longitude of the corner points on the satellite map. Stitch the number of tiles with the satellite map to generate an elevation map. Based on the pixel coordinates of each feature point in the fifth set of matching points matched in the satellite map, and the corresponding satellite map and elevation map, obtain the three-dimensional coordinates of each feature point in the fifth set of matching points. The three-dimensional coordinates include longitude, latitude, and altitude. Step 5: Obtain the three-axis attitude error based on the three-dimensional coordinates of each feature point in the fifth set of matching points. Use the three-axis attitude error as the quantity to be optimized and construct a least squares problem. By solving the least squares problem, obtain the attitude error and use the attitude error to compensate for the current attitude of the UAV.

2. The method according to claim 1, characterized in that, Step 1 includes: Step 11: Using a feature matching method, the first image frame is matched with each of the multiple image frames to obtain the first pixel coordinate set of all feature points in the first image frame and the second pixel coordinate set of all feature points in each image frame; Step 12: Based on the first heading angle and the second heading angle corresponding to the first image frame and each image frame captured by the camera mounted on the UAV, the pixel coordinates of all feature points in the first pixel coordinate set and the second pixel coordinate set are rotated to the same direction to obtain the third pixel coordinate set and the fourth pixel coordinate set, respectively; Step 13: Based on the third pixel coordinate set and the fourth pixel coordinate set, the pixel translation amount of each image frame relative to the first image frame is obtained; Step 14: Based on the pixel translation amount of each image frame relative to the first image frame, the second image frame is selected from the multiple image frames.

3. The method according to claim 2, characterized in that, Step 12 includes: the drone's onboard camera capturing the first image frame and the first heading angle corresponding to each of the multiple image frames. Second heading angle The pixel coordinates of all feature points in the first and second pixel coordinate sets are rotated to a uniform direction using the following formulas to obtain the third and fourth pixel coordinate sets respectively: (1) Among them, Let i be the coordinate of the i-th pixel in the third set of pixel coordinates. Let i be the coordinate of the i-th pixel in the fourth set of pixel coordinates. Let be the pixel coordinates of the i-th feature point in the first pixel coordinate set within the first image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively. Let i be the pixel coordinates of the i-th feature point in the second set of pixel coordinates in each image frame. and These are the coordinates on the u-axis and v-axis in the UV coordinate system, respectively; step 13 includes: according to... and The pixel shift of each of the multiple image frames relative to the first image frame can be obtained using the following formula: (2) Among them, The pixel shift for each image frame relative to the first image frame. The number of successfully matched feature points; step 14 includes: shifting the pixel distance of each image frame in the multiple image frames relative to the first image frame by a preset threshold. Compare and select pixel shift amount Greater than the preset threshold The image frame is used as the second keyframe.

4. The method according to claim 1, characterized in that, Step 3 includes: obtaining a binary index map of the same size as the second image frame, where the initial value of all pixels in the binary index map is a first value; establishing a pixel coordinate group for each feature point, the pixel coordinate group including the pixel coordinates of the feature point in the first image frame, the pixel coordinates in the second image frame, and the pixel coordinates in the satellite map; traversing each pixel coordinate group in the intersection of the third matching point set and the fourth matching point set; if the pixel value of the binary index map at the pixel coordinates of the second image frame is the first value, then a corresponding square region is constructed with the pixel coordinates of the second image frame as the center and a side of a preset width as a square, and all pixel values ​​in the square region of the binary index map are set to the second value; if the pixel value of the binary index map at the pixel coordinates of the second image frame is the second value, then the pixel coordinate group corresponding to the pixel coordinates of the second image frame in the intersection is deleted; after traversing all pixel coordinate groups in the intersection of the third matching point set and the fourth matching point set, a fifth matching point set is obtained.

5. The method according to claim 1, characterized in that, Step 4 includes: Step 41: Obtain the tile index number and tile quantity based on the latitude and longitude of the upper left and lower right corners of the satellite map; stitch the tiles onto the satellite map based on the tile index number and tile quantity to obtain an elevation map; Step 42: Obtain the longitude and latitude coordinates of each feature point in the fifth matching point set based on the pixel coordinates of each feature point in the satellite map; Step 43: Obtain the altitude coordinates of each feature point in the fifth matching point set based on the longitude and latitude coordinates of each feature point in the fifth matching point set.

6. The method according to claim 5, characterized in that, Step 41 includes: determining the latitude and longitude of the upper left corner of the satellite map. Latitude and longitude of the bottom right corner point The tile index number is calculated using the following formula: (6) Among them, The longitude tile index number of the top left corner point on the satellite map. This is the longitude tile index number of the point in the lower right corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This refers to the latitude tile index number of the point in the upper left corner of the satellite map. This is the floor operator. The longitude of the point in the upper left corner of the satellite map. The latitude of the top left corner of the satellite map. The longitude of the point in the lower right corner of the satellite map. The latitude of the point in the lower right corner of the satellite map. For elevation map tile levels; in the x-axis direction to y-axis direction to The number of tiles within the area formed is By piecing together the tiles corresponding to the specified number, an elevation map can be obtained. Step 42 includes: determining the pixel coordinates of the k-th feature point in the satellite map based on the fifth matching point set. The longitude and latitude of the k-th feature point in the fifth matching point set on the satellite map are obtained using the following formula: (7) Among them, Let k be the longitude of the k-th feature point in the fifth matching point set on the satellite map. Let k be the latitude of the k-th feature point in the fifth set of matching points on the satellite map. For satellite map resolution, Given the current latitude of the drone; based on the pixel coordinates of the kth feature point in the satellite map, the longitude and latitude of the kth feature point in the fifth matching point set are obtained using the following formula: (8) (9) Among them, Rotate the satellite map around the center The pixel coordinates x after the angle Rotate the satellite map around the center The pixel coordinates y after the angle, The u-axis pixel coordinates of the feature point matched in the satellite map for the k-th feature point in the fifth matching point set. Let v-axis pixel coordinates be the feature point in the satellite map that matches the k-th feature point in the fifth matching point set. This refers to the number of pixels in the width direction of the satellite map. This represents the number of pixels in the height direction of the satellite map. Here are the longitude coordinates of the k-th feature point in the fifth matching point set. Let k be the latitude coordinates of the k-th feature point in the fifth matching point set. The current longitude of the drone. The scaling factor is used; step 43 includes: according to 、 and The height of the k-th feature point in the fifth matching point set can be obtained using the following formula: (10) (11) Among them, The longitude tile index number of the k-th feature point. The latitude tile index number of the k-th feature point. For high-resolution elevation maps, The height of the k-th feature point in the fifth matching point set.

7. The method according to claim 1, characterized in that, Step 5 includes: Step 51: Based on the three-dimensional coordinates of each feature point in the fifth matching point set, obtain the pixel coordinates of each feature point in the first image frame and the pixel coordinates in the second image frame; Step 52: Reproject the pixel coordinates of each feature point in the fifth matching point set from the first image frame to the second image frame to obtain the first projected pixel coordinates and the second projected pixel coordinates; Step 53: Based on the pixel coordinates of each feature point in the fifth matching point set from the first image frame and the second image frame, as well as the first projected pixel coordinates and the second projected pixel coordinates, obtain the three-axis attitude error; Step 54: With minimizing the three-axis attitude error as the objective, construct a least squares problem, use the LM algorithm to solve for the attitude angle to be optimized, obtain the optimized attitude angle, and obtain the camera mounting angle error based on the attitude angle before optimization and the optimized attitude angle.

8. The method according to claim 7, characterized in that, Step 51 includes: Step 511: Converting the three-dimensional coordinates of each feature point in the fifth matching point set to geocentric rectangular coordinates; Step 512: Obtaining the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system based on the three-dimensional coordinates of the feature points in the first and second image frames of the fifth matching point set; Step 513: Obtaining the coordinates of each feature point in the fifth matching point set in the camera coordinate system based on the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system, and the geocentric rectangular coordinates.

9. The method according to claim 8, characterized in that, Step 511 includes: setting the latitude, longitude, and altitude coordinates of the kth feature point in the fifth matching point set. Convert to geocentric rectangular coordinates ； (12) Among them, The semi-major axis of the Earth's ellipse. Let N be the eccentricity of the Earth's ellipse and N be the radius of curvature of the primordial circle. Step 512 includes: obtaining the relative transformation relationship between the camera coordinate system and the geocentric rectangular coordinate system based on the latitude, longitude, and altitude coordinates of the feature points in the fifth matching point set in the first and second image frames; the relative transformation relationship includes the attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when the first image frame was captured. The attitude matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the second image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. The translation matrix of the camera coordinate system relative to the geocentric rectangular coordinate system when capturing the first image frame. Step 513 includes: obtaining the coordinates of each feature point in the fifth matching point set in the camera coordinate system in the first image frame and the second image frame using the following formula: (13) Among them, Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system used to capture the first image frame. Let be the coordinates of the k-th feature point in the fifth matching point set in the camera coordinate system of the second image frame. This represents the total number of feature points in the fifth matching point set.

10. The method according to claim 8, characterized in that, Step 52 includes obtaining the first projected pixel coordinates and the second projected pixel coordinates using the following formula: (14) Among them, and Let the k-th feature point in the fifth matching point set be the first projected pixel coordinates and the second projected pixel coordinates on the first image frame and the second image frame, respectively. The camera intrinsic parameter matrix is ​​used; step 53 includes obtaining the three-axis attitude error using the following formula: (15) Among them, For three-axis attitude error, Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the first image frame. Let k be the pixel coordinates of the k-th feature point in the fifth matching point set on the second image frame. The symbol represents the calculation of the L2 norm; step 54 includes: minimizing the three-axis attitude error. To achieve the desired outcome, a least-squares problem is constructed, and the LM algorithm is used to solve for the attitude angles to be optimized, yielding the optimized attitude angles. The attitude angles include the yaw, pitch, and roll angles. The LM algorithm is used to calculate the required Jacobian matrix. 、 、 、 、 、 The LM algorithm terminates optimization after iterating to the expected termination condition. The attitude angle before optimization is... The difference between the attitude angle before and after optimization is the camera mounting angle error. (16) Among them, and These are the heading angles of the first image frame before and after optimization, respectively. and These are the heading angles of the second image frames before and after optimization, respectively. and These represent the pitch angles of the first image frame before and after optimization. and These are the pitch angles of the second image frame before and after optimization, respectively. and These are the roll angles of the first image frame before and after optimization. and These are the roll angles of the second image frames before and after optimization. For three-axis attitude error, 、 、 These are the heading angle error, pitch angle error, and roll angle error, respectively.

Images