A UAV integrated navigation method based on visual geolocation and odometer / IMU fusion

Abstract

A UAV integrated navigation method based on visual geolocation and odometry / IMU fusion is disclosed, involving UAV positioning. The method includes a coordinate system definition and transformation step; a remote sensing image database construction step; a visual geolocation step; a relative motion compensation step; and a factor graph fusion positioning step. The method retrieves remote sensing images of the corresponding area from the remote sensing image database and matches them with images captured by the UAV to obtain visual geolocation. The coordinate system of the lidar (or visual) odometry is transformed to the visual geolocation coordinate system. Relative motion compensation is applied to the obtained visual geolocation based on the lidar (or visual) odometry. A factor graph optimization method is used to fuse the visual geolocation, odometry, and IMU to obtain the optimal pose estimate. This method solves the problem of UAVs relying on GNSS for positioning, improves the reliability of UAV navigation and positioning, and is economical, reliable, and accurate.

Description

Technical Field

[0001] This invention relates to the field of UAV positioning technology, specifically to a UAV integrated navigation method based on visual geolocation and odometry / IMU fusion. Background Technology

[0002] With the continuous development of drone technology, drones are increasingly being used in fields such as intelligence reconnaissance, search and rescue, land mapping, and environmental monitoring. Accurate drone positioning is crucial for the effective execution of these functions. Currently, drone positioning technologies mainly employ GNSS (Global Navigation Satellite System), visual odometry, and lidar odometry. Although GNSS provides high positioning accuracy, in actual flight missions, unpredictable factors such as tall buildings blocking the signal or electromagnetic interference can easily cause GNSS signals to be interfered with or lost, thus affecting drone positioning. Visual odometry and lidar odometry can provide relatively accurate drone positioning over short periods and distances, but as the flight time and distance increase, positioning drift inevitably occurs in both methods.

[0003] Visual navigation and positioning technologies based on visual geolocation and remote sensing imagery can solve the aforementioned problems. As a passive sensor, the camera acquires information by receiving ground signals, eliminating reliance on external systems and avoiding the drawbacks associated with GNSS. Furthermore, because the positioning information obtained through image matching in a visual geolocation system is absolute, it resolves the positioning drift issues that occur with visual odometry and lidar odometry over long distances and over extended periods. However, since visual geolocation provides discrete positioning, continuous positioning for UAVs requires fusion with positioning information from other sensors to achieve continuous positioning output. Summary of the Invention

[0004] The purpose of this invention is to address the problems of GNSS positioning failure on UAVs in complex scenarios and the inevitable positioning drift when using visual odometry or lidar odometry alone. This invention provides a UAV positioning method based on visual geolocation, lidar (or visual) odometry, and IMU fusion. The method first transforms the positioning information obtained from the lidar (or visual) odometry into a visual geolocation coordinate system; then, a remote sensing image database is constructed; next, visual geolocation is obtained by matching images taken by the UAV with the remote sensing image database; finally, the visual geolocation, the transformed lidar (or visual) odometry positioning information, and the IMU are fused using a factor graph optimization method to obtain continuous positioning information.

[0005] A UAV integrated navigation method based on visual geolocation and odometry / IMU fusion includes the following steps:

[0006] 1) A method for establishing a remote sensing image database: using a two-stage remote sensing image segmentation method based on Nona-Tree, the original remote sensing image is segmented into image blocks of 1024×1024 pixels, and further divided into sub-image blocks of 512×512 pixels to form an extended remote sensing image database.

[0007] 2) A visual geolocation method: A two-stage, multi-scale scene matching method based on graph neural networks SuperPoint & SuperGLUE is used to achieve visual geolocation of UAVs;

[0008] 3) A method for relative motion compensation in visual geolocation: using a lidar odometry to calculate the relative displacement between the start time stamp and the current time stamp of visual geolocation, and compensating for the visual geolocation results;

[0009] 4) A method for fusion of visual geolocation, odometry and IMU based on factor graph: Define a state sequence including position, orientation, accelerometer random walk noise and gyroscope random walk noise; construct visual geolocation factor residuals, odometry factor residuals and IMU pre-integration factor residuals; construct a factor graph optimization objective function, use the iSAM2 optimization library for state estimation, and obtain the positioning result of integrated navigation.

[0010] In step 1), the method for establishing a remote sensing image database includes the following specific steps:

[0011] (1.1) A two-stage remote sensing image segmentation method based on Nona-Tree is used;

[0012] (1.2) The original remote sensing image is segmented into 1024×1024 pixel remote sensing images using a step size of 1024 pixels and a non-overlapping method;

[0013] (1.3) Expand a set of 4×4 remote sensing images using the Nona-Tree method;

[0014] (1.4) After creating an extended remote sensing image database with 1024×1024 pixels, each of these map tiles was further subdivided into remote sensing images of 512×512 pixels.

[0015] (1.5) Each group of 4×4 remote sensing images (512×512 pixels in size) is expanded using the Nona-Tree method to obtain the final remote sensing image database.

[0016] In step 2), the visual geolocation method includes the following two stages;

[0017] (2.1) The first stage realizes multi-scale image matching of remote sensing images and UAV images at different resolutions in group i, and stitches the matching results of the three groups of images together. The specific steps are as follows:

[0018] (2.1.1) Based on the initial positioning and fused positioning, retrieve the corresponding remote sensing images from the remote sensing image database, and acquire images taken by the camera on the UAV. Simultaneously scale the two images to i different resolution scales. ],[ ] represents i different scaling scales for the length and width of the original image; the value of i can range from 1 to 5;

[0019] (2.1.2) The scaled i groups of images are fed into a graph neural network for image matching to obtain the set of pixel coordinates of the matching point pairs at different scales. , ];

[0020] (2.1.3) Map the set of pixel coordinates of the matching point pairs obtained at different scales back to the original image. The mapping formula is as follows:

[0021]

[0022] in, This represents the matching key points mapped back to the original-size image. These represent the matching keypoints mapped back to the original size image. Axis coordinates and Axis coordinates and They represent the original state before mapping. Axis coordinates and Axis coordinates; after mapping the pixel coordinate sets of three sets of matching point pairs at different scales back to the original image, the three sets are concatenated to obtain the key point pair matching result of the first stage. , ].

[0023] (2.2) In the second stage, the most relevant regions of the remote sensing image and the UAV image are clustered and cropped. Image matching is performed again, and the results are stitched together with the image matching results from the first stage. Finally, the homography matrix and the localization of the UAV are calculated based on the matching results. The specific steps are as follows:

[0024] (2.2.1) Based on the mutually matching keypoint pairs obtained in the first stage [ , The DBSCAN method was used to cluster the regions with the highest degree of mutual matching of key point pairs, which are the regions most similar to the UAV images and satellite images. These regions were then bounded with bounding boxes and cropped to obtain the most relevant UAV images and satellite images in the local area.

[0025] (2.2.2) The obtained locally most relevant UAV image and satellite image are fed into the graph neural network again for image matching to obtain the second set of image matching pixel coordinates. , The second set of image matching pixel coordinates is then mapped back to the original image to obtain the keypoint pair matching results for the second stage. , ];

[0026] (2.2.3) Match the image results from the first stage with those from the second stage. , ]and[ , The images are stitched together to obtain the final keypoint pair matching results between the UAV image and the satellite image. , ];

[0027] (2.2.4) Based on the key point matching results [ , Calculate the relationship between the pixel plane of the UAV imagery and the pixel plane of the satellite map tile, i.e., the homography matrix. The homography matrix is ​​calculated using a robust RANSAC-based method. Based on experience, the RANSAC reprojection error is set to 5 and the number of iterations is set to 1000. After obtaining the homography matrix from the UAV image pixel coordinate system to the remote sensing image pixel coordinate system, the pixel coordinates of the UAV on the satellite image can be directly calculated.

[0028] (2.2.5) The latitude and longitude coordinates of the UAV can be calculated from the pixel coordinates of the UAV on the satellite image using the following formula:

[0029]

[0030]

[0031] in,( , () represents the latitude and longitude coordinates of the upper left corner of the satellite map. , This represents the latitude and longitude coordinates of the lower right corner of the remote sensing image. , This indicates the width and height of the satellite map. , This represents the pixel coordinates of the drone on the remote sensing image.

[0032] In step 3), the method for relative motion compensation in visual geolocation specifically includes the following steps:

[0033] (3.1) Using lidar odometry to calculate the relative displacement between the start time stamp and the current time stamp of visual geolocation, compensating for visual geolocation: the visual geolocation module receives images... Frame, save Frame-based lidar odometry positioning ; after obtaining the location output from the visual geolocation module Frame, based on the location information output by the visual geolocation module timestamps in turn up Frame-preserved LiDAR odometry positioning Based on the current Frame-based lidar odometry positioning The method for calculating relative displacement is as follows:

[0034]

[0035] (3.2) Calculate the visual geolocation of the current frame:

[0036]

[0037] in, This indicates the location output of the visual geolocation module, which receives keyframes from the image. This indicates the output location of the visual geolocation module in the current frame.

[0038] In step 4), the method for visual geolocation, odometry, and IMU based on factor graph fusion comprises the following steps:

[0039] (4.1) Define the state sequence as follows:

[0040]

[0041]

[0042] The state sequence from frame i to frame j includes the motion state of the nth frame, consisting of four state variables: position. ,direction Accelerometer random walk noise and gyroscope random walk noise ;

[0043] (4.2) Constructing visual geolocation factor residuals as follows:

[0044]

[0045] in, Observations representing visual geolocation, This indicates the compensated location after visual geolocation.

[0046] (4.3) Constructing the odometry factor residuals and IMU pre-integral factor residuals ,in, Indicates the odometer's observations, Represents the observations of the IMU;

[0047] (4.4) The objective function for factor graph optimization is constructed as follows:

[0048]

[0049] In argmin{.}, the marginalized prior residuals, IMU pre-integration factor residuals, LiDAR odometry factor residuals, and visual geolocation factor residuals are respectively (the subscripts are the covariance matrices of each). To solve the above optimization problem, the iSAM2 optimization library is used for state estimation to obtain the positioning results of the integrated navigation.

[0050] This invention addresses the current situation where unmanned aerial vehicles (UAVs) heavily rely on GNSS for positioning, and under certain conditions, GNSS loss can lead to the UAV's inability to locate itself and thus malfunction. It provides a UAV positioning method suitable for small and medium-sized UAVs, based on the fusion of visual geolocation, lidar (or visual) odometry, and IMU. First, remote sensing images of the corresponding area are retrieved from a remote sensing image database and matched with images captured by the UAV to obtain visual geolocation. Then, the lidar (or visual) odometry coordinate system is transformed to the visual geolocation coordinate system. Next, relative motion compensation is performed on the obtained visual geolocation based on the lidar (or visual) odometry. Finally, a factor graph optimization method is used to fuse the visual geolocation, odometry, and IMU to obtain the optimal pose estimate.

[0051] This invention achieves high-precision positioning of UAVs in complex scenarios by fusing data from multiple sensors, including visual geolocation, lidar (or visual) odometry, and IMU. Visual geolocation technology utilizes high-resolution remote sensing images and advanced image matching algorithms to provide accurate location information; while lidar (or visual) odometry and IMU provide data directly related to the UAV's motion state, further enhancing the accuracy and stability of positioning. This multi-sensor fusion method significantly improves the positioning accuracy of UAVs in complex environments. The addition of lidar (or visual) odometry and IMU provides the UAV with additional motion information, enhancing the system's robustness and reliability. Even in the event of complete loss of GNSS signals, the UAV can achieve continuous and stable positioning using this solution.

[0052] This invention is applicable to various complex scenarios and terrain conditions, including urban canyons, mountainous areas, and forests. Visual geolocation technology is unaffected by terrain and weather, enabling accurate measurements in diverse environments; while lidar (or visual) odometry and IMU provide data directly related to the drone's motion, allowing it to better adapt to complex environments. Through this invention, the drone can plan the optimal path based on real-time location information and mission requirements, improving flight efficiency. Simultaneously, high-precision positioning information helps the drone detect and avoid obstacles in a timely manner during flight, reducing collision risks and improving flight safety.

[0053] This invention features speed and stability, solves the problem of UAVs not relying on GNSS for positioning, improves the reliability of UAV navigation and positioning, and provides an economical, reliable and accurate UAV integrated navigation method. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall structure of an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the Nona-Tree.

[0056] Figure 3 This is a diagram illustrating visual geolocation.

[0057] Figure 4 This is a physical image of the drone used in Example 1.

[0058] Figure 5 This is a test scenario diagram for Example 1.

[0059] Figure 6 The figure shows the experimental results of Example 1. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Rather, the invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims.

[0061] An embodiment of the UAV integrated navigation method based on visual geolocation and odometry / IMU fusion according to the present invention includes the following five parts, and the overall structure can be found in [reference needed]. Figure 1 The illustration is as follows.

[0062] (1) Definition and transformation of coordinate system

[0063] The camera is mounted in the center of the drone, pointing vertically downwards, and the visual geolocation coordinate system is defined as follows: The positioning coordinate system output by a lidar (or vision) odometer is defined as follows: 3D pose of the drone express The position coordinates of the drone in the visual geolocation coordinate system.

[0064] Define lidar (or vision) odometry in The output location at time t is:

[0065]

[0066] Define the visual geolocation module in The output location at time t is:

[0067]

[0068] Because lidar (or vision) odometry and visual geolocation modules only exist in output positioning... The rotation is rigid on the axis, therefore the rotation matrix is ​​defined as:

[0069]

[0070] Where, This indicates the angle between the coordinate system of the lidar (or vision) odometer and the visual geolocation module.

[0071] The rotation matrix from the odometer positioning coordinate system to the visual geolocation coordinate system can be obtained by solving the following least squares method. :

[0072]

[0073] Where, For lidar (or vision) odometry Output positioning at any given moment; For the visual geolocation module in Output positioning at any given moment; Represents the L2 norm;

[0074] (2) Construction of remote sensing image database

[0075] To achieve large-scale visual geolocation in environments with limited GPS signals, it is desirable for the remote sensing image database to cover as large an area as possible. A two-stage remote sensing image segmentation method based on Nona-Tree is used to create a broad-coverage and efficient remote sensing image database. First, the original remote sensing image is segmented into multiple 1024×1024 pixel remote sensing images using a step size of 1024 pixels with no overlap, ensuring that each image patch contains sufficient information for subsequent visual geolocation. Then, a set of 4×4 1024×1024 pixel remote sensing images is expanded using the Nona-Tree method, creating an expanded remote sensing image database of 1024×1024 pixels. Each of these map patches is further subdivided into 512×512 pixel remote sensing images. After segmentation, each set of 4×4 512×512 pixel remote sensing images is expanded using the Nona-Tree method to obtain the final remote sensing image database. Details of the Nona-Tree image segmentation method can be found in [link to documentation]. Figure 2 The illustration illustrates this. By extending and re-segmenting the Nona-Tree method, image information can be utilized more effectively, improving the coverage and retrieval accuracy of the image database. Simultaneously, smaller image patches facilitate subsequent image matching and feature extraction.

[0076] (3) Visual geolocation module

[0077] The visual geolocation module employs a two-stage, multi-scale image matching network based on graph neural networks SuperPoint & SuperGLUE to achieve multi-scale image matching between remote sensing images and UAV images. Its specific structure can be found in [link to documentation]. Figure 3 The illustration is as follows.

[0078] 1) First stage: Achieve multi-scale image matching between remote sensing images and UAV images at three different resolutions, and stitch the three sets of image matching results together;

[0079] Retrieve remote sensing images corresponding to the initial positioning and fused positioning from the remote sensing image database, and acquire images taken by the camera on the UAV. Simultaneously scale the two images to i different resolution scales. ],in,[ [] represents the scaling scale of the original image's length and width; the value of i ranges from 1 to 5; in this embodiment, the value of i is 3, so the two images are simultaneously scaled to three different resolution scales. ],[ ],[ ].

[0080] The scaled i sets of images are fed into the graph neural networks SuperPoint & SuperGLUE for image matching, resulting in i sets of pixel coordinate sets of image matching point pairs at different scales. , ],[ , ] represents the set of pixel coordinates of matching point pairs detected in the UAV image and the remote sensing image, respectively. In this embodiment, the three scaled images are fed into the graph neural network SuperPoint & SuperGLUE for image matching, resulting in three sets of pixel coordinate sets of matching point pairs at different scales, respectively represented as [ , ],[ , ],[ , ].

[0081] Since the image has been scaled, it is necessary to remap the set of pixel coordinates of the matching point pairs obtained at different scales back to the original image size. The formula for remapping is:

[0082]

[0083] in, This represents the matching key points in the image remapped back to the original size. These represent the matching keypoints mapped back to the original size image. Axis coordinates and Axis coordinates and They represent the original state before mapping. Axis coordinates and Axis coordinates

[0084] After mapping the pixel coordinate sets of three sets of matching point pairs at different scales back to the original image, the three sets are concatenated to obtain the key point pair matching result of the first stage. , ].

[0085] 2) Second stage: Cluster and crop the most relevant regions of the remote sensing image and the UAV image, perform image matching again, and stitch the image matching results from the first stage together. Finally, calculate the homography matrix and the UAV's localization based on the matching results.

[0086] Based on the keypoint pairs that match each other obtained in the first stage [ , The DBSCAN method (density-based clustering algorithm) is used to cluster the regions with the highest degree of mutual matching of key point pairs, that is, the regions where the UAV images and satellite images are most similar. These regions are then bounded by bounding boxes (bboxes) and the bbox regions are cropped to obtain the most relevant UAV images (UAV images) and satellite images in the local area.

[0087] Next, the cropped, most relevant UAV image and the satellite image are fed back into the graph neural network for image matching again, resulting in a second set of image matching pixel coordinates. , The second set of image matching pixel coordinates is then mapped back to the original image to obtain the keypoint pair matching results for the second stage. , ].

[0088] Finally, the image matching results from the first and second stages are compared. , ]and[ , The images are stitched together to obtain the final keypoint pair matching results between the UAV image and the satellite image. , ].

[0089] After obtaining the final feature point matching results [ , Afterwards, based on the keypoint matching results, the relationship between the UAV image pixel plane and the satellite map tile pixel plane is calculated, that is, the homography matrix between the UAV image pixel plane and the satellite map tile is calculated. The method for calculating the homography matrix H is as follows:

[0090]

[0091] in, This represents the value of the homography matrix H. This indicates the key points for matching on the drone. This indicates the key points that are matched on the satellite image.

[0092] A robust RANSAC-based method was used to calculate the homography matrix, with a RANSAC reprojection error of 5 and 1000 iterations. After obtaining the homography matrix from the UAV image pixel coordinate system to the remote sensing image pixel coordinate system, the UAV's pixel coordinates on the satellite image can be directly calculated. Since each stored remote sensing image patch contains the latitude and longitude coordinates of its top-left and bottom-right edges, the UAV's latitude and longitude coordinates can be calculated from its pixel coordinates on the satellite image using the following formula:

[0093]

[0094]

[0095] in,( , () represents the latitude and longitude coordinates of the upper left corner of the satellite map. , This represents the latitude and longitude coordinates of the lower right corner of the remote sensing image. , This indicates the width and height of the satellite map. , This represents the pixel coordinates of the drone on the remote sensing image.

[0096] The two-stage method described above effectively achieves the matching of UAV images with satellite remote sensing images and the localization of the UAV. The first stage obtains candidate keypoint pairs through preliminary matching, and the second stage improves the matching accuracy through clustering, cropping, and re-matching. Finally, the homography matrix and the latitude and longitude coordinates of the UAV are calculated.

[0097] (4) Relative motion compensation

[0098] When using visual geolocation to calculate the absolute position of a drone, the drone continues to fly, causing a time lag between the visual geolocation start timestamp and the current timestamp. To address this issue, a lidar odometry is used to calculate the relative displacement between the visual geolocation start timestamp and the current timestamp, compensating for the visual geolocation error.

[0099] The visual geolocation module receives images. Frame, save Frame-based lidar odometry positioning After obtaining the location output from the visual geolocation module... Frame, based on the location information output by the visual geolocation module timestamps in turn up Frame-preserved LiDAR odometry positioning Based on the current Frame-based lidar odometry positioning The method for calculating relative displacement is as follows:

[0100]

[0101] The visual geolocation of the current frame is then calculated:

[0102]

[0103] in, This indicates the location output of the visual geolocation module, which receives keyframes from the image. This indicates the output location of the visual geolocation module in the current frame.

[0104] (5) Factor graph optimization fusion positioning

[0105] To improve the accuracy and stability of UAV positioning, a factor graph optimization method is used to fuse visual geolocation, LiDAR odometry positioning, and IMU. For the optimization sequence, the state sequence is defined as follows:

[0106]

[0107]

[0108] The state sequence from frame i to frame j includes the motion state of the nth frame, consisting of four state variables: position. ,direction Accelerometer random walk noise and gyroscope random walk noise .

[0109] IMU pre-integrating factor residuals The structure is as follows:

[0110]

[0111] in, Represents the observations of the IMU. The Jacobian matrix represents the pre-integral terms of the gyroscope's velocity, position, and attitude. The Jacobian matrix represents the pre-integral terms of the accelerometer velocity and position. This represents the extrinsic parameters of the IMU. This represents the random walk error of the gyroscope and accelerometer.

[0112] LiDAR Odometry LIO Factor Residual The structure is as follows:

[0113]

[0114] in, This represents the observed measurements of the odometer. This represents the change in position and orientation of the LIO. Similarly, a visual odometry factor can be constructed.

[0115] Visual geolocation factor residuals The structure is as follows:

[0116]

[0117] in, Observations representing visual geolocation, This indicates the location after visual geolocation has been compensated.

[0118] The objective function for factor graph optimization is constructed as follows:

[0119]

[0120] In argmin{.}, the marginalized prior residuals, IMU pre-integration factor residuals, lidar odometry factor residuals, and visual geolocation factor residuals are respectively (the subscripts are the covariance matrices of each).

[0121] The following is a specific embodiment.

[0122] Example 1: The present invention is based on a positioning method using UAV visual geolocation, lidar odometry and IMU, for navigation and positioning of UAVs in outdoor environments under GNSS denial. The specific implementation process is as follows.

[0123] Step 1: Using the Lancer EX-X4 UAV as the experimental platform, such as... Figure 4 As shown in Table 1, the equipment, sensors, and parameters used are as follows.

[0124] Table 1

[0125]

[0126] Step 2: Based on the methods and algorithms in the invention, use NVIDIA Jetson Orin NX as the host computer for the drone (Linux system and ROS environment), and follow the steps below to program and implement it.

[0127] Step 3: Define the lidar odometry coordinate system, the UAV initial coordinate system, and the visual geolocation coordinate system;

[0128] Step 4: Activate the LiDAR odometry. The LiDAR odometry outputs the positioning in the LiDAR odometry coordinate system at a frequency of 20 Hz, and simultaneously transforms the LiDAR coordinate system to the world coordinate system based on the initial positioning.

[0129] Step 5: Activate the visual geolocation module. The steps for activating the visual geolocation module are as follows:

[0130] Step 5.1: Based on the initial positioning information of the UAV, obtain the remote sensing map of the corresponding area from the remote sensing image database. The image is sent to the visual geolocation module, which takes the image captured by the camera at the current moment. Input into visual geolocation module

[0131] Step 5.2: Save the current positioning information of the lidar odometry.

[0132] Step 5.3: Send the remote sensing images into the visual geolocation module and drone camera images The first stage of the two-stage-multiscale image matching network is fed into the network to obtain the keypoint pair matching results. , ]

[0133] Step 5.4: Match the results based on the key points from the first stage. , The DBSCAN clustering method was used to obtain the region with the highest degree of mutual matching of keypoint pairs. This region was then cropped from both UAV images and remote sensing images to obtain local remote sensing images. and local drone camera images ;

[0134] Step 5.5: Transfer local remote sensing images and local drone camera images The keypoint pair matching results are obtained by feeding the data into the second stage of the image matching network. , ];

[0135] Step 5.6: Match the image results from the first stage with those from the second stage. , ]and[ , The images are stitched together to obtain the final keypoint pair matching results between the UAV camera image and the remote sensing image. , ];

[0136] Step 5.7: Based on the keypoint pair matching results of the final UAV camera image and remote sensing image [ , ] Calculate the homography matrix between UAV camera images and remote sensing images. ;

[0137] Step 5.8: Based on the homography matrix Calculate the latitude and longitude of the UAV in the remote sensing image and output its latitude and longitude positioning. ;

[0138] Step 6: Based on the positioning information of the LiDAR odometry stored in the visual geolocation module. And the current location information of the lidar odometry. Calculate the output positioning of the visual geolocation module at the current moment. ;

[0139] Step 7: Obtain the current location information from the lidar odometry. The current location output of the visual geolocation module The IMU uses a factor graph optimization method for fusion to obtain the fused UAV positioning.

[0140] Test environment such as Figure 5 As shown, the drone first flies in a straight line to the designated location, and then flies around a rectangle to collect data. The flight scenario is mainly an outdoor park scene.

[0141] Test results as follows Figure 6 As shown, red represents the ground truth positioning value, yellow represents the positioning from the visual geolocation module, and blue represents the fused positioning value. It can be seen that the positioning from the visual geolocation module is not significantly different from the ground truth positioning value, while the fused positioning value further enhances the UAV's positioning accuracy and real-time performance.

[0142] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A method for integrated navigation of unmanned aerial vehicles (UAVs) based on visual geolocation and odometry / IMU fusion, characterized in that... Includes the following steps: 1) A method for establishing a remote sensing image database: using a two-stage remote sensing image segmentation method based on Nona-Tree, the original remote sensing image is segmented into image blocks of 1024×1024 pixels, and further divided into sub-image blocks of 512×512 pixels to form an extended remote sensing image database. 2) A visual geolocation method: A two-stage, multi-scale scene matching method based on graph neural networks SuperPoint & SuperGLUE is used to realize the visual geolocation of UAVs; the first stage obtains candidate key point pairs through preliminary matching; the second stage improves the matching accuracy through clustering, cropping and rematching, and finally calculates the homography matrix and the latitude and longitude coordinates of the UAV. 3) A method for relative motion compensation in visual geolocation: using a lidar odometry to calculate the relative displacement between the start time stamp and the current time stamp of visual geolocation, and compensating for the visual geolocation results; 4) A method for fusion of visual geolocation, odometry and IMU based on factor graph: Define a state sequence including position, orientation, accelerometer random walk noise and gyroscope random walk noise; construct visual geolocation factor residuals, odometry factor residuals and IMU pre-integration factor residuals; construct a factor graph optimization objective function, use the iSAM2 optimization library for state estimation, and obtain the positioning result of integrated navigation.

2. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 1, characterized in that... In step 1), the method for establishing a remote sensing image database includes the following specific steps: (1.1) A two-stage remote sensing image segmentation method based on Nona-Tree is used; (1.2) The original remote sensing image is segmented into 1024×1024 pixel remote sensing images using a step size of 1024 pixels and a non-overlapping method; (1.3) Expand a set of 4×4 remote sensing images using the Nona-Tree method; (1.4) After creating an extended remote sensing image database with 1024×1024 pixels, each of these map tiles was further subdivided into remote sensing images of 512×512 pixels. (1.5) Each group of 4×4 remote sensing images is expanded using the Nona-Tree method to obtain the final remote sensing image database; each remote sensing image is 512×512 pixels in size.

3. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 1, characterized in that... In step 2), the visual geolocation method includes the following two stages; (2.1) In the first stage, the remote sensing image and the UAV image are matched at i different resolutions at multiple scales, and the i sets of image matching results are stitched together; (2.2) In the second stage, the most relevant regions of the remote sensing image and the UAV image are clustered and cropped, the image matching is performed again, and the image matching results of the first stage are stitched together. Finally, the homography matrix and the UAV location are calculated based on the matching results.

4. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 3, characterized in that... In step (2.1), the first stage specifically includes the following steps: (2.1.1) Based on the initial positioning and fused positioning, retrieve the corresponding remote sensing images from the remote sensing image database, and acquire images taken by the camera on the UAV. Simultaneously scale the two images to i different resolution scales. ]; (2.1.2) The scaled i groups of images are fed into a graph neural network for image matching to obtain the set of pixel coordinates of the matching point pairs at different scales. , ];in,[ , ] These represent the sets of pixel coordinates of matching point pairs detected in UAV images and remote sensing images, respectively; (2.1.3) Map the set of pixel coordinates of the matching point pairs obtained at different scales back to the original image. The mapping formula is as follows: in, This represents the matching key points mapped back to the original-size image. These represent the matching keypoints mapped back to the original size image. Axis coordinates and Axis coordinates and They represent the original state before mapping. Axis coordinates and Axis coordinates; after mapping the pixel coordinate sets of i sets of matching point pairs at different scales back to the original image, the i sets are concatenated to obtain the key point pair matching results of the first stage. , ].

5. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 3, characterized in that... In step (2.2), the second stage specifically includes the following steps: (2.2.1) Based on the mutually matching keypoint pairs obtained in the first stage [ , The DBSCAN method was used to cluster the regions with the highest degree of mutual matching of key point pairs, which are the regions most similar to the UAV images and satellite images. These regions were then bounded with bounding boxes and cropped to obtain the most relevant UAV images and satellite images in the local area. (2.2.2) The obtained locally most relevant UAV image and satellite image are fed into the graph neural network again for image matching to obtain the second set of image matching pixel coordinates. , The second set of image matching pixel coordinates is then mapped back to the original image to obtain the keypoint pair matching results for the second stage. , ]; (2.2.3) Match the image results from the first stage with those from the second stage. , ]and[ , The images are stitched together to obtain the final keypoint pair matching results between the UAV image and the satellite image. , ]; (2.2.4) Based on the key point matching results [ , Calculate the relationship between the pixel plane of the UAV imagery and the pixel plane of the satellite map tile, i.e., the homography matrix. The homography matrix is ​​calculated using a robust RANSAC-based method. If the RANSAC reprojection error of a point is less than a set threshold, the point is considered to be a correctly matched point. The process is iterated multiple times, and the candidate homography matrix with the most correctly matched points is recorded as the final homography matrix each time. After obtaining the homography matrix from the UAV image pixel coordinate system to the remote sensing image pixel coordinate system, the pixel coordinates of the UAV on the satellite image are directly calculated. (2.2.5) The latitude and longitude coordinates of the UAV can be calculated from the pixel coordinates of the UAV on the satellite image using the following formula: in,( , () represents the latitude and longitude coordinates of the upper left corner of the satellite map. , This represents the latitude and longitude coordinates of the lower right corner of the remote sensing image. , This indicates the width and height of the satellite map. , This represents the pixel coordinates of the drone on the remote sensing image.

6. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 5, characterized in that... In step (2.2.4), the threshold is set to 5 and the number of iterations is set to 1000.

7. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 1, characterized in that... In step 3), the specific steps of the visual geolocation relative motion compensation method are as follows: (3.1) Using lidar odometry to calculate the relative displacement between the start time stamp and the current time stamp of visual geolocation, compensating for visual geolocation: the visual geolocation module receives images... Frame, save Frame-based lidar odometry positioning ; after obtaining the location output from the visual geolocation module Frame, based on the location information output by the visual geolocation module timestamps in turn up Frame-preserved LiDAR odometry positioning Based on the current Frame-based lidar odometry positioning The method for calculating relative displacement is as follows: (3.2) Calculate the visual geolocation of the current frame: in, This indicates the location output of the visual geolocation module, which receives keyframes from the image. This indicates the output location of the visual geolocation module in the current frame.

8. The UAV integrated navigation method based on visual geolocation and odometry / IMU fusion as described in claim 1, characterized in that... In step 4), the method for visual geolocation, odometry, and IMU based on factor graph fusion specifically includes the following steps: (4.1) Define the state sequence as follows: The state sequence from frame i to frame j includes the motion state of the nth frame, consisting of four state variables: position. ,direction Accelerometer random walk noise and gyroscope random walk noise ; (4.2) Constructing visual geolocation factor residuals as follows: in, Observations representing visual geolocation, This indicates the compensated location after visual geolocation. (4.3) Constructing the odometry factor residuals and IMU pre-integral factor residuals ,in, Indicates the odometer's observations, Represents the observations of the IMU; (4.4) The objective function for factor graph optimization is constructed as follows: In argmin{.}, the marginalized prior residuals, IMU pre-integration factor residuals, lidar odometry factor residuals, and visual geolocation factor residuals are respectively represented by their respective subscripts as covariance matrices. The iSAM2 optimization library is used for state estimation to obtain the positioning results of the integrated navigation.

Images