Method, device and equipment for positioning unmanned aerial vehicle under GPS denial condition

Abstract

The invention relates to a method, a device and equipment for positioning an unmanned aerial vehicle under a GPS denial condition. The method comprises the steps of off-line synthetic data generation and on-line visual positioning. In the off-line synthetic data generation stage, comprehensive synthetic data including an RGB image and a depth map are generated from the texture three-dimensional model; in the corresponding online visual positioning stage, for each query image, a related image is retrieved by using an image retrieval algorithm in combination with rotation sensor prior, then a 2D-3D corresponding relation between the query image and the related image is established by using a feature detection and matching algorithm, and the attitude of the unmanned aerial vehicle is obtained by using a PnP RANSAC algorithm based on gravity guidance. By adopting the method, accurate positioning of the unmanned aerial vehicle under the condition of GPS denial is realized, and more accurate 6-DoF positioning of the unmanned aerial vehicle is realized by fully utilizing sensor priori.

Description

Technical Field

[0001] The present application relates to the field of unmanned aerial vehicle positioning technology, and in particular to a method, device and equipment for positioning an unmanned aerial vehicle under GPS denial conditions. Background Art

[0002] Global positioning of unmanned aerial vehicles (UAVs) plays an important role in various applications, such as cargo transportation, surveillance, and search and rescue. Most UAVs rely on the Global Navigation Satellite System (GNSS) for global positioning and navigation, but signals can be lost or erroneous due to environmental or human factors, affecting the accuracy of positioning.

[0003] Compared with ground-based visual localization, less attention has been paid to the pose estimation of UAVs in the absence of GPS signals. Some previous works have treated UAV localization as a scene recognition task, where the goal is to identify the image most similar to the query image in a geo-tagged database. The performance of this approach is affected by the density and distribution of images in the database, which may lead to significant errors. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device and equipment for positioning unmanned aerial vehicles under GPS denial conditions to address the above technical problems.

[0005] A method for positioning an unmanned aerial vehicle under GPS denial conditions, the method comprising:

[0006] Generate comprehensive synthetic reference data based on the textured 3D reference model; the synthetic reference data includes RGB images and depth maps.

[0007] According to the RGB image and a given query image, the rotation prior data of the UAV's built-in inertial measurement unit is used to restrict the search space and determine the related images of the query image.

[0008] Based on the depth map, query image and related images, feature detection and matching algorithms are used to establish a 2D-3D correspondence between the query image and the related images.

[0009] According to the 2D-3D correspondence, the gravity-guided PnP RANSAC method is adopted to obtain the 6-DoF positioning result based on the unmanned aerial vehicle.

[0010] In one embodiment, the textured 3D reference model reconstruction step includes:

[0011] High-resolution oblique photography images were captured by an unmanned aerial vehicle equipped with a five-eye oblique-viewing SHAREP SDK 102s camera. During the shooting process, the flight control system of the unmanned aerial vehicle automatically executed a pre-planned grid-fashion flight path.

[0012] Based on the oblique photography images, a textured mesh model is generated using 3D reconstruction technology, and it is aligned with the real geographic coordinate system using built-in RTK measurement and ground control points to obtain a textured 3D reference model.

[0013] In one embodiment, a scalable semi-automatic annotation method is used to generate a pseudo ground truth GT pose for a query image; wherein the specific steps of the scalable semi-automatic annotation method include:

[0014] Based on the oblique photography images and a query sequence consisting of multiple query images, a sparse point cloud is reconstructed using the structure-motion method.

[0015] Several tie points between the query sequence and the oblique photography images are manually selected in the sparse point cloud, and the registration is performed based on all tie points.

[0016] According to the registration result, bundle adjustment is used to refine the SFM block to obtain a complete 3D model including the oblique photography image and the query image.

[0017] In one embodiment, comprehensive synthetic reference data is generated based on the textured 3D reference model, including:

[0018] Based on the geographic boundaries of the 3D map, determine the area that requires a synthetic view.

[0019] Arrange virtual pilots on the three-dimensional map, establish two hyperparameters: horizontal spacing distance and vertical height, and dynamically adjust the two hyperparameters to evenly cover the entire area that requires synthetic views.

[0020] Setting multiple θ pitch and an appropriate yaw interval α theta to cover 360° of the same location.

[0021] A multi-view and multi-height rendering method is used to generate RGB images and depth maps.

[0022] In one embodiment, according to the RGB image and a given query image, the rotation prior data of the inertial measurement unit built into the unmanned aerial vehicle is used to limit the search space and determine the related images of the query image, including:

[0023] The rotation information of the UAV’s built-in inertial measurement unit is used as prior information; the restriction imposed on each rotation angle in the rotation information is:

[0024]

[0025]

[0026]

[0027]

[0028] Among them, γ o is the direction threshold, R S The rotation matrix representing the prior information, θ r represents the rotation angle of the reference image, θ roll ,θ yaw ,θ pitch are the roll angle, yaw angle, and pitch angle of the database image respectively.

[0029] Incorrect candidate data in the RGB image are pre-filtered according to prior information to obtain correct candidate data.

[0030] Based on the given query image and the correct candidate data, the relevant images of the query image are retrieved.

[0031] In one embodiment, a 2D-3D correspondence between the query image and the related image is established by using a feature detection and matching algorithm based on the depth map, the query image, and the related image, including:

[0032] A learnable feature matching method is adopted to establish 2D-2D correspondences between query images and related images.

[0033] According to the depth map and the 2D-2D correspondence, the 2D-3D correspondence between the query image and the related image is obtained through back-projection transformation.

[0034] In one embodiment, a gravity-guided PnP RANSAC method is used according to the 2D-3D correspondence to obtain a 6-DoF positioning result based on the unmanned aerial vehicle, including:

[0035] A gravity verification module is added to the PnP RANSAC process. In each RANSAC iteration, the deviation between the sensor posture and the gravity direction of the assumed posture is calculated. The deviation is:

[0036]

[0037] Among them, d ∈ is the deviation, is the sensor posture, Assume the posture.

[0038] If the deviation is less than the predefined stopping threshold, the iteration of RANSAC is stopped and a 6-DoF positioning result based on the UAV is obtained.

[0039] In one of the embodiments, the method further includes: using a hierarchical tracking system to track a designated object located on the ground according to a 6-DoF positioning result based on the unmanned aerial vehicle.

[0040] The hierarchical tracking system includes two cameras, one of which is a fixed wide-angle lens camera and the other is a zoom lens camera; the fixed wide-angle lens camera is used to; the hierarchical tracking system is used to use the fixed wide-angle lens camera to capture images for online 6-DoF positioning, and use the zoom lens to capture images for specific target detection. The target occupies a sufficiently large area, and then uses ray tracing technology to project the two-dimensional target on the image plane to its absolute position in the three-dimensional map.

[0041] A positioning device for an unmanned aerial vehicle under GPS denial conditions, the device comprising:

[0042] The synthetic reference data generation module is used to generate comprehensive synthetic reference data based on the textured three-dimensional reference model; the synthetic reference data includes an RGB image and a depth map.

[0043] The image retrieval module is used to determine the related images of the query image based on the RGB image and the given query image by using the rotation prior data of the self-inertial measurement unit built into the unmanned aerial vehicle to limit the search space.

[0044] The 2D-3D correspondence determination module is used to establish a 2D-3D correspondence between the query image and the related image using a feature detection and matching algorithm based on the depth map, the query image and the related image.

[0045] The positioning module of the unmanned aerial vehicle is used to obtain a 6-DoF positioning result based on the unmanned aerial vehicle by adopting a PnPRANSAC method based on gravity guidance according to the 2D-3D correspondence relationship.

[0046] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above method embodiments when executing the computer program.

[0047] The above-mentioned positioning method, device and equipment for unmanned aerial vehicles under GPS denial conditions, the method includes offline synthetic data generation and online visual positioning. In the offline synthetic data generation stage, comprehensive synthetic data including RGB images and depth maps are generated from the textured three-dimensional model; corresponding to the online visual positioning stage, for each query image, an image retrieval algorithm is used in combination with a rotation sensor prior to retrieve relevant images, and then a feature detection and matching algorithm is used to establish a 2D-3D correspondence between the query image and the relevant image, and a gravity-guided PnP RANSAC algorithm is used to obtain the posture of the UAV. This method is used to achieve accurate positioning of unmanned aerial vehicles under GPS denial conditions, and to make full use of sensor priors to achieve more accurate 6-DoF positioning of UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a method for positioning an unmanned aerial vehicle under GPS denial conditions in one embodiment;

[0049] Figure 2 A method flow for positioning an unmanned aerial vehicle under large data sets and GPS denial conditions in an embodiment;

[0050] Figure 3 A method flow for positioning an unmanned aerial vehicle under GPS denial conditions in an embodiment;

[0051] Figure 4 A schematic diagram of a hierarchical tracking system in another embodiment;

[0052] Figure 5 : are positioning error results of different trajectories in another embodiment, where (a) is the offset of the drone position in sequence 1, (b) is the offset of the drone angle in sequence 1, (c) is the offset of the drone tracking target in sequence 1, (d) is the offset of the drone position in sequence 2, (e) is the offset of the drone angle in sequence 2, and (f) is the offset of the drone tracking target in sequence 2;

[0053] Figure 6 is a structural block diagram of a positioning device for an unmanned aerial vehicle under GPS denial conditions in one embodiment;

[0054] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] Compared with ground-based visual localization, less attention has been paid to the pose estimation of UAVs in GPS-denied conditions (i.e., in environments where GPS signals are absent). Some previous works have treated UAV localization as a scene recognition task, where the goal is to identify the image most similar to the query image in a geotagged database. The performance of this approach is affected by the density and distribution of images in the database, which may lead to significant errors. Existing research results can be roughly divided into two categories: the first category involves using neural networks to implicitly encode the scene and regress the absolute pose of the input image. The advantages of these methods are small memory usage and fast inference time, but limited localization accuracy. The second category follows HLoc, explicitly using maps to render synthetic views on Google Earth and recover the 6-DoF pose through image retrieval, feature matching, and PnP RANSAC. This method explores richer rendering viewpoints to represent the scene and makes full use of sensor priors to achieve more accurate UAV 6-DoF localization.

[0057] Given a textured reference 3D model M with real-world geographic coordinates and a query image I q Sensor information S q , our goal is to estimate the query image I q 6-Dof Stance ζ q . To this end, a two-stage localization approach is proposed. Offline synthetic data generation is performed in the first stage, and online visual localization is performed in the second stage. In the offline stage, rendering techniques are used to obtain synthetic views and depth maps at different virtual viewpoints (different heights and directions) to fully present a dense 3D model of the scene. In the online visual localization stage, inspired by the recent terrestrial visual localization method SensLoc, the camera information is combined with prior data from the UAV device. Specifically, the rotation information of the built-in IMU is first used to restrict the search space to determine the relevant reference views of the query image. Then, with the help of the depth map, feature matching between the query image and the retrieved top-k database images is established and promoted to 2D-3D correspondences. Finally, gravity-guided PnP RANSAC is used to estimate the camera pose.

[0058] In one embodiment, Figure 1 As shown, a method for positioning an unmanned aerial vehicle under GPS denial conditions is provided, the method comprising the following steps:

[0059] Step 100: Generate comprehensive synthetic reference data based on the textured 3D reference model; the synthetic reference data includes an RGB image and a depth map.

[0060] Specifically, the dataset used in this method is a large dataset, which includes textured 3D reference models reconstructed from aerial oblique photography, covering an area of ​​about 2.5 million square meters. It contains various urban and rural scenes, including buildings, streets, vegetation, and lakes. The large dataset and the positioning method flow of unmanned aerial vehicles under GPS denial conditions are as follows: Figure 2 described.

[0061] A variety of synthetic data can be generated based on the textured 3D reference model, including rendered RGB and depth images, and digital surface maps (DSM).

[0062] Step 102: Based on the RGB image and a given query image, the rotation prior data of the built-in inertial measurement unit of the unmanned aerial vehicle is used to limit the search space and determine the related images of the query image.

[0063] Specifically, the unmanned aerial vehicle may be, but is not limited to, a drone.

[0064] The query image collection process includes: using a DJI M300RTK drone equipped with a DJI H20T camera to collect all query images {I q To improve the diversity of query images, images spanning different regions, flight altitudes, and capture angles were manually captured. In addition, to simulate the different shooting habits of drone pilots, two modes were used to capture query images: continuous shooting with an interval of 2 seconds and random manual shooting.

[0065] The query images were collected and sampled from five different flight trajectories, varying in altitude (50-200 m), viewpoint (15-70 degrees pitch), and acquisition location. Rather than relying on the drone’s built-in sensor information, such as Global Positioning System (GPS) or Real-Time Kinematics (RTK), to provide 3-DoF ground truth positions, the query sequences were registered to a reference map using manual tie points to obtain more accurate 6-DoF ground truth positions. In addition, sensor data, namely rotation priors from the Inertial Measurement Unit (IMU), were recorded to assist further localization algorithms.

[0066] There is a one-to-one correspondence between RGB images and depth images. The retrieval is only performed in the RGB image. If the corresponding RGB is not retrieved, its corresponding depth image will not proceed to the next stage.

[0067] Step 104: Using feature detection and matching algorithms based on the depth map, the query image, and the related images, a 2D-3D correspondence relationship between the query image and the related images is established.

[0068] Step 106: According to the 2D-3D correspondence, a PnP RANSAC method based on gravity guidance is adopted to obtain a 6-DoF positioning result based on the UAV.

[0069] In the above-mentioned method for positioning an unmanned aerial vehicle under GPS denial conditions, the method includes offline synthetic data generation and online visual positioning. In the offline synthetic data generation stage, comprehensive synthetic data including RGB images and depth maps are generated from the textured three-dimensional model; corresponding to the online visual positioning stage, for each query image, an image retrieval algorithm is used in combination with a rotation sensor prior to retrieve relevant images, and then a feature detection and matching algorithm is used to establish a 2D-3D correspondence between the query image and the relevant image, and a gravity-guided PnP RANSAC algorithm is used to obtain the posture of the UAV. This method is used to achieve accurate positioning of the unmanned aerial vehicle under GPS denial conditions, and to make full use of sensor priors to achieve more accurate 6-DoF positioning of the UAV.

[0070] In one embodiment, the textured three-dimensional reference model reconstruction step in step 100 includes: taking high-resolution oblique photography images by an unmanned aerial vehicle; the unmanned aerial vehicle is equipped with a five-eye oblique-viewing SHAREP SDK 102s camera; during the shooting process of the unmanned aerial vehicle, its flight control system automatically executes a pre-planned grid-fashion flight path; based on the oblique photography images, a textured mesh model is generated using three-dimensional reconstruction technology, and it is aligned with the real geographic coordinate system using built-in RTK measurement and ground control points to obtain a textured three-dimensional reference model.

[0071] Specifically, in order to draw the reference map, a DJI M300 RTK drone was used to capture high-resolution aerial images {I o}, the drone was equipped with a five-eye squint SHAREP SDK 102s camera. To ensure adequate and uniform coverage of the area, a grid-fashion flight path was pre-planned and automatically executed by the drone's flight control system. Then, modern 3D reconstruction techniques were applied to generate a textured mesh model, which was aligned to a real geographic coordinate system using built-in RTK measurements and ground control points.

[0072] In one embodiment, the query image in step 102 includes a pseudo ground truth GT pose. A scalable semi-automatic annotation method is used to generate a pseudo ground truth GT pose for the query image; wherein the specific steps of the scalable semi-automatic annotation method include: reconstructing a sparse point cloud using a structure-motion method based on an oblique photography image and a query sequence consisting of multiple query images; manually selecting a number of connection points between the query sequence and the oblique photography image in the sparse point cloud, and aligning all the connection points; and refining the SFM block using a bundle adjustment based on the result of the alignment to obtain a complete 3D model including the oblique photography image and the query image.

[0073] Specifically, in this work, we adopt a scalable semi-automatic annotation method to generate pseudo ground truth GT poses for query images. Our method can generate pose annotations for hundreds of query images with minimal human intervention. The main steps of the method are as follows: First, we use the structure-from-motion (SfM) method to extract the poses from the oblique photography images {I o} and multiple query sequences {I q}. Next, we manually select some tie points between the query sequence and the oblique images and perform registration based on these tie points. Finally, we refine the SfM block using bundle adjustment to obtain a complete 3D model that contains both the oblique images and the query image.

[0074] To evaluate the accuracy of the textured 3D reference model M, we report the median reprojection error and the median reprojection error per tie point, which are 0.82 pixels and 0.62 pixels, respectively. In addition, to ensure the accuracy of the pseudo-GT pose, we also perform a visual inspection and select images where the pseudo-GT pose rendered images are aligned with the original query image.

[0075] In one embodiment, step 100 includes: determining an area requiring a synthetic view according to the geographic boundaries of a three-dimensional map; arranging virtual pilots on the three-dimensional map, establishing two hyperparameters of a horizontal spacing distance and a vertical height, and dynamically adjusting the two hyperparameters to uniformly cover the entire area requiring a synthetic view; setting multiple θ pitch and an appropriate yaw interval α theta To cover 360° of the same location; a multi-view and multi-height rendering method is used to generate RGB images and depth maps.

[0076] Specifically, the accuracy of drone positioning is greatly affected by the dataset. Existing drone datasets face several challenges that limit their effectiveness in advancing the field. These challenges include: 1) Limited degrees of freedom. Many datasets, such as University-1652 and ALTO, only provide 3 degrees of freedom (DoF) for positioning, which may not meet the needs of some applications. 2) Bird's-eye view. Some datasets only focus on overhead images generated by Google Earth or real drones. However, these datasets do not represent the various scenarios in which drones are used in the real world. 3) Regression-based methods Datasets such as CrossLoc provide a rich data source for regression-based positioning methods, including real images, synthetic images, and semantic information. However, these methods may not be suitable for structure-based positioning algorithms.

[0077] Different from the ground positioning case, the query image I q and the oblique image {I o} are often taken at very different positions and orientations, which poses a huge challenge to image retrieval and feature matching algorithms. To address this challenge, we generate comprehensive synthetic reference data inspired by the work on view synthesis, including RGB images {I r} and depth map {D r} to represent a dense map of the scene.

[0078] Specifically, we first use the geographic boundary M of the 3D map to determine the area where the synthetic view is needed. Then, the generation of synthetic data can be divided into two parts:

[0079] 1) For translation, place virtual viewpoint ζ above the 3D map and establish a horizontal spacing distance α t and vertical height H, which are dynamically adjusted to ensure uniform coverage of the entire geographic area.

[0080] 2) For rotation, we set multiple θ pitch and an appropriate yaw interval α theta To cover 360° of the same location. The rendering method based on multi-view and multi-height can enhance the adaptability to the scale invariance and rotation invariance of the query image.

[0081] In one embodiment, step 102 includes: using the rotation information of the UAV's built-in inertial measurement unit as prior information; wherein the restriction imposed on each rotation angle in the rotation information is:

[0082]

[0083] Among them, γ o is the direction threshold, R SThe rotation matrix representing the prior information, θ r represents the rotation angle of the reference image, θ roll ,θ yaw ,θ pitch are the roll angle, yaw angle, and pitch angle of the database image respectively.

[0084] Incorrect candidate data in the RGB image are pre-filtered according to prior information to obtain correct candidate data; and search is performed according to a given query image and the correct candidate data to obtain images related to the query image.

[0085] Specifically, given a query image I q , our goal is to identify a set of commonly seen images s L={I r1 ,I r2 ,...I rk}. The general solution consists in using an embedding function f(·) to transform the image I q ,{Ir} is mapped to a compact feature space and then the distance metric d(f(I p ),f(I r )), search I q However, simply using global features to find retrieval terms may lead to some errors because different drone images have obvious perspective and scale variations.

[0086] Inspired by SensLoc, to improve the accuracy of retrieval, we use the metadata of the drone sensor S, especially the rotation information, as prior information to pre-filter incorrect reference candidates. Considering the richer perspective of the drone, we impose restrictions on each rotation angle, including θ roll ,θ yaw ,θ pitch , as shown in formula (1). After using the rotation information as prior information to pre-filter the incorrect candidate data, according to d(f(I q ),f( p L)) Efficiently and accurately determine k nearest neighbors s L.

[0087] In one embodiment, step 104 includes: using a learnable feature matching method to establish a 2D-2D correspondence between the query image and the related image; and obtaining a 2D-3D correspondence between the query image and the related image through back-projection transformation based on the depth map and the 2D-2D correspondence.

[0088] Specifically, a learnable feature matching technique is used to q and the retrieved synthetic image sA 2D-2D correspondence is established between the depth map L. s D is also rendered during the offline stage of synthetic data generation, so these 2D-2D correspondences can be converted to 2D-3D correspondences via back-projection.

[0089] In one embodiment, step 106 includes: adding a gravity verification module to the PnP RANSAC process, and calculating the deviation between the sensor pose and the gravity direction of the assumed pose in each RANSAC iteration, where the deviation is:

[0090]

[0091] Among them, d ∈ is the deviation, is the sensor posture, Assume the posture.

[0092] If the deviation is less than the predefined stopping threshold, the iteration of RANSAC is stopped and a 6-DoF positioning result based on the UAV is obtained.

[0093] In one embodiment, the method further includes: using a hierarchical tracking system to track a designated object (such as a pedestrian or a vehicle) located on the ground according to the 6-DoF positioning result based on the unmanned aerial vehicle. Figure 4 The layered tracking system shown includes two cameras, one of which is a fixed wide-angle lens camera and the other is a zoom lens camera; the fixed wide-angle lens camera is used to; the layered tracking system is used to use the fixed wide-angle lens camera to capture images for online 6-DoF positioning, and use the zoom lens to capture images for specific target detection, the target occupies a sufficiently large area, and then use ray tracing technology to project the two-dimensional target on the image plane to its absolute position in the three-dimensional map.

[0094] Specifically, both the fixed wide-angle lens camera and the zoom lens camera are carefully calibrated. The wide-angle lens has a wider field of view and can be used for global and local feature extraction, so the wide-angle lens camera is used to recover the 6-DoF attitude of the drone, while the zoom lens camera is used to accurately detect the target. Ray tracing technology is used to track the specified target in three-dimensional space.

[0095] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In a verification embodiment, during the generation of offline synthetic reference data, the absolute heights H=100m and 150m and the horizontal interval α are generated respectively. t =50m and α t = 75m virtual viewpoint, and generate 16 images at different angles for each same position ( or 45°, α theta =45°). In the image retrieval step, the angle threshold γ o In the gravity-guided PnP solution step, we set the gravity angle threshold γ ∈ = 2°. All experiments were performed on a computer equipped with an Intel Core i9-11900K processor, a GTX 3090 graphics card (24GB memory), and an Ubuntu 18.04 operating system. The proposed method was implemented using only Python and Pytorch.

[0097] It is worth mentioning that gravity refers to the pitch angle and roll angle. When the drone is flying, the drift will not be very large at these two angles.

[0098] Baseline model of this example: The performance of multiple global descriptors in the image retrieval stage is evaluated, including NetVLAD, AP-GeM, and OpenIBL. In addition, a control experiment is performed to verify the effectiveness of using the rotation sensor prior.

[0099] (1) Evaluation scheme: If the retrieval result has enough overlap with the query image and the overlap percentage P overlap If the overlap is greater than 50%, the retrieval result is considered correct. The overlap area between the two images is calculated by converting the pixels from one image to the other based on the depth map.

[0100] (2) Results: The retrieval results are shown in Table 1. A hierarchical list of size k is maintained for each query, and the recall and precision metrics are reported. The experimental results show that all three representative retrieval methods can find the correct retrieval items, even when the rotation sensor is not adjusted. This can be attributed to the generation of a comprehensive virtual view of the entire scene. Among all the options, OpenIBL, with a hierarchical list of k = 3 and the use of sensor priors, leads the benchmark by a large margin and is therefore selected as the method for finding the visible reference set in the localization method.

[0101] Table 1 Search results

[0102]

[0103] (3) Visual positioning

[0104] The performance of various keypoint descriptors, including SIFT and Superpoint, as well as nearest neighbor matchers and learnable matchers such as SuperGlue, are evaluated. The results are also compared with those obtained using the detector-free matcher LoFTR. For pose estimation, a PnP RANSAC solver is used, both with and without gravity information.

[0105] The evaluation of this embodiment follows the standard positioning benchmark protocol.

[0106] The quantitative results are shown in Table 2. Super-point+Superglue outperforms other similar algorithms in all indicators except (1m, 1°). This may be due to the fact that the query images of the large dataset (UAVD4L) are taken under good lighting conditions and have rich textures, which is conducive to super-point detection of key points. In addition, we also found that the gravity-guided PnP RANSAC method can improve the positioning accuracy regardless of the matching algorithm used.

[0107] To evaluate the object tracking performance of our method, a person and a car are captured from two trajectories and the RTK recorded data are used as their GT positions.

[0108] The translation difference between the ground truth 3D coordinates and the calculated 3D coordinates is used to represent the target tracking estimation error.

[0109] The positioning error results of different trajectories are as follows: Figure 5 As shown, Figure 5 Two sequential results of drone localization and target tracking are shown, with and without using sensor information for localization. Figure 5(a) is the offset of the drone position in sequence 1, (b) is the offset of the drone angle in sequence 1, (c) is the offset of the drone tracking target in sequence 1, (d) is the offset of the drone position in sequence 2, (e) is the offset of the drone angle in sequence 2, and (f) is the offset of the drone tracking target in sequence 2. Due to the accurate positioning of the drone, the target position reaches meter-level accuracy. In this embodiment, the accuracy of target tracking when using rotation matrix guidance and when not using rotation guidance is compared in two different sequences. The target tracking accuracy based on the rotation matrix guidance positioning result is higher and the offset from the real-world position is smaller.

[0110] In order to fully understand the impact of different levels of rendering settings on visual localization, we evaluated three rendering settings, and the ablation experiment results are shown in Table 3. Only single-layer height and angle rendering achieved the lowest accuracy, while multi-layer pose and angle rendering achieved the highest accuracy. This shows that adding rendering layers by height and angle will improve the results. We conducted the above experiments on UAVD4L using the same image retrieval (OpenIBL), feature point detection and matching (SPP+SPG), and gravity-guided PnPRANSAC.

[0111] Table 3. Ablation experiment results

[0112]

[0113] In one embodiment, Figure 6 As shown, a positioning device for an unmanned aerial vehicle under GPS denial conditions is provided, comprising: a synthetic reference data generation module, an image retrieval module, a 2D-3D correspondence relationship determination module and an unmanned aerial vehicle positioning module, wherein:

[0114] The synthetic reference data generation module is used to generate comprehensive synthetic reference data based on the textured three-dimensional reference model; the synthetic reference data includes an RGB image and a depth map.

[0115] The image retrieval module is used to determine the related images of the query image based on the RGB image and the given query image by using the rotation prior data of the self-inertial measurement unit built into the unmanned aerial vehicle to limit the search space.

[0116] The 2D-3D correspondence determination module is used to establish a 2D-3D correspondence between the query image and the related image using a feature detection and matching algorithm based on the depth map, the query image and the related image.

[0117] The positioning module of the unmanned aerial vehicle is used to obtain a 6-DoF positioning result based on the unmanned aerial vehicle by adopting a PnPRANSAC method based on gravity guidance according to the 2D-3D correspondence relationship.

[0118] In one of the embodiments, the textured three-dimensional reference model reconstruction step in the synthetic reference data generation module includes: taking high-resolution oblique photography images by an unmanned aerial vehicle; the unmanned aerial vehicle is equipped with a five-eye oblique-viewing SHAREPSDK 102s camera; during the shooting process of the unmanned aerial vehicle, its flight control system automatically executes a pre-planned grid-fashion flight path; based on the oblique photography images, a textured mesh model is generated using three-dimensional reconstruction technology, and it is aligned with the real geographic coordinate system using built-in RTK measurement and ground control points to obtain a textured three-dimensional reference model.

[0119] In one embodiment, a query image in an image retrieval module includes a pseudo ground truth GT pose. A scalable semi-automatic annotation method is used to generate a pseudo ground truth GT pose for the query image; wherein the specific steps of the scalable semi-automatic annotation method include: reconstructing a sparse point cloud using a structure-motion method based on an oblique photography image and a query sequence consisting of multiple query images; manually selecting a number of connection points between the query sequence and the oblique photography image in the sparse point cloud, and aligning all the connection points; and refining the SFM block using a bundle adjustment based on the result of the alignment to obtain a complete 3D model including the oblique photography image and the query image.

[0120] In one embodiment, the synthetic reference data generation module is further used to determine the area requiring synthetic view according to the geographical boundary of the three-dimensional map; arrange virtual pilots on the three-dimensional map, establish two hyperparameters of horizontal spacing distance and vertical height, and dynamically adjust the two hyperparameters to evenly cover the entire area requiring synthetic view; set multiple θ pitch and an appropriate yaw interval α theta To cover 360° of the same location; a multi-view and multi-height rendering method is used to generate RGB images and depth maps.

[0121] In one embodiment, the image retrieval module is further used to use the rotation information of the built-in inertial measurement unit of the unmanned aerial vehicle as prior information; wherein the restriction imposed on each rotation angle in the rotation information is as shown in formula (1); incorrect candidate data in the RGB image is pre-filtered according to the prior information to obtain correct candidate data; and a search is performed according to a given query image and the correct candidate data to obtain a related image of the query image.

[0122] In one of the embodiments, the 2D-3D correspondence determination module is further used to establish a 2D-2D correspondence between the query image and the related image using a learnable feature matching method; and obtain the 2D-3D correspondence between the query image and the related image through back projection transformation based on the depth map and the 2D-2D correspondence.

[0123] In one of the embodiments, the positioning module of the unmanned aerial vehicle is also used to add a gravity verification module to the PnP RANSAC process. In each RANSAC iteration, the deviation between the sensor posture and the gravity direction of the assumed posture is calculated, and the deviation is shown in formula (2); if the deviation is less than a predefined stop threshold, the iteration of RANSAC is stopped to obtain a 6-DoF positioning result based on the unmanned aerial vehicle.

[0124] In one embodiment, the device further includes a target tracking module, which is used to track a specified object located on the ground using a hierarchical tracking system based on the 6-DoF positioning result based on the unmanned aerial vehicle. The hierarchical tracking system includes two cameras, one of which is a fixed wide-angle lens camera and the other is a zoom lens camera; the fixed wide-angle lens camera is used to; the hierarchical tracking system is used to use the fixed wide-angle lens camera to capture images for online 6-DoF positioning, and use the zoom lens to capture images for specific target detection, the target occupies a sufficiently large area, and then use ray tracing technology to project the two-dimensional target on the image plane to its absolute position in the three-dimensional map.

[0125] For the specific definition of the positioning device of the unmanned aerial vehicle under GPS denial conditions, please refer to the definition of the positioning method of the unmanned aerial vehicle under GPS denial conditions mentioned above, which will not be repeated here. The various modules in the positioning device of the unmanned aerial vehicle under GPS denial conditions mentioned above can be implemented in whole or in part through software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for positioning an unmanned aerial vehicle under GPS denial conditions is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0127] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for positioning an unmanned aerial vehicle under GPS denial conditions, characterized in that: The method comprises: Generating comprehensive synthetic reference data based on the textured three-dimensional reference model; the synthetic reference data includes an RGB image and a depth map; According to the RGB image and a given query image, a priori rotation data of an inertial measurement unit built into the unmanned aerial vehicle is used to limit the search space and determine the related images of the query image; Establishing a 2D-3D correspondence between the query image and the related image using a feature detection and matching algorithm based on the depth map, the query image, and the related image; According to the 2D-3D correspondence, a PnP RANSAC method based on gravity guidance is adopted to obtain a 6-DoF positioning result based on the unmanned aerial vehicle.

2. The method according to claim 1, characterized in that The textured three-dimensional reference model reconstruction step comprises: High-resolution oblique photography images are captured by an unmanned aerial vehicle; the unmanned aerial vehicle is equipped with a five-eye oblique-viewing SHAREP SDK 102s camera; during the capture process of the unmanned aerial vehicle, its flight control system automatically executes a pre-planned grid-fashion flight path; A textured mesh model is generated based on the oblique photography image using a three-dimensional reconstruction technique, and is aligned with a real geographic coordinate system using built-in RTK measurements and ground control points to obtain a textured three-dimensional reference model.

3. The method according to claim 1, characterized in that A scalable semi-automatic annotation method is used to generate a pseudo ground truth GT pose for the query image; wherein the specific steps of the scalable semi-automatic annotation method include: Reconstructing a sparse point cloud using a structure-motion method according to the oblique photography image and a query sequence consisting of a plurality of query images; Manually selecting a number of connection points between the query sequence and the oblique photography image in the sparse point cloud, and performing registration based on all the connection points; According to the registration result, bundle adjustment is used to refine the SFM block to obtain a complete 3D model including the oblique photography image and the query image.

4. The method according to claim 1, characterized in that: Generate comprehensive synthetic reference data from textured 3D reference models, including: Determine the area requiring the synthetic view according to the geographic boundaries of the three-dimensional map; Arrange virtual pilots on the 3D map, establish two hyperparameters: horizontal spacing and vertical height, and dynamically adjust the two hyperparameters to evenly cover the entire area that requires synthetic views. Setting multiple θ pitch and an appropriate yaw interval α theta To cover 360° of the same location; A multi-view and multi-height rendering method is used to generate RGB images and depth maps.

5. The method according to claim 1, characterized in that According to the RGB image and a given query image, the rotation prior data of the built-in inertial measurement unit of the unmanned aerial vehicle is used to limit the search space and determine the related images of the query image, including: The rotation information of the UAV’s built-in inertial measurement unit is used as prior information; the restriction imposed on each rotation angle in the rotation information is: Among them, γ o is the direction threshold, R S The rotation matrix representing the prior information, θ r represents the rotation angle of the reference image, θ roll ,θ yaw ,θ pitch are the roll angle, yaw angle, and pitch angle of the database image respectively; Pre-filtering incorrect candidate data in the RGB image according to the prior information to obtain correct candidate data; A search is performed based on a given query image and the correct candidate data to obtain images related to the query image.

6. The method according to claim 1, characterized in that Using a feature detection and matching algorithm according to the depth map, the query image, and the related image, a 2D-3D correspondence relationship between the query image and the related image is established, including: Using a learnable feature matching method, a 2D-2D correspondence relationship is established between the query image and the related image; A 2D-3D correspondence relationship between the query image and the related image is obtained through back-projection transformation according to the depth map and the 2D-2D correspondence relationship.

7. The method according to claim 1, characterized in that According to the 2D-3D correspondence, a PnP RANSAC method based on gravity guidance is adopted to obtain a 6-DoF positioning result based on the unmanned aerial vehicle, including: A gravity verification module is added to the PnP RANSAC process. In each RANSAC iteration, the deviation between the sensor posture and the gravity direction of the assumed posture is calculated. The deviation is: Among them, d ∈ is the deviation, is the sensor posture, To assume a posture; If the deviation is less than a predefined stopping threshold, the iteration of RANSAC is stopped to obtain a 6-DoF positioning result based on the unmanned aerial vehicle.

8. The method according to claim 1, characterized in that The method further comprises: According to the 6-DoF positioning results based on the UAV, a hierarchical tracking system is used to track the designated object on the ground; The hierarchical tracking system includes two cameras, one of which is a fixed wide-angle lens camera and the other is a zoom lens camera; the fixed wide-angle lens camera is used to; the hierarchical tracking system is used to use the fixed wide-angle lens camera to capture images for online 6-DoF positioning, and use the zoom lens to capture images for specific target detection, the target occupies a sufficiently large area, and then use ray tracing technology to project the two-dimensional target on the image plane to its absolute position in the three-dimensional map.

9. A positioning device for unmanned aerial vehicles under GPS denial conditions, characterized in that: The device comprises: A synthetic reference data generation module, used to generate comprehensive synthetic reference data according to the textured three-dimensional reference model; the synthetic reference data includes an RGB image and a depth map; An image retrieval module is used to determine the related images of the query image based on the RGB image and a given query image by using the rotation prior data of the self-inertial measurement unit built into the unmanned aerial vehicle to limit the search space; a 2D-3D correspondence determination module, configured to establish a 2D-3D correspondence between the query image and the related image using a feature detection and matching algorithm according to the depth map, the query image and the related image; The unmanned aerial vehicle positioning module is used to obtain a 6-DoF positioning result based on the unmanned aerial vehicle by adopting a PnPRANSAC method based on gravity guidance according to the 2D-3D correspondence relationship.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images