A method and apparatus for UAV visual localization based on combined guided model image matching

Abstract

This invention discloses a method and apparatus for UAV visual localization based on image matching using a combined guided model. The method includes: constructing a dense scene model based on a mesh map; using images captured during real-time UAV flight as query images and inputting these query images into an image dataset for matching to find multiple feature images similar to the query images; using a combined guided model to estimate the correspondence between the query images and each feature image to generate multiple sets of 2D-2D matching image pairs; rendering the corresponding depth maps in real time; converting each 2D-2D matching pair into a 2D-3D matching pair based on the depth maps; and calculating the corresponding attitude information of the UAV using the 2D-3D matching pairs to achieve UAV localization. This invention can reduce storage space overhead, lower the overall model runtime, and exhibits strong robustness and scalability, while also improving the accuracy of UAV visual localization.

Description

Technical Field

[0001] This invention relates to the field of UAV visual positioning technology, specifically to a UAV visual positioning method and apparatus based on combined guided model image matching. Background Technology

[0002] Visual positioning technology for unmanned aerial vehicles (UAVs) is a method that uses computer vision to estimate the position and attitude of UAVs. It integrates multiple functions of computer vision, including measurement, image matching, and spatial perception, and can be applied to fields such as navigation for autonomous vehicles, localization of unmanned aerial vehicles (UAVs), and path planning in complex environments. The core objective of visual positioning technology is to accurately determine the position of objects by analyzing images captured by cameras in a predetermined environment. UAVs can exhibit a variety of roll angles and flight speeds during flight, demonstrating superior flexibility and agility compared to other unmanned platforms.

[0003] Currently, UAV visual positioning typically relies on Structure from Motion (SfM) to generate sparse point cloud models. This involves determining the spatial and geometric relationships of a target based on a series of images captured during camera movement. This approach can be broadly categorized into two types: The first type uses Convolutional Neural Networks (CNNs) to directly regress camera pose information. While CNN-based methods allow for end-to-end training, their generalization ability is poor, and performance heavily depends on the distribution of scene image data. Therefore, when expanding to new scenes, additional training or adaptive adjustments are usually required. The second type aims to obtain 2D and 3D coordinate matching between query images and database models, using algorithms such as PnP to estimate camera pose. This typically involves first extracting feature points and descriptors from the images to match two related images, then calculating the 3D coordinates of the feature points through triangulation to obtain sparse point cloud information. In the SfM methods described above, the constructed sparse point cloud is tied to the selected feature point extraction and matching method. Changing to other methods requires recalculating the matching pairs between database images, which can be extremely time-consuming for large-scale scene construction. Meanwhile, in the UAV pose estimation stage, a dense 3D model corresponding to a depth map is required. Therefore, it's necessary to further utilize sparse point clouds to construct a dense point cloud, a process that also requires significant time and storage space. Thus, traditional SfM-based UAV visual localization methods consume substantial time and storage resources, and the accuracy of feature matching algorithms limits their effectiveness in reconstructing the 3D scene. Other visual localization methods, such as Meshloc based on dense 3D meshes, utilize mesh models and employ image retrieval and feature matching techniques to recover the camera's pose; similarly, they are limited by the accuracy of feature matching algorithms.

[0004] Traditional image matching algorithms, often designed for specific application scenarios, suffer from poor versatility and low accuracy in other applications. For example, the LoFTR (Local Feature Transformer) algorithm addresses poor matching results in weakly textured regions by combining a Transformer with contextual information, while SDF2 (Signed Distance Function 2) tackles long-term visual localization problems by classifying feature points using semantic information. However, these individual algorithms are only suitable for image matching in specific visual problems and are not applicable to other scenarios. Directly using pre-trained models for various application scenarios may lead to numerous mismatches, while training separate models for different scenarios increases implementation complexity. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a UAV visual positioning method and device based on combined guided model image matching, which has low time and storage resource overhead, high positioning accuracy, strong robustness and wide applicability. It can reduce the data storage space and time occupation, and improve the accuracy and flexibility of UAV visual positioning.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A UAV visual localization method based on combined guided model image matching includes:

[0008] S1, Construct a dense scene model based on a mesh map using scene point cloud data obtained from 3D LiDAR scanning on a drone;

[0009] S2, take the images captured during the real-time flight of the UAV as the query images, and input the query images into the image dataset for matching, so as to find multiple feature images that are close to the query images in the image dataset;

[0010] S3, use a combined guided model to estimate the correspondence between the query image and each feature image to generate multiple sets of 2D-2D matching image pairs. The combined guided model is composed of two or more matching models, and one of the matching models is selected as the guided model. The other models besides the guided model are used to filter the preliminary matching pairs during the matching process, and then the preliminary matching pairs are filtered a second time to obtain the final matching pairs.

[0011] S4. A depth map is rendered in real time based on the dense scene model. Each 2D-2D matching pair is converted into a 2D-3D matching pair based on the depth map. The corresponding attitude information of the UAV is calculated through the 2D-3D matching pair to achieve UAV positioning.

[0012] Further, step S1 includes:

[0013] 3D point cloud data of real-world scenes are acquired using radar carried by drones;

[0014] By reconstructing the 3D point cloud data using a Poisson surface, a triangular mesh surface corresponding to the 3D point cloud data is generated, thereby obtaining a dense scene model based on a mesh map corresponding to the real scene.

[0015] Furthermore, step S2 also includes constructing an image dataset, the steps of which include:

[0016] Images are captured at multiple locations in a simulated environment using cameras mounted on a simulated drone. During the capture process, the drone's yaw angle, pitch angle, and altitude are continuously adjusted, and the camera's real-time 6D attitude information and intrinsic parameters are recorded to obtain multiple simulated images labeled with the drone's attitude information. These multiple simulated images constitute an image dataset of the model environment.

[0017] Further, in step S2, inputting the query image into the image dataset for matching includes:

[0018] Local features of each captured image are obtained and clustered to obtain global features of each captured image;

[0019] The global features of each captured image are matched with the feature descriptors of each image in the image dataset to find the multiple feature images that are closest to each captured image.

[0020] Further, step S3 includes:

[0021] Two or more matching models are combined, and one of the matching models is selected as the guiding model to form a combined guiding model.

[0022] Based on the combined guidance model, multiple bidirectional matching pairs are obtained by matching the query image and each feature image using each matching model respectively;

[0023] Among the bidirectional matching pairs obtained by matching using the various matching models, matching pairs that satisfy the cycle consistency constraint are selected. A guided model is used to estimate the basis matrix. Based on the basis matrix, epipolar geometric constraints are applied to each of the bidirectional matching pairs obtained by matching using the other models besides the guided model, so as to initially select multiple preliminary matching pairs between the query image and each of the feature images.

[0024] Select matching pairs that satisfy the cycle consistency constraint from each of the preliminary matching pairs to estimate the fundamental matrix again, and select matching pairs that satisfy the epipolar geometry constraint again based on the fundamental matrix to obtain the final matching pairs.

[0025] Furthermore, the expression for the cycle consistency constraint is:

[0026]

[0027] ,

[0028] In the above formula, To determine the coordinates of point i in image A during forward matching from the captured image to the feature image, Let i be the coordinates of the corresponding point i in feature image B. Let i be the coordinates of point i in image A during backward matching from feature image to captured image. Let i be the coordinates of the corresponding point i in feature image B. This is the first threshold.

[0029] Furthermore, the expression for the epipolar geometric constraint is:

[0030]

[0031] ,

[0032] In the above formula, To map the feature points in image A to the straight lines in feature image B, Based on the matrix, The polar distance. This is the second threshold.

[0033] Further, step S4 includes:

[0034] The pixel coordinates of multiple feature images corresponding to key points of the query image in each 2D-2D matching pair are calculated using the depth map, and their corresponding three-dimensional coordinate values ​​in the dense scene model are obtained. The three-dimensional coordinate values ​​are then processed by minimizing the reprojection error to obtain the final three-dimensional coordinates, thereby converting the 2D-2D matching pair into a 2D-3D matching pair.

[0035] The pose of the UAV camera is solved by using the coordinates of the 3D points in the 2D-3D matching pair, the coordinates of the corresponding 2D points, and the intrinsic parameter matrix, so as to calculate the pose information of the UAV at the time of the current image capture and realize UAV positioning.

[0036] Furthermore, the present invention also provides a UAV visual positioning device based on combined guided model image matching, comprising:

[0037] The dense scene model building module is used to build a dense scene model based on a mesh map based on scene point cloud data obtained by scanning with a 3D LiDAR mounted on a drone.

[0038] The first image matching module is used to take the images captured during the real-time flight of the UAV as query images and input the query images into the image dataset for matching, so as to find multiple feature images that are close to the query images in the image dataset;

[0039] The second image matching module is used to estimate the correspondence between the query image and each feature image using a combined guided model to generate multiple sets of 2D-2D matching image pairs. The combined guided model is composed of two or more matching models, and one of the matching models is selected as the guided model. The other models besides the guided model are used to filter the preliminary matching pairs during the matching process, and then the preliminary matching pairs are filtered a second time to obtain the final matching pairs.

[0040] The information extraction module is used to render a depth map in real time based on the dense scene model, convert each 2D-2D matching pair into a 2D-3D matching pair based on the depth map, and calculate the corresponding attitude information of the UAV through the 2D-3D matching pair to achieve UAV positioning.

[0041] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to execute the above-described UAV visual localization method based on combined guided model image matching.

[0042] Compared with the prior art, the advantages of the present invention are as follows:

[0043] This invention constructs a dense scene model based on a mesh map and obtains a depth map through rendering. By rendering the depth map in real time instead of storing all depth maps, storage space overhead and overall model runtime can be reduced. Simultaneously, by matching real-time captured images as query images with an image dataset, feature images in the image dataset that are similar to the query image are found. This allows for preliminary filtering of similar images, reducing the number of images processed and improving efficiency. Furthermore, a combined guided model is used to estimate the correspondence between the query image and each feature image, generating 2D-2D matching image pairs. This overcomes the low matching accuracy of single algorithms, resulting in robust, repeatable, and accurate matching results. The guided model further improves the efficiency and accuracy of matching by guiding the selection process of matching image pairs, thereby enhancing the accuracy and efficiency of UAV visual positioning. Attached Figure Description

[0044] Figure 1 This is a flowchart of the UAV visual localization method based on combined guided model image matching in this embodiment.

[0045] Figure 2 This is a schematic diagram illustrating the process of image matching using a combined guided model in a specific application embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of the overall process of the UAV visual positioning method based on combined guided model image matching in a specific application embodiment of the present invention. Detailed Implementation

[0047] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0048] like Figure 1 As shown, the steps of the UAV visual localization method based on combined guided model image matching in this embodiment include:

[0049] S1, Construct a dense scene model based on a mesh map using scene point cloud data obtained from 3D LiDAR scanning on a drone;

[0050] S2, take the images captured during the real-time flight of the drone as query images, and input the query images into the image dataset for matching, so as to find multiple feature images that are similar to the captured images in the image dataset;

[0051] S3. Use a combined guided model to estimate the correspondence between the query image and each feature image to generate multiple sets of 2D-2D matching image pairs. The combined guided model is composed of two or more matching models, and one of the matching models is selected as the guided model. The other models except the guided model are used to filter the preliminary matching pairs during the matching process, and then the preliminary matching pairs are filtered a second time to obtain the final matching pairs.

[0052] S4 renders a depth map in real time based on the dense scene model, converts each 2D-2D matching pair into a 2D-3D matching pair based on the depth map, and calculates the corresponding attitude information of the UAV through the 2D-3D matching pair to achieve UAV positioning.

[0053] It is understood that this embodiment constructs a dense scene model based on a mesh map and obtains a depth map through rendering. By rendering the depth map in real time instead of storing all depth maps, the storage space overhead can be reduced and the overall running time of the model can be shortened. At the same time, by matching the real-time captured images as query images with the image dataset, feature images in the image dataset that are close to the query images can be found. Similar images can be initially filtered to reduce the number of images processed and improve efficiency. Furthermore, by using a combined guided model to estimate the correspondence between the query images and each feature image to generate 2D-2D matching image pairs, the shortcomings of low matching accuracy of single algorithms can be overcome, resulting in matching results with strong robustness, high repeatability, and high matching accuracy. Moreover, by using the guided model to guide the selection process of matching image pairs, the matching efficiency and accuracy can be further improved, thereby improving the accuracy and efficiency of UAV visual positioning.

[0054] In this embodiment, step S1 includes:

[0055] Step S101. Acquire 3D point cloud data of the real scene using radar carried by the drone;

[0056] Step S102. Generate the triangular mesh surface corresponding to the 3D point cloud data through Poisson surface reconstruction to obtain a dense scene model based on the mesh map corresponding to the real scene.

[0057] In a specific application embodiment, a mesh-based dense scene model is built on the XTDrone drone simulation environment. Specifically, triangular meshes are used to represent 3D objects in space to construct a 3D mesh map of the drone simulation scene. This process utilizes the 3D LiDAR onboard the drone to scan and acquire 3D point cloud data of the scene. After acquiring the point cloud data, the Poisson 3D surface reconstruction algorithm is applied to generate an accurate 3D mesh model, i.e., a mesh map. After construction, a depth map is obtained through Open3D rendering.

[0058] It is understandable that since mesh-based visual localization methods (such as the mesh-based dense scene model mentioned above) construct dense models, they are not tied to feature point extraction and matching methods. Therefore, feature matching algorithms are not needed during scene modeling. Furthermore, because learning-based feature extraction methods can not only establish matching relationships between two real images, but also between real images and mesh-based images, as well as images rendered from the original surface, mesh-based visual localization can avoid storing the actual captured database images. Instead, it can use synthetic images (depth maps) rendered from the dense 3D model in subsequent matching processes, thereby reducing storage space overhead.

[0059] In step S2 of this embodiment, the method further includes constructing an image dataset. The steps for constructing an image dataset include:

[0060] Images are captured at multiple locations in a simulated environment using cameras mounted on a simulated drone. During the capture process, the drone's yaw, pitch, and altitude are continuously adjusted, and the camera's real-time 6D attitude information and intrinsic parameters are recorded to obtain multiple simulated images labeled with the drone's attitude information. These multiple simulated images constitute an image dataset of the simulated environment.

[0061] In a specific application embodiment, after generating a dense scene model, compared to the real-world scene, the simulated environment displays less texture detail and more prominent feature points. Therefore, a new image dataset is created, containing images taken by a drone-mounted camera at 86 different locations within the simulated environment. During capture, the drone's yaw, pitch, and altitude are systematically adjusted, with pitch angles set to 20°, 0°, and -20°, yaw angles varying in 30° increments from 0° to 360°, and altitude randomly varying between 1.5m and 3m. Simultaneously, the camera's real-time 6D attitude information, along with its intrinsic parameters, is recorded. This image dataset covers 13 simulated building structures, including additional elements such as vehicles, grass terrain, and other objects. The database contains a total of 3096 simulated images, each labeled with precise drone attitude information.

[0062] It is understandable that after the dense scene model is built, in order to reduce the large amount of storage resources occupied and make it closer to the actual scene data, the images that may be collected from various angles during the actual flight of the drone are stored in the dense scene model to form an image dataset. This facilitates the direct comparison of the real captured images with the images in the image dataset during subsequent image matching, thereby reducing the overall time overhead.

[0063] In step S2 of this embodiment, inputting the query image into the image dataset for matching includes:

[0064] Step S201. Obtain the local features of each captured image and perform clustering to obtain the global features of each captured image;

[0065] Step S202. Match the global features of each captured image with the feature descriptors of each image in the image dataset to find the multiple feature images that are closest to each captured image.

[0066] In a specific application embodiment, NetVlad is first used to cluster the local features of each query image to obtain global features. Then, the Faiss open-source library is used to match the feature descriptors of the query image with those of images in the image dataset to find the K images closest to the query image, denoted as feature images. It is understandable that since the image dataset contains many images, directly using a matching algorithm would consume excessive system resources. Therefore, the above steps are used to find several feature images closest to the query image to facilitate further matching, thereby balancing overall computing power and reducing total time consumption.

[0067] In this embodiment, step S3 includes:

[0068] Step S301. Combine two or more matching models and select one of them as the guiding model to form a combined guiding model;

[0069] Step S302. Based on the combined guided model, multiple bidirectional matching pairs are obtained by matching the query image and each feature image using each matching model respectively.

[0070] Step S303. Select matching pairs that satisfy the cycle consistency constraint from the bidirectional matching pairs obtained by matching with each matching model, and estimate the basis matrix using the guided model. Based on the basis matrix, apply epipolar geometric constraints to each bidirectional matching pair obtained by matching with other models besides the guided model to initially select multiple preliminary matching pairs between the query image and each feature image.

[0071] Step S304. Filter the preliminary matching pairs that satisfy the cycle consistency constraint to estimate the fundamental matrix again, and filter the matching pairs that satisfy the epipolar geometry constraint again based on the fundamental matrix to obtain the final matching pairs.

[0072] This embodiment uses a combined guided model to match the query image and each feature image separately using various matching models. This allows for the simultaneous acquisition of multiple bidirectional matching pairs. By using the guided model for cyclic consistency constraints, obviously erroneous matching pairs can be effectively filtered out, thus improving the accuracy of the matching.

[0073] In this embodiment, the expression for the cycle consistency constraint is specifically as follows:

[0074] (1)

[0075] In the above formula, To determine the coordinates of point i in image A during forward matching from the captured image to the feature image, Let i be the coordinates of the corresponding point i in feature image B. Let i be the coordinates of point i in image A during backward matching from feature image to captured image. Let i be the coordinates of the corresponding point i in feature image B. This is the first threshold.

[0076] In this embodiment, the expression for the epipolar geometric constraint is specifically as follows:

[0077] (2)

[0078] In the above formula, To map the feature points in image A to the straight lines in feature image B, Based on the matrix, The polar distance. This is the second threshold.

[0079] The query image has already been obtained. and K related database images Subsequently, to complete pose estimation, it is necessary to establish correct correspondences between these images. To improve the accuracy of visual localization, this embodiment employs a matching algorithm based on a combined guided model to estimate the correspondences between the query image and each feature image, generating 2D-2D matching image pairs. For example... Figure 2 As shown, four advanced image matching algorithms are first used to obtain rich bidirectional matching pairs, and then the matching pairs are filtered using the cycle consistency condition. Then, a matching model (such as COTR (Correspondence Transformer for Matching Across Images)) is selected as the guiding model. The matching pairs obtained from the guiding model are first used... Estimate the fundamental matrix between two images This yields matching pairs that satisfy the fundamental matrix. ,based on For the remaining three methods, epipolar constraints were used to perform preliminary screening to obtain matching pairs. Based on the above results Estimate the fundamental matrix again And filter out matching pairs The fundamental matrix between the two images is estimated again, and then epipolar constraints are used to filter out outlier matching pairs; finally, accurate matching pairs are obtained.

[0080] In this embodiment, the specific steps of the matching algorithm based on the combined guided model to estimate the correspondence between the query image and each feature image and generate 2D-2D matching image pairs are as follows:

[0081] 1) Two-way matching

[0082] To obtain rich bidirectional matching pairs, four advanced image matching algorithms are used to estimate the correspondence between image pairs: such as the detection-based feature detection algorithm SuperPoint, the feature matching algorithm Superglue, and the direct matching algorithms AspanFormer, Patch2pix, and COTR. Given two images... and This allows us to obtain the correspondence between them. ,in . express arrive Matching pairs, express arrive The number of matching pairs. To ensure a sufficient number of matching pairs, this embodiment uses bidirectional matching, resulting in the following matching pairs: .

[0083] 2) Cyclic consistency constraints

[0084] Although many matching pairs are generated, some incorrect and inaccurate matches exist. This embodiment filters out obviously incorrect matching pairs by forcing the matching pairs to satisfy the cycle consistency constraint. Specifically, based on the obtained bidirectional matching pairs, the matching pairs are filtered. For two images... and The coordinates of the query point in graph A during forward matching. The corresponding coordinates in Figure B The coordinates of the query point in graph A during backward matching. The corresponding coordinates in Figure B An ideal matching pair should satisfy the following formula:

[0085] (3)

[0086] However, the above formula (3) is usually not strictly satisfied. In this embodiment, the constraint conditions are further scaled to form the constraint conditions shown below, so as to retain the matching pairs that satisfy the following formula and discard those that do not.

[0087] (4)

[0088] 3) Polar geometric constraints

[0089] In multi-view geometry, a special relationship exists between the imaging planes of two cameras. The corresponding point of any pixel in one image must lie on a straight line called the epipolar line, which can be described by epipolar geometry. To filter out outliers, this embodiment selects correspondences that conform to epipolar geometry and further filters using epipolar geometric constraints to obtain the final accurate matching pair. Specifically, given a matching point... and fundamental matrix Its polar distance The geometric error of the matching points was measured:

[0090] (5)

[0091] The estimation of the fundamental matrix described above can be accomplished using the RANSAC (Random Sample Consensus) algorithm. Since the accuracy of the initial fundamental matrix estimation is relatively low, this embodiment sets a lenient threshold. This eliminates mismatched pairs that do not conform to epipolar geometry constraints, thereby further enhancing the robustness of subsequent attitude estimation.

[0092] It is understood that the image matching method based on the combined guided model in this embodiment integrates multiple advanced image matching algorithms, combines the characteristics of various methods, and compensates for the shortcomings of a single algorithm by combining multiple algorithms. It can obtain robust, highly repeatable, and high matching accuracy results, thereby improving the accuracy and robustness of matching query images and reference images. By guiding the matching process through the guided model, the matching efficiency and accuracy can be further improved. Moreover, this model is easy to expand and can integrate more advanced algorithms. By replacing different image matching algorithms, matching pairs can be obtained, thereby improving the accuracy of UAV positioning.

[0093] In this embodiment, step S4 includes:

[0094] Step S401. Render a depth map based on the dense scene model, and calculate the three-dimensional coordinates of the pixel coordinates of multiple feature images corresponding to the key points of the query image in each 2D-2D matching pair in the dense scene model using the method of minimizing reprojection error; process the multiple three-dimensional coordinates by minimizing reprojection error to obtain the final three-dimensional coordinates, so as to convert the 2D-2D matching pair into a 2D-3D matching pair.

[0095] Step S402. Solve the pose of the UAV camera based on the coordinates of the 3D points in the 2D-3D matching, the coordinates of the corresponding 2D points, and the intrinsic parameter matrix, so as to calculate the pose information of the UAV at the time of the current image capture and realize UAV positioning.

[0096] In specific application embodiments, after image matching, multiple 2D-2D matching pairs can be obtained. Combined with the depth map obtained using the Open3D rendered mesh model, the 3D coordinates of the matching points in the 2D-2D matching pairs can be directly calculated by accessing the depth. Let the coordinates of a point in the world coordinate system be denoted as the world coordinates. Let the pixel coordinates on the depth map be (u, v), the pixel value at that point (i.e., the Z-axis component of that point in the camera coordinate system) be d, and the camera's intrinsic parameter matrix be K. Then the following correspondence exists:

[0097] (6)

[0098] For mesh-based methods, since the query image is matched with multiple images (i.e., feature images) in the database, keypoints in the query image will match a pixel in each feature image. Noise can cause a single keypoint to correspond to multiple 3D coordinates. This embodiment reduces noise error by minimizing reprojection error and calculates the optimal 3D coordinates. Specifically, let M(q) be the set of matching points between a point in the query image and multiple feature images. Due to the possibility of mismatches, a subset with consistency is first selected. Then, the coordinates pi of all corresponding 3D points in the subset are obtained. Each pi is projected back into the database image corresponding to the subset, and the pixel coordinate error is calculated. After removing 3D points greater than a threshold, the reprojection error of the remaining 3D points is minimized, and the final average 3D coordinates are obtained as the final 3D coordinates. If there are fewer than two points greater than the threshold, all matching pairs in M(q) are retained.

[0099] Furthermore, common-view filtering and position averaging can be performed as post-processing to improve the accuracy of pose estimation, transforming the obtained 2D-2D matching pairs into 2D-3D matching pairs. Finally, the PnP algorithm is used to estimate the UAV pose information corresponding to the current image capture. For example, the P3P (Perspective-Three-Point) algorithm can calculate the camera pose change using three pairs of 3D-2D matching points with known world coordinates, and verify the correct value among multiple solutions using a pair of validation points.

[0100] Figure 3The overall flow of the UAV visual localization method based on image matching using a combined guided model is shown in a specific application embodiment. The specific steps are as follows: First, point clouds acquired by the radar on the UAV are used to reconstruct a 3D model of the scene using Poisson reconstruction. Next, a database (i.e., an image dataset) of the UAV simulated environment scene is constructed using images captured by the visible light camera on the UAV during pre-flight. Then, query images captured by the UAV during real-time flight are retrieved from the database, and K most relevant feature images are obtained. Next, the image matching task is completed through a combined guided model to obtain 2D-2D matching pairs between the query images and the database images, and then the rendered depth map is used to convert them into 2D-3D matching. Finally, multiple 2D-3D correspondences are used to estimate the attitude information of the UAV.

[0101] Furthermore, the present invention also provides a UAV visual positioning device based on combined guided model image matching, comprising:

[0102] The dense scene model building module is used to build a dense scene model based on a mesh map based on scene point cloud data obtained by scanning with a 3D LiDAR mounted on a drone.

[0103] The first image matching module is used to take the images captured during the real-time flight of the UAV as query images and input the query images into the image dataset for matching, so as to find multiple feature images that are close to the query images in the image dataset;

[0104] The second image matching module is used to estimate the correspondence between the query image and each feature image using a combined guided model to generate multiple sets of 2D-2D matching image pairs. The combined guided model is composed of two or more matching models, and one of the matching models is selected as the guided model. The other models besides the guided model are used to filter the preliminary matching pairs during the matching process, and then the preliminary matching pairs are filtered a second time to obtain the final matching pairs.

[0105] The information extraction module is used to render a depth map in real time based on the dense scene model, convert each 2D-2D matching pair into a 2D-3D matching pair based on the depth map, and calculate the corresponding attitude information of the UAV through the 2D-3D matching pair to achieve UAV positioning.

[0106] In addition, the present invention provides a computer-readable storage medium storing a computer program for being programmed or configured by a microprocessor to execute the above-described UAV visual localization method based on combined guided model image matching.

[0107] The apparatus and medium of the present invention, corresponding to the methods described above, also have the advantages described above.

[0108] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. Computer-readable media include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0109] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for visual positioning of a UAV based on combined guided model image matching, the method comprising: The method comprises the following steps: S1, constructing a dense scene model based on a mesh map according to scene point cloud data scanned by a 3D laser radar mounted on a UAV; S2, taking a shooting image obtained in real time during flight of the UAV as a query image, and inputting the query image into an image dataset for matching to find a plurality of feature images close to the query image in the image dataset; S3, estimating a corresponding relationship between the query image and each feature image using a combined guide model to generate a plurality of 2D-2D matching image pairs, the combined guide model being composed of two or more matching models, and selecting one of the matching models as a guide model, and using the guide model to screen preliminary matching pairs from the remaining models other than the guide model in the matching process, and performing secondary screening on the preliminary matching pairs to obtain final matching pairs; S4, rendering a depth map in real time according to the dense scene model, converting each 2D-2D matching pair into a 2D-3D matching pair according to the depth map, and calculating corresponding attitude information of the UAV through the 2D-3D matching pair to realize UAV positioning; Wherein, step S3 comprises: Combining two or more matching models, and selecting one of the matching models as a guide model to form a combined guide model; Based on the combined guide model, each matching model is used to match the query image and each feature image respectively to obtain a plurality of bidirectional matching pairs; From the bidirectional matching pairs obtained by using the matching models, matching pairs satisfying the cyclic consistency constraint are screened out, the guide model is used to estimate the fundamental matrix, and each bidirectional matching pair obtained by matching the remaining models other than the guide model is subjected to epipolar geometric constraint based on the fundamental matrix to preliminarily screen out a plurality of preliminary matching pairs between the query image and each feature image; The matching pairs satisfying the cyclic consistency constraint in each preliminary matching pair are screened out to estimate the fundamental matrix again, and the matching pairs satisfying the epipolar geometric constraint are screened out again based on the fundamental matrix to obtain the final matching pairs.

2. The UAV visual positioning method based on combined guided model image matching according to claim 1, characterized in that, Step S1 comprises: Acquiring 3D point cloud data in a real scene by a radar mounted on a UAV; Generating a triangular mesh grid surface corresponding to the 3D point cloud data by Poisson surface reconstruction to obtain a dense scene model based on a mesh map corresponding to the real scene. 3.The UAV visual positioning method based on combined guided model image matching according to claim 1, wherein, In step S2, an image dataset is also constructed, and the steps of constructing the image dataset comprise: Shooting images by simulating a camera mounted on a UAV at a plurality of different positions in a simulated environment, continuously adjusting the yaw angle, pitch angle and height of the UAV during shooting, and recording the real-time 6D attitude information and intrinsic parameters of the camera to obtain a plurality of simulated images labeled with UAV attitude information, and the image dataset of the simulated environment is composed of the plurality of simulated images.

4. The unmanned aerial vehicle visual positioning method based on combined guided model image matching according to claim 1, characterized in that, In step S2, inputting the query image into the image dataset for matching comprises: Obtaining local features of each query image and clustering to obtain global features of each query image; matching global features of each of the query images with feature descriptors of each image in the image dataset to find a plurality of feature images closest to each of the query images.

5. The unmanned aerial vehicle visual positioning method based on combined guided model image matching according to claim 1, characterized in that, An expression of the cyclic consistency constraint is: ， In the above formulae, is the coordinate of point i in the photograph image A for the forward matching of photograph images to feature images, is the coordinate of the corresponding point i in the feature image B, is the coordinate of point i in the photograph image A for the backward matching of feature images to photograph images, is the coordinate of the corresponding point i in the feature image B, is a first threshold value.

6. The unmanned aerial vehicle visual positioning method based on combined guided model image matching according to claim 1, characterized in that, An expression of the epipolar geometry constraint is: ， In the above formula, is a line in the feature image B to which the feature point in the captured image A corresponds, is a fundamental matrix, is an epipolar distance, is a second threshold value.

7. The unmanned aerial vehicle visual positioning method based on combined guide model image matching according to any one of claims 1-6, characterized in that, The step S4 of converting each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map and calculating corresponding attitude information of the UAV through the 2D-3D matching pair includes: The step S4 of converting each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map and calculating corresponding attitude information of the UAV through the 2D-3D matching pair includes: The step S4 of converting each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map and calculating corresponding attitude information of the UAV through the 2D-3D matching pair includes:

8. An unmanned aerial vehicle (UAV) visual positioning device based on combined guided model image matching, the device being applied to the method of any one of claims 1-7, characterized in that, The device comprises: A dense scene model construction module for constructing a mesh map-based dense scene model according to scene point cloud data scanned by a 3D laser radar mounted on the UAV; A first image matching module for taking a photographed image acquired in real-time flight of the UAV as a query image and inputting the query image into an image dataset for matching to find a plurality of feature images close to the query image in the image dataset; A second image matching module for estimating a corresponding relationship between the query image and each feature image using a combined guide model to generate a plurality of 2D-2D matching image pairs, the combined guide model being composed of two or more matching models, and selecting one of the matching models as a guide model, and using the guide model to screen the remaining models other than the guide model to obtain preliminary matching pairs in the matching process, and performing secondary screening on the preliminary matching pairs to obtain final matching pairs; An information extraction module for rendering a depth map in real time according to the dense scene model, converting each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map, and calculating corresponding attitude information of the UAV through the 2D-3D matching pair to realize UAV positioning.

9. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program is used for programming or configuring a microprocessor to execute the UAV visual positioning method based on the combined guide model image matching of any one of claims 1-7.

Images