Unmanned aerial vehicle visual positioning method and device based on combined guide model image matching
By combining the combined guidance model and the intensive scene model, the problem of large time and storage resource overhead in drone visual positioning is solved, and high-precision and efficient drone positioning is achieved, which is suitable for a variety of scenarios.
Patent Information
- Application Number
- CN202410916260.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-09
AI Technical Summary
The existing drone visual positioning technology has problems such as large overhead of time and storage resources, low positioning accuracy and poor versatility. Especially when it is necessary to frequently adapt to different scenarios, traditional methods require a lot of time and storage space, and the accuracy of feature matching algorithms is affected. limit.
Using an image matching method based on the combined guidance model, intensive scene models are constructed through 3D lidar scanning, depth maps are rendered in real time, and multiple matching models are used for screening, 2D-3D matching pairs are generated, and drone attitude information is calculated to achieve efficient and efficient Accurate visual positioning.
It reduces data storage space and time overhead, improves the accuracy and flexibility of visual positioning of drones, enhances robustness and scope of application, and improves matching efficiency and accuracy.
Smart Images

Figure CN118887280A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicle visual positioning, and in particular to a method and device for unmanned aerial vehicle visual positioning based on combined guidance model image matching. Background Art
[0002] UAV visual positioning technology is a method of estimating the position and attitude of a UAV using computer vision technology. Visual positioning technology combines multiple functions of computer vision, including measurement, image matching and spatial perception, and can be applied to areas such as navigation of autonomous vehicles, positioning of unmanned aerial vehicles (UAVs), and path planning in complex environments. The core goal of visual positioning technology is to accurately determine the position of an object by analyzing images captured by a camera in a predetermined environment. UAVs are able to exhibit a variety of roll angles and flight speeds during flight, and have superior flexibility and agility compared to other unmanned platforms.
[0003] At present, UAV visual positioning is usually based on the structure from motion (Structure from Motion, SfM) to generate a sparse point cloud model, and determine the spatial and geometric relationship of the target based on a series of images taken when the camera moves. It can be mainly divided into the following two methods: The first is to use convolutional neural networks (CNNs) to directly regress the camera pose information. The CNN-based method can achieve end-to-end training, but the generalization ability is poor, and the performance depends largely on the distribution of scene image data. Therefore, when the application is extended to a new scene, additional training or adaptive adjustment is usually required. The second method aims to obtain 2D and 3D coordinate matching between the query image and the database model, and use PnP algorithms to estimate the camera pose. Usually, feature points and descriptors in the image are first extracted to match two related images, and then the three-dimensional coordinates of the feature points are calculated by triangulation to obtain sparse point cloud information. In the above SfM method, during the matching process, the constructed sparse point cloud is bound to the selected feature point extraction and matching method. If other methods are replaced, the matching pairs between the database images need to be recalculated. For the construction of large scenes, this process will take a lot of time. At the same time, in the drone pose estimation stage, the depth map corresponding to the dense three-dimensional model is required, so it is necessary to further use the sparse point cloud to construct a dense point cloud. This process also takes a lot of time, and storing the depth map also requires a certain amount of space. Therefore, the traditional SfM method-based drone visual positioning requires a lot of time and storage resources, and a feature matching algorithm is required when reconstructing the three-dimensional scene, which will be limited by the accuracy of the feature matching algorithm. Other visual positioning methods such as Meshloc based on dense 3D grids use grid models and use image retrieval and feature matching technologies to restore the camera's pose, which is also limited by the accuracy of the feature matching algorithm.
[0004] However, traditional image matching algorithms are usually targeted at specific application scenarios, so they have poor versatility and low accuracy in other application scenarios. For example, the LoFTR (Local Feature Transformer) algorithm is aimed at the poor image matching results in weak texture areas, and proposes to use Transformer combined with contextual information to improve it. SDF2 (Signed Distance Function 2) is aimed at long-term visual positioning problems, and proposes to use semantic information to classify feature points. The above single algorithm can only match images under specific visual problems and is not suitable for other scenarios. For various application scenarios, directly using pre-trained models may lead to a large number of mismatches, and training separate models for different scenarios will make the implementation complicated. Summary of the invention
[0005] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a method and device for unmanned aerial vehicle visual positioning based on combined guidance model image matching is provided, which has low time and storage resource overhead, high positioning accuracy, strong robustness and a wide range of applications, and can reduce the space and time occupied by data storage, and improve the accuracy and flexible adaptability of unmanned aerial vehicle visual positioning.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A UAV visual positioning method based on combined guidance model image matching, comprising: S1, builds a dense scene model based on the mesh map according to the scene point cloud data obtained by scanning with the 3D laser radar carried by the drone; S2, taking the captured image obtained during the real-time flight of the UAV as a query image, and inputting the query image into the image dataset for matching, so as to find a plurality of feature images close to the query image in the image dataset; S3, using a combined guidance model to estimate the correspondence between the query image and each feature image to generate multiple groups of 2D-2D matching image pairs, wherein the combined guidance model is composed of two or more matching models, and one of the matching models is selected as a guidance model, and the remaining models except the guidance model are screened by using the guidance model during the matching process to obtain preliminary matching pairs, and then the preliminary matching pairs are screened again to obtain final matching pairs; S4, rendering a depth map in real time according to the dense scene model, converting each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map, and calculating corresponding posture information of the drone through the 2D-3D matching pairs to achieve drone positioning.
[0007] Further, step S1 includes: Acquire 3D point cloud data in real scenes through the radar carried by drones; The triangular mesh surface corresponding to the 3D point cloud data is generated by Poisson surface reconstruction to obtain a dense scene model based on the mesh map corresponding to the real scene.
[0008] Furthermore, step S2 further includes constructing an image data set, and the step of constructing the image data set includes: Images are captured by a camera carried by a simulated drone at multiple different locations in a simulated environment. During the shooting process, the yaw angle, pitch angle and altitude of the drone are continuously adjusted, and the real-time 6D attitude information and intrinsic parameters of the camera are recorded to obtain multiple simulated images annotated with the drone attitude information. The multiple simulated images constitute an image data set of the simulated scene model.
[0009] Furthermore, in step S2, inputting the query image into the image dataset for matching includes: Acquire local features of each of the captured images and perform clustering to obtain global features of each of the captured images; The global features of each of the captured images are matched with the feature descriptors of each image in the image data set to find a plurality of feature images that are closest to each of the captured images.
[0010] Further, step S3 includes: Combining two or more matching models, and selecting one of the matching models as a guiding model to form a combined guiding model; Based on the combined guidance model, the query image and each feature image are matched respectively by using each matching model to obtain a plurality of bidirectional matching pairs; Screening out matching pairs that satisfy the cycle consistency constraint from the bidirectional matching pairs obtained by matching the respective matching models, using the guided model to estimate the basic matrix, and performing epipolar geometric constraints on the respective bidirectional matching pairs obtained by matching the remaining models except the guided model based on the basic matrix, so as to preliminarily screen out a plurality of preliminary matching pairs between the query image and the respective feature images; The matching pairs satisfying the cycle consistency constraint in each of the preliminary matching pairs are screened to estimate the basic matrix again, and the matching pairs satisfying the epipolar geometry constraint are screened again based on the basic matrix to obtain the final matching pairs.
[0011] Furthermore, the expression of the cycle consistency constraint is:
[0012] , In the above formula, is the coordinate of point i in the captured image A in the forward matching from the captured image to the feature image, is the coordinate of the corresponding point i in the feature image B, is the coordinate of point i in the captured image A in the backward matching from the feature image to the captured image, is the coordinate of the corresponding point i in the feature image B, is the first threshold.
[0013] Furthermore, the expression of the epipolar geometric constraint is:
[0014] , In the above formula, The straight line corresponding to the feature point in the captured image A and the feature image B is is the basic matrix, is the epipolar distance, is the second threshold.
[0015] Further, step S4 includes: Calculating the pixel coordinates of a plurality of feature images corresponding to the key points of the query image in each of the 2D-2D matching pairs through the depth map to correspond to the three-dimensional coordinate values in the dense scene model, and processing the plurality of three-dimensional coordinate values to obtain final three-dimensional coordinates by minimizing the reprojection error, so as to convert the 2D-2D matching pairs into 2D-3D matching pairs; The position and posture of the drone camera are solved according to the coordinates of the 3D points in the 2D-3D matching, the coordinates of the corresponding 2D points and the internal reference matrix to calculate the position and posture information of the drone corresponding to the current image shooting and realize the positioning of the drone.
[0016] In addition, the present invention also provides a UAV visual positioning device based on combined guidance model image matching, comprising: The dense scene model building module is used to build a dense scene model based on the mesh map according to the scene point cloud data obtained by scanning with the 3D laser radar carried by the drone; A first image matching module is used to use a captured image obtained during the real-time flight of the UAV as a query image, and input the query image into an image data set for matching, so as to find a plurality of feature images close to the query image in the image data set; a second image matching module, configured to estimate the correspondence between the query image and each feature image using a combined guiding model to generate a plurality of 2D-2D matching image pairs, wherein the combined guiding model is composed of a combination of two or more matching models, and one of the matching models is selected as a guiding model, and the remaining models except the guiding model are screened using the guiding model during the matching process to obtain preliminary matching pairs, and then the preliminary matching pairs are screened for a second time to obtain final matching pairs; The information extraction module is used to render a depth map in real time according to the dense scene model, convert each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map, and calculate the corresponding posture information of the drone through the 2D-3D matching pairs to realize drone positioning.
[0017] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the above-mentioned drone visual positioning method based on combined guidance model image matching.
[0018] Compared with the prior art, the advantages of the present invention are: The present invention constructs a dense scene model based on a mesh map and obtains a depth map by rendering. By rendering the depth map in real time instead of storing all the depth maps, the storage space overhead and the overall running time of the model can be reduced. At the same time, by matching the real-time captured image with the image data set as the query image, the feature image in the image data set that is close to the query image is found, and the similar images can be preliminarily filtered first, thereby reducing the number of image processing and improving efficiency. Then, the combined guidance model is used to estimate the correspondence between the query image and each feature image to generate a 2D-2D matching image pair, which can make up for the defect of low matching accuracy of a single algorithm and obtain a matching result with strong robustness, high repeatability and high matching accuracy. Moreover, by using the guidance model to guide the screening process of the matching image pair, the matching efficiency and accuracy can be further improved, thereby improving the accuracy and efficiency of the visual positioning of the unmanned aerial vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a flow chart of the UAV visual positioning method based on combined guidance model image matching in this embodiment.
[0020] Figure 2 It is a schematic diagram of the process of combining the guidance model for image matching in a specific application embodiment of the present invention.
[0021] Figure 3 The figure is a schematic diagram of the overall process of the UAV visual positioning method based on combined guidance model image matching in a specific application embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0023] like Figure 1 As shown, the steps of the UAV visual positioning method based on combined guidance model image matching in this embodiment include: S1, builds a dense scene model based on the mesh map according to the scene point cloud data obtained by scanning with the 3D laser radar carried by the drone; S2, taking the captured image obtained during the real-time flight of the UAV as the query image, and inputting the query image into the image dataset for matching, so as to find multiple feature images close to the captured image in the image dataset; S3, using a combined guidance model to estimate the correspondence between the query image and each feature image to generate multiple groups of 2D-2D matching image pairs, the combined guidance model is composed of two or more matching models, and one of the matching models is selected as the guidance model. The remaining models except the guidance model are screened by the guidance model during the matching process to obtain preliminary matching pairs, and then the preliminary matching pairs are screened again to obtain final matching pairs; S4, renders a depth map in real time according to the dense scene model, converts each 2D-2D matching pair into a 2D-3D matching pair according to the depth map, and calculates the corresponding posture information of the drone through the 2D-3D matching pairs to realize the positioning of the drone.
[0024] It can be understood that this embodiment constructs a dense scene model based on a mesh map and obtains a depth map through rendering. By rendering the depth map in real time instead of storing all depth maps, the storage space overhead and the overall running time of the model can be reduced; at the same time, by matching the real-time captured image with the image data set as the query image, the feature image in the image data set that is close to the query image is found, and similar images can be preliminarily filtered first to reduce the number of image processing and improve efficiency; and then the combined guidance model is used to estimate the correspondence between the query image and each feature image to generate a 2D-2D matching image pair, which can make up for the defect of low matching accuracy of a single algorithm and obtain a matching result with strong robustness, high repeatability and high matching accuracy. Moreover, by using the guidance model to guide the screening process of the matching image pairs, the matching efficiency and accuracy can be further improved, thereby improving the accuracy and efficiency of the UAV visual positioning.
[0025] In this embodiment, step S1 includes: Step S101. Acquire 3D point cloud data in a real scene through a radar carried by a drone; Step S102: Generate a triangular mesh surface corresponding to the 3D point cloud data through Poisson surface reconstruction to obtain a dense scene model based on the mesh map corresponding to the real scene.
[0026] In a specific application embodiment, a grid-based dense scene model is established on the drone simulation environment XTDrone. Specifically, a three-dimensional mesh map of the drone simulation scene is constructed by using a triangular mesh to represent the three-dimensional objects in the space. This process uses the 3D laser radar scan carried by the drone to obtain the 3D point cloud data of the scene. After obtaining the point cloud data, the Poisson three-dimensional surface reconstruction algorithm is applied to generate an accurate three-dimensional mesh model, i.e., a mesh map. After the construction is completed, the depth map is obtained by Open3D rendering.
[0027] It can be understood that since the visual positioning method based on the grid (mesh map) (such as the above-mentioned dense scene model based on the grid) builds a dense model, it has no binding relationship with the feature point extraction and matching method, so no feature matching algorithm is required in the scene modeling process. In addition, since the learning-based feature extraction method can not only establish a matching relationship between two real images, but also can be used for matching between real images and images based on the mesh model and images rendered on the original surface, this makes it possible for the visual positioning based on the mesh map to not store the real database images, but to use the synthetic images (depth maps) obtained by dense 3D model rendering instead in the subsequent matching process, thereby reducing the storage space overhead.
[0028] In step S2 of this embodiment, the image data set is also constructed. The step of constructing the image data set includes: Images are captured by a camera carried by a simulated drone at multiple different locations in the simulated environment. During the shooting process, the yaw angle, pitch angle and altitude of the drone are continuously adjusted, and the real-time 6D attitude information and intrinsic parameters of the camera are recorded to obtain multiple simulated images annotated with the drone attitude information. Multiple simulated images constitute an image dataset of the simulated scene model.
[0029] In a specific application embodiment, after generating a dense scene model, compared with the real-world scene, since the simulated environment shows fewer texture details and more prominent feature points, a new image dataset is created, which contains images taken by a drone-mounted camera at 86 different locations in the simulated environment. During the capture process, the yaw, pitch, and height of the drone are systematically adjusted, the pitch angle is set to 20°, 0°, and -20°, the yaw angle changes from 0° to 360° every 30°, and the height changes randomly between 1.5m and 3m. At the same time, the real-time 6D attitude information of the camera is recorded, as well as its intrinsic parameters. The image dataset covers 13 simulated building structures, including additional elements such as vehicles, grass terrain, and other objects. The database contains a total of 3096 simulated images, each of which is annotated with precise drone attitude information.
[0030] It can be understood that after the dense scene model is built, in order to reduce the large amount of storage resources occupied and make it closer to the actual scene data, the images that may be collected at various angles during the actual flight of the drone are stored in the dense scene model to form an image data set, which is beneficial for subsequent image matching by directly comparing the real captured images with the images in the image data set, thereby reducing the overall time cost.
[0031] In step S2 of this embodiment, inputting the query image into the image data set for matching includes: Step S201. Acquire local features of each captured image and perform clustering to obtain global features of each captured image; Step S202: Match the global features of each captured image with the feature descriptors of each image in the image data set to find multiple feature images that are closest to each captured image.
[0032] In a specific application embodiment, the local features of each query image are first clustered by NetVlad to obtain global features. Subsequently, the Faiss open source library is used to match the feature descriptors of the query image with the images in the image dataset to find the K images closest to the query image, which are recorded as feature images. It can be understood that since there are many images in the image dataset, if the matching algorithm is used directly for matching, it will consume too many system resources. Therefore, the above steps are first used to find several feature images that are closest to the query image, so as to facilitate subsequent further matching, balance the overall computing power, and reduce the total consumption time.
[0033] In this embodiment, step S3 includes: Step S301. Combining two or more matching models, and selecting one of the matching models as a guiding model to form a combined guiding model; Step S302. Based on the combined guidance model, the query image and each feature image are matched respectively by using each matching model to obtain a plurality of bidirectional matching pairs; Step S303. Select matching pairs that satisfy the cycle consistency constraint from the bidirectional matching pairs obtained by matching the matching models, and use the guided model to estimate the basic matrix. Based on the basic matrix, perform epipolar geometric constraints on each bidirectional matching pair obtained by matching the remaining models except the guided model, so as to preliminarily select a plurality of preliminary matching pairs between the query image and each feature image. Step S304: Filter the matching pairs that satisfy the cycle consistency constraint in each preliminary matching pair to estimate the basic matrix again, and filter out the matching pairs that satisfy the epipolar geometry constraint based on the basic matrix again to obtain the final matching pairs.
[0034] This embodiment can simultaneously obtain multiple bidirectional matching pairs by matching the query image and each feature image using each matching model based on the combined guidance model, and then use the guidance model to perform cyclic consistency constraints to effectively filter out obviously erroneous matching pairs and improve the matching accuracy.
[0035] In this embodiment, the expression of the cycle consistency constraint is specifically: (1) In the above formula, is the coordinate of point i in the captured image A in the forward matching from the captured image to the feature image, is the coordinate of the corresponding point i in the feature image B, is the coordinate of point i in the captured image A in the backward matching from the feature image to the captured image, is the coordinate of the corresponding point i in the feature image B, is the first threshold.
[0036] In this embodiment, the expression of the epipolar geometric constraint is specifically: (2) In the above formula, The straight line corresponding to the feature point in the captured image A and the feature image B is is the basic matrix, is the epipolar distance, is the second threshold.
[0037] After obtaining the query image and K related database images Finally, in order to complete the pose estimation, it is necessary to establish the correct correspondence between these images. In order to improve the accuracy of visual positioning, this embodiment uses a matching algorithm based on a combined guidance model to estimate the correspondence between the query image and each feature image to generate a 2D-2D matching image pair. Figure 2 As shown in the figure, four advanced image matching algorithms are first used to obtain abundant bidirectional matching pairs, and the matching pairs are screened using the cycle consistency condition to obtain Then select a matching model (such as the COTR (Correspondence Transformer for Matching Across Images) model) as the guide model, first use the matching pairs obtained by the guide model Estimate the fundamental matrix between two images , and obtain matching pairs that satisfy the basic matrix ,based on For the remaining three methods, the epipolar constraints are used to complete the preliminary screening and obtain matching pairs. ; Combined with the above results Estimate the fundamental matrix again And filter out matching pairs The basic matrix between the two images is estimated again, and then the epipolar constraints are used to filter out abnormal matching pairs; finally, accurate matching pairs are obtained.
[0038] In this embodiment, the specific steps of estimating the correspondence between the query image and each feature image to generate a 2D-2D matching image pair based on the matching algorithm of the combined guidance model are as follows: 1) Bidirectional matching In order to obtain rich bidirectional matching pairs, four advanced image matching algorithms are used to estimate the correspondence between image pairs: for example, the detection-based feature detection algorithm SuperPoint and the feature matching algorithm Superglue, as well as the direct matching algorithms AspanFormer, Patch2pix and COTR. Given two images and , we can get the corresponding relationship between them ,in . express arrive The matching pairs of express arrive To ensure that there are sufficient matching pairs, this embodiment uses bidirectional matching, and the matching pairs finally obtained are .
[0039] 2) Cycle consistency constraints Although many matching pairs are generated, there are some wrong matches and inaccurate matches. This embodiment filters out obviously wrong matching pairs by forcing the matching pairs to satisfy the cycle consistency constraint. Specifically, based on the obtained bidirectional matching pairs, the matching pairs are screened. For two images and , the coordinates of the query point in graph A in forward matching , the corresponding coordinates in Figure B , the coordinates of the query point in graph A in backward matching , the corresponding coordinates in Figure B , the ideal matching pair should satisfy the following equation: (3) However, the above formula (3) is usually not strictly satisfied. In this embodiment, the constraint condition is further scaled to form the constraint condition shown below, so as to retain the matching pairs that satisfy the following formula and discard those that do not.
[0040] (4) 3) Polar geometry constraints In multi-view geometry, there is a special relationship between the imaging planes of two cameras. The corresponding point of any pixel in one image in the other image must lie on a straight line called an epipolar line, which can be described by epipolar geometry. In order to filter out outliers, this embodiment selects the corresponding relationship that is consistent with the epipolar geometry, and further filters it using the epipolar geometry constraint to obtain the final accurate matching pair. Specifically, given a matching point and the fundamental matrix , its epipolar distance Measures the geometric error of the matching points: (5) The estimation of the above basic matrix can be completed using the RANSAC (Random Sampling Consensus) algorithm. Since the accuracy of the initial basic matrix estimation is relatively low, this embodiment specifically sets a loose threshold , to eliminate mismatched pairs that do not conform to the epipolar geometric constraints, thereby further enhancing the robustness of subsequent pose estimation.
[0041] It can be understood that the image matching method based on the combined guidance model of this embodiment integrates multiple advanced image matching algorithms, comprehensively considers the characteristics of various methods, and makes up for the defects of a single algorithm through the combination of multiple algorithms, so as to obtain a robust, highly repeatable, and high matching accuracy matching result, thereby improving the accuracy and robustness of the matching between the query image and the reference image. By guiding the matching process through the guidance model, the matching efficiency and accuracy can be further improved. Moreover, the model is easy to expand, and more advanced algorithms can be integrated. By replacing different image matching algorithms, matching pairs can be obtained, thereby improving the accuracy of drone positioning.
[0042] In this embodiment, step S4 includes: Step S401. Render a depth map according to the dense scene model, and calculate the three-dimensional coordinate values of the pixel coordinates of multiple feature images corresponding to the key points of the query image in each 2D-2D matching pair by minimizing the reprojection error method; process the multiple three-dimensional coordinate values by minimizing the reprojection error to obtain the final three-dimensional coordinates, so as to convert the 2D-2D matching pair into a 2D-3D matching pair; Step S402. Solve the posture of the drone camera according to the coordinates of the 3D point in the 2D-3D matching, the coordinates of the corresponding 2D point and the internal parameter matrix to calculate the posture information of the drone corresponding to the current image shooting and realize drone positioning.
[0043] In a specific application embodiment, after image matching, multiple sets of 2D-2D matching pairs can be obtained, and combined with the depth map obtained by using the Open3D rendering mesh model, the depth of the matching points in the 2D-2D matching pairs can be directly accessed to solve the three-dimensional coordinates. Let the coordinates of a point in the world coordinate system be the world coordinates , the coordinates in the pixel coordinate system on the depth map are (u, v), the pixel value at that location (i.e. the Z-axis component of the point in the camera coordinate system) is d, and the camera's intrinsic parameter matrix is K. Then there is the following corresponding relationship: (6) For mesh-based methods, since the query image will match multiple images (i.e., feature images) in the database, the key points in the query image will all match a certain pixel in each feature image, and noise will cause a key point to correspond to multiple 3D coordinates. This embodiment reduces the noise error by minimizing the reprojection error and calculates the optimal 3D coordinates. Specifically, let the set of matching pairs of points on a certain point on the query image and multiple feature images be M(q). Since there may be mismatches, a subset with consistency is first selected; then, all corresponding three-dimensional point coordinates pi in the subset are obtained; each pi is projected back to the database image corresponding to the subset in turn, and the pixel coordinate error is calculated; after removing the three-dimensional points greater than the threshold, the reprojection error of the remaining three-dimensional points is minimized, and the final three-dimensional coordinate average value is obtained as the final 3D coordinate. If there are less than 2 points greater than the threshold, all matching pairs in M(q) are retained.
[0044] In addition, common view filtering and position averaging can be performed as post-processing to improve the accuracy of attitude estimation, and the obtained 2D-2D matching pairs can be converted into 2D-3D matching pairs. Finally, the PnP algorithm is used to estimate the pose information of the drone corresponding to the current image capture. For example, the P3P (Perspective-Three-Point) algorithm can be used to calculate the camera pose change through 3 pairs of 3D-2D matching point pairs with known coordinates in the world coordinate system, and a pair of verification points can be used to verify the correct value in multiple P3P solutions.
[0045] Figure 3 The overall process of the UAV visual positioning method based on combined guidance model image matching in a specific application embodiment is shown, and the specific steps are as follows: first, the point cloud obtained by the radar carried by the UAV is used to obtain a three-dimensional model of the scene using Poisson reconstruction; then, the images taken by the visible light camera carried by the UAV during the pre-flight are used to build a database (i.e., image data set) of the UAV simulation scene; then, the query image taken by the UAV in real-time flight is retrieved from the database, and K most relevant feature images are obtained; then, the image matching task is completed by combining the guidance model to obtain a 2D-2D matching pair between the query image and the database image, and then the rendered depth map is used to convert it into a 2D-3D match; finally, multiple 2D-3D correspondences are used to estimate the posture information of the UAV.
[0046] In addition, the present invention also provides a UAV visual positioning device based on combined guidance model image matching, comprising: The dense scene model building module is used to build a dense scene model based on the mesh map according to the scene point cloud data obtained by scanning with the 3D laser radar carried by the drone; The first image matching module is used to use the captured image obtained during the real-time flight of the UAV as a query image, and input the query image into the image data set for matching, so as to find multiple feature images close to the query image in the image data set; The second image matching module is used to estimate the correspondence between the query image and each feature image using a combined guiding model to generate multiple groups of 2D-2D matching image pairs, wherein the combined guiding model is composed of two or more matching models, and one of the matching models is selected as the guiding model. The remaining models except the guiding model are screened by the guiding model during the matching process to obtain preliminary matching pairs, and then the preliminary matching pairs are screened for a second time to obtain final matching pairs; The information extraction module is used to render a depth map in real time according to the dense scene model, convert each 2D-2D matching pair into a 2D-3D matching pair according to the depth map, and calculate the corresponding posture information of the drone through the 2D-3D matching pairs to realize drone positioning.
[0047] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the above-mentioned drone visual positioning method based on combined guidance model image matching.
[0048] The device and medium of the present invention correspond to the above method and also have the advantages described in the above method.
[0049] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0050] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A UAV visual positioning method based on combined guidance model image matching, It is characterized in that include: S1, builds a dense scene model based on the mesh map according to the scene point cloud data obtained by scanning with the 3D laser radar carried by the drone; S2, taking the captured image obtained during the real-time flight of the UAV as a query image, and inputting the query image into the image dataset for matching, so as to find a plurality of feature images close to the query image in the image dataset; S3, using a combined guidance model to estimate the correspondence between the query image and each feature image to generate multiple groups of 2D-2D matching image pairs, wherein the combined guidance model is composed of two or more matching models, and one of the matching models is selected as a guidance model, and the remaining models except the guidance model are screened by using the guidance model during the matching process to obtain preliminary matching pairs, and then the preliminary matching pairs are screened again to obtain final matching pairs; S4, rendering a depth map in real time according to the dense scene model, converting each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map, and calculating corresponding posture information of the drone through the 2D-3D matching pairs to achieve drone positioning.
2. The UAV visual positioning method based on combined guidance model image matching according to claim 1, It is characterized in that Step S1 includes: Acquire 3D point cloud data in real scenes through the radar carried by drones; The triangular mesh surface corresponding to the 3D point cloud data is generated by Poisson surface reconstruction to obtain a dense scene model based on the mesh map corresponding to the real scene.
3. The UAV visual positioning method based on combined guidance model image matching according to claim 1, It is characterized in that Step S2 also includes constructing an image data set, and the steps of constructing the image data set include: Images are captured by a camera carried by a simulated drone at multiple different locations in a simulated environment. During the shooting process, the yaw angle, pitch angle and altitude of the drone are continuously adjusted, and the real-time 6D attitude information and intrinsic parameters of the camera are recorded to obtain multiple simulated images annotated with the drone attitude information. The multiple simulated images constitute an image data set of the simulated scene model.
4. The UAV visual positioning method based on combined guidance model image matching according to claim 1, It is characterized in that In step S2, inputting the query image into the image dataset for matching includes: Acquire local features of each of the query images and perform clustering to obtain global features of each of the query images; The global features of each of the query images are matched with the feature descriptors of each image in the image dataset to find a plurality of feature images that are closest to each of the query images.
5. The UAV visual positioning method based on combined guidance model image matching according to claim 1, It is characterized in that Step S3 includes: Combining two or more matching models, and selecting one of the matching models as a guiding model to form a combined guiding model; Based on the combined guidance model, the query image and each feature image are matched respectively by using each matching model to obtain a plurality of bidirectional matching pairs; Screening out matching pairs that satisfy the cycle consistency constraint from the bidirectional matching pairs obtained by matching the respective matching models, using the guided model to estimate the basic matrix, and performing epipolar geometric constraints on the respective bidirectional matching pairs obtained by matching the remaining models except the guided model based on the basic matrix, so as to preliminarily screen out a plurality of preliminary matching pairs between the query image and the respective feature images; The matching pairs satisfying the cycle consistency constraint in each of the preliminary matching pairs are screened to estimate the basic matrix again, and the matching pairs satisfying the epipolar geometry constraint are screened again based on the basic matrix to obtain the final matching pairs.
6. The UAV visual positioning method based on combined guidance model image matching according to claim 5, It is characterized in that The expression of the cycle consistency constraint is: , In the above formula, is the coordinate of point i in the captured image A in the forward matching from the captured image to the feature image, is the coordinate of the corresponding point i in the feature image B, is the coordinate of point i in the captured image A in the backward matching from the feature image to the captured image, is the coordinate of the corresponding point i in the feature image B, is the first threshold.
7. The UAV visual positioning method based on combined guidance model image matching according to claim 5, It is characterized in that The expression of the epipolar geometric constraint is: , In the above formula, The straight line corresponding to the feature point in the captured image A and the feature image B is is the basic matrix, is the epipolar distance, is the second threshold.
8. The UAV visual positioning method based on combined guidance model image matching according to any one of claims 1 to 7, It is characterized in that In step S4, each of the 2D-2D matching pairs is converted into a 2D-3D matching pair according to the depth map, and corresponding posture information of the drone is calculated through the 2D-3D matching pairs, including: Calculating the pixel coordinates of a plurality of feature images corresponding to the key points of the query image in each of the 2D-2D matching pairs through the depth map to correspond to the three-dimensional coordinate values in the dense scene model, and processing the plurality of three-dimensional coordinate values to obtain final three-dimensional coordinates by minimizing the reprojection error, so as to convert the 2D-2D matching pairs into 2D-3D matching pairs; The position and posture of the drone camera are solved according to the coordinates of the 3D points in the 2D-3D matching, the coordinates of the corresponding 2D points and the internal reference matrix to calculate the position and posture information of the drone corresponding to the current image shooting and realize the positioning of the drone.
9. A UAV visual positioning device based on combined guidance model image matching, It is characterized in that include: The dense scene model building module is used to build a dense scene model based on the mesh map according to the scene point cloud data obtained by scanning with the 3D laser radar carried by the drone; A first image matching module is used to use a captured image obtained during the real-time flight of the UAV as a query image, and input the query image into an image data set for matching, so as to find a plurality of feature images close to the query image in the image data set; a second image matching module, configured to estimate the correspondence between the query image and each feature image using a combined guiding model to generate a plurality of 2D-2D matching image pairs, wherein the combined guiding model is composed of a combination of two or more matching models, and one of the matching models is selected as a guiding model, and the remaining models except the guiding model are screened using the guiding model during the matching process to obtain preliminary matching pairs, and then the preliminary matching pairs are screened for a second time to obtain final matching pairs; The information extraction module is used to render a depth map in real time according to the dense scene model, convert each of the 2D-2D matching pairs into a 2D-3D matching pair according to the depth map, and calculate the corresponding posture information of the drone through the 2D-3D matching pairs to realize drone positioning.
10. A computer-readable storage medium having a computer program stored therein. It is characterized in that The computer program is used to be programmed or configured by a microprocessor to execute the UAV visual positioning method based on combined guidance model image matching as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Image positioning method and device based on ray model three-dimensional reconstruction
CN105844696A
Visual positioning method, system and device based on parallel search 2D-3D matching
CN111583332A
Visual repositioning method and system
CN112132900A
Phase-control-free live-action three-dimensional modeling surveying and mapping method and system based on unmanned aerial vehicle
CN112985361A
Analysis method and system based on autonomous flight situation awareness trend of unmanned aerial vehicle
CN113568435A
Cited By
Attitude estimation method, system and equipment of fundus surgical needle and navigation positioning method
CN119887914A
Methods, systems, and devices for needle attitude estimation and navigation positioning in retinal surgery
CN119887914B
Unmanned aerial vehicle indoor positioning method and system based on three-dimensional Gaussian splash rendering scene
CN120318481A
Road target object visual positioning system and method based on urban streetscape
CN120472006A
Unmanned aerial vehicle high-precision space positioning and virtual-real mapping three-dimensional reconstruction method
CN122066869A
