An unmanned aerial vehicle visual positioning method based on inclined three-dimensional model

Abstract

The application provides an unmanned aerial vehicle visual positioning method based on an inclined three-dimensional model, and an inclined three-dimensional model constructed by using inclined photogrammetry technology is used as an a priori geographic space database; firstly, the three-dimensional model is rendered to generate an optical image library of an orthographic perspective and a corresponding digital surface model; subsequently, for a query image captured in real time by the unmanned aerial vehicle, the system sequentially performs two key steps: a first step of fast image retrieval based on feature similarity, which efficiently screens the most similar candidate images from the massive image library; and a second step of fine image matching based on the candidate images and the query image, so as to establish a high-quality 2D-3D matching pair. Finally, the spatial corresponding relationship of the matching pair is solved, so that the three-dimensional position of the unmanned aerial vehicle image in the world coordinate system can be calculated in real time and accurately. The method realizes high-precision and fully autonomous position and attitude estimation of the unmanned aerial vehicle in a known wide range of scenes.

Description

Technical Field

[0001] This invention belongs to the field of visual positioning, specifically a visual positioning method for unmanned aerial vehicles (UAVs) based on a tilted 3D model. Background Technology

[0002] In recent years, drone technology has been increasingly widely used in smart cities, agricultural monitoring, and emergency rescue. However, its accurate positioning in complex environments still faces severe challenges, such as the susceptibility of satellite signals to blockage, dynamic interference, and the high cost of high-precision positioning equipment. Overcoming these technological bottlenecks requires the deep integration of artificial intelligence, advanced manufacturing, and unmanned systems technology, and a strong push for innovation in intelligent sensing and autonomous control technologies. Driven by both policy and technology, developing a low-cost, high-precision positioning method that does not rely on satellite signals is crucial for improving the autonomous operation capabilities and reliability of drones in complex scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a UAV visual positioning method based on a tilted 3D model to improve the real-time performance and accuracy of UAV position perception in complex environments, effectively solving the problems of traditional satellite positioning methods such as failure in signal obstruction areas, weak anti-interference capability, and high cost of high-precision equipment. The UAV visual positioning method includes:

[0004] The process involves loading pre-built tilted 3D model data and generating a digital orthorectified reference image library and a digital surface model covering the target area using rendering technology. Real-time UAV-acquired images are used as query images. A pre-trained deep convolutional neural network is used to extract global feature vectors from both the query images and the images in the orthorectified reference image library. The similarity between the two global feature vectors is calculated, and the top K images with high similarity are selected as candidate reference images, where K is a natural number greater than 0. Cross-view image matching is performed between the query images and the candidate reference images to obtain a set of 2D-2D matching pairs. Then, based on the coordinate correspondence between the orthorectified reference image library and the digital surface model, a set of 2D-3D matching pairs between the query images and the digital surface model is obtained. The 2D-3D matching pair set is iteratively optimized using a random sampling consensus algorithm to obtain optimal image pose transformation parameters. Based on the optimal image pose transformation parameters, a geometric solution method is used to calculate the 3D position coordinates and three attitude angle parameters of the query image in the world coordinate system, resulting in the UAV pose estimation result.

[0005] Preferably, the cross-view image matching is divided into a coarse matching stage and a fine matching stage, specifically including:

[0006] First, a feature pyramid network is used to extract multi-scale feature representations of the query image and candidate reference images.

[0007] Then, in the coarse matching stage, a global association between the query image and the candidate reference image is established through self-attention and cross-attention mechanisms. Initial matching pairs are generated at each position of the feature map and the matching confidence is calculated. Matching pairs with confidence scores higher than a preset threshold are selected as coarse matching results.

[0008] Finally, in the fine matching stage, the coarse matching result is used as a priori, and the matching position is refined within the local feature window through a multilayer perceptron network. Mismatches are eliminated based on the fundamental matrix constraints, resulting in a set of 2D-2D matching pairs between the query image and the candidate image.

[0009] Preferably, the matching confidence score is calculated using a double softmax operation, specifically:

[0010] ,

[0011] in, This represents the matching confidence of the matching pair (i,j). This represents the original similarity score between the feature vector of point i in the query image and the feature vector of point j in the candidate reference image. This represents the sum of the exponential similarities between a single point i in the query image and all possible matching points in the candidate reference image. This represents the sum of the exponential similarities of a single point j in the candidate reference image with all possible matching points in the query image.

[0012] Preferably, when calculating the similarity between the global feature vector of the query image and the global feature vector of the image in the orthophoto reference image library, all global feature vectors are first subjected to L2 normalization and then calculated using cosine similarity.

[0013] Preferably, the iterative optimization of the 2D-3D matching pair set using the random sampling consensus algorithm specifically includes:

[0014] Initialize the RANSAC algorithm parameters, including setting the maximum number of iterations, the interior point decision threshold, and the confidence level.

[0015] The minimum sample set is randomly selected from the initial 2D-3D matching pair set. Using the selected minimum sample set, the model parameter assumptions for the current iteration are calculated to generate a preliminary pose transformation model.

[0016] Based on the model parameter assumptions of the current iteration, the reprojection error of all matching pairs is calculated. Matching points with errors less than a preset threshold are marked as inliers, and the current number of inliers is counted. Then, multiple iterations are performed to dynamically update the optimal model and retain the model parameters with the largest set of inliers.

[0017] The model with the most inliers is selected as the final estimation result. The model parameters are refined using all inliers, and all outliers are removed. The optimized 2D-3D matching pairs and the optimal image pose transformation parameters are output.

[0018] Preferably, the geometric solution method can employ spatial forward intersection or perspective n-point positioning.

[0019] Preferably, perspective n-point localization is used to calculate the three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system. Specific steps include: constructing a mathematical model for perspective n-point localization, establishing the projection geometric relationship between two-dimensional pixel coordinates and three-dimensional spatial coordinates, and clarifying the camera extrinsic parameters to be solved; using the direct linear transformation method to solve the initial solution of the perspective n-point localization problem, and calculating a coarse estimate of the camera pose by constructing a system of linear equations; based on the initial solution, establishing a nonlinear optimization objective function to minimize the reprojection error, and refining the camera pose parameters through an iterative optimization algorithm; using the optimized pose parameters to calculate the precise three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system; verifying the rationality of the solution results, checking whether the reprojection error is within the allowable range, and outputting the UAV pose estimation result.

[0020] Preferably, the step of generating a digital orthophoto reference image library and a digital surface model covering the target area using rendering technology specifically includes:

[0021] Based on the loaded tilted 3D model data, a virtual orthophoto camera is constructed, its projection matrix is ​​set to cover the geographical range of the target area, and the resolution of the output image is determined.

[0022] Configure the frame buffer object, create a color attachment to store orthophotos, and a depth attachment to record the height information of the digital surface model, thus establishing a complete off-screen rendering pipeline.

[0023] Perform orthophoto rendering, draw the 3D model to the frame buffer, generate an orthophoto containing the true color information of the ground surface, and obtain the ground elevation value corresponding to each pixel.

[0024] The rendering results are read from the frame buffer, the color buffer data is saved as an orthophoto, the depth buffer data is converted into actual elevation values, and a digital surface model is constructed.

[0025] The present invention also provides a UAV visual positioning system based on a tilted three-dimensional model, comprising at least a microprocessor and a memory, wherein the microprocessor is programmed or configured to perform the steps of the above-described UAV visual positioning method, or the memory stores a computer program programmed or configured to perform the above-described UAV visual positioning method.

[0026] The present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform the above-described UAV visual positioning method.

[0027] This invention addresses the problems of insufficient positioning accuracy and poor stability of UAVs relying on GPS signals in complex environments. It proposes a UAV visual positioning method based on a tilted 3D model. This method deeply integrates the prior 3D model with real-time visual perception, achieving high-precision, fully autonomous position and attitude estimation of UAVs in known large-scale scenes, and outputting positioning results with confidence assessment. This provides key technical support for reliable navigation of UAVs in tasks such as urban patrol and emergency rescue. Attached Figure Description

[0028] Figure 1 This is a flowchart of the method described in an embodiment of the present invention.

[0029] Figure 2 This is a visualization of the image retrieval results of the method described in the embodiments of the present invention.

[0030] Figure 3 This is a visualization of the cross-view image matching results of the method described in the embodiments of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] like Figure 1 As shown, the UAV visual positioning method involved in this embodiment includes the following steps:

[0033] The first step is to load the pre-built tilted 3D model data and generate a digital orthophoto library and digital surface model covering the target area using rendering technology.

[0034] This step, reading the tilted 3D model data, includes the following steps:

[0035] First, the osgDB module in the OSG (OpenSceneGraph) library is used to read the tilted 3D model data in OSGB format and load it into a scene graph node structure.

[0036] Then, using the osgUtil module of the OSG library, the scene graph is traversed to extract the geometric information of the model, including vertices, normals, and vertex indices that make up the triangular facets.

[0037] Finally, the OSG texture class from the OSG library is used to read and bind the texture image corresponding to the model. Simultaneously, the texture coordinates of the geometry are parsed to establish a mapping between vertex geometric coordinates and texture coordinates.

[0038] The process of generating a digital orthophoto reference image library and a digital surface model covering the target area using rendering technology specifically includes: constructing a virtual orthophoto camera based on the loaded tilted 3D model data using the osg::Camera class in the OSG library, setting its projection matrix to cover the geographical range of the target area, and determining the resolution of the output image; configuring a frame buffer object, creating a color attachment to store the orthophoto image, and a depth attachment to record the height information of the digital surface model, establishing a complete off-screen rendering pipeline; performing orthophoto rendering, drawing the 3D model onto the frame buffer, generating an orthophoto image containing real surface color information, and simultaneously obtaining the ground elevation value corresponding to each pixel; reading the rendering results from the frame buffer, saving the color buffer data as an orthophoto image, converting the depth buffer data into actual elevation values, and constructing the digital surface model; and post-processing the generated digital orthophoto reference image library (DOM) and digital surface model (DSM), including coordinate system unification and data format standardization, registering them in a priori spatial database, and establishing a spatial index for subsequent retrieval.

[0039] The second step is a fast hierarchical image retrieval based on feature similarity. Images captured in real-time by the UAV are used as query images. A pre-trained deep convolutional neural network is used to extract global feature vectors from both the query image and images in the orthorectified reference image database. The similarity between the two global feature vectors is calculated, and the top K images with the highest similarity are selected as candidate reference images, where K is a natural number greater than 0.

[0040] First, a pre-trained deep convolutional neural network is used as a feature extractor to perform forward propagation on the query images collected in real time by the UAV and the images in the orthophoto reference image library to extract high-dimensional deep feature vectors.

[0041] Then, the extracted feature vectors are subjected to L2 normalization to eliminate the influence of feature magnitude on similarity calculation, and a query feature vector and reference feature vector library are constructed.

[0042] Secondly, cosine similarity is used as the metric to calculate the similarity score between the query feature vector and each feature vector in the reference feature vector library:

[0043] ,

[0044] The dot product of vectors A and B represents the degree of matching of the values ​​of the two vectors in corresponding dimensions. and Let A and B represent the Euclidean norms (i.e., the length or modulus of the vectors) of vectors A and B, respectively. They are obtained by calculating the square root of the sum of the squares of the vector components. and These are the components of vectors A and B in the i-th dimension, respectively.

[0045] Finally, the similarity scores are sorted in descending order, and the top K reference images with the highest similarity are selected as candidate matching results to form a candidate image list.

[0046] The third step is to perform cross-view image matching between the query image and the candidate reference image to obtain a set of 2D-2D matching pairs between the query image and the candidate image. Then, based on the coordinate correspondence between the orthophoto reference image library and the digital surface model, a set of 2D-3D matching pairs between the query image and the digital surface model is obtained.

[0047] The cross-view image matching is divided into a coarse matching stage and a fine matching stage, specifically including:

[0048] First, a feature pyramid network is used to extract multi-scale feature representations of the query image and candidate reference images.

[0049] Then, in the coarse matching stage, a global association between the query image and the candidate reference image is established through self-attention and cross-attention mechanisms. Initial matching pairs are generated at each position of the feature map and the matching confidence is calculated. Matching pairs with confidence scores higher than a preset threshold are selected as coarse matching results.

[0050] Finally, in the fine matching stage, the coarse matching result is used as a priori, and the matching position is refined within the local feature window through a multilayer perceptron network. Mismatches are eliminated based on the fundamental matrix constraints, resulting in a set of 2D-2D matching pairs between the query image and the candidate image.

[0051] The matching confidence score is calculated using a double softmax operation, specifically:

[0052] ,

[0053] in, This represents the matching confidence of the matching pair (i,j). This represents the original similarity score between the feature vector of point i in the query image and the feature vector of point j in the candidate reference image. This represents the sum of the exponential similarities between a single point i in the query image and all possible matching points in the candidate reference image. This represents the sum of the exponential similarities of a single point j in the candidate reference image with all possible matching points in the query image.

[0054] The fourth step involves using a random sampling consensus algorithm to iteratively optimize the 2D-3D matching pair set to obtain the optimal image pose transformation parameters.

[0055] In this step, Random Sampling Consensus (RANSAC) is used for optimization iteration and robust estimation, specifically including the following sub-steps:

[0056] (1) Initialize the RANSAC algorithm parameters, including setting the maximum number of iterations, the interior point decision threshold and the confidence level, to provide termination conditions for the iteration process.

[0057] (2) Randomly select a minimum sample set from the initial 2D-3D matching pair set. The size of the sample set depends on the minimum number of parameters required to estimate the model.

[0058] (3) Using the extracted minimum sample set, calculate the model parameter assumptions for the current iteration and generate a preliminary pose transformation model.

[0059] (4) Based on the current model assumptions, calculate the reprojection error of all matching pairs, mark the matching points with errors less than the preset threshold as interior points, and count the current number of interior points.

[0060] (5) Repeat steps (2) to (4) for multiple iterations, dynamically update the optimal model, and retain the model parameters with the largest set of interior points.

[0061] (6) Select the model with the most inliers as the final estimation result, refine the model parameters using all inliers through an optimization algorithm, remove all outliers, and output the optimized 2D-3D matching pairs and robust pose estimation results.

[0062] The fifth step involves using the optimal image pose transformation parameters to calculate the three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system using a geometric solution method, thereby obtaining the UAV pose estimation result.

[0063] In this step, the geometric solution method can employ spatial forward intersection or perspective n-point positioning (PnP). This embodiment uses a PnP-based UAV pose calculation method, with specific steps including:

[0064] (1) Prepare a set of high-quality 2D-3D matching pairs filtered by RANSAC, where 2D points are the pixel coordinates in the query image and 3D points are the corresponding three-dimensional spatial coordinates in the world coordinate system.

[0065] (2) Construct a mathematical model for perspective n-point positioning, establish the projection geometric relationship between two-dimensional pixel coordinates and three-dimensional spatial coordinates, and clarify the camera extrinsic parameters to be solved.

[0066] (3) The initial solution to the PnP problem is obtained by using the direct linear transformation method, and a rough estimate of the camera pose is calculated by constructing a system of linear equations.

[0067] (4) Based on the initial solution, a nonlinear optimization objective function for minimizing the reprojection error is established, and the camera pose parameters are refined through an iterative optimization algorithm.

[0068] (5) Using the optimized pose parameters, calculate the precise three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system.

[0069] (6) Verify the rationality of the solution results, check whether the reprojection error is within the allowable range, and output the final UAV pose estimation result.

[0070] To verify the effectiveness of the method in this embodiment, this embodiment visualizes the results of the fast hierarchical image retrieval based on feature similarity and the cross-view image matching results in the second step described above. (Appendix) Figure 2 This is a visualization of the fast hierarchical image retrieval results based on feature similarity. From left to right, it shows: the query image from the perspective of a drone, a large-scale map from the orthophoto perspective, and candidate reference images from the orthophoto perspective with similarity ranking 1-3. Figure 3 A visualization of cross-view image matching results.

Claims

1. A visual positioning method for unmanned aerial vehicles (UAVs) based on a tilted 3D model, characterized in that, Load pre-built tilted 3D model data and generate a digital orthophoto library and digital surface model covering the target area using rendering technology; The images collected in real time by the UAV are used as query images. A pre-trained deep convolutional neural network is used to extract the global feature vectors of the query images and the images in the orthophoto reference image library, respectively. The similarity between the two global feature vectors is calculated, and the top K images in the orthophoto reference image library with high similarity are selected as candidate reference images, where K is a natural number greater than 0. Cross-view image matching is performed between the query image and the candidate reference image to obtain a set of 2D-2D matching pairs between the query image and the candidate image. Then, based on the coordinate correspondence between the orthophoto reference image library and the digital surface model, a set of 2D-3D matching pairs between the query image and the digital surface model is obtained. The optimal image pose transformation parameters are obtained by iteratively optimizing the 2D-3D matching pair set using the random sampling consensus algorithm. Based on the optimal image pose transformation parameters, the three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system are calculated using a geometric solution method to obtain the UAV pose estimation result. The cross-view image matching is divided into a coarse matching stage and a fine matching stage, specifically including: First, a feature pyramid network is used to extract multi-scale feature representations of the query image and candidate reference images; Then, in the coarse matching stage, a global association between the query image and the candidate reference image is established through self-attention mechanism and cross-attention mechanism. Initial matching pairs are generated at each position of the feature map and the matching confidence is calculated. Matching pairs with confidence scores higher than a preset threshold are selected as coarse matching results. Finally, in the fine matching stage, the coarse matching result is used as a priori, and the matching position is refined through a multilayer perceptron network within the local feature window. Mismatches are eliminated based on the fundamental matrix constraints, and a set of 2D-2D matching pairs between the query image and the candidate image is obtained. The matching confidence score is calculated using a double softmax operation, specifically: ， in, This represents the matching confidence of the matching pair (i,j). This represents the original similarity score between the feature vector of point i in the query image and the feature vector of point j in the candidate reference image. This represents the sum of the exponential similarities between a single point i in the query image and all possible matching points in the candidate reference image. This represents the sum of the exponential similarity of a single point j in the candidate reference image with all possible matching points in the query image. The generation of a digital orthophoto library and a digital surface model covering the target area using rendering technology specifically includes: Based on the loaded tilted 3D model data, a virtual orthophoto camera is constructed, its projection matrix is ​​set to cover the geographical range of the target area, and the resolution of the output image is determined. Configure the frame buffer object, create a color attachment to store orthophotos, and a depth attachment to record the height information of the digital surface model, thus establishing a complete off-screen rendering pipeline. Perform orthophoto rendering, draw the 3D model to the frame buffer, generate an orthophoto containing real color information of the ground surface, and obtain the ground elevation value corresponding to each pixel. The rendering results are read from the frame buffer, the color buffer data is saved as an orthophoto, the depth buffer data is converted into actual elevation values, and a digital surface model is constructed.

2. The UAV visual positioning method according to claim 1, characterized in that, When calculating the similarity between the global feature vector of the query image and the global feature vector of the image in the orthophoto reference image library, L2 normalization is first performed on all global feature vectors, and then cosine similarity is used for calculation.

3. The UAV visual positioning method according to claim 1, characterized in that, The iterative optimization of the 2D-3D matching pair set using the random sampling consensus algorithm specifically includes: Initialize the RANSAC algorithm parameters, including setting the maximum number of iterations, the interior point decision threshold, and the confidence level; The minimum sample set is randomly selected from the initial 2D-3D matching pair set. Using the selected minimum sample set, the model parameter assumptions for the current iteration are calculated to generate a preliminary pose transformation model. Based on the model parameter assumptions of the current iteration, calculate the reprojection error of all matching pairs, mark the matching points with errors less than a preset threshold as inliers, count the current number of inliers, and then perform multiple iterations to dynamically update the optimal model, retaining the model parameters with the largest set of inliers. The model with the most inliers is selected as the final estimation result. The model parameters are refined using all inliers, and all outliers are removed. The optimized 2D-3D matching pairs and the optimal image pose transformation parameters are output.

4. The UAV visual positioning method according to claim 1, characterized in that, The geometric solution method can employ spatial forward intersection or perspective n-point positioning.

5. The UAV visual positioning method according to claim 4, characterized in that, The three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system are calculated using perspective n-point positioning. The specific steps include: Construct a mathematical model for perspective n-point localization, establish the projection geometric relationship between two-dimensional pixel coordinates and three-dimensional spatial coordinates, and clarify the camera extrinsic parameters to be solved; The initial solution to the perspective n-point localization problem is obtained by using the direct linear transformation method, and a rough estimate of the camera pose is calculated by constructing a system of linear equations. Based on the initial solution, a nonlinear optimization objective function that minimizes the reprojection error is established, and the camera pose parameters are refined through an iterative optimization algorithm. Using the optimized pose parameters, the precise three-dimensional position coordinates and three attitude angle parameters of the query image in the world coordinate system are calculated. Verify the rationality of the solution results, check whether the reprojection error is within the allowable range, and output the UAV pose estimation results.

6. A UAV visual positioning system based on an inclined 3D model, comprising at least a microprocessor and a memory, characterized in that, The microprocessor is programmed or configured to perform the steps of the UAV visual positioning method according to any one of claims 1 to 5, or the memory stores a computer program programmed or configured to perform the UAV visual positioning method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is programmed or configured to perform the UAV visual positioning method according to any one of claims 1 to 5.

Images