Novel multi-scale scene matching positioning algorithm

Abstract

The invention discloses a novel multi-scale scene matching positioning algorithm. The algorithm comprises the following steps: S1, obtaining an image shot by an unmanned aerial vehicle in real time and a pre-stored reference satellite image; s2, inputting the registered real-time image and the reference satellite image into an improved local feature matching model for matching to obtain a high-precision matching point pair set; s3, based on the matching point pair set obtained in the S2, estimating a homography matrix from the shot real-time image to the reference satellite image by adopting a random sampling consistency algorithm; and projecting a central point of a shot real-time image to a reference satellite image coordinate system through the homography matrix, and calculating a real-time geographic position coordinate of the unmanned aerial vehicle in combination with geographic coordinate information of the reference satellite image. Compared with the prior art, the method has the advantages that large-scale scene changes can be effectively processed, and the new multi-scale scene matching and positioning algorithm is higher in precision and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and unmanned aerial vehicle navigation, and particularly relates to a new multi-scale scene matching positioning algorithm. BACKGROUND

[0002] In the field of global positioning and navigation of unmanned aerial vehicles, global navigation satellite system (GNSS) and inertial navigation system (INS) are two widely used core technologies.

[0003] GNSS can calculate the spatial position and velocity of a carrier by receiving signals from multiple satellites, and has the advantage of providing continuous high-precision positioning information covering the entire globe.

[0004] However, the performance of GNSS is highly dependent on the quality of satellite signal reception. When an unmanned aerial vehicle is in a complex environment such as a city, a canyon, a dense forest, a tunnel, or an environment with strong electromagnetic interference, the signal is easily affected by shielding, reflection, or multipath effects, resulting in a significant decrease in positioning accuracy or even complete failure, which can seriously affect the mission reliability and survivability of the unmanned aerial vehicle.

[0005] INS mainly relies on inertial measurement units such as gyroscopes and accelerometers to calculate the attitude, velocity, and position of the unmanned aerial vehicle through integration.

[0006] The biggest advantage of INS is its strong autonomy and independence from any external signals. However, the errors in the position and velocity parameters calculated by INS will continue to accumulate and diverge over time,

[0007] Therefore, INS alone cannot meet the long-term and high-precision positioning requirements of unmanned aerial vehicles. With the rapid development of computer vision and artificial intelligence technology, visual positioning technology has become the focus of research on unmanned aerial vehicle positioning solutions.

[0008] Visual positioning technology uses an on-board camera to continuously capture image sequences of the ground below, and uses computer vision technology to compare real-time images with pre-stored reference satellite maps that have geographic coordinate information, thereby calculating the precise position of the unmanned aerial vehicle in the global coordinate system.

[0009] However, due to factors such as the camera parameters of the unmanned aerial vehicle, changes in flight attitude, changes in flight altitude, changes in roll and pitch angles caused by shaking of the unmanned aerial vehicle body, and other factors, there are often multi-scale differences between the real-time images captured by the unmanned aerial vehicle and the pre-stored reference satellite maps, which leads to a decrease in the matching accuracy of traditional matching algorithms and makes it difficult to meet the requirements of real-time and high-precision positioning.

[0010] Therefore, there is an urgent need for an unmanned aerial vehicle scene matching and positioning solution that can effectively handle large-scale scene changes and has higher accuracy and robustness. SUMMARY

[0011] The technical problem solved by the present application is to solve the problem of low matching accuracy of unmanned aerial vehicles in multi-scale scenes, which affects the decline of positioning accuracy. The present application proposes an image registration method based on inverse mapping and cubic convolution interpolation, and improves the LoFTR feature matching method combined with DBSCAN clustering and triple attention mechanism to realize high-precision, low-delay unmanned aerial vehicle scene matching and autonomous positioning.

[0012] To solve the above technical problems, the technical solution provided by the present application is: an image registration method based on inverse mapping and cubic convolution interpolation, which performs direction and scale registration on real-time images taken by an unmanned aerial vehicle, then improves the LoFTR feature matching algorithm combined with DBSCAN clustering and triple attention mechanism, matches the real-time images taken by the unmanned aerial vehicle with the reference satellite images, and maps the center pixel point of the real-time images to the reference satellite images, solving the problem of scale and direction difference between images, so that the unmanned aerial vehicle can accurately realize scene matching and autonomous positioning.

[0013] The present application adopts the following technical solution: an image registration method based on inverse mapping and cubic convolution interpolation combined with DBSCAN clustering and triple attention mechanism to improve the LoFTR feature matching algorithm, which includes the following steps:

[0014] Obtain the real-time images taken by the unmanned aerial vehicle and the pre-stored reference satellite images, use the on-board sensor data to correct the distortion, direction registration and scale registration of the real-time images taken by the unmanned aerial vehicle, so that they are aligned in geometric properties with the reference satellite images. This step is a pre-processing step for image registration;

[0015] Input the real-time images taken by the unmanned aerial vehicle and the reference satellite images after registration into the improved local feature matching model for matching to obtain a high-precision matching point pair set. The improved local feature matching model introduces the DBSCAN clustering algorithm in the LoFTR algorithm framework to eliminate false matches and crop the common view area, and uses the triple attention mechanism to enhance and match the features in the common view area. This step is a feature matching step based on the improved LoFTR algorithm;

[0016] Based on the matching point pair set obtained in step 2, estimate the homography matrix of the real-time images taken by the unmanned aerial vehicle to the reference satellite images using the random sample consensus algorithm, project the center point of the real-time images taken by the unmanned aerial vehicle to the reference satellite image coordinate system through the homography matrix, and then combine the geographic coordinate information of the reference satellite image to calculate the real-time geographic position coordinates of the unmanned aerial vehicle.

[0017] Compared with the prior art, the present application has the following advantages:

[0018] The present application greatly eliminates the differences in scale and rotation between source images through a precise geometric registration preprocessing step, laying a good foundation for subsequent matching. The improved matching algorithm further improves the matching accuracy in complex scenes, and finally realizes the positioning accuracy of meters;

[0019] The DBSCAN clustering can effectively suppress false matching, and the triple attention mechanism enhances the discrimination ability of the features in the common view area, so that the present application has better robustness to interference factors such as light changes, partial occlusions and repeated textures in the scene;

[0020] Compared with pure deep learning methods, the present application combines sensor-assisted geometric correction, reduces the over-reliance on data-driven models, meets the strict requirements of unmanned aerial vehicles on volume, weight and power consumption, and has high practical application value.

[0021] Compared with pure deep learning methods, the present application combines sensor-assisted geometric correction, reduces the over-reliance on data-driven models, meets the strict requirements of unmanned aerial vehicles on volume, weight and power consumption, and has high practical application value. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The present application is a general flowchart

[0023] Figure 2 The present application is a multi-scale scene image registration flowchart based on reverse mapping and cubic convolution interpolation

[0024] Figure 3 The present application is an improved LoFTR feature matching algorithm network framework diagram

[0025] Figure 4 The structure diagram of the triple attention mechanism. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below with reference to the accompanying drawings.

[0027] The specific implementation is as follows:

[0028] During flight, the onboard camera captures real-time scene images at a specific frame rate;

[0029] The onboard computer reads the attitude angle data from the IMU through the OSDK interface, including the yaw angle, pitch angle, roll angle, and reads the height data of the current unmanned aerial vehicle from the barometer;

[0030] Load the pre-stored reference satellite image into the onboard memory;

[0031] At this point, the data acquisition is complete;

[0032] A multi-scale scene image registration process based on reverse mapping and cubic convolution interpolation is shown in Figure 2 and specifically includes the following steps:

[0033] The pre-calibrated camera intrinsic parameters and distortion coefficients are called to correct the distortion of each captured real-time scene image,

[0034] including radial distortion and tangential distortion;

[0035] The mathematical model of radial distortion is

[0036] The mathematical model of tangential distortion is

[0037] where (x, y) is the ideal coordinate position without distortion, (x0, y0) is the coordinate position after distortion, k1, k2 and k3 are radial distortion parameters, r is the distance between (x, y) and the principal point, and satisfies the relationship of 2 2 2

[0038] According to the current heading angle, roll angle and pitch angle, a geometric transformation matrix in the image plane is constructed, and then reverse mapping is performed on the scene image after distortion correction.

[0039] The size of the output image after registration is set, and each pixel point of the output image is traversed. The coordinates of each pixel point in the original image are calculated using the inverse matrix.

[0040] If the coordinates in the original image are non-integer coordinates, the 16 pixel points in the 4x4 neighborhood around the coordinates are taken, and the 16 weights corresponding to the 16 pixel points are calculated according to the cubic convolution interpolation weight kernel function. The pixel value of each pixel point in the output image is obtained by weighted summation.

[0041] Similarly, the real distance of the real-time image taken by the UAV and the real distance of the reference satellite image are calculated, and the corresponding scale ratio is calculated. The image after direction registration is resampled to complete scale registration, and thus the image registration is completed.

[0042] The registered real-time image and the reference satellite image are input into the improved LoFTR model which has been trained, as shown in Figure 3 and specifically includes the following steps:

[0043] First, the scene image after registration and the reference satellite image are subjected to feature extraction and feature reconstruction.

[0044] Then, coarse-grained matching, DBSCAN clustering and common view area cropping are performed inside the model to find the best feature matching pair.

[0045] ​​​Finally, triple attention feature enhancement, secondary matching and fusion, fine-grained matching (final matching) and other steps are performed to obtain fine matching point pairs, Figure 4 It is a triple attention mechanism structure diagram, and thus the feature matching step is completed.

[0046] The RANSAC algorithm is used for error matching point elimination to provide robust geometric constraints for positioning.

[0047] The homographic transformation matrix H is calculated to convert the real-time image center pixel coordinates obtained by the unmanned aerial vehicle into the reference satellite image

[0048] The corresponding pixel coordinates of the image pair are as follows:

[0049]

[0050] In the formula, (x u , y u ) is the real-time image center pixel coordinates obtained by the unmanned aerial vehicle, (x s , y s ) is the corresponding pixel coordinates in the reference satellite image, and the homographic transformation matrix

[0051] Finally, according to the geographic transformation parameters (such as affine transformation parameters or corner longitude and latitude) of the reference satellite image, the pixel coordinates and longitude and latitude of the reference satellite image are linearly mapped to obtain the position of the unmanned aerial vehicle.

[0052] The protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0053] The present application greatly eliminates the differences in scale and rotation between source images through precise geometric registration preprocessing steps, lays a good foundation for subsequent matching, and further improves the matching accuracy in complex scenes by proposing an improved LoFTR matching algorithm. The improved LoFTR matching algorithm enhances the discrimination ability of the common view area features through DBSCAN clustering and triple attention mechanism, thereby having better robustness to interference factors such as light changes, partial occlusions and scene repeated textures. The method of the present application meets the strict requirements of unmanned aerial vehicles on volume, weight and power consumption, and has high practical application value.

[0054] The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

[0055] It should be noted that like reference numerals and characters refer to like elements throughout the several views of the drawings, and that, unless otherwise indicated, like reference numerals and characters in different figures represent similar items.

[0056] In the present application, unless otherwise explicitly specified and limited, "on" or "under" of a first feature to a second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, "on", "above" and "over" of a first feature to a second feature includes that the first feature is directly above and obliquely above the second feature, or only means that the first feature is higher than the second feature in horizontal height. "Under", "below" and "underneath" of a first feature to a second feature includes that the first feature is directly below and obliquely below the second feature, or only means that the first feature is lower than the second feature in horizontal height.

[0057] The above description of the application and its embodiments is not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired by it, without departing from the purpose of the application, without creative design, similar structure and embodiments of the technical solution can be designed, which should belong to the protection scope of the application.

Claims

1. A novel multi-scale scene matching and localization algorithm, characterized in that, Includes the following steps: S1: Acquire real-time images captured by the drone and pre-stored reference satellite images; Using airborne sensor data, distortion correction, orientation registration, and scale registration are performed on the real-time images captured by the UAV to align them geometrically with the reference satellite images. S2: The registered real-time image and the reference satellite image are input into the improved local feature matching model for matching to obtain a high-precision set of matching point pairs; the improved local feature matching model introduces the DBSCAN clustering algorithm in the LoFTR algorithm framework to remove mismatches and clip common view areas, and uses a triple attention mechanism to enhance and perform secondary matching on common view area features; S3: Based on the set of matching point pairs obtained in S2, the homography matrix from the captured real-time image to the reference satellite image is estimated using the random sampling consensus algorithm; the center point of the captured real-time image is projected onto the coordinate system of the reference satellite image using the homography matrix, and then the real-time geographical coordinates of the UAV are calculated by combining the geographic coordinate information of the reference satellite image.

2. The novel multi-scale scene matching and localization algorithm according to claim 1, characterized in that: The orientation registration in S1 specifically includes: S1.1: The inverse geometric transformation matrix from the onboard camera coordinate system to the onboard reference system based on the UAV attitude angle obtained from the IMU; S1.2: Using the reverse mapping method, for each target pixel in the registered output image, calculate its corresponding source coordinates in the original real-time image. These source coordinates are non-integer coordinates. S1.3: Using cubic convolution interpolation, based on the gray values ​​and weights of the 16 neighboring pixels around the source coordinates, the gray value of the target pixel is calculated to complete the image rotation. The weights are determined by the interpolation weight kernel function based on the distances in the horizontal and vertical directions.

3. A novel multi-scale scene matching and localization algorithm according to claim 2, characterized in that: The scale registration in S1 specifically includes: S1.4: Calculate the real-world distance covered by the real-time images captured by the drone based on the drone's flight altitude obtained from the barometer and the camera's field of view. S1.5: Based on the latitude and longitude information of the corner points of the reference satellite image, calculate the actual distance covered by the reference satellite image using the Haversine formula; S1.6: Calculate the ground sampling distances of the real-time images captured by the UAV and the reference satellite images respectively, and use the ratio of the two as the downsampling ratio; S1.7: Using inverse mapping and cubic convolution interpolation, the real-time images captured by the UAV are resampled according to the downsampling ratio in S1.6 to match the resolution scale of the reference satellite images.

4. A novel multi-scale scene matching and localization algorithm according to claim 1, characterized in that: The improved local feature matching model in S2 specifically includes: S2.1: Use a feature extraction network to obtain low-resolution feature maps of the image, reconstruct the features through the Transformer module, and then perform preliminary matching to obtain initial matching point pairs; S2.2: Perform DBSCAN clustering on the initial matching point pairs described in S2.1, divide the matching points in the feature space into several clusters, select the cluster with the highest pixel density as the effective common viewing area, and filter out the mismatched points in the low-density clusters. S2.3: Crops out the common viewing region on the original image based on the effective matching point pairs, and applies a triple attention mechanism to the image features of the common viewing region. This mechanism captures the cross-dimensional interaction between the channel and the spatial dimension through three parallel branches, and reconstructs the feature representation. S2.4: The results of the secondary matching are fused with the results of the primary matching to obtain coarse-grained matching results. Further fine matching is performed within a local window on the high-resolution feature map of the image. The final matching point pair with sub-pixel precision is obtained by calculating the expected value.

5. A novel multi-scale scene matching and localization algorithm according to claim 1, characterized in that: The UAV positioning calculation in S3 specifically includes: S3.1: The RANSAC algorithm is used to remove mismatched points, providing robust geometric constraints for localization; S3.2: Calculate the homography transformation matrix H to convert the center pixel coordinates of the real-time image acquired by the UAV into the pixel coordinates corresponding to the reference satellite image. The corresponding relationship is shown in the following formula: In the formula, (x u ,y u (x) represents the center pixel coordinates of the real-time image acquired by the UAV. s ,y s ( ) represents the pixel coordinates in the corresponding reference satellite image, and the homography transformation matrix. S3.3: Based on the geographic transformation parameters of the reference satellite image, including affine transformation parameters or corner latitude and longitude, the pixel coordinates of the reference satellite image are linearly mapped to latitude and longitude to obtain the position of the UAV.

Images