A Visual Continuous Positioning Method and System for Unmanned Aerial Vehicles Based on Satellite Remote Sensing Imagery

Abstract

This invention discloses a method and system for continuous visual localization of unmanned aerial vehicles (UAVs) based on satellite remote sensing imagery. The method comprises: a front-end visual tracking module calculating relative positioning in pixel coordinates based on image matching feature point sets, and then transferring the result to UTM coordinates; when a set number of frames are accumulated or visual image matching fails, a back-end satellite imagery absolute localization module uses UAV-captured images and satellite remote sensing imagery to perform absolute localization and pose correction for the UAV; a back-end forward correction module performs forward correction based on the absolute localization information obtained from the back-end satellite imagery absolute localization module, eliminating the accumulated error of the front-end visual tracking result. The system includes a front-end visual tracking module, a back-end satellite imagery absolute localization module, and a back-end forward correction module. This invention improves the positioning accuracy of UAVs using satellite remote sensing imagery, features a simple algorithm, strong real-time performance, and enables continuous visual localization of UAVs.

Description

Technical Field

[0001] This invention relates to the fields of satellite navigation and UAV positioning technology, and in particular to a UAV visual continuous positioning method and system based on satellite remote sensing imagery. Background Technology

[0002] Unmanned aerial vehicles (UAVs), or drones for short, are unmanned aircraft controlled by radio remote control equipment and their own program control devices. Intelligence is one of the core aspects of UAV development, with core technologies including intelligent sensing, positioning, and control. Currently, UAV positioning mainly relies on satellite navigation systems to provide high-precision positioning information. However, satellite navigation information is easily interfered with or blocked, severely affecting UAV mission execution. With the development of satellite remote sensing technology, imaging resolution has reached sub-meter level, providing favorable conditions for satellite image-assisted positioning.

[0003] Traditional UAV positioning in GNSS-denied environments often relies on visual odometry without incorporating historical satellite remote sensing imagery. In purely visual positioning, the lack of absolute correction often leads to significant accumulated errors. However, historical satellite imagery can provide absolute positioning information, thereby correcting the accumulated errors in visual tracking positioning. Therefore, exploring satellite remote sensing imagery-assisted positioning is of great significance.

[0004] Invention CN 111288989 A discloses a visual positioning method for small unmanned aerial vehicles (UAVs). This method extracts visual features of the UAV's surrounding environment from multiple cameras to achieve visual positioning of small UAVs in complex environments. However, it does not incorporate historical satellite imagery and relies solely on existing environmental images for visual positioning, thus limiting its application scope. Invention CN118506197A discloses a UAV visual positioning method suitable for multi-seasonal scenarios. It trains a seasonal invariant feature extraction network for UAV aerial images and satellite images using UAV aerial image datasets and satellite image datasets. A lightweight seasonal invariant feature extraction network is used to perform feature matching on the preprocessed UAV aerial images and satellite images to achieve positioning. This solves the parallax problem caused by scale and rotation variations between aerial and satellite images, as well as the appearance inconsistency caused by seasonal changes between aerial and satellite images, achieving high-precision visual positioning of UAVs. However, the algorithm is relatively complex and difficult to perform dynamic continuous positioning. Summary of the Invention

[0005] The purpose of this invention is to provide a visual continuous positioning method and system for unmanned aerial vehicles (UAVs) based on satellite remote sensing imagery, which features a simple algorithm, strong real-time performance, and high positioning accuracy.

[0006] The technical solution to achieve the objective of this invention is as follows: a UAV visual continuous positioning method based on satellite remote sensing imagery, comprising a front-end visual tracking module, a back-end satellite imagery absolute positioning module, and a back-end forward correction module. The method steps are as follows:

[0007] Step 1: The front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then transforms it into the UTM coordinate system to obtain the positioning result.

[0008] Step 2: When the set number of frames accumulated or visual image matching and tracking fails, the backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between the UAV-captured images and satellite remote sensing images to obtain the current UAV absolute positioning information and corrected pose.

[0009] Step 3: The back-end forward correction module performs forward correction based on the absolute positioning information obtained by the back-end satellite image absolute positioning module to eliminate the cumulative error of the front-end visual tracking results.

[0010] Further, in step 1, the front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then converts it to the UTM coordinate system to obtain the positioning result, as follows:

[0011] Step 1.1: Use XFeat to match the images of consecutive frames to obtain the matching feature point sets P1 and P2 of the two images, where P1 = {p 1_i {i = 1, 2, ..., N1}, P2 = {p 2_i ,i=1,2,...,N2},N1 andN2 are the number of matching feature points in the matching feature point sets P1 and P2, respectively;

[0012] Step 1.2: Solve for the displacement matrix dt using the centroids of the two feature point sets, where dx and dy are the x-axis and y-axis displacements in the pixel coordinate system, respectively.

[0013]

[0014] dt=[dx dy] (2)

[0015] Decentralize the matching feature points:

[0016]

[0017] Where P 1_center P 2_center Let P1 and P2 be the point set distributions of their respective centroid coordinate systems;

[0018] The covariance matrix of the two matching feature point sets is calculated as follows:

[0019]

[0020] Perform singular value decomposition on the covariance matrix:

[0021] H = U·S·V T (5)

[0022] Where U, S, and V are the left singular matrix, singular value matrix, and right singular matrix of H, respectively;

[0023] Calculate the rotation matrix R between two images in pixel coordinates. u The relative transformation angle dθ between the two frames:

[0024] R u =V T ·U T (6)

[0025] dθ=arctan2(R u [1,0],R u [0,0]) (7)

[0026] Among them, R u [1,0]、R u [0,0] represent R respectively u The sum of the items in the second row and first column of R u The item in the first row and first column;

[0027] Step 1.3: Determine the position of the second image in pixel coordinates:

[0028] P pic_2 =P pic_1 +(dx dy dθ) T (8)

[0029] Where P pic_1 P represents the position of the first image in pixel coordinates. pic_2 This represents the position of the second image in pixel coordinates.

[0030] Transform the pixel coordinate system to the UTM coordinate system:

[0031] P W2 =k(R) Wu P pic_2 +t Wu (9)

[0032] Where P W2 R represents the coordinates of the second image in the world coordinate system. Wu t WuThese are the rotation and translation matrices for transforming the pixel coordinate system to the world coordinate system, respectively, and k is the scale transformation coefficient.

[0033] Furthermore, in step 2, when the accumulated set number of frames or visual image matching tracking fails, the backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between the UAV-captured images and satellite remote sensing images to obtain the current UAV absolute positioning information and corrected pose, as follows:

[0034] Step 2.1, Preprocessing and Correction: Based on the current UAV heading information, pre-rotate and correct the image;

[0035] Step 2.2 Feature Extraction and Matching: Use the deep learning SuperGlue model to calculate key points and descriptors of the currently captured image and satellite remote sensing image. Match the two to generate matching feature point pairs, and use the Random Sample Consensus (RANSAC) algorithm to filter out outliers in the matching.

[0036] Step 2.3: Calculate the geographic longitude (Lon) and latitude (Lat) of the image. The formula is:

[0037]

[0038] Lat t Lon t It is the geographic longitude and latitude coordinates of the top left pixel of the satellite image, Lat b Lon b It is the geographic longitude and latitude coordinates of the bottom right pixel in the satellite image; C x C y C represents the pixel coordinates of feature points in a UAV image. x C represents the x-axis coordinates of the feature point. y y is the y-axis coordinate of the feature point; w and h are the width and height of the satellite image, respectively.

[0039] Furthermore, in step 3, the back-end forward correction module performs forward correction based on the absolute positioning information obtained by the back-end satellite image absolute positioning module to eliminate the cumulative error of the front-end visual tracking results, as follows:

[0040] The forward propagation positioning correction is performed based on the satellite absolute positioning result, using the following formula:

[0041] dP W =P' Wk -P Wk (12)

[0042] P' Wi =P Wi +dP W,i∈(k,k+1....n) (13)

[0043] Among them, P' Wk P is the absolute positioning result of the back-end satellite image of the k-th frame. Wk This is the front-end visual tracking and localization result of the k-th frame, where n is the current total number of frames, and dP W P is the change in pose correction. Wi For the front-end visual tracking and localization result of the i-th frame, P' Wi This is the corrected positioning result for the i-th frame.

[0044] A satellite remote sensing image-assisted UAV visual continuous positioning system is disclosed. This system implements the aforementioned satellite remote sensing image-assisted UAV visual continuous positioning method. The system includes a front-end visual tracking module, a back-end satellite image absolute positioning module, and a back-end forward correction module, wherein:

[0045] The front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then transforms it to the UTM coordinate system to obtain the positioning result.

[0046] The backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between images captured by the UAV and satellite remote sensing images to obtain the current absolute positioning information and corrected pose of the UAV.

[0047] The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching back-end satellite images, eliminating the cumulative error of the front-end visual tracking results.

[0048] A mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned UAV visual continuous positioning method based on satellite remote sensing imagery.

[0049] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the UAV visual continuous positioning method based on satellite remote sensing imagery.

[0050] Compared with the prior art, the significant effects of this invention are: (1) the use of a back-end satellite remote sensing image positioning algorithm combined with the SuperGlue algorithm improves the accuracy of satellite remote sensing image positioning; (2) the adoption of a scheme based on front-end visual tracking positioning and back-end satellite remote sensing image absolute positioning has better navigation accuracy compared with pure visual odometry positioning; (3) the algorithm is simple, has strong real-time performance, and can perform continuous visual positioning of UAVs. Attached Figure Description

[0051] Figure 1 This is a structural block diagram of a UAV visual continuous positioning system based on satellite remote sensing imagery, according to the present invention.

[0052] Figure 2 This is a structural block diagram of the backend satellite image absolute positioning module in this invention.

[0053] Figure 3 This is a structural block diagram of the back-end forward correction module in this invention. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0055] Combination Figure 1 The present invention provides a satellite remote sensing image-assisted UAV visual continuous positioning system, comprising a front-end visual tracking module, a back-end satellite image absolute positioning module, and a back-end forward correction module.

[0056] The front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then transforms it to the UTM coordinate system to obtain the positioning result.

[0057] The backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between images captured by the UAV and satellite remote sensing images to obtain the current absolute positioning information and corrected pose of the UAV.

[0058] The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching back-end satellite images, eliminating the cumulative error of the front-end visual tracking results.

[0059] This invention discloses a continuous method for visual absolute positioning of unmanned aerial vehicles (UAVs) based on satellite remote sensing imagery, comprising the following steps:

[0060] Step 1: The front-end visual tracking module uses image matching technology to obtain a set of matching feature points from consecutive frames captured during the drone's flight. It then calculates the drone's pose changes between adjacent frames to obtain relative positioning in pixel coordinates, and finally converts this to UTM coordinates to get the positioning result. The details are as follows:

[0061] Step 1.1: Use XFeat to match the images of consecutive frames to obtain the matching feature point sets P1 and P2 of the two images, where P1 = {p 1_i {i = 1, 2, ..., N1}, P2 = {p 2_i ,i=1,2,...,N2},N1 andN2 are the number of matching feature points in the matching feature point sets P1 and P2, respectively;

[0062] Step 1.2: Solve for the displacement matrix dt using the centroids of the two feature point sets, where dx and dy are the x-axis and y-axis displacements in the pixel coordinate system, respectively.

[0063]

[0064] dt=[dx dy] (2)

[0065] Where N1 and N2 are the number of the two feature point sets;

[0066] Decentralize the matching feature points:

[0067]

[0068] Where P 1_center P 2_center Let P1 and P2 be the point set distributions of their respective centroid coordinate systems.

[0069] The covariance matrix of the two feature point sets is calculated as follows:

[0070]

[0071] Perform singular value decomposition on the covariance matrix:

[0072] H = U·S·V T (5)

[0073] U, S, and V are the left singular matrix, singular value matrix, and right singular matrix of H, respectively.

[0074] Calculate the rotation matrix R between two images in pixel coordinates. u The relative transformation angle dθ between the two frames:

[0075] R u =V T ·U T (6)

[0076] dθ=arctan2(R u [1,0],R u [0,0]) (7)

[0077] Step 1.3: Determine the position of the second image in pixel coordinates:

[0078] P pic_2 =P pic_1 +(dx dy dθ) T (8)

[0079] Where P pic_1 P represents the position of the first image in pixel coordinates. pic_2This represents the position of the second image in pixel coordinates.

[0080] Step 1.4: Transform the pixel coordinate system into the UTM coordinate system:

[0081] P W2 =k(R) Wu P pic_2 +t Wu (9)

[0082] Where R Wu t Wu These are the rotation and translation matrices for transforming the pixel coordinate system to the world coordinate system, respectively, and k is the scale transformation coefficient.

[0083] Where P W2 These are the coordinates of the second image in the world coordinate system.

[0084] Step 2: When the accumulated set number of frames or visual image matching tracking fails, the backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between the UAV-captured images and satellite remote sensing images to obtain the current UAV absolute positioning information and corrected pose, such as... Figure 2 As shown, the details are as follows:

[0085] Step 2.1, Preprocessing and Correction: Based on the current UAV heading information, pre-rotate and correct the image;

[0086] Step 2.2 Feature Extraction and Matching: Use the deep learning SuperGlue model to calculate key points and descriptors of the currently captured image and satellite remote sensing image. Match the two to generate matching feature point pairs, and use the Random Sample Consensus (RANSAC) algorithm to filter out outliers in the matching.

[0087] Step 2.3: Calculate the geographic longitude (Lon) and latitude (Lat) of the image. The formula is:

[0088]

[0089] Lat t Lon t It is the geographic longitude and latitude coordinates of the top left pixel of the satellite image, Lat b Lon b It is the geographic longitude and latitude coordinates of the bottom right pixel in the satellite image, C x C y C represents the pixel coordinates of feature points in a UAV image. x C represents the x-axis coordinates of the feature point. y Let w be the y-axis coordinate of the feature point, and w and h be the width and height of the satellite image, respectively.

[0090] Step 3: The backend forward correction module performs forward correction based on the absolute positioning information obtained by the backend satellite imagery absolute positioning module, eliminating the cumulative error of the frontend visual tracking results, such as... Figure 3 As shown, the details are as follows:

[0091] After obtaining the satellite absolute positioning result, a forward propagation positioning correction is performed, correcting the positioning results of all frames from the query frame to the current frame. The formula is as follows:

[0092] dP W =P' Wk -P Wk (12)

[0093] P' Wi =P Wi +dP W ,i∈(k,k+1....n) (13)

[0094] Where P' Wk P is the absolute positioning result of the back-end satellite image of the k-th frame. Wk This is the front-end visual tracking and localization result of the k-th frame, where n is the current total number of frames, and dP W P is the change in pose correction. Wi For the front-end visual tracking and localization result of the i-th frame, P' Wi This is the corrected positioning result for the i-th frame.

[0095] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method for visual absolute positioning of unmanned aerial vehicles based on satellite remote sensing imagery.

[0096] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps in the satellite remote sensing image-assisted UAV visual absolute positioning method.

[0097] The above specific embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for continuous visual positioning of unmanned aerial vehicles (UAVs) based on satellite remote sensing imagery, characterized in that, The method comprises a front-end visual tracking module, a back-end satellite imagery absolute positioning module, and a back-end forward correction module. The steps are as follows: Step 1: The front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then transforms it into the UTM coordinate system to obtain the positioning result. Step 2: When the set number of frames accumulated or visual image matching and tracking fails, the backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between the UAV-captured images and satellite remote sensing images to obtain the current UAV absolute positioning information and corrected pose. Step 3: The back-end forward correction module performs forward correction based on the absolute positioning information obtained by the back-end satellite image absolute positioning module to eliminate the cumulative error of the front-end visual tracking results. In step 1, the front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then converts it to the UTM coordinate system to obtain the positioning result, as follows: Step 1.1: Use XFeat to match the images of consecutive frames to obtain the matching feature point set of the two images. , ,in , , and These are the matching feature point sets. , The number of matching feature points in the data; Step 1.2: Solve for the displacement matrix using the centroids of the two feature point sets. ,in , In pixel coordinate system Axial displacement and Axial displacement: (1) (2) Decentralize the matching feature points: ； (3) in , For point set , Distribution of point sets in their respective centroid coordinate systems; The covariance matrix of the two matching feature point sets is calculated as follows: (4) Perform singular value decomposition on the covariance matrix: (5) in, , , They are respectively The left singular matrix, singular value matrix, and right singular matrix; Calculate the rotation matrix between two images in pixel coordinates. The relative transformation angle between the two frames : (6) (7) in, , They represent The sum of the items in the second row and first column The item in the first row and first column; Step 1.3: Determine the position of the second image in pixel coordinates: (8) in This represents the position of the first image in pixel coordinates. This represents the position of the second image in pixel coordinates. Transform the pixel coordinate system to the UTM coordinate system: (9) in These are the coordinates of the second image in the world coordinate system. , These are the rotation and translation matrices, respectively, for transforming the pixel coordinate system to the world coordinate system. These are the scaling transformation coefficients; In step 3, the backend forward correction module performs forward correction based on the absolute positioning information obtained by the backend satellite image absolute positioning module, eliminating the cumulative error of the frontend visual tracking results, as follows: The forward propagation positioning correction is performed based on the satellite absolute positioning result, using the following formula: (12) (13) in, This is the absolute positioning result of the back-end satellite image in the k-th frame. This is the front-end visual tracking and localization result of the k-th frame. This represents the current total number of frames. This is the amount of change in pose correction. For the first Frame front-end visual tracking and localization results For the first Positioning results after frame correction.

2. The UAV visual continuous positioning method based on satellite remote sensing imagery as described in claim 1, characterized in that, In step 2, when the accumulated set number of frames or visual image matching tracking fails, the backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between the UAV-captured images and satellite remote sensing images to obtain the current UAV absolute positioning information and corrected pose, as follows: Step 2.1, Preprocessing and Correction: Based on the current UAV heading information, pre-rotate and correct the image; Step 2.2 Feature Extraction and Matching: Use the deep learning SuperGlue model to calculate key points and descriptors of the currently captured image and satellite remote sensing image. Match the two to generate matching feature point pairs, and use the Random Sample Consensus (RANSAC) algorithm to filter out outliers in the matching. Step 2.3: Calculate the geographical longitude of the image. ,latitude The formula is: (10) (11) in , It is the geographic longitude and latitude coordinates of the top left pixel of the satellite image. , It is the geographic longitude and latitude coordinates of the bottom right pixel in the satellite image; , These are the pixel coordinates of feature points in the drone image. The x-axis coordinates of the feature points The y-coordinate of the feature point; and These are the width and height of the satellite image, respectively.

3. A visual continuous positioning system for unmanned aerial vehicles (UAVs) based on satellite remote sensing imagery, characterized in that, This system is used to implement the UAV visual continuous positioning method based on satellite remote sensing imagery as described in any one of claims 1 to 2. The system includes a front-end visual tracking module, a back-end satellite imagery absolute positioning module, and a back-end forward correction module, wherein: The front-end visual tracking module obtains a set of matching feature points based on image matching technology, calculates the relative positioning in the pixel coordinate system, and then transforms it to the UTM coordinate system to obtain the positioning result. The backend satellite image absolute positioning module uses the deep learning method SuperGlue to perform image matching between images captured by the UAV and satellite remote sensing images to obtain the current absolute positioning information and corrected pose of the UAV. The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching back-end satellite images, eliminating the cumulative error of the front-end visual tracking results.

4. A mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the UAV visual continuous positioning method based on satellite remote sensing imagery as described in any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the UAV visual continuous positioning method based on satellite remote sensing imagery as described in any one of claims 1 to 2.

Images