Unmanned aerial vehicle visual continuous positioning method and system based on satellite remote sensing image assistance

Abstract

The invention discloses an unmanned aerial vehicle visual continuous positioning method and system based on satellite remote sensing image assistance. The method comprises the following steps: a front-end visual tracking module calculates relative positioning under a pixel coordinate system according to an image matching feature point set, and turns to a UTM coordinate system to obtain a positioning result; when the accumulated set frame number or visual picture matching tracking fails, a rear-end satellite image absolute positioning module performs unmanned aerial vehicle absolute positioning and posture correction by using an unmanned aerial vehicle shooting image and a satellite remote sensing image; and the rear-end forward correction module performs forward correction based on absolute positioning information obtained by matching of the rear-end satellite image absolute positioning module, and eliminates an accumulated error of a front-end visual tracking result. The system comprises a front-end visual tracking module, a rear-end satellite image absolute positioning module and a rear-end forward correction module. The satellite remote sensing image positioning precision of the unmanned aerial vehicle is improved, the algorithm is simple, the real-time performance is high, and unmanned aerial vehicle visual continuous positioning can be carried out.

Description

Technical Field

[0001] The present invention relates to the field of satellite navigation and unmanned aerial vehicle positioning technology, and in particular to a method and system for unmanned aerial vehicle visual continuous positioning based on satellite remote sensing image assistance. Background Art

[0002] Unmanned aircraft, also known as drones, are unmanned aircraft that are controlled by radio remote control equipment and self-contained program control devices. UAV intelligence is one of the core technologies of UAV development, and its core technologies include intelligent perception, positioning and control. At present, UAV positioning mainly relies on satellite navigation systems to provide high-precision positioning information, but satellite navigation information is easily interfered with or blocked, which seriously affects the execution of UAV missions. With the development of satellite remote sensing technology, imaging resolution has entered the sub-meter level, which provides favorable conditions for satellite image-assisted positioning.

[0003] Traditional drone positioning in GNSS-denied environments often uses visual odometers without combining satellite historical remote sensing images. In pure visual positioning, due to the lack of absolute correction, the error tends to accumulate to a large extent. However, absolute positioning information can be obtained through satellite historical images to correct the accumulated error of visual tracking positioning. Therefore, it is of great significance to explore satellite remote sensing image-assisted positioning.

[0004] Invention CN 111288989 A discloses a visual positioning method for a small UAV. The method realizes visual positioning of a small UAV in a complex environment by extracting visual features of the UAV's surrounding environment from multiple cameras. However, it does not combine historical satellite images and only relies on existing environmental images for visual positioning, so its application scope is greatly limited. Invention CN118506197A discloses a visual positioning method for a UAV suitable for multi-season scenes. A season-invariant feature extraction network for UAV aerial images and satellite images is trained using a UAV aerial image dataset and a satellite image dataset. A lightweight season-invariant feature extraction network is used to perform feature matching on the pre-processed UAV aerial images and satellite images, thereby realizing positioning. The parallax problem caused by scale changes and rotation changes between aerial images and satellite images during UAV visual positioning and the appearance inconsistency problem caused by seasonal changes between aerial images and satellite images during UAV visual positioning are solved, and high-precision visual positioning of the UAV is realized. However, the algorithm is relatively complex and it is difficult to perform dynamic continuous positioning. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for unmanned aerial vehicle visual continuous positioning based on satellite remote sensing images, which has a simple algorithm, strong real-time performance and high positioning accuracy.

[0006] The technical solution to achieve the purpose of the present invention is: a method for continuous visual positioning of unmanned aerial vehicles based on satellite remote sensing images, which is equipped with a front-end visual tracking module, a back-end satellite image absolute positioning module and a back-end forward correction module. The method steps are as follows:

[0007] Step 1: The front-end visual tracking module obtains the matching feature point set based on the image matching technology, calculates the relative positioning in the pixel coordinate system, and converts it to the UTM coordinate system to obtain the positioning result;

[0008] Step 2: When the set number of frames is accumulated or the visual image matching and tracking fails, the back-end satellite image absolute positioning module uses the deep learning method SuperGlue to match the drone image with the satellite remote sensing image to obtain the current drone absolute positioning information and corrected posture;

[0009] Step 3: The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite image absolute positioning module to eliminate the accumulated error of the front-end visual tracking result.

[0010] Furthermore, in step 1, the front-end visual tracking module obtains a set of matching feature points based on the image matching technology, calculates the relative positioning in the pixel coordinate system, and converts it to the UTM coordinate system to obtain the positioning result, as follows:

[0011] Step 1.1, use XFeat to match the previous and next frame images, and obtain the matching feature point sets P1 and P2 of the two images, where P1 = {p 1_i ,i=1,2,...,N1},P2={p 2_i ,i=1,2,...,N2}, N1 and N2 are the number of matching feature points in the matching feature point sets P1 and P2 respectively;

[0012] Step 1.2, solve the displacement matrix dt through the centroids of the two feature point sets, where dx and dy are the x-axis displacement and y-axis displacement in the pixel coordinate system respectively:

[0013]

[0014] dt=[dx dy] (2)

[0015] Decentralize the matching feature points:

[0016]

[0017] Where P 1_center , P 2_center is the point set distribution of point sets P1 and P2 in their respective centroid coordinate systems;

[0018] The covariance matrix of two matching feature point sets is calculated as:

[0019]

[0020] Perform singular value decomposition on the covariance matrix:

[0021] H=U·S·V T (5)

[0022] Among them, U, S, and V are the left singular matrix, singular value matrix, and right singular matrix of H respectively;

[0023] Calculate the rotation matrix R between the two images in the pixel coordinate system u And the relative transformation angle dθ of the two frames:

[0024] R u =V T ·U T (6)

[0025] dθ=arctan2(R u [1,0],R u [0,0]) (7)

[0026] Among them, R u [1,0], R u [0,0] represents R u The second row and first column of the item and R u The item in the first row and first column of ;

[0027] Step 1.3, solve the position of the second image in the pixel coordinate system:

[0028] P pic_2 =P pic_1 +(dx dy dθ) T (8)

[0029] Where P pic_1 is the position of the first image in the pixel coordinate system, P pic_2 is the position of the second image in the pixel coordinate system;

[0030] Convert pixel coordinates to UTM coordinates:

[0031] P W2 = k(R Wu P pic_2 +t Wu ) (9)

[0032] Where P W2 is the coordinate of the second image in the world coordinate system, R Wu ,t WuThey are the rotation matrix and displacement matrix from the pixel coordinate system to the world coordinate system, and k is the scale transformation coefficient.

[0033] Furthermore, in step 2, when the set number of frames is accumulated or the visual image matching tracking fails, the back-end satellite image absolute positioning module uses the deep learning method SuperGlue to match the drone image with the satellite remote sensing image to obtain the current drone absolute positioning information and corrected posture, as follows:

[0034] Step 2.1, pre-processing correction: pre-rotate and correct the image according to the current UAV heading information;

[0035] Step 2.2, feature extraction and matching: Use the deep learning SuperGlue model to calculate the key points and descriptors of the current captured image and satellite remote sensing image, match the two to generate matching feature point pairs, and use the random sampling consistency RANSAC algorithm to filter out abnormal points in the match;

[0036] Step 2.3, calculate the geographical longitude Lon and latitude Lat of the image, the formula is:

[0037]

[0038] Among them, Lat t , Lon t is the geographic longitude and latitude coordinates of the upper left pixel of the satellite image. b , Lon b is the geographic longitude and latitude coordinates of the lower right corner pixel of the satellite image; C x , C y is the pixel coordinate of the feature point in the drone image, C x is the x-axis coordinate of the feature point, C y is the y-axis coordinate of the feature point; w and h are the width and height of the satellite image respectively.

[0039] Furthermore, in step 3, the back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite image absolute positioning module to eliminate the cumulative error of the front-end visual tracking result, as follows:

[0040] Forward propagation positioning correction is performed based on the satellite absolute positioning result. The formula is:

[0041] dP W =P' Wk -P Wk (12)

[0042] P' Wi =P Wi +dP W,i∈(k,k+1....n) (13)

[0043] Among them, P' Wk is the absolute positioning result of the backend satellite image of the kth frame, P Wk is the front-end visual tracking and positioning result of the kth frame, n is the current total number of frames, dP W is the change in posture correction, P Wi is the front-end visual tracking and positioning result of the i-th frame, P' Wi is the corrected positioning result of the i-th frame.

[0044] A satellite remote sensing image-assisted unmanned aerial vehicle visual continuous positioning system, the system is used to implement the satellite remote sensing image-assisted unmanned aerial vehicle visual continuous positioning method, the system includes a front-end visual tracking module, a back-end satellite image absolute positioning module and a back-end forward correction module, wherein:

[0045] The front-end visual tracking module obtains a matching feature point set based on image matching technology, calculates relative positioning in a pixel coordinate system, and converts it to a UTM coordinate system to obtain a positioning result;

[0046] The back-end satellite image absolute positioning module matches the drone image with the satellite remote sensing image based on the deep learning method SuperGlue to obtain the current drone absolute positioning information and corrected posture;

[0047] The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite images to eliminate the accumulated errors of the front-end visual tracking results.

[0048] A mobile terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for continuous visual positioning of an unmanned aerial vehicle based on the assistance of satellite remote sensing images is implemented.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for continuous visual positioning of unmanned aerial vehicles based on the assistance of satellite remote sensing images.

[0050] Compared with the prior art, the present invention has the following significant effects: (1) the back-end satellite remote sensing image positioning algorithm combined with the SuperGlue algorithm is used to improve the accuracy of satellite remote sensing image positioning; (2) a solution based on front-end visual tracking positioning and back-end satellite remote sensing image absolute positioning is adopted, which has better navigation accuracy than pure visual odometer positioning; (3) the algorithm is simple, has strong real-time performance, and can perform continuous visual positioning of unmanned aerial vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The present invention is a structural block diagram of a satellite remote sensing image-assisted unmanned aerial vehicle visual continuous positioning system.

[0052] Figure 2 It is a structural block diagram of the back-end satellite image absolute positioning module in the present invention.

[0053] Figure 3 It is a structural block diagram of the back-end forward correction module in the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0055] Combination Figure 1 The present invention discloses a satellite remote sensing image-assisted unmanned aerial vehicle visual continuous positioning system, comprising a front-end visual tracking module, a rear-end satellite image absolute positioning module and a rear-end forward correction module;

[0056] The front-end visual tracking module obtains a matching feature point set based on image matching technology, calculates relative positioning in a pixel coordinate system, and converts it to a UTM coordinate system to obtain a positioning result;

[0057] The back-end satellite image absolute positioning module matches the drone image with the satellite remote sensing image based on the deep learning method SuperGlue to obtain the current drone absolute positioning information and corrected posture;

[0058] The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite images to eliminate the accumulated errors of the front-end visual tracking results.

[0059] The present invention provides a continuous method for visual absolute positioning of an unmanned aerial vehicle based on satellite remote sensing images, comprising the following steps:

[0060] Step 1: The front-end visual tracking module uses image matching technology to obtain a set of matching feature points by matching the front and back frames taken during the flight of the drone. The pose change of the drone in two adjacent frames is calculated to obtain relative positioning in the pixel coordinate system, and then transferred to the UTM coordinate system to obtain the positioning result, as follows:

[0061] Step 1.1, use XFeat to match the previous and next frame images, and obtain the matching feature point sets of the two images. The matching feature point sets P1 and P2 of the two images are obtained, where P1 = {p 1_i ,i=1,2,...,N1},P2={p 2_i ,i=1,2,...,N2}, N1 and N2 are the number of matching feature points in the matching feature point sets P1 and P2 respectively;

[0062] Step 1.2, solve the displacement matrix dt through the centroids of the two feature point sets, where dx and dy are the x-axis displacement and y-axis displacement in the pixel coordinate system respectively:

[0063]

[0064] dt=[dx dy] (2)

[0065] Where N1 and N2 are the number of two feature point sets;

[0066] Decentralize the matching feature points:

[0067]

[0068] Where P 1_center , P 2_center is the point set distribution of point sets P1 and P2 in their respective centroid coordinate systems.

[0069] The covariance matrix of two feature point sets is calculated as:

[0070]

[0071] Perform singular value decomposition on the covariance matrix:

[0072] H=U·S·V T (5)

[0073] U, S, and V are the left singular matrix, singular value matrix, and right singular matrix of H, respectively.

[0074] Calculate the rotation matrix R between the two images in the pixel coordinate system u And the relative transformation angle dθ of the two frames:

[0075] R u =V T ·U T (6)

[0076] dθ=arctan2(R u [1,0],R u [0,0]) (7)

[0077] Step 1.3, solve the position of the second image in the pixel coordinate system:

[0078] P pic_2 =P pic_1 +(dx dy dθ) T (8)

[0079] Where P pic_1 is the position of the first image in the pixel coordinate system, P pic_2is the position of the second image in the pixel coordinate system.

[0080] Step 1.4: Convert pixel coordinate system to UTM coordinate system:

[0081] P W2 = k(R Wu P pic_2 +t Wu ) (9)

[0082] Where R Wu ,t Wu They are the rotation matrix and displacement matrix from the pixel coordinate system to the world coordinate system, and k is the scale transformation coefficient.

[0083] Where P W2 is the coordinate of the second image in the world coordinate system.

[0084] Step 2: When the set number of frames is accumulated or the visual image matching tracking fails, the back-end satellite image absolute positioning module uses the deep learning method SuperGlue to match the drone image with the satellite remote sensing image to obtain the current drone absolute positioning information and corrected posture, such as Figure 2 As shown, the details are as follows:

[0085] Step 2.1, pre-processing correction: pre-rotate and correct the image according to the current UAV heading information;

[0086] Step 2.2, feature extraction and matching: Use the deep learning SuperGlue model to calculate the key points and descriptors of the current captured image and satellite remote sensing image, match the two to generate matching feature point pairs, and use the random sampling consistency RANSAC algorithm to filter out abnormal points in the match;

[0087] Step 2.3, calculate the geographical longitude Lon and latitude Lat of the image, the formula is:

[0088]

[0089] Among them, Lat t , Lon t is the geographic longitude and latitude coordinates of the upper left pixel of the satellite image. b , Lon b is the geographic longitude and latitude coordinates of the lower right pixel of the satellite image, C x , C y is the pixel coordinate of the feature point in the drone image, C x is the x-axis coordinate of the feature point, C y is the y-axis coordinate of the feature point, w and h are the width and height of the satellite image respectively.

[0090] Step 3: The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite image absolute positioning module to eliminate the cumulative error of the front-end visual tracking result, such as Figure 3 As shown, the details are as follows:

[0091] After obtaining the satellite absolute positioning result, forward propagation positioning correction is performed to correct all frame positioning results from the query frame to the current frame. The formula is:

[0092] dP W =P' Wk -P Wk (12)

[0093] P' Wi =P Wi +dP W ,i∈(k,k+1....n) (13)

[0094] Where P' Wk is the absolute positioning result of the backend satellite image of the kth frame, P Wk is the front-end visual tracking and positioning result of the kth frame, n is the current total number of frames, dP W is the change in posture correction, P Wi is the front-end visual tracking and positioning result of the i-th frame, P' Wi is the corrected positioning result of the i-th frame.

[0095] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for visual absolute positioning of an unmanned aerial vehicle assisted by satellite remote sensing images is implemented.

[0096] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for visual absolute positioning of an unmanned aerial vehicle assisted by satellite remote sensing images are implemented.

[0097] The above specific implementation methods are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, a person of ordinary skill in the art should understand that the technical method of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for continuous visual positioning of unmanned aerial vehicles based on satellite remote sensing images, characterized in that: A front-end visual tracking module, a back-end satellite image absolute positioning module and a back-end forward correction module are provided, and the method steps are as follows: Step 1: The front-end visual tracking module obtains the matching feature point set based on the image matching technology, calculates the relative positioning in the pixel coordinate system, and converts it to the UTM coordinate system to obtain the positioning result; Step 2: When the set number of frames is accumulated or the visual image matching and tracking fails, the back-end satellite image absolute positioning module uses the deep learning method SuperGlue to match the drone image with the satellite remote sensing image to obtain the current drone absolute positioning information and corrected posture; Step 3: The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite image absolute positioning module to eliminate the accumulated error of the front-end visual tracking result.

2. The method for continuous unmanned aerial vehicle visual positioning based on satellite remote sensing image assistance according to claim 1 is characterized in that: In step 1, the front-end visual tracking module obtains the matching feature point set based on the image matching technology, calculates the relative positioning in the pixel coordinate system, and converts it to the UTM coordinate system to obtain the positioning result, as follows: Step 1.1, use XFeat to match the previous and next frame images to obtain the matching feature point sets P1 and P2 of the two images, where P1 = {p1_i, i = 1, 2, ..., N1}, P2 = {p2_i, i = 1, 2, ..., N2}, N1 and N2 are the number of matching feature points in the matching feature point sets P1 and P2 respectively; Step 1.2, solve the displacement matrix dt through the centroids of the two feature point sets, where dx and dy are the x-axis displacement and y-axis displacement in the pixel coordinate system respectively: dt=[dx dy] (2) Decentralize the matching feature points: Where P 1_center , P 2_center is the point set distribution of point sets P1 and P2 in their respective centroid coordinate systems; The covariance matrix of two matching feature point sets is calculated as: Perform singular value decomposition on the covariance matrix: H=U·S·V T (5) Among them, U, S, and V are the left singular matrix, singular value matrix, and right singular matrix of H respectively; Calculate the rotation matrix R between the two images in the pixel coordinate system u And the relative transformation angle dθ of the two frames: R u =V T ·U T (6) dθ=arctan2(R u [1,0],R u [0,0]) (7) Among them, R u [1,0], R u [0,0] represents R u The second row and first column of the item and R u The item in the first row and first column of ; Step 1.3, solve the position of the second image in the pixel coordinate system: P pic_2 =P pic_1 +(dx to dθ) T (8) Where P pic_1 is the position of the first image in the pixel coordinate system, P pic_2 is the position of the second image in the pixel coordinate system; Convert pixel coordinates to UTM coordinates: P W2 =k(R Wu P pic_2 +t Wu ) (9) Where P W2 is the coordinate of the second image in the world coordinate system, R Wu ,t Wu They are the rotation matrix and displacement matrix from the pixel coordinate system to the world coordinate system, and k is the scale transformation coefficient.

3. The method for continuous unmanned aerial vehicle visual positioning based on satellite remote sensing image assistance according to claim 2 is characterized in that: In step 2, when the set number of frames is accumulated or the visual image matching tracking fails, the back-end satellite image absolute positioning module uses the deep learning method SuperGlue to match the drone image with the satellite remote sensing image to obtain the current drone absolute positioning information and corrected posture, as follows: Step 2.1, pre-processing correction: pre-rotate and correct the image according to the current UAV heading information; Step 2.2, feature extraction and matching: Use the deep learning SuperGlue model to calculate the key points and descriptors of the current captured image and satellite remote sensing image, match the two to generate matching feature point pairs, and use the random sampling consistency RANSAC algorithm to filter out abnormal points in the match; Step 2.3, calculate the geographical longitude Lon and latitude Lat of the image, the formula is: Among them, Lat t , Lon t is the geographic longitude and latitude coordinates of the upper left pixel of the satellite image. b , Lon b is the geographic longitude and latitude coordinates of the lower right corner pixel of the satellite image; C x , C y is the pixel coordinate of the feature point in the drone image, C x is the x-axis coordinate of the feature point, C y is the y-axis coordinate of the feature point; w and h are the width and height of the satellite image respectively.

4. The method for continuous unmanned aerial vehicle visual positioning based on satellite remote sensing image assistance according to claim 3 is characterized in that: In step 3, the back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite image absolute positioning module to eliminate the cumulative error of the front-end visual tracking result, as follows: Forward propagation positioning correction is performed based on the satellite absolute positioning result. The formula is: dP W =P' Wk -P Wk (12) P' Wi =P Wi +dP W ,i∈(k,k+1....n) (13) Among them, P' Wk is the absolute positioning result of the backend satellite image of the kth frame, P Wk is the front-end visual tracking and positioning result of the kth frame, n is the current total number of frames, dP W is the change in posture correction, P Wi is the front-end visual tracking and positioning result of the i-th frame, P' Wi is the corrected positioning result of the i-th frame.

5. A satellite remote sensing image-assisted unmanned aerial vehicle visual continuous positioning system, characterized in that: The system is used to implement the satellite remote sensing image-assisted unmanned aerial vehicle visual continuous positioning method according to any one of claims 1 to 4, and the system includes a front-end visual tracking module, a back-end satellite image absolute positioning module and a back-end forward correction module, wherein: The front-end visual tracking module obtains a matching feature point set based on image matching technology, calculates relative positioning in a pixel coordinate system, and converts it to a UTM coordinate system to obtain a positioning result; The back-end satellite image absolute positioning module matches the drone image with the satellite remote sensing image based on the deep learning method SuperGlue to obtain the current drone absolute positioning information and corrected posture; The back-end forward correction module performs forward correction based on the absolute positioning information obtained by matching the back-end satellite images to eliminate the accumulated errors of the front-end visual tracking results.

6. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for continuous visual positioning of unmanned aerial vehicles based on the assistance of satellite remote sensing images as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the steps of the method for continuous visual positioning of unmanned aerial vehicles based on the assistance of satellite remote sensing images as described in any one of claims 1 to 4 are implemented.

Images