A UAV autonomous positioning method based on monocular vision
Through the image hybrid registration and PnP algorithm of a monocular camera, the signal dependence and error accumulation problems in the drone positioning technology are solved, and high-precision autonomous positioning in the denial environment is achieved.
Patent Information
- Application Number
- CN202111242626.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-10-25
AI Technical Summary
The existing drone positioning technology relies on external signals and is prone to failure under signal occlusion or interference. There are error accumulation problems in visual fusion inertial odometers, which affects accuracy.
Using a monocular camera for image mixing registration and PnP algorithm, the position of the drone in the earth coordinate system is calculated by registering remote sensing images and images collected by the drone, and the SLAM coordinates are corrected using the conversion matrix to achieve high accuracy Autonomous positioning.
In the denial environment, the drone is realized, which overcomes the error accumulation problem and improves positioning accuracy and reliability.
Smart Images

Figure CN114066972B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of autonomous flight of unmanned aerial vehicles, and in particular refers to an autonomous positioning method of unmanned aerial vehicles based on monocular vision. Background Art
[0002] With the rapid development of information science, drones have been widely used in people's lives, and drones have gradually expanded their application areas and research scope, such as post-disaster search and rescue, aerial photography, crop monitoring, etc. Among them, the positioning system is an important guarantee for drone carriers to successfully complete their tasks.
[0003] At present, the main way to implement drone positioning technology is the Global Positioning System (GPS). GPS positioning has many advantages, such as: the positioning method is relatively mature and easy to integrate, and the positioning accuracy is high when the outdoor signal is good. But it has one major disadvantage: it relies on external signals. If the GPS signal is blocked, interfered or missing, the positioning will fail, and the drone will lose control or even fall. It can be seen that realizing autonomous positioning technology is an important task to achieve real-time, accurate and autonomous positioning of drones.
[0004] Vision fusion inertial navigation odometer is a widely used method for autonomous positioning of UAVs. This method can estimate the UAV's posture parameters and obtain sparse 3D reconstruction of the scene. However, since this method calculates the relative position of adjacent moments, there will be cumulative errors in the system during operation, which seriously restricts the accuracy of vision fusion inertial navigation odometer. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a UAV autonomous positioning method based on monocular vision. The method adopts a monocular camera as the only sensor and realizes high-precision autonomous positioning based on image hybrid registration and PnP. It can overcome the error accumulation problem of existing autonomous positioning systems and has good practical value.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is:
[0007] A method for autonomous positioning of a UAV based on monocular vision comprises the following steps:
[0008] (1) Control the drone to climb to multiple locations at different altitudes;
[0009] (2) Image acquisition is performed at each location, and SLAM autonomous positioning is performed based on the image information to obtain the coordinates of the UAV in the SLAM coordinate system;
[0010] (3) Load the remote sensing image of the UAV flight area, perform image hybrid registration on the remote sensing image and the image collected by the UAV, and calculate the feature points with the same name;
[0011] (4) Based on the latitude and longitude coordinates and elevation information of the feature points with the same name in the remote sensing image, as well as the two-dimensional positions of the feature points with the same name in the image collected by the UAV, the coordinates of the UAV in the geodetic coordinate system are solved by the PnP algorithm;
[0012] (5) Based on the coordinates of the UAV in the SLAM coordinate system and the coordinates in the earth coordinate system, calculate and solve the transformation matrix between the two coordinate systems and the SLAM scale information;
[0013] (6) Use the transformation matrix to convert the SLAM coordinates into geodetic coordinates, and use the coordinates of the UAV in the geodetic coordinate system obtained in step (4) to correct the coordinates obtained by SLAM transformation.
[0014] Furthermore, the specific method of step (2) is:
[0015] (201) Start the camera through the image acquisition program, collect image sequences, and publish the images in the form of ROS nodes;
[0016] (202) Subscribe to image information through a positioning program;
[0017] (203) performing feature detection on the image sequence to obtain location information and descriptor information of feature points;
[0018] (204) Tracking feature points in the image using a feature tracking method to obtain coordinates of the same feature point in different images;
[0019] (205) Calculate the pose transformation between different images by multi-view geometry method;
[0020] (206) The bundle adjustment method is used to optimize the UAV posture and obtain the SLAM positioning result, which is then published in the form of a ROS node.
[0021] Furthermore, the specific method of step (3) is:
[0022] (301) Perform multi-feature extraction on remote sensing images and images collected by UAVs, including SIFT feature extraction, SURF feature extraction, ORB feature extraction, edge feature extraction, and descriptor extraction;
[0023] (302) Point feature and edge feature matching is achieved through similarity detection;
[0024] (303) performing a combined analysis of multiple feature points on the feature distribution in the image collected by the drone, and if the feature distribution is uneven, discarding the frame of the image collected by the drone;
[0025] (304) Acquire feature points with the same name from remote sensing images and images collected by UAVs.
[0026] Furthermore, the specific method of step (4) is:
[0027] (401) reading the three-dimensional coordinates of the feature point with the same name according to the pixel position in the remote sensing image, and obtaining the two-dimensional position coordinates of the feature point with the same name in the image collected by the drone;
[0028] (402) The PnP algorithm is used to solve the position of the UAV in the geodetic coordinate system, and the position is published in the form of a ROS node.
[0029] Furthermore, the specific method of step (5) is:
[0030] (501) subscribing to the positioning result published in step (206) and the position published in step (402) through a fusion program, and synchronizing them according to the timestamp information;
[0031] (502) The point cloud alignment algorithm is used to calculate the SLAM scale information and the transformation matrix between the SLAM coordinate system and the earth coordinate system.
[0032] Furthermore, in step (6), the extended Kalman filter method is used to correct the coordinates obtained by SLAM conversion.
[0033] The beneficial effects of the present invention compared to the prior art are:
[0034] 1. The present invention provides an effective method for autonomous positioning of unmanned aerial vehicles, which solves the matching problem caused by the large difference between remote sensing images and images taken by unmanned aerial vehicles through an image hybrid registration algorithm. After obtaining the feature points with the same name, the PnP algorithm is used to calculate the position and posture of the unmanned aerial vehicle in the geodetic coordinate system, so as to correct the drift problem caused by error accumulation in the actual operation of monocular vision SLAM, thereby realizing high-precision autonomous positioning of unmanned aerial vehicles in a denied environment.
[0035] 2. Heterogeneous image registration has always been a research hotspot in this field. The present invention realizes accurate registration of heterogeneous images with large differences through effective fusion of multiple features, which is an important innovation to the existing technology.
[0036] 3. The drift problem caused by the accumulation of SLAM errors has always been a thorny problem that hinders its application in autonomous navigation and positioning of UAVs. The present invention proposes to solve the UAV posture through remote sensing image registration and PnP algorithm, and then fuse and correct the obtained posture with the posture obtained by visual SLAM, so as to achieve autonomous positioning in a completely denied environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 Flow chart of the autonomous positioning method of a UAV in an embodiment of the present invention. Specific implementation methods
[0038] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not intended to limit the scope of protection of the present invention.
[0039] A monocular vision-based autonomous positioning method for a UAV is provided. The method is based on a ROS robot operating system and an Ubuntu operating system and is implemented by running multiple programs in the ROS robot operating system and the Ubuntu operating system. The method includes the following steps:
[0040] (1) Manually operate the drone to climb to different heights and locations.
[0041] (2) Start the image acquisition program and publish the image in the form of a ROS node. The autonomous positioning program subscribes to the image information in real time and performs SLAM autonomous positioning. Finally, the positioning result is published in the form of a ROS node.
[0042] (3) Load the remote sensing image of the drone flight area and subscribe to the image collected in step (2), perform image hybrid registration on the remote sensing image and the image taken by the drone, and calculate the feature points with the same name.
[0043] (4) Based on the latitude and longitude coordinates and elevation information of the feature points with the same name in the remote sensing image, and the two-dimensional position of the feature points with the same name in the image taken by the drone, the position of the drone is solved using the PnP (perspective-n-point) algorithm.
[0044] (5) Based on the coordinates of the UAV in the SLAM coordinate system and the coordinates in the earth coordinate system, the transformation matrix between the two coordinate systems and the SLAM scale information are calculated and solved.
[0045] (6) The SLAM coordinates are converted into geodetic coordinates using the transformation matrix. Since errors accumulate during the operation of SLAM, the coordinates obtained by SLAM conversion are corrected using the UAV geodetic coordinates obtained through image hybrid registration and PnP during the positioning process.
[0046] The specific method of step (2) is as follows:
[0047] (201) Start the camera and publish the image in the form of a ROS node;
[0048] (202) Locating program subscription image node;
[0049] (203) performing feature detection on the onboard camera image sequence to obtain location information and descriptor information of feature points;
[0050] (204) Tracking feature points in the image using a feature tracking method to obtain coordinates of the same feature point in different images;
[0051] (205) Calculate the pose transformation between different camera images by multi-view geometry method;
[0052] (206) The bundle adjustment method is used to optimize the UAV posture and three-dimensional point cloud coordinates to obtain the position in the position SLAM coordinate system, and the positioning result is published in the form of a ROS node.
[0053] The specific method of step (3) is:
[0054] (301) Perform multiple feature extraction on remote sensing images and images taken by drones, including SIFT feature extraction, SURF feature extraction, ORB feature extraction, edge feature extraction, and descriptor extraction;
[0055] (302) performing similarity detection to achieve point feature and edge feature matching;
[0056] (303) Through the combined analysis of multiple feature points, a certain number of matching features are achieved within each image range;
[0057] (304) Obtaining feature points with the same name.
[0058] The specific method of step (4) is:
[0059] (401) According to the pixel position of the feature point with the same name in the remote sensing image, read its three-dimensional coordinates, recorded as (X i , Y i , Z i ), where i is the number of the feature point in the same image; the image coordinates of the feature point with the same name in the image taken by the drone are (u i , v i );
[0060] (402) The PnP algorithm is used to solve the position of the UAV in the geodetic coordinate system, and the position is published in the form of a ROS node.
[0061] The specific method of step (5) is:
[0062] (501) The fusion algorithm subscribes to the position node published by the SLAM program and the position node published in step (402), and synchronizes them according to the timestamp information;
[0063] (502) The point cloud alignment algorithm is used to calculate the SLAM scale information and the transformation matrix between the two coordinate systems.
[0064] Here is a more specific example:
[0065] like Figure 1 As shown, a UAV autonomous positioning method based on monocular vision includes the following steps:
[0066] Step 1: Manually operate the drone to climb to different heights and positions;
[0067] Step 2: The onboard computing unit (NVIDIANX) starts the image acquisition program and publishes the image in the form of a ROS node. The autonomous positioning program subscribes to the image information in real time and performs SLAM autonomous positioning. Finally, the positioning result is published in the form of a ROS node.
[0068] Step 2.1, start the camera acquisition program and publish the image in the form of ROS node;
[0069] Step 2.2, start the positioning program and subscribe to the image node;
[0070] Step 2.3, perform feature detection on the camera image sequence to obtain the location information and descriptor information of the feature points;
[0071] Step 2.4, using feature tracking method to track feature points in the image, and obtain the coordinates of the same feature point in different images;
[0072] Step 2.5, calculate the pose transformation between different camera images by multi-view geometry method;
[0073] Step 2.6: Use the BundleAdjustment bundle adjustment method to optimize the UAV posture and 3D point cloud coordinates to obtain the position Publish the positioning results in the form of ROS nodes.
[0074] Step 3: load the remote sensing image of the drone flight area, subscribe to the image collected in step (2), perform image hybrid registration on the remote sensing image and the image taken by the drone, and calculate the feature points with the same name:
[0075] Step 3.1: Perform multiple feature extraction on remote sensing images and images taken by drones, including SIFT feature extraction, SURF feature extraction, ORB feature extraction, edge feature extraction, and descriptor extraction. For specific algorithms, see references [1]-[4].
[0076] [1]David G.Lowe.Distinctive Image Features from Scale-InvariantKeypoints[J].International Journal of ComputerVision, 2004, 60(2):91-110.
[0077] [2]BAY H, ESS A, TUYTELAARS T, et al.Speeded-Up Robust Features(SURF)[J].Computer vision and image understanding:CVIU, 2008, 110(3):346-359.
[0078] [3]ETHAN RUBLEE, VINCENT RABAUD, KURT KONOLIGE, et al.ORB: an efficient alternative to SIFT or SURF[C]. / / 2011 International Conference on ComputerVision.[v.3].:IEEE, 2011:2564-2571.
[0079] [4]Canny J FA computational Approach to Edge Detection[J].IEEETransactions on Pattern Analysis and Machine Intelligence, 1986, 8(6):679-698.
[0080] Step 3.2, perform similarity detection to achieve point feature and edge feature matching;
[0081] Step 3.3, through the combined analysis of multiple feature points, a certain number of matching features are achieved within each image range;
[0082] Step 3.4, obtain feature points with the same name.
[0083] Step 4: Based on the latitude and longitude coordinates and elevation information of the feature points with the same name in the remote sensing image, and the two-dimensional position of the feature points with the same name in the image taken by the drone, the position of the drone is solved by the PnP algorithm. The specific algorithm is shown in reference [5]:
[0084] [5] Vincent Lepetit, Francesc Moreno-Nuguer, and Pascal Fua. Epnp: Anaccurate o(n)solution to the pnp problem. International journal of computervision, 81(2):155–166, 2009.
[0085] Step 4.1: Read the three-dimensional coordinates of the feature points with the same name according to their pixel positions in the remote sensing image, recorded as (X i , Y i , Z i ), where i is the number of the feature point in the same image; the image coordinates of the same feature point in the image taken by the drone are (u i , v i );
[0086] In step 4.2, the PnP algorithm is used to solve the position of the UAV in the geodetic coordinate system and publish the position in the form of a ROS node.
[0087] Step 5: Calculate the transformation matrix between the two coordinate systems and the SLAM scale information based on the coordinates of the drone in the SLAM coordinate system and the coordinates in the earth coordinate system:
[0088] Step 5.1, the fusion algorithm subscribes to the location nodes published by the SLAM program and the location nodes published in step (4.2), and synchronizes them according to the timestamp information;
[0089] Step 5.2: Use the point cloud alignment algorithm to calculate the SLAM scale information and the transformation matrix between the two coordinate systems. For the specific algorithm, see reference [6]:
[0090] [6]KSArun,TSHuang&S.D.Blostein.Least-Squares Fitting of Two 3-DPoint Sets.IEEE Transactions on PatternAnalysis and Machine Intelligence,vol.9,no.5,pages 698–700,Sept 1987
[0091] Step 6: Use the transformation matrix to convert the SLAM coordinates into geodetic coordinates. Since there is error accumulation during the operation of SLAM, during the positioning process, the coordinates obtained by SLAM conversion are corrected using the UAV geodetic coordinates obtained through image hybrid registration and PnP.
[0092] In summary, the present invention provides an effective autonomous positioning method for unmanned aerial vehicles, which uses a monocular camera as the only sensor to achieve autonomous positioning of unmanned aerial vehicles. The present invention uses an image hybrid registration algorithm to solve the matching problem caused by the large difference between remote sensing images and images taken by unmanned aerial vehicles. After obtaining feature points with the same name, the PnP algorithm is used to calculate the position and posture of the unmanned aerial vehicle in the geodetic coordinate system, so as to correct the drift problem caused by error accumulation in the actual operation of monocular vision SLAM, thereby achieving high-precision autonomous positioning of unmanned aerial vehicles in a denied environment.
[0093] The above description is only an individual embodiment of the present invention and is not intended to limit the present invention. Any modification, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A UAV autonomous positioning method based on monocular vision, characterized in that: The following steps are involved: (1) Control the drone to climb to multiple locations at different altitudes; (2) Image acquisition is performed at each position, and SLAM autonomous positioning is performed based on the image information to obtain the coordinates of the UAV in the SLAM coordinate system; the specific method is as follows: (201) Start the camera through the image acquisition program, collect image sequences, and publish the images in the form of ROS nodes; (202) Subscribe to image information through a positioning program; (203) performing feature detection on the image sequence to obtain location information and descriptor information of feature points; (204) Tracking feature points in the image using a feature tracking method to obtain coordinates of the same feature point in different images; (205) Calculate the pose transformation between different images by multi-view geometry method; (206) Use the bundle adjustment method to optimize the UAV posture, obtain the SLAM positioning result, and publish the SLAM positioning result in the form of ROS node; (3) Load the remote sensing image of the UAV flight area, perform image hybrid registration on the remote sensing image and the image collected by the UAV, and calculate the feature points with the same name; the specific method is as follows: (301) Perform multi-feature extraction on remote sensing images and images collected by UAVs, including SIFT feature extraction, SURF feature extraction, ORB feature extraction, edge feature extraction, and descriptor extraction; (302) Point feature and edge feature matching is achieved through similarity detection; (303) performing a combination analysis of multiple feature points on the feature distribution in the image collected by the drone, and if the feature distribution is uneven, discarding the image collected by the drone; (304) Acquire feature points with the same name from remote sensing images and images collected by drones; (4) Based on the latitude and longitude coordinates and elevation information of the feature points with the same name in the remote sensing image, as well as the two-dimensional position of the feature points with the same name in the image collected by the drone, the coordinates of the drone in the geodetic coordinate system are solved by the PnP algorithm; the specific method is as follows: (401) reading the three-dimensional coordinates of the feature point with the same name according to the pixel position in the remote sensing image, and obtaining the two-dimensional position coordinates of the feature point with the same name in the image collected by the drone; (402) using the PnP algorithm to solve the position of the UAV in the geodetic coordinate system, and publishing the position in the form of a ROS node; (5) Based on the coordinates of the drone in the SLAM coordinate system and the coordinates in the earth coordinate system, the transformation matrix between the two coordinate systems and the SLAM scale information are calculated and solved; the specific method is: (501) subscribing to the positioning result published in step (206) and the position published in step (402) through a fusion program, and synchronizing them according to the timestamp information; (502) Calculating SLAM scale information and a transformation matrix between the SLAM coordinate system and the earth coordinate system using a point cloud alignment algorithm; (6) Use the transformation matrix to convert the SLAM coordinates into geodetic coordinates, and use the coordinates of the UAV in the geodetic coordinate system obtained in step (4) to correct the coordinates obtained by SLAM transformation.
2. The method for autonomous positioning of unmanned aerial vehicles based on monocular vision according to claim 1, characterized in that: In step (6), the extended Kalman filter method is used to correct the coordinates obtained by SLAM conversion.
