A method for autonomous guided landing of UAV based on monocular vision navigation

Abstract

The present invention discloses a method for autonomous guided landing of a UAV based on monocular vision navigation, which belongs to the technical field of UAV landing control systems. The method uses a camera onboard a UAV to obtain a runway image; the acquired image is used to identify the airport through a YOLOv3 model, and the area containing the airport runway is cropped according to the detection frame; the airport runway is then segmented through a segnet model and the runway edge is linearly fitted to the segmented airport runway mask to obtain the pixel coordinates of the runway vertex; the longitude and latitude information of the runway vertex and the attitude angle of the UAV are then combined to calculate the positioning coordinates of the UAV. Finally, the UAV is guided to correct its course and complete the landing. The present invention solves the problems of low UAV autonomous landing accuracy and poor universality when relying solely on visual navigation technology, meets the actual needs of autonomous and precise UAV landing, and has a positioning accuracy that is not lower than or even better than satellite positioning accuracy, and has high practical value.

Description

Technical Field

[0001] The present invention relates to an autonomous guided landing method for an unmanned aerial vehicle (UAV) based on monocular vision navigation, and belongs to the technical field of UAV landing control. Background Art

[0002] For unmanned aircraft, flight control relies crucially on precise navigation systems and robust flight control systems. During takeoff and landing, a precise and reliable navigation system is particularly crucial, replacing the pilot's control to prevent accidents. Currently, common methods for guiding drone landings include GPS, GPS / INS (inertial navigation system), and computer vision. GPS-based methods rely heavily on satellites, which can lead to navigation failure when satellites are interfered with. INS-based methods suffer from positioning errors that increase over time. While visual navigation methods address these shortcomings, they currently still suffer from insufficient aircraft positioning accuracy and significant system errors. In actual landing guidance, inertial navigation systems or GPS positioning systems are often required to correct for these errors. Algorithms for assisting landings solely based on visual navigation are currently immature. Furthermore, current visual navigation methods are mostly based on a cooperative target model, requiring a series of feature modifications to the runway, which are then used to calculate the drone's position and posture for landing assistance. However, these modifications are time-consuming and labor-intensive, and to a certain extent, reduce the algorithm's general applicability. Summary of the Invention

[0003] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a method for autonomous guided landing of a UAV based on monocular vision navigation. The purpose is to rely only on the basic characteristics of the runway, namely the runway length information and the longitude and latitude coordinates of the four vertices, to obtain high-precision UAV positioning coordinates through solution, and guide the UAV to land.

[0004] The technical solution of the present invention is:

[0005] A method for autonomously guiding the landing of an unmanned aerial vehicle (UAV) based on monocular vision navigation, the method comprising the following steps:

[0006] Step 1: The drone collects runway image data through its onboard camera, and trains a YOLOv3 model and a SegNet model based on the collected runway image data. The YOLOv3 model is used to detect the runway, and the region S containing the runway is cropped based on the detected runway detection frame. The SegNet model is then used to segment the cropped region S, and the runway segmentation result mask is output.

[0007] Step 2: Perform a morphological closing operation on the runway segmentation result mask output in step 1 to smooth the irregular edges in the runway segmentation result mask. Then, use the least squares method to fit the left and right edge lines of the runway segmentation result mask, calculate the pixel coordinates of the four vertices of the airport runway, and then map the pixel coordinates of the four vertices of the airport runway back to the airport runway image collected in step 1 to obtain the pixel position coordinates of the airport runway on the collected airport runway image.

[0008] Step 3: Calculate the azimuth angle α and pitch angle β of the camera relative to the geographic coordinate system using the attitude angle information provided by the onboard gyroscope and the attitude angle of the camera relative to the drone.

[0009] Step 4: Calculate the coordinates of the four vertices of the runway relative to the camera coordinate system by solving a set of equations based on the length and width information of the airport runway and the pixel coordinates of the four vertices of the runway mapped in step 2.

[0010] Step 5: Based on the altitude of the UAV and the pitch angle of the camera in the geographic coordinate system obtained in step 3, the coordinates of the projection point of the extended field of view center on the plane of the airport runway in the camera coordinate system are obtained;

[0011] Step 6: Based on the coordinates of the four vertices of the runway obtained in step 4 relative to the camera coordinates and the coordinates of the projection point of the center of the field of view obtained in step 5 on the plane of the airport runway in the camera coordinate system, obtain the Euclidean distances [L1, L2, L3, L4] from the projection point to the four vertices on the airport runway;

[0012] Step 7: Based on the longitude and latitude of the four vertices of the airport runway and the Euclidean distance [L1, L2, L3, L4] obtained in step 6, the longitude and latitude of the projection point are solved by the longitude and latitude distance calculation formula. At this point, the drone positioning is converted into the longitude and latitude of the known projection point and the azimuth and pitch angle of the camera relative to the geographic coordinate system in step 3 to calculate the longitude and latitude of the drone;

[0013] Step 8: Based on the altitude of the drone and the azimuth and pitch angle of the camera in the geographic coordinate system obtained in step 3, calculate the distance from the drone to the projection point in the east direction and the distance in the north direction. Further calculate the latitude and longitude differences between the drone and the projection point, and then calculate the drone positioning coordinates based on the longitude and latitude of the projection point obtained in step 7.

[0014] Step 9: Feedback the drone's positioning coordinate information obtained in step 8 to the drone's flight control system to guide the drone to correct its course and complete landing.

[0015] The geographic coordinate system is a right-handed system defined with the drone as the origin, the north direction as the positive direction of the X-axis, the east direction as the positive direction of the Y-axis, and the Z-axis perpendicular to the ground and pointing to the sky.

[0016] The aircraft coordinate system is defined as follows: the aircraft is the coordinate origin, the nose of the aircraft is forward as the positive direction of the X coordinate axis, the Z coordinate axis is perpendicular to the ground and pointing to the sky as the positive direction, and the Y coordinate axis is rightward in the right-handed system.

[0017] The camera coordinate system is a right-handed system with the camera optical center as the coordinate origin, the two sides parallel to the imaging plane as the positive directions of the X and Y coordinate axes, and the positive direction of the Z axis perpendicular to the imaging plane.

[0018] The field of view center refers to the projection point of the camera's optical center on the imaging plane.

[0019] In step 3, the method for solving the positioning coordinates of the drone is the azimuth angle α and the pitch angle β. The specific calculation method is as follows:

[0020] (1) First, according to the current attitude angle of the UAV, the transformation matrix S from the UAV coordinate system to the geographic coordinate system is obtained. The calculation formula of S is shown in formula (1):

[0021]

[0022] In formula (1), ψ is the UAV heading angle, γ is the UAV pitch angle, and θ is the UAV roll angle;

[0023] (2) Without considering the roll of the drone, the pitch angle and azimuth angle of the drone's camera relative to the drone are known. The direction vector D of the camera in the geographic coordinate system is Dir, so Dir = D*S. The azimuth angle α and pitch angle β of the drone's camera in the geographic coordinate system are further solved.

[0024] In step 4, the coordinate calculation method of the four vertices of the runway in the camera coordinate system is as follows:

[0025] (1) Based on the pixel coordinates (x, y) of the runway vertices in the image, find the corresponding coordinate expressions in the camera coordinate system (Z*x*u / f, Z*y*u / f, Z), where u is the pixel size of the camera, f is the focal length of the camera, and Z is an unknown quantity. The coordinate expressions of the four vertices A, B, C, and D of the runway are obtained;

[0026] (2) When the length, width, and diagonal length of the runway are known, six quadratic equations are listed according to the distance formula to obtain an overdetermined system of equations. Finally, the stable solution of the overdetermined system of equations, i.e., the values ​​of ZA, ZB, ZC, and ZD, is obtained by using the Levenberg-Marquarelt algorithm and substituted into the coordinate expression in step (1) to obtain the vertex coordinates.

[0027] In step 5, the coordinate W of the projection point of the extended field of view center on the airport runway plane in the camera coordinate system is calculated by the following formula:

[0028] W=(0,0,H / cosβ)

[0029] Where H is the current altitude of the drone, and β is the pitch angle of the camera in geographic coordinates;

[0030] In step 8, the method for solving the drone positioning coordinates is:

[0031] (1) According to the altitude of the UAV, the pitch angle and azimuth angle of the camera in step (1), the distance Δx from the projection point to the due north of the UAV and the distance Δy from the due east of the projection point to the UAV are calculated as follows:

[0032] Δx=-1*Q*cos(β)*sin(α)

[0033] Δy=-1*Q*cos(β)*cos(α)

[0034] Where Q is the distance from the drone to the projection point of the center of the field of view, Q = H / cosβ;

[0035] (2) According to the distance Δx from the projection point to the north and the distance Δy from the projection point to the UAV, the difference in latitude Δati and longitude Δlong between the projection point, the UAV and the center of the field of view are calculated as follows:

[0036] Δati=Δy / rl

[0037] Along=Δx / ree

[0038] Where rl is the radius of the Earth near the target, and ree is the radius of the circle of the Earth parallel to the equatorial plane near the target.

[0039] Beneficial effects of the present invention:

[0040] (1) The method of the present invention proposes an autonomous guided landing method for UAVs based on monocular vision navigation. The positioning accuracy of this method can reach within 10m, which is better than the positioning accuracy of satellite navigation.

[0041] (2) The method of the present invention enables a UAV to accurately guide and land on an airport runway in the absence of a satellite positioning system, relying solely on visual navigation capabilities.

[0042] (3) The method of the present invention continuously corrects the positioning error of the drone based on the camera image during the landing process. Compared with inertial navigation, the error accumulates over time during the guided landing process, and the present invention has more advantages.

[0043] (4) The method of the present invention adopts deep learning and uses the powerful target recognition ability of the YOLOv3 model to first select the airport target from the camera image, effectively reducing the impact of environmental changes on the segmentation effect in the subsequent airport segmentation process. Then, the segnet network is used for semantic segmentation to accurately segment the airport runway.

[0044] (5) Compared with other models based on cooperative goals, the method of the present invention only assumes that the airport is rectangular, and uses the length information of the airport runway and the longitude and latitude coordinates of the four vertices to solve the longitude and latitude coordinates of the UAV positioning. Therefore, the method of the present invention can guide the landing of the UAV without making any specific modifications to the airport, which reflects the universality of the present invention.

[0045] (6) The present invention discloses a method for autonomously guiding the landing of a UAV based on monocular vision navigation, which belongs to the technical field of UAV landing control systems. The method uses a camera onboard a UAV to acquire a runway image; after image preprocessing the acquired image, the airport is identified using YOLOv3, and the region of interest containing the airport runway is cropped based on the detection frame; the runway is then segmented using a segnet model, and the runway edge is linearly fitted to the segmented runway mask to obtain the pixel coordinates of the runway vertex; the UAV positioning coordinates are then calculated by combining the longitude and latitude information of the runway vertex and the UAV attitude angle. Finally, the UAV is guided to correct its course and complete the landing.

[0046] (7) The method of the present invention solves the problem of low accuracy of autonomous landing of UAVs and poor universality of the method when relying solely on visual navigation technology. It meets the actual needs of autonomous and precise landing of UAVs, and its positioning accuracy is not lower than or even better than that of satellite positioning, and has high practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the method flow of the present invention;

[0048] Figure 2 Schematic diagram of the UAV positioning model in an embodiment of the present invention;

[0049] Figure 3Schematic diagram of a camera pinhole imaging model according to an embodiment of the present invention, wherein the coordinate system XYZ is the camera coordinate system, point a is a point in an image of an airport runway captured by the camera, and point A is the corresponding point on the airport runway corresponding to point a;

[0050] Figure 4 This is a flow chart of calculating the current latitude and longitude coordinates of a drone based on the runway vertex pixel coordinates in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] like Figure 1 As shown, the present invention provides a method for autonomously guiding the landing of a UAV based on monocular vision navigation in the following order:

[0053] Step 1: Starting from the drone's landing preparation stage, the drone obtains runway images through its onboard camera.

[0054] Step 2: Use the YOLOv3 model to identify the airport in the acquired image and crop the area S containing the airport runway based on the detection frame. The main purpose of this process is to minimize the impact of environmental factors on the segmentation of the airport runway and reduce the difficulty of subsequent segmentation.

[0055] Step 3: Use SegNet semantic segmentation to segment the cropped area S and output the model segmentation result.

[0056] Step 4: The SegNet segmentation results are not very accurate in determining the runway vertices, with large errors. Appropriate data processing methods are required to obtain the pixel coordinates of the runway vertices. The method adopted by this invention first performs morphological operations on the SegNet segmentation results, first dilating them, then eroding them, to smooth the irregular edges after segmentation. The least squares method is then used to fit the left and right edge lines of the runway, thereby solving for the pixel coordinates of the four runway vertices.

[0057] Step 5: First, according to the current attitude angle of the UAV, obtain the transformation matrix S from the UAV coordinates to the geographic coordinate system. The calculation formula of S is shown in formula (1):

[0058]

[0059] In formula (1), ψ is the UAV heading angle, γ is the UAV pitch angle, and θ is the UAV roll angle.

[0060] Without considering the roll of the drone, the direction vector of the drone's camera relative to the drone is D. This can be obtained based on the pitch angle and azimuth angle of the drone's camera relative to the drone in the aircraft coordinate system. The camera's direction vector in the geographic coordinate system is Dir, and Dir = D*S. The azimuth angle α and pitch angle β of the drone's camera in the geographic coordinate system can be further solved, as follows: Figure 2 shown.

[0061] Without considering the camera distortion, the camera imaging model can be regarded as an ideal pinhole imaging. The pixel coordinates (x, y) of the runway vertex on the image are as follows: Figure 3 In the pinhole imaging model shown, the corresponding coordinates in the camera coordinate system can be written as (Z*x*u / f,Z*y*u / f,Z). Among them, u is the pixel size of the camera, f is the focal length of the camera, and Z is an unknown quantity that needs to be solved. The coordinate expressions of the four vertices A, B, C, and D of the runway can be obtained. Furthermore, combined with the length and width of the runway and the length of the diagonal line, a set of overdetermined equations consisting of six quaternary quadratic equations can be listed through the Euclidean distance calculation formula. Considering that the Levenberg-Marquarelt algorithm is more stable than the solution obtained by the Newton method, this algorithm is used here to solve the above-mentioned quaternary quadratic overdetermined equations to obtain the coordinates of the four vertices of the runway in camera coordinates. The center of the field of view refers to the projection point of the optical center of the camera on the image plane. Combined with Figure 2 It can be seen that point M is the projection point of the field of view center onto the runway. The coordinates of M in the camera coordinate system can be expressed as (0, 0, H / cosβ). The distances from the field of view center point M to the coordinates of each runway vertex [L1, L2, L3, L4] can be calculated. Combined with the longitude and latitude distance calculation formula, an overdetermined system of four quadratic equations is obtained. This system of overdetermined quadratic equations is solved using the Levenberg-Marquarelt algorithm to obtain the longitude and latitude coordinates of the field of view center M.

[0062] exist Figure 2 The distance Δx between the center of the field of view M and the drone in the north direction of the X axis and the distance Δy between the center of the field of view M and the drone in the east direction of the Y axis can be obtained by formula (2). The difference in longitude Δlong and latitude Δati between the drone and the projection point of the center of the field of view can be solved by formula (3):

[0063] Δx=-1*Q*cos(β)*sin(α) (2)

[0064]

[0065] In equation (2), Q can be calculated from the current altitude H of the drone, i.e., Q = H / cosβ. In equation (3), rl is the radius of the Earth near the target, and ree is the radius of the circle parallel to the equatorial plane near the target.

[0066] Finally, the latitude and longitude of the center of the field of view plus the difference between it and the latitude and longitude of the drone will get the positioning coordinates of the drone, and the final calculation result will be output. In step 5, the latitude and longitude of the drone is calculated based on the pixel coordinates of the runway vertex on the image. The overall processing flow is as follows: Figure 4 shown.

[0067] Step 6: Based on the calculated latitude and longitude coordinates of the drone, guide the drone to correct its course until it completes the guided landing.

[0068] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A method for autonomously guiding the landing of an unmanned aerial vehicle based on monocular vision navigation, characterized in that The steps of the method include: Step 1: Obtain the semantic segmentation results of the airport runway through deep learning methods; Step 2, obtaining the pixel position coordinates of the airport runway on the airport runway image; Step 3, obtain the azimuth angle α and pitch angle β of the camera relative to the geographic coordinate system; Step 4: Get the coordinates of the four vertices of the airport runway relative to the camera coordinate system; Step 5, obtaining the coordinates of the projection point of the extended field of view center on the airport runway plane in the camera coordinate system; Step 6: Obtain the Euclidean distance from the projection point to the four vertices on the airport runway; Step 7, solve the latitude and longitude of the projection point; Step 8: Calculate the distance from the UAV to the projection point in the east direction and the distance in the north direction, as well as the latitude difference and longitude difference between the UAV and the projection point, and then calculate the UAV positioning coordinates; Step 9: Feedback the drone's positioning coordinate information to the drone's flight control system to guide the drone to correct its course and complete landing; In step 7, the method for solving the longitude and latitude of the projection point is: according to the longitude and latitude of the four vertices of the airport runway and the Euclidean distance [L1, L2, L3, L4] obtained in step 6, the longitude and latitude of the projection point are solved by the longitude and latitude distance calculation formula. At this point, the UAV positioning is converted into the longitude and latitude of the known projection point and the azimuth and pitch angle of the camera relative to the geographic coordinate system in step 3 to solve the longitude and latitude of the UAV; In step 8, the distance between the drone and the projection point in the east direction and the distance in the north direction are calculated based on the altitude of the drone and the azimuth and pitch angle of the camera in the geographic coordinate system obtained in step 3. The latitude difference and longitude difference between the drone and the projection point are further calculated, and the positioning coordinates of the drone are calculated based on the longitude and latitude of the projection point obtained in step 7. The method to solve the UAV positioning coordinates is: (1) According to the altitude of the UAV, the pitch angle and azimuth angle of the camera in step (1), the distance ∆x between the projection point and the UAV in the due north and the distance ∆y in the due east are calculated as follows: Where Q is the distance from the drone to the projection point of the center of the field of view, Q=H / cosβ; (2) Based on the distance ∆x to the north and the distance ∆y to the east between the projection point and the drone, the difference in latitude ∆ati and longitude ∆long between the projection point, the drone and the center of the field of view are calculated as follows: Where rl is the radius of the Earth near the target, and ree is the radius of the circle of the Earth parallel to the equatorial plane near the target.

2. The method for autonomously guided landing of an unmanned aerial vehicle based on monocular vision navigation according to claim 1, characterized in that: In step 1, the method for obtaining the semantic segmentation result of the airport runway by deep learning method is: The drone collects airport runway image data through its onboard camera, and trains the YOLOv3 model and SegNet model based on the collected airport runway image data. The YOLOv3 model is used to detect the airport runway, and the area S containing the airport runway is cropped based on the detection frame of the detected airport runway. The cropped area S is then segmented by the SegNet model, and the runway segmentation result mask is output.

3. The method for autonomously guided landing of an unmanned aerial vehicle based on monocular vision navigation according to claim 2, characterized in that: In step 2, the method for obtaining the pixel position coordinates of the airport runway on the airport runway image is: Perform morphological closing operation on the runway segmentation result mask, then use the least squares method to fit the left and right edge lines of the runway segmentation result mask, calculate the pixel coordinates of the four vertices of the runway, and then map the pixel coordinates of the four vertices of the runway back to the runway image collected in step 1 to obtain the pixel position coordinates of the runway on the collected runway image.

4. The method for autonomously guided landing of an unmanned aerial vehicle based on monocular vision navigation according to claim 3, characterized in that: In step 3, the method for obtaining the azimuth angle α and the pitch angle β of the camera relative to the geographic coordinate system is: calculating the azimuth angle α and the pitch angle β of the camera relative to the geographic coordinate system using the attitude angle information provided by the onboard gyroscope and the attitude angle of the camera relative to the drone; In step 4, the method for obtaining the coordinates of the four vertices of the airport runway relative to the camera coordinate system is: based on the length and width information of the airport runway and the pixel coordinates of the four vertices of the runway mapped in step 2, the coordinates of the four vertices of the runway relative to the camera coordinate system are calculated by solving a group of equations.

5. The method for autonomously guided landing of an unmanned aerial vehicle based on monocular vision navigation according to claim 4, characterized in that: In step 3, the method for solving the positioning coordinates of the drone is the method for the azimuth angle α and the pitch angle β as follows: First, according to the current attitude angle of the UAV, the transformation matrix S from the UAV coordinate system to the geographic coordinate system is obtained. The calculation formula of S is shown in formula (1): In formula (1), is the UAV heading angle, γ is the UAV pitch angle, and θ is the UAV roll angle; Secondly, according to the pitch angle and azimuth angle of the drone's camera relative to the drone in the aircraft coordinate system, the camera's direction vector D is calculated. The camera's direction vector in the geographic coordinate system is Dir, then , and further solve the azimuth angle α and pitch angle β of the drone camera in the geographic coordinate system.

6. The method for autonomously guided landing of an unmanned aerial vehicle based on monocular vision navigation according to claim 5, characterized in that: In step 4, the coordinate calculation method of the four vertices of the runway in the camera coordinate system is: First, based on the pixel coordinates (x, y) of the runway vertices in the image, find the corresponding coordinate expressions in the camera coordinate system (Z*x*u / f, Z*y*u / f, Z), where u is the camera pixel size, f is the camera focal length, and Z is the unknown quantity. This gives the coordinate expressions for the four vertices A, B, C, and D of the runway. Secondly, according to the length, width and diagonal length of the runway, six quadratic equations are listed according to the distance formula to obtain an overdetermined system of equations. Finally, the stable solution of the overdetermined system of equations is obtained by the Levenberg-Marquarelt algorithm, that is, the values ​​of ZA, ZB, ZC, and ZD, which are substituted into the coordinate expression in step (1) to obtain the vertex coordinates.

7. The method for autonomously guiding the landing of an unmanned aerial vehicle based on monocular vision navigation according to any one of claims 4 to 6, characterized in that: In step 5, the method for obtaining the coordinates of the projection point of the extended field of view center on the airport runway plane in the camera coordinate system is: according to the altitude of the UAV and the pitch angle of the camera in the geographic coordinate system obtained in step 3, the coordinates of the projection point of the extended field of view center on the airport runway plane in the camera coordinate system are obtained; In step 6, the method for obtaining the Euclidean distances from the projection point to the four vertices on the airport runway is as follows: based on the coordinates of the four vertices of the runway relative to the camera coordinates obtained in step 4 and the coordinates of the projection point of the field of view center extended on the airport runway plane obtained in step 5 in the camera coordinate system, the Euclidean distances [L1, L2, L3, L4] from the projection point to the four vertices on the airport runway are obtained.

8. The method for autonomously guided landing of an unmanned aerial vehicle based on monocular vision navigation according to claim 7, characterized in that: In step 5, the coordinate W of the projection point of the extended field of view center on the airport runway plane in the camera coordinate system is calculated by the following formula: W = (0, 0, H / cosβ) Where H is the current altitude of the drone, and β is the pitch angle of the camera in geographic coordinates.

Images