A monocular vision aircraft runway pose estimation method based on line features

Abstract

The present invention discloses a method for estimating the runway pose of a monocular vision aircraft based on line features. The method includes: using the least squares method and the angle bisector method to correct the left, right, and center line equations of the aircraft runway in the on-board monocular vision image; calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right, and center line equations of the aircraft runway and the information of the starting line of the aircraft runway; using a Butterworth filter to optimize the position and attitude information to obtain a reliable and stable estimation result of the aircraft runway pose. The present invention only uses the line features extracted from the image to estimate the aircraft attitude, avoiding the large number of feature point matching problems in the PnP (Perspective-n-Point) algorithm, and has good real-time performance and accuracy, so as to provide support for subsequent real-time autonomous landing.

Description

Technical Field

[0001] The present invention relates to the technical field of pose estimation, and particularly relates to a monocular vision aircraft runway pose estimation method based on line features. Background Art

[0002] Autonomous flight is the basis for realizing the intelligence of aircraft, including three stages: autonomous takeoff, autonomous cruise, and autonomous landing. Among them, autonomous landing is the most challenging and one of the research difficulties in the whole flight process. To ensure the successful completion of autonomous landing, during the approach and landing phase of the aircraft, it is necessary to calculate the positioning of the aircraft relative to the runway through the data collected by sensors, that is, the pose estimation of the aircraft relative to the runway (including the position and attitude relative to the runway), so as to better control the aircraft to land.

[0003] Currently, the commonly used positioning methods for realizing autonomous landing of aircraft include satellite-based, radar-based, inertial-based, vision-based, and multi-sensor information fusion positioning methods. Among them, vision positioning technology uses the aircraft-borne vision imaging equipment (visible light, infrared, etc.) of the aircraft to obtain the ground scene image information within the sensor's field of view, extracts the feature information of the objects in the scene through image processing, thereby realizing the estimation of the navigation parameters of the flight vehicle, and then controlling the flight of the aircraft, which has the advantages of low cost, low power consumption, and portability. The applicable scenarios of autonomous landing of aircraft are extensive. For example, in the case of no ground guidance system at small and medium-sized airports or pilot incapacitation, this technology can help the aircraft land safely.

[0004] In vision navigation technology, the navigation technology based on monocular vision has the characteristics of convenient operation, simple equipment, and low cost. Moreover, due to the small size and light weight of the monocular camera, it is the only choice for some specific application scenarios with space size limitations or payload limitations. Therefore, designing an aircraft runway pose estimation method with wide application scenarios, high accuracy, and high real-time performance can effectively lay the foundation for the flight control of autonomous landing. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a monocular vision aircraft runway pose estimation method based on line features, and the method includes:

[0006] Step S1, using the least squares method to correct the left, right, and center line equations of the aircraft runway in the airborne monocular vision image;

[0007] Step S2, based on the corrected left, right, and center line equations of the aircraft runway and the aircraft runway start line information, calculating the position and attitude information of the aircraft relative to the aircraft runway;

[0008] Using a Butterworth filter to optimize the position and attitude information to obtain the aircraft runway pose estimation result.

[0009] Optionally, in step S1, the process of using the least squares method to correct the left, right, and center line equations of the aircraft runway in the airborne monocular vision image specifically includes:

[0010] Determine the left, right, and center line equations in the airborne monocular vision image;

[0011] Calculate the slopes and intercepts of the left, right, and center lines in the airborne monocular vision image;

[0012] Calculate the least squares solution of the three straight lines of the left, right, and center line equations, and use the least squares solution as the vanishing point;

[0013] Use the vanishing point to correct the left, right, and center line equations of the aircraft runway in the airborne monocular vision image.

[0014] Optionally, in step S2, the content of calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right, and center line equations of the aircraft runway and the information of the starting line of the aircraft runway specifically includes:

[0015] S21. Solve the attitude angle based on the corrected left, right, and center line equations of the aircraft runway and the information of the starting line of the aircraft runway to obtain the angle parameter;

[0016] S22. Calculate the position and attitude information of the aircraft relative to the aircraft runway based on the angle parameter. Optionally, in S21, the content of solving the attitude angle based on the corrected left, right, and center line equations of the aircraft runway and the information of the starting line of the aircraft runway to obtain the angle parameter specifically includes:

[0017] S211. Calculate the vanishing point using the intersection point of the left and right lines Homogeneous coordinates: where 0i, 1i, and 2i respectively represent the coordinate representations of the left, middle, and right line images;

[0018] S212. Calculate the vanishing line l ∞ of the normalized coordinates:

[0019] l ∞i =[(l 0i ×l 2i ) T (l 1i ×l 2i )]l 1i +2[(l 0i ×l 1i ) T (l 2i ×l 1i )]l 2i ;

[0020] S213. Calculate the camera intrinsic matrix K and its inverse K -1 . Transpose K T ;

[0021] S214. Calculate K -1 P ∞i , K T l ∞i , K -1 P ∞i ×K T l ∞i , and rewrite it into the corresponding row vector form:

[0022]

[0023] where g1, g2, g3, h1, h2, h3, e1, e2, e3 respectively represent the three row vectors of the three matrices;

[0024] S215. Solve for α and β in the following equation: In the formula, p ∞ is the vanishing point, l ∞ is the vanishing line, is the rotation matrix, is the direction of the runway line in the object space, n π is the normal vector of the runway plane in the object space;

[0025] S216. Solve the attitude matrix C ij is the element in the i-th row and j-th column of the attitude matrix , where (i, j = 1, 2, 3);

[0026] S217. Solve the angular parameters using the attitude angles:

[0027]

[0028] Optionally, in S22, the content of calculating the position and attitude information of the aircraft relative to the aircraft runway based on the angular parameters specifically includes:

[0029] S221. Calculate the runway height and lateral position of the aircraft relative to the camera intrinsic matrix;

[0030] S222. Calculate the longitudinal position of the aircraft relative to the runway based on the intersection points of the left runway and the starting line, the right runway and the starting line, and the vanishing point coordinates;

[0031] S223. Calculate the pose estimation result of the aircraft relative to the runway based on the corrected left, right, and centerline equations of the aircraft runway, the relative height and lateral position of the aircraft with respect to the runway, and the longitudinal position of the aircraft relative to the runway.

[0032] Optionally, in S221, the content of calculating the relative height and lateral position of the aircraft with respect to the runway based on the camera internal parameter matrix specifically includes:

[0033] Using the camera internal parameter matrix and the attitude angle calculation result of the attitude angle calculation module of the pose calculation module Calculate And

[0034] Solve the equation for α0 and α2, where a ii is an element of matrix A, and l 0x , l 0y , l 0z represent the coefficients of the left runway line equation, and l 2x , l 2y , l 2z represent the coefficients of the right runway line equation; d left , d right represents the distance from the runway edge line to the center line;

[0035] Solve for the height z and lateral position y of the aircraft relative to the runway: where l 0x , l 0y , l 0z represent the coefficients of the left runway line equation, and l 2x , l 2y , l 2z represent the coefficients of the right runway line equation; d right represents the distance from the runway edge line to the center line.

[0036] Optionally, in S222, the content of calculating the longitudinal position of the aircraft relative to the runway based on the intersection points of the left runway of the aircraft and the starting line, the intersection points of the right runway and the starting line, and the vanishing point coordinates specifically includes:

[0037] Calculate the intersection point p2(u2, v2) of the left runway line and the starting line, the intersection point p3(u3, v3) of the right runway line and the starting line, and the vanishing point coordinates p(u p , v p );

[0038] Calculate where f is the focal length;

[0039] Calculate for k2 and k3 in;

[0040] According to the calculated k2 and k3, and p2(u2, v2) and p3(u3, v3) in step S1, calculate in

[0041] Given the runway width W and the half-length l of the runway, calculate P3 ω ;

[0042] Using the calculated P3 ω , P3 c , Solve to obtain the position parameter matrix From this, the longitudinal position x can be obtained.

[0043] Optionally, in step S3, the order of the Butterworth filter is 8.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] The present invention uses the left and right sidelines, the center line, and the starting line features of the runway in the airborne monocular vision image to calculate the accurate aircraft pose estimation information corresponding to the image at the current moment. The present invention only uses the line features extracted from the image to estimate the aircraft attitude, avoiding a large number of feature point matching problems in the PnP (Perspective-n-Point) algorithm, and has good real-time performance and accuracy, so as to provide support for subsequent real-time autonomous landing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the present invention, the following briefly introduces the drawings required in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is the overall flow chart of the aircraft pose estimation method of the present invention;

[0048] Figure 2 It is the flow chart of the line equation correction of the aircraft pose estimation method of the present invention;

[0049] Figure 3 It is the comparison diagram of the line equation correction effect of the aircraft pose estimation method of the present invention;

[0050] Figure 4 It is the flow chart of the pose calculation of the aircraft pose estimation method of the present invention;

[0051] Figure 5It is a step diagram of the attitude angle calculation module for pose calculation in the aircraft pose estimation method of the present invention;

[0052] Figure 6 It is a step diagram of the lateral position and altitude calculation in the pose calculation of the aircraft pose estimation method of the present invention;

[0053] Figure 7 It is a step diagram of the longitudinal position calculation in the pose calculation of the aircraft pose estimation method of the present invention;

[0054] Figure 8 It is a result diagram of the pose calculation in the aircraft pose estimation method of the present invention;

[0055] Figure 9 It is a comparison diagram of using Butterworth and not using Butterworth in the present invention. Detailed implementation manners

[0056] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0057] A monocular vision aircraft runway pose estimation method based on line features, as Figure 1 shown, the method includes:

[0058] Step S1, using the least squares method to correct the left, right, and middle line equations of the aircraft runway in the airborne monocular vision image.

[0059] Referring to Figure 2 , after obtaining the slopes k l , k r , k m and intercepts h l , h r , h m of the left, right, and middle lines in the airborne monocular vision image, use the equations of the left, right, and middle lines to find the least squares solution of the three lines, use this least squares solution to replace the vanishing point, and use the line passing through the fixed point (new vanishing point) to correct the slopes and intercepts in the left and right line equations of the runway, obtaining the new left and right line slopes and intercepts as k lNew , k rNew , h lNew , h rNew , so that they not only retain certain original line segment features but also pass through the new vanishing point. After obtaining the new left and right lines, find their angle bisector to correct the middle line of the runway, obtaining the slope and intercept k mNew , h mNew of the new middle line for subsequent use by the pose calculation module.

[0060] Referring to Figure 3 , Comparison of the line equation correction effect of the aircraft pose estimation method of the present inventionFigure 3 a, Figure 3 c is the attitude angle solution result graph and error result graph obtained after the runway left line, right line, and center line are corrected and optimized by the line equation. In both graphs, "1" in the abscissa represents the roll angle, "2" represents the pitch angle, "3" represents the yaw angle, and the ordinate unit is °; Figure 3 b, Figure 3 d is the position solution result graph and error result graph obtained after the runway left line, right line, and center line are corrected and optimized by the line equation. In both graphs, "4" in the abscissa represents the lateral position, "5" represents the altitude, "6" represents the longitudinal position, and the ordinate unit is meters. From Figure 3 a, Figure 3 b, it can be seen that the pose solution result obtained after the runway left line, right line, and center line are corrected and optimized by the line equation correction module is closer to the true value line. Figure 3 c, Figure 3 d, in the error result graph, the error of the pose solution result obtained after the runway left line, right line, and center line are corrected and optimized by the line equation is smaller. Therefore, the pose estimation result is more accurate after being corrected and optimized by the line equation.

[0061] Refer to Figure 4 , the pose solution process of the pose estimation method of the aircraft of the present invention is as follows: First, solve 3 attitude angles. The attitude angles can be solved from the attitude matrix, and the attitude matrix can be solved from the runway left and right lines, center line, hidden line, vanishing point, and image conjugate equation. After obtaining the angle parameters, solve the position parameters. The present invention solves the position parameters by the lateral position, altitude, and longitudinal position respectively, and uses the runway starting line to obtain the complete 3 position parameters. Among them, the lateral position and altitude can be obtained from the runway left and right lines, straight line model, pitch angle, yaw angle, and yaw angle obtained previously; the longitudinal position is obtained by relying on the image conjugate equation and the runway starting line.

[0062] Step S2: Calculate the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right, and center line equations of the aircraft runway and the aircraft runway starting line information.

[0063] Refer to Figure 5 , the steps for solving the attitude angle of the pose solution are as follows.

[0064] Step 1: Calculate the vanishing point using the intersection of the left and right lines Homogeneous coordinates: 0, 1, 2 represent the left, middle, and right lines respectively.

[0065] Step 2: Calculate the hidden line l ∞ of the normalized coordinates:

[0066] l ∞i = [(l 0i × l2i ) T (l 1i ×l 2i )]l 1i +2[(l 0i ×l 1i ) T (l 2i ×l 1i )]l 2i 。

[0067] Step 3: Calculate the camera intrinsic matrix K and its inverse K -1 and transpose K T 。

[0068] Step 4: Calculate KP -1 P ∞i 、K T l ∞i 、K -1 P ∞i ×K T l ∞i and rewrite them in the corresponding row vector form:

[0069]

[0070] Step 5: Solve for α and β in the following equation: where p ∞ is the vanishing point, l ∞ is the vanishing line, is the rotation matrix, is the direction of the runway line in the object space, and n π is the normal vector of the runway plane in the object space.

[0071] Step 6: Solve for the pose matrix

[0072] Step 7: Solve for each angle (roll, pitch, yaw) using the pose angles:

[0073]

[0074] Refer to Figure 6 for the following steps of the lateral position and altitude calculation module in pose calculation.

[0075] Step 1: Calculate and using the camera intrinsic matrix and the pose angle calculation result of the pose angle calculation module in the pose calculation module

[0076] Step 2: Solve for α0 and α2 in the equation where aii is an element of matrix A, l 0x , l 0y , l 0z represent the coefficients of the left runway line equation, l 2x , l 2y , l 2z represent the coefficients of the right runway line equation. d left , d right represents the distance from the runway sideline to the center line.

[0077] Step 3: Solve for the height z and lateral position y of the aircraft relative to the runway: In the formula, α0 can be obtained from Step 2, l 0x , l 0y , l 0z represent the coefficients of the left runway line equation, l 2x , l 2y , l 2z represent the coefficients of the right runway line equation. d right represents the distance from the runway sideline to the center line.

[0078] Refer to Figure 7 , and the steps of the longitudinal position calculation module of the pose calculation module are as follows.

[0079] Step 1: Calculate the intersection point p2(u2, v2) of the left runway line and the starting line, the intersection point p3(u3, v3) of the right runway line and the starting line, and the coordinates of the vanishing point p(u p , v p ).

[0080] Step 2: Calculate where f is the focal length.

[0081] Step 3: Calculate k2 and k3 in .

[0082] Step 4: According to k2 and k3 obtained in Step 3, p2(u2, v2) and p3(u3, v3) in Step 1, calculate in P3 c .

[0083] Step 5: Given the runway width W and the half-length l of the runway, calculate P3 ω .

[0084] Step 6: Using the calculated P3 ω , P3 c , Solve to obtain the position parameter matrix From this, the longitudinal position x can be obtained.

[0085] Referring to Figure 8 , after line equation correction and pose calculation, the relative pose estimation result of the aircraft with respect to the runway is relatively accurate. Figure 8 a is the calculation result of a single picture. Figure 8 b is the calculation result of video data. It can be seen that the calculation result on the video data fluctuates relative to the true value. Therefore, it is necessary to perform subsequent filtering to suppress interference.

[0086] Step S3: Use a Butterworth filter to optimize the position and attitude information to obtain the aircraft runway pose estimation result.

[0087] Referring to Figure 9 , Figure 9 a - f in are the calculation parameter results of the roll angle, pitch angle, yaw angle, lateral position, height position, and longitudinal position with and without using the Butterworth filter. The abscissa of each sub - figure represents the current frame number, and the ordinate represents the calculation result of the parameter corresponding to the sub - figure title. The present invention uses a Butterworth filter to filter the calculation signal. In this scenario, the order of the Butterworth filter is set to 8, the sampling frequency is 30 hz, and the cut - off frequency is set to 1 hz. In the final result figure obtained, the dotted line is the true value, the solid line is the aircraft pose calculation result before filtering, and the dashed - dotted line is the result after filtering. It can be clearly seen that the sudden change results in the calculation result are removed after filtering, and the high - frequency signal interference is well removed, ensuring the stability of the pose calculation result.

[0088] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A monocular vision aircraft runway pose estimation method based on line features, characterized in that The method includes: Step S1: Using the least squares method to correct the left, right, and center line equations of the aircraft runway in the airborne monocular vision image; Step S2: Calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right, and center line equations of the aircraft runway and the information of the starting line of the aircraft runway. Specifically, it includes: S21: Solving the attitude angles based on the corrected left, right, and center line equations of the aircraft runway and the information of the starting line of the aircraft runway to obtain angle parameters. Specifically, it includes: S211. Calculate the vanishing point using the intersection points of the left and right lines Homogeneous coordinates: where 0i, 1i, and 2i respectively represent the coordinate representations of the left, middle, and right line images; S212: Calculating the normalized coordinates of the vanishing line using the intersection points of the left, right, and center lines: l ∞i ＝[(l 0i ×l 2i ) T (l 1i ×l 2i )]l 1i +2[(l 0i ×l 1i ) T (l 2i ×l 1i )]l 2i ； S213. Calculate the camera internal parameter matrix K and its inverse K -1 and the transpose of K T ; S214. Calculate K -1 P ∞i 、K T l ∞i 、K -1 P ∞i ×K T l ∞i , and rewrite it in the corresponding row vector form: Where g1, g2, g3, h1, h2, h3, e1, and e2 respectively represent the three row vectors of the three matrices; S215: Solving for α and β in the following equation: where p ∞i is the vanishing point, l ∞i is the vanishing line, is the rotation matrix, is the direction of the runway straight line in the object space, and n π is the normal vector of the runway plane in the object space; S216. Solve the attitude matrix where C ij is the element in the i-th row and j-th column of the attitude matrix , where (i, j = 1, 2, 3); S217: Solving the angle parameters using the attitude angles; S22: Calculating the position and attitude information of the aircraft relative to the aircraft runway based on the angle parameters; Specifically, it includes: S221: Calculating the runway height and lateral position of the aircraft relative to the camera based on the camera internal parameter matrix; S222: Calculating the longitudinal position of the aircraft relative to the runway based on the intersection points of the left runway and the starting line, the right runway and the starting line, and the coordinates of the vanishing point; S223: Calculating the pose estimation result of the aircraft relative to the runway based on the corrected left, right, and center line equations of the aircraft runway, the runway height and lateral position of the aircraft relative to the runway, and the longitudinal position of the aircraft relative to the runway. Step S3: Using a Butterworth filter to optimize the position and attitude information to obtain the pose estimation result of the aircraft runway.

2. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, wherein In the step S1, the process of using the least squares method to correct the left, right, and center line equations of the aircraft runway in the airborne monocular vision image specifically includes: Determining the left, right, and center line equations in the airborne monocular vision image; Calculating the slopes and intercepts of the left, right, and center lines in the airborne monocular vision image; Calculating the least squares solution of the three straight lines of the left, right, and center line equations, and taking the least squares solution as the vanishing point; Using the vanishing point to correct the left, right, and center line equations of the aircraft runway in the airborne monocular vision image.

3. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, wherein In the S221, the content of calculating the runway height and lateral position of the aircraft relative to the camera based on the camera internal parameter matrix specifically includes: The attitude angle calculation results of the attitude angle calculation module using the camera internal parameter matrix and the pose calculation module Calculate And Solve the equation for α0 and α2, where a ii is an element of matrix A, and l 0x , l 0y , l 0z represent the coefficients of the left runway line equation, and l 2x , l 2y , l 2z represent the coefficients of the right runway line equation; d left , d right represent the distance from the runway sideline to the center line; Solving for the height z and lateral position y of the aircraft relative to the runway: wherein, l 0x , l 0y , l 0z represent the coefficients of the left runway line equation, and l 2x , l 2y , l 2z represent the coefficients of the right runway line equation; d right represents the distance from the runway sideline to the center line.

4. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, characterized in that In the S222, the content of calculating the longitudinal position of the aircraft relative to the runway based on the intersection points of the left runway and the starting line, the right runway and the starting line, and the coordinates of the vanishing point specifically includes: Calculate the intersection point p2(u2, v2) of the left runway line and the starting line, the intersection point p3(u3, v3) of the right runway line and the starting line, and the coordinates of the vanishing point p(u p , v p ); Calculation where f is the focal length; Calculation of k2 and k3 therein; Calculate based on the calculated k2 and k3, and p2(u2, v2) and p3(u3, v3) in step S1 in P3 c ; Given the runway width W and the semi-length l of the runway, calculate P3 ω ; Using the calculated P3 ω , P3 c , the position parameter matrix obtained by solving From this, the longitudinal position x can be obtained.

5. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, characterized in that In the step S3, the order of the Butterworth filter is 8.

Images