Monocular vision aircraft runway pose estimation method based on line features

Abstract

The invention discloses a monocular vision aircraft runway pose estimation method based on line features. The method comprises the following steps: correcting left, right and middle line equations of an aircraft runway in an airborne monocular vision image by using a least square method and an angular bisector method; calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right and middle line equations of the aircraft runway and the initial line information of the aircraft runway; and optimizing the position and attitude information by using a Butterworth filter to obtain a reliable and stable aircraft runway pose estimation result. According to the method, the aircraft attitude is estimated only by using the line features extracted from the image, the problem of matching of a large number of feature points in a PnP (Perspective-n-Point) algorithm is avoided, and the method has relatively good real-time performance and accuracy so as to provide support for real-time autonomous landing subsequently.

Description

Technical Field

[0001] The present invention relates to the technical field of posture estimation, and in particular to a monocular vision aircraft runway posture estimation method based on line features. Background Art

[0002] Autonomous flight is the basis for realizing the intelligence of aircraft, including autonomous take-off, autonomous cruising and autonomous landing. Autonomous landing is the most challenging and one of the research difficulties in the whole process of flight. In order to ensure the smooth completion of autonomous landing, during the landing phase of the aircraft, it is necessary to calculate the position of the aircraft relative to the runway through the data collected by the sensor, that is, the position and attitude estimation of the aircraft relative to the runway (including the position and attitude relative to the runway), so as to better control the landing of the aircraft.

[0003] At present, the commonly used positioning methods for achieving autonomous landing of aircraft include satellite-based, radar-based, inertial-based, vision-based, and multi-sensor information fusion positioning methods. Among them, visual positioning technology uses the aircraft's onboard visual imaging equipment (visible light, infrared, etc.) to obtain ground scene image information within the sensor's field of view, and extracts feature information of objects in the scene through image processing, thereby realizing the estimation of the navigation parameters of the flight carrier, and then controlling the flight of the aircraft. It has the advantages of low cost, low power consumption, and portability. Autonomous landing of aircraft is applicable to a wide range of scenarios. For example, in the absence of ground guidance systems at small and medium-sized airports, when pilots are disabled, etc., this technology can help aircraft land safely.

[0004] In visual navigation technology, navigation technology based on monocular vision has the characteristics of easy operation, simple equipment and low cost. In addition, due to the small size and light weight of monocular cameras, it is the only choice for some specific applications with space size restrictions or load restrictions. Therefore, designing an aircraft runway pose estimation method with wide application, high accuracy and high real-time performance can effectively lay the foundation for autonomous landing flight control. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a monocular vision aircraft runway pose estimation method based on line features, the method comprising:

[0006] Step S1, using the least square method to correct the left, right and center line equations of the aircraft runway in the airborne monocular vision image;

[0007] Step S2, calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information;

[0008] The position and attitude information is optimized using a Butterworth filter to obtain an aircraft runway attitude estimation result.

[0009] Optionally, in step S1, the process of correcting the left, right and center line equations of the aircraft runway in the airborne monocular vision image using the least squares method specifically includes:

[0010] Determine the equations of left, right, and center lines in airborne monocular vision images;

[0011] Calculating the slopes and intercepts of the left, right and center lines in the airborne monocular vision image;

[0012] Calculate the least squares solution of the three straight line equations of the left line, the right line and the center line, and use the least squares solution as the vanishing point;

[0013] The vanishing points are used to correct the left, right and center line equations of the aircraft runway in the airborne monocular vision image.

[0014] Optionally, in step S2, the content of calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information specifically includes:

[0015] S21, solving the attitude angle based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information to obtain an angle parameter;

[0016] S22, calculating the position and attitude information of the aircraft relative to the aircraft runway based on the angle parameter. Optionally, in S21, the attitude angle is calculated based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information, and the content of obtaining the angle parameter specifically includes:

[0017] S211, calculate the vanishing point using the intersection of the left and right lines Homogeneous coordinates: Among them, 0i, 1i, and 2i represent the coordinate representation of the left, middle, and right line images respectively;

[0018] S212, calculate the hidden line l using the intersection of the left, right and center lines ∞ The normalized coordinates of are:

[0019] l ∞i =[(l 0i × 2i ) T (l 1i × 2i )] 1i +2[(l 0i × 1i ) T (l 2i × 1i )] 2i ;

[0020] S213, calculate the camera intrinsic parameter matrix K and its inverse K -1 , transpose K T ;

[0021] S214. Calculate K -1 P ∞i , K T l ∞i , K -1 P ∞i ×K T l ∞i , and rewrite it into the corresponding row vector form:

[0022]

[0023] Among them, g1, g2, g3, h1, h2, h3, e1, e2, and e3 represent the three row vectors of the three matrices respectively;

[0024] S215. Solve the following equations for α and β: In the formula, p ∞ is the vanishing point, l ∞ is the hidden line, is the rotation matrix, is the direction of the runway line in object space, n π is the normal vector of the runway plane in object space;

[0025] S216, solving the attitude matrix C ij is the posture matrix The element in the i-th row and j-th column of , where (i, j = 1, 2, 3);

[0026] S217. Using the attitude angle to solve the angle parameter:

[0027]

[0028] Optionally, in S22, the content of calculating the position and attitude information of the aircraft relative to the aircraft runway based on the angle parameter specifically includes:

[0029] S221, calculating the relative runway height and lateral position of the aircraft based on the camera intrinsic parameter matrix;

[0030] S222, calculating the longitudinal position of the aircraft relative to the runway based on the intersection of the left runway and the starting line, the intersection of the right runway and the starting line, and the coordinates of the vanishing point;

[0031] S223. Calculate the estimated result of the aircraft's posture relative to the runway based on the corrected left, right and center line equations of the runway, the height and lateral position of the aircraft relative to the runway, and the longitudinal position of the aircraft relative to the runway.

[0032] Optionally, in S221, the content of calculating the relative runway height and lateral position of the aircraft based on the camera intrinsic parameter matrix specifically includes:

[0033] The attitude angle calculation results of the attitude angle calculation module using the camera intrinsic parameter matrix and the attitude calculation module calculate as well as

[0034] Solving equations α0, α2, where a ii is the matrix element of A, l 0x , l 0y , l 0z The coefficient of the left runway line equation, l 2x , l 2y , l 2z The coefficient of the right runway line equation; d left ,d right Indicates the distance from the runway edge to the center line;

[0035] Solve for the height z and lateral position y of the aircraft relative to the runway: In the formula, l 0x , l 0y , l 0z The coefficient of the left runway line equation, l 2x , l 2y , l 2z The coefficient of the right runway line equation; d right Indicates the distance from the runway edge to the center line.

[0036] Optionally, in S222, the content of calculating the longitudinal position of the aircraft relative to the runway based on the intersection of the left runway of the aircraft with the starting line, the intersection of the right runway of the aircraft with the starting line, and the coordinates of the vanishing point specifically includes:

[0037] Calculate the intersection point p2(u2,v2) between the left runway line and the starting line, the intersection point p3(u3,v3) between the right runway line and the starting line, and the coordinates of the vanishing point p(u p ,v p );

[0038] calculate Where f is the focal length;

[0039] calculate k2, k3 in

[0040] Based on the calculated k2 and k3, p2(u2,v2) and p3(u3,v3) in step S1, calculate In

[0041] Given the runway width W and the runway half-length l, calculate P3 ω ;

[0042] Using the calculated P3 ω , P3 c , Solve to get the position parameter matrix The longitudinal position x can be obtained from this.

[0043] Optionally, in step S3, the order of the Butterworth filter is 8.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention uses the left and right runway lines, the center line and the runway start line features in the airborne monocular vision image to calculate the accurate aircraft posture estimation information corresponding to the image at the current moment. The present invention only uses the line features extracted from the image to estimate the aircraft attitude, avoiding the problem of matching a large number of feature points in the PnP (Perspective-n-Point) algorithm, and has good real-time and accuracy, so as to provide support for real-time autonomous landing in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a schematic diagram of the overall process of the method for estimating the position and posture of an aircraft of the present invention;

[0048] Figure 2 A flow chart of line equation correction for the method for estimating the position and attitude of an aircraft according to the present invention;

[0049] Figure 3 A comparison diagram of the line equation correction effect of the aircraft posture estimation method of the present invention;

[0050] Figure 4 A flow chart of posture solution of the method for estimating posture of an aircraft of the present invention;

[0051] Figure 5It is a step diagram of the attitude angle solution module of the posture solution of the aircraft posture estimation method of the present invention;

[0052] Figure 6 A diagram showing the steps of calculating the lateral position and height of the posture of the aircraft posture estimation method of the present invention;

[0053] Figure 7 A diagram showing the longitudinal position solution steps of the posture solution of the aircraft posture estimation method of the present invention;

[0054] Figure 8 A result diagram of posture solution of the method for estimating posture of an aircraft according to the present invention;

[0055] Fig. 9 This is a comparison diagram between the present invention using Butterworth and not using Butterworth. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] A monocular vision aircraft runway pose estimation method based on line features, such as Figure 1 As shown, the method includes:

[0058] Step S1, using the least square method to correct the left, right and center line equations of the aircraft runway in the airborne monocular vision image.

[0059] Reference Figure 2 , after obtaining the slopes k of the left, right and center lines in the airborne monocular vision image l , k r , k m and the intercept h l 、h r 、h m Finally, the equations of the left, right, and center lines are used to find the least squares solution of the three straight lines. The least squares solution is used to replace the vanishing point, and the straight line passing through the fixed point (new vanishing point) is used to correct the slope and intercept in the equations of the left and right lines of the runway, and the new slope and intercept of the left and right lines are obtained as k. lNew , k rNew 、h lNew 、h rNew , so that they retain certain original line segment features while passing through the new vanishing point. After obtaining the new left and right lines, find their angle bisectors to correct the runway centerline and obtain the slope and intercept k of the new centerline mNew 、h mNew , so that it can be used by the subsequent pose solving module.

[0060] Reference Figure 3 , Comparison of the line equation correction effect of the aircraft posture estimation method of the present invention Figure 3 a. Figure 3 c is the attitude angle solution result diagram and error result diagram obtained after the left, right and center lines of the runway are corrected and optimized by the linear equation. In the two figures, "1" on the horizontal axis represents the roll angle, "2" on the horizontal axis represents the pitch angle, and "3" on the vertical axis represents the yaw angle. The unit of the vertical axis is °. Figure 3 b. Figure 3 d is the position solution result diagram and error result diagram obtained after the line equation is corrected and optimized for the left, right and center lines of the runway. In the two diagrams, "4" in the horizontal axis represents the horizontal position, "5" represents the height, and "6" represents the vertical position. The vertical axis unit is meters. Figure 3 a. Figure 3 In b, we can see that the pose solution obtained after the line equation correction module optimizes the left, right and center lines of the runway is closer to the true value line. Figure 3 c. Figure 3 In the error result diagram in d, the error of the pose solution result obtained after the line equation correction and optimization of the left line, right line and center line of the runway is smaller, so the pose estimation result is more accurate after the line equation correction and optimization.

[0061] Reference Figure 4 The posture solution process of the aircraft posture estimation method of the present invention is as follows: first solve the three posture angles, the posture angles can be obtained by solving the posture matrix, and the posture matrix can be solved by solving the left and right lines, center line, vanishing line, vanishing point, and image conjugate equation of the runway. After obtaining the angle parameters, solve the position parameters. The present invention uses the lateral position and height as well as the longitudinal position to solve the position parameters respectively, and the three complete position parameters can be obtained using the runway starting line. Among them, the lateral position and height can be obtained by the left and right lines of the runway, the straight line model, the pitch angle, yaw angle and yaw angle obtained previously; the longitudinal position is obtained by relying on the image conjugate equation and the runway starting line.

[0062] Step S2: Calculate the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information.

[0063] Reference Figure 5 ,The steps of posture solution to calculate the attitude angle are as follows.

[0064] Step 1: Calculate the vanishing point using the intersection of the left and right lines Homogeneous coordinates: 0, 1, and 2 represent the left, center, and right lines, respectively.

[0065] Step 2: Calculate the hidden line l using the intersection of the left, right and center lines ∞ The normalized coordinates of are:

[0066] l ∞i =[(l 0i ×2i ) T (l 1i × 2i )] 1i +2[(l 0i × 1i ) T (l 2i × 1i )] 2i .

[0067] Step 3: Calculate the camera intrinsic parameter matrix K and its inverse K -1 , transpose K T .

[0068] Step 4: Calculate K -1 P ∞i , K T l ∞i , K -1 P ∞i ×K T l ∞i , and rewrite it into the corresponding row vector form:

[0069]

[0070] Step 5: Solve the following equations for α and β: In the formula, p ∞ is the vanishing point, l ∞ is the hidden line, is the rotation matrix, is the direction of the runway line in object space, n π is the normal vector of the runway plane in object space.

[0071] Step 6: Solve for the attitude matrix

[0072] Step 7: Use the attitude angle to solve each angle (roll, pitch, yaw):

[0073]

[0074] Reference Figure 6 ,The steps of the lateral position and height solution module of posture solution are as follows.

[0075] Step 1: Use the camera intrinsic matrix and the attitude angle calculation module to calculate the attitude angle of the pose calculation module calculate as well as

[0076] Step 2: Solve the equation α0, α2, where aii is the matrix element of A, l 0x , l 0y , l 0z The coefficient of the left runway line equation, l 2x , l 2y , l 2z Represents the coefficient of the right runway line equation. d left ,d right Indicates the distance from the runway edge to the center line.

[0077] Step 3: Solve the height z and lateral position y of the aircraft relative to the runway: In the formula, α0 can be obtained from step 2, l 0x , l 0y , l 0z The coefficient of the left runway line equation, l 2x , l 2y , l 2z Represents the coefficient of the right runway line equation. d right Indicates the distance from the runway edge to the center line.

[0078] Reference Figure 7 ,The steps of the longitudinal position solution module of the posture solution module are as follows.

[0079] Step 1: Calculate the intersection point p2(u2,v2) between the left runway line and the starting line, the intersection point p3(u3,v3) between the right runway line and the starting line, and the coordinates of the vanishing point p(u p ,v p ).

[0080] Step 2: Calculation Where f is the focal length.

[0081] Step 3: Calculation k2 and k3 in .

[0082] Step 4: Based on the k2 and k3 calculated in step 3, p2(u2,v2) and p3(u3,v3) in step 1, calculate In P3 c .

[0083] Step 5: Given the runway width W and the runway half-length l, calculate P3 ω .

[0084] Step 6: Using the calculated P3 ω , P3 c , Solve to get the position parameter matrix The longitudinal position x can be obtained from this.

[0085] Reference Figure 8 After adopting the line equation correction and posture solution, the relative posture estimation result of the aircraft relative to the runway is relatively accurate. Figure 8 a is the solution result of a single image. Figure 8 b is the solution result of the video data. It can be seen that the solution result on the video data will fluctuate relative to the true value, so it is necessary to perform subsequent filtering to suppress interference.

[0086] Step S3: Use a Butterworth filter to optimize the position and attitude information to obtain an aircraft runway attitude estimation result.

[0087] Reference Fig. 9 , Fig. 9 AF are the solution parameter results of the roll angle, pitch angle, yaw angle, lateral position, height position, and longitudinal position using Butterworth and not using Butterworth, respectively. The horizontal axis of each sub-graph indicates the current frame number, and the vertical axis indicates the result of the parameter solution corresponding to the sub-graph title. The present invention uses a Butterworth filter to filter the solution signal. In this scenario, the order of the Butterworth filter is set to 8, the sampling frequency is 30 Hz, and the cutoff frequency is set to 1 Hz. In the final result diagram, the dotted line is the true value, the solid line is the solution result of the aircraft posture before filtering the wavefront, and the dotted line is the result after filtering. It can be clearly seen that the sudden change result in the settlement result is removed after filtering, and the high-frequency signal interference is well removed, ensuring the stability of the posture solution result.

[0088] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A monocular vision aircraft runway pose estimation method based on line features, characterized in that: The method comprises: Step S1, using the least square method to correct the left, right and center line equations of the aircraft runway in the airborne monocular vision image; Step S2, calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information; Step S3: Use a Butterworth filter to optimize the position and attitude information to obtain an aircraft runway attitude estimation result.

2. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, characterized in that: In step S1, the process of correcting the left, right and center line equations of the aircraft runway in the airborne monocular visual image using the least squares method specifically includes: Determine the equations of left, right, and center lines in airborne monocular vision images; Calculating the slopes and intercepts of the left, right and center lines in the airborne monocular vision image; Calculate the least squares solution of the three straight line equations of the left line, the right line and the center line, and use the least squares solution as the vanishing point; The vanishing points are used to correct the left, right and center line equations of the aircraft runway in the airborne monocular vision image.

3. The monocular vision aircraft runway pose estimation method based on line features according to claim 2, characterized in that: In step S2, the content of calculating the position and attitude information of the aircraft relative to the aircraft runway based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information specifically includes: S21, solving the attitude angle based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information to obtain an angle parameter; S22. Calculate the position and attitude information of the aircraft relative to the aircraft runway based on the angle parameter.

4. The monocular vision aircraft runway pose estimation method based on line features according to claim 3, characterized in that: In S21, the attitude angle is solved based on the corrected left, right and center line equations of the aircraft runway and the aircraft runway start line information, and the content of the angle parameter obtained specifically includes: S211, calculate the vanishing point using the intersection of the left and right lines Homogeneous coordinates: Among them, 0i, 1i, and 2i represent the coordinate representation of the left, middle, and right line images respectively; S212, calculate the hidden line l using the intersection of the left, right and center lines ∞ The normalized coordinates of are: l ∞i ＝[(l 0i ×l 2i ) T (l 1i ×l 2i )]l 1i +2[(l 0i ×l 1i ) T (l 2i ×l 1i )]l 2i ； S213, calculate the camera internal parameter matrix K and its inverse K -1 , transpose K T ; S214. Calculate K -1 P ∞i , K T l ∞i , K -1 P ∞i ×K T l ∞i , and rewrite it into the corresponding row vector form: Among them, g1, g2, g3, h1, h2, h3, e1, e2, and e3 represent the three row vectors of the three matrices respectively; S215. Solve the following equations for α and β: In the formula, p ∞ is the vanishing point, l ∞ is the hidden line, is the rotation matrix, is the direction of the runway line in object space, n π is the normal vector of the runway plane in object space; S216, solving the attitude matrix Among them, C ij is the posture matrix The element in the i-th row and j-th column of , where (i, j = 1, 2, 3); S217. Using the attitude angle to solve the angle parameter:

5. The monocular vision aircraft runway pose estimation method based on line features according to claim 3, characterized in that: In S22, the content of calculating the position and attitude information of the aircraft relative to the aircraft runway based on the angle parameter specifically includes: S221, calculating the relative runway height and lateral position of the aircraft based on the camera intrinsic parameter matrix; S222, calculating the longitudinal position of the aircraft relative to the runway based on the intersection of the left runway and the starting line, the intersection of the right runway and the starting line, and the coordinates of the vanishing point; S223. Calculate the estimated result of the aircraft's posture relative to the runway based on the corrected left, right and center line equations of the runway, the height and lateral position of the aircraft relative to the runway, and the longitudinal position of the aircraft relative to the runway.

6. The monocular vision aircraft runway pose estimation method based on line features according to claim 5, characterized in that: In S221, the contents of calculating the relative runway height and lateral position of the aircraft based on the camera intrinsic parameter matrix specifically include: The attitude angle calculation results of the attitude angle calculation module using the camera internal parameter matrix and the attitude calculation module calculate as well as Solving equations α0, α2, where a ii is the matrix element of A, l 0x , l 0y , l 0z The coefficient of the left runway line equation, l 2x , l 2y , l 2z The coefficient of the right runway line equation; d left ,d right Indicates the distance from the runway edge to the center line; Solve for the height z and lateral position y of the aircraft relative to the runway: In the formula, l 0x , l 0y , l 0z The coefficient of the left runway line equation, l 2x , l 2y , l 2z The coefficient of the right runway line equation; d right Indicates the distance from the runway edge to the center line.

7. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, characterized in that: In S222, the content of calculating the longitudinal position of the aircraft relative to the runway based on the intersection of the left runway of the aircraft with the starting line, the intersection of the right runway of the aircraft with the starting line, and the coordinates of the vanishing point specifically includes: Calculate the intersection point p2(u2,v2) between the left runway line and the starting line, the intersection point p3(u3,v3) between the right runway line and the starting line, and the coordinates of the vanishing point p(u p ,v p ); calculate Where f is the focal length; calculate k2, k3 in Based on the calculated k2 and k3, p2(u2,v2) and p3(u3,v3) in step S1, calculate In Given the runway width W and the runway half-length l, calculate Using the calculated Solve to get the position parameter matrix The longitudinal position x can be obtained from this.

8. The monocular vision aircraft runway pose estimation method based on line features according to claim 1, characterized in that: In step S3, the order of the Butterworth filter is 8.

Images