Vision-based autonomous landing method and indication picture design method

Abstract

The invention belongs to the technical field of machine vision, and particularly relates to a vision-based autonomous landing method and an indication picture design method.A runway and a horizon are detected and tracked through a vision sensor, image information of a target runway corner and the horizon is obtained, and then a coordinate conversion relation between the runway image information and physical dimensions is established; the position and attitude of the aircraft relative to the runway are calculated by using a PNP algorithm, more accurate roll angle information is calculated by using a horizontal line, and navigation parameters are drawn and displayed on a head-up display, so that a driver can see the navigation information of the aircraft relative to the runway while looking at the scene outside the aircraft horizontally, the workload of the pilot can be reduced, and the navigation efficiency is improved. Therefore, the flight safety is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision, and in particular relates to a vision-based autonomous landing method and an indication screen design method. Background Art

[0002] Currently, CCAR 25 transport aircraft have achieved automatic flight from takeoff altitudes above 400ft to descent to the decision altitude through "flight management + automatic flight system". During takeoff and landing below 400ft, manual operation by the pilot is still the main method. In order to reduce or even replace the reliance of aircraft below 400ft on precision instrument navigation, Airbus launched the Autonomous Takeoff and Landing (ATTOL) project, which aims to explore autonomous flight technology and conducted a fully automatic visual takeoff test on the A350-1000 model.

[0003] Machine vision technology uses computer vision algorithms through onboard image sensors and other equipment, and can obtain the aircraft's current navigation and location information, just like traditional ILS and other navigation facilities and airport sensor equipment, to provide input for guidance algorithms, reduce dependence on airport facilities, and expand the scope of application of aircraft landing functions. In the future, as machine vision-based landing guidance technology matures and develops, it can assist aircraft in achieving assisted, enhanced, and fully intelligent flight. Summary of the invention

[0004] In view of this, in order to solve the problem of limitations of external navigation equipment, the present invention provides a vision-based autonomous landing method and an indication screen design method, which obtains navigation information based on a visual sensor and draws the navigation information on a head-up display to assist the pilot in completing the landing of the aircraft, thereby reducing the dependence on airport facilities and alleviating the pilot's driving burden.

[0005] The technical solution of the present invention is:

[0006] A vision-based autonomous landing method and an indication screen design method, the autonomous landing method comprising the following steps:

[0007] S101: The aircraft detects and tracks the four corner image information and horizon image information of the airport runway in real time based on the visual sensor;

[0008] S102: Acquire internal reference information of the visual sensor;

[0009] S103: Mapping the geographical coordinates of the current runway into pixel coordinates of the runway by combining the internal reference information and the four corner point image information;

[0010] S104: Calculating the position and attitude information of the aircraft relative to the runway based on the geographic coordinates of the current runway, the identified pixel coordinates, and the mapping relationship between the geographic coordinates and the identified pixel coordinates;

[0011] S105: Based on the position information of the aircraft relative to the runway, solve the lateral deviation and longitudinal deviation of the aircraft relative to the runway.

[0012] Furthermore, the autonomous landing method further includes:

[0013] The accurate roll angle of the aircraft is obtained based on the pixel coordinates identified by the horizon image information to replace the roll angle in the attitude information.

[0014] Furthermore, in S103, the method for establishing the coordinate system is specifically as follows:

[0015] World Coordinate System O w X w Y w Z w Fixed; origin O w It is the intersection of the left sideline of the runway and the end line of the runway at a distance. ′w The axis is in the same direction as the runway edge, X w Axis and Z ′w The axis lies in the plane of the runway and is perpendicular to the Z w Axis, pointing to the right of the runway, Y w The axis is perpendicular to the runway plane, and the coordinate system satisfies the right-hand rule;

[0016] The image coordinate system is divided into two types: image pixel coordinate system (u, v) and image physical coordinate system (x, y), which are defined as:

[0017] Image pixel coordinate system (u, v): The image pixel coordinate system is a rectangular coordinate system with the upper left corner of the image as the origin and pixels as the coordinate unit; u and v represent the column and row numbers of the pixel in the digital image respectively;

[0018] Image physical coordinate system (x, y): The image physical coordinate system is a rectangular coordinate system with the intersection of the optical axis and the image plane as the origin and in millimeters; its x and y axes are parallel to the u and v axes of the image pixel coordinate system respectively and have the same direction;

[0019] Camera coordinate system O c X c Y c Z c Take the origin O c is the optical center of the camera, Z c The X axis coincides with the optical axis of the camera, and the camera direction is taken as the positive direction. c , Y c The axis is parallel to the x and y axes of the image physical coordinate system and has the same direction; the camera coordinate system satisfies the right-hand rule.

[0020] Furthermore, in S103, the mapping relationship is established by:

[0021] Establish the mapping relationship between the runway's geographic coordinates and pixel coordinates as follows:

[0022]

[0023] K is the intrinsic parameter matrix of the camera, T is the coordinate of the origin of the world coordinate system in the camera coordinate system; The transformation matrix between the world coordinate system and the camera coordinate system.

[0024] Furthermore, in S104, the method for calculating the position and attitude information of the aircraft relative to the runway is specifically as follows:

[0025] The position and attitude information of the aircraft relative to the runway are calculated using the PNP algorithm based on the internal reference information of the visual sensor, the pixel coordinates of the four corner points of the runway, and the runway size information.

[0026] Furthermore, the calculation method of the precise roll angle is:

[0027] The precise roll angle of the aircraft is calculated using the two-point coordinates of the horizon end line based on image recognition.

[0028] Furthermore, the method for solving the lateral deviation and longitudinal deviation of the aircraft relative to the runway is specifically as follows:

[0029] Calculate the lateral deviation and longitudinal deviation: the lateral deviation parameter Loc = PW / 2, where P is the distance from the aircraft to the left sideline of the runway, and W is the runway width; the longitudinal deviation parameter Gs = h-tan(ψ)*Z, where Z is the distance from the projection of the aircraft on the runway plane to the runway entrance point, h is the vertical distance from the aircraft to the runway plane, and the angle ψ is the standard angle of the aircraft at the runway entrance point.

[0030] Furthermore, the indication screen design method includes:

[0031] Set the pitch ladder symbol, horizontal line symbol, heading scale symbol, altitude symbol, longitudinal deviation symbol, distance and lateral deviation symbol;

[0032] The symbols are used to display in real time:

[0033] The position information of the aircraft relative to the runway, the replaced attitude information, and the lateral deviation and longitudinal deviation of the aircraft relative to the runway.

[0034] Furthermore, the internal parameters include focal length, principal point coordinates, and distortion coefficients.

[0035] The beneficial effects of the present invention are:

[0036] 1) No external equipment is required, only visual sensors are needed to complete autonomous navigation and guide the aircraft to land.

[0037] 2) Design a simple navigation screen, utilize the equiangular display performance of the head-up display, and simultaneously observe the visual information outside the window to display navigation information in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 It is a schematic diagram of the visually assisted aircraft landing scheme involved in the present invention.

[0040] Figure 2 is a schematic diagram of the symbolic appearance of this method. DETAILED DESCRIPTION

[0041] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0042] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0043] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0044] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0045] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0046] In one embodiment of the present invention, a vision-based autonomous landing method and an indication screen design method are proposed, and the autonomous landing method comprises the following steps:

[0047] S101: The aircraft detects and tracks the four corner image information and horizon image information of the airport runway in real time based on the visual sensor;

[0048] S102: Acquire internal reference information of the visual sensor;

[0049] S103: Mapping the geographical coordinates of the current runway into pixel coordinates of the runway by combining the internal reference information and the four corner point image information;

[0050] S104: Calculating the position and attitude information of the aircraft relative to the runway based on the geographic coordinates of the current runway, the identified pixel coordinates, and the mapping relationship between the geographic coordinates and the identified pixel coordinates;

[0051] S105: Based on the position information of the aircraft relative to the runway, solve the lateral deviation and longitudinal deviation of the aircraft relative to the runway.

[0052] In this embodiment, the autonomous landing method further includes:

[0053] The accurate roll angle of the aircraft is obtained based on the pixel coordinates identified by the horizon image information to replace the roll angle in the attitude information.

[0054] In this embodiment, in S103, the method of establishing the coordinate system is specifically:

[0055] World Coordinate System O w X w Y w Z w Fixed; origin O w It is the intersection of the left sideline of the runway and the end line of the runway in the distance. w The axis is in the same direction as the runway edge, X w Axis and Z w The axis lies in the plane of the runway and is perpendicular to the Z wAxis, pointing to the right of the runway, Y w The axis is perpendicular to the runway plane, and the coordinate system satisfies the right-hand rule;

[0056] The image coordinate system is divided into two types: image pixel coordinate system (u, v) and image physical coordinate system (x, y), which are defined as:

[0057] Image pixel coordinate system (u, v): The image pixel coordinate system is a rectangular coordinate system with the upper left corner of the image as the origin and pixels as the coordinate unit; u and v represent the column and row numbers of the pixel in the digital image respectively;

[0058] Image physical coordinate system (x, y): The image physical coordinate system is a rectangular coordinate system with the intersection of the optical axis and the image plane as the origin and in millimeters; its x and y axes are parallel to the u and v axes of the image pixel coordinate system respectively and have the same direction;

[0059] Camera coordinate system O c X c Y c Z c Take the origin O c is the optical center of the camera, Z c The X axis coincides with the optical axis of the camera, and the camera direction is taken as the positive direction. c , Y c The axis is parallel to the x and y axes of the image physical coordinate system and has the same direction; the camera coordinate system satisfies the right-hand rule.

[0060] In this embodiment, in S103, the method for establishing the mapping relationship is:

[0061] Establish the mapping relationship between the runway's geographic coordinates and pixel coordinates as follows:

[0062]

[0063] K is the intrinsic parameter matrix of the camera, T is the coordinate of the origin of the world coordinate system in the camera coordinate system; The transformation matrix between the world coordinate system and the camera coordinate system.

[0064] In this embodiment, in S104, the method for calculating the position and attitude information of the aircraft relative to the runway is specifically as follows:

[0065] The position and attitude information of the aircraft relative to the runway are calculated using the PNP algorithm based on the internal reference information of the visual sensor, the pixel coordinates of the four corner points of the runway, and the runway size information.

[0066] In this embodiment, the calculation method of the precise roll angle is:

[0067] The precise roll angle of the aircraft is calculated using the two-point coordinates of the horizon end line based on image recognition.

[0068] In this embodiment, the method for solving the lateral deviation and longitudinal deviation of the aircraft relative to the runway is specifically as follows:

[0069] Calculate the lateral deviation and longitudinal deviation: lateral deviation parameter Loc = PW / 2, where P is the distance from the aircraft to the left sideline of the runway, and W is the runway width; longitudinal deviation parameter Gs = h-tan(ψ)*Z, where Z is the distance from the projection of the aircraft on the runway plane to the runway entrance point, h is the vertical distance from the aircraft to the runway plane, and angle ψ is the standard angle of the aircraft at the runway entrance point.

[0070] In this embodiment, the indication screen design method includes:

[0071] Set the pitch ladder symbol, horizontal line symbol, heading scale symbol, altitude symbol, longitudinal deviation symbol, distance and lateral deviation symbol;

[0072] The symbols are used to display in real time:

[0073] The position information of the aircraft relative to the runway, the replaced attitude information, and the lateral deviation and longitudinal deviation of the aircraft relative to the runway.

[0074] In this embodiment, the internal parameters include focal length, principal point coordinates, and distortion coefficients.

[0075] This embodiment provides a vision-based autonomous landing and instruction screen design method, that is, obtaining navigation information based on a visual sensor and drawing the navigation information on a head-up display to assist the pilot in completing the landing of the aircraft, thereby reducing the dependence on airport facilities and alleviating the pilot's driving burden.

[0076] See also Figure 1-Figure 2 This embodiment adopts the following steps:

[0077] 1) Establish a coordinate system as follows:

[0078] World Coordinate System O w X w Y w Z w Fixed. Origin O w It is the intersection of the left sideline of the runway and the end line of the runway in the distance. w The axis is in the same direction as the runway edge, X w Axis and Z w The axis lies in the plane of the runway and is perpendicular to the Z w Axis, pointing to the right of the runway, Y w The axis is perpendicular to the runway plane. The coordinate system satisfies the right-hand rule.

[0079] The image coordinate system is divided into two types: image pixel coordinate system (u, v) and image physical coordinate system (x, y), which are defined as follows:

[0080] Image pixel coordinate system (u, v): The image pixel coordinate system is a rectangular coordinate system with the upper left corner of the image as the origin and pixels as the coordinate unit. u and v represent the column and row numbers of the pixel in the digital image respectively.

[0081] Image physical coordinate system (x, y): The image physical coordinate system is a rectangular coordinate system with the intersection of the optical axis and the image plane as the origin and in millimeters. Its x and y axes are parallel to the u and v axes of the image pixel coordinate system, respectively, and have the same direction.

[0082] Camera coordinate system O c X c Y c Z c Take the origin O c is the optical center of the camera, Z c The X axis coincides with the optical axis of the camera, and the camera direction is taken as the positive direction. c , Y c The axis is parallel to the x and y axes of the image physical coordinate system and has the same direction. The coordinate system satisfies the right-hand rule.

[0083] 2) Establish the mapping relationship between the runway's geographic coordinates and pixel coordinates, as shown below:

[0084]

[0085] K is the intrinsic parameter matrix of the camera, and T is the coordinate of the origin of the world coordinate system in the camera coordinate system. The transformation matrix between the world coordinate system and the camera coordinate system.

[0086] 3) The position and attitude information of the aircraft relative to the runway are calculated using the PNP algorithm based on the internal reference information of the visual sensor, the pixel coordinates of the four corner points of the runway, and the runway size information.

[0087] 4) The aircraft roll angle is calculated using the two-point coordinates of the horizon end line based on image recognition.

[0088] 5) Calculate the lateral deviation and longitudinal deviation. The lateral deviation parameter Loc = PW / 2, where P

[0089] is the distance from the aircraft to the left sideline of the runway, W is the runway width; longitudinal deviation parameter Gs =

[0090] h-tan(ψ)*Z, where Z is the distance from the projection of the aircraft on the runway plane to the runway entrance point, h

[0091] is the vertical distance from the aircraft to the runway plane, and angle ψ is the standard angle of the aircraft at the runway entrance point.

[0092] 6) Symbol design: pitch ladder, used to display pitch and roll angles; horizontal line and heading scale, used to display relative heading; altitude and longitudinal deviation symbol, used to display altitude and longitudinal deviation information in combination; distance and lateral deviation symbol.

[0093] This embodiment uses an algorithm to calculate and display the position and attitude of the aircraft relative to the runway, thereby reducing dependence on airport facilities and alleviating the pilot's driving burden, thereby ensuring safe landing of the aircraft.

[0094] 1) Use image recognition algorithm to identify the image information of the four corner points and horizontal line of the runway.

[0095] 2) Calculate the position and attitude information of the aircraft relative to the runway based on the physical information of the runway.

[0096] 3) Based on the calculated position and attitude information of the aircraft relative to the runway, the lateral deviation and longitudinal deviation are further calculated.

[0097] 4) Draw the screen according to the navigation information. The symbol design is as follows:

[0098] Design requirements for pitch ladders

[0099] The pitch ladder is in the form of a step with a scale set every 5°. The pitch ladder should move or roll according to the pitch angle and roll angle.

[0100] Heading design requirements

[0101] The current heading pointer is fixedly displayed above the center of the horizon, pointing to the relative heading.

[0102] Height and longitudinal deviation design requirements

[0103] The rectangular vertical bar is fixed in position, while the aircraft symbol and deviation bar move with altitude changes;

[0104] The rectangular vertical bar of 300mrad represents a height of 600m, the top represents 600 meters, and the bottom represents 0 meters;

[0105] The number above the aircraft symbol indicates the current altitude (meters);

[0106] The 80mrad deviation bar on the right side of the rectangular vertical bar indicates a vertical deviation of 40 meters, 20 meters above and below, with the upper side being positive and the lower side being negative. The upper and lower "20" and "-20" indicate the maximum deviation range;

[0107] The symbol "3" on the right side of the rectangular vertical bar indicates the current true vertical deviation, which changes with the true deviation value;

[0108] The dotted line on the rectangular vertical bar indicates the current height, which is converted to the corresponding position of the rectangular vertical bar along with the height value.

[0109] The aircraft symbol moves relative to the deviation rod based on the wing. When the wing coincides with the horizontal dotted line, it means the deviation is 0.

[0110] Design requirements for distance and lateral deviation symbols

[0111] 1. The rectangular horizontal bar is fixed, and the aircraft symbol and deviation bar move with the distance; 2. The rectangular horizontal bar represents a distance scale of 5000m, the leftmost side represents 5000m, and the rightmost side represents 0m;

[0112] 3. The characters on the left side of the aircraft symbol identify the current actual deviation, and the characters "3000" on the right side indicate the current distance to the runway;

[0113] 4. The 50mrad length of the deviation bar indicates a 100m deviation, 50m above and below; the intersection of the rectangular crossbar and the deviation bar indicates the current distance from the runway;

[0114] 5. The aircraft symbol moves up and down relative to the deviation bar, indicating the deviation in real time; when the aircraft coincides with the deviation bar, the deviation is 0.

[0115] 5) Display the symbol image on the head-up display to assist the pilot in landing the aircraft.

[0116] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A vision-based autonomous landing method and an indication screen design method, characterized in that: The autonomous landing method comprises the following steps: S101: The aircraft detects and tracks the four corner image information and horizon image information of the airport runway in real time based on the visual sensor; S102: Acquire internal reference information of the visual sensor; S103: Mapping the geographical coordinates of the current runway into pixel coordinates of the runway by combining the internal reference information and the four corner point image information; S104: Calculating the position and attitude information of the aircraft relative to the runway based on the geographic coordinates of the current runway, the identified pixel coordinates, and the mapping relationship between the geographic coordinates and the identified pixel coordinates; S105: Based on the position information of the aircraft relative to the runway, solve the lateral deviation and longitudinal deviation of the aircraft relative to the runway.

2. The vision-based autonomous landing method and instruction screen design method according to claim 1, characterized in that: The autonomous landing method further comprises: The accurate roll angle of the aircraft is obtained based on the pixel coordinates identified by the horizon image information to replace the roll angle in the attitude information.

3. The vision-based autonomous landing method and instruction screen design method according to claim 2, characterized in that: In S103, the method for establishing the coordinate system is specifically: World Coordinate System O w X w Y w Z w Fixed; origin O w It is the intersection of the left sideline of the runway and the end line of the runway in the distance. w The axis is in the same direction as the runway edge, X w Axis and Z w The axis lies in the plane of the runway and is perpendicular to the Z w Axis, pointing to the right of the runway, Y w The axis is perpendicular to the runway plane, and the coordinate system satisfies the right-hand rule; The image coordinate system is divided into two types: image pixel coordinate system (u, v) and image physical coordinate system (x, y), which are defined as: Image pixel coordinate system (u, v): The image pixel coordinate system is a rectangular coordinate system with the upper left corner of the image as the origin and pixels as the coordinate unit; u and v represent the column and row numbers of the pixel in the digital image respectively; Image physical coordinate system (x, y): The image physical coordinate system is a rectangular coordinate system with the intersection of the optical axis and the image plane as the origin and in millimeters; its x and y axes are parallel to the u and v axes of the image pixel coordinate system respectively and have the same direction; Camera coordinate system O c X c Y c Z c Take the origin O c is the optical center of the camera, Z c The X axis coincides with the optical axis of the camera, and the camera direction is taken as the positive direction. c , Y c The axis is parallel to the x and y axes of the image physical coordinate system and has the same direction; the camera coordinate system satisfies the right-hand rule.

4. The vision-based autonomous landing method and instruction screen design method according to claim 3, characterized in that: In S103, the mapping relationship is established by: Establish the mapping relationship between the runway's geographic coordinates and pixel coordinates as follows: K is the intrinsic parameter matrix of the camera, T is the coordinate of the origin of the world coordinate system in the camera coordinate system; The transformation matrix between the world coordinate system and the camera coordinate system.

5. The vision-based autonomous landing method and instruction screen design method according to claim 4, characterized in that: In S104, the method for calculating the position and attitude information of the aircraft relative to the runway is specifically as follows: The position and attitude information of the aircraft relative to the runway are calculated using the PNP algorithm based on the internal reference information of the visual sensor, the pixel coordinates of the four corner points of the runway, and the runway size information.

6. The vision-based autonomous landing method and instruction screen design method according to claim 5, characterized in that: The calculation method of the precise roll angle is: The precise roll angle of the aircraft is calculated using the two-point coordinates of the horizon end line based on image recognition.

7. The vision-based autonomous landing method and instruction screen design method according to claim 6, characterized in that: The method for solving the lateral deviation and longitudinal deviation of the aircraft relative to the runway is as follows: Calculate the lateral deviation and longitudinal deviation: the lateral deviation parameter Loc = Z*sin(θ), where Z is the distance from the projection of the aircraft on the runway plane to the runway entrance point, and the angle θ is the angle between the projection of the aircraft on the runway plane and the centerline of the runway; the longitudinal deviation parameter Gs = h-tan(ψ)*Z, where h is the vertical distance from the aircraft to the runway plane, and the angle ψ is the standard angle of the aircraft at the runway entrance point.

8. The vision-based autonomous landing method and instruction screen design method according to claim 7, characterized in that: The indication screen design method comprises: Set the pitch ladder symbol, horizontal line symbol, heading scale symbol, altitude symbol, longitudinal deviation symbol, distance and lateral deviation symbol; The symbols are used to display in real time: The position information of the aircraft relative to the runway, the replaced attitude information, and the lateral deviation and longitudinal deviation of the aircraft relative to the runway.

9. The vision-based autonomous landing method and instruction screen design method according to claim 8, characterized in that: The internal parameters include focal length, principal point coordinates, and distortion coefficient.

Images