An aircraft automatic take-off and landing pose perception method and system based on multi-sensor fusion

Abstract

The application discloses an automatic take-off and landing pose sensing method and system for an airplane based on multi-sensor fusion, which comprises the following steps: (1) sensing the stage of the airplane in real time according to the data collected by IMU, a camera and a GPS sensor, including a static stage, a taxiing take-off stage, an ascending stage, a cruising stage, a landing stage and a landing deceleration stage; (2) in the taxiing take-off stage and the landing deceleration stage, the runway centerline is detected by using Yolo, and the deflection angle and the offset distance of the airplane relative to the runway centerline are calculated; (3) in the ascending stage and the landing stage, the data of the IMU, the camera and the GPS sensor are fused, and the six-degree-of-freedom pose data corresponding to the position and the orientation of the airplane relative to the runway are calculated. The application uses the multi-sensor fusion technology to sense the stage of the airplane in real time and calculate the pose information, so that the calculation precision and the system robustness can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, and in particular to an aircraft automatic take-off and landing pose perception method and system based on multi-sensor fusion. BACKGROUND

[0002] With the continuous progress of aviation technology, the degree of automation of aircraft is gradually improved, and usually in the cruising stage, automatic driving can be realized with the help of GPS. However, in the process of taking off and landing, due to various complex situations that may occur on the runway, the aircraft pilot still needs to operate manually. In order to realize the automation of the process of taking off and landing, accurate pose perception is very important. The traditional aircraft perception system mainly relies on a single sensor for pose detection, such as Global Positioning System (GPS) or Inertial Measurement Unit (IMU). However, the reliability and accuracy of a single sensor in a complex environment are difficult to guarantee. For example, GPS signals may be affected by urban buildings or adverse weather conditions, and IMU may be affected by cumulative errors, resulting in deviation of pose estimation.

[0003] In order to solve these problems, multi-sensor fusion technology emerges as the times require. By combining the data of multiple sensors such as lidar, vision sensor and IMU, the accuracy and reliability of pose perception can be effectively improved. Multi-sensor fusion not only reduces the limitations of a single sensor, but also provides redundant data to enhance the fault tolerance of the system.

[0004] It is a hot research direction to perceive the environment using multi-sensor fusion. DEMO(Zhang J, Kaess M, Singh S. Real-time depth enhanced monocular odometry[C] 2014 IEEE / RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014:4973-4980.), V-LOAM(Zhang J, Singh S. Visual-lidar odometry and mapping: Low-drift, robust, and fast[C] 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015:2174-2181.) fuse camera and lidar data, use image information obtained by camera and structure information obtained by lidar to improve the accuracy of the system; MSCKF(Mourikis A I, Roumeliotis S I. A multi-state constraint Kalman filter for vision-aided inertial navigation[C] Proceedings 2007 IEEE International Conference on Robotics and Automation. IEEE, 2007:3565-3572.), VINS-Mono(Qin T, Li P, Shen S. Vins-mono: A robust and versatile monocular visual-inertial state estimator[J]. IEEE Transactions on Robotics, 2018, 34(4):1004-1020.) fuse camera and IMU, use IMU to provide scale information for the picture without scale information, so as to better predict the pose information; LOAM(Zhang J, Singh S. Visual-lidar odometry and mapping: Low-drift, robust, and fast[C] 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015:2174-2181.)), LIO-Mapping (Ye H, Chen Y, Liu M. Tightly Coupled 3D Lidar Inertial Odometry and Mapping[J]. arXiv preprint arXiv: 1904.06993, 2019.) fuses lidar and IMU; VINS-Fusion (Qin T, Cao S, Pan J, et al. A General Optimization-based Framework for Global Pose Estimation with Multiple Sensors[J]. arXiv preprint arXiv: 1901.03642, 2019.) proposes a fusion framework that can fuse IMU, camera, lidar, GPS and other multi-sensor data, so as to maintain high robustness in variable environments.

[0005] However, the above algorithms are mainly applied to ground autonomous driving scenes. Due to the significant difference between the motion characteristics of the aircraft and the ground vehicle, the data to be perceived in different stages of the aircraft operation is significantly different from that of the ground vehicle, and it is difficult to directly apply these algorithms to the aircraft to meet the complex motion conditions and precision requirements of the aircraft. SUMMARY

[0006] In view of the shortcomings of the prior art, the present application provides a kind of aircraft automatic take-off and landing pose perception method and system based on multi-sensor fusion, uses multi-sensor fusion technology to perceive the stage of aircraft in real time and calculates pose information, which can effectively improve the calculation precision and system robustness.

[0007] A kind of aircraft automatic take-off and landing pose perception method based on multi-sensor fusion, comprising the following steps:

[0008] (1) according to the data collected by IMU, camera and GPS sensor, the stage of aircraft is perceived in real time, including static stage, taxiing take-off stage, ascending stage, cruising stage, landing stage, landing deceleration stage;

[0009] (2) in taxiing take-off stage and landing deceleration stage, the center line of runway is detected using Yolo, and the deflection angle and offset distance of the aircraft relative to the center line of runway are calculated;

[0010] (3) in ascending stage and landing stage, fuse IMU, camera and GPS sensor data, calculate the six-degree-of-freedom pose data corresponding to the position and orientation of the aircraft relative to the runway.

[0011] Further, the specific process of step (1) is:

[0012] (1.1) The aircraft is in a static phase at the beginning, and the absolute value of the difference between the IMU reading and the gravity acceleration g is less than a set threshold T m , and the longitude and latitude data measured by the GPS should be near the start of the runway.

[0013] (1.2) In the static phase, it is detected that the IMU reading changes significantly, and the difference with the gravity acceleration g exceeds the set threshold T m , and the aircraft enters the taxiing takeoff phase; in the taxiing takeoff phase, the distance and height information of the aircraft at this time are calculated according to the IMU, camera and GPS sensor data, and after the aircraft height exceeds the set threshold T h1 , the aircraft enters the ascending phase; in the ascending phase, after the aircraft height exceeds the set threshold T h2 , the aircraft enters the cruising phase.

[0014] (1.3) In the cruising phase, it is detected that the distance between the GPS reading and the runway GPS coordinates is less than the set threshold T w , and the aircraft enters the landing phase.

[0015] (1.4) In the landing phase, the calculated height of the aircraft relative to the runway is less than the set threshold T h1 , and the aircraft enters the landing deceleration phase; in the deceleration phase, it is detected that the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m , and the aircraft enters the static phase.

[0016] The specific process of step (2) is as follows:

[0017] (2.1) Yolo is used to detect the pictures collected by the camera sensor to determine the pixel coordinates of the center line of the runway in the pictures;

[0018] (2.2) According to the pixel coordinates of the center line of the runway and the internal parameter data of the camera, the offset angle and offset distance of the aircraft relative to the center line of the runway are calculated;

[0019] (2.3) According to the IMU, camera and GPS sensors, the position and orientation of the aircraft are calculated, and the calculation results in (2.2) are fused to obtain more accurate calculation results.

[0020] The specific process of step (3) is as follows:

[0021] (3.1) The acceleration zero offset b a and the gyroscope zero offset b g; the IMU pose corresponding to the first frame of image taken by the camera as a reference system, pre-integrating and integrating the IMU data to preliminarily obtain the position p, orientation R and velocity v of each frame of image in the local coordinate system; extracting feature points from the images collected by the camera and tracking the feature points between adjacent frames of image, combining the IMU data, and using a triangulation method to preliminarily calculate the inverse depth λ of each feature point; and using the preliminary calculation results p, R, v, λ, b an ,b gn As the optimization variable χ, the actual observation value z obtained by the IMU, the camera and the GPS and the corresponding theoretical calculation value h(χ) are used to construct a constraint term to optimize χ, that is, to obtain the accurate position and orientation of the aircraft in the local coordinate system corresponding to the six-degree-of-freedom pose data.

[0022] (3.2) According to the position information of the aircraft in the local coordinate system calculated in step (3.1) and the position information of the aircraft in the global coordinate system calculated according to the GPS data, the conversion relationship between the local coordinate system and the global coordinate system with the target landing point as the origin is calculated.

[0023] (3.3) According to the aircraft pose data corresponding to the latest frame calculated in step (3.1), the data read by the IMU sensor is integrated to obtain the real-time pose data of the aircraft in the local coordinate system; and according to the conversion relationship between the local coordinate system and the global coordinate system calculated in step (3.2), the pose data in the local coordinate system is converted to the global coordinate system.

[0024] In step (3.1), the actual observation value z obtained by the IMU, the camera and the GPS and the corresponding theoretical calculation value h(χ) are used to construct a constraint term to optimize χ, specifically:

[0025]

[0026] In the formula, z i and h i (χ) represent the actual observation value and the corresponding theoretical calculation value of the i-th frame of image, respectively.

[0027] A kind of aircraft automatic take-off and landing pose perception system based on multi-sensor fusion, comprising a memory and one or more processors, the memory has executable code stored therein, the one or more processors execute the executable code, to implement the above-mentioned aircraft automatic take-off and landing pose perception method.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] The application uses a multi-sensor fusion technology to perceive a flight stage of an aircraft in real time and calculate pose information, which can effectively improve calculation accuracy and system robustness, so that the system can effectively operate in a complex and changeable environment and continue to operate in the case of partial sensor failure. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A flow chart of an automatic take-off and landing pose perception method of an aircraft based on multi-sensor fusion is provided for an embodiment of the application.

[0031] Figure 2 A Yolo network structure used in the application is provided.

[0032] Figure 3 A flow chart of aircraft pose calculation in the take-off and landing stages is provided. DETAILED DESCRIPTION

[0033] The application will be further described in detail below in combination with the drawings and embodiments, and it should be noted that the following embodiments are intended to facilitate the understanding of the application and do not limit the application in any way.

[0034] The core of the application is to use a multi-sensor fusion technology to perceive a flight stage of an aircraft in real time and calculate aircraft pose data, which is used by a subsequent control module to achieve automatic driving of the aircraft during the take-off and landing process.

[0035] As shown in Figure 1 , an automatic take-off and landing pose perception method of an aircraft based on multi-sensor fusion includes the following steps:

[0036] (1) Real-time perception of the stage of the aircraft using data collected by sensors, including the static stage, the taxiing take-off stage, the take-off stage, the cruising stage, the landing stage, and the landing deceleration stage. The transition relationship between stages and the operation required in each stage are shown in Figure 1 .

[0037] (1.1) Determine the state of the aircraft according to the data of IMU, GPS, and camera sensors. The aircraft is initially in the static stage, at which time the absolute value of the difference between the IMU reading and the gravitational acceleration g is less than a set threshold T m , and the latitude and longitude data measured by the GPS should be located near the start of the runway.

[0038] (1.2) In the static stage, after detecting that the IMU reading changes significantly and the difference from the gravitational acceleration g exceeds the set threshold T m , the aircraft enters the taxiing take-off stage. In the take-off stage, according to the IMU, image, and GPS data, the distance and height information of the aircraft at this time are calculated. In the take-off stage, when the height of the aircraft exceeds a set threshold T h1After that, the aircraft enters the take-off phase. In the take-off phase, the aircraft height exceeds a set threshold T h2 After that, the aircraft enters the cruising phase.

[0039] (1.3) In the cruising phase, it is detected that the distance between the GPS reading and the runway GPS coordinates is less than a set threshold T w After that, the aircraft enters the landing phase. In the landing phase, the IMU, camera and GPS sensor data are fused to calculate the aircraft's deviation angle and position relative to the runway in each dimension.

[0040] (1.4) In the landing phase, the calculated height of the aircraft relative to the runway is less than a set threshold T h1 After that, the aircraft enters the post-landing deceleration phase. In the deceleration phase, it is detected that the difference between the IMU reading and the gravitational acceleration g is less than a set threshold T m After that, the aircraft enters the stationary phase.

[0041] (2) In the take-off phase and the landing deceleration phase, Yolo is used to detect the runway centerline and calculate the aircraft's deviation angle and offset distance relative to the runway centerline.

[0042] (2.1) Yolo is used to process the pictures collected by the camera sensor to identify the runway corner points and the runway centerline position. The Yolo network structure used is as shown in Figure 2 .

[0043] For an input picture I∈R H×W×3 , the network finally outputs the result as , where the first 5 dimensions represent the confidence, horizontal and vertical coordinate offset, and length and width offset of the detection point, and the last 6 dimensions represent which category (corresponding to the four corner points of the runway and the two endpoints of the centerline, respectively).

[0044] For the four corner points of the runway, from the output results of the Yolo model, the targets with the category label of the four corner points of the runway are filtered out, and for each corner point, the k points with the highest confidence are taken as candidate points to form k 4 corner point combinations. The combinations that are not convex quadrilaterals are filtered out, and the combination with the largest IoU with the previous frame result is selected as the final prediction of the runway corner points.

[0045] For the runway centerline endpoints, from the output results of the Yolo model, the targets with the category label of the runway centerline endpoints are filtered out, and the point with the highest confidence is selected as the prediction result.

[0046] (2.2) According to the pixel coordinates of the two endpoints of the runway centerline and the camera's intrinsic data, the offset angle and offset distance of the aircraft relative to the runway centerline are calculated.

[0047] The pixel coordinate system is defined with the upper left corner of the image as the origin, the right as the u axis, and the downward as the v axis. The camera coordinate system is defined with the camera as the origin, the vertical imaging plane forward as the z axis, the right as the x axis, and the downward as the y axis. The conversion relationship between the two coordinate systems is as follows:

[0048]

[0049] In the above formula, (x, y, z) is the spatial coordinate of the point in the camera coordinate system, (u, v) is the pixel coordinate of the point, and f x ,f y ,c x ,c y is the camera internal parameter.

[0050] When the camera imaging plane is perpendicular to the ground, for a point on the ground, if its pixel coordinates (u, v) and the camera height h are known, the coordinates of the point on the ground in the camera coordinate system can be calculated:

[0051]

[0052] For each frame of image obtained by the camera sensor, the center line of the runway is detected using the method in (2.1). The two three-point points of the center line are taken, and their pixel coordinates are recorded as (u1, v1) and (u2, v2). The coordinates in the camera coordinate system are calculated using the above formula and recorded as P1(x1, h, z1) and P2(x2, h, z2).

[0053] Assuming the aircraft's leftward deviation angle is positive, the deviation angle α is

[0054]

[0055] Let (0,h,0) be point O, and the distance the camera deviates from the center line is

[0056]

[0057] To determine the orientation of the camera relative to the center line, the point closest to the camera in P1 and P2 is called P n , only keep P n The coordinates on the xz plane are denoted as (X n ,Z n ), similarly, let the point farthest from the camera be P f (X f ,Z f ), let the origin of the xz plane (i.e. the camera position) be O.

[0058]

[0059] If c is a positive value, the camera is to the right of the center line; if c is a negative value, the camera is to the left of the center line.

[0060] (2.3) According to the IMU, image, GPS data, the current position and orientation of the aircraft are calculated, and the calculation results in (2.2) are fused to obtain more accurate calculation results.

[0061] (3) In the take-off and landing stage, the IMU, camera, GPS and other sensor data are fused to calculate the six-degree-of-freedom pose information corresponding to the position and orientation of the aircraft relative to the runway, and the overall calculation process is as shown in Figure 3 .

[0062] (3.1) The acceleration zero offset b a and the gyroscope zero offset b g of the IMU are measured in advance; the IMU data is pre-integrated and integrated with the IMU pose corresponding to the first frame of image as the reference system, to preliminarily obtain the position p, orientation R and velocity v of the aircraft at each time in the local coordinate system; the feature points are extracted from the images collected by the camera, and the feature point tracking is performed on the adjacent frames of images, combined with the IMU data, and the inverse depth λ of each feature point is preliminarily calculated using the triangulation method.

[0063] To obtain the accurate pose data of the aircraft, the preliminary calculation results p, R, v, λ, b gn are used as optimization variables, and the constraint terms are constructed according to the IMU, camera and GPS data to optimize these variables. To improve the calculation efficiency, when new image data arrives, the sliding window method is used to marginalize the older data, and the image feature point matching results and the pre-integration results during the last ten frames are retained to optimize the above optimization variables, and the formula is as follows:

[0064]

[0065] In the above formula, χ represents all optimization variables, and {r p ,Η p} represents the prior information from the last marginalization, and represents the IMU residual and visual residual terms, is the global positioning residual, wherein and are observation values, and are covariance matrices. Solving the above optimization problem can obtain the accurate real-time six-degree-of-freedom pose in the local reference system.

[0066] (3.2) According to the position information x l of the aircraft in the local coordinate system calculated in (3.1) and the position information xg , the conversion relationship between the local coordinate system and the global coordinate system with the target landing point as the origin is calculated, and the calculation formula is as follows:

[0067]

[0068] With the continuous calculation of new x l and x g data, iterative optimization is continuously performed according to the above formula, and a more accurate conversion relationship T * between the local coordinate system and the global coordinate system can be obtained.

[0069] (3.3) According to the latest frame corresponding to the aircraft pose data calculated in (3.1), the data read by the IMU sensor is integrated to obtain the real-time pose data of the aircraft in the local coordinate system. According to the conversion relationship between the local coordinate system and the global coordinate system calculated in (3.2), the real-time pose data in the local coordinate system is transformed to the global coordinate system. When the camera sensor reads new image data, the frames in the sliding window in (3.1) are marginalized and then the least squares problem in (3.1) is solved again to obtain the accurate latest frame pose.

[0070] The integral calculation of the IMU is performed using the following formula:

[0071]

[0072] In the above formula, R, v, and p represent orientation, velocity, and position, respectively, and b g , b a are the readings of the gyroscope and accelerometer in the IMU, respectively, and b gs , b ad are the zero offsets of the gyroscope and accelerometer in the IMU, respectively, and g is the gravity vector. According to the above formula, the aircraft pose can be calculated in real time.

[0073] The above-described embodiments have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above-described embodiments are only specific embodiments of the present application and are not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A method for sensing the posture of an aircraft for automatic takeoff and landing based on multi-sensor fusion, characterized in that: The steps include: (1) Real-time perception of the aircraft's current phase based on data collected by the IMU, camera, and GPS sensors, including the stationary phase, taxiing takeoff phase, liftoff phase, cruising phase, landing phase, and landing deceleration phase; (2) During the taxiing takeoff phase and landing deceleration phase, Yolo is used to detect the runway centerline and calculate the aircraft's deviation angle and offset distance relative to the runway centerline; (3) During the takeoff and landing phases, the IMU, camera, and GPS sensor data are integrated to calculate the six-degree-of-freedom pose data corresponding to the aircraft's position and orientation relative to the runway; The specific process is: (3.1) Pre-measure the IMU acceleration bias b a and gyroscope bias b g Using the IMU pose corresponding to the first frame captured by the camera as the reference system, pre-integrate and integrate the IMU data to initially obtain the position p, orientation R, and velocity v in the local coordinate system corresponding to each frame. Extract feature points from the image captured by the camera and track feature points in adjacent frames. Combined with the IMU data, use the triangulation method to preliminarily calculate the inverse depth λ of each feature point. The results of the preliminary calculation are p,R,v,λ, As the variable to be optimized, χ is used as a constraint term based on the actual observation value z obtained by the IMU, camera, and GPS and the corresponding theoretical calculated value h(χ) to optimize χ, that is, to obtain the six-degree-of-freedom pose data corresponding to the accurate position and orientation of the aircraft in the local coordinate system; (3.2) Based on the position information of the aircraft in the local coordinate system calculated in step (3.1) and the position information of the aircraft in the global coordinate system calculated based on the GPS data, calculate the transformation relationship between the local coordinate system and the global coordinate system with the target landing point as the origin; (3.3) Based on the aircraft posture data corresponding to the latest frame calculated in step (3.1), the data read by the IMU sensor is integrated to obtain the real-time posture data of the aircraft in the local coordinate system; and based on the transformation relationship between the local coordinate system and the global coordinate system calculated in step (3.2), the posture data in the local coordinate system is transformed to the global coordinate system.

2. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: The specific process of step (1) is: (1.1) The aircraft is in a stationary state at the beginning, and the absolute value of the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m , the longitude and latitude data measured by GPS should be near the start of the runway; (1.2) During the static phase, a significant change in the IMU reading is detected, and the difference from the gravity acceleration g exceeds the set threshold T m After that, the aircraft enters the taxiing takeoff phase. During the taxiing takeoff phase, the distance and altitude of the aircraft are calculated based on the data from the IMU, camera, and GPS sensors. When the aircraft altitude exceeds the set threshold T h1 After that, the aircraft enters the lift-off phase; during the lift-off phase, when the aircraft altitude exceeds the set threshold T h2 After that, the aircraft enters the cruising phase; (1.3) During the cruise phase, the distance between the GPS reading and the runway GPS coordinates is detected to be less than the set threshold T w After that, the aircraft enters the landing phase; (1.4) During the landing phase, the calculated height of the aircraft relative to the runway is less than the set threshold T h1 After that, the aircraft enters the landing deceleration phase; during the deceleration phase, the difference between the IMU reading and the gravity acceleration g is detected to be less than the set threshold T m After that, the aircraft entered a stationary phase.

3. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: The specific process of step (2) is: (2.1) Use Yolo to detect the image captured by the camera sensor and determine the pixel coordinates of the runway centerline in the image; (2.2) Calculate the aircraft's offset angle and offset distance relative to the runway centerline based on the pixel coordinates of the runway centerline and the camera's intrinsic parameter data; (2.3) Calculate the position and orientation of the aircraft based on the IMU, camera, and GPS sensor, and fuse them with the results calculated in (2.2) to obtain more accurate calculation results.

4. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: In step (3.1), based on the actual observation value z obtained by IMU, camera, and GPS and the corresponding theoretical calculated value h(χ), a constraint term is constructed to optimize χ, specifically: Where z i and h i (χ) represents the actual observed value and the corresponding theoretical calculated value of the i-th frame image, respectively.

5. An aircraft automatic take-off and landing posture perception system based on multi-sensor fusion, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the aircraft automatic take-off and landing posture perception method according to any one of claims 1 to 4.

Images