Airplane automatic take-off and landing pose sensing method and system based on multi-sensor fusion

Abstract

The invention discloses an aircraft automatic take-off and landing pose sensing method and system based on multi-sensor fusion, and the method comprises the steps: (1) sensing the stages of an aircraft in real time according to the data collected by an IMU, a camera and a GPS sensor, including a static stage, a taxiing take-off stage, a lift-off stage, a cruising stage, a landing stage and a landing deceleration stage; (2) in a taxiing takeoff stage and a landing deceleration stage, using Yolo to detect a runway center line and calculating a deflection angle and an offset distance of the aircraft relative to the runway center line; and (3) in the lift-off stage and the landing stage, data of the IMU, the camera and the GPS sensor are fused, and six-degree-of-freedom pose data corresponding to the position and orientation of the aircraft relative to the runway are calculated. According to the method, the multi-sensor fusion technology is used for sensing the stage of the aircraft in real time and calculating the pose information, and the calculation precision and the system robustness can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method and system for sensing the posture of an aircraft automatically taking off and landing based on multi-sensor fusion. Background Art

[0002] With the continuous advancement of aviation technology, the degree of automation of aircraft has gradually improved. Usually, automatic driving can be achieved with the help of GPS during the cruising phase. However, during take-off and landing, due to various complex situations that may arise on the runway, manual operation by the pilot is still required. To automate the take-off and landing process, accurate posture sense is very important. Traditional aircraft perception systems mainly rely on a single sensor for posture detection, such as the global positioning system (GPS) or inertial measurement unit (IMU). However, the reliability and accuracy of a single sensor in a complex environment are difficult to guarantee. For example, GPS signals may be affected by urban buildings or severe weather conditions, while IMUs may be affected by cumulative errors, resulting in deviations in posture estimation.

[0003] In order to solve these problems, multi-sensor fusion technology came into being. By combining the data of multiple sensors, such as lidar, visual sensors, and IMU, the accuracy and reliability of posture perception can be effectively improved. Multi-sensor fusion can not only reduce the limitations of a single sensor, but also provide redundant data and enhance the fault tolerance of the system.

[0004] Using multi-sensor fusion to perceive the environment is a popular research direction. Among them, DEMO (Zhang J, Kaess M, Singh S. Real-time depth enhanced monocular odometry [C] 2014 IEEE / RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014: 4973-4980.) and V-LOAM (Zhang J, Singh S. Visual-lidar odometry and mapping: Low-drift, robust, and fast [C] 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015: 2174-2181.) fuse camera and lidar data to improve the accuracy of the system using the image information obtained by the camera and the structural information obtained by the lidar; MSCKF (Mourikis AI, Roumeliotis S IA multi-state constraint Kalman filter for vision-aided inertial navigation [C] Proceedings 2007 IEEE International Conference on Robotics and Automation) Automation.IEEE, 2007:3565-3572.), VINS-Mono (Qin T, Li P, Shen S. Vins-mono: A robust and versatile monocular visual-inertial state estimator [J]. IEEE Transactions on Robotics, 2018, 34 (4): 1004-1020.) fuses the camera and IMU, and uses the IMU to provide scale information for images without scale information, so as to better predict the pose information; LOAM (Zhang J, Singh S. Visual-lidar odometry and mapping: Low-drift, robust, and fast [C] 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015: 2174-2181.), LIO-Mapping (Ye H, Chen Y, Liu M. Tightly Coupled 3D Lidar Inertial Odometry and Mapping [J]. arXiv preprint arXiv: 1904.06993, 2019.) fuses LiDAR and IMU; VINS-Fusion (Qin T, Cao S, Pan J, et al. A General Optimization-based Framework for Global Pose Estimation with Multiple Sensors [J]. arXiv preprint arXiv: 1901.03642, 2019.) proposes a fusion framework that can fuse multiple sensor data such as IMU, camera, LiDAR, GPS, etc., so as to maintain high robustness in a changing environment. .

[0005] However, the above algorithms are mainly used in ground-based autonomous driving scenarios. Since the motion characteristics of aircraft are significantly different from those of ground vehicles, the data that needs to be sensed at different stages of aircraft operation is significantly different from that of ground vehicles. Directly applying these algorithms to aircraft is difficult to meet the complex motion conditions and accuracy requirements of aircraft. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method and system for sensing the posture of an aircraft for automatic take-off and landing based on multi-sensor fusion. The multi-sensor fusion technology is used to sense the stage of the aircraft in real time and calculate the posture information, which can effectively improve the calculation accuracy and system robustness.

[0007] A method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion comprises the following steps:

[0008] (1) Based on the data collected by the IMU, camera, and GPS sensors, the aircraft can be perceived in real time at the current stage, including the stationary stage, taxiing and takeoff stage, lift-off stage, cruising stage, landing stage, and landing deceleration stage;

[0009] (2) During the taxiing takeoff phase and landing deceleration phase, Yolo is used to detect the runway centerline and calculate the aircraft's deviation angle and offset distance relative to the runway centerline;

[0010] (3) During the launch and landing phases, the IMU, camera, and GPS sensor data are fused to calculate the six-degree-of-freedom posture data corresponding to the aircraft's position and orientation relative to the runway.

[0011] Furthermore, the specific process of step (1) is as follows:

[0012] (1.1) The aircraft is in a stationary stage at the beginning, and the absolute value of the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m , the longitude and latitude data measured by GPS should be near the start of the runway;

[0013] (1.2) During the static phase, a significant change in the IMU reading is detected, and the difference with the gravity acceleration g exceeds the set threshold T m After that, the aircraft enters the taxiing takeoff phase. During the taxiing takeoff phase, the aircraft’s distance and altitude information are calculated based on the IMU, camera, and GPS sensor data. When the aircraft altitude exceeds the set threshold T h1 After that, the aircraft enters the lift-off phase. During the lift-off phase, when the aircraft altitude exceeds the set threshold T h2 After that, the aircraft enters the cruising phase;

[0014] (1.3) During the cruise phase, it is detected that the distance between the GPS reading and the runway GPS coordinates is less than the set threshold T w After that, the aircraft enters the landing phase;

[0015] (1.4) During the landing phase, the calculated height of the aircraft relative to the runway is less than the set threshold T h1 After that, the aircraft enters the landing deceleration stage; during the deceleration stage, it is detected that the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m After that, the aircraft entered a stationary phase.

[0016] The specific process of step (2) is as follows:

[0017] (2.1) Use Yolo to detect the images captured by the camera sensor and determine the pixel coordinates of the runway centerline in the image;

[0018] (2.2) Calculate the offset angle and offset distance of the aircraft relative to the runway centerline based on the pixel coordinates of the runway centerline and the camera's internal reference data;

[0019] (2.3) Based on the IMU, camera and GPS sensor, the position and orientation of the aircraft are calculated and fused with the calculation results in (2.2) to obtain more accurate calculation results.

[0020] The specific process of step (3) is as follows:

[0021] (3.1) Pre-measure the IMU acceleration bias b a and gyro bias b g; Take the IMU pose corresponding to the first frame of the camera as the reference system, perform pre-integration and integration operations on the IMU data, and preliminarily obtain the position p, orientation R and velocity v in the local coordinate system corresponding to each frame of the image; extract feature points from the image captured by the camera and track feature points in adjacent frames, combine the IMU data, and use the triangulation method to preliminarily calculate the inverse depth λ of each feature point; the preliminarily calculated results p, R, v, λ, b an ,b gn As the variable to be optimized, χ is used to construct constraint terms based on the actual observation value z obtained by IMU, camera, and GPS and the corresponding theoretical calculated value h(χ) to optimize χ, that is, to obtain the six-degree-of-freedom posture data corresponding to the accurate position and orientation of the aircraft in the local coordinate system;

[0022] (3.2) Calculating the transformation relationship between the local coordinate system and the global coordinate system with the target landing point as the origin based on the position information of the aircraft in the local coordinate system calculated in step (3.1) and the position information of the aircraft in the global coordinate system calculated based on the GPS data;

[0023] (3.3) Based on the aircraft posture data corresponding to the latest frame calculated in step (3.1), the data read by the IMU sensor is integrated to obtain the real-time posture data of the aircraft in the local coordinate system; and based on the transformation relationship between the local coordinate system and the global coordinate system calculated in step (3.2), the posture data in the local coordinate system is transformed to the global coordinate system.

[0024] In step (3.1), based on the actual observation value z obtained by IMU, camera, and GPS and the corresponding theoretical calculated value h(χ), a constraint term is constructed to optimize χ, specifically:

[0025]

[0026] In the formula, z i and h i (χ) represent the actual observed value and the corresponding theoretical calculated value of the i-th frame image respectively.

[0027] A multi-sensor fusion-based aircraft automatic take-off and landing posture perception system comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the above-mentioned aircraft automatic take-off and landing posture perception method.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention uses multi-sensor fusion technology to sense the stage of the aircraft in real time and calculate the position and posture information, which can effectively improve the calculation accuracy and system robustness, so that the system can operate effectively in a complex and changeable environment and continue to operate when some sensors fail. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to an embodiment of the present invention;

[0031] Figure 2 The Yolo network structure used in the present invention;

[0032] Figure 3 The present invention provides a flow chart for calculating the aircraft posture during the lift-off and landing phases. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0034] The core of the present invention is to use multi-sensor fusion technology to perceive the flight phase of the aircraft in real time and calculate the aircraft posture data for use by subsequent control modules to achieve automatic driving during the aircraft take-off and landing process.

[0035] like Figure 1 As shown, a method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion comprises the following steps:

[0036] (1) Use the data collected by sensors to perceive the aircraft's current stage in real time, including the stationary stage, taxiing takeoff stage, lift-off stage, cruising stage, landing stage, and landing deceleration stage. The transition relationship between stages and the operations required in each stage are as follows: Figure 1 shown.

[0037] (1.1) Determine the aircraft's state based on the data from the IMU, GPS, and camera sensors. The aircraft is initially stationary, and the absolute value of the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m , the longitude and latitude data measured by GPS should be near the start of the runway.

[0038] (1.2) During the static phase, a significant change in the IMU reading is detected, and the difference with the gravity acceleration g exceeds the set threshold T m After that, the aircraft enters the taxiing takeoff phase. During the takeoff phase, the aircraft’s distance and altitude are calculated based on IMU, image, and GPS data. h1After that, the aircraft enters the lift-off phase. During the lift-off phase, when the aircraft altitude exceeds the set threshold T h2 After that, the aircraft entered the cruising phase.

[0039] (1.3) During the cruise phase, it is detected that the distance between the GPS reading and the runway GPS coordinates is less than the set threshold T w After that, the aircraft enters the landing phase. During the landing phase, the IMU, camera and GPS sensor data are fused to calculate the aircraft's deflection angle and position in various dimensions relative to the runway.

[0040] (1.4) During the landing phase, the calculated height of the aircraft relative to the runway is less than the set threshold T h1 After that, the aircraft enters the deceleration phase after landing. During the deceleration phase, the difference between the IMU reading and the gravity acceleration g is detected to be less than the set threshold T m After that, the aircraft entered a stationary phase.

[0041] (2) During the takeoff and landing deceleration phases, Yolo is used to detect the runway centerline and calculate the aircraft's deviation angle and offset distance relative to the runway centerline.

[0042] (2.1) Use Yolo to process the images collected by the camera sensor and identify the runway corner points and runway centerline positions. The Yolo network structure used is as follows: Figure 2 shown.

[0043] For the input image I∈R H×W×3 The final output of the network is Among them, the first 5 dimensions represent the confidence of the detection point, the offset corresponding to the horizontal and vertical coordinates, and the offset corresponding to the length and width, and the last 6 dimensions represent which category it belongs to (corresponding to the four corner points of the runway and the two endpoints of the center line).

[0044] For the four corner points of the runway, the targets with the categories marked as the four corner points of the runway are selected from the output results of the Yolo model. For each corner point, the k points with the highest confidence are selected as candidate points to form a k-point network. 4 Corner point combinations. Filter out the combinations that are not convex quadrilaterals, and select the combination with the largest IoU with the previous frame result from the remaining combinations as the prediction of the final runway corner point.

[0045] For the runway centerline endpoint, from the output results of the Yolo model, filter out the targets with the category labeled as the runway centerline endpoint, and select the point with the highest confidence as the prediction result.

[0046] (2.2) Based on the pixel coordinates of the two endpoints of the runway centerline and the camera's internal parameter data, the aircraft's offset angle and offset distance relative to the runway centerline are calculated.

[0047] The pixel coordinate system is defined with the upper left corner of the image as the origin, the right as the u axis, and the downward as the v axis. The camera coordinate system is defined with the camera as the origin, the vertical imaging plane forward as the z axis, the right as the x axis, and the downward as the y axis. The conversion relationship between the two coordinate systems is as follows:

[0048]

[0049] In the above formula, (x, y, z) is the spatial coordinate of the point in the camera coordinate system, (u, v) is the pixel coordinate of the point, and f x ,f y ,c x ,c y is the camera internal parameter.

[0050] When the camera imaging plane is perpendicular to the ground, for a point on the ground, if its pixel coordinates (u, v) and the camera height h are known, the coordinates of the point on the ground in the camera coordinate system can be calculated:

[0051]

[0052] For each frame image obtained by the camera sensor, the center line of the runway is detected using the method in (2.1). The two three-point points of the center line are taken, and their pixel coordinates are recorded as (u1, v1) and (u2, v2). The coordinates in the camera coordinate system are calculated using the above formula, recorded as P1(x1, h, z1) and P2(x2, h, z2).

[0053] Assuming that the aircraft's leftward deviation angle is positive, the deviation angle α is

[0054]

[0055] Let (0,h,0) be point O, and the distance the camera deviates from the center line is

[0056]

[0057] To determine the orientation of the camera relative to the midline, the point closest to the camera in P1 and P2 is denoted as P n , only keep P n The coordinates on the xz plane are denoted as (X n ,Z n ), similarly, let the point farthest from the camera be P f (X f ,Z f ), let the origin of the xz plane (i.e. the camera position) be O.

[0058]

[0059] If c is a positive value, the camera is to the right of the center line; if c is a negative value, the camera is to the left of the center line.

[0060] (2.3) Calculate the current position and orientation of the aircraft based on the IMU, image, and GPS data, and fuse them with the calculation results in (2.2) to obtain a more accurate calculation result.

[0061] (3) During the takeoff and landing phases, the IMU, camera, GPS and other sensor data are integrated to calculate the six-degree-of-freedom posture information corresponding to the position and orientation of the aircraft relative to the runway. The overall calculation process is as follows: Figure 3 shown.

[0062] (3.1) Pre-measure the IMU acceleration bias b a and gyro bias b g ; Taking the IMU pose corresponding to the first frame of image as the reference system, pre-integrate and integrate the IMU data to preliminarily obtain the position p, orientation R and speed v of the aircraft in the local coordinate system at each moment; extract feature points from the image captured by the camera and track feature points in adjacent frame images. Combined with the IMU data, use the triangulation method to preliminarily calculate the inverse depth λ of each feature point.

[0063] In order to obtain accurate posture data of the aircraft, the results p, R, v, λ, b gn As variables to be optimized, constraints are constructed based on IMU, camera, and GPS data, and these variables are optimized. To improve computational efficiency, each time new image data arrives, a sliding window method is used to marginalize older data, retain the image feature point matching results of ten frames and the pre-integration results during the period, and optimize the above variables to be optimized. The formula is as follows:

[0064]

[0065] In the above formula, χ represents all variables to be optimized, and {r p , Η p} represents the prior information from the last marginalization, and Represents the IMU residual and visual residual terms, is the global positioning residual, where and refers to the observed value, and Refers to the covariance matrix. Solving the above optimization problem can obtain the accurate and real-time six-degree-of-freedom pose in the local reference frame.

[0066] (3.2) According to the position information x of the aircraft in the local coordinate system calculated in (3.1), l and the position information x of the aircraft in the global coordinate system calculated based on GPS datag , calculate the transformation relationship between the local coordinate system and the global coordinate system with the target landing point as the origin. The calculation formula is as follows:

[0067]

[0068] As we continue to calculate new x l and x g Data, according to the above formula, continuous iterative optimization can obtain a more accurate transformation relationship between the local coordinate system and the global coordinate system T * .

[0069] (3.3) According to the aircraft pose data corresponding to the latest frame calculated in (3.1), the data read by the IMU sensor is integrated to obtain the real-time pose data of the aircraft in the local coordinate system. According to the transformation relationship between the local coordinate system and the global coordinate system calculated in (3.2), the real-time pose data in the local coordinate system is transformed to the global coordinate system. When the camera sensor reads new image data, the frames in the sliding window in (3.1) are marginalized and the least squares problem in (3.1) is solved again to obtain the accurate latest frame pose.

[0070] Use the following formula to calculate the IMU integration:

[0071]

[0072] In the above formula, R, v, and p represent direction, speed, and position, respectively. are the readings of the gyroscope and accelerometer in the IMU, respectively, g 、b a are the zero bias corresponding to the gyroscope and accelerometer in the IMU, η gs , η ad are the Gaussian white noises corresponding to the gyroscope and accelerometer in the IMU, and g is the gravity vector. The aircraft posture can be calculated in real time according to the above formula.

[0073] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion, characterized in that: The steps include: (1) Based on the data collected by the IMU, camera, and GPS sensors, the aircraft can be perceived in real time at the current stage, including the stationary stage, taxiing takeoff stage, lift-off stage, cruising stage, landing stage, and landing deceleration stage; (2) During the taxiing takeoff phase and landing deceleration phase, Yolo is used to detect the runway centerline and calculate the aircraft's deviation angle and offset distance relative to the runway centerline; (3) During the launch and landing phases, the IMU, camera, and GPS sensor data are fused to calculate the six-degree-of-freedom posture data corresponding to the aircraft's position and orientation relative to the runway.

2. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: The specific process of step (1) is as follows: (1.1) The aircraft is in a stationary stage at the beginning, and the absolute value of the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m , the longitude and latitude data measured by GPS should be near the start of the runway; (1.2) During the static phase, a significant change in the IMU reading is detected, and the difference with the gravity acceleration g exceeds the set threshold T m After that, the aircraft enters the taxiing takeoff phase. During the taxiing takeoff phase, the aircraft’s distance and altitude information are calculated based on the IMU, camera, and GPS sensor data. When the aircraft altitude exceeds the set threshold T h1 After that, the aircraft enters the lift-off phase. During the lift-off phase, when the aircraft altitude exceeds the set threshold T h2 After that, the aircraft enters the cruising phase; (1.3) During the cruise phase, it is detected that the distance between the GPS reading and the runway GPS coordinates is less than the set threshold T w After that, the aircraft enters the landing phase; (1.4) During the landing phase, the calculated height of the aircraft relative to the runway is less than the set threshold T h1 After that, the aircraft enters the landing deceleration stage; during the deceleration stage, it is detected that the difference between the IMU reading and the gravity acceleration g is less than the set threshold T m After that, the aircraft entered a stationary phase.

3. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: The specific process of step (2) is as follows: (2.1) Use Yolo to detect the images captured by the camera sensor and determine the pixel coordinates of the runway centerline in the image; (2.2) Calculate the offset angle and offset distance of the aircraft relative to the runway centerline based on the pixel coordinates of the runway centerline and the camera's internal reference data; (2.3) Based on the IMU, camera and GPS sensor, the position and orientation of the aircraft are calculated and fused with the calculation results in (2.2) to obtain more accurate calculation results.

4. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: The specific process of step (3) is as follows: (3.1) Pre-measure the IMU acceleration bias b a and gyro bias b g ; Using the IMU pose corresponding to the first frame of the camera as the reference system, pre-integrate and integrate the IMU data to preliminarily obtain the position p, orientation R and velocity v in the local coordinate system corresponding to each frame of the image; extract feature points from the image captured by the camera and track feature points in adjacent frames of the image. Combined with the IMU data, use the triangulation method to preliminarily calculate the inverse depth λ of each feature point; The preliminary calculated results p,R,v,λ, As the variable to be optimized, χ is used to construct constraint terms based on the actual observation value z obtained by IMU, camera, and GPS and the corresponding theoretical calculated value h(χ) to optimize χ, that is, to obtain the six-degree-of-freedom posture data corresponding to the accurate position and orientation of the aircraft in the local coordinate system; (3.2) Calculating the transformation relationship between the local coordinate system and the global coordinate system with the target landing point as the origin based on the position information of the aircraft in the local coordinate system calculated in step (3.1) and the position information of the aircraft in the global coordinate system calculated based on the GPS data; (3.3) Based on the aircraft posture data corresponding to the latest frame calculated in step (3.1), the data read by the IMU sensor is integrated to obtain the real-time posture data of the aircraft in the local coordinate system; and based on the transformation relationship between the local coordinate system and the global coordinate system calculated in step (3.2), the posture data in the local coordinate system is transformed to the global coordinate system.

5. The method for sensing the posture of an aircraft taking off and landing automatically based on multi-sensor fusion according to claim 1, characterized in that: In step (3.1), based on the actual observation value z obtained by IMU, camera, and GPS and the corresponding theoretical calculated value h(χ), a constraint term is constructed to optimize χ, specifically: In the formula, z i and h i (χ) represent the actual observed value and the corresponding theoretical calculated value of the i-th frame image respectively.

6. An aircraft automatic take-off and landing posture perception system based on multi-sensor fusion, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the aircraft automatic take-off and landing posture perception method according to any one of claims 1 to 5.

Images