Automatic parking system for aircraft

Abstract

An automatic parking system for an aircraft comprises a multi-sensing system, a target detecting and tracking module, an information fusion center and an automatic auxiliary suite, and the multi-sensing system collects image information and point cloud data of the aircraft to be parked through a camera, a laser radar and a millimeter wave radar; the target detection and tracking module carries out target identification and detection and target tracking and cognitive processing in sequence according to the image information and the point cloud data to obtain the state information of the aircraft to be parked and the model of the aircraft, and the information fusion center carries out data fusion and information noise reduction on heterogeneous data from different sensors and sends the data to the target detection and tracking module; and the automatic auxiliary suite sends the aircraft model, position, state information and parking position information comprehensively processed by the information fusion center to an aircraft cockpit to guide a driver to berth. The unmanned aircraft intelligent parking system is constructed in combination with the sensor information fusion technology, the precision and safety of aircraft parking are improved, and then the requirement of an intelligent airport is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an automatic parking system for an aircraft. BACKGROUND

[0002] The existing aircraft parking auxiliary system reduces the dependence on ground staff during the aircraft parking process by using various modern sensors, thereby improving the aircraft parking accuracy and parking efficiency. The system uses sensors such as radars to obtain the state information of the aircraft and transmits the information to the ground staff through a display screen to assist the ground staff in guiding the aircraft parking. However, the aircraft model recognition of some existing systems is still manually input by the tower and the staff, that is, the specified stopping position of the aircraft is determined based on the results of manual observation. The existing parking system has a strong dependence on the raw data of various sensors and cannot comprehensively process the noise and other external influences. In the case of insufficient light and bad weather, personnel intervention is required, and the system parameters are fixed values. The adaptation of the system in different scenarios depends on manual parameter updating. SUMMARY

[0003] The present application improves the high degree of staff dependence, insufficient utilization of sensor information and other defects of the existing aircraft parking system, and proposes an automatic aircraft parking system. The system combines sensor information fusion technology to construct an unmanned aircraft intelligent parking system, improves the accuracy and safety of aircraft parking, and meets the requirements of intelligent airport.

[0004] The present application is achieved by the following technical solutions:

[0005] The present application relates to an automatic aircraft parking system, which comprises a multi-sensor system, a target detection and tracking module, an information fusion center and an automatic auxiliary kit. The multi-sensor system collects image information and point cloud data of the aircraft to be parked through a camera, a laser radar and a millimeter wave radar. The target detection and tracking module performs target recognition detection and target tracking and cognitive processing according to the image information and the point cloud data, obtains the state information and the aircraft model of the aircraft to be parked. The information fusion center performs data fusion and information noise reduction on the heterogeneous data from different sensors, stores and analyzes the information of target recognition and detection and the results of target tracking, determines the principal components in the raw data through principal component analysis, eliminates the redundant information from different sensors, and ensures the consistency and coordination of the information. The automatic auxiliary kit sends the aircraft model, position, state information and parking position information processed by the information fusion center to the aircraft cockpit to guide the pilot to park. TECHNICAL EFFECTS

[0006] The application combines the human pose estimation algorithm PifPaf idea, makes full use of different parameter characteristics of different models, and defines 12 different key points for each aircraft. Through the detection, splicing and Yolo algorithm training of the key points, the aircraft skeleton recognition is completed. The application introduces reinforcement learning into the unscented Kalman filter, constructs five different action update process noise covariance matrices and observation noise covariance matrices, and seven different action update Sigma point parameters. The application fuses the data of the two laser radars through the RANSAC clustering method of the double laser radars. Compared with a single laser radar, the double laser radars have more dense point clouds, lower error levels and smaller error fluctuations in the aircraft front wheel area, and can provide more consistent and reliable results in continuous data processing. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 It is a schematic diagram of the system of the application;

[0008] Figure 2 It is a schematic diagram of 12 key points of an aircraft;

[0009] Figure 3 It is a schematic diagram of the total reward change curve with the training number;

[0010] Figure 4 It is a schematic diagram of the average reward change curve every five training;

[0011] Figures 5-7 It is a schematic diagram of the test results in three scenes of the embodiment;

[0012] Figure 8 It is a schematic diagram of the simulation aircraft taxiing process scene in the embodiment;

[0013] Figure 9 It is a schematic diagram of the simulation aircraft ground taxiing kinematics model in the embodiment;

[0014] Figure 10 It is a schematic diagram of the RANSAC processing effect of a single laser radar in the embodiment;

[0015] Figure 11 It is a schematic diagram of the RANSAC processing effect of two laser radars in the embodiment;

[0016] Figure 12 It is a comparison chart of the error between the processing value and the true value of a single laser radar and two laser radars in 50 sampling times in the embodiment;

[0017] Figure 13 It is a schematic diagram of the effect of the automatic auxiliary kit. DETAILED DESCRIPTION

[0018] As Figure 1As shown, the present embodiment relates to an aircraft automated docking system, which comprises a multi-sensor system, a target detection and tracking module, an information fusion center and an automated auxiliary suite.

[0019] The target detection and tracking module comprises a target recognition and detection unit and a target tracking and cognition unit based on reinforcement learning, wherein: the target recognition and detection unit reads out the skeleton parameters of the aircraft to be parked in the image according to the image data collected by the camera, using a bottom-up multi-person 2D human pose estimation method (PifPaf) combined with a Yolo neural network image recognition algorithm, and according to the point cloud data collected by the laser radar, uses a random sample consensus (RANSAC) clustering method to segment the point cloud data falling on the front wheels of the aircraft to be parked, and obtains the center point as the information of the front wheels, and further obtains the state information of the aircraft to be parked, and then compares with the existing data to obtain the aircraft model based on the information calibration of the millimeter wave radar; the target tracking and cognition unit uses a reinforcement learning-based unscented Kalman filter (RL-UKF) to track the target according to the observation data of the sensor.

[0020] As shown in Figure 13 , the automated auxiliary suite transmits the data processed by the information fusion center to the aircraft and displays it through the onboard display device to guide the aircraft pilot in a visual manner. At the same time, since the guidance information is sent to the aircraft in the form of data, it can be directly obtained and processed by other auxiliary driving systems and data analysis systems carried on the aircraft, providing a platform and interface for the application of other automatic driving technologies.

[0021] The image recognition algorithm specifically comprises:

[0022] 1) Combine the human pose estimation algorithm PifPaf idea, define 12 joint points for each aircraft, as shown in Figure 2 , detect and determine the key points using part intensity field. In each output position , Pif predicts a confidence , a vector pointing to the key point, a scale and a diffusion degree , expressed as .

[0023] 2) Convolve and weight the confidence of the regression target from Pif by using an unnormalized Gaussian kernel , to generate a high-resolution part position confidence map .

[0024] 3) After the spatial relationship and directionality between key points are analyzed by the Part Association Feild, the detected key points are associated with each other, and the aircraft attitude including the key point position and its connection is constructed, the Yolo network is trained to obtain more accurate aircraft attitude recognition results.

[0025] The random sample consensus (RANSAC) clustering method can explain and smooth data containing a large number of gross errors by generating candidate solutions using the minimum number of observations required to estimate the underlying model parameters. Considering that the installation position of the laser radar is generally in the front of the aircraft taxiing, most of the point clouds will fall on the front wheel surface, the point clouds falling on the front wheel are segmented by using the RANSAC clustering method, and the center point is calculated from the inlier set as the information of the front wheel, which specifically includes:

[0026] 1) Random sampling: select a minimum data subset, usually two points or more, to generate an initial estimate of the model;

[0027] 2) Model fitting: estimate the model parameters using the randomly selected minimum data subset;

[0028] 3) Error minimization: use the current model for all data points and calculate their error with the model prediction;

[0029] 4) Consistency check: determine the error of the data points with the current model is less than the preset value, these points are considered as inliers, that is, data points consistent with the model;

[0030] 5) Model verification and update: use the inlier set determined in the consistency check to re-estimate the model parameters to improve the accuracy of the model;

[0031] 6) Iterative process: repeat the steps of random sampling, model fitting, error minimization and consistency check until the best model is found or the predetermined number of iterations is reached;

[0032] 7) Best model selection: among all iterations, select the model with the largest inlier data point set as the final output, and the model is considered as the model that best represents the data.

[0033] The existing data refers to the parameter information of different types of aircraft, as shown in Table 1.

[0034] Table 1

[0035] The target tracking is specifically a reinforcement learning-based unscented Kalman filter algorithm (RL-UKF). Reinforcement learning is introduced into the unscented Kalman filter, and the parameter update action is constructed, so that the Sigma parameter and the process noise parameter and the measurement noise parameter can be autonomously updated, and the tracking of dynamic targets can be better processed. The DQN parameter setting of the reinforcement learning is shown in Table 2:

[0036] Table 2

[0037] The parameter update action includes:

[0038] The parameters of the UKF are initialized, the learning rate is 0.001, the reward scaling factor is 0.1, and after 1000 episodes of training, the total reward change of the training number is as shown in Figure 3 From the figure, it can be seen that the convergence of the total reward of the agent, and the convergence of the reward means that the training direction and significance of the reinforcement learning are clear.

[0039] The unscented Kalman filter uses reinforcement learning to control the filter parameters, and realizes online update of the filter parameters.

[0040] The data fusion is specifically: according to the covariance matrix of the data collected by different sensors, the abnormal values, i.e. outliers, in the data are determined by weighting and principal component analysis; after removing the outliers, the measurement information of different sensors is combined to form a unified information set, i.e. data fusion, and the common state estimation is realized through the unscented Kalman filter algorithm to obtain more accurate state estimation.

[0041] The information noise reduction is specifically: the measurement noise matrix and the process noise matrix in the unscented Kalman filter are autonomously updated through reinforcement learning, so that more accurate estimation of noise is realized, thereby achieving the effect of information noise reduction.

[0042] The aircraft cockpit receives the guidance instructions sent by the wireless communication network (such as 5G or dedicated VHF data link) from the airport, and displays them through the onboard display system.

[0043] Through specific actual experiments, the Falcon Kinetic laser radar of Innovusion Company is used for simulation, and the parameters of the laser radar model used for simulation are shown in Table 3.

[0044] Table 3

[0045] To test the tracking effect of the system in different environments, the test scenarios shown in Table 4 are constructed:

[0046] Table 4

[0047] In Table 4: represents the distance standard deviation of the lidar, represents the angle standard deviation of the lidar, and the point cloud loss scenario simulates rainy days and other situations where the lidar does not work well. (Different scenarios mainly affect the data acquisition ability of the sensor)

[0048] As shown in Figures 5-7 , three different scenarios together constitute a variety of test environments. Scenario one simulates the running environment under normal noise conditions, providing a benchmark performance indicator for the system; Scenario two introduces strong noise to test the robustness of the system under more severe conditions, ensuring stability in extreme situations; Scenario three further challenges the fault tolerance of the system by simulating point cloud data loss, which is crucial for evaluating the system's performance when facing sensor failures or signal interference. The evaluation parameters for different scenarios are shown in Tables 5-7.

[0049] Table 5 Comparison of RMSE (Root Mean Square Error) of three UKFs under scenario one

[0050] Table 6 Comparison of RMSE (Root Mean Square Error) of three UKFs under scenario two

[0051] Table 7 Comparison of RMSE (Root Mean Square Error) of three UKFs under scenario three

[0052] From Figures 5-7From the above comparison table, under the condition of good environment and normal sensor operation, RL-UKF can achieve better target tracking effect, and the result is better than the commonly used NUKF (Normal Unscented Kalman Filter); under the condition of strong environmental noise, the tracking effect of RL-UKF is similar to that of AUKF (Adaptive Unscented Kalman Filter) which has strong noise robustness; under the condition of poor sensor operation and loss of point cloud data, the target tracking effect of RL-UKF is optimal. Therefore, RL-UKF has better comprehensive processing capability in different scenarios and can adapt to target tracking in different scenarios. This proves that RL-UKF can adjust its internal parameters in real time through training to adapt to the dynamically changing noise environment. The adaptive characteristics of RL-UKF make it more flexible and effective in processing different environmental noise.

[0053] Table 8 Error analysis

[0054] As shown in Figure 8 , an aircraft landing scene is built for simulation verification. One laser radar is installed on each side of the runway, which continuously observes the state information of the aircraft during the landing and parking process of the aircraft.

[0055] As shown in Figure 9 , it is a schematic diagram of the aircraft ground taxi kinematics model. The model regards the aircraft as a simplified system composed of front and rear wheels, and the acceleration and deceleration operations of the aircraft are controlled by the engine input and the brake torque of the rear wheel, respectively, while the steering operation is realized by adjusting the steering angle of the front wheel. This model can more realistically simulate the nonlinear trajectory of the aircraft and introduce the relevant information of the front wheel of the aircraft observed by the laser radar, making the simulation more realistic.

[0056] As shown in Figure 10 and Figure 11 , they are the results of observing the front wheel of the aircraft using a single laser radar and using double laser radars to observe the front wheel of the aircraft, respectively. When processing point cloud data, although the RANSAC algorithm can effectively identify inliers and suppress outliers, its performance is limited under the condition of strong noise with large angle error. In contrast, the configuration of two laser radars shows better data fusion effect, enhances the suppression ability of abnormal values, and thus improves the overall data processing quality. The double laser radar system is more accurate in estimating the fitting point of the edge midpoint of the front wheel of the aircraft, and can obtain more accurate aircraft position information.

[0057] As shown in Figure 12As shown, compared with a single laser radar, the double laser radar system provides more point cloud data, significantly increases the amount of information of the inlier set when the algorithm is sampled, and thus improves the accuracy of model fitting and segmentation effect. At the same time, since the error distribution of the double laser radar system presents a clear concentration trend, most error values are closely distributed around zero, which shows that the mean of the error is close to zero, and the error fluctuation is small, showing high estimation accuracy and stability, which is conducive to improving the accuracy of the subsequent target tracking link.

[0058] In summary, with the gradual increase of environmental interference, the accuracy of different tracking algorithms is reduced, which is also consistent with daily experience. At the same time, it can be seen that, compared with traditional algorithms such as fixed parameter unscented Kalman filter (Normal UKF), the unscented Kalman filter based on reinforcement learning (RL-UKF) has better tracking accuracy, and shows better anti-interference ability and accuracy.

[0059] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraint of the present application.

Claims

1. An automated aircraft berthing system, characterized in that: include: A multi-sensor system, a target detection and tracking module, an information fusion center, and an automated assistance suite. The multi-sensor system uses cameras, lidar, and millimeter-wave radar to collect image information and point cloud data of the aircraft to be parked. The target detection and tracking module sequentially performs target identification and detection, target tracking, and cognitive processing based on the image information and point cloud data to obtain the status information and aircraft model of the aircraft to be parked. The information fusion center fuses and reduces noise on heterogeneous data from different sensors, stores and analyzes target identification, detection, and tracking results, determines the principal components in the raw data through principal component analysis, and eliminates redundant information from different sensors to ensure information consistency and coordination. The automated assistance suite sends the aircraft model, location, status and parking position information processed by the information fusion center to the aircraft cockpit to guide the pilot to the parking position.

2. The aircraft automated berthing system according to claim 1, characterized in that: The target detection and tracking module includes: a target recognition and target detection unit and a target tracking and cognition unit based on reinforcement learning, wherein: the target recognition and target detection unit respectively reads the skeleton parameters of the aircraft to be parked in the image based on the image data collected by the camera using a bottom-up multi-person 2D human posture estimation method combined with a Yolo neural network image recognition algorithm, and uses a random sampling consistent clustering method to segment the point cloud data of the front wheels of the aircraft to be parked based on the point cloud data collected by the lidar, and obtains the center point as the information of the front wheel, thereby obtaining the status information of the aircraft to be parked, and then obtains the aircraft model after information calibration and comparison based on the existing data based on the millimeter wave radar; the target tracking and cognition unit obtains the observation data of the target based on the sensor, and uses an unscented Kalman filter (RL-UKF) based on reinforcement learning to track the target.

3. The aircraft automated berthing system according to claim 2, wherein: The image recognition algorithm specifically includes: 1) Combining the human posture estimation algorithm PifPaf, 12 joint points are defined for each aircraft, and the key points are detected and determined using the Part Intensity Field. , Pif predicts a confidence , a vector pointing to the keypoint , a scale and a diffusion , expressed as ; 2) By using an unnormalized Gaussian kernel , convolve the regression target from Pif and weight its confidence , generating a high-resolution local position confidence map ; 3) The spatial relationship and directionality between key points are analyzed through the component association field, and the detected key points are associated with each other. After constructing the aircraft attitude including the key point positions and their connections, it is trained through the Yolo network to obtain more accurate aircraft attitude recognition results.

4. The aircraft automated berthing system according to claim 2, wherein: The random sampling consensus clustering method generates candidate solutions by using the minimum observation value required to estimate the underlying model parameters. It can explain and smooth data containing a large number of gross errors. Considering that the installation position of the lidar is generally in front of the taxiing aircraft, most point clouds will fall on the surface of the front wheel. The point cloud falling on the front wheel is segmented using the RANSAC clustering method, and the center point is obtained from the inner point set as the information of the front wheel.

5. The aircraft automated berthing system according to claim 2 or 4, characterized in that: The random sampling consistent clustering method specifically includes: 1) Random sampling: Select a minimum subset of data, usually two points or more, to generate an initial estimate of the model; 2) Model fitting: using a randomly selected minimal subset of data to estimate model parameters; 3) Error minimization: Apply the current model to all data points and calculate the error between them and the model prediction; 4) Consistency check: Determine if the error between the data point and the current model is less than a preset value. These points are considered to be inliers, that is, data points that are consistent with the model; 5) Model verification and updating: Use the inlier set identified in the consistency check to re-estimate the model parameters to improve the accuracy of the model; 6) Iterative process: Repeat the random sampling, model fitting, error minimization and consistency check steps until the optimal model is found or the predetermined number of iterations is reached; 7) Best model selection: Among all iterations, the model with the largest set of consistent data points is selected as the final output. The model is considered to be the one that best represents the data.

6. The aircraft automated berthing system according to claim 1 or 2, characterized in that: The target tracking is specifically: based on the unscented Kalman filter algorithm of reinforcement learning, reinforcement learning is introduced into the unscented Kalman filter, and the parameter update action is constructed so that its Sigma parameter as well as the process noise parameter and the measurement noise parameter can be updated autonomously, so as to better handle the tracking of dynamic targets.

7. The aircraft automated berthing system according to claim 6, characterized in that: The parameter updating action includes: , .

8. The aircraft automated berthing system according to claim 1, wherein: The data fusion is specifically as follows: based on the covariance matrix of the data collected by different sensors, the outliers in the data are determined by weighting and principal component analysis; after removing them, the measurement information of different sensors is combined into a unified information set, i.e. data fusion, and a common state estimation is achieved through the unscented Kalman filter algorithm to obtain a more accurate state estimation.

9. The aircraft automated berthing system according to claim 1, wherein: The information denoising is specifically as follows: the measurement noise matrix and the process noise matrix in the unscented Kalman filter are autonomously updated through reinforcement learning to achieve a more accurate estimation of the noise, thereby achieving the effect of information denoising.

10. The aircraft automated berthing system according to claim 1, wherein: The automated assistance kit transmits the data processed by the information fusion center to the aircraft and displays it through the onboard display device, guiding the aircraft pilot to his or her parking position in a visual manner. At the same time, since the guidance information is sent to the aircraft in the form of data, it is directly acquired and processed by other assisted driving systems and data analysis systems on the aircraft, providing a platform and interface for the application of other autonomous driving technologies.

Images