Airplane automatic landing method based on multi-modal image recognition

Abstract

The invention provides an automatic aircraft landing method based on multi-modal image recognition, and relates to the technical field of aviation navigation and automatic control, and the method comprises the steps: fusing optical, infrared and millimeter wave radar data through a multi-modal sensor, and carrying out the preprocessing to generate a fused image sequence; detecting a runway, an obstacle and a meteorological entity based on an improved YOLOv8 model, and extracting an environment dynamic adaptation factor (EDAF) and a multi-modal fusion confidence coefficient (MFC); predicting and controlling an MPC path planning weight by combining a flight state data dynamic optimization model, and generating a landing instruction; the flight attitude is adjusted in real time by using a redundant PID controller, and control parameters are dynamically corrected through an EDAF; continuously monitoring the system state, and triggering emergency operation based on the decision tree model. Through multi-modal sensing fusion, dynamic parameter optimization and an intelligent emergency mechanism, the problems of low reliability, control parameter staticization and emergency delay of a single sensor are solved, and the landing precision, stability and safety in a complex environment are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of aviation navigation and automatic control technology, and in particular to an automatic aircraft landing method based on multimodal image recognition. Background Art

[0002] In the aviation field, the reliability of aircraft automatic landing technology is directly related to flight safety and operational efficiency. Current mainstream technologies rely on single-modality visual navigation systems, such as optical camera-based runway recognition solutions. However, these systems are susceptible to lighting conditions in low-visibility environments (such as fog, heavy rain, and at night), and are unable to accurately detect runway boundaries and obstacles, resulting in increased deviations in landing path planning. In addition, when the runway is covered with rain or snow, the temperature resolution of traditional infrared sensors is insufficient, making it difficult to promptly identify ice or water areas on the runway surface, further exacerbating safety risks.

[0003] Existing path planning algorithms often use fixed optimization weights, making it impossible to dynamically adjust control strategies based on real-time environmental disturbances (such as sudden changes in wind speed or runway temperature gradients). For example, in strong crosswinds, static weights can lead to cumulative path tracking errors, increasing the risk of runway deviation. Furthermore, traditional flight control systems rely on preset PID parameters and lack adaptability to dynamic environmental changes. This makes control lag or oscillations more likely to occur, particularly in complex weather conditions, compromising landing stability.

[0004] Furthermore, emergency decision-making mechanisms are often based on threshold judgments or manual intervention, resulting in insufficient response speed and automation. When sensor data reliability degrades (e.g., millimeter-wave radar interference due to rain or fog) or anomalies occur (e.g., engine thrust failure), existing systems struggle to promptly generate and execute emergency commands such as go-arounds and obstacle avoidance, resulting in insufficient safety redundancy.

[0005] The above technical defects limit the application scope and reliability of the automatic landing system in complex environments. There is an urgent need for a comprehensive solution that can integrate multimodal perception, dynamic parameter optimization and intelligent decision-making. Summary of the Invention

[0006] In order to solve the technical problems in the existing technology, such as the insufficient reliability of single-modal perception systems in low-visibility environments, poor adaptability to dynamic disturbances caused by fixed weights in path planning algorithms, control lag and oscillation caused by static flight control parameters, and response delay and low degree of automation of emergency decision-making mechanisms, the present invention provides an aircraft automatic landing method based on multimodal image recognition.

[0007] The technical solutions provided by the present invention are as follows:

[0008] The present invention provides an aircraft automatic landing method based on multimodal image recognition, comprising:

[0009] S1. Collecting multi-source image data of the aircraft's surrounding environment in real time using a multi-modal sensor, wherein the multi-modal sensor includes a high-resolution optical camera, an infrared thermal imager, and a millimeter-wave radar;

[0010] S2. Preprocessing the multi-source image data, including denoising, distortion correction, and spatiotemporal alignment, to generate a fused multimodal image sequence;

[0011] S3. Performing target detection and semantic segmentation on the multimodal image sequence based on a deep learning model to identify runway boundaries, obstacles, and meteorological interference entities, and extracting two key parameters: environmental dynamic adaptation factor (EDAF) and multimodal fusion confidence factor (MFC);

[0012] S4. Generate landing path planning instructions using a decision model based on the aircraft's current flight status data, including speed, altitude, and attitude angle. The decision model dynamically adjusts path optimization weights based on EDAF and MFC.

[0013] S5. Executing the landing path planning instruction through the flight control module, correcting the aircraft's pitch angle, roll angle, and thrust output in real time to ensure the aircraft lands stably along the planned path;

[0014] S6. During the landing process, the system continuously monitors environmental changes and system operating status. If an anomaly is detected, the emergency decision-making mechanism is triggered to execute a go-around or emergency obstacle avoidance operation.

[0015] Furthermore, the spatiotemporal alignment in S2 adopts a SLAM algorithm based on feature point matching, specifically including:

[0016] S201, extracting ORB feature points from the multimodal image;

[0017] S202, eliminating mismatched points using the RANSAC algorithm;

[0018] S203, construct a 6-DOF posture transformation matrix to achieve spatiotemporal synchronization of multi-sensor data

[0019] Furthermore, the EDAF calculation method steps are as follows:

[0020] S301, extracting the ambient temperature gradient matrix T based on infrared thermal imaging data grad ;

[0021] S302. Calculate the wind speed disturbance coefficient W based on millimeter wave radar data dist ;

[0022] S303. Calculate EDAF using the following formula:

[0023]

[0024] Among them, α and β are preset empirical coefficients, and the T grad Indicates the temperature change rate per square meter on the runway, in ℃ / m 2 °, W dist It represents the wind speed component perpendicular to the runway, in m / s.

[0025] Furthermore, the MFC calculation method steps are as follows:

[0026] S311. Assign confidence scores C to the detection results of optical cameras, infrared thermal imagers, and millimeter-wave radars. opt 、C inf 、C rad ;

[0027] S312. Calculate the multimodal fusion confidence MFC using the following formula:

[0028]

[0029] Among them, C opt ,C inf ,C rad ∈[0,1].

[0030] Furthermore, the deep learning model in S3 adopts an improved YOLOv8 algorithm and improves the accuracy of small target detection by introducing a channel attention mechanism.

[0031] Furthermore, the generation of the path planning instruction in S4 is based on a model predictive control (MPC) algorithm, whose objective function is:

[0032]

[0033] Among them, e k is the path tracking error at step k, Δu k is the rate of change of the control variable, λ is the smoothing factor, and N is the prediction time domain step size.

[0034] Furthermore, the flight control module in S5 adopts a redundant PID controller, and its control law is:

[0035]

[0036] Among them, K p , K i , K d is the controller gain parameter, which is dynamically adjusted according to EDAF.

[0037] Furthermore, the emergency decision mechanism in S6 is based on a decision tree model, and the node rules of the decision tree include: if MFC is lower than a threshold θ and EDAF is higher than a threshold β, a go-around instruction is triggered.

[0038] Furthermore, the values ​​of the empirical coefficients α and β are obtained through offline reinforcement learning optimization, and the optimization goal is to minimize the landing path deviation.

[0039] Furthermore, the confidence score specifically includes:

[0040] The confidence score C opt 、C inf 、C rad The calculation is based on the matching degree between the detection target and the prior map. The matching degree formula is:

[0041]

[0042] Among them, P detected To detect the target position, P prior is the corresponding target position in the prior map, D max is the maximum allowable position deviation.

[0043] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0044] (1) In this invention, through the collaborative operation of multimodal sensors and data fusion, the system accurately identifies runway boundaries, obstacles, and meteorological interference entities in low visibility, rain, snow, fog, and haze, as well as at night. The multimodal fusion confidence dynamic assessment mechanism suppresses single sensor failure or noise interference, ensuring the robustness of environmental perception results and significantly reducing landing path deviations and safety risks caused by perception errors.

[0045] (2) In this invention, the runway temperature gradient and wind speed disturbance intensity are evaluated in real time based on the environmental dynamic adaptation factor, and the path planning weights and flight control parameters are dynamically adjusted. By synergizing the model predictive control algorithm with redundant controllers, the system adaptively matches the control requirements under different environmental disturbances, optimizes path tracking accuracy and operational smoothness, and avoids the oscillation or lag caused by sudden environmental changes in traditional fixed parameter control.

[0046] (3) In the present invention, a decision tree model analyzes multimodal fusion confidence, environmental dynamic adaptation factors, and flight status data in real time, determining risk levels within milliseconds and triggering go-around or emergency obstacle avoidance instructions. The seamless switching and parameter synchronization mechanism of redundant controllers significantly reduces emergency operation delays, eliminating the need for human intervention and significantly improving autonomous response capabilities in emergency scenarios such as sensor anomalies, sudden obstacles, or engine failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A schematic diagram of a flow chart of an automatic aircraft landing method based on multimodal image recognition provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a spatiotemporal alignment process in an aircraft automatic landing method based on multimodal image recognition provided by an embodiment of the present invention;

[0050] Figure 3 A schematic flow chart of an EDAF calculation method in an aircraft automatic landing method based on multimodal image recognition provided by an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of a flow chart for dynamically adjusting controller gain parameters in an aircraft automatic landing method based on multimodal image recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0053] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0054] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, the meanings they convey are the same.

[0055] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] Reference Manual Figure 1 , which shows a flow chart of an aircraft automatic landing method based on multimodal image recognition provided by an embodiment of the present invention.

[0058] An embodiment of the present invention provides an automatic aircraft landing method based on multimodal image recognition. The processing flow may include the following steps:

[0059] S1. Collect multi-source image data of the aircraft's surrounding environment in real time through multi-modal sensors.

[0060] It should be noted that multimodal sensors include a high-resolution optical camera, an infrared thermal imager, and a millimeter-wave radar. The high-resolution optical camera is used to capture runway textures, obstacles, and ground markings in the visible light band; the infrared thermal imager generates an image of the ambient temperature distribution by detecting infrared radiation, which is used to identify runway icing or heat source interference caused by temperature differences; the millimeter-wave radar obtains the distance, speed, and direction information of the target object by transmitting and receiving millimeter-wave signals, and is particularly suitable for environmental perception in low-visibility conditions such as rain and fog. The sensor array needs to be installed on the front and belly of the aircraft to ensure coverage of the forward and downward fields of view during the aircraft's landing process, and an anti-shake bracket and shock-absorbing device must be used to eliminate the impact of flight vibration on data collection.

[0061] It's important to note that multimodal sensors require joint calibration in a laboratory environment to ensure that the focal length of the optical camera, the temperature sensitivity of the infrared thermal imager, and the ranging accuracy of the millimeter-wave radar meet aviation-grade standards. This calibration process includes determining the intrinsic parameter matrix and distortion coefficients of the optical camera, correcting the nonlinear temperature response of the infrared thermal imager, and jointly calibrating the range and velocity of the millimeter-wave radar.

[0062] Specifically, the optical camera calibration uses a 9×6 corner checkerboard calibration plate, and collects at least 20 sets of images at different distances (1 meter to 20 meters) and angles (pitch angle ±30°, yaw angle ±45°). The intrinsic parameter matrix (focal length f x ,f y 、Main point c x ,c y ) and distortion coefficients (radial distortion k1, k2, tangential distortion p1, p2), and the calibration error must be less than 0.1 pixel.

[0063] Furthermore, the infrared thermal imager calibration uses a blackbody radiation source to generate temperature calibration points in the range of -20℃ to 100℃ at intervals of 5℃, collects the infrared image grayscale value at each temperature point, and fits the temperature-grayscale response curve through a third-order polynomial. The fitting residual must be less than 0.5℃.

[0064] To further calibrate the millimeter-wave radar, metal reflectors were placed at 50, 100, and 200 meters in a windless environment to collect distance and velocity data from the radar echo signals. The least-squares method was used to calibrate the distance measurement error (±0.1 meter) and Doppler velocity error (±0.05 m / s).

[0065] It should be noted that the installation location of multimodal sensors must be optimized based on their functional characteristics and data collection requirements. A high-resolution optical camera is installed within the aircraft's front fairing, with its lens facing forward and downward. It covers the runway's forward field of view during landing (horizontal field of view ≥60°, vertical field of view ≥40°), ensuring a complete optical image of the runway texture, ground markings, and obstacles. An infrared thermal imager is mounted on the aircraft's belly, near the landing gear. Its lens is aimed vertically at the runway surface. Its short focal length (focal length ≤50mm) enables high-resolution temperature distribution monitoring of the runway area, avoiding interference from thermal radiation from the fuselage. The millimeter-wave radar utilizes a distributed multi-probe layout, with the main probe mounted below the nose and secondary probes symmetrically distributed on either side of the wings. Its tilted installation (pitch angle 10° to 15°) allows for both long-range target detection and near-ground obstacle identification, eliminating detection blind spots in rainy and foggy environments.

[0066] All sensors are secured via anti-shake brackets and shock-absorbing devices. The brackets incorporate built-in active stabilization mechanisms (such as piezoelectric ceramic actuators) to counteract image blur or data jitter caused by flight vibrations in real time. Multi-sensor clock signals are uniformly calibrated via an onboard synchronization module, with time deviations controlled to within ±1 microsecond, ensuring precise matching of data frames during spatiotemporal alignment. The fields of view of the optical camera and infrared thermal imager overlap by ≥70%, while the millimeter-wave radar's detection range extends 20 meters beyond the boundaries of the optical and infrared fields of view, forming a seamless fusion foundation for multimodal data.

[0067] S2. Preprocess the multi-source image data to generate a fused multimodal image sequence.

[0068] Furthermore, the preprocessing includes the following sub-steps:

[0069] Denoising: For optical camera images, a frequency-domain filtering algorithm based on wavelet transform is used to eliminate random noise. For infrared thermal imaging data, a median filter is used to remove isolated noise points. For millimeter-wave radar data, a sliding window averaging method is used to reduce signal fluctuations.

[0070] Distortion Correction: Based on the intrinsic parameter matrix and distortion coefficients of the optical camera, the Zhang Zhengyou calibration method is used to correct radial and tangential distortion of the image. The geometric distortion of the infrared thermal imager is compensated using a pre-calibrated nonlinear mapping table.

[0071] Spatiotemporal alignment: The SLAM (Simultaneous Localization and Mapping) algorithm based on feature point matching is used to achieve spatiotemporal synchronization of multi-sensor data. Figure 2 As shown, specifically including:

[0072] S201, extracting Oriented FAST and Rotated BRIEF (ORB) feature points from the optical image and the infrared image;

[0073] S202, eliminating mismatched point pairs using the Random Sample Consensus (RANSAC) algorithm;

[0074] S203: Construct a 6-DOF pose transformation matrix to map the millimeter-wave radar data to the optical camera coordinate system, ensuring that the timestamps of all sensor data are consistent with the spatial coordinates.

[0075] S3. Perform target detection and semantic segmentation on multimodal image sequences based on deep learning models, and extract the Environmental Dynamic Adaptation Factor (EDAF) and Multimodal Fusion Confidence (MFC).

[0076] It should be noted that for object detection and semantic segmentation, the improved You Only Look Once version 7 (YOLOv8) algorithm is used, and the Channel Attention Mechanism is introduced to enhance the model's detection capabilities for runway edges, obstacles, and meteorological entities (such as rain and snow). Specific improvements include:

[0077] Embed a channel attention module in the Feature Pyramid Network (FPN) to improve the small object feature response through adaptive weight distribution;

[0078] The dynamic non-maximum suppression (DNMS) algorithm is used to optimize the screening threshold of overlapping target boxes.

[0079] It should be noted that the improved YOLOv8 model needs to be trained using a labeled dataset containing rain, snow, fog, haze, and night runway scenes. The dataset covers different lighting conditions, meteorological interference, and runway surface conditions (such as dry, wet, and icy).

[0080] Specifically, the dataset construction involves synchronously collecting image data covering a variety of complex scenarios using multimodal sensors, including runway environments in rain, snow, fog, haze, nighttime conditions, and strong lighting conditions, as well as image sequences of different runway surface conditions (dry, wet, and icy). The annotated data must include pixel-level semantic segmentation labels for runway boundaries, obstacles, and meteorological entities (such as rain and snow patches), as well as object detection bounding boxes.

[0081] The training strategy utilizes a phased approach. Initially, pre-training is performed using a large-scale general aviation dataset and simulations, followed by fine-tuning using domain-specific data in the second phase. The learning rate is dynamically adjusted using a cosine annealing strategy, with a higher initial learning rate to accelerate convergence and a gradually lower learning rate during the fine-tuning phase to refine model parameters.

[0082] To achieve the effect of data augmentation, random geometric transformations (including rotation, scaling, and translation) and photometric transformations (adding Gaussian noise, simulating motion blur, and adjusting contrast) are applied to the training data to change weather conditions and terrain, thereby improving the model's robustness to lighting changes and motion jitter.

[0083] The optimizer configuration uses an adaptive momentum optimization algorithm (Adam variant), combined with a weight decay strategy to prevent overfitting. The loss function uses focal loss to balance the training weights of small objects and background categories.

[0084] It should be noted that the loss function uses focal loss to alleviate the category imbalance problem in small target detection, and the model robustness is improved through data enhancement (random rotation, scaling, and adding noise) during training.

[0085] Specifically, if Figure 3 As shown, the EDAF calculation method is as follows:

[0086] S301, extracting the temperature gradient matrix T of the runway area based on infrared thermal imaging data grad , which is defined as the maximum temperature difference per square meter (unit: ℃ / m 2 );

[0087] S302: Calculate the wind speed disturbance coefficient W perpendicular to the runway based on the radial velocity data of the millimeter wave radar. dist , its value is the component of wind speed in the direction normal to the runway (unit: m / s);

[0088] S303: Through formula

[0089]

[0090] Calculate EDAF, where α and β are empirical coefficients optimized by offline reinforcement learning, and the optimization objective is to minimize the historical landing path deviation.

[0091] It should be noted that the optimization of the empirical coefficients α and β is completed through a simulation environment, and the simulation scenario needs to cover various extreme conditions such as high temperature runway, crosswind disturbance, and low visibility.

[0092] It should be noted that the optimization algorithm uses proximal policy optimization (PPO), which iteratively adjusts the values ​​of α and β to minimize the correlation coefficient between the EDAF and the actual landing path. The simulation data includes more than 1,000 landing cases in different environmental combinations to ensure the universality of the optimization results.

[0093] Specifically, the MFC calculation method is as follows:

[0094] S311: Assign a confidence score C to the detection results of the optical camera, infrared thermal imager, and millimeter wave radar respectively opt 、C inf 、C rad , the scoring is based on the matching degree between the detection target and the prior map. The matching degree formula is:

[0095]

[0096] Among them, P detected To detect the target position coordinates, P prior is the standard coordinate of the corresponding target in the prior map, D max is the maximum allowable position deviation threshold (for example, set to 5 meters);

[0097] It should be noted that the prior map is generated by high-precision LiDAR scanning of airport runways and contains the three-dimensional coordinates of runway boundaries, taxiway markings, and fixed obstacles. This map data must be integrated with the airport's real-time management system to ensure that changes to obstacles (such as temporary construction fences) are updated within 24 hours. The map resolution is at least 0.1 meters, and the coordinate system uses the WGS-84 standard, ensuring seamless compatibility with onboard navigation systems.

[0098] S312: By formula

[0099]

[0100] The multimodal fusion confidence is calculated, and the denominator is designed to suppress the influence of abnormally high scores of a single sensor.

[0101] S4. Generate landing path planning instructions based on the aircraft flight status data.

[0102] Specifically, flight status data: real-time acquisition of the aircraft's airspeed (unit: m / s), pressure altitude (unit: meter) and three-axis attitude angles (pitch angle, roll angle, yaw angle, unit: degree).

[0103] Decision model: The Model Predictive Control (MPC) algorithm is used to generate path planning instructions, and its objective function is:

[0104]

[0105] Among them, e k is the path tracking error at step k, Δu k is the rate of change of the control variable, λ is the smoothing factor (range 0.1 to 1.0), and N is the prediction time domain step (usually set to 10 to 20 steps).

[0106] Dynamic weight adjustment: The path optimization weights in the objective function are adjusted in real time based on EDAF and MFC. For example, when EDAF increases (indicating increased environmental disturbances), the weight of the tracking error term is increased; when MFC decreases (indicating decreased sensor reliability), the weight of the control variable smoothing term is increased.

[0107] S5. Execute the landing path planning instruction through the flight control module.

[0108] Specifically, the control law design uses a redundant proportional-integral-derivative (PID) controller, whose control law is:

[0109]

[0110] Among them, K p , K i , K d is the controller gain parameter, which is dynamically adjusted according to EDAF. For example, when EDAF exceeds the threshold, K is increased. d To enhance anti-interference ability.

[0111] Redundancy mechanism: The main controller and the backup controller run in parallel. When the main controller fails, the backup controller takes over control within 10 milliseconds to ensure continuous execution of instructions.

[0112] It should be noted that the status data of the active and standby controllers are synchronized in real time via the high-speed bus with a synchronization period of 1 millisecond. The synchronization content includes the current control parameters (such as K p , K i , Kd ), flight status data, and real-time EDAF / MFC values. The synchronization protocol uses aviation-grade Time-Triggered Ethernet (TTE) to ensure deterministic and low-latency data transmission and avoid command jumps during handover.

[0113] It should be noted that if Figure 4 As shown, the controller gain parameter K p , K i , K d The detailed steps for dynamic adjustment based on EDAF are as follows:

[0114] S501, initial gain parameter: preset proportional gain K according to aircraft model and flight control requirements p , integral gain K i , differential gain K d The initial value of K p =1.2, K i =0.05, K d =0.8, the EDAF threshold is divided into multiple intervals, and the adjustment strategies for different intervals are defined:

[0115] Low disturbance range: EDAF≤2.0;

[0116] Medium disturbance range: 2.0 <EDAF≤4.0;

[0117] High disturbance range: EDAF>4.0.

[0118] Furthermore, control scenarios (such as strong crosswind and runway icing) under different EDAF values ​​are simulated in a flight simulator, the system response after gain parameter adjustment is recorded, and the piecewise function coefficients are optimized.

[0119] S502, calculate the gain adjustment coefficient in real time based on EDAF, the adjustment coefficient formula is: According to the current value of EDAF, calculate the adjustment coefficient ΔK of each gain parameter through the piecewise function p , ΔK i , ΔK d :

[0120] Low disturbance range: ΔK p =0.1·EDAF,ΔK d =0.05·EDAF;

[0121] High disturbance range: ΔK p =0.3·(EDAF-4.0), ΔK d =0.2·(EDAF-4.0).

[0122] Integral gain adjustment: To prevent integral saturation, ΔK i Dynamic adjustment based on the accumulated value of system error, the formula is:

[0123] ΔK i =min(0.02·|e(t)|,0.1)

[0124] Where e(t) is the current tracking error.

[0125] Furthermore, the gain adjustment algorithm needs to run in the real-time operating system of the onboard computer, with a calculation cycle of no more than 10 milliseconds to ensure the timeliness of the control instructions.

[0126] S503. Apply the dynamically adjusted gain parameter. Gain update formula: Add the adjustment coefficient to the initial gain parameter to generate the real-time control parameter:

[0127]

[0128] Anti-saturation constraints: impose upper and lower limits on the gain parameters:

[0129] K p ∈[0.5,2.5]

[0130] K i ∈[0.01,0.1]

[0131] K d ∈[0.3, 1.5]

[0132] S504, real-time verification and adaptive optimization, control performance evaluation: Evaluate the control effect after adjustment through the following indicators:

[0133] Path tracking error e(t) = ||actual position - target position||

[0134] Control quantity smoothness Δu(t)=||u(t)-u(t-1)||

[0135] Adaptive feedback mechanism: If the tracking error continues to exceed the threshold (e.g., e(t)>1.0 m), a secondary adjustment of the gain parameters is triggered:

[0136]

[0137] Where sgn is the sign function, which ensures that the gain adjustment direction is consistent with the error trend.

[0138] S505. Redundant Controller Parameter Synchronization: Master and Standby Controller Synchronization: The master controller's real-time gain parameters are transmitted to the standby controller via a high-speed bus (e.g., Time-Triggered Ethernet (TTE)). The synchronization cycle is 1 millisecond. Switchover consistency is ensured: If the master controller fails, the standby controller immediately adopts the gain parameters from the last synchronization, ensuring seamless control command transitions.

[0139] S6. Continuously monitor the environment and system status and trigger emergency decision-making mechanisms.

[0140] Anomaly detection: Real-time monitoring of the MFC, EDAF, sensor health status, and control instruction execution deviations. If the MFC is lower than a threshold θ (e.g., 0.6) and the EDAF is higher than a threshold γ (e.g., 3.0), a high-risk state is determined.

[0141] Emergency decision-making: Select emergency strategies based on the decision tree model. Decision rules include:

[0142] If the runway is blocked by an obstacle, emergency obstacle avoidance is performed and an evasive flight path is generated;

[0143] If the engine thrust is abnormal, a go-around command will be triggered to control the aircraft to climb to a safe altitude and re-plan the approach path.

[0144] Perform emergency operations: During the go-around, the flight control module increases the thrust output to the maximum continuous thrust and adjusts the flaps and landing gear status to ensure that the climb gradient meets safety standards.

[0145] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0146] (1) In this invention, through the collaborative operation of multimodal sensors and data fusion, the system accurately identifies runway boundaries, obstacles, and meteorological interference entities in low visibility, rain, snow, fog, and haze, as well as at night. The multimodal fusion confidence dynamic assessment mechanism suppresses single sensor failure or noise interference, ensuring the robustness of environmental perception results and significantly reducing landing path deviations and safety risks caused by perception errors.

[0147] (2) In this invention, the runway temperature gradient and wind speed disturbance intensity are evaluated in real time based on the environmental dynamic adaptation factor, and the path planning weights and flight control parameters are dynamically adjusted. By synergizing the model predictive control algorithm with redundant controllers, the system adaptively matches the control requirements under different environmental disturbances, optimizes path tracking accuracy and operational smoothness, and avoids the oscillation or lag caused by sudden environmental changes in traditional fixed parameter control.

[0148] (3) In the present invention, a decision tree model analyzes multimodal fusion confidence, environmental dynamic adaptation factors, and flight status data in real time, determining risk levels within milliseconds and triggering go-around or emergency obstacle avoidance instructions. The seamless switching and parameter synchronization mechanism of redundant controllers significantly reduces emergency operation delays, eliminating the need for human intervention and significantly improving autonomous response capabilities in emergency scenarios such as sensor anomalies, sudden obstacles, or engine failures.

[0149] The above content is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0150] There are a few points to note:

[0151] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0152] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present invention are exaggerated or reduced, that is, these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or intervening elements may be present.

[0153] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to form new embodiments.

[0154] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An aircraft automatic landing method based on multimodal image recognition, characterized in that: include: S1. Collecting multi-source image data of the aircraft's surrounding environment in real time using a multi-modal sensor, wherein the multi-modal sensor includes a high-resolution optical camera, an infrared thermal imager, and a millimeter-wave radar; S2. Preprocessing the multi-source image data, including denoising, distortion correction, and spatiotemporal alignment, to generate a fused multimodal image sequence; S3. Performing target detection and semantic segmentation on the multimodal image sequence based on a deep learning model to identify runway boundaries, obstacles, and meteorological interference entities, and extracting two key parameters: the environmental dynamic adaptation factor (EDAF) and the multimodal fusion confidence factor (MFC); S4. Generate landing path planning instructions using a decision model based on the aircraft's current flight status data, including speed, altitude, and attitude angle. The decision model dynamically adjusts path optimization weights based on EDAF and MFC. S5. Executing the landing path planning instruction through the flight control module, correcting the aircraft's pitch angle, roll angle, and thrust output in real time to ensure the aircraft lands stably along the planned path; S6. During the landing process, the system continuously monitors environmental changes and system operating status. If an anomaly is detected, the emergency decision-making mechanism is triggered to execute a go-around or emergency obstacle avoidance operation.

2. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: The EDAF calculation method in S3 specifically includes: S301, extracting the ambient temperature gradient matrix T based on infrared thermal imaging data grad ; S302. Calculate the wind speed disturbance coefficient W based on millimeter wave radar data dist ; S303. Calculate EDAF using the following formula: Among them, α and β are preset empirical coefficients, and the T grad Indicates the temperature change rate per square meter on the runway, in ℃ / m 2 °, W dist It represents the wind speed component perpendicular to the runway, in m / s.

3. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: The MFC calculation method in S3 is specifically summarized as follows: S311. Assign confidence scores C to the detection results of optical cameras, infrared thermal imagers, and millimeter-wave radars. opt 、C inf 、C rad ; S312. Calculate the multimodal fusion confidence MFC using the following formula: Among them, C opt ,C inf ,C rad ∈[0,1].

4. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: Said S3 further comprises: The deep learning model adopts the improved YOLOv8 algorithm and improves the accuracy of small target detection by introducing a channel attention mechanism.

5. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: The S4 specifically includes: The generation of the path planning instructions is based on the model predictive control MPC algorithm, whose objective function is: Among them, e k is the path tracking error at step k, Δu k is the rate of change of the control variable, λ is the smoothing factor, and N is the prediction time domain step size.

6. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: Said S5 further comprises: The flight control module adopts a redundant PID controller, and its control law is: Among them, K p , K i , K d is the controller gain parameter, which is dynamically adjusted according to EDAF.

7. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: Said S6 further comprises: The emergency decision mechanism is based on a decision tree model, and the node rules of the decision tree include: if the MFC is lower than a threshold θ and the EDAF is higher than a threshold γ, a go-around instruction is triggered.

8. The method for automatic aircraft landing based on multimodal image recognition according to claim 2, characterized in that: include: The values ​​of the empirical coefficients α and β are obtained through offline reinforcement learning optimization, and the optimization goal is to minimize the landing path deviation.

9. The method for automatic aircraft landing based on multimodal image recognition according to claim 3, characterized in that: The confidence score further includes: The confidence score C opt 、C inf 、C rad The calculation is based on the matching degree between the detection target and the prior map. The matching degree formula is: Among them, P detected To detect the target position, P prior is the corresponding target position in the prior map, D max is the maximum allowable position deviation.

10. The method for automatic aircraft landing based on multimodal image recognition according to claim 1, characterized in that: Said S2 further comprises: The spatiotemporal alignment adopts a SLAM algorithm based on feature point matching, which specifically includes: S201, extracting ORB feature points from the multimodal image; S202, eliminating mismatched points using the RANSAC algorithm; S203: Construct a 6-DOF posture transformation matrix to achieve spatiotemporal synchronization of multi-sensor data.

Images