Flight Delay Prediction Method Based on Multimodal Data Fusion and Large Model Fine-tuning

Abstract

This invention belongs to the field of flight delay prediction technology and discloses a flight delay prediction method based on multimodal data fusion and large model fine-tuning. The method first establishes a state-space model for flight delay prediction. By dividing flight operation into stages and analyzing the generation and propagation of delays in each stage, it effectively captures the dynamic and stochastic characteristics of flight delays, thus focusing on the system state and observation data crucial for prediction. Secondly, the LoRA method is used to fine-tune the Qwen large model, utilizing historical flight data for fine-tuning and combining it with real-time flight data for real-time delay prediction, thereby improving the prediction accuracy and response speed of the large model. Based on the prediction results obtained from the fine-tuned Qwen large model inference, a weighted fusion is performed with the processed state-space model estimate to obtain the final flight delay prediction result, effectively ensuring the accuracy and reliability of the prediction results.

Description

Technical Field

[0001] This invention belongs to the field of flight delay prediction technology, and specifically relates to a flight delay prediction method based on multimodal data fusion and large model fine-tuning. Background Technology

[0002] Flight delay prediction offers a more reliable flight management approach due to its ability to improve air transport efficiency, reduce passenger wait times, optimize airline operating costs, and enhance airport resource utilization. Fine-tuning models using real-time data is a common method in flight delay prediction, especially in complex, large-scale delay scenarios. Combining real-time flight data with a state-space model simplifies the system architecture and improves prediction accuracy. Currently, Support Vector Regression (SVR) and time series analysis are frequently used in flight delay prediction because they can learn simple delay patterns and directly map historical data to predicted results. Furthermore, methods based on large-scale model fine-tuning can accumulate experience through continuous interaction with real-time flight data to optimize the prediction model, thereby achieving a more efficient and flexible delay prediction process, particularly in complex environments and tasks.

[0003] Most existing flight delay prediction methods employ a single model trained on a single dataset. While this approach can simulate delay scenarios to some extent, the diversity and complexity of the data often fail to fully reflect real-world variations. Due to a lack of sufficient training data scenarios, multi-source data fusion, and unforeseen circumstances, the efficiency and quality of experience collection are limited, leading to poor model performance when facing real-world uncertainties. Furthermore, acquiring large-scale real-world flight data is costly, limiting the avenues for experience acquisition. Therefore, effectively expanding and diversifying training scenarios and enhancing the ability to extract information from multi-source data has become a major challenge for the application of multimodal data fusion methods in flight delay prediction.

[0004] For dynamic data, data fusion and model calibration are crucial, but the problem of insufficient data fusion is particularly prominent in flight delay prediction scenarios. Existing methods, when faced with unknown delay environments or complex scenarios, often tend to rely on previously learned suboptimal strategies, making it difficult to escape local optima. Especially when dealing with sparse observation data or dynamic environments, the ability to fuse dynamic data is insufficient, and the model is prone to falling into the trap of "premature convergence." Furthermore, since new data fusion strategies may come with high risks, especially in time-sensitive applications like flight delay prediction, models tend to adopt conservative strategies, further exacerbating the insufficient fusion. Therefore, how to enhance the model's data fusion capability, balance fusion and accuracy, and avoid falling into local optima is another important challenge for dynamic data fusion in flight delay prediction. Summary of the Invention

[0005] To address the shortcomings of existing flight delay prediction models that fail to adequately reflect the dynamic and stochastic nature of flight operations, this invention proposes a flight delay prediction method based on multimodal data fusion and large model fine-tuning. This method calibrates the parameters of the large model using real-time flight data and LoRA fine-tuning to improve the prediction accuracy and adaptability of the large model. Simultaneously, it combines the estimated values ​​from the state-space model for weighted fusion, which helps to improve the accuracy and reliability of flight delay prediction.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The flight delay prediction method based on multimodal data fusion and large model fine-tuning includes the following steps:

[0008] Step 1. Establish a state-space model for flight delay prediction;

[0009] The state-space model uses dynamic systems methods to associate the potential states of flight delays with observable data, and uses state space to describe the state transition behavior of dynamic systems and the relationship between state transitions and observed variables.

[0010] Step 2. Perform multi-source data processing and design prompt words;

[0011] First, historical flight data from the airport is collected and multi-source data processing is performed. This involves first stratifying the collected data according to different scenarios and modalities, establishing the correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. Next, prompt word templates are designed and the multi-source data is structured according to the prompt word templates.

[0012] Step 3. Fine-tune the Qwen large model based on the LoRA method, only updating the low-rank adapter parameters. By processing multi-source historical flight data, setting loss functions and optimization strategies, the Qwen large model is adapted to the flight delay prediction task.

[0013] Step 4. Perform dynamic prediction of flight delays;

[0014] Collect real-time flight data from the airport, perform multi-source data processing as in step 2, generate prompt words according to the prompt word template, use the fine-tuned Qwen large model to infer the real-time flight delay prediction result, and perform weighted fusion with the state space model estimate obtained by processing the real-time flight data in step 1 to obtain the final flight delay prediction result.

[0015] The present invention has the following advantages:

[0016] As described above, this invention discloses a flight delay prediction method based on multimodal data fusion and large-scale model fine-tuning. First, this invention establishes a state-space model for flight delay prediction. By dividing flight operations into stages and analyzing the generation and propagation of delays in each stage, it effectively captures the dynamic and stochastic characteristics of flight delays, thus focusing on the system state and observational data crucial for prediction. This modeling approach improves the understanding of complex delay phenomena and helps make more accurate predictions in dynamic environments. Second, this invention uses the LoRA method to fine-tune the Qwen large-scale model, utilizing historical flight data to fine-tune the large model and combining it with real-time flight data for real-time delay prediction, improving the prediction accuracy and response speed of the large model. Furthermore, the prediction model in this invention incorporates incremental learning and multimodal data processing and fusion operations, enhancing the adaptability and accuracy of the prediction model. Incremental learning continuously learns delay patterns from historical data to generate internal prediction logic, thereby continuously optimizing the generated prediction results. This prediction logic guides the system to focus on key factors during the prediction process, avoiding getting trapped in local optima. Simultaneously, by combining real-time data, it enhances the prediction depth and breadth of the model in complex environments, promoting model optimization. This invention applies a flight delay prediction method based on multimodal data fusion and large model fine-tuning in a real airport environment. By employing data fusion technology to integrate multi-source data, it ensures the consistency of prediction results and effectively reduces bias caused by data noise. This method improves the stability and performance of the prediction system in complex environments, increases prediction efficiency, and comprehensively enhances the accuracy and reliability of the prediction system. Airport environments are complex and diverse, encompassing various scenarios such as single-aircraft multi-tasking, continuous arrivals and departures, and delays within the airport network. Coupled with high-density flight traffic, random weather changes and airspace restrictions, ground traffic congestion, and emergencies, the flight delay prediction method proposed in this invention ensures that the prediction system can generate accurate prediction results based on real-time flight data while adhering to aviation regulations. It demonstrates excellent prediction performance and decision support capabilities in the ever-changing airport environment, thereby achieving stable and efficient flight management. Attached Figure Description

[0017] Figure 1 This is a flowchart of the flight delay prediction method based on multimodal data fusion and large model fine-tuning in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the single-machine multi-tasking delay waveform in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of continuous inbound flight delays in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the propagation analysis of OD flight delays in an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram illustrating the fine-tuning of a large model based on the LoRA method in an embodiment of the present invention;

[0022] Figure 6 This is a framework diagram of the flight delay prediction method based on multimodal data fusion and large model fine-tuning in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0024] Example 1

[0025] This embodiment describes a flight delay prediction method based on multimodal data fusion and large model fine-tuning. This method introduces a state-space model for flight delay prediction, strengthening the correlation extraction between system state and observed data, and improving the system's ability to understand key delay information. Furthermore, this invention designs a real-time data fusion architecture for flight delay prediction, where the data fine-tuning architecture fine-tunes the Qwen large model using the LoRA method, enabling the large model to predict flight delays and ensuring the model's accuracy and generalization in delay prediction. Additionally, this invention generates the final prediction result by combining the real-time prediction results of the large model with the state-space estimation values ​​in a weighted manner, guiding the system to focus on key factors during the prediction process, improving prediction accuracy, and effectively optimizing the prediction strategy. Finally, a modular design is used to integrate various modules (real-time data acquisition and processing module, large model fine-tuning module, system state-space model prediction module, real-time data prediction module, etc.) Figure 6 (As shown) to achieve the fusion of multimodal data, thereby improving the stability and performance of the prediction system in complex environments.

[0026] like Figure 1 As shown, the flight delay prediction method based on multimodal data fusion and large model fine-tuning includes the following steps:

[0027] Step 1. Establish a state-space model for flight delay prediction. The state-space model uses dynamic systems methods to associate the potential states of flight delays (such as trends and fluctuations) with observable data (such as weather and traffic). It uses state space to describe the state transition behavior of the dynamic system and the relationship between state transitions and observed variables (i.e., the evolution of the delay state is described by state transition equations, and the hidden states are connected with the observed data by observation equations).

[0028] Specifically, this embodiment includes three delay scenarios: single-aircraft multi-tasking scenario, continuous arrival and departure scenario, and airport network delay scenario. The process of establishing the state space model for flight delay prediction is as follows:

[0029] This paper analyzes the event sequence of flight delays affecting a single aircraft performing multiple tasks consecutively, and establishes the state equation and observation equation for single-aircraft multi-task flight delays. It also analyzes the event sequence of flight delays affecting multiple aircraft arriving and departing consecutively, and establishes the state equation and observation equation for consecutive arrival and departure flight delays. Furthermore, it analyzes the source airport where the flight delay occurs and the destination airport affected by the delay propagation from the source airport, and establishes the state equation and observation equation for delays within the airport network.

[0030] This invention demonstrates a single-machine multi-tasking delay waveform, as shown in the schematic diagram below. Figure 2 As shown. Suppose an aircraft performs multiple consecutive flights. When a preceding flight experiences a departure delay, the process of the delay propagating to downstream flights is as follows: Figure 2 As shown, and These represent the planned departure and arrival times, respectively. and These represent the actual departure and arrival times, respectively. Let the departure event be... The port entry incident is Then, the sequence of discrete arrival and departure events that an aircraft performs in a day can be represented as follows: Furthermore, the delay status of the next event depends only on the delay status of the current event and is independent of the delay status of past events. Using a state-space model to model the above events, the flight delay state-space model for a single-aircraft multi-tasking scenario is obtained as follows:

[0031] .

[0032] in, The current flight status is a delay variable. Let be the delay variable of the previous state. Random delays caused by uncertain factors Random delays caused by uncertain factors This is a measurement of flight delays. The noise being measured is white noise with a mean of zero. This is process noise.

[0033] We use a finite mixture model to represent the probability density function of random delays. Let the probability density function of the scene delay be... Depend on It is a mixture of several branches. Mixing density at each point It can be represented as:

[0034] .

[0035] Where the parameter vector ; and , is the The weights of the mixed branches; The parameter is The A density function.

[0036] If we assume If is a density function composed of a set of normally distributed mixtures, then .

[0037] in The mean, Let covariance matrix be the variance matrix. This represents the actual delay time for the i-th sample point.

[0038] The expectation-maximization algorithm is used to estimate the model parameters, first based on the parameter vector. The current value is used to continuously estimate the hidden vector. The expected value is recalculated based on the maximum likelihood hypothesis. Through repeated iterations of the E-step and M-step, the result will be obtained. The local optimal solution.

[0039] For the parameter estimation problem of a finite mixture distribution model of field delay, a parameter vector is constructed. Log-likelihood function:

[0040] .

[0041] The same method can be used to model the random delays generated by the aircraft during flight, and construct a complete state-space model for delay prediction under single-aircraft multi-tasking conditions.

[0042] This invention illustrates a continuous inbound flight delay wave, as shown in the schematic diagram below. Figure 3 As shown, this paper discusses the impact of delays between adjacent flights on a single airport flight schedule. Two adjacent aircraft on the airport flight schedule are considered as the basic structural unit of operation, called an operating aircraft group, where the first aircraft is the leading aircraft and the second aircraft is the following aircraft.

[0043] Let the port arrival delay be The minimum interval between the preceding and following aircraft is There is a buffer time between the two aircraft. , This refers to the time the following flight was delayed due to the delay of the preceding flight; it is the actual arrival time. Included In the middle, and must be satisfied. Assuming there is no priority relationship between the leading and following aircraft in the flight group, the process by which the arrival delay of the leading aircraft affects the following aircraft can be described by... Figure 3The above events are modeled using a state-space model, resulting in the following state-space model for flight delays under continuous arrival and departure scenarios:

[0044] .

[0045] in , Indicates the first , No. The number of flight delays affected; For the first The flight and the The planned interval between flights, i.e., the buffer space This can be calculated based on the airport's flight schedule; Minimum aircraft spacing may be implemented according to regulations; Random delays caused by uncertain factors; To delay the measurement value; For process noise, The noise from the previous flight is affecting the current flight. For noise measurement, all noise is white noise with a mean of zero.

[0046] By analyzing delay propagation patterns and event sequences within the airport network, a state-space model of delays within the airport network is established. This invention demonstrates the propagation analysis of OD (Original Delay) flight delays, as illustrated in the diagram below. Figure 4 As shown. The airport experiencing widespread flight delays is called the source airport, and the airports affected by the delay propagation from the source airport are called destination airports. These two airports together constitute a dynamic system called the OD (Original Departure) airport. For the source airport, delayed flights will affect the operation of its subsequent scheduled flights. Between the source and destination airports, delayed departing flights at the source airport will, to some extent, affect arriving flights at the destination airport. The OD airport delay propagation analysis under widespread delays is as follows: Figure 4 As shown.

[0047] exist Figure 4 In the diagram, solid arrows indicate the path of delay propagation. Flights that are sequential in time are called preceding and following flights; aircraft that perform consecutive flight missions are called preceding and following flights.

[0048] In actual large-scale flight delays, because preceding and subsequent flights share the same aircraft, a delayed arrival flight may directly propagate the delay to subsequent flights. Similarly, because flights on the timetable have a sequential order, a delayed arrival / departure flight may propagate the delay to subsequent arrival / departure flights.

[0049] Establish a state-space model for delay propagation in field O. The current delay state in field O depends on the delay propagation state at the previous time step, that is, the number of delayed flights at the current time step depends on the propagation capability of the delayed flights at the previous time step. The state equation is expressed as:

[0050] .

[0051] in, Let O be the state variable within the field at time t; The state variable from the previous moment; The propagation of random delays within the airport caused by uncertainties during the flight's arrival and departure process; This is process noise, which is white noise with a mean of 0. Let be the pre-delay propagation rate within Airport O, representing the probability that a delay in an arriving flight propagates to a subsequent departing flight, causing that flight to also be delayed. Because the propagation of pre-delays is discontinuous in time—meaning a delay in one flight within the current time period does not necessarily mean a delay in the next time period—the pre-delay propagation rate is consistent across all time periods within the same airport. ; For O field The forward-to-back departure delay propagation rate for a given time period indicates the propagation of delays in forward-departing flights from Airport O to subsequent-departing flights, and ultimately, their delays. Indicates the O field The degree of in-field delay propagation during a given time period is defined as the sum of the two delay propagation rates mentioned above.

[0052] The observation equation is defined as:

[0053] .

[0054] in, To delay the measurement value, This represents measurement noise, which is white noise with a mean of 0.

[0055] Establish a state space model for delay propagation between OD fields. There is delay propagation between upstream and downstream flights and delay propagation between consecutive arriving flights in D field.

[0056] The current arrival delay status at airport D depends on the arrival delay propagation status of the previous time. In other words, the number of delayed flights at the current time depends on the propagation capability of the delayed flights at the previous time. The state equation is expressed as:

[0057] .

[0058] in, Let be the state variables between the OD fields at time t; The state variable from the previous moment; Propagation of random in-flight delays caused by uncertainties during the period from departure to arrival of a flight; This represents process noise, which is white noise with a mean of 0. Let be the inter-airport upstream-downstream delay propagation rate, representing the probability that a flight delayed departing from airport O will become an arriving delayed flight at airport D, and is proportional to . Consistency is achieved by ensuring that the pre-order to post-order delay propagation rate is consistent across all time periods. ; For field D The forward-to-backward arrival delay propagation rate for a given time period indicates the propagation of delays from flights originating from Airport O to flights originating from Airport D, causing their own delays. Indicates the OD field between The degree of inter-field delay propagation during a given time period is defined as the sum of the two delay propagation rates mentioned above. The observation equation is defined as:

[0059] .

[0060] in, For delayed measurement values; For state variables; This is process noise, which is white noise with a mean of 0.

[0061] This invention addresses the problem of establishing a flight delay prediction model by employing a state-space model to describe the state transition behavior of a dynamic system and the relationship between state variables and observed variables. By using the state-space model to treat flight delays as a dynamic system, it provides a hierarchical logical basis for the processing in step 2, and at the same time provides interpretability support for prediction fusion in subsequent steps.

[0062] Step 2. Perform multi-source data processing and design prompt words.

[0063] First, historical flight data from the airport is collected and multi-source data processing is performed. This involves first stratifying the collected data according to different scenarios and modalities, establishing the correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. Next, prompt word templates are designed and the multi-source data is structured according to the prompt word templates.

[0064] The historical flight data here is mainly used for training large models to learn how to predict delay durations based on multimodal data and form basic prediction capabilities; real-time flight data is used to make instant delay predictions for the current real-time scenario using the trained large model.

[0065] The historical flight data collected here includes, for example, minute-level weather data from airport weather stations, flight traffic data, historical flight schedules, flight execution records, and expert delay analysis reports.

[0066] The different scenarios mentioned in this embodiment specifically refer to single-machine multi-tasking scenarios, continuous arrival and departure scenarios, and airport network delay scenarios. The different modalities specifically refer to text modality, temporal modality, and structured modality.

[0067] First, historical flight data is collected and processed in layers, as follows:

[0068] The system collects weather data updated every minute by the airport weather station, such as wind speed, rainfall, and visibility; traffic flow data transmitted in real time by the air traffic control system, such as flight trajectories, flight density, and runway occupancy time; and data on taxiing time and runway usage recorded by the airport ground operations system. It also collects flight plans and execution records from the past three years, including planned departure / arrival times, actual departure / arrival times, and delay duration data, as well as textual analysis reports by experts on historical delay events. The collected data is segmented into single-aircraft multi-task scenarios (multiple flight sequences operated continuously by the same aircraft daily), continuous arrival and departure scenarios (multiple flights arriving or departing from adjacent airports), and airport network delay scenarios (delay propagation between source and destination airports). Furthermore, it is divided into three modalities: text modality (such as expert analysis text and weather report text), time-series modality (weather data and traffic data arranged in time series), and structured modality (delay trends, fluctuation coefficients, random delay mean, and random delay variance parameters in a state-space model). This establishes a precise correspondence between data and scenarios, improves data processing efficiency through modal layering, adapts to the processing needs of different data types, and supports multi-source information integration using prompt word templates.

[0069] Then, the collected data undergoes data cleaning and feature extraction, as follows:

[0070] For numerical data (such as wind speed and rainfall), the 3σ principle is used to identify outliers. For example, data with rainfall greater than 50 mm / h and wind speed exceeding 60 m / s are judged as outliers and removed. The judgment principle is as follows:

[0071] .

[0072] in, Assume that the rainfall data for a certain period follows a normal distribution, with a mean of 1 / 2. The standard deviation is If the rainfall at a certain moment If the above formula is not met, it is judged as an outlier (such as extreme rainfall caused by rainstorms).

[0073] Similarly, the same applies to wind speed data.

[0074] Missing weather or traffic data is supplemented using linear interpolation, the specific interpolation method of which is as follows:

[0075] For time series data ,..., If at time Missing values ​​exist Then through time and Observations at and Perform the calculation.

[0076] Data-type data refers to meteorological data, flow data, and operational data. These data can be directly quantified into continuous or discrete numerical values. They are not considered a modality, but rather a type of data processing.

[0077] The text data undergoes simplified Chinese standardization and word segmentation, filtering out low-frequency words that appear less frequently (in this embodiment, words appearing less than 5 times are filtered out). The specific processing procedure is as follows:

[0078] The input original text set is converted into simplified Chinese to obtain a simplified text list; for each simplified text, special symbols, punctuation marks, etc. need to be removed, and the processed text is segmented into word sequences to obtain a word set of all texts;

[0079] Based on the obtained word segmentation set, the frequency of each word is counted, a word frequency dictionary is generated, and words with a frequency of less than 5 are filtered out to form a low-frequency word list. For each simplified text, words are re-segmented, and words not in the low-frequency word list are retained. The filtered words are concatenated with spaces to obtain the processed text list.

[0080] After the above processing, text features are constructed, and the expert analysis text is labeled according to delay types such as weather causes, mechanical failures, and traffic congestion. A token sequence of length 256 is then generated using the Qwen large model's tokenizer.

[0081] The preprocessed simplified Chinese text is segmented into a word sequence using the Qwen large model's word segmenter, resulting in a word list. Each word in the word list is then mapped to a numeric ID that the model can recognize, generating a numeric ID sequence.

[0082] If the number sequence is greater than 256, then the stage sequence is used, and only the first 256 words are retained. If the number sequence is less than 256, then padding IDs are used to fill the sequence to 256, resulting in a standardized ID sequence with a fixed length of 256.

[0083] For the standardized ID sequence, the padding part needs to be marked as 0 (representing padding bits ignored by the model), and the rest needs to be marked as 1 (representing real words that the model needs to focus on). Based on this, a mask list is generated to obtain the attention mask, which is used to indicate the location of effective information when the model processes it.

[0084] For time-series feature extraction, a 12-hour sliding window is used to calculate statistical measures such as mean, variance, and maximum value for weather data. The specific operation process is as follows:

[0085] Assuming the time series data is .in Indicates time The window size for weather observations (such as rainfall, wind speed, etc.) is Then for the current moment... The observation interval included in the sliding window is: The observed value within the window is Various statistics can be calculated within the sliding window, such as:

[0086] .

[0087] in, Indicates time The average value of the data within the time window is used to reflect recent weather trends.

[0088] The traffic data is decomposed into daily and weekly components using Fourier transform.

[0089] During structured feature processing, the delay trend (e.g., delay of 15 minutes), fluctuation coefficient (e.g., 0.3), random delay mean, and random delay variance parameters output by the state-space model are converted into JSON format for easy parsing by the Qwen large model.

[0090] Next, we design the prompt word template. Flight delay prediction involves multi-source data such as weather, air traffic, and status parameters, and includes different modalities such as text, time series, and structured data. The prompt word template uses categorical fields to structure the scattered information, enabling large models to uniformly understand different types of input and solving the data fragmentation problem. The specific prompt word template is as follows:

[0091] [Scenario Type]: {Single-machine multi-tasking / continuous arrival / departure / airport network};

[0092] [Real-time Flight Data]: Weather = {Wind Speed, Rainfall, Visibility}, Traffic Flow = {Runway Occupancy Rate, Flight Density}, Status Parameters = {Delay Trend, Fluctuation Coefficient};

[0093] [Historical Case]: {Summary of M near-term delay events in similar scenarios} where M is a natural number, for example, 3;

[0094] [Prediction Target]: The probability and duration of delay in the next N hours, where N is a natural number, for example, a value of 2.

[0095] Since the delay propagation mechanisms differ across different operating scenarios in Step 1, scenario layering allows data to be split according to actual scenarios. This facilitates the subsequent use of a large model to learn delay patterns for specific scenarios. The data is layered according to scenarios such as single-machine multi-tasking and modalities such as text, time series, and structured data. Furthermore, the collected data undergoes data cleaning and feature extraction, and a prompt word template is designed to structure the multi-source heterogeneous data, making it compatible with the input format of the Qwen large model. This template utilizes the underlying capabilities of the Qwen large model to autonomously define field definitions and data organization methods to adapt to the prediction of specific tasks.

[0096] Step 3. Fine-tune the Qwen large model based on the LoRA method, only updating the low-rank adapter parameters. By processing multi-source historical flight data, setting loss functions and optimization strategies, the Qwen large model is adapted to the flight delay prediction task.

[0097] In this embodiment, the Qwen large model (e.g., the Qwen-7B model) is selected as the basic flight delay prediction model.

[0098] Since flight delay prediction involves multi-source heterogeneous information such as weather text, traffic data, and expert analysis, large models can efficiently capture the complex relationships between different modalities through self-attention mechanisms. For example, they can extract key semantic features that affect delays from weather description texts and integrate them with numerical traffic data to learn delay patterns that are difficult for traditional models to discover.

[0099] Meanwhile, the generalization ability formed by the large model in the pre-training of massive amounts of data can adapt to the dynamic changes in flight operation scenarios, such as extreme weather or temporary air traffic control, and dynamically adjust the prediction logic through contextual understanding.

[0100] Qwen's large-scale model optimizes word vector representations for the characteristics of the Chinese language, enabling it to more accurately understand specialized terms in the aviation field (such as "runway occupancy rate" and "air traffic control") and avoid semantic ambiguity. Furthermore, Qwen supports efficient prompt word engineering, allowing the integration of state-space model parameters, real-time flight data, and other information into the input through structured prompt word templates.

[0101] like Figure 5 As shown, the process of fine-tuning a large model using the LoRA method is as follows:

[0102] LoRA (Low-Rank Adaptation) is a method that, based on freezing the weights of the pre-trained model, significantly reduces the number of trainable parameters in downstream tasks by injecting trainable low-rank decomposition matrices into each layer of the Transformer architecture. Let the weight matrix of a certain layer in the Qwen model be... , express The matrix, when fine-tuning all parameters, yields the incremental parameter matrix as follows: .

[0103] The LoRA method assumes It is low-rank, meaning it can be approximately decomposed into the product of two matrices A and B with small parameter values. ,in , Its initialization follows a normal distribution. This represents a normal distribution with a mean of 0 and a variance of . ; An initial matrix consisting entirely of zeros. In this way, the number of parameters that originally needed to be trained during full parameter fine-tuning is d. 2 However, after adopting LoRA, the number of parameters that need to be trained becomes 2dr.

[0104] During training, the LoRA method only calculates the gradients of matrices A and B, while freezing the pre-trained weights. The gradient is not updated; during the inference phase, the BA can be directly merged into... In the context of inference, the model's weights are: .

[0105] Flight delay prediction is essentially a continuous value regression problem (e.g., predicting a delay of 5 minutes). The mean squared error loss (MSE), calculated by the squared difference between the predicted and actual values, effectively measures the model's accuracy in fitting specific numerical values. A larger error results in a larger loss, guiding the model towards optimization that reduces absolute error. The specific formula is as follows:

[0106] .

[0107] in, For loss function, For the sample size, The delay time predicted by the model. The actual delay time is represented by MSE. At the same time, MSE is a differentiable convex function, which makes it easy to use optimization algorithms such as gradient descent to solve for the optimal parameters. It is also sensitive to outliers, which can encourage the model to focus on the prediction accuracy of severe delay scenarios.

[0108] Step 4. Perform dynamic prediction of flight delays and model calibration.

[0109] This invention achieves real-time inference and dynamic prediction by combining real-time flight data, the predictive power of large models, and the physical logic of state-space models. Furthermore, it addresses model prediction bias caused by dynamic changes in the flight operating environment through incremental learning and continuously optimizes the model by learning new data distributions to achieve model calibration.

[0110] Real-time flight data from the airport is collected, preprocessed, and prompts are generated according to the prompt template. The fine-tuned Qwen large model is used to infer the real-time flight delay prediction result. The real-time flight delay prediction result is then weighted and fused with the state space model estimate obtained from the real-time flight data in step 1 (the model construction in step 1 is based on a physically and logically independent modeling process. The estimated value of the delay state can be directly calculated through the state space model in step 1 and the preprocessed real-time flight data) to obtain the final flight delay prediction result.

[0111] Specifically, the preprocessing of real-time flight data, including collected weather, air traffic, flight operation status, and flight schedules, is as follows: Real-time weather data (updated minute-by-minute), airport runway occupancy rate, flight density and traffic flow data, and current flight operation status data are acquired through the airport meteorological monitoring system and air traffic control management system. Multi-source data processing is performed according to step 2. Then, prompts for the large model are generated according to the designed prompt template to facilitate better input into the large model. Simultaneously, for missing data in real-time flight data, such as when traffic flow data for a certain period is not available in time, prior estimates from the flight delay prediction state space model established in step 1 are used to fill in the missing data. For example, the missing traffic flow value is predicted based on the traffic flow trend of the previous hour, ensuring that the data input into the large model is complete and conforms to the format requirements, providing a reliable input basis for subsequent delay prediction inference.

[0112] Then, state-space fusion is performed by combining real-time flight data. This embodiment constructs a weighted fusion model to organically combine the prediction results of the large model with the estimated values ​​of the state-space model, as shown in the following formula:

[0113] .

[0114] Final flight delay prediction results Real-time flight delay prediction results from a large model for the current scenario , and the state-space model estimate for the corresponding scenario The weighted average is obtained.

[0115] The current scenarios include single-machine multi-tasking scenarios, continuous arrival and departure scenarios, or airport network delay scenarios.

[0116] α is a dynamically adjusted weight, and its value is automatically optimized based on data completeness: when real-time flight data such as weather and air traffic are complete, α is increased to give full play to the Qwen large model's ability to recognize patterns in multimodal data; when data is missing, α is decreased to increase the prior estimation weight of the state space model and make up for the lack of data by taking advantage of its dynamic system modeling.

[0117] The weighted fusion mechanism proposed in this embodiment not only retains the ability of the Qwen large model to capture complex nonlinear relationships, but also enhances the interpretability of the prediction results by leveraging the physical logic of the state space model for flight delay prediction. Especially in special scenarios such as extreme weather or sudden traffic congestion, it can avoid prediction bias caused by insufficient training data in the Qwen large model by extrapolating the trend of the state space model, thereby improving the overall stability and accuracy of the prediction.

[0118] In addition, step 4 introduces the concept of incremental model calibration learning. Through a caching mechanism, real-time prediction data and actual delay results are continuously collected. When the sample size of a single scenario reaches a preset threshold, the LoRA incremental fine-tuning process is automatically triggered.

[0119] Specifically, the LoRA incremental fine-tuning process is automatically triggered when the cumulative sample size for a single scenario reaches a preset threshold. For example, when the cumulative sample size for a single-aircraft multi-task scenario reaches 200 consecutive flight delay samples of the same aircraft; when the cumulative sample size for a continuous arrival / departure scenario reaches 300 adjacent flight delay samples at airports; when the cumulative sample size for an airport network scenario reaches 500 delay samples between source and destination airports; or when extreme weather occurs continuously (such as continuous heavy rain or other extreme weather in the past), the new samples (predicted data + actual delay results) for that scenario are extracted from the cache, reorganized according to the prompt word template format in step 2, and the parameters of matrices A and B in step 3 are optimized with the actual delay results of the new data as the target, minimizing the deviation between the predicted delay duration and the actual value. During this process, the weights of the pre-trained model remain frozen, and only a small number of new parameters are added iteratively to adapt to the new data distribution. For example, when extreme weather occurs consecutively, the correlation pattern between the cause of delay and the duration may change significantly. If the original model is not updated in time, it will lead to prediction bias due to reliance on historical distribution. In this case, incremental calibration is required. Some rainstorm delay samples are extracted, organized according to the prompt word template, and the parameters of matrices A and B in step 3 are updated. The actual delay duration is used as the target, and the deviation between the predicted delay duration and the actual value is minimized using the loss function MSE mentioned in step 3. The new correlation between "rainfall and delay duration" is learned, enabling the model to quickly learn the new weather and delay correlation features, avoiding the time and resource consumption of retraining the entire model, thereby achieving dynamic optimization of prediction capabilities.

[0120] Furthermore, to verify the effectiveness of the flight delay prediction method proposed in this invention, a series of experiments were conducted on continuous mission flight delay prediction, continuous arrival flight delay prediction, and airport network delay prediction, successfully verifying the feasibility and accuracy of the flight delay prediction method based on multimodal data fusion and large model fine-tuning.

[0121] This invention focuses on flight delay prediction based on multimodal data fusion and large model fine-tuning. By combining multimodal data processing, it significantly improves the accuracy and efficiency of predictions. Through the application of data fusion and LoRA fine-tuning of large models, the system can process and calibrate large amounts of observation data in real time, accelerating prediction convergence and reducing the time and cost required to retrain a specialized large model. Furthermore, the modular architecture allows the system to operate on large datasets and in complex environments, further enhancing the adaptability and generalization ability of flight delay prediction. This invention also combines multi-source data processing operations and incremental learning methods to further enhance the diversity and effectiveness of the prediction model. The combination of expert analysis and historical data enables comprehensive analysis of historical data and expert modules, ensuring greater accuracy in the global prediction process. With the addition of incremental learning, the system not only improves prediction efficiency but also enhances the robustness and generalization ability of the model, making it perform better in complex flight delay prediction tasks. The design of the model fine-tuning module significantly improves prediction accuracy and comprehensively enhances the reliability and efficiency of the flight delay prediction system, enabling the prediction results to provide decision-making support for airlines or airports in practice.

[0122] Example 2

[0123] This embodiment 2 describes a flight delay prediction system based on multimodal data fusion and large model fine-tuning. This system is based on the same inventive concept as the flight delay prediction method based on multimodal data fusion and large model fine-tuning in embodiment 1.

[0124] like Figure 6 As shown, the flight delay prediction system based on multimodal data fusion and large model fine-tuning in this embodiment includes:

[0125] The system state-space model prediction module is used to establish a state-space model for flight delay prediction.

[0126] The state-space model uses dynamic systems methods to associate the potential states of flight delays with observable data, and uses state space to describe the state transition behavior of dynamic systems and the relationship between state transitions and observed variables.

[0127] The real-time data acquisition and processing module is used for multi-source data processing and prompt word design;

[0128] First, historical flight data from the airport is collected and multi-source data processing is performed. This involves first stratifying the collected data according to different scenarios and modalities, establishing the correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. Next, prompt word templates are designed and the multi-source data is structured according to the prompt word templates.

[0129] The large model fine-tuning module is used to fine-tune the Qwen large model based on the LoRA method;

[0130] When fine-tuning the Qwen large model based on the LoRA method, only the low-rank adapter parameters are updated. By processing multi-source historical flight data, setting loss functions and optimization strategies, the Qwen large model can be adapted to flight delay prediction tasks.

[0131] It also includes a real-time data prediction module for dynamic prediction of flight delays.

[0132] Collect real-time flight data from the airport, perform multi-source data processing as in step 2, generate prompt words according to the prompt word template, use the fine-tuned Qwen large model to infer the real-time flight delay prediction result, and perform weighted fusion with the state space model estimate obtained by processing the real-time flight data in step 1 to obtain the final flight delay prediction result.

[0133] It should be noted that the implementation process of the functions and roles of each functional module in the flight delay prediction system described in this embodiment 2 is detailed in the implementation process of the corresponding steps of the method in the above embodiment 1, and will not be repeated here.

[0134] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A flight delay prediction method based on multimodal data fusion and large model fine-tuning, characterized in that, Includes the following steps: Step 1. Establish a state-space model for flight delay prediction; The state-space model uses dynamic systems methods to associate the potential states of flight delays with observable data, and uses state space to describe the state transition behavior of dynamic systems and the relationship between state transitions and observed variables. We analyze the sequence of flight delay impact events when a single aircraft continuously performs multiple missions, and establish the state equation and observation equation for single-aircraft multi-mission flight delays. The sequence of flight delays affecting multiple aircraft arriving and departing in succession is analyzed to establish the state equation and observation equation for consecutive flight delays. The source airport where the flight delay occurs and the destination airport affected by the delay propagation from the source airport are analyzed to establish the state equation and observation equation for delays within the airport network. Step 2. Perform multi-source data processing and design prompt words; First, historical flight data from the airport is collected and multi-source data processing is performed. This involves first stratifying the collected data according to different scenarios and modalities, establishing the correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. Next, prompt word templates are designed and the multi-source data is structured according to the prompt word templates. Step 3. Fine-tune the Qwen large model based on the LoRA method, only updating the low-rank adapter parameters. By processing multi-source historical flight data, setting loss functions and optimization strategies, the Qwen large model is adapted to the flight delay prediction task. Step 4. Perform dynamic prediction of flight delays; Collect real-time flight data from the airport, perform multi-source data processing as in step 2, generate prompt words according to the prompt word template, use the fine-tuned Qwen large model to infer the real-time flight delay prediction result, and perform weighted fusion with the state space model estimate obtained by processing the real-time flight data in step 1 to obtain the final flight delay prediction result.

2. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1, characterized in that, In step 2, the process of collecting and stratifying historical flight data is as follows: Collect flight schedules and execution records, weather data, traffic data, and expert analysis reports on historical delay events from the past three years; The collected data was broken down into single-machine multi-tasking scenarios, continuous arrival and departure scenarios, and airport network delay scenarios. It was also divided into three modalities: text modality, time-series modality, and structured modality, to establish a precise correspondence between data and scenarios.

3. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 2, characterized in that, In step 2, the process of data cleaning and feature extraction of the collected historical flight data is as follows: For numerical data, outliers are identified and removed using the 3σ principle, and missing data is supplemented using linear interpolation. The text data is simplified and standardized, and then segmented to filter out low-frequency words. When constructing text features, the expert analysis text is labeled according to the delay type, and a token sequence of length 256 is generated by the tokenizer of the Qwen large model. In terms of time series feature extraction, a 12-hour sliding window is used to calculate statistics including mean, variance, and maximum value for weather data, and Fourier transform is used to decompose traffic data into daily and weekly periodic components. When processing structured features, the delay trend, fluctuation coefficient, random delay mean, and random delay variance parameters in the flight delay prediction state space model are converted into JSON format for easier parsing by large models.

4. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 2, characterized in that, In step 2, the design process of the prompt word template is as follows: Flight delay prediction involves multi-source data, including weather, traffic, and status parameters, and includes different modalities; prompt word templates structure scattered information through classification fields, enabling large models to uniformly understand different types of input; The prompt word template for the Qwen large model is as follows: [Scenario Type]: {Single-machine multi-tasking / continuous arrival / departure / airport network}; [Real-time Flight Data]: Weather = {Wind Speed, Rainfall, Visibility}, Traffic Flow = {Runway Occupancy Rate, Flight Density}, Status Parameters = {Delay Trend, Fluctuation Coefficient}; [Historical Case]: {Summary of nearly M delay events in similar scenarios}, where M is a natural number; [Prediction Target]: The probability and duration of delays in the next N hours, where N is a natural number.

5. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1, characterized in that, In step 3, the process of fine-tuning the large model using the LoRA method is as follows: Let the weight matrix of a certain layer in the Qwen large model be... ; in express The matrix, when fine-tuning all parameters, yields the incremental parameter matrix as follows: ; The LoRA method assumes It is low rank, meaning it can be decomposed into the product of two matrices A and B with small parameter counts; During training, the LoRA method only calculates the gradients of matrices A and B, while freezing the pre-trained weights. The gradient is not updated; during the inference phase, the BA is directly merged into... In the middle, that is, the weights of the model during inference are .

6. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1, characterized in that, In step 3, the loss function is set. The calculation formula is as follows: ； in For the sample size, The delay time predicted by the model. This refers to the actual delay time. In the loss function calculation, the mean squared error loss (MSE) is used to calculate the squared difference between the predicted value and the true value, which effectively measures the model's fitting accuracy to a specific numerical value. MSE is a differentiable convex function, which facilitates the use of gradient descent optimization algorithms to solve for the optimal parameters.

7. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1, characterized in that, In step 4, the preprocessing of the collected real-time flight data is as follows: The system acquires minute-by-minute updated weather data, airport runway occupancy rate, flight density and flow data, and current flight operation status data through the airport meteorological monitoring system and air traffic control flow management system. It then performs multi-source data processing operations according to the processing method in step 2, and generates prompt words for the large model according to the designed prompt word template. Meanwhile, to address any missing data in the real-time flight data, prior estimates from the flight delay prediction state space model established in step 1 are used to fill in the gaps, ensuring that the data input into the large model is complete and meets the format requirements.

8. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1, characterized in that, In step 4, the calculation process for the final flight delay prediction result is as follows: A weighted fusion model is constructed to organically combine the prediction results of the large model with the estimates of the state-space model, as shown in the following formula: ； Final flight delay prediction results Real-time flight delay prediction results from a large model for the current scenario , and the state-space model estimate for the corresponding scenario Weighted average; The current scenarios include single-machine multi-tasking scenarios, continuous arrival and departure scenarios, or airport network delay scenarios.

9. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 5, characterized in that, In step 4, a caching mechanism is used to store the predicted data and the actual delay results. When the number of samples in a single scenario reaches a preset number or extreme weather occurs consecutively, LoRA incremental fine-tuning is triggered to achieve dynamic optimization of the large model. The process is as follows: Extract the new samples for the current scenario from the cache, namely the cached prediction data and actual delay results, and reorganize them according to the prompt word template format in step 2. Use the actual delay results of the new data as the target to optimize the parameters of A and B in step 3 and minimize the deviation between the predicted delay time and the actual value. During this process, the weights of the pre-trained model remain frozen, and only a small number of new parameters are added to iteratively adapt to the new data distribution so that the model can quickly learn new weather and delay correlation features.

Images