Flight delay prediction method based on multi-modal data fusion and large model fine tuning

Abstract

The invention belongs to the technical field of flight delay prediction, and discloses a flight delay prediction method based on multi-modal data fusion and large model fine tuning. According to the method, firstly, a flight delay prediction state space model is established, and dynamic characteristics and random characteristics of flight delay are effectively captured by dividing flight operation stages and analyzing delay generation and propagation modes of each stage, so that system states and observation data which are crucial to prediction are focused on. Secondly, performing fine adjustment on the Qwen large model by adopting a LoRA method, performing fine adjustment on the large model by utilizing historical flight data, and performing real-time delay prediction in combination with real-time flight data so as to improve the prediction precision and response speed of the large model; a final flight delay prediction result is obtained by performing weighted fusion on a prediction result obtained by reasoning based on the fine-tuned Qwen large model and a state space model estimated value obtained through processing, and the accuracy and reliability of the prediction result are effectively ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of flight delay prediction, and in particular relates to a flight delay prediction method based on multimodal data fusion and large model fine-tuning. Background Art

[0002] Flight delay prediction offers a more reliable approach to flight management, as it can improve air transport efficiency, reduce passenger waiting time, optimize airline operating costs, and increase airport resource utilization. Fine-tuning models with real-time data is a common approach for flight delay prediction, especially in complex, large-scale delay scenarios. By combining real-time flight data with state-space models, it simplifies the system architecture and improves prediction accuracy. Currently, support vector regression (SVR) and time series analysis methods are commonly used in flight delay prediction because they can directly map from historical data to predictions by learning simple delay patterns. Large model fine-tuning methods, on the other hand, can optimize prediction models through continuous interaction with real-time flight data, accumulating experience and enabling a more efficient and flexible delay prediction process, especially in complex environments and tasks.

[0003] Most existing flight delay prediction methods use a single model trained on a single dataset. While this approach can simulate delay scenarios to a certain extent, the diversity and complexity of the data often fail to fully reflect real-world variations. The lack of sufficient training data scenarios, multi-source data fusion, and unexpected situations limits the efficiency and quality of experience collection, resulting in poor model performance when faced with real-world uncertainties. Furthermore, the high cost of acquiring large-scale, actual flight data limits the means of gaining experience. Therefore, effectively expanding and diversifying training scenarios and improving the ability to extract information from multi-source data pose major challenges for the application of multimodal data fusion methods in the field of flight delay prediction.

[0004] For dynamic data itself, data fusion and model calibration are key, but in the flight delay prediction scenario, the problem of insufficient data fusion is particularly prominent. When encountering unknown delay environments or complex scenarios, existing methods usually tend to use previously learned suboptimal strategies, making it difficult to escape the local optimal solution. Especially when faced with sparse observation data or dynamic environments, the fusion ability of dynamic data is insufficient, and the model is prone to falling into the trap of "premature convergence". In addition, because new data fusion strategies may be accompanied by high risks, especially in applications such as flight delay prediction that require extremely high timeliness, the model is more inclined to conservative strategies, further exacerbating the lack of fusion. Therefore, how to enhance the model's data fusion capabilities, balance fusion and accuracy, and avoid falling into local optimality is another important challenge for dynamic data fusion in flight delay prediction. Summary of the Invention

[0005] In response to the deficiency that existing flight delay prediction models cannot well reflect the dynamics and randomness of the flight operation process, the present invention proposes a flight delay prediction method based on multimodal data fusion and large model fine-tuning. This method calibrates the large model parameters through real-time flight data and LoRA fine-tuning to improve the prediction accuracy and adaptability of the large model. At the same time, it combines the estimated values ​​of the state-space model for weighted fusion, which is conducive to improving the accuracy and reliability of flight delay prediction.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions: The flight delay prediction method based on multimodal data fusion and large model fine-tuning includes the following steps: Step 1. Establish a flight delay prediction state space model; The state space model uses a dynamic system approach to associate the potential state of flight delays with observable data, and uses the state space to describe the state transition behavior of the dynamic system and the relationship between state transformations and observed variables; Step 2. Process multi-source data and design prompt words; First, historical flight data from the airport is collected and multi-source data processing is performed. This involves stratifying the collected data according to different scenarios and modalities, establishing a correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. A prompt word template is then designed and used to structure the multi-source data. Step 3. Fine-tune the Qwen large model based on the LoRA method, updating only the low-rank adapter parameters. By processing multi-source historical flight data and setting the loss function and optimization strategy, the Qwen large model is adapted to the flight delay prediction task. Step 4. Perform dynamic prediction of flight delays; Collect real-time flight data from the airport, perform multi-source data processing operations according to the method in step 2, and generate prompt words according to the prompt word template. Use the fine-tuned Qwen large model to infer the real-time flight delay prediction results, and perform weighted fusion with the state-space model estimate obtained by processing the real-time flight data in step 1 to obtain the final flight delay prediction results.

[0007] The present invention has the following advantages: As described above, the present invention describes a flight delay prediction method based on multimodal data fusion and large-scale model fine-tuning. First, the present invention establishes a flight delay prediction state-space model. By dividing flight operation phases and analyzing the generation and propagation of delays in each phase, it effectively captures the dynamic and stochastic characteristics of flight delays, thereby focusing on system states and observational data that are critical for prediction. This modeling approach improves understanding of complex delay phenomena and facilitates more accurate predictions in dynamic environments. Second, the present invention employs the LoRA method to fine-tune the Qwen large-scale model. This model is fine-tuned using historical flight data and combined with real-time flight data for real-time delay prediction, improving the large-scale model's prediction accuracy and response speed. Furthermore, the present invention incorporates incremental learning and multimodal data processing and fusion into the prediction model, enhancing its adaptability and accuracy. Incremental learning continuously learns delay patterns from historical data, generates internal prediction logic, and continuously optimizes the generated prediction results. This prediction logic guides the system to focus on key factors during the prediction process, avoiding local optima. Furthermore, the integration of real-time data enhances the prediction depth and breadth of the prediction model in complex environments, promoting model optimization. The present invention applies a flight delay prediction method based on multimodal data fusion and large model fine-tuning in an actual airport environment, adopts data fusion technology, integrates multi-source data, ensures the consistency of the prediction results, and effectively reduces the deviation caused by data noise. This method improves the stability and performance of the prediction system in complex environments, improves the prediction efficiency, and comprehensively enhances the accuracy and reliability of the prediction system. The airport environment is complex and diverse, including multiple scenarios such as single-aircraft multi-tasking, continuous arrival and departure, and delays within the airport network, accompanied by high-density flight traffic, random weather changes and airspace restrictions, ground traffic congestion, and emergency situations. The flight delay prediction method proposed in the present invention ensures that the prediction system can generate accurate prediction results based on real-time flight data while complying with aviation rules, and demonstrates good prediction effects and decision support capabilities in a changing airport environment, thereby achieving stable and efficient flight management. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Flowchart of a flight delay prediction method based on multimodal data fusion and large model fine-tuning in an embodiment of the present invention; Figure 2 Schematic diagram of a single-machine multi-task delay wave in an embodiment of the present invention; Figure 3 Schematic diagram of a continuous inbound flight delay wave in an embodiment of the present invention; Figure 4 Schematic diagram of OD flight delay propagation analysis in an embodiment of the present invention; Figure 5Schematic diagram of fine-tuning a large model based on the LoRA method in an embodiment of the present invention; Figure 6 This is a framework diagram of a flight delay prediction method based on multimodal data fusion and large model fine-tuning in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 This embodiment describes a flight delay prediction method based on multimodal data fusion and large model fine-tuning. The method introduces a flight delay prediction state space model, strengthens the correlation extraction between system status and observation data, and improves the system's ability to understand key delay information. In addition, the present invention also designs a real-time data fusion architecture for flight delay prediction, in which the data fine-tuning architecture fine-tunes the Qwen large model through the LoRA method, so that the large model has the ability to predict flight delays, ensuring the accuracy and generalization of the model's delay prediction. In addition, the present invention generates the final prediction result by combining the real-time prediction results of the large model with the state space estimation value in a weighted manner, guiding the system to focus on key factors in the prediction process, improving prediction accuracy, and effectively optimizing the prediction strategy. Finally, each module (real-time data acquisition and processing module, large model fine-tuning module, system state space model prediction module, real-time data prediction module) is combined through modular design. Figure 6 The fusion of multimodal data is achieved, which improves the stability and performance of the prediction system in complex environments.

[0010] like Figure 1 As shown in Figure 2, the flight delay prediction method based on multimodal data fusion and large model fine-tuning includes the following steps: Step 1. Build a state-space model for flight delay prediction. This model leverages a dynamic systems approach to link the underlying states of flight delays (such as trends and fluctuations) with observable data (such as weather and traffic). It uses state space to describe the dynamic system's state transitions and the relationship between state transformations and observed variables (i.e., describing the evolution of delay states through state transition equations and connecting latent states with observed data through observation equations).

[0011] Specifically, in this embodiment, there are three types of delay scenarios: a single-aircraft multi-task scenario, a continuous arrival and departure scenario, and an airport network delay scenario. The process of establishing the flight delay prediction state space model is as follows: The flight delay propagation event sequence of a single aircraft performing multiple tasks continuously is analyzed, and the state equation and observation equation of single-aircraft multi-task flight delay are established; the flight delay propagation event sequence of multiple aircraft continuously arriving and departing is analyzed, and the state equation and observation equation of continuous arriving and departing flight delay are established; the source airports where flight delays occur and the destination airports affected by the propagation of delays at the source airport are analyzed, and the state equation and observation equation of delays within the airport network are established.

[0012] The present invention shows a single machine multi-task delay wave, the schematic diagram of which is as follows Figure 2 Assume that an aircraft performs multiple flight missions in succession. When the preceding flight is delayed, the delay propagates to the downstream flights as shown in the following figure: Figure 2 As shown, and Represent the planned departure and arrival times, and Represent the actual departure and arrival times respectively. Let the departure event be , the port entry event is , then the sequence of discrete events of arrival and departure that an aircraft must perform in a day can be expressed as , and the delay status of the next event is only related to the delay status of the current event, and has nothing to do with the delay status of past events. Using the state space model to model the above events, the flight delay state space model for the single-aircraft multi-task case is as follows: .

[0013] in, is the current flight status delay variable, is the delay variable of the previous state, Random delays caused by uncertain factors, Random delays caused by uncertain factors, is the flight delay measurement value, is the measurement noise, which is white noise with zero mean, is the process noise.

[0014] Use the finite mixture model to represent the probability density function of random delays. Let the probability density function of field delay be Depend on A mixture of branches. The mixture density of the points It can be expressed as: .

[0015] The parameter vector ; and , is the The weight of the hybrid branch; The parameter is No. A density function.

[0016] If we assume is a density function consisting of a set of normal mixture distributions, then .

[0017] in is the mean, is the covariance matrix, Indicates the actual delay duration of the i-th sample point.

[0018] Use the expectation maximization algorithm to estimate the model parameters, first according to the parameter vector The current value of the hidden vector is continuously estimated Recalculate the maximum likelihood hypothesis with the expected value of , and after continuous iteration of the E step and the M step, we will get local optimal solution.

[0019] Aiming at the parameter estimation problem of the finite mixture distribution model of surface delay, the parameter vector The log-likelihood function of : .

[0020] The same method can be used to model the random delays caused by an aircraft during flight and construct a complete delay prediction state space model for a single aircraft with multiple missions.

[0021] The present invention shows a continuous inbound flight delay wave, the schematic diagram of which is as follows Figure 3 As shown in Figure 2, the delay spread issue between adjacent flights on a single airport's flight schedule is discussed. Two adjacent aircraft on an airport's flight schedule are considered the basic structural unit of an operation, called an operating aircraft group, where the first aircraft is the leading aircraft and the second aircraft is the trailing aircraft.

[0022] Assume that the arrival delay is , the minimum aircraft separation time between the preceding aircraft and the following aircraft is , there is a buffer time between the two aircraft , is the time the following aircraft is delayed due to the delay of the preceding aircraft, and is the actual arrival time. Included in Assuming that there is no priority relationship between the preceding and following aircraft in the flight group, the delay of the preceding aircraft on the following aircraft can be affected by the following aircraft. Figure 3 Represented. The state space model is used to model the above events, and the flight delay state space model under the condition of continuous arrival and departure is obtained as follows: .

[0023] in 、 Indicates the , No. The amount of delays caused by flights; For the Flight 1 and Flight 2 The planned interval between flights, i.e. the buffer , which can be calculated based on the airport’s flight schedule; It is the minimum aircraft separation and can be implemented according to regulations; Random delays caused by uncertain factors; is the delay measurement value; is the process noise, is the process noise of the previous flight, which affects the current flight. is the measurement noise, which is white noise with zero mean.

[0024] By analyzing the delay wave and event sequence in the airport network, a delay state space model in the airport network is established. The present invention shows the OD flight delay propagation analysis, the schematic diagram of which is as follows Figure 4 As shown. The airport where large-scale flight delays occur is called the original airport, and the airport affected by the delay propagation of the source airport is called the destination airport. These two airports together constitute a dynamic system called the OD field. For the source airport, delayed flights will affect the operation of its subsequent scheduled flights. For the connection between the source airport and the destination airport, the delayed flights departing from the source airport will affect the arriving flights of the destination airport to a certain extent. The OD field delay propagation analysis under large-scale delays is shown as follows: Figure 4 shown.

[0025] exist Figure 4 In the figure, solid arrows indicate the propagation path of delays. Flights that follow one another in time are called preceding and succeeding flights; aircraft operating consecutive flights are called preceding and succeeding flights.

[0026] In actual large-scale flight delays, a delayed preceding arrival flight may directly propagate the delay to a subsequent flight, as the preceding and subsequent flights share the same aircraft. Due to the sequential nature of flights on the flight schedule, a delayed preceding arrival-departure flight may also propagate the delay to a subsequent arrival-departure flight.

[0027] The delay propagation state space model of field O is established. The delay state of field O at the current moment depends on the delay propagation state at the previous moment. That is, the number of delayed flights at the current moment depends on the propagation capacity of the delayed flights at the previous moment. The state equation is expressed as: .

[0028] in, is the state variable in the O field at time t; is the state variable at the previous moment; It is the random delay propagation caused by airport uncertainty factors during the period from arrival to departure of a flight; is the process noise, which is white noise with a mean of 0. is the delay propagation rate from preceding to succeeding flight within airport O, representing the probability that the delay of the preceding arriving flight will propagate to the succeeding departing flight, causing its own delay. Since the propagation of successive flights is discontinuous in time, that is, the delay of the preceding flight in the current time period does not necessarily cause the delay of the succeeding flight in the next time period, the delay propagation rate from preceding to succeeding flight is consistent across time periods within the same airport. ; For O field The preceding-following departure delay propagation rate for a time period indicates the propagation of the preceding departure flight delay at Airport O to the following departure flight, resulting in its own delay; Indicates the internal O field The degree of delay propagation within a time period is defined as the sum of the above two delay propagation rates.

[0029] The observation equation is defined as: .

[0030] in, is the delay measurement value, Represents measurement noise, which is white noise with mean 0.

[0031] A state space model of delay propagation between OD airports is established. There are delay propagation between upstream and downstream flights and delay propagation between preceding and following flights of consecutive arrivals at D airports.

[0032] The current arrival delay state of Airport D depends on the arrival delay propagation state at the previous moment. That is, the number of delayed arrival flights at the current moment depends on the propagation capacity of the delayed arrival flights at the previous moment. The state equation is expressed as: .

[0033] in, is the state variable between OD fields at time t; is the state variable at the previous moment; It is the propagation of random delays in the air caused by uncertain factors during the period from departure to arrival of a flight; Represents process noise, which is white noise with a mean of 0. is the upstream-downstream delay propagation rate between airports, which indicates the probability that a flight with a departure delay at airport O will be transformed into an arrival delayed flight at airport D. Consistent, take the pre-order-post-order delay propagation rate of each time period to be consistent, ; For D field The forward-backward arrival delay propagation rate for a period of time indicates that the delay of the preceding flight at airport O is propagated to the following flight at airport D, causing its delay. Indicates that the OD field is The degree of delay propagation between fields in a time period is defined as the sum of the two delay propagation rates mentioned above. The observation equation is defined as: .

[0034] in, is the delay measurement value; is a state variable; is the process noise, which is white noise with a mean of 0.

[0035] To address the problem of establishing a flight delay prediction model, the present invention adopts a state-space model to describe the state transition behavior of a dynamic system and the relationship between state variables and observation variables. The state-space model is used to regard flight delays as a dynamic system, providing a layered logical basis for the processing in step 2 and providing explainable support for prediction fusion in subsequent steps.

[0036] Step 2. Process multi-source data and design prompt words.

[0037] First, historical flight data obtained by the airport is collected and multi-source data processing operations are performed. That is, the collected data is first layered according to different scenarios and different modes, and the correspondence between data and scenarios is established. Then, the collected data is pre-processed by data cleaning and feature extraction; then a prompt word template is designed and the multi-source data is structured according to the prompt word template.

[0038] Historical flight data is mainly used for large-scale model training to learn to predict delay duration based on multimodal data and form basic prediction capabilities. Real-time flight data is used to use the trained large-scale model to make instant delay predictions for the current real-time scenario.

[0039] The historical flight data collected here includes, for example, minute-level weather data from airport weather stations, flight traffic data, historical flight plans, flight execution records, and expert delay analysis report texts.

[0040] The different scenarios mentioned in this embodiment specifically refer to single-aircraft multi-tasking scenarios, continuous arrival and departure scenarios, and airport network delay scenarios; the different modalities specifically refer to text modality, time sequence modality, and structured modality.

[0041] First, historical flight data is collected and processed in layers. The process is as follows: The system collects weather data such as wind speed, rainfall, and visibility updated every minute by the airport meteorological station, traffic data such as flight trajectory, flight density, and runway occupancy time transmitted in real time by the air traffic control system, and data such as taxiing time and runway usage recorded by the airport ground operation system. It also collects flight planning and execution records for the past three years, including planned take-off / arrival times, actual take-off / arrival times, delay duration data, and analysis reports from experts on historical delay events. The collected data is split according to single-aircraft multi-task scenarios (multiple flight sequences operated continuously by the same aircraft every day), continuous arrival and departure scenarios (multiple flights arriving or departing from adjacent airports), and airport network delay scenarios (delay propagation between source and destination airports). It is also divided into three modes: text mode (such as expert analysis text and weather report text), time series mode (weather data and traffic data arranged in time series), and structured mode (delay trend, fluctuation coefficient, random delay mean, and random delay variance parameter in the state-space model). A precise correspondence between data and scenarios is established, and data processing efficiency is improved through modal stratification. At the same time, it adapts to the processing requirements of different types of data and supports prompt word templates for multi-source information integration.

[0042] Then, the collected data is cleaned and features are extracted. The process is as follows: For numerical data (such as wind speed and rainfall), the 3σ principle is used to identify outliers. For example, data with rainfall greater than 50 mm / h and wind speed exceeding 60 m / s are considered abnormal and removed. The judgment principle is as follows: .

[0043] in, Assume that the rainfall data for a certain period follows a normal distribution with a mean of , the standard deviation is , if the rainfall at a certain moment If the above formula is not satisfied, it is determined to be an outlier (such as extreme rainfall caused by heavy rain).

[0044] The same is true for wind speed data.

[0045] Missing weather or traffic data are supplemented by linear interpolation, and the specific interpolation method is as follows: Among them, for time series data ,..., , if at time There are missing values , then through the time and Observations at and Perform calculations.

[0046] Data type data refers to meteorological data, flow data, and operation data. These data can be directly quantified into continuous or discrete numerical data. They are not considered a mode, but are processed as class data.

[0047] The text data is standardized in simplified Chinese and segmented, and low-frequency words with low frequency of occurrence are filtered out (in this embodiment, for example, words with a frequency of occurrence of less than 5 times are filtered out). The specific processing process is as follows: Convert the input original text set into simplified Chinese to obtain a simplified text list; for each simplified text, remove special symbols, punctuation, etc., and segment the processed text into word sequences to obtain a word segmentation set of all texts; Based on the obtained word segmentation set, the frequency of each word is counted to generate a word frequency dictionary, and words with an appearance frequency less than 5 are filtered out to form a low-frequency word list; for each simplified text, the word segmentation is repeated, and the words not in the low-frequency word list are retained. The filtered words are spliced ​​with spaces to obtain a processed text list.

[0048] After the above operations, text features are constructed. The expert analysis text is annotated according to delay types such as weather reasons, mechanical failure, and traffic congestion. The tokenizer of the Qwen large model is used to generate a token sequence of length 256: The preprocessed simplified Chinese text is segmented into word sequences using the Qwen model's word segmenter to obtain a word list. Each word in the word list is mapped to a digital ID recognizable by the model to generate a digital ID sequence. If the number sequence is greater than 256, the sequence is staged and only the first 256 words are retained. If the number sequence is less than 256, the padding ID is used to fill it to 256, resulting in a standardized ID sequence with a fixed length of 256. For the standardized ID sequence, the mark of the padded part needs to be set to 0 (representing the padded bits ignored by the model), and the rest needs to be set to 1 (representing the real words that the model needs to pay attention to). Based on this combination, a mask list is generated to obtain the attention mask, which is used to indicate the effective information location during model processing.

[0049] In terms of time series feature extraction, a 12-hour sliding window is used to calculate statistics such as mean, variance, and maximum value for weather data. The specific operation process is as follows: Assume that the time series data is .in Indicates time The window size of weather observation values ​​(such as rainfall, wind speed, etc.) is . Then for the current moment , the observation interval contained in the sliding window is: , then the observation value in the window is , various statistics can be calculated within the sliding window, such as: .

[0050] in, Indicates time The mean of the data within the time window is used to reflect the recent weather trend.

[0051] The flow data is decomposed into daily and weekly components through Fourier transform.

[0052] During structured feature processing, the delay trend (e.g., 15-minute delay), fluctuation coefficient (e.g., 0.3), random delay mean, and random delay variance parameters output by the state-space model are converted into JSON format to facilitate parsing by the Qwen large model.

[0053] Next, we designed a prompt word template. Flight delay prediction involves multiple data sources, including weather, traffic, and state parameters, and includes different modalities such as text, time series, and structured data. The prompt word template uses categorized fields to structure the scattered information, enabling the large model to uniformly understand different types of input and addressing data fragmentation. The specific prompt word template is as follows: [Scene Type]: {Single-aircraft multi-task / Continuous arrival and departure / Airport network}; [Real-time flight data]: Weather = {wind speed, rainfall, visibility}, Traffic = {runway occupancy rate, flight density}, Status parameters = {delay trend, fluctuation coefficient}; [Historical Case]: {Summary of recent M delay events in similar scenarios} where M is a natural number, for example, 3; [Prediction target]: Delay probability and duration range for the next N hours, where N is a natural number, for example, 2.

[0054] Because delay propagation mechanisms vary across different operational scenarios in Step 1, scenario stratification allows the data to be broken down by specific scenario. This facilitates subsequent use of the large model to learn delay patterns for specific scenarios. The data is then stratified by scenarios such as single-machine multi-tasking and modalities such as text, time series, and structured. Furthermore, the collected data undergoes data cleaning and feature extraction, and a prompt word template is designed to structure the multi-source, heterogeneous data to adapt it to the input format of the Qwen large model. This template leverages the underlying capabilities of the Qwen large model to independently define field definitions and data organization to accommodate predictions for specific tasks.

[0055] Step 3. Fine-tune the Qwen large model based on the LoRA method, only updating the low-rank adapter parameters. By processing multi-source historical flight data, setting the loss function and optimization strategy, the Qwen large model is adapted to the flight delay prediction task.

[0056] This embodiment selects the Qwen large model (eg, the Qwen-7B model) as the basic flight delay prediction model.

[0057] Since flight delay prediction involves multi-source heterogeneous information such as weather text, traffic data, and expert analysis, large models can efficiently capture the complex associations between different modalities through the self-attention mechanism. For example, they can extract key semantic features that affect delays from weather description texts and integrate them with digitized traffic data to learn delay patterns that are difficult for traditional models to discover.

[0058] At the same time, the generalization ability of the large model formed through pre-training on massive data can adapt to the dynamic changes in flight operation scenarios, such as special circumstances such as extreme weather or temporary traffic control, and dynamically adjust the prediction logic through context understanding.

[0059] The Qwen large-scale model optimizes word vector representations for the specific characteristics of the Chinese language, enabling more accurate understanding of specialized flight terminology (such as "runway occupancy" and "air traffic control") while avoiding semantic ambiguity. Furthermore, Qwen supports efficient prompt word engineering, integrating state-space model parameters, real-time flight data, and other information into input through structured prompt word templates.

[0060] like Figure 5 As shown in Figure 2, the process of fine-tuning a large model using the LoRA method is as follows: The core idea of ​​the LoRA (Low-Rank Adaptation) method is to inject a trainable low-rank decomposition matrix into each layer of the Transformer architecture on the basis of freezing the weights of the pre-trained model, thereby significantly reducing the number of trainable parameters in the downstream tasks. Suppose the weight matrix of a layer in the Qwen large model is , express When all parameters are fine-tuned, the incremental parameter matrix is .

[0061] LoRA method assumptions It is low rank, that is, it can be approximately decomposed into the product of two matrices A and B with small parameters, that is, ,in , , whose initialization follows a normal distribution, Represents a normal distribution with a mean of 0 and a variance of ; , which is initially a matrix of all 0s, In this way, the number of parameters that need to be trained when fine-tuning all parameters is d 2 , and after adopting LoRA, the number of parameters that need to be trained becomes 2dr.

[0062] During training, the LoRA method only calculates the gradients of matrices A and B, while freezing the pre-trained weights. The gradient of , is not updated; in the inference phase, BA can be directly incorporated into , that is, the weight of the model during inference is: .

[0063] Flight delay duration prediction is essentially a continuous-value regression problem (e.g., predicting a 5-minute delay). The mean squared error (MSE) effectively measures the model's accuracy in fitting specific values ​​by calculating the squared difference between the predicted value and the true value. The larger the error, the greater the loss, guiding the model toward optimization that reduces absolute error. The specific formula is as follows: .

[0064] in, is the loss function, is the sample size, is the delay duration predicted by the model, is the actual delay duration; at the same time, MSE is a differentiable convex function, which facilitates the use of optimization algorithms such as gradient descent to solve the optimal parameters. It is also sensitive to outliers, which can prompt the model to focus on the prediction accuracy of severe delay scenarios.

[0065] Step 4. Perform dynamic flight delay prediction and model calibration.

[0066] The present invention realizes real-time reasoning and dynamic prediction based on the dynamic combination of real-time flight data, the predictive ability of large models and the physical logic of state-space models. It also solves the model prediction deviation caused by dynamic changes in the flight operating environment based on the incremental learning method, and continuously optimizes the model by continuously learning the new data distribution to achieve model calibration.

[0067] Real-time flight data from the airport is collected, preprocessed, and prompt words are generated according to the prompt word template. The fine-tuned Qwen large model is used to infer the real-time flight delay prediction results. The real-time flight delay prediction results are weighted and fused with the state-space model estimate obtained by processing the real-time flight data in step 1 (the model construction in step 1 is an independent modeling process based on physical logic. The estimated value of the delay state can be directly calculated through the state-space model in step 1 and the preprocessed real-time flight data) to obtain the final flight delay prediction results.

[0068] Specifically, the process of preprocessing the collected real-time flight data, including weather, traffic, flight operation status, and flight schedules, is as follows: minute-by-minute weather data, airport runway occupancy rates, flight density and flow data, and the current flight operation status data are obtained in real time through the airport meteorological monitoring system and air traffic control flow management system. Multi-source data processing operations are performed according to step 2, and then prompt words for the large model are generated according to the designed prompt word template to facilitate input into the large model. At the same time, for missing data in real-time flight data, such as traffic data for a certain period that cannot be obtained in a timely manner, the prior estimates of the flight delay prediction state space model established in step 1 are used to fill in the gaps. For example, the current missing traffic value is predicted based on the traffic change trend of the previous hour. This ensures that the data input into the large model is complete and meets the format requirements, providing a reliable input basis for subsequent delay prediction reasoning.

[0069] Then, state space fusion is performed in combination with real-time flight data. This embodiment constructs a weighted fusion model to achieve an organic combination of the large model prediction results and the state space model estimation value. The formula is as follows: .

[0070] Final flight delay prediction results Real-time prediction results of flight delays for the current scenario by the large model , and the estimated value of the state space model under the corresponding scenario Weighted.

[0071] The current scenario is a single-aircraft multi-tasking scenario, a continuous arrival and departure scenario, or an airport network delay scenario.

[0072] α is a dynamically adjusted weight, and its value is automatically optimized based on data integrity: when real-time flight data such as weather and traffic are complete, α increases, focusing on the Qwen large model's pattern recognition ability for multimodal data; when data is missing, α decreases to increase the prior estimation weight of the state-space model, leveraging its dynamic system modeling advantages to make up for data shortages.

[0073] The weighted fusion mechanism proposed in this embodiment not only retains the Qwen large model's ability to capture complex nonlinear relationships, but also enhances the interpretability of the prediction results by leveraging the physical logic of the flight delay prediction state space model. Especially in special scenarios such as extreme weather or sudden traffic congestion, the trend extrapolation of the state space model can avoid the prediction deviation of the Qwen large model caused by insufficient training data, thereby improving the stability and accuracy of the overall prediction.

[0074] In addition, the idea of ​​incremental model calibration learning is introduced in step 4. Real-time prediction data and actual delay results are continuously collected through a caching mechanism. When the cumulative sample size of a single scenario reaches a preset threshold, the LoRA incremental fine-tuning process is automatically triggered.

[0075] Specifically, real-time prediction data and actual delay results are continuously collected through a caching mechanism. When the sample size of a single scenario reaches a preset threshold, for example, when the single-aircraft multi-task scenario accumulates 200 flight delay samples of the same aircraft continuously operating; when the continuous arrival and departure scenario accumulates 300 delay samples of flights between adjacent airports; when the airport network scenario accumulates 500 delay samples between source and destination airports; or when extreme weather occurs continuously (such as continuous heavy rain and other extreme weather in a certain period in the past), the LoRA incremental fine-tuning process is automatically triggered, and new samples of the scenario (predicted data + actual delay results) are extracted from the cache. They are reorganized according to the prompt word template format of step 2, and the actual delay results of the new data are used as the target. The parameters of matrices A and B in step 3 are optimized to minimize the deviation between the delay prediction duration and the actual value. During this process, the weights of the pre-trained model remain frozen and only a small number of new parameters are iteratively added to adapt to the new data distribution. For example, when extreme weather occurs continuously, the correlation pattern between the cause of delay and duration may change significantly. If the original model is not updated in time, it will lead to prediction deviation due to reliance on historical distribution. In this case, incremental calibration is required. Some rainstorm delay samples are extracted and organized according to the prompt word template. The matrix A and B parameters in step 3 are updated. The actual delay duration is used as the target. The loss function MSE mentioned in step 3 is used to minimize the deviation between the predicted delay duration and the true value, and learn the new correlation between "rainfall-delay duration". This allows the model to quickly learn new weather and delay correlation features, avoiding the time and resource consumption of retraining the entire model, thereby achieving dynamic optimization of prediction capabilities.

[0076] In addition, in order to verify the effectiveness of the flight delay prediction method proposed in the present invention, the present invention conducted a series of experiments on continuous mission flight delay prediction, continuous inbound flight delay prediction and delay prediction within the airport network, and successfully verified the feasibility and accuracy of the flight delay prediction method based on multimodal data fusion and large model fine-tuning.

[0077] This invention uses multimodal data fusion and large-scale model fine-tuning to predict flight delays. Combined with multimodal data processing, it significantly improves prediction accuracy and efficiency. By applying data fusion and the LoRa fine-tuning large-scale model method, the system can process and calibrate large amounts of observational data in real time, accelerating prediction convergence and reducing the time and cost required to retrain a specialized large-scale model. Furthermore, its modular architecture allows the system to operate on large datasets and complex environments, further enhancing the adaptability and generalization capabilities of flight delay prediction. This invention also combines multi-source data processing operations with incremental learning to further enhance the diversity and effectiveness of the prediction model. Expert analysis is combined with historical data to enable comprehensive analysis of historical data and expert modules, ensuring a more accurate prediction process across the board. Combined with incremental learning, the system not only improves prediction efficiency but also enhances the model's robustness and generalization capabilities, enabling it to perform even better in complex flight delay prediction tasks. The design of the model fine-tuning module significantly improves prediction accuracy and comprehensively enhances the reliability and efficiency of the flight delay prediction system, enabling practical prediction results that can provide decision-making support for airlines or airports.

[0078] Example 2 This embodiment 2 describes a flight delay prediction system based on multimodal data fusion and large model fine-tuning. This system is based on the same inventive concept as the flight delay prediction method based on multimodal data fusion and large model fine-tuning in embodiment 1.

[0079] like Figure 6 As shown, the flight delay prediction system based on multimodal data fusion and large model fine-tuning in this embodiment includes: System state space model prediction module, used to establish a flight delay prediction state space model; The state space model uses a dynamic system approach to associate the potential state of flight delays with observable data, and uses the state space to describe the state transition behavior of the dynamic system and the relationship between state transformations and observed variables; Real-time data acquisition and processing module, used for multi-source data processing and prompt word design; First, historical flight data from the airport is collected and multi-source data processing is performed. This involves stratifying the collected data according to different scenarios and modalities, establishing a correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. A prompt word template is then designed and used to structure the multi-source data. Large model fine-tuning module, used to fine-tune the Qwen large model based on the LoRA method; When fine-tuning the Qwen large model based on the LoRA method, only the low-rank adapter parameters are updated. By processing multi-source historical flight data and setting the loss function and optimization strategy, the Qwen large model is adapted to the flight delay prediction task. And a real-time data prediction module for dynamic prediction of flight delays.

[0080] Collect real-time flight data from the airport, perform multi-source data processing operations according to the method in step 2, and generate prompt words according to the prompt word template. Use the fine-tuned Qwen large model to infer the real-time flight delay prediction results, and perform weighted fusion with the state-space model estimate obtained by processing the real-time flight data in step 1 to obtain the final flight delay prediction results.

[0081] It should be noted that, in the flight delay prediction system described in this embodiment 2, the implementation process of the functions and effects of each functional module is detailed in the implementation process of the corresponding steps of the method in the above embodiment 1, and will not be repeated here.

[0082] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above-mentioned embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with this field under the guidance of this specification fall within the substantive scope of this specification and should be protected by the present invention.

Claims

1. A flight delay prediction method based on multimodal data fusion and large model fine-tuning, characterized by: The steps include: Step 1. Establish a flight delay prediction state space model; The state space model uses a dynamic system approach to associate the potential state of flight delays with observable data, and uses the state space to describe the state transition behavior of the dynamic system and the relationship between state transformations and observed variables; Step 2. Process multi-source data and design prompt words; First, historical flight data from the airport is collected and multi-source data processing is performed. This involves stratifying the collected data according to different scenarios and modalities, establishing a correspondence between data and scenarios, and then performing data cleaning and feature extraction preprocessing on the collected data. A prompt word template is then designed and used to structure the multi-source data. Step 3. Fine-tune the Qwen large model based on the LoRA method, updating only the low-rank adapter parameters. By processing multi-source historical flight data and setting the loss function and optimization strategy, the Qwen large model is adapted to the flight delay prediction task. Step 4. Perform dynamic prediction of flight delays; Collect real-time flight data from the airport, perform multi-source data processing operations according to the method in step 2, and generate prompt words according to the prompt word template. Use the fine-tuned Qwen large model to infer the real-time flight delay prediction results, and perform weighted fusion with the state-space model estimate obtained by processing the real-time flight data in step 1 to obtain the final flight delay prediction results.

2. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1 is characterized in that: In step 1, the process of establishing the flight delay prediction state space model is as follows: The flight delay propagation event sequence of a single aircraft performing multiple tasks continuously is analyzed, and the state equation and observation equation of single-aircraft multi-task flight delay are established; the flight delay propagation event sequence of multiple aircraft continuously arriving and departing is analyzed, and the state equation and observation equation of continuous arriving and departing flight delay are established; the source airports where flight delays occur and the destination airports affected by the propagation of delays at the source airport are analyzed, and the state equation and observation equation of delays within the airport network are established.

3. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1 is characterized in that: In step 2, the process of collecting and layering historical flight data is as follows: Collect flight plan and execution records, weather data, traffic data, and expert analysis reports on historical delay incidents over the past three years; The collected data is split according to single-aircraft multi-task scenarios, continuous arrival and departure scenarios, and airport network delay scenarios, and is divided into three modes: text mode, time series mode, and structured mode to establish a precise correspondence between data and scenarios.

4. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 3 is characterized in that: In step 2, the process of data cleaning and feature extraction of the collected historical flight data is as follows: For numerical data, the 3σ principle was used to identify and eliminate outliers, and missing data were supplemented by linear interpolation; The text data is standardized in simplified Chinese and segmented to filter out low-frequency words. When constructing text features, the expert analysis text is annotated according to the delay type, and a token sequence of 256 in length is generated using the Qwen large model's tokenizer. In terms of time series feature extraction, a 12-hour sliding window is used to calculate statistics including mean, variance, and maximum value for weather data, and Fourier transform is used to decompose flow data into daily and weekly components. When processing structured features, the delay trend, fluctuation coefficient, random delay mean, and random delay variance parameters in the flight delay prediction state space model are converted into JSON format to facilitate large model parsing.

5. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 3 is characterized in that: In step 2, the design process of the prompt word template is as follows: Flight delay prediction involves multiple data sources, including weather, traffic, and state parameters, and encompasses different modalities. The prompt word template structures the scattered information through categorical fields, enabling the large model to uniformly understand different types of input. The prompt word template for the designed Qwen large model is as follows: [Scene Type]: {Single-aircraft multi-task / Continuous arrival and departure / Airport network}; [Real-time flight data]: Weather = {wind speed, rainfall, visibility}, Traffic = {runway occupancy rate, flight density}, Status parameters = {delay trend, fluctuation coefficient}; [Historical Cases]: {Summary of recent M delay events in similar scenarios}, where M is a natural number; [Prediction target]: Delay probability and duration range in the next N hours, where N is a natural number.

6. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1 is characterized in that: In step 3, the process of fine-tuning the large model using the LoRA method is as follows: Assume that the weight matrix of a certain layer in the Qwen model is ; in express When all parameters are fine-tuned, the incremental parameter matrix is ; LoRA method assumptions It is low-rank, that is, it can be decomposed into the product of two matrices A and B with small parameters; During training, the LoRA method only calculates the gradients of matrices A and B, and freezes the pre-trained weights. The gradient of , is not updated; BA is directly merged into , in the inference phase , that is, the weight of the model during inference is .

7. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1 is characterized in that: In step 3, set the loss function The calculation formula is as follows: ； in is the sample size, is the delay duration predicted by the model, is the actual delay duration; The mean square error (MSE) is used in the loss function calculation to calculate the square difference between the predicted value and the true value, which effectively measures the model's fitting accuracy to a specific value. MSE is a differentiable convex function, which facilitates the use of the gradient descent optimization algorithm to solve for the optimal parameters.

8. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1 is characterized in that: In step 4, the process of pre-processing the collected real-time flight data is as follows: The airport meteorological monitoring system and air traffic control flow management system are used to obtain minute-by-minute weather data, runway occupancy rates, flight density and flow data, and current flight operation status data in real time. Multi-source data processing is performed according to the processing method in step 2, and then the prompt words of the large model are generated according to the designed prompt word template. At the same time, in order to fill in the missing data in the real-time flight data, the prior estimated values ​​of the flight delay prediction state space model established in step 1 are used to ensure that the data input into the large model is complete and meets the format requirements.

9. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 1 is characterized in that: In step 4, the calculation process of the final flight delay prediction result is as follows: A weighted fusion model is constructed to achieve an organic combination of the prediction results of the large model and the estimated values ​​of the state space model. The formula is as follows: ； Final flight delay prediction results Real-time prediction results of flight delays for the current scenario by the large model , and the estimated value of the state space model under the corresponding scenario Weighted The current scenarios include single-aircraft multi-tasking scenarios, continuous arrival and departure scenarios, or airport network delay scenarios.

10. The flight delay prediction method based on multimodal data fusion and large model fine-tuning according to claim 6, characterized in that: In step 4, the predicted data and actual delay results are stored with the help of a cache mechanism. When the number of single-scene samples reaches a preset number or extreme weather conditions occur continuously, LoRA incremental fine-tuning is triggered to achieve dynamic optimization of the large model. The process is as follows: New samples for the current scenario, namely the cached predicted data and actual delay results, are extracted from the cache and reorganized according to the prompt word template format of step 2. The actual delay results of the newly added data are used as the target, and the parameters A and B in step 3 are optimized to minimize the deviation between the predicted delay duration and the true value. During this process, the pre-trained model weights remain frozen, and only a small number of new parameters are iteratively added to adapt to the new data distribution, allowing the model to quickly learn new weather and delay correlation characteristics.

Images