Airport group flight schedule optimization method and system based on deep reinforcement learning

Abstract

The present invention provides an airport group flight schedule optimization method and system based on deep reinforcement learning, including the following steps: pre-processing actual airport group flight operation data to construct a flight operation data set; Markov process modeling to determine the agent, state space, action space, reward function, and environment simulator; constructing an airport group flight schedule optimization model based on DQN; using the actual flight operation data set to verify the rationality of the optimization model. The airport group flight schedule optimization method of the present invention uses DQN to optimize the flight schedule, and uses a deep neural network to replace the traditional reinforcement learning table, combining the decision-making ability of reinforcement learning with the data analysis ability of deep learning, making the task of processing large-scale data space more efficient; DQN can perform long-term planning and consider future rewards. Compared with traditional heuristic algorithms, it can better avoid falling into local optimal solutions, which is conducive to guiding the optimization of actual flight operation times.

Description

Technical Field

[0001] The present invention belongs to the technical field of flight schedule optimization for air traffic control, and specifically relates to a method and system for optimizing flight schedules for airport clusters based on deep reinforcement learning. Background Art

[0002] In the long run, the demand for air transport will continue to grow, but airspace resources such as airport capacity and waypoint capacity are limited. As a result, the demand for flight schedules will increase, but airports do not have sufficient available resources. The contradiction between the two will lead to traffic congestion and flight delays.

[0003] An airport cluster is a spatial cluster formed between airports and cities within a specific region, with one or more hub airports as the core. As urban areas expand, airports form mutually constrained clusters. Airspace resources are highly shared across airport clusters, necessitating coordinated development and joint regulation. Regulating airspace resource allocation with airport clusters as the core will enable more scientific and rational utilization of airport resources and promote the coordinated development of airport clusters. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for optimizing flight schedules of airport clusters based on deep reinforcement learning.

[0005] In order to solve the above technical problems, the first aspect of the present invention provides an airport group flight schedule optimization method based on deep reinforcement learning, comprising:

[0006] Preprocess the actual flight operation data of the airport cluster and construct a flight operation dataset;

[0007] Markov process modeling, determining the agent, state space, action space, reward function, and environment simulator;

[0008] Construct a DQN-based airport cluster flight schedule optimization model;

[0009] The actual flight operation data set is used to verify the rationality of the optimization model.

[0010] A second aspect of the present invention provides an airport group flight schedule optimization system based on deep reinforcement learning, comprising:

[0011] The flight operation data set construction module is used to pre-process the actual airport group flight operation data and construct the flight operation data set;

[0012] Markov process modeling module, used to perform Markov process modeling, determine the intelligent agent, state space, action space, reward function, and environment simulator;

[0013] Optimization model building module, used to build a DQN-based airport cluster flight schedule optimization model;

[0014] The rationality verification module is used to verify the rationality of the optimization model using the actual flight operation data set.

[0015] The beneficial effects of the present invention are:

[0016] The airport cluster flight schedule optimization method based on deep reinforcement learning of the present invention uses DQN to optimize flight schedules, replacing traditional reinforcement learning tables with deep neural networks. It combines the decision-making ability of reinforcement learning with the data analysis ability of deep learning, making the task of processing large-scale data spaces more efficient; DQN can perform long-term planning and consider future rewards. Compared with traditional heuristic algorithms, it can better avoid falling into local optimal solutions, which is conducive to guiding the optimization of real-world flight operation schedules.

[0017] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The objectives and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of the airport group flight schedule optimization method based on deep reinforcement learning of the present invention;

[0020] Figure 2 is a flow chart of the environmental simulator of the present invention;

[0021] Figure 3 This is the DQN-based airport group flight schedule optimization model of the present invention;

[0022] Figure 4 Comparison of the total amount of time offset in each round under different GAMMAs in the embodiment of the present invention;

[0023] Figure 5 This is a comparison of airport group schedules before and after optimization in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to more clearly illustrate the purpose, technical solutions and advantages of the embodiments of the present invention, the following will be combined with the accompanying drawings and examples to describe in detail the implementation process of the technical solutions of the present invention. The examples described here are only used to explain the core principles of the present invention, but are not used to limit the present invention.

[0025] See also Figure 1 In one embodiment of the present invention, the specific steps of the airport group flight schedule optimization method based on deep reinforcement learning are as follows:

[0026] Step 1: Preprocess the actual airport group flight operation data to construct a flight operation dataset.

[0027] Since the actual flight operation dataset includes a lot of attribute information related to flight operations, not all of which are required by this embodiment, it is necessary to filter the attributes. The original airport group operation dataset includes flight data such as flight number, flight number, aircraft type, execution date, departure airport, planned departure time, actual departure time, arrival airport, planned arrival time, and actual arrival time.

[0028] The flight operation dataset we constructed includes three categories: unique identifier information, flight takeoff and landing information, and time information. The unique identifier is the flight number; flight takeoff and landing information includes the departure and arrival airports; and time information includes the flight date, scheduled departure time, and scheduled arrival time.

[0029] When constructing a dataset, the raw data needs to be preprocessed, including data cleaning and data conversion, processing data with missing attribute values ​​or abnormal data length, converting string attribute data into discrete digital representation, and separating date and time from time information.

[0030] Based on the pre-processed data, the waypoint related data are associated according to the flight number.

[0031] Step 2: Markov process modeling, determine the agent, state space, action space, reward function, and environment simulator.

[0032] The intelligent agent is set as the Greater Bay Area airport cluster, and the intelligent agent makes action choices based on the status.

[0033] The state space describes the current flight schedule of the airport group. The airport capacity is divided into time slices of 15 minutes. There are three airports with 96 time slices each. An array of (3, 96) is constructed. The array elements represent the remaining capacity of the corresponding time slice of the airport. The initial value is the published capacity per unit time of the airport. The composition of is the remaining available capacity of each time slice of each airport:

[0034] ;

[0035] in, To allocate Flight time after Time slice airports have allocated capacity;

[0036] The action space determines all actions that can be taken in a certain state, and divides the time slice into 5-minute units. The action attribute value means the time offset. If the allocated time is the same as the application time, it is 0, and if the next time slice is allocated, it is 1. is the adjustment amount to the planned take-off and landing time of the flight, that is, the time offset:

[0037] ;

[0038] in, For the Flight time offset;

[0039] The reward function is used to evaluate the quality of an action. The objectives of the model include two parts: one is to minimize the total offset of all flights, and the other is to minimize the maximum offset of a single flight. Therefore, the reward function includes the weighted sum of the total time offset and the maximum time offset. An additional penalty factor is added to increase the penalty amount when the constraints are not met. The purpose is to ensure that the constraints are met. This problem needs to meet a total of three constraints: the airport capacity constraint means that the number of flights scheduled per unit time (15 minutes or 1 hour) cannot exceed the airport time capacity; the connecting flight constraint means that the turnaround time of two flights of an aircraft is within a certain range; the waypoint capacity constraint means that the number of flights per unit time cannot exceed the capacity constraint of the waypoints. The smaller the time offset, the greater the reward, and the larger the time offset, the smaller the reward. Reward function The negative of the weighted sum of the total time offset and the maximum time offset, plus the penalty factor , which consists of three parts:

[0040] ;

[0041] in, is the weight of the maximum adjustment amount;

[0042] Based on the defined state space, action space, and reward function, build a simulator to simulate the flight schedule optimization environment; see Figure 2 , the steps of the environment simulator are as follows:

[0043] (1) Initialize the environment;

[0044] (2) Receive the action selected by the model;

[0045] (3) Check whether the selected action satisfies the constraints. If not, the penalty factor takes effect.

[0046] (4) Update the environment status;

[0047] (5) Return rewards to the agent;

[0048] (6) Check whether this round of training has been completed. If it has been completed, return the end information to the agent to end this round. If it has not been completed, return to step (2) to continue this round of training.

[0049] In this example, Markov process modeling transforms the optimization problem into a Markov decision process, providing a systematic mathematical framework for incorporating reinforcement learning into the solution to flight schedule optimization. This describes the reinforcement learning decision-making process, simplifies the subsequent modeling complexity, and makes the model description and solution more intuitive. The Markov process modeling results are used to design the input and output of the subsequent reinforcement learning-based optimization model, as well as the optimization method.

[0050] Step 3: Build a DQN-based airport cluster flight schedule optimization model.

[0051] The specific expression of the airport group flight schedule optimization problem is as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] Where, For the Flight time offset, is the weight of the maximum adjustment amount, To allocate Flight time after Time slice Airports have allocated capacity, For the Time slice Airports announce capacity, For the Flight turnaround time, is the minimum / maximum turnaround time for connecting flights, To allocate Flight time after Time slice Waypoints have allocated capacity, For the Time slice Waypoint capacity.

[0057] Constraints For the Time slice The allocated capacity of an airport must not be greater than the maximum capacity of the airport. For connecting flights, the maximum / minimum turnaround time constraints must be met. For the Time slice The allocated capacity of each waypoint must not be greater than the maximum capacity of the waypoint.

[0058] After mathematically modeling the optimization problem, the goal is to minimize time offset, which consists of two components: total offset and maximum offset. While striving to achieve this goal, several constraints must still be met. These constraints ensure that the solution does not exceed the total amount of finite resources in reality, ensuring that the solution is realistic and feasible.

[0059] Follow the steps below to build and train the model:

[0060] (1) If Figure 3 As shown in the figure, the structure and parameters of the DQN-based airport group flight schedule optimization model are set, the structure of the deep neural network is determined, the network structure such as the input layer, hidden layer and output layer is established, and the experience replay unit structure is set; the flight operation dataset is used as the input of the model.

[0061] (2) Set parameter indicators such as the number of neurons in each layer of the network to determine the total capacity of the experience replay unit and the number of sampling samples.

[0062] (3) Input flight operation data set to train the model and optimize the flight schedule.

[0063] (4) Check whether the optimized flight schedule achieves the expected effect and satisfies the constraints. If so, output the optimized flight schedule. If not, return to step (3) and continue training until the target is met.

[0064] The expected effect is to adjust the number of flights during peak hours so that the number of flights at each airport and waypoint in each time slot is no more than the maximum capacity, and the total amount of flight time offsets in the optimized timetable is within the predetermined minimum range.

[0065] The DQN-based airport cluster flight schedule optimization model uses the DQN algorithm and a custom environment simulator. Its components include the environment, experience replay unit, current value network, target value network, and error function. The specific working process is as follows:

[0066] (1) The agent interacts with the environment, performs actions and observes the results. The environment is implemented by an environmental simulator, which simulates the external environment of the agent in the scenario of flight schedule optimization of airport clusters.

[0067] (2) The acquired experience data is stored in the experience replay unit. The stored experience data includes state, action, reward, and next state. When the capacity of the experience replay unit is full, the new data will overwrite the earliest data in all the data.

[0068] (3) Randomly extract a batch of data from the experience playback unit to train the current value network. The current value network is a three-layer deep neural network. The input layer is the current state of the airport cluster. , set up a hidden layer, use the Tanh activation function, and the output of the output layer is to perform each action in this state The Q value is calculated using the error function, and the SmoothL1 Loss loss function is used.

[0069] (4) Regularly update the target value network. The target value network has the same structure as the current value network, and the parameters are regularly copied from the current value network.

[0070] Repeat steps (1)-(4) to output the optimized flight schedule.

[0071] In terms of the DQN-based airport group flight schedule optimization model, the number of training rounds is set to 100, the learning rate is 0.001, the number of hidden layer neurons is 256, the total capacity of the experience replay unit is 1000, the number of sampling samples is 64, and the network update frequency is 10.

[0072] Step 4: Use the actual flight operation data set to verify the rationality of the optimization model.

[0073] In this embodiment, the data of the actual flight operation data set can be derived from the actual flight operation data set of Guangzhou Baiyun International Airport (ZGGG), Shenzhen Bao'an International Airport (ZGSZ), and Zhuhai Jinwan Airport (ZGSD) in the domestic Greater Bay Area airport cluster in 2023.

[0074] In this embodiment, the rationality evaluation model performance of the verification optimization model adopts the total time offset generated after the flight schedule is optimized. The total time offset means the sum of the offsets of all flight application times of the airport group in a single day. The offset of a single flight uses the difference between the flight allocation time and the flight application time to calculate the difference, and then all flight offsets are added up to obtain the total amount.

[0075] The discount factor measures the importance of future rewards. A smaller discount factor prioritizes current rewards, while a larger discount factor prioritizes future rewards. An appropriate discount factor can balance the trade-off between exploration and exploitation, making it easier for the reinforcement learning algorithm to converge to a stable strategy. In this example, the discount factors GAMMA were set to 0.99 / 0.95 / 0.9 / 0.89 for comparative experiments.

[0076] We selected different discount factors GAMMA and conducted 4 groups of experiments. The total amount of time offset of the experimental evaluation index is shown in Table 1. The total amount of time offset of each round is shown in Table 1. Figure 4 As shown in the figure, it is found that the proposed airport group flight schedule optimization model can optimize the flight schedule application while ensuring that the schedule offset is within the predetermined minimum range, and can reach convergence within 100 rounds.

[0077] Table 1

[0078]

[0079] Comparison of the timetables of Guangzhou Baiyun Airport, Shenzhen Bao'an Airport and Zhuhai Jinwan Airport before and after optimization when the discount factor is 0.9 Figure 5 shown.

[0080] From the comparison of the flight schedules of the three airports before and after the optimization, it can be seen that in response to the situation where there were too many flight applications at a certain time before the optimization, the optimized flight schedule adjusted the flight schedule during peak hours, which significantly reduced the flight schedule applications during peak hours.

[0081] Based on the above embodiment, an embodiment of the present invention further provides an airport group flight schedule optimization system based on deep reinforcement learning, including:

[0082] The flight operation data set construction module is used to pre-process the actual airport group flight operation data and construct the flight operation data set;

[0083] Markov process modeling module, used to perform Markov process modeling, determine the intelligent agent, state space, action space, reward function, and environment simulator;

[0084] Optimization model building module, used to build a DQN-based airport cluster flight schedule optimization model;

[0085] The rationality verification module is used to verify the rationality of the optimization model using the actual flight operation data set.

[0086] In summary, the airport group flight schedule optimization method based on deep reinforcement learning of the present invention has the following beneficial effects:

[0087] (1) In terms of the input factors of the airport cluster flight schedule optimization model, a deep neural network is used to process the input flight operation data, which can effectively handle high-dimensional input space. Considering the mutual constraints between airport clusters, waypoint-related features are added to the training set.

[0088] (2) In terms of time optimization models, DQN is used to optimize flight schedules, and deep neural networks are used to replace traditional reinforcement learning tables. This combines the decision-making ability of reinforcement learning with the data analysis ability of deep learning, making the task of processing large-scale data spaces more efficient; DQN can make long-term plans and consider future rewards. Compared with traditional heuristic algorithms, it can better avoid falling into local optimal solutions.

[0089] (3) In terms of example verification, taking the flight operation data of the Greater Bay Area airport cluster as an example, it is verified that the airport cluster flight schedule optimization model based on deep reinforcement learning can effectively optimize the flight schedule and has good performance.

[0090] (4) The airport group flight schedule optimization method proposed in this invention is helpful to guide the optimization of actual flight operation schedules.

[0091] With the above-described preferred embodiments of the present invention as inspiration, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A method for optimizing flight schedules for airport groups based on deep reinforcement learning, characterized in that: include: Preprocess the actual flight operation data of the airport group and construct the flight operation data set; Markov process modeling, determining the agent, state space, action space, reward function, and environment simulator; Construct a DQN-based airport group flight schedule optimization model; Use actual flight operation data sets to verify the rationality of the optimization model; The flight operation data set includes: unique identification information, flight take-off and landing information and time information; The Markov process modeling method includes: The agent is set as the Greater Bay Area airport cluster, and the agent makes action choices based on the status; The state space describes the current flight schedule of the airport group. The airport capacity is divided into time slices of 15 minutes. There are 3 airports with 96 time slices each. An array of (3, 96) is constructed. The array elements represent the remaining capacity of the corresponding time slice of the airport. The initial value is the published capacity per unit time of the airport. The composition of is the remaining available capacity of each airport in each time slice: ； in, To allocate Flight schedule after Time slice airports have allocated capacity; The action space determines all actions that can be taken in a certain state, and divides the time slice into 5-minute units. The meaning of the action attribute value is the time offset. If the allocated time is the same as the application time, it is 0, and if the next time slice is allocated, it is 1; action is the adjustment amount to the planned take-off and landing time of the flight, that is, the time offset: ； in, For the The time offset of each flight; The reward function is used to evaluate the quality of an action. The model's objectives include two parts: one is to minimize the total deviation of all flights, and the other is to minimize the maximum deviation of a single flight. The reward function includes the weighted sum of the total time deviation and the maximum time deviation, and adds a penalty factor to increase the penalty amount when the constraints are not met. The reward function It is the negative of the weighted sum of the total time offset and the maximum time offset, plus the penalty factor , which consists of three parts: ； in, is the weight of the maximum adjustment, is the set of all flights in the airport group; The steps of the environment simulator are as follows: (1) Initialize the environment; (2) Receive the action selected by the model; (3) Check whether the selected action satisfies the constraints. If not, the penalty factor takes effect. (4) Update the environment status; (5) Return rewards to the agent; (6) Check whether this round of training has been completed. If completed, return the end information to the agent to end this round. If not completed, return to step (2) to continue this round of training. The method for constructing a DQN-based airport group flight schedule optimization model includes: The specific statement of the airport group flight schedule optimization problem is as follows: ； ； ； ； In the formula, For the Flight time offset, is the weight of the maximum adjustment, To allocate Flight schedule after Time slice Airports have allocated capacity, For the Time slice Airports publish capacity, For the The turnaround time for flights, is the minimum / maximum turnaround time for connecting flights, To allocate Flight schedule after Time slice Waypoints have allocated capacity, For the Time slice Waypoint capacity; Constraints For the Time slice The allocated capacity of an airport must not be greater than the maximum capacity of the airport. The connecting flights need to meet the maximum / minimum turnaround time constraints. For the Time slice The allocated capacity of each waypoint needs to be no greater than the maximum capacity of the waypoint; and Follow the steps below to build and train the model: (1) Set the structure and parameters of the DQN-based airport group flight schedule optimization model, determine the structure of the deep neural network, establish the input layer, hidden layer and output layer network structure, and set the experience playback unit structure; use the flight operation data set as the input of the model; (2) Set the parameter index of the number of neurons in each layer of the network and determine the total capacity and number of sampling samples of the experience playback unit; (3) Input flight operation data set to train the model and optimize the flight schedule; (4) Check whether the optimized flight schedule achieves the expected effect and satisfies the constraints. If so, output the optimized flight schedule. If not, return to step (3) and continue training until it meets the requirements. The expected effect means that the number of flights during peak hours is adjusted so that the number of flights at each airport and waypoint in each time slot is no greater than the maximum capacity, and the total amount of flight time deviations in the optimized timetable is within the predetermined minimum range; The DQN-based airport group flight schedule optimization model includes: an environment, an experience playback unit, a current value network, a target value network and an error function; wherein (1) The agent interacts with the environment, performs actions and observes the results; the environment is implemented by the environment simulator, which is used to simulate the external environment of the agent in the scenario of flight schedule optimization of airport clusters; (2) The acquired experience data is stored in the experience replay unit; the stored experience data includes state, action, reward, and next state. When the capacity of the experience replay unit is full, the new data will overwrite the earliest data among all the data; (3) A batch of data is randomly selected from the experience playback unit to train the current value network; the current value network is a three-layer deep neural network, and the input layer input is the current state of the airport group , set a hidden layer, use the Tanh activation function, and the output of the output layer is to perform each action in this state Q value; use the error function to calculate the loss, and use the Smooth L1Loss loss function; (4) Regularly update the target value network. The target value network has the same structure as the current value network, and its parameters are regularly copied from the current value network. Repeat steps (1)-(4) to output the optimized flight schedule; The method for verifying the rationality of the optimization model includes: using the total amount of time deviation generated after the flight schedule is optimized for evaluation.

2. An airport group flight schedule optimization system based on deep reinforcement learning, characterized in that: include: The flight operation data set construction module is used to pre-process the actual airport group flight operation data and construct the flight operation data set; Markov process modeling module, used to perform Markov process modeling, determine the agent, state space, action space, reward function, and environment simulator; The optimization model building module is used to build a DQN-based airport group flight schedule optimization model; The rationality verification module is used to verify the rationality of the optimization model using the actual flight operation data set; The airport group flight schedule optimization system based on deep reinforcement learning adopts the airport group flight schedule optimization method based on deep reinforcement learning as described in claim 1.

Images