A method for airport capacity allocation based on reinforcement learning

Abstract

A method for airport capacity configuration based on reinforcement learning belongs to the technical field of air traffic planning and management. It includes the following steps: step S1, defining the airport capacity configuration time period; step S2, building a Bayesian network model; step S3, defining the reinforcement learning environment for airport capacity configuration; step S4, configuring the reinforcement learning hyperparameters for airport capacity configuration; step S5, initializing the training environment and Q table; step S6, training through the reinforcement learning method, iteratively updating the Q value; step S7, judging whether the model training termination condition is met; step S8, obtaining the airport capacity configuration plan according to the Q table. Through the airport capacity configuration method based on reinforcement learning of this application, the temporal correlation and difference of the uncertainty service capacity in each time period are comprehensively considered, which promotes the efficiency and ease of operation of the capacity configuration plan generation, and provides a basis for optimizing airport resource utilization and reducing flight delays.

Description

Technical Field

[0001] The present invention relates to the technical field of air traffic planning and management, and in particular to an airport capacity configuration method based on reinforcement learning. Background Art

[0002] Schedule management is a universal management method for ensuring safe, efficient, and orderly flight operations. A rational schedule allocation process maximizes the utilization of airport and airspace resources, improving the overall efficiency of the air traffic system. Airport published capacity is crucial for schedule allocation, directly limiting the amount of resources available during the allocation process.

[0003] Current methods for determining published airport capacity primarily include those based on historical statistical data, mathematical models, computer simulation models, and controller workload. These methods result in constant capacity settings, assigning equal capacity values ​​to each time period throughout the day. To account for the uncertainty of actual service capacity during the operational phase, this value is typically set at 85-90% of maximum throughput. However, these constant capacity allocation methods fail to account for the differences in uncertainty patterns across time periods and ignore the correlations between them. This can easily lead to underutilized airport resources and frequent delays. Therefore, an airport capacity allocation method is proposed that fully considers the temporal correlations between time periods and the differences in airport service capacity due to uncertainty, effectively generating capacity allocation plans. This is key to improving airport resource utilization and reducing flight delays. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: to overcome the shortcomings of the existing technology and provide an airport capacity configuration method based on reinforcement learning, which comprehensively considers the temporal correlation and differences of uncertain service capabilities in each time period, promotes the efficiency and ease of operation of the capacity configuration plan generation, and provides a basis for optimizing airport resource utilization and reducing flight delays.

[0005] The technical solution adopted by the present invention to solve the technical problem is: the airport capacity configuration method based on reinforcement learning includes the following steps:

[0006] Step S1, defining the airport capacity allocation period;

[0007] Step S2: collecting and extracting the actual service capacity data of the airport during the capacity allocation period, constructing a Bayesian network model of the airport service capacity, and inferring the probability distribution of the actual service capacity of the airport in each period;

[0008] Step S3, defining the reinforcement learning environment for airport capacity configuration, including service capacity generation, actions and action space, delay calculation, and reward function;

[0009] Step S4, setting the reinforcement learning hyperparameters for airport capacity configuration, including strategy selection method, maximum number of rounds, termination condition, learning rate, and discount factor;

[0010] Step S5, initializing the training environment and Q table;

[0011] Step S6, training by reinforcement learning method, iteratively updating Q value;

[0012] Step S7, determining whether the model training termination condition is met, if the model training termination condition is met, executing step S8, otherwise returning to step S6;

[0013] Step S8: Obtain the airport capacity configuration plan according to the Q table.

[0014] Preferably, step S2 specifically includes the following steps:

[0015] S2-1, collect the actual service capacity data of the airport in each capacity configuration period,

[0016] S2-2, each capacity configuration period The actual service capacity is represented as a node in the network;

[0017] S2-3, establish service capabilities for each time period through edge connections Dependencies between Figure 2 As shown;

[0018] S2-4 uses maximum likelihood estimation to train the model, estimate the conditional probability distribution of the Bayesian network, and obtain the Bayesian network model of airport service capacity.

[0019] Preferably, step S3 specifically includes the following steps:

[0020] S3-1, through the Bayesian network model of airport service capacity, define Time service capability t ;

[0021] S3-2. Define the action space , the action is the decision of capacity setting, that is, the capacity set for each period , ;

[0022] S3-3. Define time period Delays The calculation method is , yes Delay during the period, c t Indicates time period service capabilities;

[0023] S3-4. Definition , indicating that the total demand is achieved in the current period After that, the total demand that can be realized is less than the target demand. probability;

[0024] S3-5. Define the reward function .

[0025] Preferably, in step 3-1, define Real-time service capabilities The calculation method is:

[0026] Inferring time periods through Bayesian networks Conditional probability distribution of service capability status , yes The service capability value of the parent node;

[0027] Sampling from the conditional probability distribution to generate Service capabilities at all times:

[0028] ;

[0029] in, It's time The possible values ​​of the service capability, is the conditional probability of each service capability value.

[0030] Preferably, in step S3-5, the reward function It consists of delay reward and demand reward, i.e. ,in , is the weight coefficient of delay, is the delay for the current period, is the delay penalty index; , is the weight coefficient of demand; .

[0031] Preferably, in step S3-4, The calculation is as follows:

[0032] Time Take action The probability of , , Abbreviated as ;

[0033] , where the convolution operation ∗ is to convolve the probability distribution of the current moment with the distribution of the previous moment, that is, , and is the distribution of actions at two moments; Sub-convolution operation;

[0034] aggregate demand Expressed as:

[0035] ;

[0036] Where t represents the time period.

[0037] Preferably, step S6 specifically includes the following steps:

[0038] S6-1, select the reinforcement learning Q-learning algorithm to train the model;

[0039] S6-2, for Time period, generate random numbers , epsilon-greedy method, select the action in time t ,Right now:

[0040] ;

[0041] in, For the period Take action The probability of is the exploration rate,

[0042] S6-3, according to the selected action , calculate the corresponding reward ;

[0043] S6-4, update the Q value.

[0044] Preferably, in step S6-4, the Q value is updated using the following formula:

[0045] ;

[0046] in, is the Q value at time t+1, is the Q value at time period t, ,in , is the weight coefficient of delay, is the delay for the current period, is the delay penalty index; , is the weight coefficient of demand; , is the discount factor.

[0047] Preferably, in step S8, the action with the largest Q value in each time period is extracted according to the Q table, which is the airport capacity configuration plan. This includes: when the Q value converges or reaches a predetermined number of training rounds, the optimal action in each time period is extracted, that is, the action corresponding to the largest Q value in each time period is selected. The optimal action is the final airport capacity configuration plan: .

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] Through the reinforcement learning-based airport capacity allocation method proposed in this application, the temporal correlation and differences of uncertain service capabilities in each time period are comprehensively considered, which promotes the efficiency and ease of operation of the capacity allocation plan generation, and provides a basis for optimizing airport resource utilization and reducing flight delays.

[0050] This reinforcement learning-based airport capacity allocation method dynamically adjusts capacity settings for each time period, fully accounting for the uncertainty and temporal dependencies of airport service capabilities. Compared to traditional constant capacity allocation methods, this method intelligently adjusts capacity based on real-time changes in airport service capabilities, effectively improving airport resource utilization and avoiding excessive or insufficient resources. By optimizing capacity allocation, flight delays caused by insufficient or inappropriate capacity allocation are reduced.

[0051] Within the framework of the present invention, a reinforcement learning algorithm infers the optimal capacity settings for each time period based on historical data and actual service capabilities, which can effectively reduce the occurrence of delays and improve flight punctuality. The present invention combines reinforcement learning with Bayesian networks to generate efficient and accurate capacity allocation plans in complex airport operating environments. The reinforcement learning training process ensures that the model can gradually learn the optimal capacity allocation strategy, making capacity allocation more flexible and in line with actual needs. The method itself is simple and easy to operate, reduces reliance on human experience, improves the automation and accuracy of plan generation, and is easy to apply in actual airport operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of the airport capacity configuration method based on reinforcement learning;

[0053] Figure 2 Bayesian network diagram for airport service capacity;

[0054] Figure 3 Schematic diagram of the maximum Q value change during the training process;

[0055] Figure 4 This is a schematic diagram of the airport capacity configuration plan. DETAILED DESCRIPTION

[0056] Figure 1-Figure 4 The best embodiment of the present invention is shown below in conjunction with the attached Figure 1-Figure 4 The present invention is further described.

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] like Figure 1 As shown in FIG, a method for airport capacity configuration based on reinforcement learning includes the following steps:

[0059] Step S1, defining the airport capacity allocation period;

[0060] Determine the total duration and unit duration of airport capacity allocation, and define the time period for capacity allocation, including: determining the airport capacity coordination period, and determining the total duration of capacity allocation. , determine the capacity unit duration , divide the total duration into capacity configuration period, denoted as For example, if the start and end time of airport capacity coordination is 6:00-23:59, the total duration L=18 hours. hours, divided into There are three capacity configuration periods, namely 6:00, 7:00…, 23:00, recorded as .

[0061] Step S2, constructing an airport service capability representation model based on the YeBais network;

[0062] The actual service capacity data of the airport during the capacity allocation period is collected and extracted, and a Bayesian network model of the airport service capacity is constructed to infer the probability distribution of the actual service capacity of the airport in each period. The specific steps include the following:

[0063] S2-1, collect the actual service capacity data of the airport during each capacity configuration period.

[0064] S2-2, each capacity configuration period The actual service capacity is represented as a node in the network.

[0065] S2-3, establish service capabilities for each time period through edge connections Dependencies between Figure 2 shown.

[0066] S2-4 uses maximum likelihood estimation to train the model, estimate the conditional probability distribution of the Bayesian network, and obtain the Bayesian network model of airport service capacity.

[0067] Step S3, defining a reinforcement learning environment for airport capacity configuration;

[0068] Define the reinforcement learning environment for airport capacity configuration, including service capacity generation, actions and action space, delay calculation, reward function, etc. The specific steps include the following:

[0069] S3-1, based on the Bayesian network model of airport service capacity, define Real-time service capabilities The calculation method is:

[0070] Inferring the time period through the Bayesian network Conditional probability distribution of service capability status , yes The service capability value of the parent node;

[0071] The service capacity at the moment is generated by sampling from the conditional probability distribution, that is,

[0072] ;

[0073] in, It's time The possible values ​​of the service capability, is the conditional probability of each service capability value; in the temporal relationship, all parent nodes are upstream nodes of this node, so the value of the parent node can be obtained recursively through this step.

[0074] S3-2. Define the action space , the action is the decision of capacity setting, that is, the capacity set for each period , the capacity is set to discrete data, ; The demand in each period is different, and the action space can change in different periods.

[0075] S3-3. Define time period Delays The calculation method is , yes Delay during the period; c t express Time service capability.

[0076] S3-4. Definition , indicating that the total demand is achieved in the current period After that, the total demand that may be realized is less than the target demand. The probability of . Among them, , is a constant, The calculation is as follows:

[0077] Time Take action The probability of , , Abbreviated as ;

[0078] , where the convolution operation ∗ is to convolve the probability distribution of the current moment with the distribution of the previous moment, that is, , and is the distribution of actions at two moments; Sub-convolution operation;

[0079] S3-5. Define the reward function , which consists of delay reward and demand reward, i.e. ,in , is the weight coefficient of delay, is the delay for the current period, is the delay penalty index; , is the weight coefficient of demand; .

[0080] Step S4, configuring the reinforcement learning hyperparameters for airport capacity configuration;

[0081] Set the reinforcement learning hyperparameters for airport capacity configuration, including the strategy selection method, maximum number of rounds, termination condition, learning rate, discount factor, etc. The specific steps include the following:

[0082] S4-1 uses the epsilon-greedy method as the strategy selection method and defines the exploration rate for:

[0083] ;

[0084] in, is the initial exploration rate, is the decay rate. As the number of training rounds increases, the exploration rate gradually decreases.

[0085] S4-2 Setting the learning rate , which controls the step size of the model in each update.

[0086] S4-3 Set discount factor ,determine the weights of future rewards to integrate the model’s impact on long-term rewards;

[0087] S4-4 sets the maximum number of training rounds , to avoid overtraining;

[0088] S4-5 Set the convergence condition parameters and , that is, if the continuous In the iterations, the maximum change of the Q value in each iteration is less than the threshold , the algorithm is considered to have converged, which can be expressed as:

[0089] ;

[0090] in, is the current iteration round, No. The Q value after iterations is No. The Q value after iterations is Indicates the time period, , To divide the total duration into capacity allocation periods, The capacity set for each time period, .

[0091] Step S5, initializing the training environment and Q table;

[0092] Initialize the environment and Q table, which is used to store the Q value of each period and each possible action, that is, Indicates the time period Select Action The expected return after . During initialization, all Q values ​​are set to 0.

[0093] Step S6, model training and iterative update of Q value;

[0094] Training is performed using reinforcement learning methods to iteratively update the Q value, including the following steps:

[0095] S6-1, select the reinforcement learning Q-learning algorithm to train the model;

[0096] S6-2, for Time period, generate random numbers , epsilon-greedy method, select the action in time t ,Right now:

[0097] ;

[0098] in, For the period Take action The probability of is the exploration rate,

[0099] S6-3, according to the selected action , calculate the corresponding reward ;

[0100] S6-4, update the Q value using the following formula,

[0101] ;

[0102] in, is the Q value at time t+1, is the Q value at time period t, ,in , is the weight coefficient of delay, is the delay for the current period, is the delay penalty index; , is the weight coefficient of demand; , is the discount factor.

[0103] Step S7, determining whether the model training termination condition is met;

[0104] The termination condition of model training is that the Q value converges or the maximum number of rounds is reached. It is judged whether the termination condition of model training is met. If the termination condition of model training is met, step S8 is executed, otherwise return to step S6.

[0105] Step S8, obtaining the airport capacity configuration plan according to the Q table;

[0106] According to the Q table, the action with the largest Q value in each period is extracted, which is the airport capacity configuration plan, including: when the Q value converges or reaches the predetermined training rounds, such as Figure 3 As shown, the optimal action for each period is extracted, that is, the action corresponding to the maximum Q value in each period is selected.

[0107] like Figure 4 As shown; the optimal action is the final airport capacity configuration plan: ; In this way, we try to ultimately obtain the optimal capacity setting for each time period, ensuring that delays are minimized and the total demand target is met.

[0108] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0109] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for airport capacity configuration based on reinforcement learning, characterized in that: The steps include: Step S1, defining the airport capacity allocation period; Step S2: collecting and extracting the actual service capacity data of the airport during the capacity allocation period, constructing a Bayesian network model of the airport service capacity, and inferring the probability distribution of the actual service capacity of the airport in each period; Step S3, defining the reinforcement learning environment for airport capacity configuration, including service capacity generation, actions and action space, delay calculation, and reward function; Step S4, setting the reinforcement learning hyperparameters for airport capacity configuration, including strategy selection method, maximum number of rounds, termination condition, learning rate, and discount factor; Step S5, initializing the training environment and Q table; Step S6, training by reinforcement learning method, iteratively updating Q value; Step S7, determining whether the model training termination condition is met, if the model training termination condition is met, executing step S8, otherwise returning to step S6; Step S8, obtaining the airport capacity configuration plan according to the Q table; Step S2 specifically includes the following steps: S2-1, collect the actual service capacity data of the airport in each capacity configuration period, S2-2, each capacity configuration period The actual service capacity is represented as a node in the network; S2-3, establish service capabilities for each time period through edge connections Dependencies between them; S2-4 uses maximum likelihood estimation to train the model, estimate the conditional probability distribution of the Bayesian network, and obtain the Bayesian network model of airport service capacity; Step S3 specifically includes the following steps: S3-1, through the Bayesian network model of airport service capacity, define Time service capability t ; S3-2. Define the action space , the action is the decision of capacity setting, that is, the capacity set for each period , ; S3-3. Define time period Delays The calculation method is , yes Delay during the period, c t Indicates time period service capabilities; S3-4. Definition , indicating that the total demand is achieved in the current period After that, the total demand that can be realized is less than the target demand. probability; S3-5. Define the reward function .

2. The airport capacity configuration method based on reinforcement learning according to claim 1 is characterized in that: In step 3-1, define Real-time service capabilities The calculation method is: Inferring time periods through Bayesian networks Conditional probability distribution of service capability status , yes The service capability value of the parent node; Sampling from a conditional probability distribution to generate Service capabilities at all times: ； in, It's time The possible values ​​of the service capability, is the conditional probability of each service capability value.

3. The airport capacity configuration method based on reinforcement learning according to claim 1, characterized in that: In step S3-5, the reward function It consists of delay reward and demand reward, i.e. ,in , is the weight coefficient of delay, is the delay for the current period, is the delay penalty index; , is the weight coefficient of demand; .

4. The airport capacity configuration method based on reinforcement learning according to claim 3 is characterized in that: In step S3-4, The calculation is as follows: Time Take action The probability of , , Abbreviated as ; , where the convolution operation ∗ is to convolve the probability distribution of the current moment with the distribution of the previous moment, that is, , and is the distribution of actions at two moments; Sub-convolution operation; aggregate demand Expressed as: ； Where t represents the time period.

5. The airport capacity configuration method based on reinforcement learning according to claim 1, characterized in that: Step S6 specifically includes the following steps: S6-1, select the reinforcement learning Q-learning algorithm to train the model; S6-2, for Time period, generate random numbers , epsilon-greedy method, select the action in time t ,Right now: ； in, For the period Take action The probability of is the exploration rate, S6-3, according to the selected action , calculate the corresponding reward ; S6-4, update the Q value.

6. The airport capacity configuration method based on reinforcement learning according to claim 4 is characterized in that: In step S6-4, the Q value is updated using the following formula: ； in, is the Q value at time t+1, is the Q value at time period t, ,in , is the weight coefficient of delay, is the delay for the current period, is the delay penalty index; , is the weight coefficient of demand; , is the discount factor.

7. The airport capacity configuration method based on reinforcement learning according to claim 1, characterized in that: In step S8, the action with the largest Q value in each time period is extracted according to the Q table, which is the airport capacity configuration plan. This includes: when the Q value converges or reaches the predetermined training rounds, the optimal action in each time period is extracted, that is, the action corresponding to the largest Q value in each time period is selected. The optimal action is the final airport capacity configuration plan: .

Images