An airport taxiing intelligent scheduling method based on multi-agent reinforcement learning

Abstract

The application provides an airport taxiing intelligent scheduling method based on multi-agent reinforcement learning, comprising the following steps: step 1, receiving the state information of each aircraft on the airport surface; step 2, using a multi-agent reinforcement learning algorithm based on a centralized learning distributed execution framework to perform distributed policy training on the state information of all aircraft, and obtaining a distributed policy for each aircraft on the surface; step 3, selecting an action for the corresponding aircraft according to the generated distributed policy and sending the action to the aircraft on the surface, and the aircraft on the surface taxiing according to the action generated by the distributed policy; and step 4, repeating the above learning process by executing steps 1 to 3, so that finally all aircraft can taxi according to the optimal path trajectory under the premise of avoiding taxiing conflicts. The method can determine the moving direction and moving speed of each aircraft according to the airport surface information, and minimize the taxiing time of the aircraft under the condition of avoiding conflicts.

Description

Technical Field

[0001] This invention belongs to the field of air traffic management, specifically relating to an intelligent airport taxiing scheduling method based on multi-agent reinforcement learning. Background Technology

[0002] With the growth of air traffic volume, the operation and management of civil aviation airports are facing increasing pressure. The expansion cycle of airport infrastructure is far behind the growth of air transport demand, so the urgent task is to improve the operational efficiency of the entire airport system and resolve the conflict of airport resources in a relatively economical way.

[0003] Traditional dispatching methods rely on manual assignment of taxiing routes. Due to the high workload of tower controllers, who need to plan routes while monitoring and directing aircraft operations, and limited by the complex environment and personnel factors, these methods suffer from high safety risks and low efficiency. The existing technology "Application of Multi-Agent Reinforcement Learning in Helicopter Airport Dispatch" (Computer Engineering and Applications, September 8, 2022) designed a simplified helicopter dispatching experimental platform and implemented several multi-agent reinforcement learning algorithms for helicopter path guidance based on this platform. However, the airport dispatching environment used is only a two-dimensional grid environment, far less complex than the large airports in real-world scenarios. Furthermore, the multi-agent reinforcement learning algorithms used are simply reused from single-agent reinforcement learning algorithms in a multi-agent environment. As the spatial dimension of the environment increases, it becomes difficult to train an effective dispatching strategy, thus limiting its application in the dispatching of large airports. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to address the low scheduling efficiency of existing airports by providing an intelligent airport taxiing scheduling method based on multi-agent reinforcement learning.

[0005] To address the aforementioned technical problems, this invention discloses an intelligent airport taxiing scheduling method based on multi-agent reinforcement learning, comprising the following steps:

[0006] Step 1: Receive the status information of each aircraft on the airport surface. The status information includes the aircraft's current position information, previous position information, and taxiing end point position information.

[0007] Step 2: Use a multi-agent reinforcement learning algorithm based on a centralized learning and distributed execution framework to train distributed policies on the state information of all aircraft, and obtain the distributed policies of each aircraft on the field.

[0008] Step 3: Select actions for the corresponding aircraft according to the generated distributed strategy, and send the actions corresponding to each aircraft to the aircraft on the field. The aircraft on the field then taxi according to the actions generated by the distributed strategy.

[0009] Step 4: Repeat steps 1 to 3 to learn the above process. Eventually, all aircraft will be able to taxi along the optimal path trajectory while avoiding taxiing conflicts.

[0010] Furthermore, in step 2, the multi-agent reinforcement learning algorithm adopts the multi-agent proximal policy optimization (MAPPO) algorithm.

[0011] Furthermore, in step 1, there are N aircraft on the airport surface, where N≥2, and the i-th aircraft is denoted as E. i , i=1,...,N, i-th aircraft E i The state information includes the position x at time t. i (t), the position x at the previous time t-1 i (t-1) and the specified finish line position

[0012] The i-th aircraft E i The action design is a tuple For each time slot t, the i-th aircraft E i Along The direction of the velocity v i (t) moves, and the joint action space of all aircraft is defined as a t =(a 1，t a 2，t , ..., a N，t The data z that each aircraft can observe i，t This includes the positions of all aircraft on the field at time slot t, and the user's own position at the previous moment and the designated taxiing end point.

[0013]

[0014] in Let x represent the current positions of all aircraft on the field at time t as observed by the i-th aircraft. i (t-1) represents the position at the previous time t-1. This indicates the specified end point of the glide.

[0015] Furthermore, step 2 includes the following steps:

[0016] Step 2-1: Set the reward function for each aircraft at time t;

[0017] Step 2-2: Calculate the joint value function based on the state information received from all aircraft. The joint value function includes the state value function and the dominance function.

[0018] Steps 2-3: Update the distributed policy of each aircraft according to the joint value function.

[0019] Further, step 2-1 includes:

[0020] Let A be the advantage function of the i-th aircraft. i (s t a t ), where s t This represents the state of all aircraft at the airport at time t. t This represents the set of actions taken by all aircraft at time t at the airport.

[0021] Consider the safety constraints during the taxiing process of multiple aircraft. If the distance between aircraft is less than the safe distance during taxiing, a penalty term is deducted from the reward. Therefore, basic aircraft scheduling knowledge is combined with safety constraints and integrated into the reward function, let the reward function for the i-th aircraft at time t be:

[0022] r i，t (s i，t a i，t )=r1 i，t (s i，t a i，t )+r2 i，t (s i，t a i，t (2)

[0023] Among them, r1 i，t (s i，t a i，t ) is the reward function term used to measure the distance between the aircraft and the destination. If the direction of the aircraft's movement at time t... If the direction of movement is closer to the finish line, the reward value is 0; if the direction of movement is other directions farther from the finish line, the reward value is -2; furthermore, when the aircraft is stationary, the reward value is -1, and the corresponding expression is as follows:

[0024]

[0025] r2 i，t (s i，t a i，tThis is an additional reward / penalty item used to measure whether an aircraft reaches the finish line or causes a taxiway collision. When an aircraft taxis to the designated finish line, it is given a positive reward item C, where 100 ≤ C ≤ 1000. If a collision occurs between aircraft on the ground, each aircraft will receive a negative penalty item -C, as shown in the following expression:

[0026]

[0027] Furthermore, step 2-2 includes:

[0028] In the initial stage at each time t, based on the global state s t Using a distributed strategy π i Give the corresponding aircraft E i , Choose an action a i，t ~π i (a i，t |s t ), This represents the set of all aircraft within the airport area, and sends the obtained actions to the corresponding aircraft on the area for execution; assuming that through communication with all aircraft, the actions of each aircraft (E) are obtained. i of (z) i，t a i，t r i，t ), thereby obtaining the overall (s) of all aircraft on the field. t a t r t ); where s t = This represents the global state information of all aircraft on the field at time t. This represents the set of all state information of the environment, and all aircraft share the same reward function r(s). t a t ) = r t After obtaining the overall state information of all aircraft on the field, a distributed strategy for controlling taxiing scheduling is generated for each aircraft through centralized learning; centralized learning is based on sampled trajectories. This is achieved by, where T represents the sampling length; for the i-th aircraft E i The corresponding distributed strategy is π. i (a i，t |s t ), making the discount reward Where γ is the discount factor, and τ is the time difference between the current time and time t, if we use π={π1(a 1，t |s t ), π2(a 2，t |s t ), ..., πN (a N，t |s t ) represents the joint policy, and the state-value function V π Defined as:

[0029]

[0030] Where s t+1 This represents the global state information at time t+1. Indicates in s t The expected cumulative reward of a series of state transition trajectories obtained using policy π.

[0031] Advantage function A π Defined as

[0032] A π (s t a t )=Q π (s t a t )-V π (s t (6)

[0033] in

[0034] Furthermore, steps 2-3 include: using To update each policy, where θ i Let π represent the strategy π of the i-th aircraft. i The parameters, This indicates that the parameter θ i Find the gradient. R represents the sample average. t (θ i The function f(r) represents the ratio of the updated policy to the old policy of the i-th aircraft at time t. t (θ i A i (s t a t ))=min{r t (θ i A i (s t a t ), c ε (r t (θ i ))A i (s t a t )}, where the clipping function c ε (r i (θ i )) will rt (θ i The value of ) is limited to the interval [1-ε, 1+ε], where ε is a given constant, 0<ε≤0.2.

[0035] Furthermore, in step 2-2, let ρ0(s0) denote the distribution of the initial state s0, and define the expected reward function for reinforcement learning as:

[0036]

[0037] in The expected value of the state-value function obtained using policy π under s0 is expressed as:

[0038]

[0039] Let the ratio of the strategies before and after the update be . Formula (8) is transformed into:

[0040]

[0041]

[0042] Where π old This represents the current joint strategy, specifically the current strategy of the i-th aircraft. It is a parameter Differentiable functions, π i It is the parameter θ i Differentiable functions, Represents π old Find the expected value, and the clipping function c. ε (x) restricts x to the interval [1-ε, 1+ε], A i (s t a t )yes The estimated value;

[0043] Solving formula (9) using the strategy iteration method, the gradient Δθ of the objective function is equal to:

[0044]

[0045] where θ = {θ1, θ2,..., θ N}, θ represents all π i The set of parameters, therefore the strategy π for each aircraft. i The gradient is:

[0046]

[0047] Use gradients to update each policy:

[0048]

[0049] in A represents the sample average. i (s t a t )yes The estimated value.

[0050] Furthermore, step 3 includes the following steps:

[0051] Step 3-1: After generating the distributed strategy, use the distributed strategy π at time t. i With global state s t Select an action as input. Where v i (t) and Let v represent the velocity and direction of movement of the i-th aircraft at time t, respectively. For each aircraft, the available directions of movement are different each time, and the number of available directions depends on the aircraft's current taxiway position. Furthermore, each aircraft can also pause to avoid an impending taxiway conflict; in this case, v... i (t) = 0; When aircraft on the field detect a conflict, the conflict-related aircraft can change... Choose an alternative route to bypass the area where conflict is about to occur, or v i (t) is reduced to 0 to execute a pause action to avoid oncoming aircraft.

[0052] Step 3-2 involves sending the actions generated by the distributed strategy to the corresponding aircraft on the field. The aircraft on the field adjust their taxiing direction and speed according to the received actions to complete the taxiing task at the current time t. The goal of all aircraft learning is to find the taxiing trajectory with the shortest taxiing time while avoiding conflicts.

[0053] Furthermore, step 4 includes the following steps:

[0054] Step 4-1: After receiving data from all aircraft on the field at time t, calculate the reward r based on the direction of motion of each aircraft, whether a collision has occurred on the field, and whether any aircraft has reached the finish line. i，t =r1 i，t +r2 i，t , where r i，t Let r1 represent the reward value obtained by the i-th aircraft after completing the state transition at time t. The reward function is given by r1. i，t and r2 i，t It consists of two parts, r1 i，t This reflects the distance of the aircraft's direction of motion from the destination. When the aircraft moves along the shortest path, r1... i，t =0, when the aircraft chooses to pause, r1i，t =-1, when the aircraft chooses to move in another direction, r1 i，t =-2; r2 i，t The response is whether the aircraft has collided or reached its destination. If the aircraft reaches its destination at time t, r2 i，t =C, where C represents the reward constant for reaching the destination; if the aircraft collides at time t, r2 i，t =-C, where -C represents the collision penalty constant;

[0055] Step 4-2: In each learning round, each aircraft taxis from its initial position to a given destination according to the given flight schedule. The goal is to learn a taxiing trajectory that maximizes reward while avoiding conflict. When all aircraft have taxied to the destination or when there is a conflict between aircraft, this learning round ends and the next learning round begins.

[0056] Beneficial Effects: This invention specifically presents an intelligent aircraft taxiing scheduling design method based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm in complex airport surface environments. It enables multiple aircraft to detect and avoid potential conflict events during taxiing, and then performs path planning for all aircraft to ensure they taxi along optimal trajectories. Compared with existing technologies, its significant advantages are: This invention is applicable to the problem of efficient route scheduling in large airport surface environments with complex taxiway layouts, numerous taxiing conflict characteristics, and many airport surface control constraints. By designing taxiing schemes for multiple aircraft, it effectively solves the trajectory planning problem in large-scale airports that balances safety, efficiency, and fairness for multiple active objectives. The intelligent airport surface taxiing scheduling scheme proposed in this invention utilizes a centralized learning distributed execution framework, which has the advantages of high convergence speed, high conflict resolution rate, and low conflict resolution time. Attached Figure Description

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0058] Figure 1 This is an application scenario diagram of the multi-agent gliding scheduling of the present invention.

[0059] Figure 2 This is a framework diagram of the multi-agent learning algorithm for the gliding scene of the present invention.

[0060] Figure 3 This is a graph showing the taxiing arrival rate and collision rate for multiple batches of flights according to the present invention.

[0061] Figure 4 This is a graph showing the total taxiing time for multiple flights according to the present invention.

[0062] Figure 5 This is a performance comparison chart between the airport taxiing scheduling method based on multi-agent reinforcement learning of the present invention and the traditional scheduling method. Detailed Implementation

[0063] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0064] Optimizing the taxiing process of aircraft in the taxiway system can improve the safety and efficiency of airport surface resource operations. By optimizing the aircraft taxiing process through scheduling methods such as assigning taxiing paths, it is possible to effectively reduce aircraft taxiing costs, improve taxiing efficiency, thereby improving the operational efficiency of aircraft on the airport surface, increasing the dynamic capacity of the airport surface, reducing aircraft surface congestion, and ensuring the operational safety of aircraft.

[0065] By introducing intelligent algorithms for autonomous route planning, various environmental factors can be comprehensively considered. Utilizing air traffic control big data fusion and based on methods such as deep learning and reinforcement learning, this approach addresses the problems of high safety risks and low scheduling efficiency under traditional scheduling methods in scenarios such as intelligent conflict management, intelligent air traffic flow management, intelligent planning management, intelligent arrival and departure sequencing, and intelligent airport operation. It also assists in business decision-making, thereby improving the operational efficiency of the airport system.

[0066] Large airports employing intelligent learning algorithms exhibit the following characteristics: 1) complex taxiway layouts and a large number of moving targets; 2) numerous characteristics of various taxiing conflicts within the airport, requiring different resolution strategies for different types of conflicts; and 3) numerous constraints on airport surface control behavior. Predicting trajectories under these characteristics requires comprehensive optimization of safety, efficiency, and fairness in a multi-objective scenario. Therefore, researching taxi trajectory planning algorithms based on multi-objective optimization is beneficial for comprehensively improving the operational efficiency of the airport surface, making the application of multi-agent learning algorithms imperative.

[0067] This application proposes a complex airport taxiing intelligent scheduling scheme based on multi-agent reinforcement learning, building upon the single-agent taxiing scheduling scheme. It minimizes the taxiing time of each aircraft while satisfying the safety constraints of each aircraft, and the effectiveness of the scheduling scheme design has been verified through computer simulation. For the design of multiple aircraft, the movement direction and speed of the aircraft are jointly optimized. A centralized learning distributed execution framework is used, which decomposes the traditional fully centralized strategy into multiple policy sets and selects actions based on the distributed policy sets, reducing the learning difficulty of the strategy and the action space dimension of aircraft at the airport. This invention employs a solution based on multi-agent reinforcement learning. In the proposed solution, each aircraft sends its position information to the central controller. Then, the central controller uses MAPPO to obtain the mapping relationship between the system state and movement strategy of each aircraft based on the state information of all received aircraft. Based on the global state information, it selects actions using a trained distributed strategy and then sends the obtained action set to the corresponding aircraft on the airfield. The aircraft on the airfield then execute the actions sent by the central controller, thereby enabling multiple aircraft to detect and avoid potential conflict events during taxiing. Based on this, path planning for multiple aircraft is completed to ensure that multiple aircraft taxi along the optimal taxiing trajectory.

[0068] Application scenarios of the present invention are as follows Figure 1 As shown: In a complex airport environment, multiple aircraft need to taxi from a given starting position to a designated destination according to a flight schedule. Considering the risk of taxiing conflicts and collisions among multiple aircraft during their movement, the objective of the aircraft on the ground is to successfully taxi to the designated destination under the safety constraint of avoiding taxiing conflicts and collisions, and to learn a taxiing path with the shortest taxiing time. The main feature of this application is that there are strong safety constraints between each aircraft. The aircraft sends its own state information to the central controller, which uses centralized training to generate a distributed movement strategy for each aircraft. Each aircraft uses the distributed strategy to generate its movement speed and direction based on its current observations.

[0069] In the multi-agent airport taxiing intelligent scheduling scheme of the present invention, N aircraft E iA central controller, i = 1, ..., N, N≥2, is used in conjunction with the system to achieve taxiing route planning and conflict resolution in complex airport scenarios. Due to the intricate structure of taxiways and the variety of taxiing conflicts in large airports, inefficient scenario path planning algorithms are highly susceptible to collisions during peak passenger flow periods. Furthermore, considering the prohibitive cost of aircraft collisions in real-world scenarios, each aircraft requires strong safety constraints during taxiing, posing a significant challenge to scenario path planning algorithms. In traditional policy gradient-based fully centralized deep reinforcement learning, the policy input is global state information, and the output is the joint action of all aircraft. Considering the learning difficulties that high-dimensional action spaces may present, this application's embodiments study centralized training distributed execution deep reinforcement learning based on policy decomposition. In this architecture, the traditional centralized policy network π(·|s t It is reduced to multiple low-dimensional policy networks. Where s t This represents the global state information of all aircraft on the field at time t, and the strategy. Only decide on aircraft E i The actions that should be performed, thus utilizing global information s t Building upon this foundation, the action space dimension of the strategy is reduced, thereby significantly lowering the learning difficulty of the strategy. In this invention, aircraft on the ground can send their state data to the central controller during taxiing. Based on a reinforcement learning algorithm with a centralized training and distributed execution architecture, the central controller uses all collected datasets to centrally train the optimal movement strategy to guide each aircraft to find the optimal taxiing trajectory.

[0070] This application discloses an intelligent airport taxiing scheduling method based on multi-agent reinforcement learning, comprising the following steps:

[0071] Step 1: Receive the status information of each aircraft on the airport surface. The status information includes the aircraft's current position information, previous position information, and taxiing end point position information.

[0072] Under the reinforcement learning algorithm, the i-th aircraft E i The state information includes the position x at time t. i (t), the position x at the previous time t-1 i (t-1) and the specified finish line position All the above information is sent to the central controller; the action of the i-th aircraft can be designed as a tuple. For each time slot t, the i-th aircraft E i Along The direction of the velocity v i(t) moves, and the joint action space of all aircraft is defined as a t =(a 1，t a 2，t , ..., a N，t The data z that each aircraft can observe i，t This includes the positions of all aircraft on the field at time slot t, and the user's own position at the previous moment and the designated taxiing end point.

[0073]

[0074] in Let x represent the current positions of all aircraft on the field at time t as observed by the i-th aircraft. i (t-1) represents the position of the i-th aircraft at the previous time t-1. This indicates the designated taxiing endpoint position for the i-th aircraft.

[0075] Step 2: Use a multi-agent reinforcement learning algorithm based on a centralized learning and distributed execution framework to train distributed policies on the state information of all aircraft, obtaining the distributed policy of each aircraft on the field, including:

[0076] Consider the safety constraints during the taxiing process of multiple aircraft. If the distance between aircraft is less than the safe distance during taxiing, a penalty term is deducted from the reward. Therefore, we combine basic aircraft scheduling knowledge with safety constraints and integrate them into the reward function, letting the reward function for the i-th aircraft at time t be:

[0077] r i，t (s i，t a i，t )=r1 i，t (s i，t a i，t )+r2 i，t (s i，t a i，t (2)

[0078] Among them, r1 i，t (s i，t a i，t ) is the reward function term used to measure the distance between the aircraft and the destination. If the direction of the aircraft's movement at time t... If the direction of movement is closer to the finish line, the reward value is 0; if the direction of movement is further away from the finish line, the reward value is -2. Furthermore, when the aircraft is stationary, the reward value is -1, as shown in the following expression:

[0079]

[0080] r2 i，t (si，t a i，t The penalty is an additional reward or penalty used to measure whether an aircraft reaches the finish line or experiences a taxiway collision. Specifically, when an aircraft taxis to the designated finish line, it receives a positive reward C, where 100 ≤ C ≤ 1000; if a collision occurs between aircraft on the ground, each aircraft receives a negative penalty C, as shown in the following expression:

[0081]

[0082] By setting an appropriate reward function, aircraft on the ground can learn taxiing trajectory planning and conflict detection and resolution strategies simultaneously during training.

[0083] In the initial phase at each time t, the central controller is based on the global state s. t Using a distributed strategy π i Give the corresponding aircraft E i , Choose an action a i，t ~π i (a i，t |s t ), This represents the set of all aircraft within the airport area, and sends the obtained actions to the corresponding aircraft on the area for execution. Assume that through communication with all aircraft, the central controller receives the action data for each aircraft (E). i of (z) i，t a i，t r i，t ), thus obtaining the overall (s) of all aircraft on the field. t a t r t ).in This represents the global state information of all aircraft on the field at time t. This represents the set of all state information of the environment, and all aircraft share the same reward function r(s). t a t ) = r t After obtaining the overall state information of all aircraft on the ground, the central controller needs to generate a distributed strategy for each aircraft to control taxiing scheduling through centralized learning. Centralized learning is based on sampled trajectories within the central controller. This is achieved by using T, where T represents the sampling length, which is 512 in this embodiment. For the i-th aircraft, the corresponding distributed strategy is π. i (a i，t |s t ), making the discount reward Where γ is the discount factor, and τ is the time elapsed since time t (the time difference between the current time and time t), if we use π={π1(a 1，t |s t ), π2(a 2，t |s t ), ..., π N (a N，t |s t ) represents the joint policy, and the state-value function V π Defined as:

[0084]

[0085] Where s t+1 This represents the global state information at time t+1. Indicates in s t The expected cumulative reward of a series of state transition trajectories obtained using policy π, where the ellipsis in the subscript indicates that in state S t The following sequence of actions and state transitions follows the application of strategy π.

[0086] Advantage function A π Defined as

[0087] A π (s t a t )=Q π (s t a t )-V π (s t (6)

[0088] in Let ρ0(s0) denote the distribution of the initial state s0, and define the expected reward function for reinforcement learning as:

[0089]

[0090] in Let represent the expected value of the state value function obtained using policy π under s0. Once the optimal distributed policy is determined, each aircraft can complete the expected taxiing plan according to the generated optimal distributed policy. Therefore, the problem is formulated as:

[0091]

[0092] Using the idea of ​​the MAPPO algorithm, let the ratio of the policy before and after the update be... Where θ i Let π represent the strategy π of the i-th aircraft. i The parameters are transformed into formula (8):

[0093]

[0094]

[0095] Where π old This represents the current joint strategy, specifically the current strategy of the i-th aircraft. It is a parameter Differentiable functions, π i It is the parameter θ i Differentiable functions, Represents π old Find the expected value, and the clipping function c. ε (x) restricts x to the interval [1-ε, 1+ε], where 0 < ε ≤ 0.2.

[0096] Equation (9) can be solved using the policy iteration method, and the gradient of the objective function is equal to:

[0097]

[0098] where θ = {θ1, θ2,..., θ N}, θ represents all π i The set of parameters, This represents the gradient with respect to the parameter θ, therefore the policy π for each aircraft. i The gradient is:

[0099]

[0100] This indicates that the parameter θ i The gradient can be calculated using formula (12) and solved using a centralized learning distributed execution training framework based on the multi-agent PPO algorithm. For example... Figure 2 As shown, in the centralized learning distributed execution framework, the central controller uses the global state information received from all aircraft to train the distributed policy. The centralized evaluator estimates the joint value function based on the global information, updates the parameters of the distributed policy using gradients, and the central controller then makes decisions based on the global state using the distributed policy, sending the corresponding decision results to the aircraft on the field. In reinforcement learning, the expected value is usually replaced by a sampled average. Therefore, the following gradients are used to update each policy:

[0101]

[0102] in A represents the sample average. i (s t a t )yes The estimated value. In each training round, the aircraft starts from a given starting point, and the goal is to learn a path that maximizes reward. The training round ends when each aircraft reaches the destination, collides with another aircraft, or exceeds the training round time limit.

[0103] Step 3: Select actions for the corresponding aircraft according to the generated distributed strategy, and send the actions corresponding to each aircraft to the aircraft on the ground. The aircraft on the ground then taxi according to the actions generated by the distributed strategy, including the following steps:

[0104] Step 3-1: After generating the distributed strategy, use the distributed strategy π at time t. i With global state s t Select an action as input. Where v i (t) and Let v represent the velocity and direction of movement of the i-th aircraft at time t, respectively. For each aircraft, the available directions of movement are different each time, and the number of selected directions depends on the aircraft's current taxiway position. Depending on the connectivity between the current taxiway node and other taxiway nodes, the number of directions can range from one to four. Furthermore, each aircraft can also pause to avoid impending taxiway conflicts; in this case, v... i (t) = 0; When aircraft on the field detect a conflict, the conflict-related aircraft can change... Choose an alternative route to bypass the area where conflict is about to occur, or v i (t) is reduced to 0 to execute a pause action to avoid oncoming aircraft.

[0105] Step 3-2: The actions generated by the distributed strategy are sent to the corresponding aircraft on the field. The aircraft on the field adjust their taxiing direction and speed according to the received actions to complete the taxiing task at the current time t.

[0106] Step 4: Repeat steps 1 to 3 to perform the above learning process. Ultimately, all aircraft will be able to taxi along the optimal path while avoiding taxiing conflicts, including the following steps:

[0107] Step 4-1: After receiving data from all aircraft on the field at time t, calculate the reward r based on the direction of motion of each aircraft, whether a collision has occurred on the field, and whether any aircraft has reached the finish line. i，t =r1 i，t +r2 i，t , where r i，t Let r1 represent the reward value obtained by the i-th aircraft after completing the state transition at time t. The reward function is given by r1. i，t and r2 i，t It consists of two parts, r1i，t This reflects the distance of the aircraft's direction of motion from the destination. When the aircraft moves along the shortest path, r1... i，t =0, when the aircraft chooses to pause, r1 i，t =-1, when the aircraft chooses to move in another direction, r1 i，t =-2; r2 i，t The response is whether the aircraft has collided or reached its destination. If the aircraft reaches its destination at time t, r2 i，t =C, where C represents the reward constant for reaching the destination; if the aircraft collides at time t, r2 i，t =-C, where -C represents the collision penalty constant;

[0108] Step 4-2: In each learning round, each aircraft taxis from its initial position to a given destination according to the given flight schedule. The goal is to learn a taxiing trajectory that maximizes reward while avoiding conflict. When all aircraft have taxied to the destination or when there is a conflict between aircraft, this learning round ends and the next learning round begins.

[0109] like Figure 3 The figure shows the learning curves for arrival rate and collision rate. The initial environment settings are as follows: the number of aircraft on the surface is set to 8, and their starting and ending points are given by the generated flight schedule. The solid line represents the average arrival rate of aircraft on the surface, and the dashed line represents the average collision rate. It can be seen that aircraft on the surface can learn the correct taxiing trajectory through the multi-agent PPO algorithm. This not only guides the aircraft to their designated destination, but also, during taxiing, the central controller makes a series of decisions to avoid potential taxiing conflicts.

[0110] like Figure 4 The figure shows the total taxiing time curve of aircraft in a single round. It can be seen that the total taxiing time is relatively high at the beginning of the learning process but shows a decreasing trend. This indicates that the central controller is gradually shortening the taxiing time by continuously trying different aircraft and optimizing taxiing strategies based on environmental feedback. Later, there is a period where the taxiing time fluctuates significantly. At this point, aircraft in the field may collide with taxiways when taxiing along the shortest path. The central controller will then optimize its taxiing strategy based on environmental feedback to find a sequence of actions that can avoid taxiway collisions. Finally, the taxiing time converges to a stable value, indicating that the taxiing strategies of the aircraft in the field are becoming stable, and the theoretically expected multi-agent taxiing trajectory planning and taxiing conflict exploration and resolution research can be completed.

[0111] like Figure 5The figure shows a performance comparison between the multi-agent taxiing intelligent scheduling method of this embodiment and the traditional scheduling method. In the simulation experiment, to measure the performance parameters of the multi-agent taxiing intelligent scheduling method, the traditional scheduling method was introduced, which involves manually guiding aircraft on the command field to taxi to the designated destination and to avoid taxiway conflicts. It can be seen that the multi-agent taxiing intelligent scheduling method of this invention takes less time than the traditional scheduling method in both total taxiing time and conflict resolution time (i.e., the total time spent by the aircraft in taking non-shortest path actions to avoid conflicts), demonstrating a significant improvement in performance indicators.

[0112] Simulation results demonstrate that the proposed intelligent airport taxiing scheduling method based on multi-agent reinforcement learning can complete path scheduling by using a central controller to uniformly learn a suitable distributed mobility strategy when none of the aircraft possesses prior knowledge of the airport taxiing situation. Furthermore, the proposed algorithm is used in the simulation scenario to obtain the aircraft's taxiing performance indicators, thereby verifying the correctness of the theory.

[0113] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding an intelligent airport taxiing scheduling method based on multi-agent reinforcement learning, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0114] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0115] This invention provides an intelligent airport taxiing scheduling method based on multi-agent reinforcement learning. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for intelligent airport taxiing scheduling based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step 1: Receive the status information of each aircraft on the airport surface. The status information includes the aircraft's current position information, previous position information, and taxiing end point position information. Step 2: Use a multi-agent reinforcement learning algorithm based on a centralized learning and distributed execution framework to train distributed policies on the state information of all aircraft, and obtain the distributed policies of each aircraft on the field. Step 3: Select actions for the corresponding aircraft according to the generated distributed strategy, and send the actions corresponding to each aircraft to the aircraft on the field. The aircraft on the field then taxi according to the actions generated by the distributed strategy. Step 4: Repeat the learning process from Step 1 to Step 3 above. Eventually, all aircraft will be able to taxi along the optimal path trajectory while avoiding taxiing conflicts. Step 2 specifically includes: Step 2-1: Set the reward function for each aircraft at time t; Step 2-2, at each time point In the initial stage, based on the global state Use a distributed strategy Give the corresponding aircraft , Choose an action , This represents the set of all aircraft on the airport surface, and sends the obtained actions to the corresponding aircraft on the surface for execution; assuming that through communication with all aircraft, the actions of each aircraft are obtained. of Thus, the overall situation of all aircraft on the field is obtained. ;in Indicates time The global status information of all aircraft on the ground. This represents the set of all state information of the environment, and all aircraft share the same reward function. ; The airport scene is in The set of actions taken by all aircraft at any given time; after obtaining the overall state information of all aircraft on the ground, a distributed strategy for controlling taxiing scheduling is generated for each aircraft through centralized learning; centralized learning is based on sampled trajectories. This is achieved through, among which Indicates the sampling length; for the first... airplane The corresponding distributed strategy is Make the discount reward ,in As a discount factor, For the current time and time... The time difference, if used Describes the joint policy, state-value function Defined as: （5） in Indicates time The global status information of all aircraft on the ground. Indicates in The expected cumulative reward of a series of state transition trajectories obtained by using the following strategy; Advantage function Defined as: （6） in ; Steps 2-3 update the distributed strategy for each aircraft, specifically including: make Represents the initial state Given the distribution of the distribution, the expected reward function for reinforcement learning is defined as: （7） in, Indicates in Use strategy The expected value of the obtained state value function; Maximizing the expected reward function yields: （8） make Formula (8) is transformed into: （9） （10） in, Indicates the first Distributed strategy of aircraft The parameters, Indicates the first The ratio of the updated strategy to the old strategy for each aircraft. Indicates the current joint strategy, the first Current joint strategy of the aircraft It is a parameter Differentiable functions, It is a parameter Differentiable functions, Indicates to Expectation; pruning function Will Limited to the range Inside, For a given constant, N represents the total number of aircraft. The gradient of the function is obtained by solving formula (9) using the strategy iteration method. for: （11） in , Represents all distributed strategies A set of parameters, therefore the strategy for each aircraft. The gradient is: （12） Update each policy using gradients: （13） in Indicates the average of the samples. Indicates the parameter Find the gradient. This represents the ratio of the updated strategy to the old strategy of the i-th aircraft at time t. yes The estimated value.

2. The airport taxiing intelligent scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, In step 2, the multi-agent reinforcement learning algorithm employs a multi-agent proximal policy optimization algorithm.

3. The airport taxiing intelligent scheduling method based on multi-agent reinforcement learning according to claim 2, characterized in that, In step 1, record that there are N aircraft on the airport field. The i-th aircraft is denoted as , , No. airplane Status information includes Location at any moment At the previous moment Location and the designated finish line position ; No. airplane The action design is a tuple For every moment , No. airplane Along the direction of movement With speed To move, and the combined action space of all aircraft is defined as Data that each aircraft can observe Including all aircraft on the field at any time The current position, the machine's position at the previous moment, and the specified taxiing end point position, i.e.: （1） in Indicates the first An aircraft observed in The current position of all aircraft on the field at any given time. Indicates the previous moment Location, This indicates the specified end point of the glide.

4. The airport taxiing intelligent scheduling method based on multi-agent reinforcement learning according to claim 3, characterized in that, Step 2-1 includes: make Time of the first The reward function for the aircraft is: （2） in, It is a reward function term used to measure the distance between the aircraft and the destination. If the aircraft is at... Direction of movement at all times If the direction of movement is closer to the finish line, the reward value is 0; if the direction of movement is other directions farther from the finish line, the reward value is -2; when the plane is paused, the reward value is -1, and the corresponding expressions are as follows: （3） It is an additional reward / penalty item used to measure whether an aircraft reaches the destination or causes a taxiway collision. When an aircraft taxis to the designated destination, a positive reward is given. , If a collision occurs between aircraft on the field, each aircraft will receive a negative penalty. The corresponding expression is as follows: （4）。 5. The airport taxiing intelligent scheduling method based on multi-agent reinforcement learning according to claim 4, characterized in that, Step 3 includes the following steps: Step 3-1: After generating the distributed strategy, use the distributed strategy at time t. global state Select an action as input. ,in and They represent the first The plane at the moment The speed and direction of movement; for each aircraft, the available directions of movement are different each time, and the number of available directions depends on the aircraft's current position on the taxiway; in addition, each aircraft can also pause to avoid an impending taxiway conflict, at which point... When aircraft on the ground detect a conflict, the aircraft involved in the conflict can change course. Choose an alternative route to bypass the area where conflict is about to occur, or The aircraft was brought to a standstill to allow oncoming traffic to pass. Step 3-2: The actions generated by the distributed strategy are sent to the corresponding aircraft on the field. The aircraft on the field adjust their taxiing direction and speed according to the received actions to complete the current moment. The gliding mission below.

6. The airport taxiing intelligent scheduling method based on multi-agent reinforcement learning according to claim 5, characterized in that, Step 4 includes the following steps: Step 4-1, when all aircraft on the field are received... After the data is transmitted in real time, the reward value is calculated based on the direction of movement of each aircraft, whether a collision has occurred on the ground, and whether any aircraft has reached the finish line. ,in Indicates the first An airplane The reward value obtained after completing the state transition at each time step is determined by... and It consists of two parts, It reflects how far the aircraft is from its destination in the direction of its flight; It reflects whether the aircraft has collided or reached its destination; Step 4-2: In each learning round, each aircraft taxis from its initial position to a given destination according to the given flight schedule. The goal is to learn a taxiing trajectory that maximizes reward while avoiding conflict. When all aircraft have taxied to the destination or when there is a conflict between aircraft, this learning round ends and the next learning round begins.

Images