A real-time aircraft routing method based on multi-agent reinforcement learning

Abstract

The present invention relates to the field of aviation control technology, and discloses a real-time routing method for aircraft based on multi-agent reinforcement learning, including: constructing a vector matrix set based on the positions of several aircraft; inputting the vector matrix set into a pre-constructed multi-agent distributed control model to obtain the real-time routing of several aircraft; wherein, when constructing the vector matrix set, a directed graph of a MAPF environment is constructed based on the position of the aircraft and the spatial information of the airport or the position of the aircraft and the spatial information of the terminal area, and the vector matrix set is obtained according to the directed graph. The present invention converts the routing problem of the aircraft ground and terminal area into a MAPF problem, and proposes a MAPF algorithm based on deep reinforcement learning, which improves the solution speed of the MAPF algorithm, and can obtain a real-time routing method for aircraft that can plan paths for all aircraft on the ground or in the terminal area in a short time and can cope with various emergencies in the airport.

Description

Technical Field

[0001] The present invention belongs to the technical field of aviation control, and in particular relates to a real-time aircraft routing method based on multi-agent reinforcement learning. Background Art

[0002] Airport ground operations include several sub-issues: parking stand allocation, aircraft ground movement, and airport arrival and departure management, among which aircraft ground movement is considered the key to connecting other issues. The aircraft ground movement problem involves the entire process of the aircraft taxiing from the parking stand to the take-off runway or taxiing to the parking stand after landing. This process is carried out within the ground movement system. The advanced ground movement, guidance and control system proposed by ICAO lists aircraft ground routing as one of the key issues of aircraft ground movement. Therefore, by finding a suitable aircraft ground routing, the airport throughput can be increased and the airport ground traffic can be improved by rationally planning the ground movement of aircraft, thereby improving the efficiency of airport ground operations.

[0003] In the prior art, routing methods for ground aircraft can be divided into two categories: sequential methods and global methods. Strong real-time performance is the advantage of the sequential method, but the disadvantage of the sequential method is that when planning a path for an aircraft, it is impossible to schedule the routes of other aircraft at the same time, because it must respect the previously planned routes and cannot take into account the routes of subsequent aircraft. This makes its search space low-dimensional and can only obtain the local optimal solution at the current moment. In the sequential method, when the routes of two aircraft conflict, although the later aircraft can solve the problem by taking a detour, when the aircraft can obtain a shorter route through mutual coordination, the detour is unreasonable, and the global method is expected to consider such coordination. The advantage of the global method is that the dimension of the search space is higher, and all routes that can solve the problem can be found. By coordinating the interaction between routes, a better solution can be found, but the disadvantage of the global method is that the computational complexity is high and it is usually not suitable for online real-time decision-making. For large airports, especially during peak hours, the interaction between many routes often requires a more complex global routing algorithm, but there is currently no global planning method that can meet the requirements of real-time operation.

[0004] On the other hand, in the airport arrival management, the current flight approach usually implements the step descent method. This method has a level flight stage during the descent process, and the controller needs to issue scheduling instructions frequently, which makes the flight schedule difficult to predict, the workload of the controller and pilot is heavy, and leads to excessive fuel consumption and carbon emissions. Continuous descent (CDO) is a new operation concept proposed in recent years. Its characteristic is that once the descent begins, it will continue to descend to the final approach fix point, the aircraft intercepts the glide path, and completes the entire continuous descent process. The continuous descent process continues to maintain slow thrust, which can effectively reduce fuel consumption and carbon emissions, reduce noise pollution, and reduce the operating load of pilots and controllers compared with the traditional step descent method. At present, many airports at home and abroad have designed CDO procedures, but due to the inability to accurately predict the trajectory of aircraft, the current CDO still has safety risks in conflict detection and conflict resolution in complex airspace. Therefore, the CDO procedure is usually only applicable during low traffic periods such as early morning. The core of continuous descent in busy terminal airspace is still the path planning and conflict resolution of incoming aircraft. A method for conflict-free routing planning based on real-time airspace situation is needed.

[0005] MAPF can find a set of conflict-free paths to the destination for all agents on a graph with multiple mobile agents. It has been applied in many fields related to path planning, such as automated warehouses, traffic control, games, parking, etc. The MAPF algorithm has also appeared many times in the study of airport ground operations, but the application of the MAPF algorithm in aircraft ground routing is still very limited. One possible reason is that the MAPF algorithm is usually applied in grid maps, and the special geometric distribution of the airport taxi system hinders this application.

[0006] Although there are attempts in the prior art to use the MAPF algorithm to solve the problem of aircraft ground routing in non-grid maps, the traditional MAPF algorithm cannot be well extended to a large number of intelligent agents.

[0007] Recently, many scholars have explored the possibility of expanding deep reinforcement learning algorithms into MAPF algorithms to solve the MAPF problem. As we all know, the advantage of deep reinforcement learning algorithms is strong real-time performance. Once the model is fully trained, it only takes a short running time to achieve good results during the testing phase. Although it has been proposed to solve the MAPF problem through deep reinforcement learning methods and try to use deep reinforcement learning algorithms for aircraft routing, the deep learning models currently proposed cannot adapt well to the multi-agent characteristics of the airport environment, because extending the concept of deep reinforcement learning to the aircraft routing problem is not a mature work. Summary of the invention

[0008] The purpose of the present invention is to overcome the problem that the method for planning aircraft ground and terminal area routes in the prior art cannot find the optimal path planning in a short time and cannot cope with airport emergencies, and to provide an aircraft real-time routing method based on multi-agent reinforcement learning.

[0009] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0010] A real-time aircraft routing method based on multi-agent reinforcement learning comprises the following steps:

[0011] constructing a vector matrix set based on the positions of a number of aircraft;

[0012] Inputting the vector matrix set into a pre-built multi-agent distributed control model to obtain real-time routes of several aircraft;

[0013] When constructing the vector matrix set, a directed graph of the MAPF environment is constructed based on the aircraft's position and the airport's spatial information or the aircraft's position and the terminal area's spatial information, and the vector matrix set is obtained according to the directed graph.

[0014] As a preferred solution of the present invention, the airport status includes spatial information of aircraft, runways, taxiways and parking spaces in the airport.

[0015] As a preferred embodiment of the present invention, the step of constructing the directed graph includes: setting the vertices of the directed graph and the ranges represented by the vertices according to the spatial information, and inserting virtual nodes between the vertices; using the connection paths between the vertices as the edges of the directed graph; and setting the motion constraints of the aircraft.

[0016] As a preferred embodiment of the present invention, the multi-agent distributed control model includes local observation, action space, reward structure and deep neural network structure; the deep neural network structure includes a policy network and a value network; the local observation is used to set the type of local observation information obtained in the directed graph; the policy network is used to decide the execution action based on the local observation information; the reward structure is used to calculate the reward corresponding to the execution action; the value network is used to calculate the value estimate based on the local observation information.

[0017] As a preferred solution of the present invention, the step of constructing the vector matrix set includes: constructing an information matrix for expressing the local observation information of the agent according to the positions of the vertices around the agent in the directed graph, and the combination of the information matrices constitutes the vector matrix set.

[0018] As a preferred solution of the present invention, the vector matrix set includes: an aircraft orientation matrix, which is used to represent the current orientation of the agent;

[0019] The local map matrix is ​​used to indicate whether there are vertices within the agent's field of view and whether the vertices are occupied;

[0020] The shortest path matrix is ​​used to represent the intersection of the agent's path and the vertex, so as to calculate the shortest path;

[0021] The surrounding agent position matrix is ​​used to indicate whether there are other agents around the agent;

[0022] The shortest path direction matrix of the surrounding agents is used to represent the intersection position of the shortest path of the surrounding agents and the boundary of the field of view.

[0023] As a preferred solution of the present invention, after all agents complete the execution action at the same time step, the strategy network collects experience and then decides the execution action of all agents at the next time step until all agents reach the end point, and constructs an experience pool based on the experience at each time step; the experience pool is used to update the strategy network and the value network; the experience includes local observation information, execution action, reward and value estimation at the current time step.

[0024] As a preferred solution of the present invention, the updating step of the policy network includes: calculating the GAE advantage at each time step according to the experience pool, and calculating the cutting objective function at each time step through the GAE advantage; calculating the policy network parameters based on the cutting objective function of adjacent time steps to realize the updating of the policy network.

[0025] As a preferred embodiment of the present invention, the step of updating the value network includes: calculating the value estimate corresponding to the execution action at each time step based on the experience pool; calculating the value network parameters based on the value estimates of adjacent time steps to achieve the update of the value network.

[0026] As a preferred solution of the present invention, the spatial information of the terminal area includes the terminal area entrance and the terminal area fusion point location; when the aircraft performs route planning in the terminal area, a terminal area MAPF environment is constructed according to the spatial information of the terminal area and the aircraft's approach trajectory, and a terminal area matrix vector set is obtained from the terminal area MAPF environment; the terminal area matrix vector set is used as the input of the multi-agent distributed control model to obtain the route planning of the aircraft in the terminal area.

[0027] Based on the same concept, the present invention also proposes a real-time aircraft routing device based on multi-agent reinforcement learning, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned real-time aircraft routing methods based on multi-agent reinforcement learning.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention discloses a real-time aircraft routing method based on multi-agent reinforcement learning, which converts the aircraft routing problem into a MAPF problem, and then constructs a multi-agent distributed control model based on a deep reinforcement learning algorithm to solve it. It can plan paths for all aircraft on the ground or in the terminal area in a short time. The method is a method that approximates the global optimal method. The policy network of the multi-agent distributed control model can consider the status of surrounding agents when making agent action decisions, and schedule the taxiing paths of all aircraft at the same time, so that the decision of the policy network is better, and the path can be re-planned for the aircraft in time when dealing with various emergencies in the airport. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a flow chart of a real-time aircraft routing method based on multi-agent reinforcement learning of the present invention;

[0031] Figure 2 The directed graph after inserting the virtual vertex in Example 2;

[0032] Figure 3 A schematic diagram showing possible conflicts that may occur during the actual movement of aircraft in Example 2;

[0033] Figure 4 is the directed graph corresponding to the vertex information matrix of Example 2;

[0034] Figure 5 This is a schematic diagram of the structure of the strategy network and the value network sharing parameters, environment and experience pool in Example 2;

[0035] Figure 6 A schematic diagram of a flow chart of training a multi-agent distributed control model in Example 2;

[0036] Figure 7 A schematic diagram of a process for obtaining real-time routing using a fully trained multi-agent distributed control model according to Example 2;

[0037] Figure 8 A grid schematic diagram including spatial information of a terminal area according to Embodiment 3;

[0038] Fig. 9 This is a local map after the terminal area of ​​Example 3 is divided into a grid map. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0040] Unless otherwise specified, in the description of the specific embodiments of the present invention, the terms indicating the orientation or position relationship such as "up", "down", "left", "right", "center", "inside", "outside", etc. are all expressions based on the orientation or position relationship shown in the drawings, or the orientation or position relationship when the invented product / equipment / device is usually used. These terms of orientation or position relationship are only for the convenience of describing the scheme of the present invention or simplifying the description in the specific embodiments, so as to facilitate the technicians to quickly understand the scheme, rather than indicating or implying that a specific device / component / element must have a specific orientation, or be constructed and operated in a specific position relationship, and therefore cannot be understood as a limitation on the present invention.

[0041] In addition, if the terms "horizontal", "vertical", "overhanging", "parallel" and the like appear, it does not mean that the corresponding devices / components / elements are required to be absolutely horizontal or vertical or overhanging or parallel, but may be slightly tilted or have deviations. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but may be slightly tilted. Alternatively, it can be simplified to mean that the corresponding devices / components / elements are set in directions such as "horizontal", "vertical", "overhanging", "parallel", etc., and can have an error / deviation of ±10% relative to the corresponding direction setting, more preferably an error / deviation within ±8%, more preferably an error / deviation within ±6%, more preferably an error / deviation within ±5%, and more preferably an error / deviation within ±4%. As long as the corresponding device / component / element is within the error / deviation range, it can still achieve its role in the scheme of the present invention.

[0042] In addition, the expressions “first”, “second”, “third”, etc. that appear in the terms are merely descriptions used to distinguish the same or similar components and should not be understood as emphasizing or implying the relative importance of specific components.

[0043] In addition, in the description of the embodiments of the present invention, "several", "plurality" and "a number" represent at least 2. It can be any number such as 2, 3, 4, 5, 6, 7, 8, 9, and even more than 9.

[0044] In addition, in the description of the technical solution of the present invention, unless otherwise clearly specified / defined / restricted, the terms "set", "install", "connect", "connected", "provided with", "laid", and "arranged" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection, and can be welding, riveting, bolting, threading, and other commonly used connection means in the field. This connection can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, and it can be the internal connection of two elements.

[0045] Example 1

[0046] like Figure 1 As shown, a real-time aircraft routing method based on multi-agent reinforcement learning includes:

[0047] constructing a vector matrix set based on the positions of a number of aircraft;

[0048] Inputting the vector matrix set into a pre-built multi-agent distributed control model to obtain real-time routes of several aircraft;

[0049] When constructing the vector matrix set, a directed graph of the MAPF environment is constructed based on the aircraft's position and the airport's spatial information or the aircraft's position and the terminal area's spatial information, and the vector matrix set is obtained according to the directed graph.

[0050] As a preferred solution of the present invention, the airport status includes spatial information of aircraft, runways, taxiways and parking spaces in the airport.

[0051] As a preferred embodiment of the present invention, the step of constructing the directed graph includes: setting the vertices of the directed graph and the ranges represented by the vertices according to the spatial information, and inserting virtual nodes between the vertices; using the connection paths between the vertices as the edges of the directed graph; and setting the motion constraints of the aircraft.

[0052] As a preferred embodiment of the present invention, the multi-agent distributed control model includes local observation, action space, reward structure and deep neural network structure; the deep neural network structure includes a policy network and a value network; the local observation is used to set the type of local observation information obtained in the directed graph; the policy network is used to decide the execution action based on the local observation information; the reward structure is used to calculate the reward corresponding to the execution action; the value network is used to calculate the value estimate based on the local observation information.

[0053] As a preferred solution of the present invention, the step of constructing the vector matrix set includes: constructing an information matrix for expressing the local observation information of the agent according to the positions of the vertices around the agent in the directed graph, and the combination of the information matrices constitutes the vector matrix set.

[0054] As a preferred solution of the present invention, the vector matrix set includes: an aircraft orientation matrix, which is used to represent the current orientation of the agent;

[0055] The local map matrix is ​​used to indicate whether there are vertices within the agent's field of view and whether the vertices are occupied;

[0056] The shortest path matrix is ​​used to represent the intersection of the agent's path and the vertex, so as to calculate the shortest path;

[0057] The surrounding agent position matrix is ​​used to indicate whether there are other agents around the agent;

[0058] The shortest path direction matrix of the surrounding agents is used to represent the intersection position of the shortest path of the surrounding agents and the boundary of the field of view.

[0059] As a preferred solution of the present invention, after all agents complete the execution action at the same time step, the strategy network collects experience and then decides the execution action of all agents at the next time step until all agents reach the end point, and constructs an experience pool based on the experience at each time step; the experience pool is used to update the strategy network and the value network; the experience includes local observation information, execution action, reward and value estimation at the current time step.

[0060] As a preferred solution of the present invention, the updating step of the policy network includes: calculating the GAE advantage at each time step according to the experience pool, and calculating the shearing objective function at each time step through the GAE advantage;

[0061] The policy network parameters are calculated based on the clipping objective function of adjacent time steps to achieve the update of the policy network.

[0062] As a preferred embodiment of the present invention, the step of updating the value network includes: calculating the value estimate corresponding to the execution action at each time step based on the experience pool; calculating the value network parameters based on the value estimates of adjacent time steps to achieve the update of the value network.

[0063] As a preferred solution of the present invention, the spatial information of the terminal area includes the terminal area entrance and the terminal area fusion point location; when the aircraft performs route planning in the terminal area, a terminal area MAPF environment is constructed according to the spatial information of the terminal area and the aircraft's approach trajectory, and a terminal area matrix vector set is obtained from the terminal area MAPF environment; the terminal area matrix vector set is used as the input of the multi-agent distributed control model to obtain the route planning of the aircraft in the terminal area.

[0064] Based on the same concept, the present invention also proposes a real-time aircraft routing device based on multi-agent reinforcement learning, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned real-time aircraft routing methods based on multi-agent reinforcement learning.

[0065] Example 2

[0066] Taking Tianfu Airport as an example, a real-time aircraft routing method based on multi-agent reinforcement learning is implemented, including:

[0067] S1, abstract the network structure of the taxiways in the airport map into a directed graph, which includes:

[0068] The intersections of runways and taxiways, the intersections between taxiways, and parking stands in the taxi system are regarded as vertices, and the connection paths between vertices are regarded as edges to construct a directed graph, in which vertices are bidirectionally linked.

[0069] This directed graph does not satisfy the assumption of the classic MAPF for the following reasons: Since the path lengths between vertices are not all equal, if two planes start from a vertex at the same time, after one time step, one plane may reach the next vertex, but the other plane is still between the vertices. At the beginning of the next time step, this state violates the assumption of the classic MAPF: all agents should be located at the vertices. To deal with this problem, we insert a virtual vertex between two vertices, and the inserted virtual vertex satisfies the following rules:

[0070] 1) An airplane is a very large entity that occupies a large physical space, so the vertex described in this embodiment is not just a point in the physical space, but represents an area with a range of 50 meters. 50 meters is chosen because the length of common civil aircraft is roughly 30, 40 or 50 meters, and an airplane can just occupy a vertex.

[0071] 2) In the taxiway, the distance between aircraft is at least about 50 meters, so a vertex is inserted every 50 meters. Not all path lengths are exactly multiples of 50 meters, but try to ensure that the deviation is not too large and the distance between vertices is as equal as possible.

[0072] 3) The positional relationship between adjacent vertices can be up, down, left, or right. This is sufficient for the taxiing system of Tianfu Airport. For airports with more irregular paths, more complex positional relationships may need to be considered. For example, there may be 8 positional relationships between adjacent vertices: up, down, left, right, upper left, lower left, upper right, and lower right.

[0073] like Figure 2 As shown in the figure, after inserting virtual vertices according to this rule, the following vertices can be obtained on the entire directed graph: runway and taxiway intersection vertices, taxiway intersection vertex parking space nodes and virtual vertices, a total of 423 nodes; in order to reduce the number of targets, we cluster multiple adjacent parking spaces into one parking space area, which can reduce the search space of path planning and speed up the training of the algorithm.

[0074] In such a directed graph, we can directly define which edges cannot be used to exclude some unreasonable behaviors of the aircraft. For example, when the aircraft is located at a vertex, we can manually define a valid set of outgoing edges to prohibit the aircraft from making unreasonable turns, backing up, etc.; for example, if an accident occurs at the airport and a certain path suddenly cannot be used, the inbound and outbound edges related to this path can be directly disabled. These features enable the model to simulate the airport surface more realistically while reducing the burden on the routing algorithm.

[0075] Figure 2 The directed graph is created by directly importing the airport map into CAD and then dividing the airport map into a 1:1 length. Therefore, the size and proportion of the entire directed graph fully reflects the actual situation of Tianfu Airport.

[0076] S2, defines the movement mode of the agent, including:

[0077] The taxiing of an aircraft is regarded as an intelligent agent moving between the vertices and edges of a directed graph. This movement must conform to the actual operation rules of the airport ground and meet the definition of the classic MAPF. The details are as follows:

[0078] Time is discretized into time steps, and at the beginning of each time step, the agent is at a vertex and can perform an action. There are two types of actions: waiting actions, which means the agent stays at the current position, and moving actions, which means the agent moves to an adjacent vertex. The duration of each action is one time step, which means that the agent that performs the moving action will reach the adjacent vertex after one time step.

[0079] This is a simplified motion model, which assumes that the speed of the agent on the edge is equal, multiple agents move synchronously, and will reach the next vertex after one time step. However, in reality, the aircraft cannot be completely synchronized, and this is when the inserted virtual vertex comes into play. Both the aircraft and the vertex have volume, so as long as part of the aircraft is located on a vertex, it is considered to have reached the vertex. If the aircraft is completely between two vertices, it is considered to be on the nearest vertex in front when doing path planning in the next time step. These assumptions ensure that the aircraft's motion fits the actual situation as much as possible, while satisfying the assumptions in the classic MAPF.

[0080] During the actual movement of multiple aircraft, conflicts such as head-on collision, tailgating, and crossing may occur. We define the following types in the directed graph to represent them: vertex conflict, edge conflict, following conflict, loop conflict, and exchange conflict. Figure 3 These conflicts are shown. It is worth noting that these conflicts are not completely independent. There is a dominance relationship between them. For example, prohibiting vertex conflicts means that edge conflicts and exchange conflicts will not occur, and prohibiting follow-up conflicts means that cycle conflicts will not occur.

[0081] In the process of algorithm solving, these types of conflicts between agents must be strictly avoided. The motion constraints of the agents are as follows:

[0082] (1) The agent is not allowed to retreat;

[0083] (2) The agent is not allowed to move outside the path of the sliding system. Such behavior is restricted by adding action masks to the agent's observations.

[0084] (3) Vertex conflicts are not allowed; this is done by removing edges in the directed graph that may cause conflicts: when the aircraft is located on a vertex, the vertex is considered occupied, and all edges leading to the vertex are temporarily disabled. The same is true when a vertex fails and is unavailable. If an edge fails and is unavailable, the edge is disabled. This method also completely prohibits edge conflicts, follow-up conflicts, loop conflicts, and exchange conflicts;

[0085] (4) Following conflicts are not allowed. There are two situations: when the agent in front has decided to move forward in the next step, the agent behind can choose to move forward; when the aircraft behind chooses an action, the aircraft in front has not made a decision, or the aircraft in front stays in place, the agent behind is not allowed to move forward.

[0086] S3, design a multi-agent distributed control model based on deep reinforcement learning, including:

[0087] We describe the multi-agent planning problem in terms of multi-agent Markov decision processes (MDPs). Assume that there are n agents in the system, S represents the joint state space of all agents, S = {o1, o2…, o n}; A={A1, A2, ...A n} represents the joint action space of all agents. In this example, we consider the partially observable case, so each agent can only observe its own state. At time t, agent i observes state o i ∈S, where S is the joint state space; then perform action a t ∈A i (o i ), where A(o i ) is a local observation o i After executing the action, we get a reward r from the environment. t = r(S t ,a t ,S t+1 )∈R, and then transfer to the new state S t+1 The process ends after a certain number of time-steps, which is called an episode.

[0088] The following is a detailed description of each agent's local observation o i , action space A i And the reward function R:

[0089] 1) Local observation

[0090] a. The direction of the aircraft. A 1×4 0 / 1 vector, indicating the direction the aircraft is currently facing. The possible directions are up, down, left, and right.

[0091] b. Local map information. A local map with a field of view of 5, centered on the current aircraft. The map shows the vertex layout around the current aircraft and the connection relationship between vertices. In order to represent the connectivity between vertices, a 0-1 variable needs to be inserted between every two vertices to represent the connectivity between the two vertices. Therefore, the final local map information is a 9×9 0-1 matrix. If some vertices within the field of view are already occupied, the corresponding position of the matrix is ​​set to 0; if there are no vertices at certain positions within the field of view, the corresponding position of the matrix is ​​set to 0; where there are vertices, the corresponding position of the matrix is ​​set to 1.

[0092] like Figure 4 As shown, we take the local map information with a field of view of 3 to explain how to insert variables representing the connectivity between vertices. Assuming that the aircraft is located at vertex 117, then within a field of view of 3, the surrounding vertex information matrix should be:

[0093]

[0094] Among them, all positions with a value of 1 indicate that there is a vertex at this position;

[0095] In order to indicate the connection relationship between vertices, if the vertices are connected, a 1 is inserted between the two vertices, and if the two vertices are not connected, a 0 is inserted. After inserting the vertices, the local map information matrix should be:

[0096]

[0097] Among them, all the positions with a value of 1 are connected to form two traversable paths. Figure 4 As shown, it can be seen that the position of vertex 133 cannot directly lead to vertex 134.

[0098] c. Shortest route. A 5×5 0-1 matrix represents the shortest route to the destination in a field of view with a range of 5, centered on the agent. The agent's shortest route is obtained through the standard A* algorithm, assuming that the agent is located at the center of a field of view with a range of 5, with the agent's current position as the starting point and the target position as the end point. The shortest route intersects with the vertices in the field of view. For the intersection, the corresponding position in the matrix will be set to 1, and the non-intersection position will be set to 0. This matrix tells the agent the shortest possible route within the field of view.

[0099] d. The position of the surrounding agents. A 5×5 0-1 matrix shows whether there is an agent at the vertex within the field of view of 5 with the current aircraft as the center. If so, the corresponding position in the matrix is ​​1, otherwise it is 0

[0100] e. The shortest route direction of the surrounding agents. A 5×5 0-1 matrix. If there is an agent within the field of view of 5 with the current aircraft as the center, the intersection position of the shortest route of the surrounding agents and the boundary of the field of view is recorded, and the corresponding position in the matrix is ​​set to 1, and the rest of the positions are set to 0.

[0101] f. Action mask: A 1×4 0-1 vector representing the actions that the agent can perform at the current position. This vector is input into the policy network, which avoids choosing infeasible actions. Compared with giving negative rewards to illegal actions, this method converges much faster, which is extremely important for accelerating convergence.

[0102] Note that all the above local observation information is provided by the environment to the agent, and there is no explicit communication between agents.

[0103] 2) Action Space

[0104] The size of the agent's action space is 4, representing driving forward, turning left, turning right, and slowing down and waiting.

[0105] There is no backward movement because it is impossible for an aircraft to move backward while taxiing.

[0106] At each time step, some actions may be invalid, such as moving outside the taxiway or crashing into other agents. During training, we use masks to sample only valid actions. This allows for more stable training than giving negative rewards to invalid actions.

[0107] 3) Reward structure

[0108] As shown in Table 1, the left side of the table is the reward value we preset for this result, and the right side refers to the result caused by the action performed by the agent. The reward structure is shown in the following table:

[0109] Table 1 Airport Reward Structure

[0110] -1 Moving forward and away from the target -0.5 Moving forward and approaching the goal -1.5 Turn and move away from the target -1 Turning and approaching the target -2 Standing still 30 Arrive at the destination

[0111] The agent is penalized for each action it takes, which encourages it to reach the goal as quickly as possible. The penalty for standing still is slightly heavier than for taking one action, which is to encourage the agent to explore. The penalty for turning is slightly heavier, which is to encourage the agent to turn as little as possible. The reward function is an implicit reflection of the goal. One of our goals is to make the plane glide as short as possible. This reward function setting is intuitive.

[0112] 4) Network structure

[0113] The network structure is as follows: the policy network and the value network share the feature extraction layer. The feature extraction layer is processed as follows: the information from the four local observations, local map information, shortest route, location of surrounding agents, and target of surrounding agents, is processed through two layers of convolution layers with a convolution kernel size of 3, and then processed into one-dimensional data through the flatten layer, and then concatenated with the 0-1 vector representing the direction of the aircraft. The 0-1 vector of the direction of the aircraft needs to be processed by a fully connected layer with an output of 12 before concatenation. The output layer of the policy network outputs a 1×4 vector, which represents the probability of this policy choosing a certain action, and the output layer of the value network outputs a floating point number, which represents the value network's score for the current state.

[0114] 5) Algorithm training architecture

[0115] The algorithm architecture is an extension of the Proximal Policy Potimization (PPO) algorithm. The PPO algorithm is an actor-critic-based algorithm and is considered to be one of the most advanced deep reinforcement learning algorithms. It combines the ideas of A2C (Advantage Actor Critic) and TRPO (Trust Region Policy Optimization), and uses multiple techniques (importance sampling, truncated policy ratio, and multi-step update) to overcome the problem that on-policy algorithms cannot use experience replay. The original PPO algorithm is only applicable to a single agent. We extend the single-agent PPO to distributed multi-agents by sharing network parameters and experience pools. This scheme allows the experience of all agents to be used to train a policy at the same time. The final result is that multiple agents jointly learn a general single-agent policy. The feasibility of this method lies in the fact that during the taxiing of multiple aircraft, all agents are homogeneous and the abilities they should master are similar. In this way, the experience of multiple agents is used to jointly learn a general single-agent policy, and the learned policy can be copied to any number of agents. It is a distributed control architecture. Sharing network parameters at the same time means that the value network and the policy network share the results of feature extraction, which can reduce the operation of feature extraction. In addition, sharing the experience pool means that the experience of all agents will be mixed in one experience pool, which will cause the experience of different trajectories to cross each other, which is not conducive to the update of the policy network and the value network. Therefore, we set a unique identifier for each agent. This unique identifier is an integer, which gradually accumulates from 1. The experience of each agent also identifies which agent this experience belongs to. Using the experience of all agents to train a strategy together, such a strategy is universal and generalized, applicable to every agent, and any aircraft that needs to plan a path can use this strategy. It is precisely because our strategy has such generalization that we can achieve distributed control of aircraft.

[0116] The distributed algorithm training framework operates as follows: Figure 5 As shown, flight n represents the nth agent, and all agents share the environment. After an agent performs an action, the shared environment will also change accordingly, ensuring that each agent can obtain real-time local observation information from the shared environment.

[0117] Specifically, in the first time step, agent 1, flight 1, obtains a local observation o from the environment 1 , the policy network makes decisions based on local observations and obtains the next action a 1 , Agent 1 performs action a 1 , get reward r from the environment1 Then the local observation o 1 Input the value network, which calculates the value estimate v(o 1 ), and then the experience (o 1 , a 1 , r 1 ,v(o 1 )) is stored in the buffer. After Agent 1 takes action, Agent 2 makes decisions and takes actions in the same way to gain experience (o 2 , a 2 , r 2 ,v(o 2 )) is also stored in the buffer until all agents have completed one time step and a cycle ends, and then the next cycle starts from agent 1; all agents’ policy networks share the parameter w, all value networks share the parameter, and all agents share the experience pool buffer. An identifier is needed to distinguish which trajectory each experience comes from, so as to calculate the GAE advantage (Generalized Advantage Estimation) when updating the network parameters.

[0118] S4, training the multi-agent distributed control model described in S3, specifically including:

[0119] like Figure 6 As shown, the entire training process is as follows:

[0120] 1) Randomly initialize the parameters of the policy network and the value network, and set the hyperparameters during the training process, such as the learning rate α = 0.0003, the multi-step update parameter n_epochs = 10, the clipping parameter ε = 0.2, the discount return factor γ = 0.99, and the GAE parameter λ = 0.95. Gradually optimize the parameters of the policy network and the value network during the training process.

[0121] 2) Initialize the training aircraft queue [f1, f2…], and store the aircraft whose paths are to be planned in the training aircraft queue. Each aircraft has its own starting and ending positions. The starting point of the incoming aircraft is the vertex of the intersection of the runway and the taxiway, and the end point is the parking area. The opposite is true for the departing aircraft. Each aircraft will randomly generate a starting point and an end point from all possible starting points and end points, but two aircraft are not allowed to use the same starting point or end point at the same time; the starting point and end point of the aircraft in the aircraft queue during training are randomly initialized, so the experience collected by reinforcement learning contains various possible situations. The multi-agent distributed control model can learn various possible situations. The multi-agent distributed control model can adapt to various actual conditions and respond to emergencies, and give the correct route.

[0122] 3) Construct an iterator iter to help us iterate the aircraft queue F sequentially; the iteration process is to let the aircraft in the aircraft queue execute the decisions made by the policy network in turn, obtain rewards, and collect experience.

[0123] 4) If the time step step is less than the update threshold T (T represents the interval at which the policy network is updated. We set T = 1024, and the size of T also corresponds to the size of the experience pool), then jump to 5) to continue collecting the experience;

[0124] If the time step is greater than the update threshold T, an update is performed and the update process is as follows:

[0125] a. Randomly sample a mini_batch of experience from the experience pool buffer. The size of the buffer is 1024, which stores the experience of the most recent 1024 time steps. In this embodiment, the size of a mini_batch is set to 64. Randomly sample a mini_batch of experience from the buffer for updating the parameters of the value network and the policy network. The sampled experience is not sampled again, and the sampling is repeated until all the data has been sampled;

[0126] b. Calculate the GAE (Generalized Advantage Estimation) advantage. The calculation formula for the GAE advantage at a certain time step k is:

[0127]

[0128] δ k =r k +γv(o k+1 )-v(o k );

[0129] Among them, 0≤t≤k≤n, n is the size of mini_batch 64, t is the current time step, o k is the local observation information of k time steps, r k is the reward at time step k, v(o k ) is the local observation information o of the value network at time step k k The value estimate of ; γ is the discount return factor, λ is the GAE parameter;

[0130] The policy network is updated by gradient ascent of the policy network parameters with the help of the clipping objective function; the value network is updated by back-propagation and gradient descent of the mean squared error (MSE) between the value network's estimate and the actual reward.

[0131] The clipping objective function is:

[0132] L clip (θ)=E[min(r t (θ)A t ,clip(r t (θ),1-ε,1+ε)A t )];

[0133] r t (θ)=π new (a t |o t ) / π old (a t |o t );

[0134] Among them, r t (θ) represents the probability ratio between the new and old strategies at time step t; A t is the GAE advantage at time step t; clip(r t (θ),1-ε,1+ε) will r t (θ) is restricted to the range [1-ε,1+ε].

[0135] The update formula of the policy network parameter θ is:

[0136]

[0137] Among them, L clip (θ) is the aforementioned clipping objective function, θ is the policy network parameter before updating, and θ' is the policy network parameter after updating. The gradient of θ is calculated for the clipping objective function L, and then θ is updated in the direction of gradient ascent.

[0138] The update formula of the value network parameter w is:

[0139]

[0140] Among them, w is the value network parameter before update, w′ is the value network parameter after update, v w (s t ) refers to the value network parameter w, the value network for the state s at t time step t estimated value.

[0141] c. Check whether the model training target is achieved. The training target is that the aircraft path finding success rate reaches an average of more than 99%. If the training target is achieved, the training ends; otherwise, continue to perform experience collection and jump to 5);

[0142] 5) Continue to collect data. Use the iterator iter to get the next action of the aircraft f, and obtain the current local observation o of the aircraft f from the environmentt , using the current policy network π θ Make a decision and get the next action a t ;

[0143] 6) Execute the next step a t , the current aircraft f will transfer from vertex v to v' and receive reward r t ; Then take the data of this step (o t ,a t ,r t ,value t ) is stored in buffer;

[0144] 7) Check if the current aircraft is the last aircraft in the queue, otherwise jump to 4) and try to iterate the next aircraft; this means that a cycle is over, all aircraft on the scene have taken one step, and then go to 8);

[0145] 8) Check whether there are any aircraft in the aircraft queue F that have reached the end point, whether there are any aircraft that are ready to enter the taxiing system, and whether there are any aircraft that are ready to be pushed out. If there are any aircraft that have reached the end point, remove the aircraft from the aircraft queue F; if there are any aircraft that are ready to enter the taxiing system or ready to be pushed out, add the aircraft to the aircraft queue F;

[0146] 9) Check whether there are still aircraft in the aircraft queue F. If there are, it means that the next cycle needs to be entered, and the aircraft in the aircraft queue are iterated, and jump to 3). If there are no aircraft, it means that all aircraft have reached the end point, jump to 2), reinitialize the environment for the next round of training; the end point of model training is that the success rate of aircraft finding the path reaches an average of more than 99%. When this success rate is reached, the training is terminated and a fully trained multi-agent distributed control model is obtained.

[0147] S5, using a fully trained multi-agent distributed control model to route aircraft;

[0148] After the training is completed, the trained value network and policy network are obtained. The value network and policy network are optimized to a suitable parameter. The policy network has the ability to find a path for the aircraft in various states. Only the policy network is needed during the execution process. Since we train according to the distributed control method during the training process, the execution process is also distributed execution; Figure 7 As shown, the actual execution process includes:

[0149] 1) When there is an aircraft in the taxiing system that needs to plan a path, if the aircraft queue F is empty at this time, the MAPF environment is initialized according to the definition, and then the aircraft queue is initialized, and all aircraft that need to plan paths are added to the aircraft queue F;

[0150] 2) Synchronize the current position of all aircraft in the real environment, the target position of the aircraft, whether each taxi path is available, whether each intersection and parking position is available, and other status information into the MAPF environment;

[0151] 3) The input of the policy network is the local observation of each aircraft i , the output is the action a in the next step of aviation i , a i Including driving forward, turning left, turning right, or slowing down and waiting. Since the policy network only plans the next step of the action of an agent at a time, the aircraft queue is iterated for multiple steps according to the previously trained strategy until each aircraft finds a complete path to the destination. If an aircraft cannot find a path to the destination within the specified number of iterations (80 steps), the controller will be alerted and the controller can check whether there is an unexpected situation, such as the parking space being occupied by mistake.

[0152] 4) The controller guides the aircraft actions by referring to the paths planned by the strategic network;

[0153] 5) After one step, check whether any aircraft has arrived at the destination and whether any aircraft is ready to enter the taxiing system or be pushed out;

[0154] 6) If no aircraft has arrived at the destination and no aircraft is ready to enter the taxiing system or to be pushed out, the environment is synchronized and paths are planned for all aircraft, that is, steps 2) to 5) are executed again; if an aircraft has arrived at the destination, the aircraft is deleted from the aircraft queue F; if an aircraft is ready to enter the taxiing system or to be pushed out, the aircraft is added to the aircraft queue F, and then the environment is synchronized, and paths are planned for all aircraft, that is, steps 2) to 5) are executed again.

[0155] During the execution of the algorithm, a step time corresponds to a real time in the real world, that is, the time it takes for an aircraft to move from one vertex to another in the MAPF environment. The distance between two vertices is about 50m, and the speed of the aircraft in the taxiing system is generally 30-50km / h. According to 30km / h, it takes about 6s for the aircraft to move from one vertex to another, so a step time can be set to 6s. (In fact, the time of a step can be set shorter. As long as the information in the real environment can be synchronized to the MAPF environment, the operation of the algorithm actually only takes tens of milliseconds. It is recommended to set it to 6s because for airports, this update frequency is sufficient in most cases, which can reduce unnecessary path updates).

[0156] The algorithm is a global algorithm. The paths of all aircraft are not generated sequentially but simultaneously. When planning the path of an aircraft, the dynamics of other aircraft are also taken into account.

[0157] The algorithm is a real-time dynamic algorithm. Through the above process, each step can re-plan the path for all aircraft according to the actual situation. This dynamicity enables the algorithm to adapt well to the uncertainty problems in the airport, such as runway changes or temporary route closures, which require changes to the target location and route. When such problems occur, it is only necessary to synchronize the environmental changes to the MAPF environment, and the aircraft can get the re-planned path in time.

[0158] The execution of the algorithm is distributed, and each aircraft independently uses the policy network to plan its path. There is no need for explicit communication between aircraft. The policy network only needs local observation of the aircraft to plan for it.

[0159] Example 3

[0160] The aircraft real-time routing method based on multi-agent reinforcement learning according to Example 2 can also be applied to the real-time routing planning of aircraft in the terminal area after adaptive modification;

[0161] Specifically, the steps for obtaining the MAPF environment in the terminal area include:

[0162] A) The flight follows continuous descent operations (CDO) approach and landing, and the flights have the same descent speed. The altitude of the flight at any track point can be determined after the entry altitude and flight time are given. Therefore, the altitude of the aircraft does not need to be considered in the track, and the three-dimensional trajectory planning can be simplified to a two-dimensional trajectory planning along the horizontal direction of the terminal area.

[0163] B) Discretize time into a series of time windows T = {t1, t2, t3...}, each time window t has the same duration, and within each time window, the aircraft receiving the approach instruction needs to re-plan the path and resolve the conflict. The length of the time window can be defined according to the busyness. When the terminal area is busy, the length of the time window can be shortened, otherwise it can be increased.

[0164] like Figure 8 As shown in the figure, the terminal airspace is divided into a number of uniformly sized, interconnected square grids horizontally, and the side length of the square grid is equal to the size of the time window multiplied by the flight speed of the aircraft in the terminal area. The flight speed regulations for aircraft in the terminal areas of different airports may be different, but the flight speeds of all aircraft in the same terminal area are considered to be the same.

[0165] In this environment, our problem can be described as follows: there may be flights from different directions entering the terminal area from different entrances, and we need to plan a continuous descent approach trajectory for all flights that receive approach instructions. The approach trajectory is divided into multiple stages, the target point of each intermediate stage is the intermediate fusion point, and the target of the last stage is the final fusion point.

[0166] Add constraints to the MAPF environment in the terminal area, including:

[0167] 1) Fusion point restriction: For safety reasons, only two routes can intersect at each fusion point at the same time.

[0168] 2) Aircraft performance limitations: All aircraft follow their own performance limitations, and the aircraft turning angle must meet actual requirements. Therefore, we stipulate that the single turning angle of the aircraft cannot exceed 45°, so the aircraft can only choose to move forward, turn left 45°, or turn right 45° each time;

[0169] 3) Conflict restriction: No head-on, crossing or rear-end collision is allowed between any two aircraft. In the MAPF environment, this means that no two aircraft are allowed to arrive at the same grid at the same time.

[0170] 4) Safety interval limit: To prevent the following aircraft on the same route from being affected by the wake of the preceding aircraft, all aircraft need to follow the control interval limit. According to civil aviation regulations, during the approach phase, the wake interval between the preceding and following heavy and medium aircraft is no less than 2 minutes. In our MAPF environment, we assume that the flight trajectory of the aircraft cannot be used by other aircraft within n time windows, where n×time window length=2 minutes.

[0171] The settings of local observation, action space and reward function in the MAPF environment in the terminal area include:

[0172] 1) Local observation

[0173] a. Aircraft direction. A 1×8 0 / 1 vector, indicating the direction the aircraft is currently facing. There are 8 possible directions: up, down, left, right, upper left, lower left, upper right, and lower right;

[0174] b. Local map information. A local map with a range of 9 grids, centered on the current aircraft. This map shows whether there are aircraft in the grids around the current aircraft or points that are inaccessible due to aircraft wake restrictions and fusion point restrictions. The map is represented in the form of a 0 / 1 matrix. If there are inaccessible points, the corresponding position of the matrix is ​​set to 0, and the remaining positions are set to 1. Fig. 9 As shown, due to the existence of aircraft 2, the local map of aircraft 1 should be expressed as:

[0175]

[0176] c. Shortest route. A 9×9 0-1 matrix represents the shortest route to the destination in a field of view with a range of 9, centered on the agent. The agent's shortest route is obtained through the standard A* algorithm, assuming that the agent is located at the center of a field of view with a range of 9, with the agent's current position as the starting point and the target position as the end point. The shortest route intersects with the vertices in the field of view. For the intersection, the corresponding position in the matrix will be set to 1, and the non-intersection position will be set to 0. This matrix tells the agent the shortest possible route within the field of view.

[0177] d. The position of the surrounding aircraft. A 9×9 0-1 matrix shows whether there is an intelligent agent on the vertex within the field of view of 9 with the current aircraft as the center. If so, the corresponding position of the matrix is ​​1, otherwise it is 0

[0178] e. The shortest route direction of the surrounding agents. A 9×9 0-1 matrix. If there is an agent within the field of view of 5 with the current aircraft as the center, the intersection position of the shortest route of the surrounding agents and the boundary of the field of view is recorded, and the corresponding position in the matrix is ​​set to 1, and the remaining positions are set to 0.

[0179] f. Action mask. A 1×3 0-1 vector representing the actions that the agent can perform at its current position. This vector is fed into the policy network, which avoids choosing infeasible actions.

[0180] 2) Action Space

[0181] The size of the action space of the agent is 3, representing forward flight, 45° left turn, and 45° right turn. Due to the existence of inaccessible grids, some actions may be invalid in each time window. During training, we use masks to sample only valid actions.

[0182] 3) Reward structure

[0183] As shown in Table 2, the left side of Table 2 is the reward value we preset for this result, and the right side refers to the result caused by the action performed by the agent. The reward structure is shown in the following table:

[0184] Table 2 Terminal Area Reward Structure

[0185] -1 Moving forward and away from the target -0.5 Moving forward and approaching the goal -1.5 Turn and move away from the target -1 Turning and approaching the target 10×n Other aircraft within visual range 30 Arrive at the destination

[0186] The agent will be penalized for each action it takes, thus encouraging it to reach the goal as quickly as possible.

[0187] Because high-altitude aircraft are faster, for safety reasons, we hope that aircraft will not get close to each other as much as possible. Therefore, when an aircraft approaches other aircraft within the field of view, it will be penalized 0.5×n, where n is determined by the distance between the two aircraft. When the distance between the two aircraft is twice the length of the grid side, n is -1 / 2, when the distance between the two aircraft is three times the length of the grid side, n is -1 / 3, and so on. Therefore, when two aircraft are very close to each other, the penalty is very heavy, and when they are far apart, the penalty is lighter, just like there is a repulsion between the two aircraft, the closer they are, the greater the repulsion.

[0188] 4) Network structure

[0189] The network structure is as follows: the policy network and the value network share the feature extraction layer. The feature extraction layer is processed as follows: the information from the four local observations, local map information, shortest route, location of surrounding intelligent agents, and target of surrounding intelligent agents, is processed through three layers of convolution layers with a convolution kernel size of 3, and then processed into one-dimensional data through the flatten layer, and then concatenated with the 0-1 vector representing the direction of the aircraft. The 0-1 vector of the direction of the aircraft needs to be processed by a fully connected layer with an output of 24 before concatenation. The output layer of the policy network outputs a 1×3 vector, which represents the probability of this policy choosing a certain action, and the output layer of the value network outputs a floating point number, which represents the value network's score for the current state.

[0190] Training process:

[0191] 1) Initialize network parameters.

[0192] 2) Initialize the training aircraft queue. This is similar to the previous approach, except that the starting point of this aircraft queue is the terminal area entry point, and the end point is the intermediate fusion point / final fusion point.

[0193] 3) Construct an iterator iter to help us iterate over the aircraft queue F sequentially.

[0194] 4) Gather experience / update network parameters based on the time step, in the same way as ground routing.

[0195] 5) Continue collecting data in the same way as for ground routing.

[0196] 6) Execute the next action, the aircraft will move from one grid point to the adjacent grid point, obtain the reward and store the data (ot, at, rt, valuet) in the buffer.

[0197] 7) Check if the current aircraft is the last aircraft in the queue, otherwise jump to 4) and try to iterate the next aircraft; this means that a cycle is over, all aircraft on the scene have taken one step, and then go to 8);

[0198] 8) Check whether any aircraft in the aircraft queue F has arrived at the destination or entered the terminal area. If any aircraft has arrived at the destination, remove the aircraft from the aircraft queue F. If any aircraft is about to enter the terminal area, add the aircraft to the aircraft queue F.

[0199] 9) Check whether there are still aircraft in the aircraft queue F. If there are, it means that the next cycle needs to be entered, and the aircraft in the aircraft queue are iterated, and jump to 3). If there are no aircraft, it means that all aircraft have reached the end point, jump to 2), reinitialize the environment for the next round of training; the end point of model training is that the success rate of aircraft finding the path reaches an average of more than 99%. When this success rate is reached, the training is terminated and a fully trained multi-agent distributed control model is obtained.

[0200] Use a trained multi-agent distributed control model to route aircraft:

[0201] 1) When an aircraft in the terminal area needs to perform an approach operation, if the aircraft queue F is empty at this time, the MAPF environment is initialized according to the definition, and then the aircraft queue is initialized, and all aircraft that need to plan paths are added to the aircraft queue F;

[0202] 2) Whether the current position of all aircraft in the real environment, the target position of the aircraft, and each grid point in the airspace can be synchronized to the MAPF environment using equal state information;

[0203] 3) The input of the policy network is the local observation of each aircraft i , the output is the action a in the next step of aviation i , a i Including flying forward, turning left, and turning right. Since the policy network only plans the next step of the action of an agent at a time, the aircraft queue is iterated for multiple steps according to the previously trained strategy until each aircraft finds a complete path to the destination. If an aircraft cannot find a path to the destination within the specified number of iterations (80 steps), the controller will be notified.

[0204] 4) The pilot flies according to the planned path

[0205] 5) After a time window, check whether there are any aircraft arriving at the destination and whether there are any aircraft ready to perform approach operations. As mentioned above, the length of the time window can be specified manually. The effect may be different if different time windows are specified according to the busyness of the airspace. (The time window here is the same as the time step of the ground path planning, both representing a period of time)

[0206] 6) If no aircraft has arrived at the destination and no aircraft is ready to perform approach operations, synchronize the environment and then plan paths for all aircraft, that is, execute steps 2) to 5) again; if an aircraft has arrived at the destination, delete the aircraft from the aircraft queue F; if an aircraft is ready to perform approach operations, add the aircraft to the aircraft queue F, then synchronize the environment, and then plan paths for all aircraft, that is, execute steps 2) to 5) again.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time aircraft routing method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: constructing a vector matrix set based on the positions of a number of aircraft; Inputting the vector matrix set into a pre-built multi-agent distributed control model to obtain real-time routes of several aircraft; Wherein, when constructing the vector matrix set, a directed graph of the MAPF environment is constructed based on the position of the aircraft and the spatial information of the airport or the position of the aircraft and the spatial information of the terminal area, and the vector matrix set is obtained according to the directed graph; The multi-agent distributed control model includes local observation, action space, reward structure and deep neural network structure; The deep neural network structure includes a policy network and a value network; the policy network and the value network share a feature extraction layer, in which the information of the four local observations, namely, local map information, the shortest route, the position of the surrounding intelligent agents, and the target of the surrounding intelligent agents, is processed by two layers of convolution layers with a convolution kernel size of 3, and then processed into one-dimensional data by a flatten layer, and then concatenated with a 0-1 vector representing the direction of the aircraft. The 0-1 vector of the direction of the aircraft is processed by a fully connected layer with an output of 12 before concatenation. The output layer of the policy network outputs a 1×4 vector representing the probability of this policy selecting a certain action, and the output layer of the value network outputs a floating point number representing the score of the value network for the current state; the policy network is used to decide on the execution of an action based on the local observation information, and the value network is used to calculate the value estimate based on the local observation information; The reward structure is used to calculate the reward corresponding to the execution action; the action performed by the agent has a corresponding reward value: the reward value of "moving forward and away from the target" is "-1"; the reward value of "moving forward and approaching the target" is "-0.5"; the reward value of "turning and moving away from the target" is "-1.5"; the reward value of "turning and approaching the target" is "-1"; the reward value of "approaching other aircraft within the field of view" is "10×n, n is determined by the distance between the two aircraft"; the reward value of "reaching the target point" is "30"; The action space includes: the action space size of the agent is 4, representing driving forward, turning left, turning right, and slowing down and waiting; The local observation is used to set the type of local observation information obtained in the directed graph, including aircraft direction, local map information, shortest route, positions of surrounding aircraft, shortest route directions of surrounding agents, and action masks; The step of constructing the vector matrix set includes: constructing an information matrix for expressing the local observation information of the agent according to the positions of the vertices around the agent in the directed graph, and the combination of the information matrices constitutes the vector matrix set, and the vector matrix set includes: The aircraft orientation matrix is ​​used to represent the current orientation of the agent. The orientation of the aircraft is represented by a 1×4 vector, where the value of the 1×4 vector is 1 or 0, representing the four directions of up, down, left, and right. The local map matrix is ​​used to indicate whether there are vertices within the agent's field of view and whether the vertices are occupied. The local map matrix represents the vertices around the current aircraft as the center, as well as the connection relationship between vertices. In the local map matrix, a value of 0 indicates that the vertex is occupied or there is no vertex at some position; a value of 1 indicates that there is a vertex at the corresponding position; in addition, if the vertices are connected, a 1 is inserted between the two vertices; if the two vertices are not connected, a 0 is inserted; The shortest route matrix is ​​used to represent the intersection of the agent's route and the vertex, so as to calculate the shortest route; the shortest distance to the end point with the agent as the center, the shortest route matrix is ​​obtained by the standard A* algorithm; the shortest route and the vertex have intersections, and for the intersection points, the corresponding positions in the shortest route matrix are set to 1, and the non-intersection positions are set to 0; The surrounding agent position matrix is ​​used to indicate whether there are other agents around the agent. If there are agents with the current aircraft as the center, the intersection of the shortest route of the surrounding agents and the boundary of the field of view is recorded, and the corresponding position in the shortest route direction matrix of the surrounding agents is set to 1, and the rest of the positions are set to 0. The shortest path direction matrix of the surrounding intelligent agents is used to represent the intersection position of the shortest path of the surrounding intelligent agents and the boundary of the visual range; It also includes an action mask matrix, which is a 1×4 vector with values ​​of 0 or 1; the 1×4 vector is used to represent the actions that the agent can perform at the current position; the 1×4 vector is used to input into the policy network.

2. The method for real-time aircraft routing based on multi-agent reinforcement learning according to claim 1, characterized in that: The spatial information of the airport includes spatial information of aircraft, runways, taxiways and parking spaces in the airport.

3. The method for real-time aircraft routing based on multi-agent reinforcement learning according to claim 2, characterized in that: The steps of constructing the directed graph include: setting the vertices of the directed graph and the ranges represented by the vertices according to the spatial information, and inserting virtual nodes between the vertices; using the connection paths between the vertices as the edges of the directed graph; and setting the motion constraints of the aircraft.

4. The method for real-time aircraft routing based on multi-agent reinforcement learning according to claim 1, characterized in that: After all agents complete the execution action at the same time step, the strategy network collects experience and then decides the execution action of all agents at the next time step until all agents reach the end point, and builds an experience pool based on the experience at each time step; the experience pool is used to update the strategy network and the value network; the experience includes local observation information, execution action, reward and value estimation at the current time step.

5. The method for real-time aircraft routing based on multi-agent reinforcement learning according to claim 4, characterized in that: The updating steps of the policy network include: Calculate the GAE advantage at each time step according to the experience pool, and calculate the shear objective function at each time step through the GAE advantage; The policy network parameters are calculated based on the clipping objective function of adjacent time steps to achieve the update of the policy network.

6. The method for real-time aircraft routing based on multi-agent reinforcement learning according to claim 5, characterized in that: The step of updating the value network includes: calculating the value estimate corresponding to the execution action at each time step according to the experience pool; calculating the value network parameters according to the value estimate of adjacent time steps to achieve the update of the value network.

7. The method for real-time aircraft routing based on multi-agent reinforcement learning according to claim 6, characterized in that: The spatial information of the terminal area includes the terminal area entrance and the terminal area fusion point location; when the aircraft performs route planning in the terminal area, a terminal area MAPF environment is constructed according to the spatial information of the terminal area and the aircraft's approach trajectory, and a terminal area matrix vector set is obtained from the terminal area MAPF environment; the terminal area matrix vector set is used as the input of the multi-agent distributed control model to obtain the route planning of the aircraft in the terminal area.

8. A real-time aircraft routing device based on multi-agent reinforcement learning, characterized in that: The invention comprises at least one processor and a memory in communication with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a real-time aircraft routing method based on multi-agent reinforcement learning as described in any one of claims 1 to 7.

Images