Scene aircraft taxiing hybrid control method combining sorting and path planning

Abstract

The invention discloses a scene aircraft taxiing hybrid control method and device combining sorting and path planning, and the method comprises the following steps: 1, analyzing scene geographic information into an undirected graph structure, pre-calculating scene structure information and an initial motion mask set in combination with an undirected graph and a scene activity prohibition specification, and constructing a reinforcement learning environment; 2, real-time position information of the aircraft is obtained, and an input observation value of the scene aircraft taxiing hybrid control method is constructed in combination with scene structure information; 3, constructing a scene aircraft taxiing hybrid control model; the scene aircraft taxiing hybrid control model comprises a path planning module, a conflict detection module and a conflict scheduling module; the path planning module is realized based on a neural network, and the structure of the path planning module is a neural network composed of multiple layers of noise adaptation layer units. And the path planning module is used for taking the observation value obtained in the step 2 as input, and obtaining an action probability vector of the aircraft after neural network processing.

Description

Technical Field

[0001] The present invention relates to the field of intelligent air traffic control technology, and in particular to a hybrid control method for taxiing of aircraft on a scene combining sequencing and path planning. Background Art

[0002] The traditional way of dispatching airport aircraft is for tower controllers to guide aircraft to take off and land safely, manage aircraft taxiing on taxiways, coordinate the intervals between flights and handle emergencies to ensure the safety and smoothness of airport operations. However, with the continuous growth of aviation demand and the long time span of airport facility expansion, the workload intensity of tower controllers has increased, and the dispatching decisions of aircraft by controllers often reduce a certain degree of efficiency.

[0003] In the path planning problem, the research on traditional scheduling algorithms such as Dijkstra and heuristic algorithms is very complete, but often lacks generality, and the research problems are only applicable to specific scenarios. As for the path planning problem of aircraft conflict resolution, modeling is difficult and the application scenarios are highly restricted. Deep reinforcement learning is a common machine learning method that is widely used to deal with decision-making problems under complex conditions. If the observations of existing scene aircraft auxiliary decision-making based on deep reinforcement learning do not cover the overall environment, the scheduling of aircraft is often not the optimal result due to the complexity of the design of scheduling and path planning reward functions; if the observations cover the overall environment, it is difficult to train effective strategies when the action space is large and the action distribution is uneven. Summary of the invention

[0004] The present invention aims at solving the problems in the prior art and provides a hybrid control method for taxiing of surface aircraft combining sequencing and path planning to assist tower controllers in making decisions.

[0005] The technical solution adopted by the present invention is:

[0006] A hybrid control method for taxiing of a surface aircraft combining sequencing and path planning comprises the following steps:

[0007] Step 1: Parse the scene geographic information into an undirected graph structure, and pre-calculate the scene structure information and the initial action mask set by combining the undirected graph and the scene activity prohibition specification to build a reinforcement learning environment;

[0008] Step 2: Obtain the real-time position information of the aircraft and construct the input observation value of the hybrid control method of the aircraft taxiing on the surface in combination with the surface structure information;

[0009] Step 3: Construct a hybrid control model for taxiing of surface aircraft;

[0010] The hybrid control model of surface aircraft taxiing includes path planning module, conflict detection module and conflict scheduling module;

[0011] The path planning module is implemented based on a neural network, and its structure is a neural network composed of multiple layers of noise adaptation layer units. The path planning module is used to take the observations obtained in step 2 as input, obtain the aircraft's action probability vector after processing by the neural network, and pass the action probability vector to the conflict scheduling module.

[0012] The conflict detection module constructs the protection area of ​​the taxiway intersection and the runway as the conflict domain, and calculates the conflict information based on the real-time position of the aircraft to determine whether the aircraft is in conflict. The conflict detection module is used to determine whether the aircraft is in conflict and pass the conflict information to the conflict scheduling module.

[0013] The conflict scheduling module obtains the action mask from the initial action mask set according to the real-time position information of the aircraft, and updates the action mask after sorting the aircraft in conflict. Finally, the next action selection of the aircraft is calculated by the action mask and the action probability vector. The conflict scheduling module is used to integrate the information submitted by the path planning module and the conflict detection module and calculate the next action selection of the aircraft.

[0014] Step 4: Use the curriculum learning training strategy to train the surface aircraft taxiing hybrid control model;

[0015] Step 5: Input the observation values ​​into the hybrid control model of the surface aircraft taxiing to obtain the aircraft's action selection and finally form the aircraft's path planning trajectory.

[0016] Furthermore, the reinforcement learning environment construction in step 1 includes the following processes:

[0017] S11: Based on the taxiway and runway features of the scene environment, the scene is abstracted into a topological graph, and the intersections between taxiways and taxiways, and between taxiways and runways are used as nodes of the topological graph, and taxiways and runways are used as edges of the topological graph. The roads of runways and taxiways are bidirectionally reachable, and the topological graph is an undirected graph structure;

[0018] S12: Divide the environmental action space into 9 actions, namely east, south, west, north, southeast, northeast, southwest, northwest, and stop;

[0019] S13: Generate an initial action mask set based on the scene geographic information. The action mask is represented by a one-hot encoding vector and initialized to [1,1,1,1,1,1,1,1,1], where each position represents the actions of east, south, west, north, southeast, northeast, southwest, northwest, and stop. The 1 in the action mask indicates an optional action, and the 0 indicates an unavailable action. The initial action mask set generation process is as follows:

[0020] The initial action mask set includes the action masks of edges and nodes. The executable actions of edges include following a certain direction of the edge, the reverse direction of the direction, and stopping. Taking the edge to the northeast as an example, the action mask is [0,0,0,0,0,1,1,0,1]. The executable actions of nodes include directions to other nodes and stopping. Taking a node that can reach other nodes to the east and north as an example, the action mask is [1,0,0,1,0,0,0,0,1];

[0021] S14: Calculate the surface structure information based on the surface geographic information, including: runway and taxiway length information; node-to-node relative position of the aircraft parking position and the location information of the two end points of the runway; and take-off or landing time information waiting on the runway.

[0022] Furthermore, the step 2 of inputting observation value construction includes the following process:

[0023] Construct aircraft position observations, including [x, y, x e ,y e ,x b ,y b ,t,v]. The center point of the scene is taken as the origin, the east direction is the positive direction of the x-axis, and the north direction is the positive direction of the y-axis. (x,y) represents the horizontal and vertical coordinates of the aircraft, (x e ,y e ) represents the horizontal and vertical coordinates of the aircraft's end point, (x b ,y b ) represents the horizontal and vertical coordinates of the aircraft target node, t represents the number of steps executed by the aircraft, and v represents the aircraft speed.

[0024] Construct the path planning observation, including the splicing of 8 path vectors, which represent the paths of the aircraft target node in the eight directions of east, south, west, north, southeast, northeast, southwest and northwest. Each path vector is represented as [l,s,n,x d ,y d ], represents the path information adjacent to the aircraft target node. l represents the path length, s represents the shortest distance from the path to the aircraft target node to other aircraft, n represents the number of aircraft on the path, and x d and d Indicates the horizontal and vertical coordinates of another node adjacent to the aircraft target node on the path.

[0025] Construct conflict observations, including [t r ,n r ,r,s r ]. r Indicates the takeoff or landing time that the aircraft is waiting on the runway, n rIt indicates the number of aircraft waiting at the same conflict node as the aircraft, r indicates the radius of the conflict domain, and indicates the shortest distance from other waiting aircraft to the conflict node.

[0026] Finally, the above-mentioned aircraft position observations with a dimension of 8, the path planning observations spliced ​​by 8 path vectors with a dimension of 5, and the conflict observations with a dimension of 4 are concatenated to construct an input observation value with a dimension of 52.

[0027] Furthermore, the path planning module, conflict detection module and conflict scheduling module in step 3 are as follows:

[0028] The path planning module is implemented based on a neural network, including an optimal path value estimation network Q1 and an optimal path auxiliary network Q2. The optimal path value estimation network Q1 outputs aircraft actions, and the optimal path auxiliary network Q2 is used to assist the training of the optimal path value estimation network Q1. The two have the same structure and initial parameters, and the structure is a neural network composed of multiple layers of noise adaptation layer units. The first layer input of the Q1 and Q2 networks is the concatenation of aircraft position observations, path planning observations, and conflict observations. The output of each layer of the subsequent noise adaptation layer units is jump-connected to the aircraft position observations as the input of the next layer of noise adaptation layer units for further feature extraction. The network finally outputs an action probability vector and passes the action probability vector to the conflict scheduling module.

[0029] The conflict detection module constructs the protection area of ​​the taxiway intersection and the runway as the conflict domain, which is a circle with the conflict node as the center. There are two types of aircraft conflicts: aircraft waiting for takeoff or landing on the same runway are judged to be in conflict; different aircraft whose Euclidean distance between the real-time position of the aircraft and the position of the conflict node is less than the radius of the conflict domain are judged to be in conflict. The conflict detection module is used to determine whether the aircraft is in conflict and pass the conflict information to the conflict scheduling module.

[0030] The conflict scheduling module obtains the action mask from the initial action mask set according to the real-time position information of the aircraft. For aircraft in conflict, the conflict scheduling module adopts the shortest time priority strategy, based on the distance to the conflict node, the conflict resolution time, the speed of the aircraft, and the runway waiting time, and takes the shortest total conflict resolution time as the optimization goal to obtain the aircraft queue at each conflict. For the aircraft in the queue, the action mask of the head aircraft remains unchanged, and the action masks of the remaining aircraft are replaced by zero vectors. Finally, the Hadamard product of the action mask and the action probability vector is calculated to obtain the aircraft's next action selection. The conflict scheduling module is used to calculate the aircraft's next action selection based on the action probability vector and conflict information.

[0031] Furthermore, the step 4 utilizes the curriculum to learn the training strategy of the path planning module parameters;

[0032] Due to the large scope of the airport, uneven distribution of movements and constraints of multiple rules, the scheduling of aircraft is very complicated. The course learning solves the problem of scheduling the path planning module in a complex scene environment by gradually increasing the density of aircraft and adding rule constraints. The course learning is divided into the following two courses:

[0033] Course 1: The first stage (1-1) of Course 1 is to train the landing aircraft to reach the parking position while shielding the adjacent taxiway aircraft and not using the conflict detection module and the scheduling module; the second stage (1-2) of Course 1 is to train the take-off aircraft to reach the runway and take off smoothly while shielding the adjacent taxiway aircraft and not using the conflict detection module and the scheduling module; the third stage (1-3) of Course 1 is to add aircraft safety distance penalties while shielding the conflict detection module and the scheduling module, to train the take-off and landing aircraft to reach the destination safely, and to gradually increase the aircraft density until the upper density limit.

[0034] Course 2: Add conflict detection module, conflict scheduling module, add runway waiting information, intersection conflict domain and other rules, increase the penalty for aircraft approaching the runway when there are other aircraft about to land or take off, train the aircraft in Course 1 to arrive at the destination safely under the constraints of the rules, and gradually increase the aircraft density until the upper limit of the density.

[0035] Furthermore, the training methods at each stage of claim 4 are as follows:

[0036] Step 3: The path planning module is trained using the D3QN reinforcement learning method. The training process is as follows:

[0037] Construct the optimal path value estimation network Q1 and the optimal path auxiliary network Q2; Q1 and Q2 have the same network structure. In each training step, Q1 outputs the state-action value function of the current step, and Q2 outputs the state-action value function of the next step.

[0038] Decision actions include: East, South, West, North, Southeast, Northeast, Southwest, Northwest, Stop, 9 optional actions;

[0039] Phases 1-1 and 1-2 give rewards for the distance from the current aircraft to the destination and the current number of steps executed; Phase 1-3 gives rewards for the distance from the current aircraft to the destination, the current number of steps executed and the relative distance to other aircraft; Course 2 training gives rewards for the distance from the current aircraft to the destination, the current number of steps executed and the rules defined in the surface activity prohibition regulations;

[0040] The loss function is calculated based on the state-action-value function of the current step and the next step and the given reward, and the network Q1 gradient is back-propagated to update the Q1 parameters.

[0041] Q2 copies the parameters of Q1 every other cycle to increase the stability of training.

[0042] Train networks Q1 and Q2 until the networks converge.

[0043] Furthermore, the neural network of the path planning module in step 3 improves the action value function calculation under mask conditions;

[0044] The conflict scheduling module obtains the action mask from the initial action mask set according to the real-time position information of the aircraft. For aircraft in conflict, the conflict scheduling module adopts the shortest time priority strategy. Based on the distance to the conflict node, the conflict resolution time, the speed of the aircraft, and the runway waiting time, the aircraft queue at each conflict is obtained with the shortest total conflict resolution time as the optimization goal. For the aircraft in the queue, the action mask of the head aircraft remains unchanged, and the action masks of the remaining aircraft are replaced by zero vectors. Finally, the Hadamard product of the action mask and the action probability vector is calculated to obtain the next action selection of the aircraft. The conflict scheduling module is used to calculate the next action selection of the aircraft based on the action probability vector and conflict information. The calculation method is as follows:

[0045] A(s,a;θ)=Q π (s,a;θ)em

[0046]

[0047] Among them, Q(s,a;θ) is the current state action value function, m represents the action mask, A(s,a;θ) is the Hadamard product of the two, and the argmax function returns the index value of the maximum value in a vector.

[0048] Furthermore, the reward functions at each stage of claim 4 are as follows:

[0049]

[0050] In the formula: reward step represents the aircraft step penalty, reward away Indicates that the aircraft is far from the destination penalty, reward completion Indicates the reward for the aircraft reaching the destination. distance Indicates the aircraft distance to the destination penalty, reward rule Indicates that the scene activities prohibit the regulation of punishment, reward air represents the relative distance penalty with other aircraft, t is the current execution step number, s and sl are the Euclidean distances of the aircraft from the previous step and the current step to the end point, respectively, d ijis the sum of the Euclidean distances from aircraft i to the nearest aircraft j on the adjacent taxiway to the target node. The surface activity prohibition specification penalties include other aircraft about to land entering the runway in the approach direction of the runway, other aircraft taxiing on the runway entering the runway, and not maintaining a safe distance from other aircraft, etc. ω1, ω2, ω3, ω4, ω5, ω6 are all weight coefficients, b is a constant, and min is a function to find the smaller value of the two.

[0051] In Course 1, the rewards for Phases 1-1 and 1-2 are step 、reward away 、reward completion and reward distance The rewards for phases 1-3 are step 、reward away 、reward completion 、reward distance and reward air In Lesson 2, the reward is step 、reward away 、reward completion 、reward distance 、reward air and reward rule .

[0052] A decision-making device for a hybrid control method for taxiing of a surface aircraft combining sequencing and path planning, comprising:

[0053] Information collection module: obtain scene structure information and initial action mask set based on scene geographic information;

[0054] Information processing module: pre-processing of surface structure information and aircraft position information;

[0055] Decision-making module: Make scene scheduling decisions based on the processed information.

[0056] The beneficial effects of the present invention are:

[0057] (1) The present invention constructs a hybrid control model for taxiing of field aircraft, including a path planning module, a conflict detection module, and a conflict scheduling module. The joint work of these three modules optimizes path planning and conflict management. The path planning module dynamically optimizes the route through reinforcement learning, so that the aircraft can select the best path according to the real-time situation. The conflict detection module accurately identifies potential conflicts to ensure timely response of the system. The conflict scheduling module optimizes instructions based on conflict information to minimize conflict resolution time and improve flight safety and efficiency. This integrated solution improves the safety, efficiency and decision-making accuracy of the aviation system as a whole.

[0058] (2) The present invention trains the hybrid control model of surface aircraft taxiing through a curriculum learning training strategy, gradually introducing complex rules and constraints, so that the curriculum learning method can effectively train the model to gradually improve the path planning ability in a complex aircraft scheduling environment, and enhance its comprehensive adaptability and reliability in conflict detection, scheduling optimization and safety distance management in actual operations.

[0059] (3) The present invention adds the calculation of the state value function of the action mask based on the scene environment reinforcement learning framework. By eliminating invalid actions in the mask when calculating the state value function action advantage function, the accuracy of path planning under mask conditions is improved, so that the model can effectively eliminate the interference of invalid actions on the calculation when faced with invalid actions, thereby optimizing the estimation of the state action value function. This improvement enables the model to more accurately evaluate the value of each action when dealing with complex scheduling and conflict management, thereby achieving more efficient and safe aircraft taxiing control. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the process of the present invention.

[0061] Figure 2 It is a schematic diagram of the structural flow of the path planning module in the present invention.

[0062] Figure 3 It is a schematic diagram of the structural flow of the conflict detection module in the present invention.

[0063] Figure 4 It is a schematic diagram of the conflict scheduling model flow in the present invention.

[0064] Figure 5 Schematic diagram of the model training strategy in the present invention.

[0065] Figure 6 It is a structural schematic diagram of the decision-making device in the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0067] like Figure 1 As shown, a hybrid control method for taxiing of a surface aircraft combining sequencing and path planning comprises the following steps:

[0068] Step 1: Parse the scene geographic information into an undirected graph structure, build a reinforcement learning environment based on the undirected graph and scene activity prohibition specifications, and pre-calculate the scene structure information and the initial action mask set;

[0069] The process of constructing the reinforcement learning environment and pre-calculating the scene structure information and the initial action mask set is as follows:

[0070] S11: Based on the taxiway and runway features of the scene environment, the scene is abstracted into a topological graph, and the intersections between taxiways and taxiways, and between taxiways and runways are used as nodes of the topological graph, and taxiways and runways are used as edges of the topological graph. The roads of runways and taxiways are bidirectionally reachable, and the topological graph is an undirected graph structure;

[0071] S12: Divide the environmental action space into 9 actions, namely east, south, west, north, southeast, northeast, southwest, northwest, and stop;

[0072] S13: Generate an initial action mask set based on the scene geographic information. The action mask is represented by a one-hot encoding vector and initialized to [1,1,1,1,1,1,1,1,1], where each position represents the actions of east, south, west, north, southeast, northeast, southwest, northwest, and stop. The 1 in the action mask indicates an optional action, and the 0 indicates an unavailable action. The initial action mask set generation process is as follows:

[0073] The initial action mask set includes the action masks of edges and nodes. The executable actions of edges include following a certain direction of the edge, the reverse direction of the direction, and stopping. Taking the edge to the northeast as an example, the action mask is [0,0,0,0,0,1,1,0,1]. The executable actions of nodes include directions to other nodes and stopping. Taking a node that can reach other nodes to the east and north as an example, the action mask is [1,0,0,1,0,0,0,0,1];

[0074] S14: Calculate the surface structure information based on the surface geographic information, including: runway and taxiway length information; node-to-node relative position of the aircraft parking position and the location information of the two end points of the runway; and take-off or landing time information waiting on the runway.

[0075] Step 2: Obtain the real-time position information of the aircraft and construct the input observation value of the hybrid control method of the aircraft taxiing on the surface in combination with the surface structure information;

[0076] The process of constructing the input observation value of the hybrid control method for surface aircraft taxiing is as follows:

[0077] Construct aircraft position observations, including [x, y, x e ,y e ,x b ,y b ,t,v]. The center point of the scene is taken as the origin, the east direction is the positive direction of the x-axis, and the north direction is the positive direction of the y-axis. (x,y) represents the horizontal and vertical coordinates of the aircraft, (x e ,y e ) represents the horizontal and vertical coordinates of the aircraft's end point, (x b ,y b) represents the horizontal and vertical coordinates of the aircraft target node, t represents the number of steps executed by the aircraft, and v represents the aircraft speed.

[0078] Construct the path planning observation, including the splicing of 8 path vectors, which represent the paths of the aircraft target node in the eight directions of east, south, west, north, southeast, northeast, southwest and northwest. Each path vector is represented as [l,s,n,x d ,y d ], represents the path information adjacent to the aircraft target node. l represents the path length, s represents the shortest distance from the path to the aircraft target node to other aircraft, n represents the number of aircraft on the path, and x d and d Indicates the horizontal and vertical coordinates of another node adjacent to the aircraft target node on the path.

[0079] Construct conflict observations, including [t r ,n r ,r,s r ]. r Indicates the takeoff or landing time that the aircraft is waiting on the runway, n r It indicates the number of aircraft waiting at the same conflict node as the aircraft, r indicates the radius of the conflict domain, and indicates the shortest distance from other waiting aircraft to the conflict node.

[0080] Finally, the above-mentioned aircraft position observations with a dimension of 8, the path planning observations spliced ​​by 8 path vectors with a dimension of 5, and the conflict observations with a dimension of 4 are concatenated to construct an input observation value with a dimension of 52.

[0081] Step 3: Construct a hybrid control model for taxiing of surface aircraft;

[0082] The intelligent path planning model for taxiing of surface aircraft includes path planning module, conflict detection module and conflict scheduling module;

[0083] The path planning module is implemented based on neural networks. First, the optimal path value estimation network Q1 and the optimal path auxiliary network Q2 are constructed. The structures and initial parameters of the two networks are the same. The structures are both neural networks composed of multiple layers of noise adaptation layer units, such as Figure 2 As shown in the figure. The first layer of the network input is the concatenation of aircraft position observations, path planning observations, and conflict observations. The output of each layer of the subsequent noise adaptation layer units is jump-connected to the aircraft position observations and serves as the input of the next layer of noise adaptation layer units for further feature extraction. The network Q1 and Q2 of the path planning module finally output two branches, the aircraft current state value function V π (s), the action value advantage function A corresponding to the current state of the aircraft π (s,a). Calculate the advantage function A of the neural network output π(s, a) and the Hadamard product of the conflict scheduling module submission mask, and the effective advantage function vector is obtained by subtracting the result vector from the mean of the result vector The effective advantage function vector Then compare it with the value function V of the current state π (s) and the state-action value function Q(s, a; θ) are obtained by summing them up. The calculation method is as follows:

[0084] V π (s) = E a: π (s) [Q π (s,a)]

[0085]

[0086] In the formula, A π (s,a) represents the advantage value function, NEA represents the number of effective actions, represents the advantage value function after masking, θ represents the network parameters, Q(s,a;θ) is the current state action value function, and m represents the action mask. Mask the action set and estimate the action value advantage function A based on the mask π (s, a), the neural network output eliminates the influence of invalid actions on the calculation of the action value function Q(s, a; θ), making the current state action value function Q(s, a; θ) more accurately estimated. The network finally calculates the action probability vector and passes it to the conflict scheduling module.

[0087] The conflict detection module judgment process is as follows Figure 3 As shown in the figure, the protection area of ​​the taxiway intersection and the runway is constructed as the conflict domain, and the conflict domain is a circle with the conflict node as the center. There are two types of aircraft conflicts: aircraft waiting for takeoff or landing on the same runway are judged to be in conflict; different aircraft whose Euclidean distance between the real-time position of the aircraft and the position of the conflict node is less than the radius of the conflict domain are judged to be in conflict. The conflict detection module is used to determine whether the aircraft is in conflict and pass the conflict information to the conflict scheduling module.

[0088] like Figure 4As shown, the conflict scheduling module obtains the action mask from the initial action mask set according to the real-time position information of the aircraft. For aircraft in conflict, the conflict scheduling module adopts the shortest time priority strategy, based on the distance to the conflict node, the conflict resolution time, the speed of the aircraft, and the runway waiting time. The shortest total conflict resolution time is used as the optimization goal to obtain the aircraft queue for each conflict. For the aircraft in the queue, the action mask of the head aircraft remains unchanged, and the action masks of the remaining aircraft are replaced by a zero vector. Finally, the Hadamard product of the action mask and the action probability vector is calculated to obtain the next instruction of the aircraft. The conflict scheduling module is used to calculate the next instruction of the aircraft based on the action probability vector and the conflict information. The calculation method is as follows:

[0089] A(s,a;θ)=Q π (s,a;θ)em

[0090]

[0091] Among them, Q(s,a;θ) is the current state action value function, m represents the action mask, A(s,a;θ) is the Hadamard product of the two, and the argmax function returns the index value of the maximum value in a vector.

[0092] Step 4: Use the curriculum learning training strategy to train the surface aircraft taxiing hybrid control model;

[0093] like Figure 5 As shown, the course training process is as follows:

[0094] Due to the large scope of the airport, uneven distribution of movements and constraints of multiple rules, the scheduling of aircraft is very complicated. The course learning solves the problem of scheduling the path planning module in a complex scene environment by gradually increasing the density of aircraft and adding rule constraints. The course learning is divided into the following two courses:

[0095] Course 1: The first stage (1-1) of Course 1 is to train the landing aircraft to arrive at the parking position without using the conflict detection module and the scheduling module, while shielding the adjacent taxiway aircraft. The second stage (1-2) of Course 1 is to train the take-off aircraft to arrive at the runway and take off smoothly without using the conflict detection module and the scheduling module. The third stage (1-3) of Course 1 is to add the aircraft safety distance penalty while shielding the conflict detection module and the scheduling module, and train the take-off and landing aircraft to arrive at the destination safely, while gradually increasing the aircraft density until the upper density limit.

[0096] Course 2: Add conflict detection module, conflict scheduling module, add runway waiting information, intersection conflict domain and other rules, increase the penalty for aircraft approaching the runway when there are other aircraft about to land or take off, train the aircraft in Course 1 to arrive at the destination safely under the constraints of the rules, and gradually increase the aircraft density until the upper limit of the density.

[0097] The specific process is as follows:

[0098] Construct the optimal path value estimation network Q1 and the optimal path auxiliary network Q2; Q1 and Q2 have the same network structure. In each training step, Q2 outputs the action and Q1 outputs the state value function value of the selected action;

[0099] In different courses and at different training stages, the reward function is adjusted according to the training objectives.

[0100]

[0101] In the formula: reward step represents the aircraft step penalty, reward away Indicates that the aircraft is far from the destination penalty, reward completion Indicates the reward for the aircraft reaching the destination. distance Indicates the aircraft distance to the destination penalty, reward rule Indicates that the scene activities prohibit the regulation of punishment, reward air represents the relative distance penalty with other aircraft, t is the current execution step number, s and sl are the Euclidean distances of the aircraft from the previous step and the current step to the end point, respectively, d ij is the sum of the Euclidean distances from aircraft i to the nearest aircraft j on the adjacent taxiway to the target node. The surface activity prohibition specification penalties include other aircraft about to land entering the runway in the approach direction of the runway, other aircraft taxiing on the runway entering the runway, and not maintaining a safe distance from other aircraft, etc. ω1, ω2, ω3, ω4, ω5, ω6 are all weight coefficients, b is a constant, and min is a function to find the smaller value of the two.

[0102] In Course 1, the rewards for Phases 1-1 and 1-2 are step 、reward away 、reward completion and reward distance The rewards for phases 1-3 are step 、reward away 、reward completion 、reward distance and reward air In Lesson 2, the reward is step、reward away 、reward completion 、reward distance 、reward air and reward rule .

[0103] Decision actions include: East, South, West, North, Southeast, Northeast, Southwest, Northwest, Stop, 9 optional actions;

[0104] At each stage, the training process of networks Q1 and Q2 is as follows:

[0105] Assume that the network parameters of Q1 are φ and Q2 are θ. The initial network parameters of Q1 and Q2 are consistent. The action value function Q(s,a,θ) of D3QN is expressed as:

[0106]

[0107] The Q2 network optimization objective is expressed as:

[0108]

[0109] y=r+γQ2(s',a max ;θ)

[0110] In the formula, r represents the reward for executing the current action, a max represents the optimal action in state s, γ is the discount factor, and s' represents the next state of state s.

[0111] When the D3QN network performs the i-th iteration, it randomly and uniformly samples from the experience pool U(D) and updates the following loss function:

[0112]

[0113] In the formula, Q1(s,a max ; φ) is the maximum value of the action value function of the current state of the Q1 network, y2 is the optimization target of the Q2 network; s t and t+1 are the characteristics of the aircraft at the current time t and t+1, the characteristics of the taxiway adjacent to the target node, the characteristics of the aircraft adjacent to the taxiway of the target node, and the fusion representation of the conflicting characteristics; r is the reward obtained at time t, and a represents the decision action made. After calculating the loss function, the gradient descent method is used to minimize the loss function and update the network parameters so that the advantage value function A of the neural network mask (s,a) continuously approaches the optimal advantage function State value function Continuously approaching the optimal state value function

[0114] The loss function is calculated based on the state-action-value function of the current step and the next step and the given reward, and the network Q1 gradient is back-propagated to update the Q1 parameters.

[0115] Q1 copies the parameters of Q2 every other cycle to increase the stability of training.

[0116] Train networks Q1 and Q2 until the networks converge.

[0117] The decision-making method of the present invention builds a topological reinforcement learning environment based on scene geographic information and uses a mask-constrained action set to calculate position observations, path planning observations and conflict observations, and splices the observations as input. Using the joint work of multiple modules such as the path planning module, the conflict detection module, and the conflict scheduling module to process the input, this integrated solution optimizes path planning and conflict management as a whole. The scene aircraft auxiliary decision observations based on deep reinforcement learning only cover part of the environment, making training fast and effective; the aircraft at the conflicting nodes are sorted based on the shortest time priority, making scheduling efficient and safe. The present invention can effectively assist tower controllers in scheduling aircraft at the scene and relieve scheduling pressure.

Claims

1. A hybrid control method for taxiing of surface aircraft combining sequencing and path planning, characterized in that: Includes the following step: Step 1: Parse the scene geographic information into an undirected graph structure, and pre-calculate the scene structure information and the initial action mask set by combining the undirected graph and the scene activity prohibition specification to build a reinforcement learning environment; Step 2: Obtain the real-time position information of the aircraft and construct the input observation value of the hybrid control method of the aircraft taxiing on the surface in combination with the surface structure information; Step 3: Construct a hybrid control model for taxiing of surface aircraft; The hybrid control model of surface aircraft taxiing includes path planning module, conflict detection module and conflict scheduling module; The path planning module is implemented based on a neural network, and its structure is a neural network composed of multiple layers of noise adaptation layer units. The path planning module is used to take the observation value obtained in step 2 as input, obtain the aircraft's action probability vector after processing by the neural network, and pass the action probability vector to the conflict scheduling module. The conflict detection module constructs the protection area of ​​the taxiway intersection and the runway as the conflict domain, and calculates the conflict information based on the real-time position of the aircraft to determine whether the aircraft is in conflict; the conflict detection module is used to determine whether the aircraft is in conflict and pass the conflict information to the conflict scheduling module; The conflict scheduling module obtains the action mask from the initial action mask set according to the real-time position information of the aircraft, and updates the action mask after sorting the aircraft in conflict. Finally, the next action selection of the aircraft is calculated by the action mask and the action probability vector. The conflict scheduling module is used to integrate the information submitted by the path planning module and the conflict detection module and calculate the next action selection of the aircraft. Step 4: Use the curriculum learning training strategy to train the surface aircraft taxiing hybrid control model; Step 5: Input the observation values ​​into the hybrid control model of the surface aircraft taxiing to obtain the aircraft's action selection and finally form the aircraft's path planning trajectory.

2. A hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 1, characterized in that: The reinforcement learning environment construction in step 1 includes the following processes: S11: Based on the taxiway and runway features of the scene environment, the scene is abstracted into a topological graph, and the intersections between taxiways and taxiways, and between taxiways and runways are used as nodes of the topological graph, and taxiways and runways are used as edges of the topological graph; the roads of runways and taxiways are bidirectionally reachable, and the topological graph is an undirected graph structure; S12: Divide the environmental action space into 9 actions, namely east, south, west, north, southeast, northeast, southwest, northwest, and stop; S13: Generate an initial action mask set according to the scene geographic information. The action mask is represented by a one-hot encoding vector and initialized to [1,1,1,1,1,1,1,1,1], where each position represents east, south, west, north, southeast, northeast, southwest, northwest, and stop actions. The 1 in the action mask indicates that an action can be selected, and the 0 indicates that an action cannot be selected. The initial action mask set generation process is as follows: The initial action mask set includes action masks of edges and nodes. The executable actions of edges include moving in a certain direction along the edge, the reverse direction of the direction, and stopping. Taking the edge to the northeast as an example, the action mask is [0,0,0,0,0,1,1,0,1]. The executable actions of nodes include directions to other nodes and stopping. Taking a node that can reach other nodes to the east and north as an example, the action mask is [1,0,0,1,0,0,0,0,1]. S14: Calculate the surface structure information based on the surface geographic information, including: runway and taxiway length information; node-to-node relative position of the aircraft parking position and the location information of the two end points of the runway; and take-off or landing time information waiting on the runway.

3. The hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 1, characterized in that: The step 2 input observation value construction includes the following process: Construct aircraft position observations, including [x, y, x e ,y e ,x b ,y b ,t,v]; the center point of the scene is taken as the origin, the east direction is the positive direction of the x-axis, and the north direction is the positive direction of the y-axis; (x,y) represents the horizontal and vertical coordinates of the aircraft, (x e ,y e ) represents the horizontal and vertical coordinates of the aircraft's end point, (x b ,y b ) represents the horizontal and vertical coordinates of the aircraft target node, t represents the number of steps executed by the aircraft, and v represents the aircraft speed; Construct path planning observations, including the concatenation of 8 path vectors, which represent the paths of the aircraft target node in the eight directions of east, south, west, north, southeast, northeast, southwest and northwest; each path vector is represented as [l,s,n,x d ,y d ], represents the path information adjacent to the aircraft target node; l represents the path length, s represents the shortest distance from the path to the aircraft target node to other aircraft, n represents the number of aircraft on the path, x d and d Indicates the horizontal and vertical coordinates of another node adjacent to the aircraft target node on the path; Construct conflict observations, including [t r ,n r ,r,s r ]; t r Indicates the takeoff or landing time that the aircraft is waiting on the runway, n r It indicates the number of aircraft waiting at the same conflict node as the aircraft, r indicates the radius of the conflict domain, and indicates the shortest distance from other waiting aircraft to the conflict node; Finally, the aircraft position observations with a dimension of 8, the path planning observations with a dimension of 8 path vectors with a dimension of 5, and the conflict observations with a dimension of 4 are concatenated to construct an input observation value with a dimension of 52.

4. The hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 1, characterized in that: The path planning module, conflict detection module and conflict scheduling module in step 3 are as follows: The path planning module is implemented based on a neural network, including an optimal path value estimation network Q1 and an optimal path auxiliary optimal path auxiliary network Q2; the optimal path value estimation network Q1 outputs aircraft actions, and the optimal path auxiliary network Q2 is used to assist the optimal path value estimation network Q1 in training; the two have the same structure and initial parameters, and the structure is a neural network composed of multiple layers of noise adaptation layer units; the first layer input of the Q1 and Q2 networks is the concatenation of aircraft position observations, path planning observations, and conflict observations, and the output of each layer of the subsequent noise adaptation layer units is jump-connected with the aircraft position observations as the input of the next layer of noise adaptation layer units for further feature extraction; the network finally outputs an action probability vector and passes the action probability vector to the conflict scheduling module; The conflict detection module constructs the protection area of ​​the taxiway intersection and the runway as the conflict domain, which is a circle with the conflict node as the center. There are two types of aircraft conflicts: aircraft waiting for takeoff or landing on the same runway are judged to be in conflict; different aircraft whose Euclidean distance between the real-time position of the aircraft and the position of the conflict node is less than the radius of the conflict domain are judged to be in conflict; the conflict detection module is used to determine whether the aircraft is in conflict and pass the conflict information to the conflict scheduling module; The conflict scheduling module obtains the action mask from the initial action mask set according to the real-time position information of the aircraft; For aircraft in conflict, the conflict scheduling module adopts the shortest time priority strategy, based on the distance to the conflict node, conflict resolution time, aircraft speed, and runway waiting time, and takes the shortest total conflict resolution time as the optimization goal to obtain the aircraft queue at each conflict point; for the aircraft in the queue, the action mask of the head aircraft remains unchanged, and the action masks of the remaining aircraft are replaced by zero vectors; finally, the Hadamard product of the action mask and the action probability vector is calculated to obtain the next action selection of the aircraft; the conflict scheduling module is used to calculate the next action selection of the aircraft based on the action probability vector and conflict information; the calculation method is as follows: A(s,a;θ)=Q π (s,a;θ)em Among them, Q(s,a;θ) is the current state action value function, m represents the action mask, A(s,a;θ) is the Hadamard product of the two, and the argmax function returns the index value of the maximum value in a vector.

5. The hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 1, characterized in that: The training strategy of the path planning module parameters using curriculum learning in step 4 is: The course solves the problem of path planning module scheduling in complex scene environments by gradually increasing the density of aircraft and adding rule constraints. The course is divided into the following two courses: Course 1: The first stage (1-1) of Course 1 is to train the landing aircraft to arrive at the parking position without using the conflict detection module and the scheduling module, while shielding the adjacent taxiway aircraft. The second stage (1-2) of Course 1 is to train the take-off aircraft to arrive at the runway and take off smoothly without using the conflict detection module and the scheduling module. The third stage (1-3) of Course 1 is to add the aircraft safety distance penalty while shielding the conflict detection module and the scheduling module, and train the take-off and landing aircraft to arrive at the destination safely, while gradually increasing the aircraft density until the upper density limit. Course 2: Add conflict detection module, conflict scheduling module, add runway waiting information, intersection conflict domain and other rules, increase the penalty for aircraft approaching the runway when there are other aircraft about to land or take off, train the aircraft in Course 1 to arrive at the destination safely under the constraints of the rules, and gradually increase the aircraft density until the upper limit of the density.

6. The hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 1, characterized in that: The training methods at each stage of claim 4 are as follows: Step 3: The path planning module is trained using the D3QN reinforcement learning method. The training process is as follows: Construct the optimal path value estimation network Q1 and the optimal path auxiliary network Q2; Q1 and Q2 have the same network structure. In each training step, Q1 outputs the state-action value function of the current step, and Q2 outputs the state-action value function of the next step; Decision actions include: East, South, West, North, Southeast, Northeast, Southwest, Northwest, Stop, 9 optional actions; Phases 1-1 and 1-2 combine the distance from the current aircraft to the destination and the current number of steps as rewards; Phase 1-3 combines the distance from the current aircraft to the destination, the current number of steps and the relative distance to other aircraft as rewards; Course 2 training combines the distance from the current aircraft to the destination, the current number of steps and the rules defined in the surface activity prohibition regulations as rewards; Calculate the loss function based on the state-action value function of the current step and the next step and the given reward, and backpropagate the network Q1 gradient to update the Q1 parameters; Q2 copies the parameters of Q1 every other cycle to increase the stability of training; Train networks Q1 and Q2 until the networks converge.

7. The hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 1, characterized in that: The neural network of the path planning module in step 3 improves the action value function calculation under mask conditions; The network Q1 and Q2 of the path planning module finally output two branches: the value function V of the current state of the aircraft π (s), the action value advantage function A corresponding to the current state of the aircraft π (s,a); calculate the advantage function A of the neural network output π (s, a) and the Hadamard product of the conflict scheduling module submission mask, and the effective advantage function vector is obtained by subtracting the result vector from the mean of the result vector The effective advantage function vector Then compare it with the value function V of the current state π (s) and then we get the state-action value function Q(s,a;θ), which is calculated as follows: V π (s)=E a:π(s) [Q π (s,a)] In the formula, A π (s,a) represents the advantage value function, NEA represents the number of effective actions, represents the advantage value function after masking, θ represents the network parameters, Q(s, a; θ) is the current state action value function, and m represents the action mask; mask the action space and estimate the action value advantage function A based on the mask π (s, a), the neural network output eliminates the influence of invalid actions on the calculation of the action value function Q(s, a; θ), so that the current state action value function Q(s, a; θ) is estimated more accurately.

8. The hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 5, characterized in that: The reward functions at each stage of claim 4 are as follows: In the formula: reward step represents the aircraft step penalty, reward away Indicates that the aircraft is far from the destination penalty, reward completion Indicates the reward for the aircraft reaching the destination. distance Indicates the aircraft distance to the destination penalty, reward rule Indicates that the scene activities prohibit the regulation of punishment, reward air represents the relative distance penalty with other aircraft, t is the current execution step number, s and sl are the Euclidean distances of the aircraft from the previous step and the current step to the end point, respectively, d ij is the sum of the Euclidean distances from aircraft i to the nearest aircraft j on the adjacent taxiway to the target node; the surface activity prohibition standard penalties include other aircraft about to land on the runway approach direction, other aircraft taxiing on the runway, and not keeping a safe distance from other aircraft, etc. ω1, ω2, ω3, ω4, ω5, ω6 are all weight coefficients, b is a constant, and min is a function to find the smaller value of the two; in course 1, the rewards for stages 1-1 and 1-2 are reward step 、reward away 、reward completion and reward distance ; The reward for stage 1-3 is reward step 、reward away 、reward completion 、reward distance and reward air ; In Course 2, the reward is reward step 、reward away 、reward completion 、reward distance 、reward air and reward rule .

9. A hybrid control method for taxiing of surface aircraft combining sequencing and path planning according to claim 5, characterized in that: The process of training networks Q1 and Q2 is as follows: Assume that the network parameters of Q1 are φ, the network parameters of Q2 are θ, and the initial network parameters of Q1 and Q2 are consistent; the action value function Q(s, a, θ) of DDQN is expressed as: The Q2 network optimization objective is expressed as: y=r+γQ2(s',a max ;θ) In the formula, r represents the reward for executing the current action, a max represents the optimal action in state s, γ is the discount factor, and s' represents the next state of state s; When the D3QN network performs the i-th iteration, it randomly and uniformly samples from the experience pool U(D) and updates the following loss function: In the formula, Q1(s,a max ; φ) is the maximum value of the action value function of the current state of the Q1 network, and y2 is the optimization target of the Q2 network; s t and t+1 They are the characteristics of the aircraft at the current time t and t+1, the characteristics of the taxiway adjacent to the target node, the characteristics of the aircraft adjacent to the taxiway of the target node, and the conflict characteristics. Fusion representation; r is the reward obtained at time t, a represents the decision action made; after calculating the loss function, the gradient descent method is used to minimize the loss function and update the network parameters, so that the advantage value function A of the neural network mask (s,a) continuously approaches the optimal advantage function State Value Function Continuously approaching the optimal state value function 10. A decision-making device using a hybrid control method for taxiing of surface aircraft combining sequencing and path planning as claimed in claim 1, characterized in that: include: Information collection module: obtain scene structure information and initial action mask set based on scene geographic information; Information processing module: pre-processing of surface structure information and aircraft position information; Decision-making module: Make scene scheduling decisions based on the processed information.

Images