Intelligent Generation Method and System of Control Instructions Based on Situation Representation and Behavior Imitation

Abstract

The present invention discloses an intelligent generation method and system for control instructions based on situation representation and behavior imitation, including: constructing a state space of aircraft representation vectors; constructing a data set for self-perception representation learning of comprehensive situations; constructing a self-perception representation learning network for comprehensive situations; constructing a backbone network of a control instruction generation network; training the control instruction generation network using a two-stage training strategy; cascading the backbone network of self-perception representation learning of comprehensive situations and the backbone network of control instruction generation, inputting civil aviation automation system data, sampling according to the instruction action probability output by the control instruction generation network, and outputting control instructions. By performing imitation learning on the replay data of control behaviors under the comprehensive situation of real sectors, the present invention makes full use of the existing excellent experience and strategies of humans to guide the training of the instruction generation network, making up for the limitations of artificially defined reward functions in the simulated environment.

Description

Technical Field

[0001] The present invention belongs to the fields of air traffic management and artificial intelligence, and particularly relates to a method and system for intelligently generating control instructions based on situation representation and behavior imitation. Background Art

[0002] In the current air traffic management system, aircraft flights are carried out on specified fixed air routes, and the task of maintaining the aircraft separation is completed by air traffic controllers. In case of special situations such as conflicts, no-fly zones, danger zones, and extremely bad weather, if it is necessary to change the flight track, it must be carried out strictly in accordance with the instructions of air traffic controllers.

[0003] However, the increasing demand for air transportation will lead to a continuous increase in air traffic flow, resulting in more crowded flight routes, increasing the risk of aircraft conflicts, threatening aviation flight safety, and posing higher requirements for the level of air traffic controllers. In fact, air traffic controllers will have difficulty coping with the sharp increase in air traffic flow in the next decade or even longer. Therefore, industry experts have called for the development of intelligent assistants to support pilots and controllers in making real-time decisions and realizing an intelligent mode with aircraft as the main body and controllers for monitoring.

[0004] In an air traffic management system with a high level of automation, a key challenge is to design an intelligent control instruction generation system to provide real-time advice for aircraft to ensure safe separation during flight and at intersections. This is of great significance for reducing the workload of controllers, improving the safety factor of air traffic control, and reducing the operating cost of air traffic control services, and is an important development direction in the future research of the world's air traffic management field.

[0005] With the development of deep learning and artificial intelligence technologies, the research on ATC (AIR TRAFFIC CONTROL) intelligent autonomy has begun to tend to adopt the method of deep reinforcement learning. Existing research has explored a lot on the reinforcement learning modeling of specific problems in the field of air traffic control decision-making, including the design of the intelligent agent environment state, the action space, the system reward function, etc. However, the control autonomy based on artificial intelligence technology is still in the stage of simulation experiments in simple scenarios and lacks research in actual application scenarios. The specific problems can be summarized as follows:

[0006] 1. The problem is overly simplified. For example, the action space only controls the speed of the aircraft.

[0007] 2. The interaction environment of reinforcement learning uses a simple simulation environment, which is quite different from the real automation system.

[0008] 3. The excellent experience strategies accumulated by humans are not utilized.

[0009] 4. The resolution of conflicts is relatively sluggish, resulting in relatively abrupt actions for the aircraft to avoid conflicts and lacking smoothness. Summary of the Invention

[0010] Aiming at the deficiencies of the above-mentioned existing technologies, the purpose of the present invention is to provide an intelligent generation method and system for control instructions based on situation representation and behavior imitation, so as to solve the defect problems brought by the control autonomy based on artificial intelligence technology in the existing technologies.

[0011] The present invention realizes the intelligent generation of control instructions through self-perception representation learning of the comprehensive situation and imitation learning of the command behaviors of air traffic controllers, thereby realizing the autonomous management of uncertain conflicts in the control sector by the ATC system.

[0012] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0013] An intelligent generation method for control instructions based on situation representation and behavior imitation of the present invention comprises the following steps:

[0014] 1) Construct the state space of the aircraft representation vector;

[0015] 2) Construct the data set for self-perception representation learning of the comprehensive situation;

[0016] 3) Construct the self-perception representation learning network for the comprehensive situation;

[0017] 4) Construct the backbone network of the control instruction generation network;

[0018] 5) Adopt a two-stage training strategy to train the control instruction generation network;

[0019] 6) Cascade the backbone network of the self-perception representation learning of the comprehensive situation and the backbone network of the control instruction generation, input the data of the civil aviation automation system, and sample according to the instruction action probability output by the control instruction generation network to output the control instruction.

[0020] Further, the step 1) specifically includes: extracting the strongly correlated items with the control decision-making behaviors (such as meteorological index x1, wake vortex level x2, longitude x3, latitude x4, altitude x5, speed x6, heading x7, aircraft type x8, target airport x9, distance to the next waypoint x10, etc.) in the comprehensive situation according to the civil aviation control rules. After the information is numerically processed, the representation vector of the i-th aircraft is obtained , as follows:

[0021]

[0022] Further, the comprehensive situation includes: the flight plan corresponding to the flight, the comprehensive track feedback after comprehensive processing by the automation system, and the meteorological situation of the waypoints.

[0023] Further, step 2) specifically includes: The dataset for comprehensive situation self-awareness representation learning consists of the simulated scenario scripts of air traffic control training extracted from a civil aviation control simulator, and the annotation information of potential conflict flights and the route point IDs where conflicts occur in the simulated scenario scripts.

[0024] Further, the annotation in step 2) is completed by the operation records of air traffic controllers in historical data (such as distance measurement, speed adjustment, etc.), or through 4D trajectory prediction and conflict detection to achieve automatic annotation.

[0025] Further, the simulated scenario script is a flight scenario of flights in a manually set airspace, including flight plans and flight situation information in the entire control sector.

[0026] Further, step 3) specifically includes: The comprehensive situation self-awareness representation learning network is:

[0027] φ(V o )→f, C1(v o i , v o j ,...),C2(v o k , v o d ,...),...C N (v o m , v o n ,...)

[0028] In the formula, V o represents the input of the network, that is, the set of aircraft representation vectors in the control sector; the output of the network includes two parts: the first part is the implicit expression of the current comprehensive situation, denoted as the vector f; the second part is the classification result based on f, denoted as C1(v o i , v o j ,...),C2(v o k , v o d ,...),...C N (v o m , v o n ,...), where N is the number of route intersection points in the control sector, represents the representation vector of the i-th aircraft, v o jThe representation vector of the j-th aircraft, v o k The representation vector of the k-th aircraft, v o d The representation vector of the d-th aircraft, v o m The representation vector of the m-th aircraft, v o n The representation vector of the n-th aircraft; the network backbone is composed of multiple layers of neural networks. Using the comprehensive situation self-awareness representation learning dataset constructed in step 2), the comprehensive situation self-awareness representation learning network is trained to obtain the parameter values of the neural nodes of its backbone network.

[0029] Further, step 4) specifically includes: using the output of the comprehensive situation self-awareness representation learning network in step 3) as the input of the control instruction generation network; the output of the control instruction generation network is the probability of the control instruction action; the backbone network of the control instruction generation network includes a bottom-layer graph neural network and a top-layer control instruction action probability prediction network; the bottom-layer graph neural network calculates the graph representation of the sub-problem, denoted as: subgraph{(v i , v j ,...), (v k , v d ,...),...(v m , v n ,...)}, where concat is a vector concatenation operator, and the top-layer control instruction action probability prediction network uses a multi-layer fully connected neural network, with the input outputting the probability of the control instruction action.

[0030] Further, step 5) specifically includes: the first stage uses a reinforcement learning algorithm for training, and the result is used for network model parameter initialization; the second stage uses imitation learning to fine-tune the network parameters.

[0031] Further, the reinforcement learning algorithm uses the Proximal Policy Optimization (PPO) algorithm.

[0032] Further, the action space of the control instruction in the reinforcement learning in step 5) is specifically: [climb altitude, descend altitude, maintain altitude, accelerate, decelerate, maintain speed].

[0033] Further, the system reward function r(s t , a t ) in the reinforcement learning in step 5) is specifically: for the given state and action (s t , at ) Give the corresponding reward value r;

[0034]

[0035] Among them, d is the distance from the current aircraft to the nearest aircraft nearby.

[0036] Furthermore, the simulation environment of the reinforcement learning in step 5) is specifically: the training simulator and its training script supporting the main air traffic control automation system.

[0037] Furthermore, the imitation learning in step 5) specifically includes: establishing a behavioral imitation learning data set, extracting the control history replay data from the replay data of the air traffic control center automation system, including the historical comprehensive situation and the corresponding control instructions, to form the data set.

[0038] Furthermore, the use of imitation learning for network parameter tuning in step 5) is specifically: using the imitation learning data set as the training data set, using the expert policy (the real operation policy of the controllers in the imitation learning data set) and the control instruction generation network output policy obtained in the first stage of training to train the system reward function discriminator; the discriminator and the control instruction generation network perform adversarial training, so as to optimize the parameters of the control instruction generation network.

[0039] The present invention also provides an intelligent control instruction generation system based on situation representation and behavior imitation, including:

[0040] A state space construction module, used to construct the state space of the aircraft representation vector;

[0041] A data set construction module, used to construct a data set for self-perception representation learning of the comprehensive situation;

[0042] A perception network construction module, used to construct a self-perception representation learning network for the comprehensive situation;

[0043] A generation network construction module, used to construct the backbone network of the control instruction generation network;

[0044] A training module, used to train the control instruction generation network using a two-stage training strategy;

[0045] A control instruction generation module, used to cascade the backbone network of the self-perception representation learning of the comprehensive situation and the backbone network of the control instruction generation, input the data of the civil aviation automation system, and sample according to the instruction action probability output by the control instruction generation network to output the control instruction.

[0046] The beneficial effects of the present invention:

[0047] By imitating and learning the replay data of control behaviors in the comprehensive situation of real sectors, the present invention makes full use of the excellent experience and strategies already available to humans to guide the training of the instruction generation network, making up for the limitations of artificially defined reward functions in the simulation environment.

[0048] By establishing self-perceived representation learning of the comprehensive situation, the present invention provides a redundant and complete input vector for further intelligent generation, enhancing the network's encoding ability for the environment and solving the problem of overly simplified problem modeling. At the same time, through self-perception of the comprehensive situation rather than directly predicting and resolving conflicts, it can effectively avoid the problem of abrupt and non-smooth actions of aircraft to avoid conflicts. Brief Description of the Drawings

[0049] Figure 1 It is a flowchart of the method of the present invention. Detailed Embodiments

[0050] For the convenience of those skilled in the art, the present invention will be further described below in conjunction with embodiments and the accompanying drawings. The content mentioned in the embodiments does not limit the present invention.

[0051] Referring to Figure 1 As shown, a method for intelligent generation of control instructions based on situation representation and behavior imitation of the present invention is as follows:

[0052] 1) Construct a state space of the aircraft representation vector based on prior rules;

[0053] In a preferred example, the step 1) specifically includes: extracting strongly correlated items with control decision-making behaviors from the comprehensive situation according to civil aviation control rules; for example: meteorological index x1, wake vortex level x2, longitude x3, latitude x4, altitude x5, speed x6, heading x7, aircraft type x8, target airport x9, distance to the next waypoint x10, etc. After the information is numerically processed, the representation vector of the i-th aircraft is obtained as follows:

[0054]

[0055] Among them, the comprehensive situation includes: the flight plan corresponding to the flight, the comprehensive track feedback after comprehensive processing by the automation system, and the meteorological situation of the waypoints.

[0056] 2) Construct a data set for self-perceived representation learning of the comprehensive situation;

[0057] Among them, the step 2) specifically includes: the data set for self-perceived representation learning of the comprehensive situation is composed of the simulation scenario script for control training extracted from the civil aviation control simulator, and annotation information such as potential conflict flights and the waypoint IDs where conflicts occur in the simulation scenario script.

[0058] Among them, the annotation in step 2) is completed by operating records of air traffic controllers in historical data (such as distance measurement, speed adjustment, etc.), or through 4D trajectory prediction and deduction, conflict detection, etc., so as to achieve automatic annotation;

[0059] Among them, the simulated scenario script is the flight scenario of flights in the artificially set airspace, including flight plans and information such as the flight situation in the entire control sector.

[0060] 3) Construct an integrated situation self-perception representation learning network φ;

[0061] In a preferred example, step 3) specifically includes: the integrated situation self-perception representation learning network is:

[0062] φ(V o )→f, C1(v o i , v o j ,...), C2(v o k , v o d ,...C N (v o m , v o n ,...)

[0063] In the formula, V o represents the input of the network, that is, the set of aircraft representation vectors in the control sector; the output of the network includes two parts: the first part is the implicit expression of the current integrated situation, denoted as the vector f; the second part is the classification result given based on f, denoted as C1(v o i , v o j ,...), C2(v o k , v o d ,...C N (v o m , v o n ,...), where N is the number of airway intersection points in the control sector, represents the representation vector of the i-th aircraft, v o jThe representation vector representing the j-th aircraft, and so on; The network backbone consists of multiple layers of neural networks. Using the comprehensive situation self-awareness representation learning dataset constructed in step 2), the comprehensive situation self-awareness representation learning network is trained to obtain the parameter values of the neural nodes of its backbone network.

[0064] 4) Construct the backbone network of the control instruction generation network;

[0065] In a preferred example, step 4) specifically includes: using the output of the comprehensive situation self-awareness representation learning network in step 3) as the input of the control instruction generation network; the output of the control instruction generation network is the probability of the control instruction action; the backbone network of the control instruction generation network includes a bottom-layer graph neural network and a top-layer control instruction action probability prediction network; the bottom-layer graph neural network calculates the graph representation of the sub-problem, denoted as: subgraph{(v i , v j ,...), (v k , v d ,...),...(v m , v n ,...)} where, concat is a vector concatenation operator, and the top-layer control instruction action probability prediction network uses a multi-layer fully connected neural network, with the input outputting the probability of the control instruction action.

[0066] 5) Train the control instruction generation network using a two-stage training strategy;

[0067] Among them, step 5) specifically includes: The first stage is trained using a reinforcement learning algorithm, and the results are used for network model parameter initialization; the second stage uses imitation learning to fine-tune the network parameters.

[0068] Among them, the reinforcement learning algorithm uses the Proximal Policy Optimization (PPO) algorithm.

[0069] Among them, the action space of the control instruction for reinforcement learning in step 5) is specifically: [climb altitude, descend altitude, maintain altitude, accelerate, decelerate, maintain speed].

[0070] Among them, the system reward function r(s t , a t ) in step 5) of the reinforcement learning is specifically: For a given state and action (s t , a t ), a corresponding reward value r is given;

[0071]

[0072] Wherein, d is the distance from the current aircraft to the nearest aircraft nearby.

[0073] Wherein, the simulation environment of the reinforcement learning in the step 5) is specifically: the training simulator and its training script supporting the main air traffic control automation system.

[0074] Wherein, the imitation learning in the step 5) specifically includes: establishing a behavior imitation learning data set, extracting the control history replay data from the replay data of the air traffic control center automation system, including the historical comprehensive situation and the corresponding control instructions, to form a data set.

[0075] Wherein, the network parameter tuning by using imitation learning in the step 5) is specifically: using the imitation learning data set as the training data set, using the expert policy (the real operation policy of the controllers in the imitation learning data set) and the output policy of the control instruction generation network obtained in the first stage of training to train the system reward function discriminator; the discriminator and the control instruction generation network perform adversarial training, so as to tune the parameters of the control instruction generation network.

[0076] 6) Cascade the backbone network of the self-aware representation learning of the comprehensive situation and the backbone network of the control instruction generation, input the data of the civil aviation automation system, and perform sampling according to the instruction action probability output by the control instruction generation network, and output the control instruction.

[0077] The present invention also provides an intelligent control instruction generation system based on situation representation and behavior imitation, including:

[0078] A state space construction module, configured to construct a state space of the aircraft representation vector;

[0079] A data set construction module, configured to construct a data set for the self-aware representation learning of the comprehensive situation;

[0080] A perception network construction module, configured to construct a self-aware representation learning network of the comprehensive situation;

[0081] A generation network construction module, configured to construct a backbone network of the control instruction generation network;

[0082] A training module, configured to train the control instruction generation network by using a two-stage training strategy;

[0083] A control instruction generation module, configured to cascade the backbone network of the self-aware representation learning of the comprehensive situation and the backbone network of the control instruction generation, input the data of the civil aviation automation system, and perform sampling according to the instruction action probability output by the control instruction generation network, and output the control instruction.

[0084] The specific application scenarios of the present invention are numerous. The above description is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements can be made, and these improvements should also be regarded as the protection scope of the present invention.

Claims

1. An intelligent generation method for control instructions based on situation representation and behavior imitation, characterized in that The steps are as follows: 1) Construct the state space of the aircraft representation vector; 2) Construct the dataset for the self-awareness representation learning of the comprehensive situation; 3) Construct the self-awareness representation learning network for the comprehensive situation; 4) Construct the backbone network of the control instruction generation network; 5) Adopt a two-stage training strategy to train the control instruction generation network; 6) Cascade the backbone network of the self-awareness representation learning of the comprehensive situation and the backbone network of the control instruction generation, input the data of the civil aviation automation system, sample according to the instruction action probability output by the control instruction generation network, and output the control instruction; The specific content of step 1) includes: according to the civil aviation control rules, extracting the strongly correlated items with the control decision-making behavior in the comprehensive situation, and after numerical processing, obtaining the representation vector of the i-th aircraft as follows: Among them, x1 is the meteorological index, x2 is the wake vortex level, x3 is the longitude, x4 is the latitude, x5 is the altitude, x6 is the speed, x7 is the heading, x8 is the aircraft type, x9 is the target airport, and x10 is the distance to the next waypoint; The specific content of step 2) includes: The dataset for the self-awareness representation learning of the comprehensive situation is composed of the simulated scenario script of the control training extracted from the civil aviation control simulator and the annotation information of the potential conflict flights and the waypoint IDs where the conflict occurs in the simulated scenario script; The simulated scenario script is the flight scenario of flights in the artificially set airspace, including the flight plan and the flight situation information in the entire control sector; The annotation in step 2) is completed through the operation records of the air traffic controllers in the historical data, or through 4D trajectory prediction and conflict detection to achieve automatic annotation; The specific content of step 3) includes: The self-awareness representation learning network for the comprehensive situation is: φ(V o )→f,C1(v o i ,v o j ,...),C2(v o k ,v o d ,...),...C N (v o m ,v o n ,...) where, V o represents the input of the network, i.e., the set of aircraft representation vectors in the control sector; the output of the network consists of two parts: the first part is the implicit expression of the current comprehensive situation, denoted as vector f; the second part is the classification result based on f, denoted as C1(v o i , v o j ,...), C2(v o k , v o d ,...),... C N (v o m , v o n ,...), where N is the number of airway intersection points in the control sector, v i o represents the representation vector of the i-th aircraft, v o j represents the representation vector of the j-th aircraft, v o k represents the representation vector of the k-th aircraft, v o d represents the representation vector of the d-th aircraft, v o m represents the representation vector of the m-th aircraft, v o n represents the representation vector of the n-th aircraft; the backbone of the network consists of multiple layers of neural networks. Using the comprehensive situation self-awareness representation learning dataset constructed in step 2), the comprehensive situation self-awareness representation learning network is trained to obtain the parameter values of the neural nodes of its backbone network; The specific steps of step 4) include: using the output of the comprehensive situation self-awareness representation learning network in step 3) as the input of the control instruction generation network; the output of the control instruction generation network is the probability of the control instruction action; the backbone network of the control instruction generation network includes a graph neural network at the bottom layer and a control instruction action probability prediction network at the high layer; the graph neural network at the bottom layer calculates the graph representation of the sub-problem, denoted as: subgraph{(v i ,v j ,...),(v k ,v d ,...),...(v m ,v n ,...)} where concat is a vector concatenation operator, and the control instruction action probability prediction network at the high layer uses a multi-layer fully connected neural network, with the input outputting the probability of the control instruction action.

2. The intelligent generation method of control instructions based on situation representation and behavior imitation according to claim 1, wherein The specific content of step 5) includes: In the first stage, a reinforcement learning algorithm is used for training, and the result is used for initializing the network model parameters; in the second stage, imitation learning is used for tuning the network parameters.

3. The intelligent generation method of control instructions based on situation representation and behavior imitation according to claim 2, wherein The imitation learning in step 5) specifically includes: Establish a behavior imitation learning dataset, extract the control historical replay data from the replay data of the automation system of the air traffic control center, including the historical comprehensive situation and the corresponding control instructions, to form the dataset.

4. The intelligent generation method of control instructions based on situation representation and behavior imitation according to claim 2, characterized in that, The specific content of using imitation learning to tune the network parameters in step 5) is: Using the imitation learning dataset as the training dataset, and using the expert strategy and the output strategy of the control instruction generation network obtained from the first stage of training to train the discriminator of the system reward function; The discriminator and the control instruction generation network perform adversarial training, thereby tuning the parameters of the control instruction generation network.

5. An intelligent generation system for control instructions based on situation representation and behavior imitation, characterized in that, It includes: A state space construction module for constructing the state space of the aircraft representation vector; A dataset construction module for constructing the dataset for the self-awareness representation learning of the comprehensive situation; A perception network construction module for constructing the self-awareness representation learning network for the comprehensive situation; A generation network construction module for constructing the backbone network of the control instruction generation network; A training module for training the control instruction generation network using a two-stage training strategy; A control instruction generation module for cascading the backbone network of the self-awareness representation learning of the comprehensive situation and the backbone network of the control instruction generation, inputting the data of the civil aviation automation system, sampling according to the instruction action probability output by the control instruction generation network, and outputting the control instruction; The state space construction module extracts the strongly correlated items with the control decision-making behavior in the comprehensive situation according to the civil aviation control rules, and obtains the representation vector of the i-th aircraft after numerical processing. As follows: Among them, x1 is a meteorological index, x2 is a wake vortex level, x3 is a longitude, x4 is a latitude, x5 is an altitude, x6 is a speed, x7 is a heading, x8 is an aircraft type, x9 is a target airport, and x10 is the distance to the next waypoint; The dataset construction module: The dataset for integrated situation self-awareness representation learning is composed of the simulated scenario scripts for air traffic control training extracted from a civil aviation control simulator, and the annotation information of the potential conflict flights and the waypoint IDs of the conflict occurrence locations in the simulated scenario scripts; The simulated scenario script is a flight scenario of flights in an artificially set airspace, including a flight plan and the flight situation information in the entire control sector; The annotation is completed through the operation records of air traffic controllers in historical data, or through 4D trajectory prediction and conflict detection to achieve automatic annotation; The perception network construction module: The integrated situation self-awareness representation learning network is as follows: φ(V o ) → f, C1(v o i , v o j ,...), C2(v o k , v o d ,...),...C N (v o m , v o n ,...) Where, V o represents the input of the network, that is, the set of aircraft representation vectors in the controlled sector; the output of the network consists of two parts: the first part is the implicit expression of the current comprehensive situation, denoted as the vector f; the second part is the classification result given based on f, denoted as C1(v o i , v o j ,...), C2(v o k , v o d ,...),... C N (v o m , v o n ,...), where N is the number of airway intersection points in the controlled sector, represents the representation vector of the i-th aircraft, v o j represents the representation vector of the j-th aircraft, v o k represents the representation vector of the k-th aircraft, v o d represents the representation vector of the d-th aircraft, v o m represents the representation vector of the m-th aircraft, v o n represents the representation vector of the n-th aircraft; the network backbone is composed of multiple layers of neural networks. Using the constructed comprehensive situation self-awareness representation learning dataset, the comprehensive situation self-awareness representation learning network is trained to obtain the parameter values of the neural nodes in its backbone network; Use the output of the integrated situation self-perception representation learning network as the input of the control instruction generation network; the output of the control instruction generation network is the probability of the control instruction action; the backbone network of the control instruction generation network includes a bottom-layer graph neural network and a top-layer control instruction action probability prediction network; the bottom-layer graph neural network calculates the graph representation of the sub-problem, denoted as: subgraph{(v i ,v j ,...),(v k ,v d ,...),...(v m ,v n ,...)} , where concat is a vector concatenation operator, and the top-layer control instruction action probability prediction network uses a multi-layer fully connected neural network, with the input Output the probability of the control instruction action.

Images