Multi-aircraft omnidirectional autonomous interval decision-making method considering sustainable operation of civil aviation

Abstract

The invention discloses a multi-aircraft omnidirectional autonomous interval decision-making method considering sustainable operation of civil aviation. The method comprises the following steps: collecting multi-aircraft flight characteristic parameters; based on the flight characteristic parameters, constructing a Markov decision process model considering a civil aviation sustainable operation target, and determining a state space, an action space and a reward function of the model; according to the Markov decision process model, adopting a multi-agent reinforcement learning algorithm to train the strategy network, and performing targeted improvement on an evaluation network structure of the algorithm; and based on the trained strategy network, carrying out real-time extension decision making on multi-aircraft flight parameters so as to complete autonomous interval management. According to the method, training and real-time decision making of multi-aircraft flight parameters are achieved through a reinforcement learning method, the problem of rapid flight parameter decision making considering civil aviation sustainable development under safe interval operation is solved, and a method reference is provided for an air traffic autonomous interval decision making operation mode in the future.

Description

Technical Field

[0001] This invention belongs to the field of civil aviation spacing management technology, specifically involving a multi-aircraft omnidirectional autonomous spacing decision-making method that considers the sustainable operation of civil aviation. Background Technology

[0002] With the continuous growth in flight volume, the current operational model of controlling flight spacing through controllers' manual voice commands faces significant challenges in terms of safety and sustainability. Traditional spacing management methods rely on control rules and controller experience, depending on radar surveillance and ground-based navigation equipment. With the continuous increase in traffic volume, it becomes difficult to achieve safe and efficient dynamic spacing adjustments. Furthermore, the increasing demands for sustainable civil aviation operations necessitate considering fuel efficiency and emissions reduction while ensuring safety, placing higher demands on the refined decision-making capabilities of spacing management technology.

[0003] Existing research generally simplifies multi-aircraft spacing management into a single aircraft trajectory optimization problem, which is insufficient for dynamic decision-making in multi-aircraft collaborative operations. Some studies, considering sustainable operation goals, employ heuristic algorithms to solve and optimize flight parameters, but these suffer from limitations in scalability, local convergence, and low optimization efficiency. Currently, artificial intelligence technology has gradually moved from theoretical research to engineering applications. To achieve safe and efficient dynamic decision-making, a few studies have used reinforcement learning algorithms to solve multi-aircraft spacing management problems. However, in omnidirectional spacing management scenarios, considering sustainability goals, existing algorithms face issues of effectiveness and applicability. Therefore, there is an urgent need for a multi-aircraft omnidirectional autonomous spacing decision-making method for sustainable civil aviation operations. This method should achieve safe multi-aircraft spacing through omnidirectional spacing management scenario modeling and intelligent decision-making algorithms, dynamically determining aircraft flight parameters to reduce operating costs while ensuring safety and promoting sustainable development in civil aviation. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-aircraft omnidirectional autonomous spacing decision-making method that considers the sustainable operation of civil aviation. This method addresses the problem that existing methods are not applicable and cannot meet the scalability and real-time requirements for decision-making in omnidirectional spacing management scenarios, especially for operations with only one route intersection point, when considering sustainability goals.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A multi-aircraft omnidirectional autonomous spacing decision-making method considering the sustainable operation of civil aviation includes the following steps:

[0007] Step 1: Collect multi-aircraft flight characteristic parameters;

[0008] Step 2: Based on the flight characteristic parameters collected in Step 1, construct a Markov decision process model that considers the goal of sustainable civil aviation operation, and determine the state space, action space and reward function of the model.

[0009] Step 3: Based on the Markov decision process model determined in Step 2, a multi-agent reinforcement learning algorithm is used to train the policy network, wherein the evaluation network structure of the algorithm is specifically improved.

[0010] Step 4: Based on the policy network trained in Step 3, make real-time extended decisions on the flight parameters of multiple aircraft to complete autonomous interval management.

[0011] Furthermore, step 1 includes the following sub-steps:

[0012] Step 1.1, Establish a method for characterizing flight characteristic parameters:

[0013] For an aircraft sequence consisting of multiple aircraft Let represent the maximum value of the aircraft sequence number. Then, the total number of aircraft contained in the aircraft sequence is . , Indicates airplane Track characteristic parameters;

[0014] in, This is the aircraft's serial number. Indicates the head machine. Indicates the preceding machine. Indicates the local machine; This represents the x-coordinate value of the horizontal profile. This represents the ordinate value of the horizontal profile. Indicates the vertical profile flight altitude. Indicates flight time;

[0015] Define an airplane The heading is Flight speedometer is Vacuum speed is The quality is The wing reference area is The inclination of the flight path is ;

[0016] Define an airplane The fuel consumption is The total global warming caused by engine emissions is ;

[0017] Step 1.2, Parameter Collection Process:

[0018] During flight, all basic flight status parameters of other aircraft are obtained through the Automatic Dependent Surveillance-Broadcast (ADS-B) data receiver, while basic flight status parameters of the aircraft itself are obtained through the flight management system. All basic flight status parameters required for the decision-making process are obtained by calculating these parameters.

[0019] Furthermore, step 2 includes the following sub-steps:

[0020] Step 2.1, Observation State Design Method and Grouping:

[0021] Mathematical operations are performed on the basic flight state parameters to obtain the observed state of the agent, and the agent is then grouped.

[0022] Step 2.2, determine the decision-making actions and the scope of change:

[0023] By precisely controlling flight speed and the angle of inclination of the aircraft's flight path It can control the flight path, thereby achieving autonomous interval decision-making in the hybrid interval management mode; and will control the change in indicated airspeed. and the change in the aircraft's flight path angle As a decision-making action of a graph agent, i.e. For a group of aircraft, its action group is denoted as... ;

[0024] Step 2.3, determine the progress status of the interval management task:

[0025] There are three possible states for the multi-machine interval management task: task termination unsuccessful, task termination successful, and task not terminated.

[0026] Step 2.4: Establish an environmental reward calculation method:

[0027] Environmental rewards for agents In real-time scenarios, the value varies at each moment, depending on the operational objective. Vertical interval reward Vertical interval rewards Lateral interval rewards Track tilt bonus Fuel consumption reward and temperature rise reward It consists of six parts, in When making decisions, the environment provides intelligent agents with... Single-step reward As shown in the following formula:

[0028]

[0029] Step 2.5: Establish a Markov decision process model.

[0030] Furthermore, in step 2.1, the agent's observed state includes the following 15 items:

[0031] (1) Longitudinal distance between front and rear machines ;

[0032] (2) Height difference between front and rear machines ;

[0033] (3) Difference in heading between the front and rear aircraft ;

[0034] (4) Vacuum speed difference between front and rear units ;

[0035] (5) Difference in inclination angle between the front and rear aircraft tracks ;

[0036] (6) Distance of the aircraft from the route intersection ;

[0037] (7) Flight status of the aircraft This includes both level flight and descent.

[0038] (8) One-hot encoding vector of this model ;

[0039] (9) Flight altitude of the aircraft ;

[0040] (10) Flight speed of the aircraft ;

[0041] (11) Flight vacuum speed of this aircraft ;

[0042] (12) Inclination of the aircraft's flight path ;

[0043] (13) Machine quality ;

[0044] (14) Reference area of ​​the aircraft wing ;

[0045] (15) Inclination angle of the aircraft track and the inclination angle of the target track ;

[0046] The above 15 flight status parameters are divided into 3 groups, as follows:

[0047] Group 1: Status group related to interval management. ;

[0048] Group 2: Status group related to aircraft performance. ;

[0049] Group 3: Critical state group, including state (2) and state (15), specifically, .

[0050] Furthermore, in step 2.2, within the continuous action space, The range of variation is limited by the aircraft's acceleration. With unit time The product is used to determine the specific range of variation. ;also ,in and These are the minimum and maximum values ​​of the change in the aircraft's trajectory inclination angle, respectively.

[0051] As part of the environment, the flight airspeed and trajectory tilt patterns of the lead aircraft are pre-generated and do not require decision-making.

[0052] Furthermore, step 3 includes the following sub-steps:

[0053] Step 3.1, Improve the evaluation network:

[0054] A separate Critic network structure is adopted; for state groups related to interval management... The group uses a Bi-LSTM network; for the critical state group Enhancement processing is performed;

[0055] Step 3.2, determine the decision network model:

[0056] The decision network model consists of a fully connected neural network, including an input layer, an output layer, and hidden layers;

[0057] Step 3.3, Determine the training hyperparameters:

[0058] First, determine the training hyperparameters, including: evaluation network learning rate, decision network learning rate, network latency update rate, reward discount, batch size, number of steps to start learning, and total number of training rounds;

[0059] Next, build and train the model;

[0060] Step 3.4: Determine the training scenario parameters and train the network.

[0061] First, determine the training scenario parameters, including: number of aircraft sequences, aircraft flight range, initial flight altitude, initial mass, lead aircraft flight parameters, route length, route direction, etc.

[0062] Then, the decision network is trained using the model constructed in step 3.3, and the changes in average reward are observed; finally, the parameters of the well-trained decision network are saved.

[0063] Furthermore, in step 3.1, for the critical state group... The method of augmentation processing is to transform decision actions Perform critical state expansion to form .

[0064] Furthermore, step 4 includes the following sub-steps:

[0065] Step 4.1, determine the decision-making scenario parameters for multi-machine omnidirectional autonomous interval management operation:

[0066] The decision scenario parameters are of the same type as the training scenario parameters, including: number of aircraft sequences, aircraft type data parameters, aircraft flight range, initial flight altitude, initial mass, lead aircraft flight parameters, route length, and route direction.

[0067] Step 4.2, Scene and Model Initialization:

[0068] First, the operational scenario is initialized based on the experimental design objectives;

[0069] Then, the agent is initialized based on the saved, fully trained decision network;

[0070] Step 4.3: Conduct decision-making experiments in the runtime scenario:

[0071] The intelligent agent makes decisions based on the decision network and observations of the scene, and calculates the decision success rate, MDP parameters, and multi-aircraft flight parameters.

[0072] This invention also provides a multi-aircraft omnidirectional autonomous spacing decision-making system considering the sustainable operation of civil aviation, comprising:

[0073] At least one processor; and

[0074] A memory that is communicatively connected to the at least one processor;

[0075] The memory stores instructions that can be executed by the at least one processor to enable the system to perform the method.

[0076] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0077] Beneficial effects:

[0078] This invention achieves multi-aircraft omnidirectional autonomous spacing decision-making through a multi-agent reinforcement learning method, and takes into account the issue of sustainable development in civil aviation, providing a reference for future autonomous spacing decision-making tasks. This invention improves the decision success rate by improving the evaluation network structure during the training process, and at the same time enhances the scalability of the method, enabling it to adapt to more complex operating scenarios, providing a methodological reference for future applications. Attached Figure Description

[0079] Figure 1 This invention presents an architecture diagram of a multi-aircraft omnidirectional autonomous spacing decision-making method for sustainable civil aviation operations.

[0080] Figure 2 This is a schematic diagram of the Markov decision-making process under the interaction between the intelligent agent and the environment in this invention.

[0081] Figure 3 This is a schematic diagram of the evaluation network construction and improvement method of the present invention;

[0082] Figure 4 This is a schematic diagram of the network training process of the strategy of the present invention;

[0083] Figure 5 This is a graph showing the convergence curve results in an embodiment of the present invention;

[0084] Figure 6 This is a flowchart of the omnidirectional interval management decision-making process based on a well-trained network, as described in this invention.

[0085] Figure 7 This is a schematic diagram of the decision results in an embodiment of the present invention. Detailed Implementation

[0086] The invention will now be further explained with reference to the accompanying drawings.

[0087] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0088] Please see Figures 1-7 As shown, this embodiment provides a multi-aircraft omnidirectional autonomous spacing decision-making method considering the sustainable operation of civil aviation. This embodiment takes the standard terminal arrival process of multiple aircraft, i.e., the aircraft approach process, as an example for verification, and it specifically includes the following steps:

[0089] Step 1: Collect multi-aircraft flight characteristic parameters;

[0090] Specifically, it includes the following sub-steps:

[0091] Step 1.1, Establish a method for characterizing flight characteristic parameters:

[0092] For an aircraft sequence consisting of multiple aircraft Let represent the maximum value of the aircraft sequence number. Then, the total number of aircraft contained in the aircraft sequence is . , Indicates airplane Track characteristic parameters;

[0093] in, This is the aircraft's serial number. Indicates the head machine. Indicates the preceding machine. Indicates the local machine; This represents the x-coordinate value of the horizontal profile. This represents the ordinate value of the horizontal profile. Indicates the vertical profile flight altitude. Indicates flight time;

[0094] Define an airplane The heading is Flight speedometer is Vacuum speed is The quality is The wing reference area is The inclination of the flight path is ;

[0095] Define an airplane The fuel consumption is According to the BADA model, the global warming (temperature rise) caused by engine emissions is... It is obtained according to the Boeing Method-II method;

[0096] Step 1.2, Parameter Collection Process:

[0097] During flight, all basic flight status parameters of other aircraft can be obtained through the Automatic Dependent Surveillance-Broadcast (ADS-B) data receiver, while basic flight status parameters of the aircraft itself can be obtained through the Flight Management System (FMS). These parameters can be used to calculate all the basic flight status parameters needed for the decision-making process.

[0098] Step 2: Based on the flight characteristic parameters collected in Step 1, construct a Markov decision process model that considers the goal of sustainable civil aviation operation, and determine the state space, action space and reward function of the model.

[0099] Specifically, it includes the following sub-steps:

[0100] Step 2.1, Observation State Design Method and Grouping:

[0101] By performing simple mathematical operations on the basic flight state parameters, the following 15 observation states of the agent are obtained:

[0102] (1) Longitudinal distance between front and rear machines ;

[0103] (2) Height difference between front and rear machines ;

[0104] (3) Difference in heading between the front and rear aircraft ;

[0105] (4) Vacuum speed difference between front and rear units ;

[0106] (5) Difference in inclination angle between the front and rear aircraft tracks ;

[0107] (6) Distance of the aircraft from the route intersection ;

[0108] (7) Flight status of the aircraft This includes both level flight and descent.

[0109] (8) One-hot encoding vector of this model ;

[0110] (9) Flight altitude of the aircraft ;

[0111] (10) Flight speed of the aircraft ;

[0112] (11) Flight vacuum speed of this aircraft ;

[0113] (12) Inclination of the aircraft's flight path ;

[0114] (13) Machine quality ;

[0115] (14) Reference area of ​​the aircraft wing ;

[0116] (15) Inclination angle of the aircraft track and the inclination angle of the target track ;

[0117] The above 15 flight status parameters are divided into 3 groups, as follows:

[0118] Group 1: Status group related to interval management. ;

[0119] Group 2: Status group related to aircraft performance. ;

[0120] Group 3: Critical state group, including state (2) and state (15), specifically, ;

[0121] Step 2.2, determine the decision-making actions and the scope of change:

[0122] By precisely controlling flight speed and the descent angle of the aircraft's trajectory It can control the flight path, thereby achieving autonomous spacing decision-making in a hybrid spacing management mode. (This refers to the change in indicated airspeed.) and the change in the descent angle of the aircraft's trajectory As a decision-making action of a graph agent, i.e. For a group of aircraft, its action group is denoted as... In a continuous action space, The range of variation is limited by the aircraft's acceleration. With unit time The product is used to determine the specific range of variation. ;also ,in and These are the minimum and maximum values ​​of the change in the aircraft's trajectory inclination, determined jointly by aircraft performance limitations and passenger comfort.

[0123] It should be noted that the flight speed and trajectory descent angle changes of the lead aircraft, as part of the environment, are pre-generated and do not require decision-making.

[0124] Step 2.3, determine the progress status of the entry interval management task:

[0125] There are three possible states for the multi-machine approach interval management task:

[0126] (1) Mission termination unsuccessfully means that in a running round, when any aircraft in the multi-aircraft sequence violates the minimum safety separation standards in the longitudinal, lateral and vertical directions at the same time, the current running round ends immediately and the omnidirectional separation management mission terminates in failure.

[0127] (2) Mission termination is successful when the multi-aircraft sequence completes the entire approach process and all aircraft meet the minimum safe interval standard in every step of their operation within a round. At this time, the omnidirectional interval management mission is successfully terminated.

[0128] (3) The mission has not been terminated, which means that at a certain moment the aircraft sequence has not completed the approach process, and each step of the previous operation has met the minimum safe interval standard. At this time, there is a next decision moment.

[0129] Step 2.4, Establish the environmental reward calculation method:

[0130] Environmental rewards for agents In real-time scenarios, the value varies at each moment, depending on the operational objective. Vertical interval reward Vertical interval rewards Lateral interval rewards Descent angle reward Fuel consumption reward and temperature rise reward Composed of six parts, in When making decisions, the environment provides intelligent agents with... Single-step reward As shown below:

[0131]

[0132] Among them, for , and Three types of interval rewards are designed, taking into account the progress status of the entry interval management task, with the following calculation value based on real-time interval parameters. and real-time interval standard calculation value The combination forms a general form of the interval reward function as follows:

[0133]

[0134] Descent track angle bonus Indicates the real-time descent angle and the target track inclination angle. The reward value resulting from interpolation is as follows:

[0135]

[0136] Fuel consumption reward and temperature rise reward as follows:

[0137]

[0138] in and They are Fuel consumption and reference fuel consumption within the vehicle. and They are Internal temperature rise and reference temperature rise;

[0139] In this embodiment, =10 seconds during level flight and descent Take 40 kg and 10 kg respectively; taking the temperature rise on a 20-year timescale as an example, during level flight and descent 2.5×10 -13 ℃ and 0.8×10 -13 ℃.

[0140] Step 2.5, establish the Markov decision process model:

[0141] like Figure 2 As shown An MDP of agents interacting with the environment in one round; agents For airplane Provide decision-making recommendations, with intelligent agents For example: In At any moment, intelligent agent Environmental conditions observed during multi-aircraft access operation. Afterwards, according to the strategy Decision-making and action Subsequently, the environment moves to the next environment state based on the action performed. Meanwhile, the environment rewards the agent to indicate the quality of the decision; finally, if the environment reaches the termination condition, the decision-making ends, otherwise it proceeds to the next moment of observation and decision-making.

[0142] Step 3: Based on the Markov decision process model determined in Step 2, the policy network is trained using the Multi-Agent Reinforcement Learning (MARL) algorithm, specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, and the evaluation network (Critic) structure of the algorithm is improved accordingly.

[0143] Specifically, it includes the following sub-steps:

[0144] Step 3.1, determine the evaluation network improvement method:

[0145] Design an evaluation network considering the following two factors:

[0146] (1) Observation state group and The observed state groups exhibited significant differences in characteristics, specifically as follows: It has strong sequence characteristics, while Then no;

[0147] (2) Critical state group It has a significant impact on decision-making;

[0148] This invention employs a separate Critic network structure; for The group uses a Bi-LSTM network; for the critical state group Enhancement processing is performed, specifically by... Perform critical state expansion to form .

[0149] Specifically, in this embodiment: Figure 3As shown, the evaluation network has one input layer, one output layer for generating Q-values, and four hidden layers for learning the relationship between the input and output. Specifically, the neurons in the HL1 layer extract... and The low-level features of the first layer consist of 22 neurons; the neurons in the HL2 layer combine the two sets of extracted low-level feature information with the action features, and also consist of 22 neurons; the HL3 layer is an access layer... The Bi-LSTM layer of the group contains 22 neurons; HL4 is a merging layer that integrates the features from both groups, containing 11 neurons. Furthermore, They are incorporated into each fully connected layer, which enhances the extended decision-making actions during training.

[0150] Step 3.2, determine the decision network model:

[0151] The decision network model consists of a fully connected neural network, including an input layer, an output layer, and hidden layers;

[0152] In this embodiment, it specifically includes: an input layer containing all observed state features, an output layer containing all action features, and two hidden layers, wherein each hidden layer is composed of a fully connected neural network containing 22 neurons;

[0153] Step 3.3, Determine the training hyperparameters:

[0154] First, determine the training hyperparameters, including: evaluation network learning rate, decision network learning rate, network latency update rate, reward discount, batch size, number of steps to start learning, total number of training rounds, etc. The specific values ​​in this embodiment are shown in Table 1.

[0155] Table 1: Training Hyperparameter Values

[0156] Parameter Description Parameter values Critic Network Learning Rate <![CDATA[10 -3 ]]> Actor Network Learning Rate <![CDATA[10 -4 ]]> Update the decay rates of the Critic and Actor networks. 0.999 Discount return rate 0.9 Learning batch processing volume 64 Initial learning step size 500 Total Learning Rounds 600000

[0157] Next, build and train the model, such as Figure 4 As shown.

[0158] Step 3.4: Determine the training scenario parameters and train the network.

[0159] First, determine the training scenario parameters, including: number of aircraft sequences, aircraft flight range, initial flight altitude, initial mass, lead aircraft flight parameters, route length, route direction, etc.

[0160] Then, the constructed model is used to train the decision network, and the changes in average reward are observed.

[0161] Finally, save the parameters of the well-trained decision network. The average reward change curve during the training process is shown below. Figure 5As shown.

[0162] Step 4: Based on the policy network trained in Step 3, make real-time extended decisions on the flight parameters of multiple aircraft to complete autonomous interval management.

[0163] Specifically, it includes the following sub-steps:

[0164] Step 4.1, determine the decision-making scenario parameters for multi-machine omnidirectional autonomous interval management operation:

[0165] The decision scenario parameters are of the same type as the training scenario parameters, including: number of aircraft sequences, aircraft type data parameters, aircraft flight range, initial flight altitude, initial mass, lead aircraft flight parameters, route length, route direction, etc.

[0166] Step 4.2, Scene and Model Initialization:

[0167] First, the operational scenario is initialized based on the decision scenario parameters;

[0168] Then, the agent is initialized based on the saved, well-trained decision network.

[0169] Step 4.3: Conduct decision-making experiments in the runtime scenario:

[0170] The intelligent agent makes decisions based on a decision network and observations of the scene. The decision-making process is as follows: Figure 6 As shown, specifically:

[0171] In a decision-making round, the agent bases its decisions on the policy and the observed state of the environment. Make action decisions For each agent, if the longitudinal, lateral, and vertical spacing between adjacent agents are all less than the minimum safe interval requirement, the current round of operation is stopped and marked as a failure; if the longitudinal spacing between adjacent agents exceeds the minimum safe interval requirement... To ensure the decision-making process is not limited by the distance between adjacent aircraft during training, speed decisions are corrected according to the algorithm. For cases where the termination state has not been reached, the speed and descent trajectory angle actions need to be updated using the algorithm based on decision action constraints. After each agent completes the necessary action updates, the aircraft sequence simultaneously executes the updated actions. The environmental state transitions to the next step. and give a reward Then proceed to the next decision. When the end time is reached, the task is completed and marked as successful.

[0172] After completing a round of decision-making, output and save the results. All decisions made by the aircraft agent and their corresponding flight parameters, as well as the reward for each decision, are detailed as follows: Figure 7 As shown.

[0173] Based on the complete process and implementation details disclosed in the above method embodiments, those skilled in the art will understand that the present invention also provides a multi-machine omnidirectional autonomous interval decision-making system and a corresponding storage medium for implementing the method.

[0174] The multi-aircraft omnidirectional autonomous spacing decision-making system, as a preferred implementation, relies on the synergy of software and hardware to achieve the aforementioned method. This system typically includes at least one processor and a memory communicatively connected to the processor. The memory stores computer program instructions, which, when executed by the processor, control the system to sequentially perform the following operations: collecting multi-aircraft flight characteristic parameters; constructing a Markov decision process model integrating the goals of sustainable civil aviation operation based on the characteristic parameters, specifically including designing and determining its state space, action space, and reward function; then training the policy network using a multi-agent reinforcement learning algorithm with targeted improvements to the evaluation network structure; finally, making real-time decisions on multi-aircraft flight parameters based on the trained policy network to achieve omnidirectional autonomous spacing management. This system, as the physical carrier of the method, can provide real-time and intelligent decision support for the autonomous spacing decision-making operation mode of future air traffic.

[0175] Accordingly, the computer program instructions (including the trained policy network model) for implementing the above method steps can be stored in a computer-readable storage medium (e.g., USB flash drive, external hard drive, read-only memory, random access memory, magnetic disk, or optical disk). When the program in the storage medium is read into a computer or dedicated processor system and executed, the autonomous interval decision-making method described in this invention can be implemented.

[0176] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-aircraft omnidirectional autonomous spacing decision-making method considering sustainable civil aviation operations, characterized in that: Includes the following steps: Step 1: Collect multi-aircraft flight characteristic parameters; Step 2: Based on the flight characteristic parameters collected in Step 1, construct a Markov decision process model that considers the goal of sustainable civil aviation operation, and determine the state space, action space and reward function of the model. Step 3: Based on the Markov decision process model determined in Step 2, a multi-agent reinforcement learning algorithm is used to train the policy network, wherein the evaluation network structure of the algorithm is specifically improved. Step 4: Based on the policy network trained in Step 3, make real-time extended decisions on the flight parameters of multiple aircraft to complete autonomous interval management.

2. The method according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 1.1, Establish a method for characterizing flight characteristic parameters: For an aircraft sequence consisting of multiple aircraft Let represent the maximum value of the aircraft sequence number. Then, the total number of aircraft contained in the aircraft sequence is . , Indicates airplane Track characteristic parameters; in, This is the aircraft's serial number. Indicates the head machine. Indicates the preceding machine. Indicates the local machine; This represents the x-coordinate value of the horizontal profile. This represents the ordinate value of the horizontal profile. Indicates the vertical profile flight altitude. Indicates flight time; Define an airplane The heading is Flight speedometer is Vacuum speed is The quality is The wing reference area is The inclination of the flight path is ; Define an airplane The fuel consumption is The total global warming caused by engine emissions is ; Step 1.2, Parameter Collection Process: During flight, all basic flight status parameters of other aircraft are obtained through the Automatic Dependent Surveillance-Broadcast (ADS-B) data receiver, while basic flight status parameters of the aircraft itself are obtained through the flight management system. All basic flight status parameters required for the decision-making process are obtained by calculating these parameters.

3. The method according to claim 1, characterized in that: Step 2 includes the following sub-steps: Step 2.1, Observation State Design Method and Grouping: Mathematical operations are performed on the basic flight state parameters to obtain the observed state of the agent, and the agent is then grouped. Step 2.2, determine the decision-making actions and the scope of change: By precisely controlling flight speed and the angle of inclination of the aircraft's flight path It can control the flight path, thereby achieving autonomous interval decision-making in the hybrid interval management mode; and will control the change in indicated airspeed. and the change in the aircraft's flight path angle As a decision-making action of a graph agent, i.e. For a group of aircraft, its action group is denoted as... ; Step 2.3, determine the progress status of the interval management task: There are three possible states for the multi-machine interval management task: task termination unsuccessful, task termination successful, and task not terminated. Step 2.4: Establish an environmental reward calculation method: Environmental rewards for agents In real-time scenarios, the value varies at each moment, depending on the operational objective. Vertical interval reward Vertical interval rewards Lateral interval rewards Track tilt bonus Fuel consumption reward and temperature rise reward It consists of six parts, in When making decisions, the environment provides intelligent agents with... Single-step reward As shown in the following formula: Step 2.5: Establish a Markov decision process model.

4. The method according to claim 3, characterized in that: In step 2.1, the agent's observation state includes the following 15 items: (1) Longitudinal distance between front and rear machines ; (2) Height difference between front and rear machines ; (3) Difference in heading between the front and rear aircraft ; (4) Vacuum speed difference between front and rear units ; (5) Difference in inclination angle between the front and rear aircraft tracks ; (6) Distance of the aircraft from the route intersection ; (7) Flight status of the aircraft It includes both level flight and descent. (8) One-hot encoding vector of this model ; (9) Flight altitude of the aircraft ; (10) Flight speed of this aircraft ; (11) Flight vacuum speed of this aircraft ; (12) Inclination of the aircraft's flight path ; (13) Machine quality ; (14) Reference area of ​​the aircraft wing ; (15) Inclination angle of the aircraft track and the inclination angle of the target track ; The above 15 flight status parameters are divided into 3 groups, as follows: Group 1: Status group related to interval management. ; Group 2: Status group related to aircraft performance. ; Group 3: Critical state group, including state (2) and state (15), specifically, .

5. The method according to claim 3, characterized in that: In step 2.2, within the continuous action space, The range of variation is limited by the aircraft's acceleration. With unit time The product is used to determine the specific range of variation. ;also ,in and These are the minimum and maximum values ​​of the change in the aircraft's trajectory inclination angle, respectively. As part of the environment, the flight airspeed and trajectory tilt patterns of the lead aircraft are pre-generated and do not require decision-making.

6. The method according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1, Improve the evaluation network: A separate Critic network structure is adopted; for state groups related to interval management... The group uses a Bi-LSTM network; for the critical state group Enhancement processing is performed; Step 3.2, determine the decision network model: The decision network model consists of a fully connected neural network, including an input layer, an output layer, and hidden layers; Step 3.3, Determine the training hyperparameters: First, determine the training hyperparameters, including: evaluation network learning rate, decision network learning rate, network latency update rate, reward discount, batch size, number of steps to start learning, and total number of training rounds; Next, build and train the model; Step 3.4: Determine the training scenario parameters and train the network. First, determine the training scenario parameters, including: number of aircraft sequences, aircraft flight range, initial flight altitude, initial mass, lead aircraft flight parameters, route length, route direction, etc. Then, the decision network is trained using the model constructed in step 3.3, and the changes in average reward are observed; finally, the parameters of the well-trained decision network are saved.

7. The method according to claim 6, characterized in that: In step 3.1, for the critical state group The method of augmentation processing is to transform decision actions Perform critical state expansion to form .

8. The method according to claim 1, characterized in that: Step 4 includes the following sub-steps: Step 4.1, determine the decision-making scenario parameters for multi-machine omnidirectional autonomous interval management operation: The decision scenario parameters are of the same type as the training scenario parameters, including: number of aircraft sequences, aircraft type data parameters, aircraft flight range, initial flight altitude, initial mass, lead aircraft flight parameters, route length, and route direction. Step 4.2, Scene and Model Initialization: First, the operational scenario is initialized based on the experimental design objectives; Then, the agent is initialized based on the saved, fully trained decision network; Step 4.3: Conduct decision-making experiments in the runtime scenario: The intelligent agent makes decisions based on the decision network and observations of the scene, and calculates the decision success rate, MDP parameters, and multi-aircraft flight parameters.

9. A multi-aircraft omnidirectional autonomous spacing decision-making system considering the sustainable operation of civil aviation, characterized in that: include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the system to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images