Multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning

Abstract

The invention discloses a real-time three-dimensional multi-machine conflict resolution method based on graph reinforcement learning, and the method comprises the steps: constructing a three-dimensional air traffic interaction scene, carrying out the modeling of the scene into a graph, and obtaining the convolution features of the graph as model input; and establishing a conflict buffer model, and designing a reward function to guide the aircraft to minimize an additional flight distance while ensuring a safety interval. A D3QN reinforcement learning algorithm is adopted for training, the optimal avoidance action of the aircraft is obtained, and real-time multi-aircraft conflict resolution in the high-density airspace is achieved.

Description

Technical Field

[0001] The invention relates to a multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning, belonging to the technical field of aviation conflict resolution. Background Art

[0002] In a highly complex airspace environment, flight efficiency and safety have become crucial issues in the aviation field. Traditional centralized methods are limited by the computing power of the human brain and are often unable to handle high-density air traffic and real-time decision-making. With the development of artificial intelligence technology, reinforcement learning technology has been proposed in the aviation field as an effective tool to solve the problem of conflict resolution in air traffic management. In existing studies, research considering the impact between aircraft mainly focuses on two-dimensional environmental scenarios, which does not conform to the actual operating environment. Research on multi-aircraft conflict resolution problems in three-dimensional scenarios often finds it difficult to handle dynamic environments, resulting in huge computational complexity and low efficiency. Therefore, there is an urgent need for a method that is suitable for three-dimensional operating scenarios and can efficiently solve the real-time conflict resolution problem of multiple aircraft. Summary of the invention

[0003] The present invention provides a multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning, which solves the problems disclosed in the background technology.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning, comprising:

[0005] Construct a three-dimensional air traffic operation scenario and generate four random polygonal airspaces 1,000 feet apart. Aircraft are initially randomly distributed and kept horizontal on the flight altitude layer. The flight plan is defined by waypoints. At each time step, the azimuth between the aircraft and the target point at the edge of the airspace is calculated to ensure that the aircraft flies towards the target point. The multi-agent flight environment is modeled as a graph G = (ν, ε), where ν represents the set of nodes and ε represents the set of edges. Aircraft in a dynamic environment are regarded as graph nodes, and potential conflict relationships between agents are modeled as edges. The multi-aircraft conflict resolution problem is modeled as a Markov decision problem, and the state space S is constructed based on the graph. t =[N t ,A t ,F t ],N t The state information of each aircraft is combined to form a node feature matrix. t is the adjacency matrix representing the conflict relationship between aircraft, F t The mask matrix is ​​used to filter out the aircraft that have reached the target point.

[0006] A conflict buffer model is established. Each aircraft is assigned a conflict protection zone and a conflict warning zone. The conflict radii are r and p and r w When aircraft at the same or adjacent altitudes invade the conflict buffer zone, it is considered to have a potential conflict. Otherwise, there is no conflict relationship between the aircraft. Design reward function Guide aircraft to learn strategies that balance safety and economy. t ,a t ) i is the flight state s of aircraft i at time t t Next select action a t After that, the reward value given by the environment is and Quantify the extent of aircraft intrusion into conflict protection areas and conflict warning areas respectively. Depending on the aircraft's action intention, the aircraft's horizontal avoidance action includes heading adjustment and speed adjustment, and the vertical avoidance action includes altitude adjustment. We expect the aircraft to fly directly to the target point in a shorter distance, so the heading choice that deviates from the target azimuth will be penalized to achieve track recovery.

[0007] Graph convolution is used to process dynamic graph networks to capture the characteristics and interaction relationships of multiple agents. Based on this constructed state space, the D3QN algorithm is used to train the agent to obtain the Q value of each collision avoidance action in the optional action space of the aircraft, and the ε-greedy greedy strategy is combined to select the best action. Real-time three-dimensional multi-aircraft conflict resolution is achieved.

[0008] Compared with the prior art, the advantages of the present invention are as follows:

[0009] 1) A three-dimensional air traffic operation scenario was constructed to generate a polygonal airspace that is more in line with the actual air traffic control. The intelligent agent performed maneuverable flights based on three-dimensional flight intentions at different altitudes.

[0010] 2) A conflict buffer model combining horizontal and vertical distances is proposed, which simplifies the conflict detection of aircraft in complex three-dimensional operating environments, avoids the calculation of three-dimensional Euclidean distance, and reduces the amount of calculation.

[0011] 3) A track recovery step is added after the aircraft completes CR to minimize the additional flight distance, reduce the training process, and reduce the burden on airspace capacity.

[0012] 4) Modeling the flight status of aircraft and the conflict relationship between aircraft as a conflict network diagram can more comprehensively represent the status information of aircraft and learn conflict strategies more effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the method of the present invention;

[0014] Figure 2 To construct a three-dimensional air traffic conflict scenario;

[0015] Figure 3 A conflict buffer model based on horizontal distance and vertical altitude layer;

[0016] Figure 4 A conflict network diagram to represent aircraft status information and conflict relationships; DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0018] Example: Figure 1 As shown, a real-time three-dimensional multi-machine conflict resolution method based on graph reinforcement learning includes the following steps:

[0019] Step 1: Build a 3D air traffic operation scenario, initialize aircraft and airspace, determine the flight plan route, and generate four random polygonal airspaces 1,000 feet apart;

[0020] Step 2: Establish a conflict buffer model and assign conflict protection areas and conflict warning areas and corresponding conflict radii to aircraft;

[0021] Step 3: Model the multi-aircraft interaction scenario as a graph. The aircraft in the conflict scenario are regarded as nodes of the graph, and the potential conflict relationships between aircraft constitute the edges of the graph. The characteristics and interactions of the aircraft are modeled using the graph representation;

[0022] Step 4: Determine the action space based on the avoidance measures of the aircraft during actual operation, design the reward function based on the safety and economy of the aircraft behavior, and guide the aircraft to learn a strategy that takes into account both safety and economy.

[0023] Step 5: Input the state space tuple constructed based on the graph convolutional network into the reinforcement learning module as the aircraft observation space, and use the D3QN algorithm to solve the Q value of the aircraft avoidance action. Combined with the ε-greedy greedy strategy, the best action is selected to achieve real-time three-dimensional multi-aircraft conflict resolution.

[0024] In step 1, a three-dimensional air traffic operation scenario is constructed. Figure 2As shown, four random polygonal airspaces are generated 1000 feet apart, and the aircraft are initially randomly distributed at the flight altitude level to maintain horizontality. The flight plan of each aircraft is a straight line from the initialized starting point to the end point at the edge of the airspace. If the aircraft flies according to the trajectory of the flight plan, some conflicts may occur during the flight. The aircraft needs to change the flight trajectory through a series of conflict avoidance actions to ensure a safe interval. On this basis, the azimuth between the current position of the aircraft and the target point is calculated at each time step, and the aircraft is guided to always fly towards the end point. The aircraft that completes the flight goal is marked as done, which means that the aircraft has completed the conflict resolution task.

[0025] In step 2, a conflict buffer model is constructed, and each aircraft is assigned a conflict protection zone and a conflict warning zone, such as Figure 3 As shown, the conflict radius is r p and r w , the height is the altitude layer occupied by the aircraft. If the aircraft's altitude change intention in the next time step is consistent with the current altitude layer, the aircraft is considered to occupy only the current altitude layer. If the aircraft has an intention to climb or descend in the next time step, the aircraft is considered to occupy both the current altitude layer and the intended altitude layer. When an aircraft invades the conflict warning zone of an adjacent aircraft at the same or adjacent altitude layer, it is considered to have a potential conflict. If it invades the conflict protection zone of an adjacent aircraft, a collision is considered to have occurred. Otherwise, it is not considered that there is a conflict relationship between aircraft.

[0026] In step 3, the multi-machine interaction scenario is modeled as a graph G = (ν, ε), such as Figure 4 As shown. Where ν is the set of nodes and ε is the set of edges. The aircraft in the scene are regarded as nodes, and each node has adjacent points. The conflict relationship between aircraft is modeled as an edge. The multi-aircraft conflict resolution problem is modeled as a Markov decision problem, and the graph representation is processed as a tuple of node features, adjacency matrix and mask matrix to construct the state space S t =[N t ,A t ,F t ],N t The node feature matrix is ​​formed by combining the feature information of each aircraft at time step t, which represents the features of the constructed scene. It is described as:

[0027]

[0028] Among them, [P i ,V i ,H i ,I i ] represents the normalized state matrix of each aircraft, P i =[x i ,y i ,Li ] is the position information of the aircraft, x i ,y i is the two-dimensional coordinate of the aircraft, L i V is the altitude level of the aircraft. i represents the speed of the aircraft, H i Indicates the current heading of the aircraft, I i Indicates the aircraft's next action intention.

[0029] A t The adjacency matrix represents the conflict relationship between aircraft, which is expressed as:

[0030]

[0031] If there is a conflict between aircraft i and aircraft j, then a ij =1, indicating that the characteristic information between the two aircraft is shared.

[0032] F t The mask matrix is ​​used to output the filtered aircraft that are performing conflict resolution tasks, expressed as:

[0033] F t =[f1,f2,…,f n ]

[0034] If aircraft i has completed its flight mission and arrived at the target point, then f i =0, otherwise f i =1.

[0035] In step 4, when traditional air traffic control is performing interval adjustment on aircraft, the horizontal adjustment used is usually speed adjustment and heading adjustment. The vertical adjustment is usually altitude adjustment. In order to ensure the smoothness of the flight track, the action space of the aircraft consists of discrete flight intentions rather than specific actions. The action space at time step t is represented as a triple:

[0036]

[0037] in, Indicates speed intention, with three options: low speed, normal speed, and high speed flight. Indicates heading intention, with three options: left yaw, direct flight to target point, and right yaw. Indicates altitude intention, with three options: climb to a higher altitude layer, maintain the current altitude layer, and descend to the next altitude layer.

[0038] Based on the above definition of action space, the action intention is encoded as:

[0039]

[0040] By combining the action intentions of the three dimensions, the aircraft will have 27 actions to choose from at time step t. Each action number represents a specific intention combination, and the relationship between them is:

[0041]

[0042] Each action value a t ∈[0,26] corresponds to a unique intention combination For example, when a t =7, It must be (1,2,0), which means the flight intention is "normal speed - right yaw - climb".

[0043] After the aircraft resolves the conflict, the trajectory will deviate from the flight plan, resulting in additional flight distance. In order to guide the aircraft to learn a strategy that takes into account both safety and economy, and minimize the additional flight distance while maintaining a three-dimensional safety interval, the reward function is designed as follows:

[0044]

[0045] Among them, r(s t ,a t ) i is the flight state s of aircraft i at time step t t Next select action a t After that, the reward value given by the environment; the reward value is based on the safety reward and and economic incentives and Quantify the extent to which an aircraft intrudes into the conflict protection zone and the conflict warning zone. The aircraft receives a bonus for recovering its flight path after deviating from the original flight plan. It depends on the aircraft's action intention. They represent the reward values ​​of speed intention, heading intention, and altitude intention respectively. In the real-time three-dimensional multi-aircraft conflict problem, it is expected that the aircraft will fly at a normal speed, that is, If the aircraft's speed intention is not normal, it will be punished. At the beginning of each time step, the aircraft's current position and the azimuth of the target point are calculated as the heading to guide the aircraft to fly towards the predetermined target point. So when a time step ends, if That is, when the aircraft's heading intention is to fly directly to the target point, the aircraft will be closer to the destination, and a bonus value will be given to the aircraft to encourage a shorter additional flight distance. When an aircraft chooses to yaw left or right, it will cause extra flight distance and be penalized. Although changing the aircraft's altitude can effectively eliminate conflicts, it may affect the stability of the airspace and cause secondary conflicts. Aircraft attempting to climb or descend to other levels will be penalized. The aircraft will receive bonus points for maintaining the current altitude.

[0046] In step 5, the node feature matrix is ​​first input into the fully connected layer, expressed as: N FC =φ FC (N t ). The processed node feature matrix N FC And the adjacency matrix A t Input into GCN through the formula Generate graph convolution features G t Among them, φ GCN represents the graph convolution operator, D t Based on A t Calculate and obtain, D ii =∑ j A ij , σ represents the ReLU activation function. In order to save storage space and improve computational efficiency, the adjacency matrix A t Convert it into a sparse matrix and store the adjacency matrix A by a triple consisting of row number, column number, and element value. t .

[0047] The graph convolution features are input into the deep reinforcement learning module, and the Q value of the aircraft's collision avoidance action is generated through mask matrix filtering. The DQN algorithm is a value-based reinforcement learning algorithm. The agent takes actions in the environment to interact with the environment and obtains the returned reward value. Q(s,a) represents the expectation of future rewards for taking action a in state s at a certain time step. The main idea of ​​the Q-value-based reinforcement learning method is to select the action that can obtain the maximum reward based on the Q value. The Q value is updated as follows:

[0048]

[0049] DQN gradually approaches the optimal Q value through the online Q network Q(s,a,θ). The agent observes the current state s in the environment and selects an action a from the online Q network Q(s,a,θ) according to the ε-greedy strategy, that is:

[0050]

[0051] Execute action a and obtain an immediate reward r and a new state s' from the environment. Store the experience sample (s, a, r, s') of taking action a in state s and getting reward r and transferring to the next state s' into the experience replay pool D. Sample a small batch (s j ,a j ,rj,sj') for training. Use the target Q network to calculate the target Q value Y for each state action j , the calculation process can be expressed as

[0052]

[0053] Use online Q network Q(s j ,a j ,θ) calculate the predicted Q value and the target Q value Y j For comparison, calculate the loss function:

[0054]

[0055] Using the back-propagation algorithm, the parameters θ of the online Q network are updated by gradient descent based on the loss function L(θ), and after a certain number of iterations, the parameters θ of the online Q network are synchronized to the parameters θ' of the target Q network.

[0056] The disadvantages of the DQN algorithm are also obvious: due to the use of the maximum operator, it is easy to cause the Q value to be over-estimated. At the same time, when the online Q network and the target Q network have the same structure, the learning and convergence efficiency are low.

[0057] Double DQN was proposed to solve the overestimation problem of DQN. The difference between it and the DQN algorithm is that the target Q value is estimated in a different way. When updating the Q value, DQN uses the same Q network to select actions and evaluate the Q value of actions. DoubleDQN separates action selection and Q value evaluation: first use the online Q network Q(s,a,θ) to select the optimal action a', and then use the target Q network Q(s',a';θ') to evaluate the Q value of the action. The formula for calculating the target Q value using Double DQN is:

[0058]

[0059] The Dueling DQN algorithm is proposed to solve the network structure optimization problem of DQN. Its main improvement over DQN is the introduction of separate estimation of the advantage function and the state-value function. In Dueling DQN, the Q value function is redefined as:

[0060]

[0061] Among them, V(s) is the state value function, which measures the overall value of state s and represents the expected long-term return in a certain state without considering the specific action. A(s,a) is the advantage function, which represents the advantage of choosing action a over other actions in state s. Dueling DQN can more effectively learn state information that does not depend on specific actions by estimating the state value function V(s) separately, and at the same time better distinguish the relative importance of each action through the advantage function A(s,a), which can effectively improve the efficiency and stability of training. The target Q value of Dueling DQN can be calculated as follows, where α' and β' are the parameters of the fully connected layer:

[0062]

[0063] The D3QN algorithm combines the improvements of Double DQN and Dueling DQN on the DQN algorithm, and uses an online Q network to select the optimal action α'=argmax in the next state s' a' Q(s',a';θ), uses the target network Q(s',a';θ') to evaluate the Q value of the selected optimal action a', whose Q value is estimated by a combination of state value and advantage function. Through this combination, D3QN avoids the over-estimation problem of Q value in traditional DQN when estimating Q value, and effectively improves the learning efficiency in complex states.

[0064] Interaction between the agent model and the environment model: The graph convolution features are used as the current environment state s observed by the aircraft at time step t t , the agent selects the aircraft in the current flight state s t The best action a under t And take action according to the current strategy π(a|s) where π(a|s) = Pr(a t =a|s t =s) indicates that the agent is in state s t Next select action a t After executing the action, the environment returns a reward value r t =R(s t ,a t ), and according to the state transfer function p(s'|s,a)=Pr[s t+1 =s',r t+1 =r|s t =s,a t =a], s∈S′, a∈A, s′∈S′ transfers to the next state s′. The state transition function represents the state of t Next, perform action a t After that, the probability of transferring to the next state s'.

[0065] Update the action-state value function Q(s,a) based on the Bellman expectation equation so that Q(s t ,a t ) is closer to the actual cumulative return. It can be expressed as:

[0066]

[0067] Among them, γ is the discount factor representing the weight of future expectations, represents the maximum Q value that can be obtained by selecting the optimal action a' in the next state s', and α represents the learning rate. From the formula, it can be seen that the current Q value is determined by the immediate reward r t and optimal future returns By continuously updating the Q value, the agent gradually approaches the optimal strategy.

[0068] After updating the Q-value, the agent can also update its strategy based on the new Q-value. Usually in each state, the agent will choose the action that maximizes the Q-value.

[0069]

[0070] By updating the strategy, it is ensured that in each state s, the agent will choose the action a that brings the maximum expected return. The agent repeats the above process in multiple time steps, constantly interacting with the environment, sensing the state, selecting actions, obtaining rewards, and updating the Q value and strategy until the Q value gradually converges.

[0071] The goal of reinforcement learning is to maximize the cumulative reward of the agent throughout the process. t represents the discounted sum of all rewards from time step t to the future:

[0072]

[0073] The agent continuously learns to maximize the cumulative reward G t .

[0074] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning, characterized by: The method comprises the following steps: Step 1: Build a 3D air traffic operation scenario, initialize aircraft and airspace, and determine the flight plan route; Step 2: Establish a conflict buffer model based on horizontal distance and vertical altitude layer, and assign conflict protection area and conflict warning area and corresponding conflict radius to aircraft; Step 3: Model the multi-aircraft interaction scenario as a graph, treat the aircraft in the conflict scenario as nodes of the graph, and the potential conflict relationships between aircraft constitute the edges of the graph. Use the graph representation to model the characteristics and interactions of the aircraft. Step 4: Determine the action space based on the avoidance measures of the aircraft during actual operation, and design the reward function based on the safety and economy of the aircraft behavior; Step 5: Input the state space tuple constructed based on the graph convolutional network into the reinforcement learning module as the aircraft observation space, and use the D3QN algorithm to solve the Q value of the aircraft avoidance action.

2. The multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning according to claim 1 is characterized by: In step 1, a three-dimensional air traffic operation scenario is constructed, and four random polygonal airspaces are generated 1,000 feet apart. Aircraft are initially randomly distributed at different flight altitudes and kept horizontal. The flight plan of each aircraft is a straight line from the initialized starting point to the end point at the edge of the airspace. If the aircraft flies according to the trajectory of the flight plan, some conflicts may occur during the flight. The aircraft needs to change the flight trajectory through a series of conflict avoidance actions to ensure a safe interval. On this basis, a reward function is designed so that the aircraft can achieve track recovery while ensuring safety, fly towards the end point with the minimum additional flight distance, and the aircraft that completes the flight goal is marked as completed, which means that the aircraft has completed the conflict resolution task.

3. The multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning according to claim 1 is characterized by: In step 2, a conflict buffer model is constructed as follows: Consider a conflict definition that combines horizontal and vertical distances. In the horizontal direction, each aircraft is assigned a conflict protection zone and a conflict warning zone, and the conflict radii are r and r, respectively. p and r w In the vertical direction, the aircraft must move at the specified altitude layer. If the aircraft's altitude change intention in the next time step is consistent with the current altitude layer, the aircraft is considered to occupy only the current altitude layer. If the aircraft has an intention to climb or descend in the next time step, the aircraft is considered to occupy both the current altitude layer and the intended altitude layer. When the horizontal distance between the two aircraft is less than the safety radius R, where R = r w , and the two aircraft are at the same or adjacent altitudes, they are considered to be in a conflict relationship.

4. The multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning according to claim 1 is characterized in that: In step 3, the multi-aircraft interaction scene is modeled as a graph G = (ν, ε), where ν is a set of nodes and ε is a set of edges. The aircraft in the scene are regarded as nodes, each node has adjacent points, the conflict relationship between aircraft is modeled as an edge, and the multi-aircraft conflict resolution problem is modeled as a Markov decision problem. The graph representation is processed as a tuple of node features, and the adjacency matrix and mask matrix are used to construct the state space S t =[N t ,A t ,F t ],N t The node feature matrix is ​​formed by combining the feature information of each aircraft at time step t, which represents the features of the constructed scene and is described as: Among them, [P i ,V i ,H i ,I i ] represents the normalized state matrix of each aircraft, P i =[x i ,y i ,L i ] is the position information of the aircraft, x i ,y i is the two-dimensional coordinate of the aircraft, L i is the altitude level of the aircraft, V i represents the speed of the aircraft, H i Indicates the current heading of the aircraft, I i Indicates the aircraft's next action intention. A t The adjacency matrix represents the conflict relationship between aircraft, which is expressed as: If there is a conflict between aircraft i and aircraft j, then a ij =1, indicating that the characteristic information between the two aircraft is shared. F t The mask matrix is ​​used to output the filtered aircraft that are performing conflict resolution tasks, expressed as: F t =[f1,f2,…,f n ] If aircraft i has completed its flight mission and arrived at the target point, then f i =0, otherwise f i =1.

5. The multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning according to claim 1 is characterized in that: In step 4, when traditional air traffic control conducts interval deployment of aircraft, the horizontal deployment used is usually speed adjustment and heading adjustment, and the vertical deployment is altitude adjustment. In order to ensure the smoothness of the flight track, the action space of the aircraft is composed of discrete flight intentions. The action space at time step t is represented as a triple: in, Indicates speed intention, with three options: low speed, normal speed, and high speed flight. Indicates the heading intention, with three options: left yaw, direct flight to the target point, and right yaw. Indicates altitude intention, with three options: climb to a higher altitude layer, maintain the current altitude layer, and descend to the next altitude layer. Based on the above definition of action space, the action intention is encoded as: By combining the action intentions of the three dimensions, the aircraft will have 27 actions to choose from at time step t. Each action number represents a specific intention combination, and the relationship between them is: Each action value a t ∈[0,26] corresponds to a unique intention combination In order to guide the aircraft to learn a strategy that takes into account both safety and economy, and to minimize the additional flight distance while maintaining a three-dimensional safety interval, the reward function is designed as follows: Among them, r(s t ,a t ) i is the flight state s of aircraft i at time step t t Next select action a t After that, the reward value given by the environment; the reward value is based on the safety reward and and economic incentives and Quantify the extent to which an aircraft intrudes into the conflict protection zone and the conflict warning zone. The aircraft receives a bonus for recovering its flight path after deviating from the original flight plan. It depends on the aircraft's action intention. They represent the reward values ​​of speed intention, heading intention, and altitude intention respectively. In the real-time three-dimensional multi-aircraft conflict problem, it is expected that the aircraft will fly at a normal speed, that is, If the aircraft's speed intention is not normal, it will be punished. At the beginning of each time step, the current position of the aircraft and the azimuth of the target point are calculated as the heading to guide the aircraft to fly towards the predetermined target point. So when a time step ends, if That is, when the aircraft's heading intention is to fly directly to the target point, the aircraft will be closer to the destination, and a bonus value will be given to the aircraft to encourage a shorter additional flight distance. When an aircraft chooses to yaw left or right, it will cause extra flight distance and be penalized. Although changing the aircraft's altitude can effectively eliminate conflicts, it may affect the stability of the airspace and cause secondary conflicts. Aircraft attempting to climb or descend to other levels will be penalized. The aircraft will receive bonus points for maintaining the current altitude.

6. The multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning according to claim 1 is characterized by: In step 5, the node feature matrix is ​​first input into the fully connected layer, expressed as: N FC =φ FC (N t ), the processed node feature matrix N FC And the adjacency matrix A t Input into GCN through the formula Generate graph convolution features G t , where φ GCN represents the graph convolution operator, D t Based on A t Calculate and obtain, D ii =∑ j A ij , σ represents the ReLU activation function. In order to save storage space and improve computational efficiency, the adjacency matrix A is t Convert it into a sparse matrix and store the adjacency matrix A by a triple consisting of row number, column number, and element value. t . The graph convolution features are input into the deep reinforcement learning module, and the Q value of the aircraft collision avoidance action is generated after filtering by the mask matrix. The DQN algorithm is a value-based reinforcement learning algorithm. The agent takes actions in the environment to interact with the environment and obtains the returned reward value. Q(s,a) represents the expectation of future rewards for taking action a in state s at a certain time step. The main idea of ​​the Q-value-based reinforcement learning method is to select the action that can obtain the maximum reward based on the Q value. The Q value is updated as follows: Y t Q =R t+1 +γargmaxQ(s t+1 ,a) DQN gradually approaches the optimal Q value through the online Q network Q(s,a,θ). The agent observes the current state s in the environment and selects an action a from the online Q network Q(s,a,θ) according to the ε-greedy strategy, that is: Execute action a, obtain immediate reward r and new state s' from the environment, store the experience sample (s, a, r, s') of taking action a in state s to get reward r and transfer to the next state s' into the experience replay pool D, and sample a small batch (s j ,a j ,rj,sj') for training, and use the target Q network to calculate the target Q value Y for each state action j The calculation process is expressed as Use online Q network Q(s j ,a j ,θ) calculate the predicted Q value and the target Q value Y j For comparison, calculate the loss function: Using the back-propagation algorithm, the parameters θ of the online Q network are updated by gradient descent based on the loss function L(θ), and after a certain number of iterations, the parameters θ of the online Q network are synchronized to the parameters θ' of the target Q network. The D3QN algorithm combines the improvements of Double DQN and Dueling DQN on the DQN algorithm, and uses an online Q network to select the optimal action α'=arg max in the next state s' a' Q(s',a';θ), uses the target network Q(s',a';θ') to evaluate the Q value of the selected optimal action a', whose Q value is estimated by a combination of state value and advantage function. Through this combination, D3QN avoids the overestimation problem of Q value in traditional DQN when estimating Q value, and effectively improves the learning efficiency in complex states.

7. The multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning according to claim 6 is characterized by: In step 5, the graph convolution feature is used as the current environment state s observed by the aircraft at time step t t , the agent selects the aircraft in the current flight state s t The best action a under t And take action according to the current strategy π(a|s) where π(a|s) = Pr(a t =a|s t =s) indicates that the agent is in state s t Next select action a t The probability that the environment returns a reward value r after executing the action t =R(s t ,a t ), and according to the state transfer function p(s'|s,a)=Pr[s t+1 =s',r t+1 =r|s t =s,a t =a], s∈S′, a∈A, s′∈S′ transfers to the next state s′. The state transfer function indicates that in state s t Next, perform action a t After that, the probability of transferring to the next state s'. Update the action-state value function Q(s,a) based on the Bellman expectation equation so that Q(s t ,a t ) is closer to the actual cumulative return, expressed as: Among them, γ is the discount factor representing the weight of future expectations, represents the maximum Q value that can be obtained by selecting the optimal action a' in the next state s', and α represents the learning rate. After updating the Q-value, the agent updates its policy based on the new Q-value, typically choosing the action that maximizes the Q-value at each state. By updating the strategy, it is ensured that in each state s, the agent will choose the action a that can bring the maximum expected reward. The agent repeats the above process in multiple time steps, constantly interacting with the environment, sensing the state, selecting actions, obtaining rewards, and updating the Q value and strategy until the Q value gradually converges. The goal of reinforcement learning is to maximize the cumulative reward of the agent throughout the process, the cumulative reward G t represents the discounted sum of all rewards from time step t to the future: Finally, the agent continues to learn to maximize the cumulative reward G t .

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning as described in any one of claims 1 to 7 above is implemented.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instruction is executed by the processor, the multi-machine real-time three-dimensional conflict resolution method based on graph reinforcement learning as described in any one of claims 1-7 is implemented.

Images