Approach conflict dynamic resolution method based on multi-head attention and double-layer reinforcement learning

Abstract

The application discloses an approach conflict dynamic resolution method based on multi-head attention and double-layer reinforcement learning, which comprises the following steps: constructing an airspace perception state space model to obtain a flight state vector of an aircraft, and jointly constructing a perception state vector; calculating spatial attention weights by using a scaled dot-product self-attention mechanism on the perception state vector, and obtaining time-space enhanced features by encoding; constructing an approach conflict dynamic resolution model, wherein an upper-layer decision network uses the time-space enhanced features to predict and deduce in the airspace in a forward-looking manner, and outputs a conflict prediction time limit; a mode switching mechanism module uses a mode standard database to judge the conflict prediction time limit, and inputs a dangerous avoidance mode into a lower-layer tactical action control module; and the lower-layer tactical action control module uses a Dueling architecture network and a C51 distributed reinforcement learning to process and output an optimal discrete avoidance maneuver action set. The application realizes adaptive switching of a hybrid control mode and emergency avoidance processing, and improves the approach conflict resolution efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent air control technology, and in particular to a method for dynamic resolution of approach conflicts based on multi-head attention and two-layer reinforcement learning. Background Technology

[0002] The approach phase of the airport terminal area (TRACON) is the core convergence airspace where aircraft transition from high-altitude cruise to runway landing. With the continuous increase in civil aviation traffic density, multiple aircraft converge at a high temporal and spatial level, significantly increasing the probability of potential flight conflicts. Efficient and real-time conflict resolution (CR) is crucial for ensuring airspace operational safety. Traditional conflict resolution methods mainly rely on geometric optimization or rule-based optimization; however, when facing complex nonlinear scenarios with high density and heterogeneous multi-aircraft convergence, traditional geometric methods lack look-ahead extrapolation capabilities at the temporal level, easily causing high-frequency oscillations in avoidance trajectories, and making it difficult to balance fuel consumption and flight comfort while ensuring safe separation.

[0003] In recent years, deep reinforcement learning has demonstrated significant advantages in sequential decision-making under uncertain and time-varying environments. However, existing single-stage deep reinforcement learning (DRL) error avoidance methods still have core limitations: First, traditional multilayer perceptron encoders cannot effectively capture the dynamic spatial dependencies between the numerous obstacle aircraft and the aircraft itself in the airspace, easily leading to feature dilution; second, multi-objective imbalance makes it difficult for the model to reconcile the contradiction between long-term safety horizon extrapolation and short-term high-frequency maneuver control, easily resulting in Q-value overestimation bias in the long prediction time domain, making it difficult for the control strategy to converge stably. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic approach conflict resolution method based on multi-head attention and two-layer reinforcement learning. A two-layer heterogeneous reinforcement learning architecture is constructed. The upper-layer decision network builds a strategic forward prediction network based on dual-delay deterministic policy gradient (TD3) and adopts a dual-independent TwinCritics evaluation network and a target policy smoothing mechanism to output a continuous conflict prediction time horizon. The lower-layer tactical action control module uses a Dueling architecture network and C51 distributed reinforcement learning to process and output the optimal discrete evasion maneuver action set, which can effectively decouple long-term strategic simulation and short-term tactical maneuver.

[0005] The objective of this invention is achieved through the following technical solution: A dynamic approach conflict resolution method based on multi-head attention and two-layer reinforcement learning, the method comprising: S1. Construct an airspace perception state space model to obtain time t, study the flight state vector of the aircraft and the flight state vector of the intruding aircraft in the aircraft defense sector, and jointly construct the perception state vector. S2. Calculate spatial attention weights from the perceptual state vector using a scaled dot product self-attention mechanism and encode them to obtain spatiotemporal augmented features. ; S3. Construct an approach conflict dynamic resolution model that includes an upper-level decision network, a mode switching mechanism module, and a lower-level tactical action control module. The upper-level decision network utilizes spatiotemporal augmentation features. Proactive forecasting and simulation within the airspace, and output of conflict prediction time horizons. The mode switching mechanism module internally stores a mode standard database containing nominal approach modes and hazard avoidance modes. The mode switching mechanism module uses the mode standard database to determine the conflict prediction time boundary. Modal assessment is performed, and the hazard avoidance mode is input into the lower-level tactical action control module; the lower-level tactical action control module receives the spatiotemporal enhancement features corresponding to the hazard avoidance mode. The optimal discrete evasion maneuver set is output by using a Dueling architecture network and C51 distributed reinforcement learning. The optimal discrete evasion maneuver set includes several discrete angle adjustment actions and several speed adjustment actions.

[0006] To better implement the present invention, in method S3, the optimal discrete evasive maneuver action set is a set of actions arranged according to action index. The discrete angle adjustment action includes five heading deflection angles, and the speed adjustment action includes three speed adjustments. The actions are independent composite actions consisting of five heading deflection angles and three speed adjustments forming a 5×3 Cartesian product.

[0007] Preferably, the lower-level tactical action control module uses a Dueling architecture network to obtain the state value stream and action advantage stream, and then outputs the expected probability distribution of 51 support sets of attoms through C51 distributed reinforcement learning and calculates the optimal discrete evasion maneuver actions at each time to form the optimal discrete evasion maneuver action set.

[0008] To better achieve the present invention, the present invention also includes the following methods: S4. The lower-level tactical action control module has a multi-target, multi-modal continuous reward function, the expression of which is as follows: ,in The final value after rewards and penalties. For collision avoidance hard constraint parameters, For near-range potential energy penalty parameters, For heading tracking penalty parameters, For speed stability penalty parameters, The reward parameter for reaching the finish line. , , , , These are the weight parameters.

[0009] Preferably, the flight state vector of the aircraft is studied. The expression is as follows: , To study the distance of an aircraft from the runway landing point, The relative azimuth of the runway. To study the aircraft's current heading angle, To study the current flight speed of the aircraft, To study the aircraft's current flight altitude, To study the maximum permissible speed of an aircraft approaching the airspace; the defense sector of the studied aircraft consists of a quarter-circular sector with a detection radius R1, which is composed of four directional sensor systems deployed in front of, behind, to the left and right of the studied aircraft. The four directional sensor systems detect the intruding aircraft corresponding to the studied aircraft and record the flight state vector of the intruding aircraft.

[0010] Preferably, the upper-level decision network internally constructs a dual-independent TwinCritics evaluation network consisting of two completely independent Critic value networks with no parameter sharing. The objective Q-value function expression of the dual-independent TwinCritics evaluation network is as follows: ,in The final value after rewards and penalties. As a return discount factor, For the i-th Critic value network output, For the spatiotemporal enhancement features at time t+1, For use by upper-level decision-making networks Process the output. To truncate Gaussian perturbation noise, This is the upper bound of the maximum forward-looking prediction duration.

[0011] Preferably, the mode switching mechanism module utilizes conflict prediction time bounds. Calculate the distance between the research aircraft and the intruding aircraft. The mode switching mechanism module has a nominal approach threshold set internally. ,like Then the mode is determined to be the nominal approach mode; if If the mode is determined to be a danger avoidance mode, the mode switching mechanism module will input the danger avoidance mode as an alarm and the corresponding spatiotemporal enhancement features of the danger avoidance mode into the lower-level tactical action control module.

[0012] Preferably, the total number of atoms in the C51 distributed reinforcement learning is 51, and its distributed state value and advantage cascade combination distribution expression are as follows: Among them, atoms Spatiotemporal enhancement features ,action Distributed Q components, For spatiotemporal enhancement features ,atom State value stream, For spatiotemporal enhancement features ,action ,atom Action advantage flow, This represents the total number of motion spaces. For all actions at the atom The mean of the advantage at each location.

[0013] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) This invention utilizes the conflict prediction time boundary solution of the upper-level decision network to calculate the real-time minimum distance through the mode switching mechanism module, and compares it with the safety threshold and critical buffer threshold for mode judgment. The danger avoidance mode is input into the lower-level tactical action control module for emergency avoidance mode processing, realizing the adaptive switching of hybrid control modes and emergency avoidance processing, and improving the efficiency of approach conflict resolution. This invention solves the problems of blind spots in state space dynamic representation, overestimation bias in long-term decision and high-frequency oscillation of geometric speed regulation in multi-machine dense intersection scenarios.

[0014] (2) This invention utilizes a multi-head self-attention mechanism to achieve precise focusing on dynamic threats, thereby improving safety robustness in high-density scenarios. Experimental data shows that in airspace density stress tests where the number of aircraft increases from 10 to 30, the conflict rate obtained by this invention increases at a moderate rate. Even in extreme scenarios with 30 aircraft and high saturation, the conflict rate is still controlled at around 5.30%, which is far superior to the 25.10% malign conflict rate of the Pure-DRL algorithm. Moreover, 75% of the conflict events are minor contacts with a distance greater than 3km, and the occurrence rate of critical collisions with a distance less than 1km is only 1%, thus constructing a solid safety defense line from a macro perspective.

[0015] (3) The present invention constructs a two-layer heterogeneous reinforcement learning architecture. The upper-layer decision network constructs a strategic forward prediction network based on the dual-delay deterministic policy gradient (TD3), and adopts a dual independent TwinCritics evaluation network and a target policy smoothing mechanism to output the continuous conflict prediction time horizon. The lower-layer tactical action control module uses the Dueling architecture network and C51 distributed reinforcement learning to process and output the optimal discrete evasion maneuver action set, which can effectively decouple long-term strategic deduction and short-term tactical maneuver. Attached Figure Description

[0016] Figure 1 This is a flowchart of the approach conflict dynamic resolution method of the present invention; Figure 2 This is a schematic diagram comparing the change in total reward in a single round with the number of conflict rounds in the training example; Figure 3 The above is a schematic diagram comparing the conflict rate distribution, trajectory deviation distribution histogram, and multi-dimensional performance scoring thermal moment in the embodiment. Figure 4 This is a comparative diagram of convergence performance in the examples; Figure 5 This is a comparative schematic diagram of the ablation experiments in the examples; Figure 6 This is a comparison chart showing the results of changes in spatial density, collision rate, and collision severity classification in the embodiments. Figure 7 This is a schematic diagram comparing the AIRG approach conflict dynamic resolution method with other algorithms in various dimensions in the embodiment; Figure 8 The image shows a comparison of the flight paths of the AIRG approach conflict dynamic resolution method and other algorithms after an approach conflict in the embodiment. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to embodiments: Example like Figure 1 As shown, an approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning is presented, the method comprising: S1. Construct an airspace perception state space model to obtain the flight state vector of the aircraft at time t, and study the flight state vector of the intruding aircraft within the aircraft's defense sector, and jointly construct the perception state vector. Study the aircraft's flight state vector. The expression is as follows: , To study the distance of an aircraft from the runway landing point, The relative azimuth of the runway. To study the aircraft's current heading angle, To study the current flight speed of the aircraft, To study the aircraft's current flight altitude, To study the permissible speed limits for aircraft approach airspace (i.e., the maximum permissible speed limits for approach airspace, which will also be used in reward and penalty constraints), the defense sector for the studied aircraft consists of a quarter-circular sector with a detection radius R1, located in front of, behind, to the left and right of the studied aircraft. The four sensor systems detect the intruding aircraft corresponding to the studied aircraft and record the intruding aircraft's flight state vector. The intruding aircraft's flight state vector can also be calculated according to the flight state vector... Similarly, it records data such as relative distance, azimuth, relative heading angle, speed, and altitude. The perception state vector (25-dimensional global perception state space vector) is obtained by concatenating and feature-expanding the flight state vectors of the studied aircraft and the intruding aircraft.

[0018] S2. Calculate spatial attention weights from the perceptual state vector using a scaled dot product self-attention mechanism and encode them to obtain spatiotemporal augmented features. In some embodiments, the perceived state vector Dimensionality enhancement is performed using a two-layer fully connected network (256 neurons per layer) (with LayerNorm feature regularization applied in between) to suppress information dissipation caused by large-scale state mutations. The output feature dimension is [dimensionality missing]. Hidden Tensors Hidden Tensor The cloned components are then multiplied by three tensor transformation weight matrices respectively. By setting up four self-attention heads, each head has an independent sub-feature dimension of 1. Within each head, a matrix dot product operation is performed and divided by... Perform numerical scaling: By The input is fed into a channel-wise Softmax distributed activation layer, the dynamic spatial association self-attention weights are calculated, and then compared with the value matrix. Perform matrix multiplication and aggregation, resulting in the output matrix. After linear projection normalization, the final output is a spatiotemporally enhanced feature that can deeply capture the spatiotemporal dependencies of multiple machines. (The introduction of a self-attention mechanism enables the network to adaptively quantify the threat level of each sector. For example, when there is a rapidly approaching intruder on the right front, the corresponding attention weight increases significantly, thereby guiding subsequent decisions.)

[0019] S3. Construct an approach conflict dynamic resolution model that includes an upper-level decision network, a mode switching mechanism module, and a lower-level tactical action control module. The upper-level decision network utilizes spatiotemporal augmentation features. Proactive forecasting and simulation within the airspace, and output of conflict prediction time horizons. In some embodiments, the upper-layer decision network contains an AttentionActor network employing a 3-layer fully connected architecture (with the number of hidden layer neurons set to 256 and 128), and the last layer uses a Sigmoid activation function to strictly constrain the output to... Within the parameter range, the dynamic mapping outputs the optimal continuous conflict prediction time bound corresponding to the current environmental state. Preferably, the upper-level decision-making network internally comprises two completely independent Critic value networks whose parameters are not shared. and The dual independent TwinCritics evaluation network is constructed as follows: The objective Q-value function expression of the dual independent TwinCritics evaluation network is as follows: , ,in This is the final value after rewards and penalties (with smaller constraints). As a return discount factor, For the i-th Critic value network output, For the spatiotemporal enhancement features at time t+1, For use by upper-level decision-making networks Process the output. To truncate Gaussian perturbation noise, To provide the upper bound for the maximum forward forecast duration, Limit the range of action values. With a mean of 0 and a variance of Standard normal Gaussian distribution (standard deviation in this example) ), This indicates that the noise will be truncated at... Interval (noise truncation in this embodiment) The parameters of the Critic value network are adaptively updated in batches by minimizing the mean squared error loss function (MSELoss). In the formula, This represents the combined total loss of the two Critics. For all trainable parameters of the first-path online Critic network, For all trainable parameters of the second-path online Critic network, This indicates a batch averaging operation. To address the non-uniform importance sampling weights fed back from the priority experience replay pool, the Policy network adopts a delayed update strategy (updating 2 steps per iteration in the main loop and 1 step per iteration in the Policy network). Implement a strong gradient descent strategy to update Actor parameters . For spatiotemporal enhancement features The Q-value of the first-way online Critic network with continuous action over time t. For spatiotemporal enhancement features Q-value of the second-way online Critic network with continuous action over time t.

[0020] The mode switching mechanism module internally stores a mode standard database containing nominal approach modes and hazard avoidance modes. The mode switching mechanism module uses the mode standard database to determine the conflict prediction time bound. Modal assessment is performed, and the danger avoidance mode is input into the lower-level tactical action control module. Preferably, the modal switching mechanism module utilizes the conflict prediction time bound. Calculate the distance between the research aircraft and the intruding aircraft. The mode switching mechanism module has a nominal approach threshold set internally. ,like If the mode is determined to be the nominal approach mode, the mode switching mechanism module issues the geometric control command with action_type='recovery', the lower-level tactical action control module is in silent sleep, and the aircraft directly executes the nominal approach mode. The calculation mechanism aligns the relative azimuth angle of the current runway landing point in real time, and the single-step heading correction angle is strictly truncated within... Within, at the same speed The acceleration smoothly approaches the standard nominal landing speed of 150 m / s². If If the mode is determined to be a danger avoidance mode, the mode switching mechanism module will input the danger avoidance mode as an alarm (triggering a red warning) and the spatiotemporal enhancement features corresponding to the danger avoidance mode into the lower-level tactical action control module, and the lower-level tactical action control module will switch to the emergency error avoidance working state.

[0021] The lower-level tactical action control module receives the spatiotemporal augmentation features corresponding to the danger avoidance mode. The optimal discrete evasion maneuver action set is output using a Dueling architecture network and C51 distributed reinforcement learning. The lower-level tactical action control module uses the Dueling architecture network to obtain the state value stream and action advantage stream, and then uses C51 distributed reinforcement learning to output the expected probability distribution of 51 support sets of atoms and calculates the optimal discrete evasion maneuvers for each time period. In some embodiments, the total number of atoms in the C51 distributed reinforcement learning is 51, and its distributed state value and advantage concatenation combination distribution expression is as follows: Among them, atoms Spatiotemporal enhancement features ,action Distributed Q components, For spatiotemporal enhancement features ,atom State value stream, For spatiotemporal enhancement features ,action ,atom Action advantage flow, This represents the total number of motion spaces. For all actions at the atom The mean of the advantage at each location.

[0022] The optimal discrete evasive maneuver set includes several discrete angle adjustment actions and several speed adjustment actions. Preferably, the optimal discrete evasive maneuver set is a set of actions arranged by action index. The discrete angle adjustment actions include five heading angles, and the speed adjustment actions include three speed adjustments. The actions are independent composite actions consisting of a 5×3 Cartesian product of the five heading angles and the three speed adjustments. Examples of the heading angle levels (i.e., turning and course change decisions) during a discrete evasive maneuver within a certain time period are as follows: The following represent, in sequence: a sharp left turn with a 20° leftward veer on the course; a slight leftward turn; a 10° leftward veer on the course; maintaining course straight with no turning; a slight rightward turn with a 10° rightward veer on the course; and a sharp rightward turn with a 20° rightward veer on the course. Examples of the three speed adjustment levels in discrete avoidance maneuvers are as follows: The values ​​represent, in order: deceleration (flight speed decreases by 10 m / s); cruise at constant speed (speed remains unchanged); and acceleration (flight speed increases by 10 m / s). In some embodiments, the lower-level tactical action control module updates the parameters by minimizing the Kullback-Leibler (KL) divergence between the projected target probability distribution and the currently estimated distribution. Of which is the total network loss. Let the probability of the target being distributed on the j-th atom be denoted as . For spatiotemporal enhancement features ,action The predicted probability. Finally, the expected value of the Q-value is calculated based on the probability distribution corresponding to each action, through... Operator accurately outputs the optimal error avoidance action in a single step. The study investigates how an aircraft updates its physical space coordinates, actual flight speed, and actual yaw angle when performing this discrete maneuver.

[0023] S4. The lower-level tactical action control module has a multi-target, multi-modal continuous reward function, the expression of which is as follows: ,in The final value after rewards and penalties. These are the hard constraint parameters for collision avoidance. Indicates the use of indicator functions Constraints study the minimum distance between the research aircraft and the intrusion aircraft (or the intrusion aircraft). With respect to the preset critical safety distance The parameter is expressed as: , These are preset parameters. Here are the near-range potential energy penalty parameters. The expression is as follows: , To maintain a safe distance, To study the distance from an aircraft to an intrusion aircraft (or an intrusion aircraft), These are preset parameters. For heading tracking penalty parameters, heading tracking penalty parameters The expression is as follows: , The heading angle of the aircraft. The heading angle pointing towards the landing point. These are preset parameters. This refers to the velocity stability penalty parameter. The expression is as follows: , , The aircraft speeds at times t and t-1 are respectively. The nominal cruising speed, and Preset parameters (example in this embodiment) The value is 0.1. The value is 0.05. The endpoint reward parameter. The expression is as follows: ,in A value of 1 indicates successful arrival at the landing area, and a value of 0 indicates otherwise. These are preset parameters. , , , , These are the weight parameters.

[0024] Figure 2To compare the change in total reward per round with the number of conflicts in training rounds, this invention conducted comparative experiments with AIRG (the dynamic approach conflict resolution method of this invention), MVP (Modified Voltage Potential), and Pure-DRL (the traditional single deep reinforcement learning network). This invention showed the highest upper limit of reward convergence, remained stable in the high range, had controllable oscillation amplitude, the lowest global average conflict, and repeatedly dropped to near 0, demonstrating the best long-term risk avoidance stability. Figure 3 Histograms of conflict rate distribution, trajectory deviation distribution, and multi-dimensional performance scoring heatmaps were used to group algorithms into three categories: AIRG (the dynamic approach conflict resolution method of this invention), MVP (Modified Voltage Potential), and Pure-DRL (the traditional single deep reinforcement learning network). AIRG samples were concentrated in the low conflict rate range, followed by MVP, while Pure-DRL samples were largely concentrated in the high collision range. The self-developed hierarchical architecture significantly reduced the probability of high-frequency obstacle collisions. Simultaneously, AIRG exhibited smaller overall trajectory deviations, closely adhering to the planned flight path; Pure-DRL showed frequent large-scale deviations and had the worst maneuverability. Figure 3 The multi-dimensional performance scoring heatmap shows that AIRG leads in all dimensions, with the best overall performance in success rate, collision avoidance, fuel consumption, and computing power. Figure 4 For the convergence performance comparison diagram, this invention uses AIRG (the dynamic approach conflict resolution method of this invention), MVP (Modified Voltage Potential), and Pure-DRL (the traditional single deep reinforcement learning network) for comparative experiments. See [link to relevant documentation]. Figure 4 It is evident that the approach conflict dynamic resolution method of this invention achieves the highest convergence reward, lowest collision rate, highest landing success rate, and lowest network loss. This invention is compared with AIRG (the approach conflict dynamic resolution method of this invention), MVP (Modified Voltage Potential), and Pure-DRL (a traditional single deep reinforcement learning network) in ablation experiments. Figure 5As shown, the approach conflict dynamic resolution method of this invention outperforms the other two algorithms in all three aspects: collision avoidance, safe distance, and mission success rate. The spatial resolution strategy of the AIRG approach conflict dynamic resolution method of this invention is excellent, successfully reducing the dense approach conflict rate to an absolute low of 1.09%, far superior to Pure-DRL's 4.67% and PPO's 3.92%; its approach landing success rate reaches 58.00%. In terms of computational efficiency, the forward computation time of AIRG is stable at 125.00, compared to 450.00 for the traditional geometric MVP method, significantly shortening the computational latency and providing absolute support for high-frequency dynamic control. This invention compares the spatial density, conflict rate changes, and conflict severity levels of AIRG, MVP (Modified Voltage Potential), and Pure-DRL (i.e., the traditional single deep reinforcement learning network). The comparison results are as follows: Figure 6 As shown, with the increase in the number of aircraft, the AIRG approach conflict dynamic resolution method of this invention exhibits a gradual increase in conflict rate, while the baseline algorithm shows an explosive increase in conflict rate. The AIRG approach conflict dynamic resolution method of this invention can avoid obstacles in advance, controlling the danger to a slight approach at a long distance, significantly reducing the probability of high-risk close-range collisions. This invention compares AIRG (the approach conflict dynamic resolution method of this invention), MVP (Modified Voltage Potential), Pure-DRL (i.e., a traditional single deep reinforcement learning network), and other algorithms in terms of collision rate, homing success rate, average safe distance from obstacles, single-step inference time, fuel efficiency, and ComprehensivePerformance radar chart, etc. Figure 7 As shown, the AIRG approach conflict dynamic resolution method of this invention has the lowest collision rate and the highest homing success rate, and its autonomous risk avoidance and route planning capabilities are comprehensively superior to all baselines; at the same time, flight track comparisons are performed, such as... Figure 8 As shown, the approach conflict dynamic resolution method of the present invention predicts the timing of the conflict in advance at the upper level and smoothly avoids the danger with a small margin in advance, while the lower level finely adjusts the course / speed, so that there is no need to make an emergency large-angle detour at close range.

[0025] In the airspace stress topological elasticity verification of this invention, under extreme stress saturation tests with 10 to 30 heterogeneous aircraft, Pure-DRL exhibited characteristic disintegration at high density, with the conflict rate rapidly exceeding the yellow warning zone and entering the red collapse danger zone, soaring to 25.10% at 30 aircraft. In contrast, the approach conflict dynamic resolution method AIRG of this invention, thanks to self-attention encoding, exhibits an extremely smooth conflict rate curve evolution, maintaining a conflict rate of 5.30% even under saturated traffic flow of 30 aircraft. Moreover, in the very few near misses that occurred, 75% of AIRG events were minor contacts with a distance greater than 3km, while the incidence of critical collisions (less than 1km) was only 1%, demonstrating extremely high macroscopic safety. In the convergence performance and strategy evolution verification, as shown in the convergence characteristic evolution curve, the cumulative reward and success rate curves of the approach conflict dynamic resolution method AIRG of this invention exhibit a clear convergence plateau after approximately 200 episodes. Its training residual (Loss) on the logarithmic coordinate axis exhibits the steepest descent slope, and there is absolutely no cliff-like divergence or multi-objective oscillation common in conventional policy gradient algorithms (such as PPO) in the later stages, demonstrating the extremely high stability of the policy. During the in-depth verification of the control mechanism in the ablation experiment, in order to quantitatively deconstruct the empowerment weights of each core scientific module, a key component stripping ablation experiment was performed: the conflict rate of the AIRG model, the complete approach conflict dynamic resolution method of this invention, was locked at 1.1%, the approach success rate remained at 58.0%, the training time was 45.0 minutes, and the policy convergence speed was 380 episodes. Stripping the multi-head self-attention mechanism (w / oAttention): the conflict rate explosively deteriorated to 8.3%, evolving into the worst safety baseline of the entire ablation group. However, the approach success rate recorded by the system abnormally rose to the highest of the entire group at 76.0%. This abnormal academic phenomenon reveals an extremely important control mechanism: due to the loss of the spatial focusing ability of the attention mechanism on the high-dimensional features of potential intercepting aircraft in multiple sectors, the agent exhibited a serious tendency to blindly rush forward. In pursuit of the maximum final-state reward at the runway finish line, it chose to ignore airspace hazards and blindly taxi forward. While this resulted in a higher absolute number of samples reaching the finish line (a success rate as high as 76%), the evasive trajectory, while penetrating multiple obstacle aircraft, also caused an unacceptable 8.3% major mid-air collision rate. This fully establishes that the self-attention encoder is the absolute scientific cornerstone for constructing a multi-aircraft error avoidance spatial defense line. Removing the long-term Look-Ahead predictor (w / oLook-Ahead): Due to the loss of long-term strategic guidance, the strategy became blindly short-sighted, severely dragging down the convergence speed to the slowest in the entire group (450 episodes), increasing the conflict rate to 3.2%, and decreasing the success rate to 52.0%.Stripping the trajectory recovery network: Due to the loss of geometric rules constraining long-distance straight flight, the aircraft was prone to heading divergence after error avoidance, and the success rate dropped to the lowest in the group at 48.0%, establishing the necessity of a dual-loop collaborative mechanism of "reinforcement learning error avoidance + geometric rule recovery".

[0026] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamic resolution of approach conflicts based on multi-head attention and two-layer reinforcement learning, characterized in that: The methods include: S1. Construct an airspace perception state space model to obtain time t, study the flight state vector of the aircraft and the flight state vector of the intruding aircraft in the aircraft defense sector, and jointly construct the perception state vector. S2. Calculate spatial attention weights from the perceptual state vector using a scaled dot product self-attention mechanism and encode them to obtain spatiotemporal augmented features. ; S3. Construct an approach conflict dynamic resolution model that includes an upper-level decision network, a mode switching mechanism module, and a lower-level tactical action control module. The upper-level decision network utilizes spatiotemporal augmentation features. Proactive forecasting and simulation within the airspace, and output of conflict prediction time horizons. The mode switching mechanism module internally stores a mode standard database containing nominal approach modes and hazard avoidance modes. The mode switching mechanism module uses the mode standard database to determine the conflict prediction time boundary. Modal assessment is performed, and the hazard avoidance mode is input into the lower-level tactical action control module; the lower-level tactical action control module receives the spatiotemporal enhancement features corresponding to the hazard avoidance mode. The optimal discrete evasion maneuver set is output by using a Dueling architecture network and C51 distributed reinforcement learning. The optimal discrete evasion maneuver set includes several discrete angle adjustment actions and several speed adjustment actions.

2. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 1, characterized in that: In method S3, the optimal discrete evasive maneuver action set is a set of actions arranged according to action index. The discrete angle adjustment action includes five heading deflection angles, and the speed adjustment action includes three speed adjustments. The actions are independent composite actions consisting of five heading deflection angles and three speed adjustments forming a 5×3 Cartesian product.

3. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 2, characterized in that: The lower-level tactical action control module uses a Dueling architecture network to obtain the state value stream and action advantage stream. Then, it outputs the expected probability distribution of 51 support sets of atmospheres through C51 distributed reinforcement learning and calculates the optimal discrete evasion maneuver actions at each time, forming the optimal discrete evasion maneuver action set.

4. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 1 or 2, characterized in that: It also includes the following methods: S4. The lower-level tactical action control module has a multi-target, multi-modal continuous reward function, the expression of which is as follows: ,in The final value after rewards and penalties. For collision avoidance hard constraint parameters, For near-range potential energy penalty parameters, For heading tracking penalty parameters, For speed stability penalty parameters, The reward parameter for reaching the finish line. , , , , These are the weight parameters.

5. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 1, characterized in that: Study the flight state vector of an aircraft The expression is as follows: , To study the distance of an aircraft from the runway landing point, The relative azimuth of the runway. To study the aircraft's current heading angle, To study the current flight speed of the aircraft, To study the aircraft's current flight altitude, To study the maximum permissible speed of an aircraft approaching the airspace; the defense sector of the studied aircraft consists of a quarter-circular sector with a detection radius R1, which is composed of four directional sensor systems deployed in front of, behind, to the left and right of the studied aircraft. The four directional sensor systems detect the intruding aircraft corresponding to the studied aircraft and record the flight state vector of the intruding aircraft.

6. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 1, characterized in that: The upper-level decision network internally constructs a dual-independent TwinCritics evaluation network, consisting of two completely independent Critic value networks with no parameter sharing. The objective Q-value function expression of the dual-independent TwinCritics evaluation network is as follows: ,in The final value after rewards and penalties. As a return discount factor, For the i-th Critic value network output, For the spatiotemporal enhancement features at time t+1, For use by upper-level decision-making networks Process the output. To truncate Gaussian perturbation noise, This represents the upper bound of the maximum forward-looking prediction duration.

7. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 1, characterized in that: The modality switching mechanism module utilizes conflict prediction time bounds. Calculate the distance between the research aircraft and the intruding aircraft. The mode switching mechanism module has a nominal approach threshold set internally. ,like Then the mode is judged as the nominal approach mode; if If the mode is determined to be a danger avoidance mode, the mode switching mechanism module will input the danger avoidance mode as an alarm and the corresponding spatiotemporal enhancement features of the danger avoidance mode into the lower-level tactical action control module.

8. The approach conflict dynamic resolution method based on multi-head attention and two-layer reinforcement learning according to claim 3, characterized in that: The total number of atoms in the C51 distributed reinforcement learning is 51, and its distributed state value and advantage cascade combination distribution expression are as follows: Among them, atoms Spatiotemporal enhancement features ,action Distributed Q components, For spatiotemporal enhancement features ,atom State value stream, For spatiotemporal enhancement features ,action ,atom Action advantage flow, This represents the total number of motion spaces. For all actions at the atom The mean of dominance at the location; The lower-level tactical action control module uses a Dueling architecture network to obtain the state value stream and action advantage stream. Then, it outputs the expected probability distribution of 51 support set Atoms through C51 distributed reinforcement learning and solves the optimal discrete evasion maneuver actions at each time to form the optimal discrete evasion maneuver action set. S1. Construct an airspace perception state space model to obtain time t, study the flight state vector of the aircraft and the flight state vector of the intruding aircraft in the aircraft defense sector, and jointly construct the perception state vector. S2. Calculate spatial attention weights from the perceptual state vector using a scaled dot product self-attention mechanism and encode them to obtain spatiotemporal augmented features. ; S3. Construct an approach conflict dynamic resolution model that includes an upper-level decision network, a mode switching mechanism module, and a lower-level tactical action control module. The upper-level decision network utilizes spatiotemporal augmentation features. Proactive forecasting and simulation within the airspace, and output of conflict prediction time horizons. The mode switching mechanism module internally stores a mode standard database containing nominal approach modes and hazard avoidance modes. The mode switching mechanism module uses the mode standard database to determine the conflict prediction time boundary. Modal assessment is performed, and the hazard avoidance mode is input into the lower-level tactical action control module; the lower-level tactical action control module receives the spatiotemporal enhancement features corresponding to the hazard avoidance mode. The optimal discrete evasion maneuver set is output by using a Dueling architecture network and C51 distributed reinforcement learning. The optimal discrete evasion maneuver set includes several discrete angle adjustment actions and several speed adjustment actions.

Images