Low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints

Abstract

The present invention discloses a low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints, which belongs to the field of aviation safety technology and includes the following steps: setting static obstacles and non-cooperative target aircraft to construct a low-altitude operation environment; constructing the state space and action space of the aircraft in the low-altitude operation environment; constructing a reward function with time constraints; based on the reward function with time constraints, the state space and action space of the aircraft in the low-altitude operation environment, low-altitude aircraft conflict resolution training is performed in the low-altitude operation environment based on a deep reinforcement learning model to obtain a trained deep reinforcement learning model; obtaining the current state of the main aircraft in performing an actual conflict resolution task, and using the trained deep reinforcement learning model to output the corresponding action when the reward value is the highest, so as to avoid conflicts while reaching the target position in time. The present invention solves the problem of insufficient conflict resolution capability and punctuality of low-altitude aircraft.

Description

Technical Field

[0001] The present invention belongs to the field of aviation safety technology, and in particular relates to a low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints. Background Art

[0002] With the rapid development of drone technology, drones have been widely used in logistics distribution, patrol monitoring, environmental monitoring, rescue and other fields. However, with the increasing number of drones and the complex and changeable operating conditions in urban environments, the air conflict problem faced by low-altitude drones is becoming more and more serious.

[0003] Conflict resolution plays a key role in aviation safety operations, especially in high-density low-altitude unmanned operations. Low-altitude unmanned operations refer to the simultaneous transfer of people or objects by a large number of unmanned aircraft in a small urban airspace. High density reduces the interval between aircraft and the reaction time for conflict resolution, and increases the mutual impact between multiple unmanned aircraft.

[0004] Tactical conflict resolution refers to real-time monitoring of the relative position of the drone and its surroundings during the flight of the drone, and dynamically adjusting the flight path to avoid possible conflicts. The traditional method of tactical conflict resolution for aircraft is to resolve conflicts through communication and decision-making between controllers and pilots, but this method of communication and decision-making is too inefficient and cannot be applied to low-altitude transportation scenarios. The existing method of conflict resolution based on reinforcement learning focuses on maximizing the success rate of avoiding collisions, but does not consider the time constraints of reaching the target location, and the on-time arrival rate is very low. If the aircraft cannot reach the target location on time, it will affect the operation plan scheduling, and will have a secondary impact on other aircraft operating in the air, greatly increasing the complexity of airspace management. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints, which realizes tactical conflict resolution between the main aircraft and static obstacles and non-cooperative target aircraft in low-altitude and high-density operation scenarios, thereby solving the problems of insufficient conflict resolution capability and punctuality of low-altitude aircraft.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] The present invention provides a low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints, comprising the following steps:

[0008] S1. Set up static obstacles and non-cooperative target aircraft to create a low-altitude operating environment;

[0009] S2. Construct the state space and action space of the aircraft in the low-altitude operation environment;

[0010] S3. Construct a reward function with time constraints based on the distance between the own aircraft and the static obstacle or non-cooperative target aircraft, and the punctual time for the own aircraft to arrive at the target location;

[0011] S4. According to the reward function with time constraints, the state space and action space of the aircraft in the low-altitude operation environment, low-altitude aircraft conflict resolution training is performed in the low-altitude operation environment based on the deep reinforcement learning model to obtain a trained deep reinforcement learning model;

[0012] S5. Obtain the current state of the main aircraft in the actual conflict resolution mission, and use the trained deep reinforcement learning model to output the corresponding action when the reward value is the highest, so as to avoid conflict and reach the target position in time.

[0013] The beneficial effects of the present invention are as follows: a low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints provided by the present invention provides an intelligent agent simulation action basis for aircraft conflict resolution training by constructing a low-altitude operating environment, as well as the state space and action space of the aircraft; by constructing a reward function with time constraints, it provides a basis for training aircraft to reach the target position in a timely and accurate manner; conflict resolution training for aircraft based on the deep reinforcement learning model improves the efficiency and accuracy of aircraft conflict resolution training, and also achieves a significant increase in the punctuality rate of arriving at the target position when the aircraft effectively avoids obstacles.

[0014] Furthermore, the S2 comprises the following steps:

[0015] S21. Divide the detection area into six sector-shaped detection sub-areas based on the detection range of the aircraft and centering on the aircraft itself;

[0016] S22, selecting information of the static obstacle or non-cooperative target aircraft closest to the own aircraft in each sector detection sub-area as state input, and obtaining the state space of the aircraft;

[0017] The calculation expression of the state space is as follows:

[0018] ,

[0019] Where S represents the state space of the aircraft, Indicates the target location, Indicates the position of the own aircraft. Indicates the speed of the own aircraft. represents the relative position of the static obstacle or non-cooperative target aircraft closest to the own aircraft in the i-th sector detection sub-area, represents the relative speed of the static obstacle or non-cooperative target aircraft closest to the own aircraft in the i-th sector detection sub-area, where i=1,2,3,4,5,6;

[0020] S23. Constructing the action space of the aircraft according to the aircraft tactical conflict resolution task;

[0021] The calculation expression of the action space is as follows:

[0022] ,

[0023] Where A represents the action space of the aircraft, represents the change in the aircraft's heading angle, Indicates the change in aircraft speed.

[0024] The beneficial effect of adopting the above-mentioned further scheme is as follows: the present invention provides a state space and action space for constructing an aircraft in a low-altitude operating environment, constrains the state changes and action changes of the aircraft when performing conflict resolution mission training in a low-altitude operating environment, ensures the stability of the low-altitude operating environment, and provides an action constraint basis for the aircraft to avoid obstacles and arrive at the target position on time.

[0025] Furthermore, S3 comprises the following steps:

[0026] S31. Constructing conflict resolution rewards based on the distance relationship between the target aircraft and the static obstacle or non-cooperative target aircraft;

[0027] The calculation expression of the conflict resolution reward is as follows:

[0028] ,

[0029] ,

[0030] ,

[0031] in, Reward for conflict resolution. Represents the distance change reward, Indicates the closest reward. Indicates the distance change reward value, Indicates the change in distance between the own aircraft and a static obstacle or non-cooperative target aircraft. Indicates the distance between the aircraft and the static obstacle or non-cooperative target aircraft at the current moment. Indicates the distance between the aircraft and the static obstacle or non-cooperative target aircraft at the last moment. represents the nearest reward scaling factor, log represents the logarithmic operation, Indicates the shortest distance between the own aircraft and a static obstacle or non-cooperative target aircraft. Indicates the detection range of the aircraft;

[0032] S32. constructing an on-time constraint reward according to the on-time and actual time of the aircraft arriving at the target location;

[0033] The calculation expression of the time constraint reward is as follows:

[0034] ,

[0035] in, represents the time-constrained reward, represents the time constraint reward scaling factor, e represents the exponential base constant, T represents the actual time when the own aircraft reaches the target position, Indicates the on-time time when the aircraft arrives at the target location. represents the reward width distribution control factor;

[0036] S33, obtaining a reward function with a time constraint based on the conflict resolution reward and the time constraint reward;

[0037] The calculation expression of the reward function with time constraint is as follows:

[0038] ,

[0039] Among them, R represents the reward function with time constraints.

[0040] The beneficial effect of adopting the above further scheme is as follows: the present invention constructs a reward function with time constraints based on the distance between the own aircraft and the static obstacle or non-cooperative target aircraft, and the punctual time for the own aircraft to arrive at the target location. Based on the reward function with time constraints, it can provide a basis for effectively improving the punctuality rate of aircraft arriving at the target location after conflict resolution training.

[0041] Furthermore, the S4 comprises the following steps:

[0042] S41. Build an experience replay pool for storing operation data;

[0043] S42, setting model training parameters;

[0044] S43, constructing a deep reinforcement learning model based on Rainbow DQN, wherein the deep reinforcement learning model includes a first neural network for predicting the action of the main drone at the next moment and a second neural network for updating the target Q value;

[0045] S44. Connect the deep reinforcement learning model to the low-altitude operation environment and the experience replay pool;

[0046] S45, according to the state space and action space of the aircraft in the low-altitude operation environment, the reward function with time constraints and the second neural network, the aircraft of the aircraft is used as an intelligent agent to repeatedly perform conflict resolution training in the low-altitude operation environment, and the operation data and target Q value corresponding to each conflict resolution training are obtained;

[0047] S46, synchronizing the network parameters of the first neural network to the second neural network;

[0048] S47. According to the reward function with time constraints, the actions of the main UAV during the conflict training process are optimized through the trained first neural network, so that the target Q value updated by the second neural network continues to increase until it stabilizes and approaches the expected target Q value, thereby obtaining a trained deep reinforcement learning model.

[0049] The beneficial effect of adopting the above-mentioned further scheme is as follows: the present invention adopts deep reinforcement learning combined with a time-constrained method to conduct conflict resolution training for low-altitude aircraft, which can enable the trained aircraft to efficiently complete tactical conflict resolution in a high-density environment. At the same time, it can also effectively improve the punctuality rate of aviation after conflict resolution.

[0050] Furthermore, the S45 comprises the following steps:

[0051] S451, setting a training number threshold for conflict resolution training and a training duration for a single conflict resolution training;

[0052] S452, setting the main body aircraft as an intelligent body at any position in the low-altitude operation environment, and using the position as the initial position during the current conflict resolution training, and initializing the state of the main body UAV;

[0053] S453. According to the state space and action space of the aircraft in the low-altitude operation environment, based on the state and action of the main UAV at the current moment, the state of the main UAV at the next moment is obtained, and according to the reward function with time constraints and the second neural network, the reward value of the main UAV at the current moment and the target Q value corresponding to the state of the main UAV at the next moment are obtained, until the main UAV reaches the target position or reaches the training time of the single conflict resolution training, and the single conflict resolution training is completed;

[0054] S454, after each round of conflict resolution training, the operation data at each moment in the single conflict resolution training is stored in the experience replay pool, and a number of operation data are randomly extracted from the experience replay pool, and the first neural network is trained with the goal of minimizing the error between the predicted output and the actual output of the first neural network in combination with the target Q value at the corresponding moment of each operation data;

[0055] S455. Determine whether the number of conflict resolution trainings performed by the main UAV has reached the training number threshold. If so, proceed to S46; otherwise, return to S452.

[0056] The beneficial effect of adopting the above further scheme is: the present invention provides a conflict resolution training method for the main body UAV based on deep reinforcement learning combined with a reward function with time constraints, and makes decisions through the Rainbow DQN reinforcement learning model, which greatly improves the ability to stably and effectively avoid obstacles and arrive at the target location on time.

[0057] Furthermore, the operating data includes the action, state, corresponding reward value and state of the main drone at the next moment, wherein the intelligent agent obtains the corresponding reward value at the current moment and the state of the intelligent agent at the next moment after completing the action at the current moment based on the state and action at the current moment.

[0058] The beneficial effect of adopting the above further scheme is: the present invention provides a specific composition of operating data, and the operating data reflects the obstacle avoidance behavior of the aircraft as an intelligent entity in a low-altitude operating environment, which can provide a basis for training deep reinforcement learning models to improve the aircraft's obstacle avoidance capability and punctuality rate.

[0059] Other advantages of the present invention will be analyzed in more detail in subsequent embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 The present invention is a flowchart of the steps of a method for resolving low-altitude aircraft conflicts based on deep reinforcement learning and time constraints in an embodiment of the present invention.

[0062] Figure 2 Schematic diagram of a low-altitude operating environment in an embodiment of the present invention.

[0063] Figure 3This is a schematic diagram of adding a reward function with a time constraint and an aircraft punctuality rate without a time constraint in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0065] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides a low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints, comprising the following steps:

[0066] S1. Set up static obstacles and non-cooperative target aircraft to create a low-altitude operating environment;

[0067] Low-altitude tactical conflicts mainly include collisions with static obstacles such as tall buildings, and collisions with dynamic non-cooperative target aircraft. Therefore, the low-altitude operation environment mainly considers adding multiple static obstacles and non-cooperative target aircraft, among which each aircraft in the low-altitude operation environment has a kinematic model that meets actual constraints.

[0068] like Figure 2 As shown, in this embodiment, the low-altitude operating environment includes 3 rectangular static obstacles and 9 non-cooperative target aircraft flying at a constant speed. The rectangular ones in the figure are static obstacles used to simulate buildings such as high-rise buildings. The solid circle in the figure is the aircraft itself, and the large dotted circle outside the aircraft itself is the range of detected obstacles and cooperative targets. If the detection distance of the aircraft is 80m, the aircraft can sense and detect static obstacles and dynamic non-cooperative target aircraft within 80m around it; in this embodiment, the update frequency of the aircraft for detection and action is set to once every 1s.

[0069] S2. Construct the state space and action space of the aircraft in the low-altitude operation environment;

[0070] The S2 comprises the following steps:

[0071] S21. Divide the detection area into six sector-shaped detection sub-areas based on the detection range of the aircraft and centering on the aircraft itself;

[0072] The state space is an important part of the reinforcement learning algorithm, which represents the characteristics of the intelligent agent model and the environment. For the aircraft tactical conflict resolution task, the state space designed in this embodiment includes the position and speed of the aircraft itself, as well as the position and speed of static obstacles and non-cooperative target aircraft within the detection distance. Since the number of static obstacles and non-cooperative target aircraft in the detection area is not fixed, the detection area is divided into 6 fan-shaped detection sub-areas to construct the state space of the aircraft;

[0073] S22, selecting information of the static obstacle or non-cooperative target aircraft closest to the own aircraft in each sector detection sub-area as state input, and obtaining the state space of the aircraft;

[0074] The calculation expression of the state space is as follows:

[0075] ,

[0076] Where S represents the state space of the aircraft, Indicates the target location, Indicates the position of the own aircraft. Indicates the speed of the own aircraft. represents the relative position of the static obstacle or non-cooperative target aircraft closest to the own aircraft in the i-th sector detection sub-area, represents the relative speed of the static obstacle or non-cooperative target aircraft closest to the own aircraft in the i-th sector detection sub-area, where i=1,2,3,4,5,6; in this embodiment, if there is no static obstacle or non-cooperative target aircraft in a sector detection sub-area, the corresponding sector detection sub-area and Set to 0;

[0077] S23. Constructing the action space of the aircraft according to the aircraft tactical conflict resolution task;

[0078] The calculation expression of the action space is as follows:

[0079] ,

[0080] Where A represents the action space of the aircraft, represents the change in the aircraft's heading angle, In this embodiment, the value range of the aircraft's heading angle change is The value of the aircraft speed change is .

[0081] S3. Construct a reward function with time constraints based on the distance between the own aircraft and the static obstacle or non-cooperative target aircraft, and the punctual time for the own aircraft to arrive at the target location;

[0082] The S3 comprises the following steps:

[0083] S31. Constructing conflict resolution rewards based on the distance relationship between the target aircraft and the static obstacle or non-cooperative target aircraft;

[0084] The calculation expression of the conflict resolution reward is as follows:

[0085] ,

[0086] ,

[0087] ,

[0088] in, Reward for conflict resolution. Represents the distance change reward, R 12 Indicates the closest reward. Indicates the distance change reward value, Indicates the change in distance between the own aircraft and a static obstacle or non-cooperative target aircraft. Indicates the distance between the aircraft and the static obstacle or non-cooperative target aircraft at the current moment. Indicates the distance between the aircraft and the static obstacle or non-cooperative target aircraft at the last moment. represents the nearest reward scaling factor, log represents the logarithmic operation, Indicates the shortest distance between the own aircraft and a static obstacle or non-cooperative target aircraft. Indicates the detection range of the aircraft;

[0089] In this embodiment, the distance change reward reflects the distance change trend between the own aircraft and the static obstacle or non-cooperative target aircraft. If the distance increases, a positive distance change reward value is obtained, and if the distance decreases, a negative distance change reward value is obtained; the closest distance reward reflects the distance relationship between the own aircraft and the nearest static obstacle or non-cooperative target aircraft. If the distance is greater than the detection distance of the aircraft, that is, in this embodiment, when the distance is greater than 80m, the closest distance reward value is positive, if the distance is less than 80m, then the closest distance reward value is negative, and as the distance decreases, the closest distance reward value will decrease rapidly.

[0090] S32. constructing an on-time constraint reward according to the on-time and actual time of the aircraft arriving at the target location;

[0091] The calculation expression of the time constraint reward is as follows:

[0092] ,

[0093] in, represents the time-constrained reward, represents the time constraint reward scaling factor, e represents the exponential base constant, T represents the actual time when the own aircraft reaches the target position, Indicates the on-time time when the aircraft arrives at the target location. represents the reward width distribution control factor;

[0094] In this embodiment, the reward scaling factor can directly adjust the size of the maximum value of the time-constrained reward; when the reward width distribution control factor controls the width of the time-constrained reward distribution, if the reward distribution control factor is smaller, the actual time for the aircraft to arrive at the target location must be closer to the punctual time in order to obtain a higher reward.

[0095] S33, obtaining a reward function with a time constraint based on the conflict resolution reward and the time constraint reward;

[0096] The calculation expression of the reward function with time constraint is as follows:

[0097] ,

[0098] Among them, R represents the reward function with time constraints.

[0099] S4. According to the reward function with time constraints, the state space and action space of the aircraft in the low-altitude operation environment, low-altitude aircraft conflict resolution training is performed in the low-altitude operation environment based on the deep reinforcement learning model to obtain a trained deep reinforcement learning model;

[0100] The S4 comprises the following steps:

[0101] S41. Build an experience replay pool for storing operation data;

[0102] In this embodiment, the experience replay pool enables the neural network model in deep reinforcement learning training to achieve rapid learning convergence based on a large amount of historical operation data; during the deep reinforcement learning training process, the experience replay pool will continue to expand and iterate, so that the deep reinforcement learning model can obtain comprehensive and valuable experience data.

[0103] S42, setting model training parameters;

[0104] In this embodiment, setting the model training parameters includes setting the learning rate to 0.00005, the discount factor to 0.99, the experience replay pool size to 1000000, the batch size to 256, the multi-step update to 5, the current network update delay to 10 steps, the target network update to update after each round of training, the total number of training rounds to 5000, and the loss function to the mean square error function.

[0105] S43, constructing a deep reinforcement learning model based on Rainbow DQN, wherein the deep reinforcement learning model includes a first neural network for predicting the action of the main drone at the next moment and a second neural network for updating the target Q value;

[0106] In this embodiment, Rainbow DQN integrates six improved deep reinforcement learning methods based on the deep Q network (DQN). The six improved methods include Double Q-learning, priority replay, Dueling network, multi-step learning, distributed RL and Noisy Nets. The improved methods can solve the problems of over-estimation of Q value, processing high-dimensional state space and low exploration efficiency of the deep Q network (DQN). Rainbow DQN is a model-free, off-policy, value-based and discrete reinforcement learning model. Based on the above six improved methods, it has significant learning efficiency, algorithm stability and generalization ability.

[0107] S44. Connect the deep reinforcement learning model to the low-altitude operation environment and the experience replay pool;

[0108] S45, according to the state space and action space of the aircraft in the low-altitude operation environment, the reward function with time constraints and the second neural network, the aircraft of the aircraft is used as an intelligent agent to repeatedly perform conflict resolution training in the low-altitude operation environment, and the operation data and target Q value corresponding to each conflict resolution training are obtained;

[0109] The S45 comprises the following steps:

[0110] S451, setting a training number threshold for conflict resolution training and a training duration for a single conflict resolution training;

[0111] S452, setting the main body aircraft as an intelligent body at any position in the low-altitude operation environment, and using the position as the initial position during the current conflict resolution training, and initializing the state of the main body UAV;

[0112] S453. According to the state space and action space of the aircraft in the low-altitude operation environment, based on the state and action of the main UAV at the current moment, the state of the main UAV at the next moment is obtained, and according to the reward function with time constraints and the second neural network, the reward value of the main UAV at the current moment and the target Q value corresponding to the state of the main UAV at the next moment are obtained, until the main UAV reaches the target position or reaches the training time of the single conflict resolution training, and the single conflict resolution training is completed;

[0113] S454, after each round of conflict resolution training, the operation data at each moment in the single conflict resolution training is stored in the experience replay pool, and a number of operation data are randomly extracted from the experience replay pool, and the first neural network is trained with the goal of minimizing the error between the predicted output and the actual output of the first neural network in combination with the target Q value at the corresponding moment of each operation data;

[0114] The operation data includes the action, state, corresponding reward value and state of the main drone at the next moment, wherein the intelligent agent obtains the corresponding reward value at the current moment and the state of the intelligent agent at the next moment after completing the action at the current moment based on the state and action at the current moment.

[0115] S455, determining whether the number of conflict resolution trainings performed by the main UAV has reached the training number threshold, if so, proceeding to S46, otherwise returning to S452;

[0116] In this embodiment, when training the first neural network, the state of the drone at the current moment in the operating data is used as the input of the first neural network, and the predicted action of the main drone is output, and the predicted action is used as the predicted output, and the action of the main drone at the current moment in the operating data is used as the actual output, so as to optimize the predicted action output by the first neural network based on a reward function with a time constraint.

[0117] S46, synchronizing the network parameters of the first neural network to the second neural network;

[0118] S47. According to the reward function with time constraints, the actions of the main UAV during the conflict training process are optimized through the trained first neural network, so that the target Q value updated by the second neural network continues to increase until it stabilizes and approaches the expected target Q value, thereby obtaining a trained deep reinforcement learning model.

[0119] S5. Obtain the current state of the main aircraft in the actual conflict resolution mission, and use the trained deep reinforcement learning model to output the corresponding action when the reward value is the highest, so as to avoid conflict and reach the target position in time.

[0120] In this embodiment, the action corresponding to the highest reward value output by the trained deep reinforcement learning model is based on the trained first neural network taking the action corresponding to the highest reward value according to the current state of the main aircraft in performing the actual conflict resolution mission.

[0121] like Figure 3 As shown in the figure, after combining the reward function with time constraints and deep reinforcement learning based on the scheme of the present invention, the on-time rate of the aircraft arriving at the target location reached 85.40%, while when the reward function with time constraints was not set, the on-time rate of the aircraft arriving at the target location was only 43.70%. This scheme greatly improves the ability to stably and effectively avoid obstacles and arrive at the target location on time.

[0122] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints, characterized in that: The steps include: S1. Set up static obstacles and non-cooperative target aircraft to create a low-altitude operating environment; S2. Construct the state space and action space of the aircraft in the low-altitude operation environment; The S2 comprises the following steps: S21. Divide the detection area into six sector-shaped detection sub-areas based on the detection range of the aircraft and centering on the aircraft itself; S22, selecting information of the static obstacle or non-cooperative target aircraft closest to the own aircraft in each sector detection sub-area as state input, and obtaining the state space of the aircraft; The calculation expression of the state space is as follows: ， in, S represents the state space of the aircraft, Indicates the target location, Indicates the position of the own aircraft. Indicates the speed of the own aircraft. Indicates i The relative position of the static obstacle or non-cooperative target aircraft closest to the own aircraft in the sector detection sub-area is Indicates i The relative speed of the static obstacle or non-cooperative target aircraft closest to the own aircraft in the sector detection sub-area, where: i =1,2,3,4,5,6; S23. Constructing the action space of the aircraft according to the aircraft tactical conflict resolution task; The calculation expression of the action space is as follows: ， in, A represents the action space of the aircraft, represents the change in the aircraft's heading angle, Indicates the change in aircraft speed; S3. Construct a reward function with time constraints based on the distance between the own aircraft and the static obstacle or non-cooperative target aircraft, and the punctual time for the own aircraft to arrive at the target location; The S3 comprises the following steps: S31. Constructing conflict resolution rewards based on the distance relationship between the target aircraft and the static obstacle or non-cooperative target aircraft; The calculation expression of the conflict resolution reward is as follows: ， ， ， in, Reward for conflict resolution. Represents the distance change reward, Indicates the closest reward. Indicates the distance change reward value, Indicates the change in distance between the own aircraft and a static obstacle or non-cooperative target aircraft. Indicates the distance between the aircraft and the static obstacle or non-cooperative target aircraft at the current moment. Indicates the distance between the aircraft and the static obstacle or non-cooperative target aircraft at the last moment. represents the nearest reward scaling factor, log represents the logarithmic operation, Indicates the shortest distance between the own aircraft and a static obstacle or non-cooperative target aircraft. Indicates the detection range of the aircraft; S32. constructing an on-time constraint reward according to the on-time and actual time of the aircraft arriving at the target location; The calculation expression of the time constraint reward is as follows: ， in, represents the time-constrained reward, represents the time constraint reward scaling factor, e represents the exponential basis constant, T Indicates the actual time when the own aircraft arrives at the target location. Indicates the on-time time when the aircraft arrives at the target location. represents the reward width distribution control factor; S33, obtaining a reward function with a time constraint based on the conflict resolution reward and the time constraint reward; The calculation expression of the reward function with time constraint is as follows: R=R1+R2, Among them, R represents the reward function with time constraints; S4. According to the reward function with time constraints, the state space and action space of the aircraft in the low-altitude operation environment, low-altitude aircraft conflict resolution training is performed in the low-altitude operation environment based on the deep reinforcement learning model to obtain a trained deep reinforcement learning model; S5. Obtain the current state of the main aircraft in the actual conflict resolution mission, and use the trained deep reinforcement learning model to output the corresponding action when the reward value is the highest, so as to avoid conflict and reach the target position in time.

2. The low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints according to claim 1 is characterized in that: The S4 comprises the following steps: S41. Build an experience replay pool for storing operation data; S42, setting model training parameters; S43, constructing a deep reinforcement learning model based on Rainbow DQN, wherein the deep reinforcement learning model includes a first neural network for predicting the action of the main drone at the next moment and a second neural network for updating the target Q value; S44. Connect the deep reinforcement learning model to the low-altitude operation environment and the experience replay pool; S45, according to the state space and action space of the aircraft in the low-altitude operation environment, the reward function with time constraints and the second neural network, the aircraft of the aircraft is used as an intelligent agent to repeatedly perform conflict resolution training in the low-altitude operation environment, and the operation data and target Q value corresponding to each conflict resolution training are obtained; S46, synchronizing the network parameters of the first neural network to the second neural network; S47. According to the reward function with time constraints, the actions of the main UAV during the conflict training process are optimized through the trained first neural network, so that the target Q value updated by the second neural network continues to increase until it stabilizes and approaches the expected target Q value, thereby obtaining a trained deep reinforcement learning model.

3. The low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints according to claim 2 is characterized in that: The S45 comprises the following steps: S451, setting a training number threshold for conflict resolution training and a training duration for a single conflict resolution training; S452, setting the main body aircraft as an intelligent body at any position in the low-altitude operation environment, and using the position as the initial position during the current conflict resolution training, and initializing the state of the main body UAV; S453. According to the state space and action space of the aircraft in the low-altitude operation environment, based on the state and action of the main UAV at the current moment, the state of the main UAV at the next moment is obtained, and according to the reward function with time constraints and the second neural network, the reward value of the main UAV at the current moment and the target Q value corresponding to the state of the main UAV at the next moment are obtained, until the main UAV reaches the target position or reaches the training time of the single conflict resolution training, and the single conflict resolution training is completed; S454. After each round of conflict resolution training, the operation data at each moment in the single conflict resolution training is stored in the experience replay pool, and a number of operation data are randomly extracted from the experience replay pool, combined with the target Q value at the corresponding moment of each operation data, to train the first neural network with the goal of minimizing the error between the predicted output and the actual output of the first neural network; S455. Determine whether the number of conflict resolution trainings performed by the main UAV reaches the training number threshold, if so, enter S46, otherwise return to S452.

4. The low-altitude aircraft conflict resolution method based on deep reinforcement learning and time constraints according to claim 3 is characterized in that: The operation data includes the action, state, corresponding reward value and state of the main drone at the next moment, wherein the intelligent agent obtains the corresponding reward value at the current moment and the state of the intelligent agent at the next moment after completing the action at the current moment based on the state and action at the current moment.

Images