A reinforcement learning-based action control agent training method

Abstract

The application provides a kind of action control intelligent agent training method based on reinforcement learning, comprising three modules of neural network construction module, reward function configuration module, sample queue management module.Neural network construction module receives expert experience and simulation experience, while receiving the current state of unmanned aerial vehicle, after the training of each sub-network, the next time unmanned aerial vehicle avoidance control instruction is generated.Compared with the prior art, the application has the advantages that the method can be used to train the unmanned aerial vehicle to avoid obstacles in the environment, and the trained unmanned aerial vehicle can avoid obstacles in the environment.

Description

Technical Field

[0001] This invention relates to a method for training action control agents based on reinforcement learning. Background Technology

[0002] With the support of intelligent technologies, autonomous action control intelligent agents are one of the development directions of modern air combat. Air combat is the key to determining the outcome of modern warfare, and its result depends on the sophistication of equipment and the rational tactical decisions of pilots. Air combat involves high-speed flight, rapid decision-making, and complex tactical operations in a very short time, which often exceeds the limits of human reaction. Pilots face the challenge of processing massive amounts of information and overcome physiological limitations, making it difficult to make optimal decisions. At the same time, pilots may face fatigue, physiological stress, and emotional fluctuations in the high-intensity air combat environment, which may affect their decision-making and execution capabilities. Especially when facing incoming missiles, the short window for decision-making and execution makes it difficult for pilots to ensure the accuracy of their decisions and maneuvers. Meanwhile, unmanned aerial vehicles (UAVs) are gradually becoming the mainstream in modern air combat due to their high maneuverability, low cost, low risk, and ease of multi-aircraft coordination.

[0003] Deep reinforcement learning has achieved great success in agent sequence decision-making scenarios and holds promise for solving real-time decision-making challenges in air combat. However, research on combining UAV decision-making with reinforcement learning to generate tactical maneuvers for UAVs to evade missile attacks is still in its early stages. Furthermore, the missile evasion problem suffers from long decision chains and sparse rewards, leading to long algorithm learning cycles and poor robustness. How to scientifically and effectively train agents to automatically make decisions and maneuver rapidly within the limited time of an incoming missile attack, thereby ensuring the safety of UAVs, has become a research hotspot. Summary of the Invention

[0004] Purpose of the invention: To address the problem of intelligent autonomous evasion of unmanned combat aircraft when missiles are approaching, the purpose of this invention is to propose an improved deep reinforcement learning method based on dual-delay deep deterministic policy gradient by incorporating expert experience, constructing a comprehensive reward function, and distinguishing the value of data in the state-action queue. This method effectively improves the learning rate of the agent, generates the optimal maneuver strategy for evading incoming missiles, and thus improves the success rate of unmanned aerial vehicles (UAVs) in evading missiles.

[0005] This invention provides a training method for action control agents based on reinforcement learning (the meaning of agent and drone is not equivalent; agent can be understood as a control program of drone), including: establishing a neural network construction module based on dual-delay deep deterministic policy gradient, a reward function configuration module, and a sample queue management module;

[0006] The neural network construction module based on dual-delay deep deterministic policy gradient is used to construct a neural network based on dual-delay deep deterministic policy gradient. It receives expert experience and simulation experience, as well as the current state of the UAV. After training the neural network, it comprehensively generates the UAV evasion control command for the next moment.

[0007] The reward function configuration module generates the immediate reward or cost obtained by taking a certain action in the current drone state by mapping from the drone state and actions to real numbers. It considers four types of rewards, including: speed reward, angle reward, angular velocity reward and task completion reward.

[0008] During the training process, the sample queue management module dynamically selects samples from the expert experience sample queue and the simulation experience sample queue, mixes them, and then uses them to train the neural network.

[0009] The neural network includes a policy network Π, a target policy network Π', a first evaluation network Q1, a second evaluation network Q2, a first target evaluation network Q'1, and a second target evaluation network Q'2. The policy network takes the current state of the UAV as input and outputs control commands. The first and second evaluation networks take the current state and the control commands output by the policy network as input and output an evaluation of the current state's actions as output. The first and second target evaluation networks are responsible for processing the state and control commands at the next moment.

[0010] The policy network and the target network adopt the same neural network structure, including an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer;

[0011] The input layer receives state information from the UAV and missile, including position, velocity, attitude, and angular velocity. The input layer has a total of 18 neurons.

[0012] The first fully connected hidden layer connects the input layer and the second hidden layer, and contains L a1 One neuron, L a1 The value is a natural number greater than 0, and the ReLU activation function is used.

[0013] The second fully connected hidden layer connects the second hidden layer and the output layer, and includes L. a2 One neuron, L a2 For natural numbers with values ​​greater than 0, the ReLU activation function is used;

[0014] The output layer contains 4 neurons and uses the tanh activation function to generate normalized UAV output control parameters, including throttle, rudder, elevator, and roll.

[0015] The optimizers for the policy network and the target network are Adam optimizers.

[0016] The first evaluation network, the second evaluation network, the first target evaluation network, and the second target evaluation network all adopt the same neural network structure, including an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer.

[0017] The input layer, consisting of 22 neurons, receives state and action information from the drone and missile.

[0018] The first fully connected hidden layer connects the input layer and the second hidden layer, and contains L c1 One neuron, L c1 For natural numbers with values ​​greater than 0, the ReLU activation function is used;

[0019] The second fully connected hidden layer connects the second hidden layer and the output layer, and includes L. c2 One neuron, L c2 For natural numbers with values ​​greater than 0, the ReLU activation function is used;

[0020] The output layer contains one neuron, and its output is the state-action value function Q(s,a) of the UAV at the current moment, which is used to evaluate the quality of the action given by the UAV based on the current state. The higher the value of the action value function, the better the action selected by the policy network in this state.

[0021] The optimizers for the first evaluation network, the second evaluation network, the target evaluation network, and the evaluation network are all Adam optimizers.

[0022] The neural network construction module based on the double-delay deep deterministic policy gradient is used to execute the neural network algorithm of the double-delay deep deterministic policy gradient, including the following steps:

[0023] Input: A simulation environment for Markov decision processes of unmanned aerial vehicles;

[0024] Output: Parameters of the policy network, parameters of the first evaluation network, the second evaluation network, the first target evaluation network, and the second target evaluation network;

[0025] Initialize the first evaluation network Q1 with random parameters, and copy the parameters to the first target evaluation network Q'1;

[0026] The second evaluation network Q2 is initialized with random parameters, and the parameters are copied to the second target evaluation network Q'2;

[0027] Initialize the policy network with random parameters and copy the parameters to the target policy network;

[0028] Initialize the simulation sample queue QS and the expert sample queue QE;

[0029] Load expert experience into the expert sample queue (QE);

[0030] Begin the loop, letting the loop variable i range from 1 to MaxT, where MaxT is the maximum simulation time:

[0031] Get the current status of the drone and missile. i ;

[0032] s i As input to the policy network, the output is the drone's action a. i ;

[0033] a i =clip(a i +ε), where ε is white noise and clip is the clipping function;

[0034] a i Input the simulation system to obtain the new state s i+1 ;

[0035] At the same time, a comprehensive reward r is obtained from the simulation environment. i And the marker bEnd indicating whether the task has ended. i The comprehensive reward is the sum of speed reward, angle reward, angular velocity reward and task completion reward.

[0036] The quintet (s) i ,a i ,s i+1 ,r i ,bEnd i Store it in the simulation sample queue QS;

[0037] The iTD3 neural network training algorithm is invoked to update the neural network parameters;

[0038] If the task is complete, the flag bEnd indicates that the task has ended. i If the value is True, then the simulation environment is reset.

[0039] The invocation of the iTD3 neural network training algorithm specifically includes:

[0040] Input: Parameters of expert sample queue QE, simulation sample queue QS, policy network Π before calling iTD3 neural network training algorithm, target policy network Π', first evaluation network Q1, second evaluation network Q2, first target evaluation network Q'1 and second target evaluation network Q'2;

[0041] Output: Network Q1, Q2, Π, Q'1, Q'2, Π';

[0042] From the expert sample queue QE, randomly select round(λ*N) samples X1 according to weight, where N represents the total number of samples selected in a single draw, and λ represents the ratio of the number of samples selected from the expert experience sample queue to N;

[0043] Randomly select round((1-λ)*N) samples X2 from the simulation sample queue QS according to the weights;

[0044] X1 and X2 form the sample set;

[0045] For any sample j(s) in the sample set j ,a j ,s j+1 ,r j ,bEnd j ):

[0046] a' j+1 =clip(π'(s) j+1 )+ε),

[0047] y j =r j +γ*(1-bEnd j )*min i (Q i (s j+1 ,a' j+1 )), where i=1,2;a' j+1 Based on state s j+1 Based on the action estimate obtained from strategy π', the clipping function clip is used to control a' j+1 Within a range; y j It is an intermediate variable, γ is a discount factor, γ∈[0,1]; min i (Q i (s j+1 ,a' j+1 Expanded to min(Q1(s) j+1 ,a' j+1 ), Q2(s j+1 ,a' j+1 That is, take Q1(s) j+1 ,a' j+1 ) and Q2(s j+1 ,a' j+1 The smaller of the two;

[0048] (s j ,a j ,s j+1 ,r j ,bEnd j) is the quintuple obtained from sample j using a neural network algorithm with a double-delay deep deterministic policy gradient;

[0049] Based on mean squared loss 1 / N*Σ j (y j -Q i (s j ,a j )) 2 Update network Q i The parameter θ i ;

[0050] cnt = cnt + 1, where cnt is a global counter with an initial value of 0, and update_count is the network update cycle;

[0051] When cnt is divisible by update_count:

[0052] Reset the global counter: cnt = 0;

[0053] Update the parameter μ of Π according to the gradient ascent maximization method;

[0054] The parameters θ'1 of the first objective evaluation network Q'1, θ'2 of the second objective evaluation network Q'2, and μ' of the objective policy network Π' are updated using a soft update strategy:

[0055] θ'1←ρ*θ'1+(1-ρ)*θ1,

[0056] θ'2←ρ*θ'2+(1-ρ)*θ2,

[0057] μ′←ρ*μ′+(1-ρ)*μ,

[0058] Where ρ is the update ratio coefficient, μ is the parameter of the policy network Π, and the symbol ← indicates assignment.

[0059] Define speed reward as rv i (i):

[0060]

[0061] Where v is the linear velocity of the UAV, D is the straight-line distance between the missile and the UAV, and K1, K2, and K3 are constants.

[0062] Define the angle reward as rα i (i):

[0063]

[0064] Where α represents the deviation angle between the missile and the drone;

[0065] Define the angular velocity bonus as rωi (i):

[0066]

[0067] in K1 and K2 are the pitch and yaw angular velocities of the UAV, respectively, and K4 is a constant.

[0068] Define the task completion reward as rc i :

[0069]

[0070] Comprehensive reward r i The sum of speed bonus, angle bonus, angular velocity bonus, and mission completion bonus:

[0071] r i =rv i +rα i +rω i +rc i .

[0072] The sample queue management module dynamically selects samples from the expert experience sample queue and the simulation experience sample queue during training, mixes them, and uses the mixture to train the neural network. Specifically, the expert experience sample queue stores the action sequences of experts controlling drones to evade missiles. The length of the expert experience sample queue is K6. During training, N samples are selected from the expert experience sample queue and the simulation experience sample queue according to a ratio λ(t) and mixed to train the neural network. The ratio λ(t) is dynamically adjusted during training, continuously decreasing the number of samples used in the expert experience queue while increasing the number of samples in the simulation experience sample queue. For example, a linearly decreasing λ(t) function can be defined.

[0073] λ(t)=λ0-(λ0-λ f )t / MaxT

[0074] Where λ0 is the initial value of λ, λ f It is the value of λ at the end time, 0≤λ f <λ0≤1; t is the current time;

[0075] At the same time, a weight w(j) is added to each sample j. The magnitude of the weight is inversely proportional to the current distance between the missile and the drone, i.e., w(j)∝1 / D(j), where D(j) is the current distance between the missile and the drone. When selecting sample j, the probability P(j) of sample j being selected is determined according to the weight, i.e., P(j)=w(j) / ∑w(j), where ∑w(j) represents the summation of the weights of all samples.

[0076] Another aspect of the present invention proposes a training device for action control intelligent agents based on reinforcement learning, including a neural network construction module based on dual-delay deep deterministic policy gradient, a reward function configuration module, and a sample queue management module.

[0077] The neural network construction module based on dual-delay deep deterministic policy gradient receives expert experience and simulation experience, as well as the current state of the UAV. After training each sub-network, it comprehensively generates the UAV evasion control command for the next moment.

[0078] The reward function configuration module generates the immediate reward or cost obtained by taking a certain action in the current drone state by mapping the drone state and actions to real numbers.

[0079] During the training process, the sample queue management module dynamically selects samples from the expert experience sample queue and the simulation experience sample queue, mixes them, and then uses them to train the neural network.

[0080] Compared with the prior art, the beneficial effects of this invention are: (1) it proposes a reward function for close-range air combat, optimizing the sparse reward problem in air combat reinforcement learning; (2) to address the slow convergence speed of algorithms caused by random sampling in deep reinforcement learning, it proposes a value-based sample priority ranking method, improving the convergence speed of the learning and training algorithm. The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0081] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0082] Figure 1 This is a schematic diagram of the neural network interconnection structure based on the dual-delay deep deterministic policy gradient of the present invention.

[0083] Figure 2 This is a schematic diagram of the deviation angle between the missile and the drone when the missile rapidly approaches the drone according to the present invention. Detailed Implementation

[0084] This invention provides a method for training an action control agent based on reinforcement learning, comprising:

[0085] like Figure 1 As shown, a neural network construction module, a reward function configuration module, and a sample queue management module based on a dual-delay deep deterministic policy gradient are established.

[0086] The neural network construction module based on dual-delay deep deterministic policy gradient is used to construct a neural network based on dual-delay deep deterministic policy gradient. It receives expert experience and simulation experience, as well as the current state of the UAV. After training the neural network, it comprehensively generates the UAV evasion control command for the next moment.

[0087] The reward function configuration module generates the immediate reward or cost obtained by taking a certain action in the current drone state by mapping from the drone state and actions to real numbers. It considers four types of rewards, including: speed reward, angle reward, angular velocity reward and task completion reward.

[0088] During the training process, the sample queue management module dynamically selects samples from the expert experience sample queue and the simulation experience sample queue, mixes them, and then uses them to train the neural network.

[0089] The neural network includes a policy network Π, a target policy network Π', a first evaluation network Q1, a second evaluation network Q2, a first target evaluation network Q'1, and a second target evaluation network Q'2. The policy network takes the current state of the UAV as input and outputs control commands. The first and second evaluation networks take the current state and the control commands output by the policy network as input and output an evaluation of the current state's actions as output. The first and second target evaluation networks are responsible for processing the state and control commands at the next moment.

[0090] The policy network and the target network use the same neural network structure, which contains four layers:

[0091] Input layer: Receives state information from the drone and missile, including position, velocity, attitude, angular velocity, etc., with a total of 18 neurons.

[0092] The first fully connected hidden layer: connects the input layer and the second hidden layer, containing L a1 10 neurons, using the ReLU activation function.

[0093] The second fully connected hidden layer: connects the second hidden layer and the output layer, containing L a2 Each neuron uses the ReLU activation function.

[0094] Output layer: Contains 4 neurons, using the tanh activation function to generate normalized UAV output control parameters, namely throttle, rudder, elevator, and roll.

[0095] The network optimizer uses the Adam optimizer.

[0096] The first evaluation network, the second evaluation network, the first target evaluation network, and the second target evaluation network all use the same neural network structure, which contains a total of four layers:

[0097] Input layer: Receives status and action information from drones and missiles, consisting of 22 neurons.

[0098] The first fully connected hidden layer: connects the input layer and the second hidden layer, containing L c1 10 neurons, using the ReLU activation function.

[0099] The second fully connected hidden layer: connects the second hidden layer and the output layer, containing L c2 Each neuron uses the ReLU activation function.

[0100] Output layer: Contains 1 neuron, the output of which is the state-action value function Q(s,a) of the UAV at this moment, which is used to evaluate the quality of the action given by the UAV based on the state at this moment. The higher the value of the action value function, the better the action selected by the policy network in this state.

[0101] The network optimizer uses the Adam optimizer.

[0102] The neural network construction module based on the double-delay deep deterministic policy gradient is used to execute the neural network algorithm of the double-delay deep deterministic policy gradient, including the following steps:

[0103] Input: A simulation environment for Markov decision processes of unmanned aerial vehicles;

[0104] Output: Parameters of the policy network, parameters of the first evaluation network, the second evaluation network, the first target evaluation network, and the second target evaluation network;

[0105] Initialize the first evaluation network Q1 with random parameters, and copy the parameters to the first target evaluation network Q'1;

[0106] The second evaluation network Q2 is initialized with random parameters, and the parameters are copied to the second target evaluation network Q'2;

[0107] Initialize the policy network with random parameters and copy the parameters to the target policy network;

[0108] Initialize the simulation sample queue QS and the expert sample queue QE;

[0109] Load expert experience into the expert sample queue (QE);

[0110] Begin the loop, letting the loop variable i range from 1 to MaxT, where MaxT is the maximum simulation time:

[0111] Get the current status of the drone and missile. i ;

[0112] s iAs input to the policy network, the output is the drone's action a. i ;

[0113] a i =clip(a i +ε), where ε is white noise and clip is the clipping function;

[0114] a i Input the simulation system to obtain the new state s i+1 ;

[0115] At the same time, a comprehensive reward r is obtained from the simulation environment. i And the marker bEnd indicating whether the task has ended. i The comprehensive reward is the sum of speed reward, angle reward, angular velocity reward and task completion reward.

[0116] The quintet (s) i ,a i ,s i+1 ,r i ,bEnd i Store it in the simulation sample queue QS;

[0117] The iTD3 neural network training algorithm is invoked to update the neural network parameters;

[0118] If the task is complete, the flag bEnd indicates that the task has ended. i If the value is True, then the simulation environment is reset.

[0119] The invocation of the iTD3 neural network training algorithm specifically includes:

[0120] Input: Parameters of expert sample queue QE, simulation sample queue QS, policy network Π before calling iTD3 neural network training algorithm, target policy network Π', first evaluation network Q1, second evaluation network Q2, first target evaluation network Q'1 and second target evaluation network Q'2;

[0121] Output: Network Q1, Q2, Π, Q'1, Q'2, Π';

[0122] From the expert sample queue QE, randomly select round(λ*N) samples X1 according to weight, where N represents the total number of samples selected in a single draw, and λ represents the ratio of the number of samples selected from the expert experience sample queue to N;

[0123] Randomly select round((1-λ)*N) samples X2 from the simulation sample queue QS according to the weights;

[0124] X1 and X2 form the sample set;

[0125] For any sample j(s) in the sample set j ,a j ,s j+1 ,r j ,bEnd j ):

[0126] a' j+1 =clip(π'(s) j+1 )+ε),

[0127] y j =r j +γ*(1-bEnd j )*min i (Q i (s j+1 ,a' j+1 )), where i=1,2;a' j+1 Based on state s j+1 Based on the action estimate obtained from strategy π', the clipping function clip is used to control a' j+1 Within a range; y j It is an intermediate variable, γ is a discount factor, γ∈[0,1]; min i (Q i (s j+1 ,a' j+1 Expanded to min(Q1(s) j+1 ,a' j+1 ), Q2(s j+1 ,a' j+1 That is, take Q1(s) j+1 ,a' j+1 ) and Q2(s j+1 ,a' j+1 The smaller of the two;

[0128] (s j ,a j ,s j+1 ,r j ,bEnd j ) is the quintuple obtained from sample j using a neural network algorithm with a double-delay deep deterministic policy gradient;

[0129] Based on mean squared loss 1 / N*Σ j (y j -Q i (s j ,a j )) 2 Update network Q i The parameter θ i ;

[0130] cnt = cnt + 1, where cnt is a global counter with an initial value of 0, and update_count is the network update cycle;

[0131] When cnt is divisible by update_count:

[0132] Reset the global counter: cnt = 0;

[0133] Update the parameter μ of Π according to the gradient ascent maximization method;

[0134] The parameters θ'1 of the first objective evaluation network Q'1, θ'2 of the second objective evaluation network Q'2, and μ' of the objective policy network Π' are updated using a soft update strategy:

[0135] θ'1←ρ*θ'1+(1-ρ)*θ1,

[0136] θ'2←ρ*θ'2+(1-ρ)*θ2,

[0137] μ′←ρ*μ′+(1-ρ)*μ,

[0138] Where ρ is the update ratio coefficient, μ is the parameter of the policy network Π, and the symbol ← indicates assignment.

[0139] Configure the reward function:

[0140] (1) Speed ​​Bonus

[0141] When a missile approaches a drone while the drone maintains high speed, the drone can escape the missile's threat area more quickly by utilizing its higher flight speed during the evasion maneuver. Therefore, the speed bonus is defined as follows:

[0142]

[0143] Where v is the linear velocity of the UAV, D is the straight-line distance between the missile and the UAV, and K1, K2, and K3 are constants.

[0144] (2) Angle Reward

[0145] like Figure 2 As shown ( Figure 2 The median angle alpha is the deviation angle, v m v represents the speed of the missile p (Representing the speed of the drone). When the missile approaches the drone quickly and the deviation angle α between the missile and the drone is small, it is more difficult for the drone to escape the missile's tracking. The angle reward is to add the influence of the deviation angle on the final attack result to the reward function.

[0146]

[0147] (3) Angular velocity bonus

[0148] When a missile approaches a UAV and the UAV's pitch or yaw angle changes rapidly, the UAV can adjust its escape direction more quickly during missile evasion, thus reaching a safe area. Therefore, the angular velocity bonus is defined as follows:

[0149]

[0150] (4) Task completion reward

[0151] Mission completion rewards are only generated at the end of the missile explosion mission, and positive rewards are only generated when the drone is far enough away from the missile. The specific calculation method is as follows:

[0152]

[0153] The overall reward is the sum of speed reward, angle reward, angular velocity reward, and mission completion reward.

[0154] r i =rv i +rα i +rω i +rc i

[0155] Managing samples and queues:

[0156] To accelerate algorithm convergence, in addition to the traditional simulation experience sample queue (queue length K5), an expert experience sample queue (queue length K6) is designed to store the action sequences of experts controlling drones to evade missiles. During training, the algorithm will take N samples from the expert experience sample queue and the simulation experience sample queue according to a ratio λ(t) and mix them to train the neural network. This ratio will be dynamically adjusted during training, continuously reducing the number of samples used in the expert experience queue and increasing the number of samples in the simulation experience sample queue, thereby ensuring that the algorithm has stronger exploratory capabilities rather than being limited by expert experience.

[0157] Meanwhile, to differentiate the value of different samples, a weight is assigned to each sample, the magnitude of which is inversely proportional to the current distance between the missile and the drone. The closer the missile is to the drone, the more critical the drone's reaction becomes, and the more valuable the corresponding empirical data. Empirical data collected at greater distances is relatively less important to the missile avoidance agent. The probability of a sample being selected is determined based on its weight.

[0158] The hyperparameters involved in this invention can be selected from Table 1 below:

[0159] Table 1

[0160]

[0161] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the content of the reinforcement learning-based action control agent training method provided by this invention, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0162] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0163] This invention provides a method for training an action control agent based on reinforcement learning. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for training an action control agent based on reinforcement learning, characterized in that, include: Establish a neural network construction module, a reward function configuration module, and a sample queue management module based on a dual-delay deep deterministic policy gradient. The neural network construction module based on dual-delay deep deterministic policy gradient is used to construct a neural network based on dual-delay deep deterministic policy gradient. It receives expert experience and simulation experience, as well as the current state of the UAV. After training the neural network, it comprehensively generates the UAV evasion control command for the next moment. The reward function configuration module generates the immediate reward or cost obtained by taking a certain action in the current drone state by mapping from the drone state and actions to real numbers. It considers four types of rewards, including: speed reward, angle reward, angular velocity reward and task completion reward. During the training process, the sample queue management module dynamically selects samples from the expert experience sample queue and the simulation experience sample queue, mixes them, and then uses them to train the neural network. The expert experience sample queue is used to store the action sequences of experts controlling drones to evade missiles. The length of the expert experience sample queue is K6. During training, N samples are taken from the expert experience sample queue and the simulation experience sample queue according to a ratio λ(t) and mixed together to train the neural network. The ratio λ(t) is dynamically adjusted during training, continuously reducing the number of samples used in the expert experience queue while increasing the number of samples in the simulation experience sample queue. A linearly decreasing λ(t) function is defined. λ(t)=λ0-(λ0-λ f )t / MaxT Where λ0 is the initial value of λ, λ f It is the value of λ at the end time, 0 ≤ λ f < λ0≤1; t is the current time; At the same time, a weight w(j) is added to each sample j. The magnitude of the weight is inversely proportional to the current distance between the missile and the drone, i.e., w(j)∝1 / D(j), where D(j) is the current distance between the missile and the drone. When selecting sample j, the probability P(j) of sample j being selected is determined according to the weight, i.e., P(j)=w(j) / ∑w(j), where ∑w(j) represents the summation of the weights of all samples.

2. The method according to claim 1, characterized in that, The neural network includes a policy network Π, a target policy network Π', a first evaluation network Q1, a second evaluation network Q2, a first target evaluation network Q'1, and a second target evaluation network Q'2. The policy network takes the current state of the UAV as input and outputs control commands. The first and second evaluation networks take the current state and the control commands output by the policy network as input and output an evaluation of the current state's actions as output. The first and second target evaluation networks are responsible for processing the state and control commands at the next moment.

3. The method according to claim 2, characterized in that, The policy network and the target network adopt the same neural network structure, including an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer; The input layer receives status information from the UAV and missile, including position, velocity, attitude, and angular velocity. The first fully connected hidden layer connects the input layer and the second hidden layer, and contains L a1 One neuron, L a1 For natural numbers with values ​​greater than 0, the ReLU activation function is used; The second fully connected hidden layer connects the second hidden layer and the output layer, and includes L. a2 One neuron, L a2 For natural numbers with values ​​greater than 0, the ReLU activation function is used; The output layer contains 4 neurons and uses the tanh activation function to generate normalized UAV output control parameters, including throttle, rudder, elevator, and roll. The optimizers for the policy network and the target network are Adam optimizers.

4. The method according to claim 3, characterized in that, The first evaluation network, the second evaluation network, the first target evaluation network, and the second target evaluation network all adopt the same neural network structure, including an input layer, a first fully connected hidden layer, a second fully connected hidden layer, and an output layer. The input layer receives status and action information from the drone and missile; The first fully connected hidden layer connects the input layer and the second hidden layer, and contains L c1 One neuron, L c1 For natural numbers with values ​​greater than 0, the ReLU activation function is used; The second fully connected hidden layer connects the second hidden layer and the output layer, and includes L. c2 One neuron, L c2 For natural numbers with values ​​greater than 0, the ReLU activation function is used; The output layer contains one neuron, and its output is the state-action value function Q(s,a) of the UAV at the current moment. The optimizers for the first evaluation network, the second evaluation network, the target evaluation network, and the evaluation network are all Adam optimizers.

5. The method according to claim 4, characterized in that, The neural network construction module based on the double-delay deep deterministic policy gradient is used to execute the neural network algorithm of the double-delay deep deterministic policy gradient, including the following steps: Input: A simulation environment for Markov decision processes of unmanned aerial vehicles; Output: Parameters of the policy network, parameters of the first evaluation network, the second evaluation network, the first target evaluation network, and the second target evaluation network; Initialize the first evaluation network Q1 with random parameters, and copy the parameters to the first target evaluation network Q'1; The second evaluation network Q2 is initialized with random parameters, and the parameters are copied to the second target evaluation network Q'2; Initialize the policy network with random parameters and copy the parameters to the target policy network; Initialize the simulation sample queue QS and the expert sample queue QE; Load expert experience into the expert sample queue (QE); Begin the loop, letting the loop variable i range from 1 to MaxT, where MaxT is the maximum simulation time: Get the current status of the drone and missiles. i ; s i As input to the policy network, the output is the drone's action a. i ; a i = clip(a i + ɛ), where ɛ is white noise and clip is the clipping function; a i Input the simulation system to obtain the new state s i+1 ; At the same time, a comprehensive reward r is obtained from the simulation environment. i And the marker bEnd indicating whether the task has ended. i The comprehensive reward is the sum of speed reward, angle reward, angular velocity reward and task completion reward. The quintet (s) i , a i , s i+1 , r i , bEnd i Store it in the simulation sample queue QS; The iTD3 neural network training algorithm is invoked to update the neural network parameters; If the task is complete, the flag bEnd indicates that the task has ended. i If the value is True, then the simulation environment is reset.

6. The method according to claim 5, characterized in that, The invocation of the iTD3 neural network training algorithm specifically includes: Input: Parameters of expert sample queue QE, simulation sample queue QS, policy network Π before calling iTD3 neural network training algorithm, target policy network Π', first evaluation network Q1, second evaluation network Q2, first target evaluation network Q'1 and second target evaluation network Q'2; Output: Network Q1, Q2, Π, Q'1, Q'2, Π'; From the expert sample queue QE, randomly select round(λ*N) samples X1 according to weight, where N represents the total number of samples selected in a single draw, and λ represents the ratio of the number of samples selected from the expert experience sample queue to N; Randomly select round((1-λ)*N) samples X2 from the simulation sample queue QS according to the weights; X1 and X2 form the sample set; For any sample j(s) in the sample set j ,a j ,s j+1 ,r j ,bEnd j ): that' j+1 = clip ( π ' ( s j+1 ) + e), y j = r j + γ*(1-bEnd j )*min i (Q i (s j+1 ,a' j+1 )), where i=1,2;a' j+1 Based on state s j+1 Based on the action estimate obtained from strategy π', the clipping function clip is used to control a' j+1 Within a range; y j It is an intermediate variable, γ is a discount factor, γ∈[0,1]; min i (Q i (s j+1 ,a' j+1 Expanded to min(Q1(s) j+1 ,a' j+1 ), Q2(s j+1 ,a' j+1 That is, take Q1(s) j+1 ,a' j+1 ) and Q2(s j+1 ,a' j+1 The smaller of the two; (s j ,a j ,s j+1 ,r j ,bEnd j ) is the quintuple obtained from sample j using a neural network algorithm with a double-delay deep deterministic policy gradient; based on mean squared loss 1 / N*Σ j (y j -Q i (s j ,a j )) 2 Update network Q i The parameter θ i ; cnt = cnt+1, where cnt is a global counter with an initial value of 0, and update_count is the network update cycle; When cnt is divisible by update_count: Reset the global counter: cnt = 0; Update the parameter μ of Π according to the gradient ascent maximization method; The parameters θ'1 of the first objective evaluation network Q'1, θ'2 of the second objective evaluation network Q'2, and μ' of the objective policy network Π' are updated using a soft update strategy: θ'1 ← ρ*θ'1 + (1 − ρ)*θ1, θ'2 ← ρ*θ'2 + (1 − ρ)*θ2, μ′ ← ρ*μ′ + (1 − ρ)*μ, Where ρ is the update ratio coefficient, μ is the parameter of the policy network Π, and the symbol ← indicates assignment.

7. The method according to claim 6, characterized in that, Define speed reward as : in Let K be the linear velocity of the UAV, D be the straight-line distance between the missile and the UAV, and K1, K2, and K3 be constants.

8. The method according to claim 7, characterized in that, Define angle reward as : in Indicates the deviation angle between the missile and the drone; Define angular velocity reward as : in , These are the pitch rate and yaw rate of the UAV, respectively, and K4 is a constant.

9. The method according to claim 8, characterized in that, The reward for completing the task is : Comprehensive Rewards The sum of speed bonus, angle bonus, angular velocity bonus, and mission completion bonus: 。

Images