A close-range air combat maneuver decision method based on improved DDPG

Abstract

The invention discloses a close-range air combat maneuver decision method based on improved DDPG. First, a kinematic model of a UAV is established, and a one-to-one close-range air combat simulation confrontation scenario and a victory and defeat judgment model are established. The air combat decision process is converted into a Markov decision process, and the state-action space of the UAV is designed. A reward function model based on the target sight angle and relative speed is designed according to the victory and defeat judgment model, and the exploration of space and the use of experience of the intelligent agent are balanced by adding attenuated noise. The four-tuple generated by each decision made according to the air combat state is stored in the memory bank, and the strategy network is sampled and trained from it. The Actor network and the Critic network are updated proportionally to improve the stability and decision-making efficiency of the air combat training process. Game confrontation training with enemy UAVs based on matrix countermeasures can generate maneuver decision instructions that meet the expected needs, and achieve the air combat advantages of occupying the enemy first, attacking the enemy first, and avoiding the enemy first.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicles, and relates to unmanned aerial vehicle air combat maneuvering decision-making, and in particular to a close-range air combat maneuvering decision-making method based on an improved DDPG. Background Art

[0002] Artificial intelligence methods based on deep reinforcement learning have shown superior performance to humans in solving actual nonlinear optimization decision-making problems, such as AlphaZero, which integrates deep reinforcement learning and Monte Carlo search trees, and AlphaDogfight, an air combat decision-making agent based on hierarchical reinforcement learning. As the main combat force in future air combat, drones are the core of autonomous air combat technology research, and they are also the key and difficult issues in existing technology research. Drone air combat decision-making requires the most favorable tactical maneuver instructions to be generated in real time based on the current air combat situation information in a fast-paced, highly confrontational air combat environment, so as to gain the air combat advantage of occupying the enemy first, attacking the enemy first, and avoiding the enemy first.

[0003] Existing air combat maneuver decision-making methods can be divided into three categories: based on game theory, based on optimization theory, and based on artificial intelligence. Due to the large space of air combat decision-making problems, the first two methods are difficult to solve and have poor real-time performance, making them difficult to apply to fast-paced air combat confrontation environments. Deep reinforcement learning is the mainstream method for air combat maneuver decision-making based on artificial intelligence. Its basic idea is to model maneuver decisions as a Markov decision process, through continuous trial and error interaction between the agent and the environment, and to continuously adjust its own strategy using the rewards fed back from the environment to obtain the maximum cumulative discounted reward. When using deep reinforcement learning to train air combat decision-making agents, there are problems such as the large state-action space that needs to be explored, which makes it difficult to converge, and the sparse rewards during the learning process. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a close-range air combat maneuver decision-making method based on an improved DDPG, designs a reward function model based on the target line of sight angle and relative speed according to the victory and defeat judgment model, and balances the intelligent agent's exploration of space and utilization of experience through attenuated noise, and uses an opponent with a matrix countermeasure based on game theory to carry out simulated confrontation, thereby achieving efficient training and learning, and solving the problems of low learning efficiency and poor decision-making effect of air combat maneuver decision-making.

[0005] A close-range air combat maneuver decision method based on improved DDPG, the specific steps are as follows:

[0006] Step 1: Construction of drone motion model and close-range combat environment

[0007] The construction of close-range air combat scenarios includes the kinematic modeling of drones as air combat confrontation entities, the setting of drone sensors and weapon capabilities, and the setting of victory and defeat determination criteria.

[0008] The following UAV motion model is established in the northeast sky coordinate system:

[0009]

[0010] In the formula, V represents the speed of the drone, They represent the velocity components in the x, y, and z directions in a three-dimensional rectangular coordinate system. θ is the pitch angle, which represents the angle between the velocity vector and the horizontal plane xoy. ψ is the yaw angle, which represents the angle between the projection of the velocity vector on the horizontal plane xoy and the positive direction of the x-axis. g represents the acceleration due to gravity, n x is the tangential overload along the speed direction, n z express

[0011] ·

[0012] Normal overload along the pitch direction, n y Indicates perpendicular to n x 、n z The lateral overload of . represents the derivative.

[0013] In a close-range air combat environment, assuming that the Red and Blue sides rely on their own airborne sensors to obtain the opponent's real-time information, seize a favorable attack position through maneuvering decisions, enable their airborne fire control radars to intercept the target, guide weapons to launch and lock, and carry out the attack when the launch conditions are met, then the attack conditions should meet:

[0014]

[0015] in, represents the attack angle of the Red side’s airborne weapons, β0 is the maximum off-axis launch angle of the airborne weapons, and R rb Represents the distance between the red and blue sides, R min , R max are the minimum and maximum inescapable attack distances of airborne weapons. lock Indicates the current target lock duration, t set The shortest time that the onboard weapon can fire after being locked. When either side meets the firing condition, that side is considered the winner of the current round. If neither side meets the firing condition at the end of the current round, it is considered a draw.

[0016] Assume that the motion state information of the red and blue drones in three-dimensional space is:

[0017] T Ω =[X,Y,Z,V,θ,ψ] (3)

[0018] The velocity vector is

[0019] v Ω =[Vcos(θ)cos(ψ),Vcos(θ)sin(ψ),Vsin(θ)] (4)

[0020] Among them, Ω=r, b represents the red side and the blue side respectively. X, Y, Z represent the position information of the UAV in three-dimensional space, and the speed of the UAV V∈[100,250]m / s. Based on the kinematic model of the UAV, the fourth-order Runge-Kutta method is used to carry out the flight trajectory confrontation between the red and blue sides, and the fourth-order Runge-Kutta method advances step length RK_Step.

[0021] Construct a one-on-one close-range air combat scenario, assuming that the performance of both drones is exactly the same, that is, the maximum overload value in each direction and the upper and lower limits of the drone speed are the same. The initial position, height, speed, pitch angle, and yaw angle of the red team are all constant, and the initial height, speed, and pitch angle of the blue team are constant, and the initial position and yaw angle are randomly selected within a certain range.

[0022] Step 2: Blue team decision-making method

[0023] The blue drone uses a matrix game strategy based on game theory to make decisions. The principle of the matrix game method is easy to understand, and the algorithm complexity is relatively low. The actions of the drone's maneuvering decision are discretized into 7 typical maneuvering actions, namely ① remain unchanged, ② maximum overload acceleration, ③ maximum overload deceleration, ④ maximum overload left turn, ⑤ maximum overload right turn ⑥ maximum overload pull up, ⑦ maximum overload dive. And assume that both sides of the battle use these 7 typical maneuvering methods as decision execution. When the red side chooses the jth maneuvering method and the blue side chooses the ith maneuvering method, after one decision step, the advantage function value adv of the blue side over the red side is calculated. ij By predicting the overall advantage value of the red and blue sides after 7 kinds of maneuver decisions, the blue side's advantage matrix A is calculated. ij :

[0024]

[0025] Calculate the advantage matrix A ij The row and maximum value of the blue team are selected, and the maneuver corresponding to the row and maximum value is selected as the blue team’s final decision to be executed in the environment.

[0026] Since the design of the advantage function will directly affect the decision accuracy made by the matrix countermeasure method, an advantage function that integrates angle, speed, height and distance is designed to enable the blue agent to have a certain decision-making level.

[0027] The angle advantage function is:

[0028]

[0029] The speed advantage function is:

[0030]

[0031] Among them, V 0 Indicates the optimal air combat speed.

[0032] The height advantage function is:

[0033]

[0034] Among them, Z 0 Indicates the optimal air combat altitude.

[0035] The distance advantage function is:

[0036]

[0037] Among them, R 0 Denotes the optimal air combat distance, D 0 is the distance attenuation coefficient.

[0038] Combining the above advantage functions, the overall advantage function is:

[0039]

[0040] Among them, α1, α2, α3, and α4 are the weighted coefficients of angle advantage, speed advantage, height advantage, and distance advantage, respectively.

[0041] Step 3: DDPG-based air combat maneuver decision architecture

[0042] The maneuver decision process of the red team's UAV is described as a Markov decision process, and the UAV's maneuver decision model is constructed using the Actor-Critic framework of DDPG, including the design of the state-action space and the network structure design.

[0043] s3.1. State-action space design

[0044] In order to fully describe the state information of the red and blue sides in the air combat and use it as the input vector of the network, the relative distance R between the red and blue sides is defined as rb =[X b -X r ,Y b -Y r ,Z b -Z r ], the relative speed v of red and blue rb =[v b -v r ], the angle between the velocity vectors of red and blue Red side attack angle Blue attack angle The unit of position information is m, the unit of speed information is m / s, and the unit of angle information is rad. The air combat state space is represented as At the same time, it can be seen from the motion equation of the drone that the motion of the drone is controlled by tangential overload, normal overload and lateral overload, so the action space is expressed as a = [n x ,n y ,n z ].

[0045] s3.2. Network structure design

[0046] Construct two fully connected feedforward neural networks with N_Layer hidden layers and N_Num neurons in each layer, which serve as the online Actor network and the online Critic network respectively. Initialize the node parameters π(s|θ π ) and the node parameters Q(s,a|θ) of the online critic network Q ), using ReLu() as the activation function, setting the network learning rate to LR, and using the error back propagation and gradient descent method to update the network node parameters. In order to make the training more stable, two target actor networks and target critic networks with the same structure and node parameters as the online actor network and the online critic network, but with different update methods, were constructed, namely π'(s|θ' π )←π(s|θ π ), Q'(s,a|θ' Q )←Q(s,a|θ Q ). The online network is used as the real-time network and updated in real time through error back propagation and gradient descent during training. The target network is soft-updated by copying some node parameters of the online network.

[0047] Step 4: Red team’s single-step decision-making process

[0048] The Red Team drone selects action a by inputting the current air combat status into the online Actor network constructed in step 3. * To improve the agent's space exploration capability, at time t, the agent uses the network to select actions based on the current air combat status. After that, in action Gaussian noise is added on the basis of The standard deviation is σ, and the action value after adding noise Since the drone has a certain overload saturation limit, the action value needs to be limited to a certain range. Action value a t =[nx ,n y ,n z ], clip(-N,N) is the truncation function, which can make n x ,n y ,n z respectively fall into [-N x ,N x ],[-N y ,N y ],[-N z ,N z ] within the required range. In order to explore the space and collect experience as much as possible, the added noise standard deviation σ satisfies:

[0049]

[0050] Among them, episode represents the number of rounds, episode end The number of rounds at which the noise standard deviation stops decaying is set, and Max_Episode represents the maximum number of rounds of training. At the beginning of training, the initial value of the noise standard deviation is σ init , in each training round, the decay is carried out in an arithmetic manner through the above method. This method can effectively balance the exploration and utilization of the intelligent agent and accelerate the convergence speed of the DDPG algorithm. At the same time, in order to ensure that the output value of the online Actor network in DDPG remains constant in the later stage of training, resulting in a single experience sample in the memory bank, which in turn leads to overfitting of the online Critic network, it is necessary to set the standard deviation to σ end When , no attenuation is performed. The action value a with noise added after the decision t After executing in the environment, the environment feedbacks the next moment air combat status s t+1 and the single-step reward r t .

[0051] The single-step reward r t It includes two parts: angle reward and speed reward:

[0052]

[0053] Among them, ω1 and ω2 are the angle rewards respectively. Speed ​​Reward v The weight of .

[0054] The purpose of the angle bonus is to guide the red team to aim at the blue team, because the smaller the sight angle to the target, the greater the advantage in the air battle. The angle bonus is designed as follows:

[0055] The purpose of the speed bonus is to guide the Red team to control the flight speed in the air combat to match that of the Blue team. Reducing the speed when turning can aim at the opponent with a smaller turning radius, and catching up with the opponent at a faster speed when chasing, so that the Blue team can stay within the attack range of the Red team's airborne weapons. The speed bonus is designed as follows:

[0056]

[0057] Step 5: Experience storage and update

[0058] When using the DDPG method as a decision-making framework, it is necessary to design a memory bank that can store state transitions. The Actor network and the Critic network are trained by extracting experience from different time nodes in the past. Assuming that the decision cycle is δ seconds, the action remains unchanged during the decision cycle. In the round, the Red side’s airborne sensor observes the air combat state as s at time t. t , select action a according to the online Actor network t After executing the action, the environment will give the next reward r t , the air combat status is transferred to s t+1 , after each decision step, a four-tuple is generated t ,a t ,r t * ,s t+1 >, store it in the temporary experience library, where r t * Represents the single-step reward return value of the current decision. Repeat step 4 until the maximum decision step time Max_Step is reached, or one of the two sides reaches the victory condition. If the red side wins, the victory global reward r is given win ,Right now If the blue team wins, the red team will be awarded a global reward r. lose ,Right now If it is judged as a tie, no operation will be performed, that is, Where t_end represents the end time of the round. t ,a t ,r t ,s t+1 >All experience points in the temporary experience bank are stored in the total memory. If the total memory does not overflow after adding all experience points in the current temporary experience bank, the experience points are directly stored in the total memory. If the total memory does overflow, the old experience points in the memory bank are deleted according to the amount of overflow experience points, and then the temporary experience bank is stored and the temporary experience bank is cleared.

[0059] Step 6: Network sampling training

[0060] ​​When the total memory in step 5 is full, the online actor network and online critic network are trained once after each decision is made by the red team. The training process is: extract N experience points from the total memory i ,a i ,r i ,s i+1 >, calculate Among them, γ is the discount factor, which represents the ability to foresee future rewards. The loss is calculated using the mean square error (MSE):

[0061]

[0062] Update the node parameters θ of the online critic network through error back propagation and gradient descent Q .

[0063] Update the node parameters of the online Actor network based on policy gradient:

[0064]

[0065] in, represents the policy gradient, Represents gradient calculation.

[0066] In order to make the training process more stable, it is necessary to update the online Actor network at a lower frequency and the online Critic network at a higher frequency. Therefore, it is set to update the Actor network once every m times the Critic network is updated. After the node parameters of the online Actor network and the online Critic network are updated, the node parameters of the target Actor network and the target Critic network are soft updated. Each time the node parameters of the online network are weighted with the node parameters of the target network, the soft update is conducive to the stability of the target network, and its update method is θ' Q ←τθ Q +(1-τ)θ' Q ,θ' π ←τθ π +(1-τ)θ' π , τ represents the soft update ratio of the target network.

[0067] If the current training round number episode is less than the maximum training round number Max_Episode, return to step 1 to obtain new experience data. Otherwise, save the current network model node parameters and end the training.

[0068] Step 7: Air Combat Maneuver Decision

[0069] ​Use the trained network model to make single-step decisions for air combat maneuvers and output action values ​​for the online Actor network To restrict, the actual action is Repeat single-step decision-making until the conditions for ending the air combat confrontation are met.

[0070] The present invention has the following beneficial effects compared with the prior art:

[0071] 1. We designed a single-step reward and global reward function based on the winner-lose decision model, which effectively solved the problem of unstable training caused by sparse rewards in deep reinforcement learning.

[0072] 2. Use the improved DDPG algorithm to achieve continuous control of the drone's overload, add noise during the agent training process, and add a noise attenuation mechanism to balance the agent's exploration of better strategies and the use of the current better strategies, taking into account both convergence speed and decision quality. At the same time, update the Actor and Critic networks at a certain frequency, effectively improving the accuracy of Critic evaluation and the stability of the actions generated by the Actor network.

[0073] 3. Through adversarial training with the blue drone based on matrix countermeasures, the ability level of the generated maneuver decision-making agent based on the improved DDPG is relatively high. At the same time, the agent has a certain generalization ability for the initial spatial position distribution of the adversarial scenario. Experiments have proved that the adversarial winning rate reaches 98%. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 Flowchart of close air combat maneuver decision-making method based on improved DDPG;

[0075] Figure 2 A three-dimensional scene confrontation diagram of the training process in the embodiment;

[0076] Figure 3 This is the cumulative round reward result of training in the embodiment;

[0077] Figure 4 This is a graph showing the winning rate of the red team during the training process in the embodiment;

[0078] Figure 5 It is a three-dimensional scene confrontation diagram of test experiment 1 in the embodiment;

[0079] Figure 6 This is a three-dimensional scene confrontation diagram of test experiment 2 in the embodiment. DETAILED DESCRIPTION

[0080] The present invention will be further described below with reference to the accompanying drawings and embodiments. Figure 1As shown in the figure, a close-range air combat maneuver decision method based on improved DDPG is proposed. First, the close-range air combat training scene is initialized. The information of the red agent in three-dimensional space is set as T r =[0,0,5000,180,0,0], the information of the blue agent is T b =[X b ,Y b ,6000,150,0,ψ b ], where the initial position of the blue agent is X b ,Y b The initial yaw angle ψ is randomly selected from 11 points sampled every 100m in [1000,2000]. b X is randomly selected from 19 points sampled every 10° in [-90,90]. b ,Y b ,ψ b The selection of the three variables is independent of each other.

[0081] After the scene is initialized, the red and blue sides conduct maneuver decision-making confrontation in the scene. After each decision cycle, the red and blue sides each complete a single-step decision. The red drone uses the online Actor network to make decisions, taking the air combat status at time t as the network input and obtaining the output action. Execute action a after noise t The blue drone uses the matrix strategy to make decisions and executes the actions after the decision. b After completing a round of dogfights, the experience generated in the process is stored in the total memory bank. When the total memory bank is full, the training process begins. After the training is completed, the online Actor network node parameters are saved. In the test scenario, the saved network node parameters are loaded, but no noise is added when making decisions at this time. After multiple rounds of single-step decisions, a round of dogfights is completed. The training model parameters are shown in Table 1 below.

[0082] Table 1 Training parameters

[0083]

[0084]

[0085] Figure 2 This is a 3D confrontation diagram of a round in the training process. The initial yaw angle of the blue team is 0°, and the initial yaw angle of the red team is 0°. In the first section, the red team's agent accelerates to turn left and climb, and the blue team's agent chooses to turn right to gain an angle advantage. In the middle section, the red team turns right and climbs to gain an angle advantage, and continues to turn right in the final section to obtain a favorable attack angle, allowing the blue team to enter the red team's attack zone, and finally the red team is determined to win.

[0086] Figure 3, Figure 4 They are the cumulative round rewards of the red agent during the training process and the statistical winning rate curve of every 500 games. Figure 3 The cumulative round rewards steadily increase and eventually stabilize, indicating that the agent is constantly learning and eventually achieves a stable effect. Figure 4 It can be clearly seen that the DDPG algorithm can eventually converge, and the final statistical winning rate can be maintained above 98%.

[0087] In order to fully test the feasibility of the method and verify its generalization ability, the initial parameters of the test scenario and the training scenario are set to be the same. The blue agent changes the decision types at different tactical levels by changing the weights of different advantage functions. The advantage weights are set as shown in Table 2. Figure 5 In order for the Blue Agent to turn left and climb away from the Red Agent, the Red Agent chooses to accelerate the left turn and climb, and then attacks from behind.

[0088] Table 2 Advantage weight table

[0089]

[0090] The advantage weights of the blue agent are set as shown in Table 3. Figure 6 In order to avoid the Red team's attack zone lock, the Blue team's intelligent agent continued to turn right. The Red team's intelligent agent chose to accelerate the left turn and climb, and then decelerate and turn right to reduce the turning radius to gain an angle advantage. However, at this time, the attack window was missed due to the long attack distance caused by the deceleration. The Red and Blue teams continued to engage in a single-loop tailing. In the end, the Red team's intelligent agent was able to balance speed and attack angle and won.

[0091] Table 3 Advantage weight table

[0092]

Claims

1. A close-range air combat maneuver decision method based on improved DDPG, characterized by: The method specifically comprises the following steps: Step 1: Construction of drone motion model and close-range combat environment A kinematic model is established for the two opposing sides, the sensor and weapon capabilities of the drones of both sides are set, and the criteria for determining the winner are formulated; the drone decision-making actions are discretized into seven maneuvers, including ① remain unchanged, ② accelerate with maximum overload, ③ decelerate with maximum overload, ④ turn left with maximum overload, ⑤ turn right with maximum overload, ⑥ pull up with maximum overload, and ⑦ dive with maximum overload, and the two opposing sides are set to use the above maneuvers as decision execution; Step 2: Blue team decision-making method The blue drone uses a matrix strategy based on game theory to make decisions. When the red drone chooses the jth maneuver and the blue drone chooses the ith maneuver, the blue drone's advantage function value adv over the red drone is calculated. ij , we get the blue side's advantage matrix A ij : The advantage matrix A ij The maneuver corresponding to the middle row and the maximum value is the action performed by the blue team in the environment as the final decision; Step 3: Red Team Decision-Making Method The maneuver decision process of the red team’s UAV is described as a Markov decision process, and the UAV’s maneuver decision model is constructed using the DDPG Actor-Critic framework, including the design of the state-action space and the network structure design: s3.

1. State-action space design Define the relative distance R between red and blue rb =[X b -X r ,Y b -Y r ,Z b -Z r ], the relative speed v of red and blue rb =[v b -v r ], the velocity vector angle between red and blue Red side attack angle Blue attack angle Among them, the unit of position information is m, the unit of speed information is m / s, and the unit of angle information is rad; the air combat state space is expressed as The action space of the drone a=[n x ,n y ,n z ],n x is the tangential overload along the speed direction, n z represents the normal overload along the pitch direction, n y Indicates perpendicular to n x 、n z lateral overload; X, Y, Z represent the position information of the drone in three-dimensional space, v represents the velocity vector, subscript r represents the red side, and b represents the blue side; s3.

2. Network structure design Construct two fully connected feedforward neural networks as the online Actor network and the online Critic network respectively; initialize the node parameters π(s|θ π ) and the node parameters Q(s,a|θ) of the online critic network Q ), using ReLu() as the activation function, and using the error back propagation and gradient descent method to update the network node parameters in real time; then construct two target Actor networks and target Critic networks with the same structure and node parameters as the online Actor network and the online Critic network, namely π'(s|θ' π )←π(s|θ π ), Q'(s,a|θ' Q )←Q(s,a|θ Q ); The target network implements soft update by copying some node parameters of the online network; Step 4: Red team’s single-step decision-making process The Red Army drone inputs the air combat status at time t into the online Actor network constructed in step 3, and the network outputs the action In Action Add a mean value based on Gaussian noise with standard deviation σ, to get the actual action performed Where clip(-N,N) is a truncation function, which is used to make the overload in the three directions of the UAV fall within the limit range of overload saturation; Set the standard deviation σ of the Gaussian noise to decay during network training as follows: σ init represents the initial value of the standard deviation, episode represents the number of training rounds, and episode end The number of training rounds to stop decay, Max_Episode indicates the maximum number of training rounds; The Red Team drone performs action a in the environment t After that, the environment feedbacks the air combat status s at the next moment t+1 and the single-step reward r t ; The single-step reward r t Including angle bonus and speed bonus: Among them, ω1 and ω2 are the angle rewards respectively. Speed ​​Reward v The weight of Step 5: Experience storage and update Assume that the decision cycle time is δ seconds. During the decision cycle, the action remains unchanged. A single-step decision is made through step 4 to generate a four-tuple t ,a t ,r t * ,s t+1 >, store it in the temporary experience library, where r t * Represents the single-step reward return value of the current decision. Repeat step 4 until the maximum decision step time Max_Step is reached, or one of the two sides reaches the victory condition; if the red side wins, the global victory reward r is given win ,Right now If the blue team wins, the red team will be awarded a global reward r. lose ,Right now If it is judged as a tie, no operation will be performed, that is, Where t_end represents the end time of the round; at t_end, the quadruple of each decision cycle in the temporary experience library is t ,a t ,r t ,s t+1 > All are stored in the total memory. If the total memory does not overflow after adding all the experience of the current temporary experience bank, it will be directly stored. If it overflows, the old experience in the memory will be deleted according to the amount of overflow experience, and then the temporary experience bank will be stored and the temporary experience bank will be cleared;​​ Step 6: Network sampling training When the total memory in step 5 is full, the online actor network and online critic network are trained once after each decision is made by the red team. The training process is as follows: N experience points are extracted from the total memory i ,a i ,r i ,s i+1 >, calculate Among them, γ is the discount factor, and the loss is calculated using the mean square error (MSE):​ Update the node parameters θ of the online critic network through error back propagation and gradient descent Q ; Update the node parameters of the online Actor network based on policy gradient: in, represents the policy gradient, Represents gradient calculation; Set the Actor network to be updated once after every m updates of the Critic network; after the node parameters of the online Actor network and the online Critic network are updated, the node parameters of the target Actor network and the target Critic network are soft updated. During each update, the node parameters of the online network and the node parameters of the target network are weighted, θ' Q ←τθ Q +(1-τ)θ' Q ,θ' π ←τθ π +(1-τ)θ' π ,τ represents the soft update ratio of the target network; If the current training round number episode is less than the maximum training round number Max_Episode, return to step 1 to obtain new experience data; otherwise, save the current network model node parameters and end the training; Step 7: Air Combat Maneuver Decision Use the trained network model to make single-step decisions for air combat maneuvers and output action values ​​for the online Actor network Overload limit is performed, and the actual action is: Repeat step 4 to make single-step decisions until the conditions for ending the air combat confrontation are met.

2. A close air combat maneuver decision method based on improved DDPG as claimed in claim 1, characterized in that: The following UAV motion model is established in the northeast sky coordinate system: In the formula, V represents the speed of the drone, They represent the velocity components in the x, y, and z axes in the three-dimensional rectangular coordinate system respectively; θ is the pitch angle, which represents the angle between the velocity vector and the horizontal plane xoy; ψ is the yaw angle, which represents the angle between the projection of the velocity vector on the horizontal plane xoy and the positive direction of the x-axis; g represents the gravitational acceleration, n x is the tangential overload along the speed direction, n z represents the normal overload along the pitch direction, n y Indicates perpendicular to n x 、n z lateral overload; · represents the derivative.

3. The close air combat maneuver decision method based on improved DDPG as claimed in claim 1, characterized in that: In a close-range air combat environment, assuming that the Red and Blue sides rely on their own airborne sensors to obtain the opponent's real-time information, seize a favorable attack position through maneuvering decisions, enable their airborne fire control radars to intercept the target, guide weapons to launch and lock, and carry out the attack when the launch conditions are met, then the attack conditions should meet: in, represents the attack angle of the Red side’s airborne weapons, β0 is the maximum off-axis launch angle of the airborne weapons, and R rb Represents the distance between the red and blue sides, R min , R max are the minimum and maximum inescapable attack distances of airborne weapons respectively; t lock Indicates the current target lock duration, t set The shortest time that an onboard weapon can be fired after being locked.

4. The close air combat maneuver decision method based on improved DDPG as claimed in claim 1, characterized in that: It is set that when either side meets the launch conditions, that side is judged as the winner of the current round; if neither side meets the launch conditions at the end of the current round, it is judged as a draw.

5. The close air combat maneuver decision method based on improved DDPG as claimed in claim 1, characterized in that: Assume that the motion state information of the red and blue drones in three-dimensional space is: T Ω =[X,Y,Z,V,θ,ψ] (8) The velocity vector is v Ω =[Vcos(θ)cos(ψ),Vcos(θ)sin(ψ),Vsin(θ)] (9) where, Ω=r、b respectively Represents the red team and the blue team; the speed of the drone V∈[100,250]m / s; based on the kinematic model of the drone, the fourth-order Runge-Kutta method is used to carry out the flight trajectory confrontation deduction of the red and blue teams, and the advancement step of the fourth-order Runge-Kutta method is RK_Step; construct a one-on-one close-range air combat scenario, assuming that the performance of the drones of both sides is exactly the same, that is, the maximum overload value in each direction and the upper and lower limits of the drone speed are the same; the initial position, height, speed, pitch angle, and yaw angle of the red team are constant, the initial height, speed, and pitch angle of the blue team are constant, and the initial position and yaw angle are randomly selected within a certain range.

6. The close air combat maneuver decision method based on improved DDPG as claimed in claim 1, characterized in that: Design the advantage function of the blue team as: Among them, α1, α2, α3, and α4 are the weighting coefficients of angle advantage, speed advantage, height advantage, and distance advantage respectively; The angle advantage function is: The speed advantage function is: Among them, V 0 Indicates the best air combat speed; The height advantage function is: Among them, Z 0 Indicates the best air combat altitude; The distance advantage function is: Among them, R 0 Denotes the optimal air combat distance, D 0 is the distance attenuation coefficient.

7. The close air combat maneuver decision method based on improved DDPG as claimed in claim 1, characterized in that: Design the Red Team's angle bonus to be: The speed bonus is:

Images