Deep reinforcement learning air combat decision-making method based on situation prediction

Abstract

The invention provides a deep reinforcement learning air combat decision-making method based on situation prediction, belongs to the technical field, and solves the technical problem that an unmanned aerial vehicle cannot realize autonomous decision-making and environment self-adaption in a complex air combat environment. According to the technical scheme, modeling is carried out through kinematics and dynamics characteristics of the unmanned aerial vehicle to adapt to the actual flight state; establishing a deep enhanced confrontation environment for unmanned aerial vehicle confrontation training; dynamically evaluating the state of the enemy unmanned aerial vehicle by using an LSTM prediction algorithm, and quickly obtaining the state of the enemy unmanned aerial vehicle at the next moment; and taking reward distribution as input of a decision maker, and inferring enemy unmanned aerial vehicle situation actions in combination with a prediction state and a reward result to realize tracking strike. According to the method, the advantage of machine intelligent calculation and human experience judgment are combined, the intention recognition accuracy of the enemy unmanned aerial vehicle is improved in a complex and uncertain environment, and the decision-making ability and tactical advantages of our unmanned aerial vehicle are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of air combat confrontation, and specifically relates to a deep reinforcement learning air combat decision-making method based on situation prediction. Background Art

[0002] In recent years, with the rapid development of drone technology, its application in military reconnaissance, surveillance and strike missions has become increasingly widespread. In order to improve the combat effectiveness and autonomy of drones, researchers have begun to explore the use of drones and artificial intelligence to optimize the behavior decision-making and strategy execution of drones. Especially in a confrontational environment, drones need to have the ability to quickly adapt to changes, predict enemy actions and make the best response.

[0003] Traditional UAV control systems mainly rely on the pilot's experience, which makes them powerless in complex and changeable actual battlefield environments and extremely high overload intensity maneuvers. In addition, although some existing UAV control methods based on deep learning have improved the autonomy and adaptability of UAVs to a certain extent, there are still some problems, such as insufficient model generalization ability, poor adaptability to dynamic environments, and lack of effective real-time situation assessment and reward mechanisms.

[0004] How to solve the above technical problems is the subject faced by the present invention. Summary of the invention

[0005] The present invention provides a deep reinforcement learning air combat decision-making method based on situation prediction, which is used to solve the technical problems of autonomous decision-making and environmental adaptation of UAVs, and aims to improve the combat effectiveness of UAVs in complex dynamic environments.

[0006] The inventive idea of ​​the present invention is as follows: the present invention proposes a UAV adversarial training framework based on deep reinforcement learning and long short-term memory network (LSTM). The framework firstly accurately models the kinematic and dynamic characteristics of the UAV so that the UAV can adapt to the actual flight state more accurately; then, by designing reasonable action, state and reward functions, a deep reinforcement adversarial environment is established to realize adversarial training between UAVs. On this basis, the state of the enemy UAV is dynamically evaluated by using the LSTM prediction algorithm so as to quickly obtain the state information of the enemy UAV at the next moment, and then, the reward is distributed according to the result of the real-time relative situation evaluation as the input of the decision maker; finally, according to the predicted state and reward result of the UAV, the situation action of the enemy UAV at the current moment is inferred by the decision maker to achieve the purpose of tracking and attacking the enemy UAV.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: a deep reinforcement learning air combat decision-making method based on situation prediction, comprising the following steps:

[0008] 1) By modeling the kinematic and dynamic characteristics of the UAV, the UAV can more accurately adapt to the actual flight state;

[0009] 2) Through the design of drone actions, states and rewards, establish a deep enhanced confrontation environment for drones and realize drone confrontation training;

[0010] 3) Dynamically evaluate the state of the enemy drone through the LSTM prediction algorithm to quickly obtain the state of the enemy drone at the next moment;

[0011] 4) Reward allocation based on the results of real-time relative situation assessment as input to the decision maker;

[0012] 5) Based on the predicted state and reward results of the drone, the decision maker is used to infer the current situation and action of the enemy drone, so as to achieve the purpose of tracking and attacking the enemy drone;

[0013] Furthermore, the step 2) comprises the following steps:

[0014] 2-1) The environment status includes the location information of the enemy and our drones (x p ,y p ,z p ); velocity information v; action information normal vector n x , tangent vector n z and roll angle roll; the reward settings include angle reward reward_angle, distance reward reward_dis and sparse reward reward_spa;

[0015] 2-2) Establish a drone confrontation environment through designed actions, states and rewards;

[0016] Furthermore, the step 3) comprises the following steps:

[0017] 3-1) To solve the problem of long sequences, an LSTM model is constructed to predict the state of the enemy UAV. The model predicts the state of the enemy UAV and the state of our UAV as the input for model training;

[0018] 3-2) In order to improve the stability of training, the spiral motion data of the enemy UAV is collected from different angles in advance, and an LSTM model is pre-trained based on these data sets. This process will generate multiple trajectory data sets as the basis for the pre-trained model. Subsequently, the pre-trained model is integrated into the tracking and strike mission to effectively predict the state of the enemy UAV.

[0019] Furthermore, the step 4) comprises the following steps:

[0020] 4-1) Segment reward configuration: The reward function is:

[0021] Reward_H=(1+α)·reward_angle+(1-α)·reward_dis+reward_spa Where α is the weight factor, reflecting the importance of the distribution, and its expression is:

[0022]

[0023] red_d=reward_dis-reward_p.dis

[0024] red_a=reward_p.angle-reward_angle

[0025] Among them, the k value controls the growth rate of the function, and reward_p.dis and reward_p.angle represent the distance reward and angle reward at the previous moment respectively.

[0026] 4-2) When the drone performs a situation transition, the reward distribution weight is adjusted using a smooth decay formula.

[0027] The attenuation formula is:

[0028]

[0029] in, and is the smoothing factor.

[0030] Furthermore, the step 5) comprises the following steps:

[0031] 5-1) Initialize the parameters of the actor and critic networks in the SAC algorithm used by the decision maker; in order to maximize the future cumulative rewards and entropy, set the most optimal strategy θ of the SAC algorithm * for:

[0032]

[0033] Among them, S t A is the drone status; t is the action of the drone in time step; ξ θ is the distribution function corresponding to the state and action. τ is the entropy regularization coefficient, which is used to control the importance of entropy. H(θ(·|S t )) represents the entropy value. A higher entropy value indicates that the agent has explored the environment to a greater extent, enabling the drone to find more effective strategies, thus facilitating subsequent strategy learning.

[0034] 5-2) Import the pre-trained LSTM model, receive the drone’s relative state, action, reward, and new state from the reinforcement learning environment, and prepare for model training.

[0035] 5-3) Determine whether Ang_blue is greater than 1. If so, the two drones are in direct confrontation and the reward weight distribution needs to be redistributed and the algorithm parameters updated. When the drone is in the tracking and attacking state, the weight gradually decays. After m environmental steps, the weight returns to the initial value.

[0036] 5-4) Save the reward, action and state information and input them into the model algorithm for training.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. Accurate modeling of the kinematic and dynamic characteristics of drones can more accurately describe the behavior of drones in different flight states, including acceleration, speed, attitude changes, etc. These models provide a more realistic training environment for deep reinforcement learning algorithms, making the strategies obtained by drones in simulated training closer to actual operations.

[0039] 2. By building an adversarial simulation environment for deep reinforcement learning of drones and designing state, action and reward functions, it is possible to simulate complex air combat environments to test and optimize the autonomous decision-making capabilities of drones, thereby significantly improving the combat effectiveness of drones in real battlefield environments.

[0040] 3. A prediction-based reinforcement learning algorithm is proposed. Through this algorithm, the behavior of enemy drones can be predicted, so that our drones can respond in advance and take more effective tactical actions. This not only improves the autonomous decision-making ability of drones, but also enhances their survivability and strike efficiency in complex battlefield environments.

[0041] 4. The implementation of situation assessment and reward allocation mechanism has been introduced. Real-time relative situation assessment can continuously monitor the status changes of both sides, and the reward allocation mechanism ensures that the drone will not ignore the long-term strategy while pursuing short-term goals. This enables our drone to quickly make the best decision on the battlefield, and maintain tactical flexibility and initiative whether attacking or retreating, thereby improving the efficiency of tracking and striking.

[0042] 5. Through the combination of deep reinforcement learning and LSTM prediction algorithm, a deep reinforcement learning air combat decision-making method based on situation prediction is proposed. This method enables the UAV to respond in advance, avoid passive situations, and can effectively track and strike enemy UAVs.

[0043] Therefore, the UAV confrontation training framework provided by the present invention is expected to significantly improve the combat effectiveness and autonomy of UAVs in complex battlefield environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification.

[0045] Figure 1 This is a structural diagram of the air combat decision-making system of the present invention;

[0046] Figure 2 It is the parameter diagram of the UAV model in the present invention;

[0047] Figure 3 It is a relative situation diagram of the UAV in the present invention;

[0048] Figure 4 This is a structural diagram of the prediction decision algorithm in the present invention;

[0049] Figure 5 This is a structural diagram of the segmented reward allocation prediction algorithm in the present invention;

[0050] Figure 6 This is a graph of drone confrontation results predicted based on LSTM situation in the present invention.

[0051] Figure 7 This is a diagram of the drone confrontation results predicted based on RNN situation in the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments.

[0053] Example 1

[0054] See also Figures 1 to 6 The technical solution provided in this embodiment is: a deep reinforcement learning air combat decision-making method based on situation prediction, the system structure diagram is as follows Figure 1 As shown, the following steps are included:

[0055] Step 1) By modeling the kinematic and dynamic characteristics of the UAV, the UAV can more accurately adapt to the actual flight state;

[0056] Step 2) Establish a deep enhanced confrontation environment for drones through the design of drone actions, states, and rewards.

[0057] Realize UAV confrontation training;

[0058] Step 3) Dynamically evaluate the state of the enemy UAV through the LSTM prediction algorithm to quickly obtain the state of the enemy UAV at the next moment;

[0059] Step 4) Reward allocation is performed based on the results of the real-time relative situation assessment as input to the decision maker;

[0060] Step 5) Based on the predicted state and reward result of the UAV, the decision maker is used to infer the situation and action of the enemy UAV at the current moment, so as to achieve the purpose of tracking and attacking the enemy UAV.

[0061] Step 1) By modeling the kinematic and dynamic characteristics of the drone, the drone is more in line with the actual flight state. The kinematic model and dynamic model formula are as follows:

[0062] Kinematic model:

[0063] Kinetic model:

[0064] Where (x p ,y p ,z p ) For drone coordinates, please refer to Figure 2 , v is the flight speed of the drone, γ, ψ, and roll represent the pitch angle, heading angle, and roll angle of the drone, respectively, and n x is the overload in the direction of the drone speed, representing the thrust and deceleration of the drone, n z Indicates the overload in the pitch direction of the drone, that is, the normal overload. x ,n z ,roll] is the control variable for controlling the UAV to perform maneuvers, where n x Control the speed of the drone, n z and roll control the direction of the drone's velocity vector.

[0065] Step 2) Through the design of drone actions, states and rewards, establish a deep enhanced confrontation environment for drones. Implement drone confrontation training. The distance reward, angle reward and sparse reward of drones, and the relative state of drones are shown in Figure 3 .

[0066] 2-1) The environment status includes the location information of the enemy and our drones (x p ,y p ,z p ); velocity information v; action information normal vector n x , tangent vector n z and roll angle rol; reward settings include angle reward reward_angle, distance reward reward_dis and sparse reward reward_spa;

[0067] Angle reward reward_angle: In order to improve the UAV's ability to track and strike enemy UAVs, it is necessary to design an angle reward function to minimize the relative angle between the two UAVs, so that the UAV's nose is as close to the enemy UAV's tail as possible, thereby optimizing the tracking and striking efficiency. The angle reward is as follows:

[0068]

[0069] Distance reward reward_dis: In order to maintain a strategic advantage in attack, the drone must maintain a specific distance from the enemy drone. This requires the implementation of a distance reward function that evaluates whether the drone's actions are effective in keeping it at the desired distance from the enemy drone. The distance reward is as follows:

[0070]

[0071] Sparse reward reward_spa: In tasks involving long step lengths (such as drone tracking and strike), sparse rewards can be used to optimize the maneuvers of drones. By using sparse rewards, specific rewards can be assigned when specific conditions are met. When the attack angle Ang_red and the azimuth angle Ang_blue, the Euclidean distance between drones When the relative height is within the appropriate range, the value of the sparse reward function reward_spa is 2.

[0072] 2-2) Establish a drone confrontation environment through designed actions, states and rewards;

[0073] Step 3) Dynamically evaluate the state of the enemy UAV through the LSTM prediction algorithm to quickly obtain the state of the enemy UAV at the next moment;

[0074] 3-1) To solve the problem of long sequences, we build an LSTM model to predict the state of the enemy drone. The model predicts the state of the enemy drone and the state of our drone as the input of the model training. Please refer to Figure 4 .

[0075] Establish an LSTM network, and its update formula is as follows:

[0076]

[0077] Among them, the subscripts i, f, and o are input gate, forget gate, and output gate respectively; σ() is a nonlinear activation function; W hf , W hi , W ha , W ho , U xf , U xi , U xa , Uxo 、V hy is the weight matrix; b f 、b i 、b c 、b o 、b y is the bias matrix; tanh() is the hyperbolic tangent function; h (t-1) and h t The output of LSTM cells at the previous moment and the current moment; x t and t is the input and output of the network at the current moment.

[0078] 3-2) The LSTM network predicts the location information of the enemy drone at the next time point by analyzing the trajectory of the enemy drone at the most recent 20 time points. The predicted location information is subtracted from the actual location of the enemy drone at the next moment, thereby updating the parameters of the LSTM network.

[0079] Step 4) Reward allocation is performed based on the results of the real-time relative situation assessment as input to the decision maker;

[0080] 4-1) Segment reward configuration: The reward function is:

[0081] Reward_H=(1+α)·reward_angle+(1-α)·reward_dis+reward_spa Where α is the weight factor, reflecting the importance of the distribution, and its expression is:

[0082]

[0083] red_d=reward_dis-reward_p.dis

[0084] red_a=reward_p.angle-reward_angle

[0085] Among them, the k value controls the growth rate of the function, and reward_p.dis and reward_p.angle represent the distance reward and angle reward at the previous moment respectively.

[0086] 4-2) When the drone performs a situation transition, the reward distribution weight is adjusted using a smooth decay formula.

[0087] The attenuation formula is:

[0088]

[0089] in, and is the smoothing factor.

[0090] Step 5) Based on the predicted state and reward result of the UAV, the decision maker is used to infer the situation and action of the enemy UAV at the current moment, so as to achieve the purpose of tracking and attacking the enemy UAV.

[0091] 5-1) Initialize the parameters of the actor and critic networks in the SAC algorithm used by the decision maker; in order to maximize the future cumulative rewards and entropy, set the most optimal strategy θ of the SAC algorithm * for:

[0092]

[0093] Among them, S t A is the drone status; t is the action of the drone in time step; ξ θ is the distribution function corresponding to the state and action. τ is the entropy regularization coefficient, which is used to control the importance of entropy. H(θ(·|S t )) represents the entropy value. A higher entropy value indicates that the agent has explored the environment to a greater extent, enabling the drone to find more effective strategies, thus facilitating subsequent strategy learning.

[0094] 5-2) Import the pre-trained LSTM model, receive the drone’s relative state, action, reward, and new state from the reinforcement learning environment, and prepare for model training.

[0095] 5-3) Determine whether Ang_blue is greater than 1. If so, the two drones are in direct confrontation and need to redistribute the reward weights and update the algorithm parameters. When the drone is in the tracking and attacking state, the weights gradually decay. After m environmental steps, the weights return to the initial value. Finally, store the rewards, actions, and states and input them into the algorithm for training. For the final training effect, please refer to Figure 6 .

[0096] Example 2

[0097] See also Figure 7 The technical solution provided in this embodiment is an air combat decision-making method based on traditional neural network situation prediction, comprising the following steps:

[0098] Step 1) By modeling the kinematic and dynamic characteristics of the UAV, the UAV can more accurately adapt to the actual flight state;

[0099] Step 2) Establish a deep enhanced confrontation environment for drones through the design of drone actions, states, and rewards.

[0100] Realize UAV confrontation training;

[0101] Step 3) Dynamically evaluate the state of the enemy UAV through the RNN prediction algorithm to quickly obtain the state of the enemy UAV at the next moment;

[0102] Step 4) Reward allocation is performed based on the results of the real-time relative situation assessment as input to the decision maker;

[0103] Step 5) Based on the predicted state and reward result of the UAV, the decision maker is used to infer the situation and action of the enemy UAV at the current moment, so as to achieve the purpose of tracking and attacking the enemy UAV.

[0104] Step 1) and step 2) are the same as those in the above-mentioned embodiment 1.

[0105] Step 3) Dynamically evaluate the state of the enemy drone through the RNN prediction algorithm to quickly obtain the state of the enemy drone at the next moment

[0106] 3-1) Build an RNN model to predict the state of the enemy UAV. The model predicts the state of the enemy UAV and the state of our UAV as the model training input.

[0107] Establish an RNN network, and its update formula is as follows:

[0108] h t =σ(W h h t-1 +W x x t +b h )

[0109] o t =f(W o h t +b o )

[0110] Among them, W h , W x , W o is the weight matrix; b h 、b o is the bias matrix, f() is the output layer activation function; σ() is the activation function; h t-1 and h t is the hidden state of the previous moment and the current moment; x t and t are the network input and output at the current moment.

[0111] 3-2) The RNN network predicts the location information of the enemy drone at the next time point by analyzing the trajectory of the enemy drone at the most recent 20 time points. The predicted location information is subtracted from the actual location of the enemy drone at the next moment, thereby updating the parameters of the RNN network.

[0112] Step 4) and step 5) are the same as those in the above-mentioned embodiment 1. The local drone situation is predicted using a recurrent neural network to obtain a result graph of the confrontation between the enemy and our drones, such as Figure 7 shown.

[0113] Will Figure 6 and Figure 7 In contrast, Figure 6 The UAV confrontation result obtained by the deep reinforcement learning air combat decision-making method based on situation prediction of this embodiment has a more obvious confrontation effect.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep reinforcement learning air combat decision-making method based on situation prediction, characterized by: The following steps are involved: 1) By modeling the kinematic and dynamic characteristics of the UAV, the UAV can accurately adapt to the actual flight state; 2) Through the design of drone actions, states and rewards, a deep enhanced confrontation environment for drones is established to achieve drone confrontation training; 3) Dynamically evaluate the state of the enemy drone through the LSTM prediction algorithm to quickly obtain the state of the enemy drone at the next moment; 4) Reward allocation based on the results of real-time relative situation assessment as input to the decision maker; 5) Based on the predicted state and reward results of the UAV, the decision maker is used to infer the current situation and action of the enemy UAV, so as to achieve the purpose of tracking and attacking the enemy UAV.

2. The deep reinforcement learning air combat decision-making method based on situation prediction according to claim 1 is characterized in that: In step 2), the drone environment features include the following steps: S21, the environment status contains the position information of the enemy and our drones (x p ,y p ,z p ); velocity information v; motion information tangent vector n x , normal vector n z , roll angle roll, pitch angle γ, heading angle ψ; reward settings include angle reward reward_angle, distance reward reward_dis and sparse reward reward_spa; S22. Establish a drone confrontation environment through designed actions, states, and rewards.

3. The deep reinforcement learning air combat decision-making method based on situation prediction according to claim 1 is characterized in that: In step 3), the enemy UAV situation assessment includes the following steps: S31. Build an LSTM model to predict the state of the enemy UAV. The model predicts the state of the enemy UAV and the state of our UAV as the model training input; S32. Collect the spiral motion data of the enemy UAV from different angles in advance, and pre-train an LSTM model based on these data sets. This process will generate multiple trajectory data sets as the basis of the pre-trained model. Subsequently, the pre-trained model is integrated into the tracking and strike mission to effectively predict the state of the enemy UAV.

4. The deep reinforcement learning air combat decision-making method based on situation prediction according to claim 1 is characterized in that: In step 4), the segmented reward allocation representation includes the following steps: S41, segment reward configuration: the reward function is: Reward_H=(1+α)·reward_angle+(1-α)·reward_dis+reward_spa Where α is the weight factor, reflecting the importance of the distribution, and its expression is: red_d=reward_dis-reward_p.dis red_a=reward_p.angle-reward_angle Among them, the k value controls the growth rate of the function, reward_p.dis and reward_p.angle represent the distance reward and angle reward of the previous moment respectively; S42. When the UAV performs a situation transition, the reward allocation weight is adjusted using a smooth decay formula; The attenuation formula is: in, and is the smoothing factor.

5. The deep reinforcement learning air combat decision-making method based on situation prediction according to claim 1 is characterized in that: In step 5), the decision maker is used to infer the situation and action of the enemy drone, including the following steps: S51. Initialize the parameters of the actor and critic networks in the SAC algorithm used by the decision maker. In order to maximize the future cumulative rewards and entropy, set the most strategic θ of the SAC algorithm. * for: Among them, S t A is the drone status; t is the action of the drone in time step; ξ θ is the distribution function corresponding to the state and action, τ is the entropy regularization coefficient used to control the importance of entropy, H(θ(·|S t )) represents the entropy value; S52, import the pre-trained LSTM model, receive the relative state of the drone from the reinforcement learning environment, and prepare for model training; S53, determine whether Ang_blue is greater than 1. If so, the two drones are in direct confrontation, and the reward weight distribution needs to be redistributed and the algorithm parameters updated; when the drone is in the tracking and attacking state, the weight gradually decays, and after m environmental steps, the weight returns to the initial value; S54: Save the reward, action and state information, and input them into the model algorithm for training.

Images