A hierarchical decision-making method based on deep reinforcement learning
By adopting a hierarchical decision-making method in air combat decision-making, and using a deep deterministic policy gradient algorithm and a deep recurrent Q-network to train the tactical decision-making layer and the intent recognition layer respectively, the problems of slow convergence speed and poor interpretability in air combat decision-making are solved, and faster training and better decision-making results are achieved.
Patent Information
- Application Number
- CN202211014161.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-23
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2042-08-23
AI Technical Summary
Existing air combat decision-making techniques based on deep reinforcement learning suffer from slow convergence speed, difficulty in shaping reward functions, and poor interpretability, which affect algorithm efficiency and training difficulty.
A hierarchical decision-making approach is adopted, which includes agents with a tactical decision-making layer and an intent recognition layer. The agents are trained independently using a deep deterministic policy gradient algorithm and a deep recurrent Q-network, respectively, and then interconnected during decision-making to improve the overall decision-making ability of the agents.
It accelerates the convergence speed of the algorithm during training, improves the interpretability of training results, and enables the agent to implement more complex tactical combinations.
Smart Images
Figure CN115392435B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of deep reinforcement learning technology, and in particular relates to a hierarchical decision-making method based on deep reinforcement learning. Background Technology
[0002] Modern fighter jets are developing towards high automation, informatization, and intelligence. The battlefield environment is complex and ever-changing, and the battlefield information is complex and diverse. Relying on pilots to make combat decisions in a short period of time is a heavy burden. There is an urgent need for an intelligent decision support system to assist pilots in making real-time decisions when facing complex battlefield situations.
[0003] Deep reinforcement learning is an artificial intelligence algorithm that does not rely on labeled samples. It learns knowledge through interaction with the environment, continuously training and iterating the model to improve the intelligence level of the decision-making system. Deep reinforcement learning mainly solves sequential decision-making problems, enabling real-time decisions based on current environmental information. It is very suitable for air combat scenarios and is currently a research hotspot. Existing deep reinforcement learning algorithms for air combat applications suffer from problems such as difficulty in shaping reward functions, slow convergence, and poor interpretability. These problems make training very difficult and affect algorithm efficiency.
[0004] In the complex, intense, and rapidly changing battlefield environment, battlefield situation assessment and target tactical intent identification technologies infer the situation through limited information perception, assess the enemy's combat intentions, the level of threat, and the chances of victory of the aircraft, and form basic judgments, which are of great significance for subsequent decision-making.
[0005] The main drawbacks of existing air combat decision-making techniques based on reinforcement learning algorithms are that the large output space leads to slow convergence speed, difficulty in reward shaping, and poor interpretability. Summary of the Invention
[0006] The purpose of this application is to provide a hierarchical decision-making method based on deep reinforcement learning for air combat simulation, and to solve the problems of difficulty in convergence, sparse rewards, and difficult interpretation of results in reinforcement learning algorithm models in air combat simulation.
[0007] To achieve the above objectives, the technical solution of this application is as follows:
[0008] A hierarchical decision-making method based on deep reinforcement learning includes:
[0009] Initialize the tactical decision-making layer agent and the intent recognition layer agent of the decision-making object;
[0010] The intent recognition agent uses a deep recurrent Q network to generate intent recognition layer behavior based on the input intent recognition observation information, and selects tactical decision layer agents based on the intent recognition layer behavior;
[0011] The tactical decision-making layer agent uses a deep deterministic policy gradient algorithm to calculate the agent's behavior based on tactical decision-making observation information;
[0012] The decision-making object executes the behavioral instructions output by the tactical decision-making layer agent and updates the decision-making object's state information.
[0013] Furthermore, the intent recognition observation information includes the state of the observed object at a preset discrete time length T, and the state of the observed object at a single moment includes:
[0014] The observation includes at least one of the following: relative distance between observation objects, relative angle between observation objects, heading angle of observation objects, survival status of observation objects, radar status of observation objects, weapon status of observation objects, and motion status of observation objects.
[0015] Furthermore, the number of agents in the tactical decision-making layer is the same as the number of actions of the intent recognition agent.
[0016] Furthermore, the tactical decision observation information includes the current state of the decision-making object and the state of the decision-making object with other observation objects, including at least one of the following: the relative distance between the decision-making object and other observation objects at the current time, the relative angle between the decision-making object and other observation objects at the current time, the heading angle between the decision-making object and other observation objects at the current time, the radar state of the decision-making object at the current time, the weapon state of the decision-making object at the current time, the survival status of the decision-making object and other observation objects at the current time, and the motion state of the decision-making object and other observation objects at the current time.
[0017] This application proposes a hierarchical decision-making method based on deep reinforcement learning. The technical solution adopted is divided into two parts: a deep reinforcement learning algorithm for agent tactical decision-making and a deep reinforcement learning algorithm for intent recognition. The former is used for the agent's bottom-level tactical decision-making, which is a decision in a continuous behavior space, while the latter is used for the agent's upper-level intent recognition, which is a decision in a discrete behavior space. The two iterate with each other to accelerate the convergence speed and thus improve the overall decision-making ability of the agent. The advantages of this scheme are: (1) The tactical decision-making layer and the intent recognition layer are independent of each other during training, but interconnected during decision-making, avoiding the problem of difficulty in convergence during simultaneous training and accelerating the convergence speed of the algorithm during training; (2) Dividing air combat decision-making into intent layer and tactical layer is more in line with human cognition, and the training results have good interpretability; (3) The hierarchical algorithm splits the problem into two interconnected sub-problems, making it possible for the agent to realize more complex tactical combinations. Attached Figure Description
[0018] Figure 1 Here is a flowchart of the hierarchical decision-making method based on deep reinforcement learning proposed in this application;
[0019] Figure 2 This is a flowchart of the hierarchical decision-making process for this application. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0021] In reinforcement learning, an agent learns knowledge through interaction with its environment. In this application, the agent's interaction with the environment involves the intent recognition agent acquiring observation information (obs) from the environment. IA Calculate and output a IA According to a IA Select a tactical decision-making agent; the tactical decision-making agent acquires observation information (obs) from the environment. DA The calculated behavior a DA The information is then fed back to the environment. The intent-recognition agent influences the environment by switching between tactical decision-making agents, which in turn interact directly with the environment. Through continuous training, the intent-recognition agent can maximize the reward function and intelligently switch between various tactical models based on situational information, thereby achieving strategy scheduling. This can be applied to hierarchical decision-making in air combat and also to intelligent operation of game AI. The following embodiments use air combat as an example for illustration.
[0022] In one embodiment, such as Figure 1 As shown, this application proposes a hierarchical decision-making method based on deep reinforcement learning, comprising:
[0023] Step S1: Initialize the tactical decision-making layer agent and the intent recognition layer agent of the decision object.
[0024] In this embodiment, environment initialization mainly involves initializing and constructing two types of intelligent agents: tactical decision-making layer agents and intent recognition layer agents. For example, the initialization yields a set of tactical decision-making layer agents as [DA0, DA1, ..., DA1]. n-1 Initialize the intent recognition layer agent IA. The construction of the tactical decision layer agent is related to tactics, and the number of tactical decision layer agents constructed is the same as the number of behaviors of the intent recognition layer agent.
[0025] The intent recognition agent identifies the enemy's intent and outputs its own intent recognition layer behavior (or tactics), including pursuit, tailing and evasion, lateral guidance, ascent, descent, and attack. Based on the output of the intent recognition agent, the appropriate tactical decision-making agent is selected. The tactical decision-making agent outputs decision-making instructions based on its own tactics, such as maneuver instructions, attack instructions, jamming switch instructions, and sensor switch instructions.
[0026] For example, tactics mainly include pursuit, tail-to-tail evasion, lateral guidance, ascent, descent, and attack. The outputs of the tactical decision-making agent mainly include maneuver commands (target waypoints), attack commands, jamming switch commands, and sensor switch commands.
[0027] It should be noted that the tactical decision-making layer agent and the intent recognition layer agent are both set on the decision-making object that needs to make a decision. For example, in air combat, they are set on our own aircraft to make decisions about our own aircraft. The decision-making object is our own aircraft.
[0028] Step S2: The intent recognition agent uses a deep recurrent Q-network to recognize the observed information obs based on the input intent. IA Generate intent recognition layer behavior a IA According to the intent recognition layer behavior a IA Select the tactical decision-making layer agent.
[0029] The observation information is obtained from real-time situational observations of the air combat environment. In the simulation environment, it is obtained through API data interfaces, while in the real battlefield environment, it is obtained through a combination of aircraft sensors, ground radar, etc.
[0030] Assume the set of actions of the agent for intent recognition is: pursuit, tail-avoidance, lateral guidance, elevation, descent, and attack. IA A value of 0 indicates that the intent-layer tactic is pursuit, which is equivalent to selecting the tactical decision-making agent corresponding to pursuit. The decision output of the tactical decision-making agent includes specific maneuver routes, jamming switches, sensor switches, and weapon doctrines during the pursuit process.
[0031] a IA A value of 1 indicates that the intention-level tactic is tail-keeping evasion, which is equivalent to selecting the tactical decision-making agent corresponding to tail-keeping evasion. The decision output of the tactical decision-making agent is the specific maneuver route, jamming switch, sensor switch, and weapon doctrine during the tail-keeping evasion process.
[0032] This step is used to implement intent recognition, employing a Deep Recurrent Q-Learning Network (DRQN) to analyze the intent recognition observation information (obs). IA Generate intent recognition layer behavior a IA .
[0033] Deep Recurrent Q-Learning Network (DRQN) is a novel artificial intelligence algorithm that combines deep learning and deep Q-network reinforcement learning. In air combat situation assessment, after obtaining a large amount of situational advantage information, the problem can be described as: based on the input information, using a network to learn its representation, thereby providing targeted decisions.
[0034] The changing air combat situation is a continuous process, and information from a single moment is insufficient to describe the situation's advantage or disadvantage. To address this issue, the DRQN algorithm replaces the fully connected layers in DQN with LSTM networks. When training the model using partial observation data and evaluating it using complete observation data, the model's performance is related to the completeness of the observation data; when training with complete observation data and evaluating with partial observation data, the performance drop of DRQN is less than that of DQN. Recurrent networks exhibit greater adaptability to changes in observation quality.
[0035] DQN's network structure cannot learn from time-series sequences, while DRQN can use an n-second situation to replace the previous 1-second situation as a state for learning and decision-making. Since the current state is not only the state of the previous time step, the entire system loses its Markov property. Similarly, its reward is not only related to the current situation information, but also to the situation over a number of previous time steps.
[0036] Air combat situational information may be partially observable in some cases, transforming a partially observable MDP (Markov Decision Process) into a partially observable Markov Decision Process (POMDP). For partially observable Markov Decision Processes, Recurrent Neural Networks (RNNs) perform better in handling incomplete observations, with DQRNs handling missing information better than DQNs.
[0037] The key challenge of DRQN is fitting a suitable network value function Q(s,a; θ) to evaluate the quality of actions. DRQN modifies the DQN structure only in its network construction, replacing the fully connected layer after the input layer with an RNN network, such as LSTM or GRU. The final output is the Q(s,a) value for each action a. During training, the fully connected layer, output layer, and recurrent neural network layer are updated and iterated together. DRQN uses a recurrent neural network as a feature extraction tool; the input for each iteration is temporal sequence situational information, and the output is the Q value for each action. The DRQN network structure includes an input layer, an LSTM / GRU layer, a fully connected layer, and an output layer.
[0038] The input situational information for DRQN is a time-series sequence, representing the states of observed objects related to the intent recognition layer agent in the environment over the past T historical moments. Selecting appropriate situational information is beneficial for improving the model's learning speed. In this embodiment, the intent recognition observation information includes the states of observed objects at a preset discrete time length of T. The state of the observed object at a single moment includes:
[0039] The observation includes at least one of the following: relative distance between observation objects, relative angle between observation objects, heading angle of observation objects, survival status of observation objects, radar status of observation objects, weapon status of observation objects, and motion status of observation objects.
[0040] Let the current time be t, then the situation information can be described as: [x t-T+1 ,x t-T+2 ,…,x t-1 ,x t Assuming the observed objects are m red team aircraft and n blue team aircraft, then the state vector x collected at a single moment is... i It contains the following elements:
[0041] Relative distance between aircraft
[0042] Relative angles between aircraft
[0043] m+n aircraft heading angles
[0044] The status and flight time of all missiles among m+n aircraft
[0045] m+n aircraft speeds
[0046] m+n aircraft detection radar status
[0047] m+n aircraft electro-optical radar states
[0048] m+n aircraft chaff states
[0049] m+n aircraft survival status
[0050] Combining the state vectors from T time points yields the final selected situational information: [x t-T+1 ,x t-T+2 ,…,x t-1 ,x t Considering that pilots in actual flight will comprehensively consider situational information over a period of time and continuously observe the battlefield before making a situational judgment, the length T of the input data time series is generally greater than 1. The observation information in this embodiment includes the states of all observed objects in the application scenario, and includes the states at T time points.
[0051] In this embodiment, the decision-making of the intent recognition agent is relatively sparse. Therefore, the results of the battle can be directly used as the basis for optimizing the intent recognition agent. The reward function is designed as follows:
[0052]
[0053]
[0054]
[0055]
[0056] The red team has m aircraft, the blue team has n aircraft, and R r The reward representing the Red Team's aircraft, R b The reward representing the blue team's aircraft, s i Let α1, α2, and α3 be the survival reward parameters for the i-th aircraft, which can be adjusted. If an friendly aircraft is shot down, there is a penalty; if an enemy aircraft is shot down, there is a reward; if the round ends and the player wins, there is a reward; if the round ends and the player loses, there is a penalty.
[0057] Training for air combat intent recognition based on the DRQN algorithm: The decision actions of the intent recognition agent are discrete actions. The DRQN algorithm uses LSTM and a fully connected network to construct a network Q(s,a;θ), and trains the neural network by optimizing the time difference error function. A probabilities ∈ [0, 1] are selected according to the tactical action a. t Or probability 1-∈ according to a t =argmax a Q(s,a;θ) selects action a t A dual-network structure is adopted, with the Q network and... The network's model structure and initial parameters remain consistent. During iterative training, the Q-network's model parameters are updated in real time, and at regular intervals, the Q-network model parameters are assigned to... Network model, The network is used to calculate the time difference error. The input situation information of the Q(s,a;θ) network is the situation sequence.
[0058] A deep reinforcement learning algorithm for air combat intent recognition enables tactical switching through situation assessment. Its aim is to solve the agent's intent recognition problem by training tactical decision-making and intent recognition separately, thereby accelerating agent convergence and addressing the reward sparsity problem. Using temporal sequence situational information as input, a recurrent neural network evaluates the value of actions and selects the action with the highest value. Each action corresponds to a tactical decision, thus determining the tactical decision-making agent corresponding to that tactic.
[0059] Step S3: The tactical decision-making layer agent adopts a deep deterministic policy gradient algorithm based on the tactical decision observation information obs. DA Calculate the behavior of the tactical decision-making agent a DA .
[0060] Tactical decision-making observation information is derived from real-time situational observation of the air combat environment. In the simulation environment, it is obtained through API data interface, while in the real battlefield environment, it is obtained through a combination of aircraft sensors, ground radar, etc.
[0061] This step is used for tactical decision-making. The Deep Deterministic Policy Gradient (DDPG) algorithm is employed to select the output instructions for the tactical decision-making agent units, enabling tactical game-theoretic decision-making in a continuous action space. The model input is the combat situation observed by the tactical decision-making agent in its current state, and the output is the decision action that the tactical decision-making agent needs to execute.
[0062] Taking air combat as an example, the tactical decision observation information in this embodiment includes the current state of the decision-making object and the state of the decision-making object and other observation objects, including at least one of the following: the relative distance between the decision-making object and other observation objects at the current time, the relative angle between the decision-making object and other observation objects at the current time, the heading angle between the decision-making object and other observation objects at the current time, the radar state of the decision-making object at the current time, the weapon state of the decision-making object at the current time, the survival status of the decision-making object and other observation objects at the current time, and the motion state of the decision-making object and other observation objects at the current time.
[0063] Assuming the observed objects are m red team aircraft and n blue team aircraft, the combat situation includes the following elements:
[0064] The relative distance between this aircraft and m+n-1 other aircraft at the current moment;
[0065] The relative angles between this aircraft and m+n-1 other aircraft at the current moment;
[0066] The heading angles of all m+n aircraft at the current moment;
[0067] The current status and flight time of all missiles on this aircraft;
[0068] Current status of the aircraft's detection radar;
[0069] Current status of the aircraft's electro-optical radar;
[0070] Current status of the aircraft's chaff and decoys;
[0071] The current speeds of m+n aircraft at the current moment;
[0072] The current status of m+n aircraft.
[0073] During the interaction between the adversarial simulation environment and the agent, the environment returns the state and rewards to the agent, while the agent returns its decision-making actions to the simulation environment and updates the aircraft's state. Through the interaction between the agent and the environment, decisions are made to control aircraft, missiles, radar, and other systems.
[0074] The reward function is the standard by which an agent judges the quality of its own behavior, and constructing a reasonable reward function R(s) is a key aspect of reinforcement learning. The reward function should comprehensively consider realizability and sparsity, achieving a balance between tactical implementation and exploration. To implement a basic tactical library, the designed reward functions include, but are not limited to: pursuit, evasion, tangential evasion, guidance, and cross-attack tactics. The following uses pursuit tactics as an example to illustrate the process of shaping the reward function. Assuming the pursuing agent is the reference frame and the relative direction angle of the pursued agent is θ, where -180≤θ≤180, the reward function R can be described as:
[0075] R(θ) = α*(90 - abs(θ)) / 90
[0076] Here, α is a constant that can be adjusted as needed. The reward function is not uniquely shaped; for numerical optimization purposes, the distribution of positive and negative rewards should ideally be balanced within the range of the independent variable.
[0077] This application constructs a continuous spatial decision-making model for air combat tactics based on the DDPG algorithm to make decisions on the actions of tactical decision-making agents. The training process of the agent's game-theoretic decision-making using the DDPG algorithm is described below.
[0078] 1. Initialize the critic network Q(s,a|θ) Q ) and actor network μ(s|θ μ Weight
[0079] 2. Initialize the weights θ of the target network Q′ and μ′. Q ′ and θ μ ′, Experience Pool R
[0080] 3. Initialize the random process Take the initial state observation s0
[0081] 4. Update step by step until s reaches the termination state.
[0082] (1) Select action a based on the current strategy and noise exploration. t
[0083] (2) Execute action a t and receive rewards from the environment. t and observations s t+1
[0084] (3) (a) t ,s t ,r t ,s t+1 Store in experience pool R
[0085] (4) Randomly sample small batches of data from the experience pool R.
[0086] (5) Update the actor and critic networks by optimizing gradient calculation.
[0087]
[0088] (6) Update the target network parameters:
[0089] θ Q′ ←τθ Q +(1-τ)θ Q′
[0090] θ μ′ ←τθμ+(1-τ)θ μ′
[0091] During initial training, steps 1 and 2 are executed. At the start of each battle, step 3 is executed. Then, during that battle, step 4 is executed repeatedly until the model training is complete. Here, state s... t a represents the operational situation observed by the tactical decision-making agent in the current state, i.e., the observations. t For the decision-making actions performed by the tactical decision-making agent under the current combat situation, r t The immediate reward s received by the tactical decision-making agent after executing a decision-making action. t+1 This refers to the operational situation returned to by the tactical decision-making agent after executing a decision-making action and entering the next state environment. The tactical decision-making agent interacts with the air combat environment through a series of s, a, and r, with the goal of maximizing cumulative reward. The air combat agent is in the current state s, and then calculates the decision action a according to the policy μ = μ(a|s). t , will make decision action a t This process continues in the environment until the battle ends. The optimal action value function is calculated as follows:
[0092]
[0093] The training of tactical decision-making models based on the DDPG algorithm has the following characteristics:
[0094] (1) The decision-making actions of the air combat agent are continuous actions. The algorithm uses a deep neural network to construct a critic network Q(s,a|θ). Q ) and actor network μ(s|θ μ ).
[0095] (2) The critic network is used to evaluate the quality of the current behavior. The critic network is trained by optimizing the time difference error function. The actor network is used to calculate the decision behavior. The actor network is trained by maximizing Q.
[0096] (3) In the early stages of training, the agent first accumulates a memory bank. The training data is stored in the memory. When the data in the memory reaches a certain capacity, the agent samples data from it and updates the model, with each sample having the same sampling probability. Simultaneously, the memory... It's also being updated, the memory bank. Only the latest N training data sets will be stored, while older data will be deleted.
[0097] (4) A dual-network structure is adopted, with the Q network and... The network model structure and initial parameters remain consistent. The Q-network model parameters are updated independently, and then the Q-network model parameters are weighted and assigned to... Network model. This involves small-scale updates. This method makes the model easier to converge.
[0098] After the tactical decision-making model is trained, a decision model is generated. Each tactic corresponds to a specific model, and these models are combined to form a library of selectable models. The intent recognition layer can easily call upon these models to switch between them and make hierarchical decisions.
[0099] Step S4: The decision-making object executes the behavioral instructions a output by the tactical decision-making layer agent. DA Update the status information of the decision object.
[0100] An aircraft is an individual in the real physical world or a simulated world, and an intelligent agent is an algorithmic model that provides behavioral decisions for the aircraft. The behavioral decisions for the aircraft are jointly achieved through the outputs of tactical and intentional intelligent agents.
[0101] As mentioned above, the intent recognition agent identifies the enemy's intent and outputs its own tactics (tactics include pursuit, tail-to-tail evasion, lateral guidance, ascent, descent, attack, etc.). The tactical decision-making agent outputs decision commands based on its own tactics, such as maneuver commands, attack commands, jamming switch commands, sensor switch commands, etc. The commands output by the tactical decision-making agent are the decision objects, that is, the commands that the aircraft ultimately needs to execute. The aircraft updates itself according to these commands. For example, if the waypoint output by the tactical decision-making agent is [1000,1000,1000], then the aircraft will fly towards the point [1000,1000,1000]. If the decision command output by the tactical decision-making agent is to turn on jamming, then the aircraft will turn on jamming.
[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A hierarchical decision-making method based on deep reinforcement learning, characterized in that, The layered decision method based on deep reinforcement learning is used for air combat confrontation simulation deduction, and comprises the following steps: Initializing a tactical decision layer agent and an intention recognition layer agent of a decision object; An intention recognition agent generates an intention recognition layer behavior according to inputted intention recognition observation information by using a deep recurrent Q network, and selects a corresponding tactical decision layer agent according to the output of the intention recognition agent; A tactical decision layer agent calculates a behavior of the tactical decision layer agent according to tactical decision observation information by using a deep deterministic policy gradient algorithm; The decision object executes a behavior instruction outputted by the tactical decision layer agent, and updates state information of the decision object; The intention recognition observation information comprises state of an observation object at a preset discrete time length T, and the state of the observation object at a single time comprises at least one of relative distance between observation objects, relative angle of observation objects, heading angle of observation objects, survival state of observation objects, radar state of observation objects, weapon state of observation objects and motion state of observation objects. The tactical decision observation information comprises state of the decision object at a current time and state of the decision object and other observation objects, and comprises at least one of relative distance between the decision object and other observation objects at the current time, relative angle of the decision object and other observation objects at the current time, heading angle of the decision object and other observation objects at the current time, radar state of the decision object at the current time, weapon state of the decision object at the current time, survival state of the decision object and other observation objects at the current time, and motion state of the decision object and other observation objects at the current time. The number of the tactical decision layer agents is the same as the number of behaviors of the intention recognition agent.
2. The deep reinforcement learning based hierarchical decision method according to claim 1, characterized in that,