Hierarchical Multi-Agent Reinforcement Learning Method for Multi-Factor Joint Accusation

Abstract

The present invention discloses a hierarchical multi-agent reinforcement learning method for multi-factor joint charge. The method models the state space, action space and reward mechanism of reinforcement learning, and conducts research and design respectively in a multi-agent environment. The idea of Actor-Critic network is integrated into the multi-agent environment to form an intelligent decision-making framework with top-level planning and bottom-level execution. The multi-level structure is used in the model to learn strategies at different levels to improve the model's ability to solve complex problems, realize parallel training of the bottom level, and thus realize autonomous action planning of the agent.

Description

Technical Field

[0001] The present invention belongs to the technical field of military confrontation, and specifically relates to a hierarchical multi-agent reinforcement learning method for multi-factor joint command. Background Art

[0002] With the rapid development of artificial intelligence, AI has evolved from saving manpower and time, improving simulation efficiency, and supporting complex simulations to creating intelligent blue forces and supporting staff in command and confrontation scenarios. The so-called intelligent blue force allows the system to act as a designated "blue force" or "green force" in exercises. Through continuous deep learning and reinforcement learning on limited sample data, the opponent becomes more professional and more similar to the hypothetical blue force, thus achieving local and even global human-machine confrontation. Reinforcement learning, a recently emerging deep learning technique that differs from unsupervised learning, presents three major practical difficulties: Setting the reward function is particularly difficult in complex scenarios because it relies solely on the developer's judgment of the game situation and analysis of the rules; due to the complexity of the environment and the generally large dimensionality of the agent's state space, the agent can only obtain rewards from a small number of actions during actual training; and for large-scale reinforcement learning problems, the network can be very unstable, making the results difficult to reproduce. Summary of the Invention

[0003] The purpose of the present invention is to overcome the above-mentioned defects and provide a hierarchical multi-agent reinforcement learning method for multi-factor joint accusation. The state space, action space and reward mechanism of reinforcement learning are modeled, and research and design are carried out separately in a multi-agent environment. The idea of the Actor-Critic network is integrated into the multi-agent environment to form an intelligent decision-making framework with top-level planning and bottom-level execution. A multi-level structure is used in the model to learn strategies at different levels to improve the model's ability to solve complex problems, realize parallel training of the bottom layer, and thus realize autonomous action planning of the agent.

[0004] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0005] A hierarchical multi-agent reinforcement learning method for multi-factor joint charge control includes the following steps:

[0006] S1 presets the agent deduction environment;

[0007] S2 builds an agent reinforcement learning model for the agent deduction environment, which takes state as input and action as output. It then uses a reward mechanism to train the agent reinforcement learning model and obtain the optimal strategy for the agent reinforcement learning model.

[0008] The agent reinforcement learning model adopts an Actor-Critic network architecture, including a discriminant layer and a decision layer corresponding to each agent. Each decision layer is used to predict the agent's action based on the agent's current local observation state, and output it to the agent's deduction environment to obtain a new overall state and input it into the discriminant layer. The discriminant layer provides feedback to the decision layer based on the current overall state, the new overall state and the reward mechanism, and updates the decision layer strategy.

[0009] Furthermore, in step S2, the state space of the input agent reinforcement learning model includes one or more of the agent's current capture point score, the agent's terrain type, the agent's observation mode, or the remaining ammunition count;

[0010] The state types and state intervals in the state space are optimized using the default verification method.

[0011] Furthermore, in step S2, the action space of the agent reinforcement learning model includes one or more simple actions such as moving and shooting; the action space adopts discrete actions or normalized continuous actions, and the method of normalizing the continuous actions is to map the continuous actions to the range [-1, 1];

[0012] When the action space adopts the normalized continuous action space, an activation function corresponding to the normalized continuous action space is provided in the discriminant layer of the Actor-Critic network in step S2.

[0013] Furthermore, in step S2, each action output by the agent reinforcement learning model is abstracted into a function form with parameters:

[0014] Action=p(delay,Unit_Selection);

[0015] Among them, Action represents each action, delay represents delay, which is used to control the effective time of the corresponding action, and Unit_Selection represents all possible target units or target locations of the agent.

[0016] Furthermore, in step S2, the reward mechanism in the agent reinforcement learning model includes one or more of encouragement to shoot, suppression state deduction, health deduction deduction, special terrain deduction, agent with soldiers on board deduction, agent with visibility of the main target point, agent with higher than the main target point, ammunition consumption deduction, or unit loss deduction.

[0017] Furthermore, in step S2, the decision layer includes an input layer, a fully connected layer, a softmax layer and a gumbel layer, and the output of the gumbel layer is connected to the discriminant layer;

[0018] The discriminant layer obtains the evaluation values of the current state and the new state according to the reward mechanism. Based on the current state and the new state evaluation values, the advantage function is used to evaluate the pros and cons of the action predicted by the decision layer. Feedback is given to the decision layer based on the judgment results of the pros and cons of the action, and the policy gradient descent algorithm is used to update the decision layer strategy.

[0019] Furthermore, the advantage function is A(s t ,a)=r(s t )+γv(s t+1 )-v(s t ), where v represents the state value function, s t Represents the current state, s t+1 Represents the new state corresponding to the action predicted by the decision layer, γ is the discount rate, γ = 0.8.

[0020] Furthermore, the decision layer includes a high-level decision maker and a low-level decision maker corresponding to the intelligent agents one-to-one respectively; the high-level decision makers in the discrimination layer and the decision layer are regarded as the high-level decision layer, and the low-level decision makers in the decision layer are regarded as the distributed execution layer. The high-level decision layer and the distributed execution layer make asynchronous decisions. The high-level decision layer executes a decision once within multiple time steps to determine the complex action tasks that each intelligent agent should currently take; the distributed execution layer takes simple actions in the action space through non-fixed time steps to complete the complex action tasks.

[0021] Furthermore, the discriminant layer also includes an experience replay pool, a random sampling module and a target network; the experience replay pool is used to collect historical experience data of the intelligent agent, and the historical experience data includes the state and the action taken; the random sampling module is used to randomly extract historical experience data from the experience replay pool during the training of the intelligent agent reinforcement learning model; the target network is the object to be learned by the decision layer, and the gradient is obtained according to the gap between the target network output and the decision layer output under the same state, and the decision layer is updated according to the gradient.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention proposes a hierarchical multi-agent reinforcement learning method for multi-factor joint accusation. It designs an agent reinforcement learning model that includes a state space, an action space, and a reward mechanism. In combination with the actual situation of the multi-agent environment, the state space, the action space, and the reward mechanism are designed to make the model more in line with actual needs.

[0024] (2) The present invention provides a hierarchical multi-agent reinforcement learning method for multi-factor joint accusation. The idea of the Actor-Critic network is integrated into the multi-agent environment, so that each agent contains an underlying execution framework, which can achieve parallel training. There is only one centralized evaluation layer. After each agent takes different actions, the centralized evaluation layer will be used to evaluate the quality of the current state, effectively improving learning efficiency.

[0025] (3) The present invention provides a hierarchical multi-agent reinforcement learning method for multi-factor joint command, which uses the Option framework to transform long-type tasks into microtasks, abstracts different levels of control layers, and controls the task types of the lower layers by the upper layer, making intelligent decisions from the bottom up, thereby improving the reliability of decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a neural network design diagram for a hierarchical multi-agent reinforcement learning method for multi-factor joint accusation according to the present invention. DETAILED DESCRIPTION

[0027] The following detailed description of the present invention will make the features and advantages of the present invention more clear and explicit.

[0028] The word "exemplary" is used exclusively herein to mean "serving as an example, example, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0029] The purpose of the present invention is to provide a hierarchical multi-agent reinforcement learning method for multi-factor joint command in a multi-agent environment for war game simulation, which solves the problem of non-stationary battlefield situation and exponential growth of troop combination in the existing technology in a multi-agent environment.

[0030] The present invention provides a hierarchical multi-agent reinforcement learning method for multi-factor joint charge, comprising the following steps:

[0031] Step 1: Preset the scenario. For multi-agent wargaming, assume there are various unique map scenarios. Victory in the game is achieved by capturing control points (earning a higher control score) or earning more kill points (killing enemy agents). Each map has n agents scattered across it. Currently, we can obtain the state s of each agent, as well as the overall state S (including time, total score of both teams, etc.). However, due to the fog of war, we can only obtain partial enemy states.

[0032] At the same time, a deduction-based AI interface should be designed for the intelligent agent to obtain the real-time status of the intelligent agent; and the map should be analyzed to determine whether hierarchical reinforcement learning should be used to simplify the current task.

[0033] Step 2: Develop reinforcement learning modeling for the wargame environment, including actions, states, and rewards. This reinforcement learning modeling is based on the characteristics of the wargame environment, including sparse rewards and discrete action environments. A centralized evaluation and distributed training approach is employed to solve the reinforcement learning model.

[0034] Step 3: Model the reinforcement learning network using a hierarchical actor-critic network in a multi-agent environment. Taking into account the characteristics of the multi-agent environment, some reinforcement learning techniques are used to enhance the applicability of the network.

[0035] This invention models the state space, action space, and reward mechanism of reinforcement learning, researching and designing each in a multi-agent environment to optimize the sub-components and achieve the ultimate goal: autonomous action planning in war games. This invention integrates the concept of actor-critic networks into a multi-agent environment, forming an intelligent decision-making framework with top-level planning and bottom-level execution. Each agent includes a bottom-level execution framework, enabling parallel training. The discriminant layer determines the overall strategy, acting as the brain of the team. After receiving overall situational information, decisions are made regarding complex actions such as attacking, taking cover, seizing control points, and where to board or disembark. Upon receiving the decision from the discriminant layer, each agent uses its corresponding decision layer to output specific actions based on the decision. The action space of the bottom-level decision makers in the decision layer is limited to simple actions such as shooting and moving to any nearby square. Finally, the high-level decision makers of each agent evaluate the quality of these actions, and the discriminant layer determines the overall strategy, achieving the effect of hierarchical intelligent decision-making.

[0036] The behavior layer uses the gradient descent method to train according to the current state of the agent. The basic form of the gradient is:

[0037]

[0038] In reinforcement learning, our goal is to maximize the reward function (i.e., find the maximum value), so we generally use the gradient ascent method to update the parameter θ. The gradient value after θ can be regarded as the basic form of the gradient derived to maximize the overall reward. θ (o t ,a t ;θ) state-action probability, v tis the sample return value. The evaluation is updated according to the TD advantage function, which is in the form of:

[0039] Loss = r t +γ·V(s t+1 )-V(s t ) (2)

[0040] where s t is the global state at the current moment, s t+1 is the global state at the next moment, V(S t+1 ) is the state value at the next moment, V(S t ) is the state value at the current moment, γ is the discount rate, r t is the reward at the current moment. The loss function of the centralized discriminant layer is calculated using the above formula.

[0041] Secondly, because the action space of the wargame environment itself is difficult to explore and is based on long time series, the concept of hierarchical reinforcement learning is adopted to transform long-term tasks into microtasks, abstracting different levels of control layers. The upper layers control the task types of the lower layers, making intelligent decisions from the bottom up. Therefore, this invention adopts the concept of hierarchical reinforcement learning based on options, transforming macro decisions into numerous micro decisions, executing decisions from the bottom up and making decisions from the top down, and implementing collaborative optimization of each decision-making layer.

[0042] In general, each intelligent agent in the present invention covers a decision-making layer, namely Figure 1 The low-level and high-level decision makers in the agent are: the decision layer is used to output actions based on the local situation of the agent, and the discriminant layer is used to judge the quality of the current state to obtain gradients and give them to the decision layer for parallel training, and continue to interact and update with the environment. Based on the sparse rewards and long-term task problems in the war game environment, this patent proposes to use the idea of hierarchical reinforcement learning to transform long-type tasks into micro-tasks, abstract different levels of control layers, and have the upper layer control the task type of the lower layer, making intelligent decisions from the bottom up.

[0043] Example 1:

[0044] Driven by the demands of cognitive domain operations, intelligent assistance has become a critical issue in the upgrade of combat simulation systems, one that cannot be ignored and even demands a critical response. Simulation simulation, as a powerful tool for studying warfare, is a crucial decision-making tool for systemic operations. It plays a vital role in improving commanders' strategic capabilities, assisting in combat decision-making, and conducting research and verification of tactics and strategies. Wargames are a primary form of this. The Prophet AI Battle Platform is a wargame simulation platform designed to explore the role of artificial intelligence in combat decision-making through combat simulation and confrontation techniques such as wargames. The overall goal of this design is to develop a tactical-level human-machine confrontation AI system based on a wargame confrontation system platform, encompassing situational analysis, strategy formulation, and action decision-making, within the rules of wargames. First, a real-time situational analysis system based on both sides' troop strength information is developed. This system assesses the overall situation based on the current troop strength comparison and outputs combat effectiveness assessments for both sides, which serve as input for strategy formulation and action decisions. This is the core component of tactical-level human-machine AI. Second, a real-time strategy formulation network based on the overall situational value assessment of both sides is developed. This network can plan the optimal strategy for each branch's current situation given incomplete situational information. Third, adaptive task decision neural network selection based on value assessment is carried out. This neural network selection and design for multi-agent collaborative task decision-making is carried out, capable of adaptively generating optimal task planning and allocation based on changes in global situational information. Fourth, situational analysis and decision neural network model testing and joint debugging are carried out. By constructing training scenarios on the intelligent game simulation platform, the network model's effectiveness is verified. The Prophet Wargame AI Competition incorporates these four key elements, making it an excellent platform for testing AI technology.

[0045] The present invention provides a hierarchical multi-agent reinforcement learning method for multi-factor joint accusation. Figure 1 shown.

[0046] In the design of wargame AI based on the Prophet Bingsheng wargame platform, if reinforcement learning methods are to be used, the design of the state space needs to be as close as possible to the gaming experience of human players when conducting wargame simulations. The state set can mainly be composed of several parts such as intelligent agent attributes and map attributes.

[0047] In the state provided by the Prophet Bingsheng War Chess platform, the following aspects are specifically considered:

[0048] (1) Based on the current state of the control point score, the movement of each agent, as well as the overall team's attack route and attack target are allocated.

[0049] (2) When encountering terrain type data, the agent determines whether it has an offensive advantage based on the terrain type it is in, so as to subsequently command the attack or retreat.

[0050] (3) In observation mode, different choices are made according to different observation modes. For example, in ground-to-air mode, the main purpose is to detect enemy aircraft and conduct air defense; in air-to-ground mode, the primary task is to detect and support ground targets.

[0051] (4) The state of the number of remaining ammunition is even more important to the intelligent agent. Only when there is enough remaining ammunition can its subsequent attack be effective.

[0052] After designing the state space or modifying the original state space, it is necessary to verify through experiments whether it achieves the expected effect. The specific strategies are:

[0053] (1) When a better policy is obtained through training, the effect of each state information is verified in a default manner, that is, other information is input normally, and the target state information is set to a fixed value within a reasonable range (such as the midpoint of the interval).

[0054] (2) Test the percentage of performance loss (e.g., loss of win rate). The greater the loss, the more critical the state information is; conversely, the less critical it is. Sometimes, performance may even increase instead of decrease, indicating that the information is interfering and is usually removed. The purpose of default verification is to eliminate useless or counterproductive states, providing guidance for further optimization of critical and weak states.

[0055] When designing an action space, the first thing to do is to provide the possibility of achieving the desired goal. For example, a tank must be able to accelerate, decelerate, turn, and brake to complete its movement. Action-based multi-agent hierarchical reinforcement learning primarily uses option-based hierarchical reinforcement learning to solve multi-agent reinforcement learning problems.

[0056] In hierarchical reinforcement learning, the following aspects need to be considered during the design of the action space:

[0057] (1) In the underlying strategy, in order to reduce the difficulty of training and improve the performance of the algorithm, the action space should be as simple and efficient as possible. Try to choose a discrete action space, that is, mutually exclusive and exhaustible, so that training will be very convenient.

[0058] (2) In high-level strategies, the design of the action space should be based on a macro perspective, such as determining the current control point to attack, determining the current grouping of agents, etc., so as to reasonably guide the low-level strategy to perform appropriate micro-tasks.

[0059] (3) Even if the intelligent agent itself is a continuous control method, you can try to discretize the interval. If you are worried about the accuracy, you can divide it into smaller parts. This may achieve better performance than continuous control. If you must use a continuous action space, you must pay attention to normalizing the intervals of each dimension, that is, mapping them to [-1, 1], and adding an activation function to the output layer of the policy network to avoid learning difficulties caused by inconsistent actual values. In order to make the action space of reinforcement learning consistent with the action space of the weapon simulation platform itself, the reinforcement learning method designed in this project abstracts various actions into a function form with parameters, that is, different actions are simulated by using different function types with different parameters. It can be simply understood as follows:

[0060] Action=p(delay,Unit_Selection) (3)

[0061] Here represents the primary action, which has 10 types, including moving, shooting, getting on and off the vehicle, etc.; Delay represents the delay, which is used to control the effective time of the corresponding action; Unit selection represents all possible target units or target locations.

[0062] Furthermore, the agent's goal is to maximize the expected cumulative reward. In reinforcement learning, this is manifested as training the agent to implement a reward and penalty mechanism. Correct actions are rewarded, while incorrect actions are penalized. In doing so, the agent attempts to minimize errors and maximize accuracy. Schaul used the DQN to prioritize experience, using a prioritized experience replay technique to achieve efficient learning and design a suitable reward mechanism.

[0063] When designing rewards, we can consider the following points based on actual needs: (1) Encourage shooting; (2) Deduct points for suppression status; (3) Deduct points for health points; (4) Add points for being in special terrain; (5) Deduct points for having soldiers on vehicles; (6) Add points for being able to see the main target point; (7) Add points for being higher than the main target point; (8) Deduct points for ammunition consumption; (9) Deduct points for unit loss.

[0064] After completing the reinforcement learning modeling for the multi-agent environment, the deep decision network of multi-agent reinforcement learning based on war game simulation is constructed. The construction of the deep decision network for war game simulation is mainly divided into two parts: the construction of the Actor network as the decision layer and the construction of the Critic network as the discriminant layer. In the multi-agent environment, since each agent can only obtain local observations (Observation) within a certain range around itself, during training, the Actor selects an action (action) based on the current local observation. The local observation contains relevant information about the environment and the agent itself. Then the centralized Critic can calculate a V value based on the current overall state s as feedback for the joint action of all agents. The Critic is trained based on the error between the estimated V value and the actual V value, corresponding to the sampled reward value v in formula (1) t The output is generated by a dedicated critic network. The actor network updates its policy based on the critic's evaluation. This is a network framework with centralized judgment and decentralized execution. During testing, the actor alone is sufficient; feedback from the critic is not required. During training, the critic phase can incorporate additional information to obtain more accurate Q-values, such as the states and actions of other agents. This is the essence of centralized training: each agent evaluates the value of its current action based not only on its own situation but also on the behavior of other agents. Decentralized execution means that once each agent is fully trained, each actor can independently take appropriate actions based on its state, without needing to know the states or actions of other agents. Furthermore, actions have an impact on the external world, which provides the agent with external rewards as feedback. This feedback acts on the higher-level network. On the higher-level time scale, the reward at each step is the accumulated discounted reward from the external environment during that fixed time step. This cycle continues, and eventually both networks converge to an equilibrium state. This model has the advantage over traditional reinforcement learning methods, such as DQN, in that training and prediction in traditional reinforcement learning methods, such as DQN, are performed by the same network, and the input information for both must remain consistent. We cannot simply incorporate information from other agents during the training phase. Therefore, the network framework of the present invention is more suitable for multi-agent environments. In the present method, the commonly used policy gradient ascent algorithm is used to update the decision-layer network.

[0065] For a specific network structure, the Actor network obtains a specified continuous action by inputting the state. The neural network structure it establishes is as follows: The input of the neural network is O i, refers to the local observation of the agent, and the output of the neural network is a, which represents the situation of each agent's maneuver. Here, the Actor's neural network represents the actor network that actually interacts with the environment. During the learning process, the actual actor network needs to be combined with the critic network for learning. The learning process first passes through the current O i Get the action predicted by the current estimation network. This estimation network represents experience. The specific meaning of experience is that the actions taken, rewards and states obtained in the previous interaction with the environment are not discarded after training. Instead, the network at that time is saved as an indicator of the agent's subsequent interaction with the environment. This can greatly reduce the variance that increases sharply during training due to uncertainty. After obtaining the predicted action, an error is generated with the actual action we take, so the gradient can be obtained. Use the V value of the critic network to update the action based on the gradient of the action. Each agent's action maintains a gradient, and the basic form of its parameter update is:

[0066]

[0067] Among them, Δθ is the parameter of the action selection strategy, α is the learning rate, and p θ (a t |s t ) is the action selection strategy at time t, and Q W (s,a) is the evaluation value of the strategy.

[0068] In the actual algorithm, Q W (s,a) has been modified to some extent, and the concept of advantage function has been introduced. That is, the advantage function A(s,a) is used to evaluate the quality of an action.

[0069] A(s,a)=Q W (s,a)-v π (s t ) (5)

[0070] The purpose of introducing the advantage function is mainly to reduce the variance. The reason for the large variance is that if the agent can obtain rewards in most actions, that is, they are all judged as positive behaviors, then the probability of choosing each behavior will increase, and there will be no requirement that some behaviors can obtain rewards but will be weakened when there are better solutions. Therefore, this invention introduces a baseline, namely v π (s t ), according to the Bellman equation

[0071] v π (s t )=π(a1|s)Q π(s,a1)+π(a2|s)Q π (s,a2) (6)

[0072] That is, the value function v of a certain state π (s t ) is equal to the average value of all action value functions with respect to action probability. π(a1|s) is the probability of an action being executed in a certain state, Q π (s,a1) is the value of this action, π(α2|s) is the probability of another action being executed in a certain state, Q π (s, α2) is the value brought by another action. We get a benchmark action, so we can use the advantage function to evaluate the value of the current action compared to the benchmark action.

[0073] The neural network construction function for the actor network consists of an input layer, a fully connected layer, a softmax layer, and a gumbel layer (the output is equal to the number of actions). The output of the gumbel layer is then connected to the critic. The input value is the state, and the output value becomes the prediction of the discrete action. If computational cost is not a concern, the actor and critic networks can also be built using an RNN structure or an LSTM network.

[0074] After completing the construction of the decentralized execution layer, we need to further optimize the network structure through hierarchical reinforcement learning. When humans solve a complex problem, they often break it down into several easily solvable sub-problems. Because the condition for winning in war games is to achieve the highest score (control points + kill points) at the final moment of the game, a game often lasts 1800 frames, with 50-60 action interactions. If only simple action designs are used, such as the agent moving in four directions, such an action space is extremely unreasonable. The agent cannot know what the overall goal is at the moment, and the environment provides sparse rewards (1 for victory, 0 for failure). There are generally two ways to solve the above problem: first, using data to improve the agent's learning, including existing data and external data; second, improving the model to enhance its ability to handle complex problems in a large state and action space. Here, the present invention uses a multi-level structure to learn different levels of strategies to improve the model's ability to solve complex problems, which is called hierarchical reinforcement learning.

[0075] In summary, according to the multidimensional time scale theory, we can divide the control of the intelligent agent into two layers: high-level and low-level. The high-level and low-level correspond to the high-level decision-making layer and the decentralized execution layer, respectively. The time scales of the high-level and low-level are inconsistent. The high-level executes a choice in multiple time steps, and the decision made by the low-level is completed in multiple time steps.

[0076] In addition, we applied some techniques to enhance the effect of reinforcement learning. First, we applied the experience replay pool (replay buffer). Secondly, we used the idea of Nature DQN, added the experience pool, random sampling and target network, and the true Q value was calculated using two target networks. The collected historical experience was stored in the memory bank, and a batch of data was taken to calculate the mean when updating; and we used a DQN-like architecture and adopted the sliding average method to update the parameters of the target network. The specific update method of the DQN architecture is as follows

[0077]

[0078]

[0079]

[0080] The formula means that the gradient of the action to the parameter in actor(θ) is da / dθ, the gradient of Q to the action in critic is dq / da, and the final gradient formula of the Q value to actor(θ) is -(dq / da*da / dθ) (the reason for the negative number is that the optimizer's direction is to minimize the loss, while our goal is to maximize the Q value). t That is TD-error, r in the formula t Represents the reward value, is the target value function value, Q ω (s t ,a t ) is the actual value function value, and the two then generate gradients for updating. t ,θ t The parameters of both neural networks refer to the network of the intelligent agent that actually interacts with the environment and the neural network of the so-called past "experience" in the experience pool. and They correspond to the gradient values of their respective networks. θ , α ω is the discount rate.

[0081] However, unlike DQN, which directly copies the parameters of the Q network to the target network periodically, we use soft updates to ensure that the parameters can be updated slowly, thereby achieving an effect similar to DQN's periodic parameter copying to improve learning stability.

[0082] In addition, we use batch normalization to address the issue of different input features having different units and data ranges. For example, different operators may choose different action dimensions. Even if the attack type is the same, the maneuver type may be different. An operator can be followed or a route can be set, and the route can be one grid or multiple grids. Obviously, they cannot be treated as the same data. By using different strategies, the differences between the behavioral strategy and the evaluation strategy can increase exploration. Furthermore, in DDPG, inspired by Noisy Net, noise is added to the deterministic strategy of the behavioral strategy to enable efficient "exploration" of the algorithm structure.

[0083] The present invention has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present invention. Those skilled in the art will appreciate that various equivalent substitutions, modifications, or improvements may be made to the technical solutions and implementations of the present invention without departing from the spirit and scope of the present invention, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be determined by the appended claims.

[0084] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A hierarchical multi-agent reinforcement learning method for multi-factor joint command and control, applied to a war game environment, characterized by: The following steps are involved: S1 presets the intelligent agent deduction environment; S2 establishes an agent reinforcement learning model with state as input and action as output for the agent deduction environment, and uses a reward mechanism to train the agent reinforcement learning model to obtain the optimal strategy of the agent reinforcement learning model; the state space input to the agent reinforcement learning model includes one or more of the agent's current capture point score, the agent's terrain type, the agent's observation mode, or the number of remaining ammunition; the state type and each state interval in the state space are optimized using a default verification method; the action space of the agent reinforcement learning model includes one or more of the simple actions of moving and shooting; the action space uses discrete actions or normalized continuous actions, and the method of normalizing the continuous action is to map the continuous action to the range [-1, 1]; The agent reinforcement learning model adopts the Actor-Critic network architecture, which includes a discriminant layer and a decision layer corresponding to each agent. Each decision layer is used to predict the action of the agent based on the current local observation state of the agent, and output it to the agent deduction environment to obtain a new overall state and input it to the discriminant layer. The discriminant layer provides feedback to the decision layer based on the current overall state, the new overall state and the reward mechanism, and updates the decision layer strategy. When the action space adopts the normalized continuous action space, the discriminant layer of the Actor-Critic network is provided with an activation function corresponding to the normalized continuous action space. The decision layer includes a high-level decision maker and a low-level decision maker, each corresponding to an intelligent agent. The high-level decision makers in the discrimination layer and the decision layer are regarded as the high-level decision layer, and the low-level decision makers in the decision layer are regarded as the decentralized execution layer. The high-level decision layer and the decentralized execution layer make asynchronous decisions. The high-level decision layer makes a decision once in multiple time steps to determine the complex action task that each intelligent agent should take at the moment. The decentralized execution layer takes simple actions in the action space in unfixed time steps to complete the complex action task. The Actor network, which serves as the decision layer, selects an action based on the current local observation, which contains information about the environment and the agent itself. The Critic network, which serves as the discriminant layer, calculates a value V based on the current overall state s as feedback for the joint action of all agents. The Critic is trained based on the error between the estimated V value and the actual V value. The Actor network updates its policy based on the evaluation given by the Critic. The policy gradient ascent algorithm is used to update the decision layer network. The Actor network obtains a specified continuous action through the input state; the neural network structure it establishes is as follows: the input amount of the neural network is O i , refers to the local observation of the agent. The output of the neural network is a, which represents the maneuver of each agent. The actor's neural network represents the actor network that actually interacts with the environment. During the learning process, the actor network learns in conjunction with the critic network. The learning process first passes through the current O i Get the action predicted by the current estimation network. This estimation network represents experience. Experience is the action taken, reward, and status obtained in previous interactions with the environment. It serves as an indicator of the agent's future interactions with the environment. After obtaining the predicted action, generate an error with the actual action to obtain the gradient. Use the V value of the critic network to update the action. Each agent's action maintains a gradient, and the basic form of its parameter update is: Among them, Δθ is the parameter of the action selection strategy, α is the learning rate, and p θ (a t |s t ) is the action selection strategy at time t, Q W (s,a) is the evaluation value of the strategy, satisfying: A(s,a)=Q W (s,a)-v π (s t ), A(s,a) is the advantage function, v π (s t )=π(a1|s)Q π (s,a1)+π(a2|s)Q π (s,a2),v π (s t ) is the value function of a certain state The number is equal to the average value of all action value functions with respect to the action probability; π(a1|s) is the probability of an action being executable in a certain state, Q π (s,a1) is the value of this action, π(α2|s) is the probability of another action being executed in a certain state, Q π (s, α2) is the value brought by another action; The Actor network's neural network construction function includes an input layer, a fully connected layer, a softmax layer, and a gumbel layer. The output of the gumbel layer is then connected to the critic. The input value is the state, and the output value becomes the prediction of the discrete action. The discriminant layer also includes an experience replay pool, a random sampling module, and a target network; the experience replay pool is used to collect historical experience data of the intelligent agent, and the historical experience data includes the state and the action taken; the random sampling module is used to randomly extract historical experience data from the experience replay pool during the training of the intelligent agent reinforcement learning model; the target network is the object to be learned by the decision layer, and the gradient is obtained based on the difference between the output of the target network and the output of the decision layer under the same state, and the decision layer is updated according to the gradient; The Nature DQN is added to the experience pool, and the true Q value is calculated using two target networks. The collected historical experience is stored in the memory bank, and a batch of data is used to calculate the average when updating. The parameters of the target network are updated using the sliding average method. The DQN architecture is updated as follows: The gradient of the action to the parameter in actor(θ) is da / dθ, the gradient of Q to the action in critic is dq / da, and the final gradient formula of Q value to actor(θ) is -(dq / da*da / dθ); where δ t is TD-error, r in the formula t Represents the reward value, is the target value function value, Q ω (s t ,a t ) is the actual value function value; ω t ,θ t The parameters of the neural network refer to the actual agent network interacting with the environment and the neural network of past "experiences" in the experience pool. and Corresponding to the gradient value of each network; α θ , α ω is the discount rate.

2. A hierarchical multi-agent reinforcement learning method for multi-factor joint charge according to claim 1, characterized in that: In step S2, each action output by the agent reinforcement learning model is abstracted into a function form with parameters: Action=p(delay,Unit_Selection); Among them, Action represents each action, delay represents delay, which is used to control the effective time of the corresponding action, and Unit_Selection represents all possible target units or target locations of the agent.

3. The hierarchical multi-agent reinforcement learning method for multi-factor joint charge according to claim 1 is characterized in that: In step S2, the reward mechanism in the agent reinforcement learning model includes one or more of the following: encouraging shooting, deducting points for suppression status, deducting points for health deduction, adding points for being in special terrain, deducting points for agents with soldiers on vehicles, adding points for agents that can see the main target point, adding points for agents that are higher than the main target point, deducting points for ammunition consumption, or deducting points for unit loss.

Images