Agent Task Allocation Method Based on Deep Reinforcement Learning

Abstract

This invention discloses a method for agent task allocation based on deep reinforcement learning, comprising the following steps: design of a strong and weak agent architecture; design of a Markov decision process for cooperative behavior; design of a phased reward mechanism; design of the agent network structure; and design of a multi-head attention mechanism and target selection. This invention proposes a "one strong agent leading N weak agents" multi-agent structure, which can reduce system complexity and eliminate the weakness of multi-agent systems prone to interaction conflicts when dealing with complex problems. Through the multi-head attention mechanism and the phased reward mechanism, the efficiency and stability of training are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning, specifically to a method for agent task allocation based on deep reinforcement learning. Background Technology

[0002] In large-scale game-based adversarial scenarios, intelligent agents need to handle many concurrent task assignments and random events, making the entire situation full of complexity and uncertainty.

[0003] Hang's "Application of Markov Decision Process in Target Assignment" (Journal of Harbin Institute of Technology, 1996, 28(2): 32-36.) argues that dynamic weapon target assignment is divided into two stages: strategy optimization and matching optimization, and that the dynamic nature of Markov can be used to solve dynamic weapon target assignment. Chen's "An Improved Algorithm of Policies Optimization of Dynamic Weapon Target Assignment Problem" (System Engineering Theory and Practice, 2007, (7): 160-165.) builds upon this by improving the hybrid iterative method based on strategy iteration and value iteration in Markov process strategy optimization to solve large-scale weapon target assignment problems. He's "Finite Stage MDP for Task Allocation in UCAVs Cooperative Control" (Fire Control & Command Control, 2012, 37(10): 99-101) transforms the task allocation problem into a phased decision process using Markov decision processes. This method has good results in smaller-scale optimization problems.

[0004] In China, Xu Hao's "Air Defense and Anti-missile Firepower Allocation Based on Fuzzy Multi-Objective Programming" (Systems Engineering and Electronics Technology, 2017, (12): 1-8.) addresses the problem that existing multi-objective allocation methods are difficult to apply to air defense and anti-missile operations under uncertain conditions, and uses fuzzy multi-objective programming to construct a task allocation model. Zhang Jun's "Resource Allocation and Optimization of Air Defense and Anti-missile Equipment System under Multi-Objective and Multi-Decision-Maker Environment" (Journal of National University of Defense Technology, 2015, 37(1): 171-178.) introduces risk management on the basis of hierarchical multi-objective analysis method and proposes a task allocation model with anti-missile as the background.

[0005] Mnih's "Human-Level Control through Deep Reinforcement Learning" (Nature, 2015, (518): 529-533) proposed the Deep Q Network (DQN), which for the first time combined Convolutional Neural Networks (CNN) in deep learning with Q-learning algorithms in reinforcement learning. Building on this, Schaul's "Prioritized Experience Replay" (Computer Science, 2015) proposed a DQN based on a priority experience replay mechanism, significantly improving the learning efficiency of the DQN network by increasing the frequency of resampling important transition processes. To address the problem of excessive estimation errors in Q-learning with large-scale data, Van's "Deep Reinforcement Learning with Double Q-Learning" (Computer Science, 2016) proposed a double-DQN, which effectively avoids overestimation and improves the stability of policy training by applying two sets of Q-networks within the DQN framework to select and evaluate actions respectively. Silver's "Mastering the game of Go without human knowledge" (Nature 2017, (550): 354-359) shows that AlphaGo defeated top human Go players by updating the network using only training data generated through reinforcement learning. It does not rely on a large amount of human sample data, requires no human guidance, and discovers many Go patterns that humans have never explored before. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a task allocation method for intelligent agents based on deep reinforcement learning, specifically including the following steps:

[0007] Step 1. Strong and Weak Agent Architecture Design

[0008] Within a certain timeframe, the global situation, referred to simply as "state," serves as the input for a strong agent with superior computing power, enabling it to obtain combat missions. A weak agent, based on tactical rules, decomposes the strong agent's combat missions according to its own situation, forming "actions" to be executed, and then outputs specific instructions for those actions to the simulation environment. After the weak agent makes a decision based on the instructions, the situation of the entire "simulation environment" changes accordingly, generating "environmental data." This "environmental data" is then transformed into "states" according to the data structure required by the strong agent, serving as the input for the strong agent.

[0009] Step 2. Design of Markov Decision Processes for Collaborative Behavior

[0010] Let the Markov process MDP have four elements (S, A, r, p): state S, action A, reward function r, and transition probability p; Markov property: p(s) t+1 |s0, a0, ..., s t , a t )=p(s t+1 |s t , a t ); Policy function π: S→A; s t Let a be the state at time t. t The action at time t;

[0011] Optimization objective: To find the optimal policy function π. * Maximize the expected cumulative reward value:

[0012]

[0013] sts t+1 ~p(·|s t , a t ), a t ~π(·|s t ), t=0,...T-1 (1)

[0014] In the formula, γ, r t T represents the reward decay coefficient, the reward at time t, and the total time, respectively, and the superscript t indicates the power exponent;

[0015] In p(s) t+1 |s t , a t Solving Markov decision processes under unknown conditions involves using a time-difference algorithm to estimate the action-value function Q. π (s, a):

[0016]

[0017] Q π(s t , a t )←Q π (s t , a t )+α[r(s t , a t )+γmax a Q π (s t+1 ,a)-Q π (s t , a t (3)

[0018] π′(s)=arg max a Q π (s, a) (4)

[0019] In the formula, s refers to the state in general, a refers to the action in general, and r(s) t , a t ) represents the state s at time t. t Execute action a under the following circumstances t The reward obtained, π′(s) represents the optimal policy in state s, and argmax is the maximum reward. a The set of independent variable points representing the maximum value of action a;

[0020] Step 3. Design of a phased reward mechanism

[0021] The phased reward mechanism adopts the method of breaking down task objectives and giving reward values ​​in stages to guide the neural network to find a winning strategy. On this basis, the reward function is optimized according to different objectives in actual tasks, and minimizing interception resource consumption is also used as a reward indicator to increase the effect of maximizing global benefits on the agent's benefits.

[0022] Step 4. Agent Network Structure Design

[0023] The situational input data is divided into four categories: the first category is the status of the Red Team's base; the second category is the status of the Red Team's radar and interceptors; the third category is the status of the Blue Team's units; and the fourth category is the status of Blue Team units that can be attacked by the Red Team. The number of units in each category is not fixed and varies with the battlefield situation.

[0024] Each type of situational data is extracted using two fully connected linear rectified loop (FC-ReLU) layers. Then, all feature vectors are merged, and the feature vectors are predefined. Multiple feature vectors are connected and merged using the concat function to form an array. Then, a global feature is generated using an FC-ReLU layer and a gated recurrent unit (GRU). The GRU introduces the concepts of reset gate and update gate to modify the calculation method of hidden states in the network, which can solve the gradient vanishing problem in neural networks.

[0025] When making decisions, neural networks need to consider both the current situation and historical information. They need to continuously interact with the global situation through GRU to choose to retain or forget information. The global features and the feature vectors of the selectable blue team units are processed through an attention mechanism to select the interception unit. Each interception unit then selects the interception timing and enemy unit based on its own state and a pre-designed rule base through attention operations.

[0026] Step 5. Multi-head attention mechanism and goal selection design

[0027] The decision-making actions are processed by multiple heads as the output of the network, that is, the actions are divided into action subject, action predicate, and action object;

[0028] When selecting interception targets, the neural network needs to focus on some important targets in a local area; the feature vectors of each fire unit state and the incoming target are calculated using an additive model to implement the attention mechanism; the "feature vectors of each fire unit state and the incoming target" are obtained from the "state" data, which is converted from the "simulation environment data";

[0029] Define X = [x1, ..., x2] N [x] is a vector formed by N input information. i Let i be the i-th incoming target that can be selected, i = 1, 2, ..., N. First, calculate the probability a of selecting the i-th target given the query vectors q and X. i , a i This is called the attention distribution, then we define a i for

[0030]

[0031] In the formula, s(x i Let q) be the attention scoring function, representing the score for selecting the i-th incoming target given the query vector q. The higher the score, the greater the probability that the i-th incoming target will be selected. An additive model is chosen for the calculation.

[0032] s(x i ,q)=v T tanh(Wx i +Uq) (7)

[0033] In the formula, the query vector q is the feature vector of each fire unit, W and U are the first and second trainable neural network parameters, v is the global situation feature vector, which is the conditional attention mechanism that allows global situation information to participate in the calculation, and tanh is the hyperbolic tangent function. The attention score of each fire unit with respect to each target is obtained, and sigmoid sampling is performed on each bit of the score vector to finally generate the overall decision.

[0034] In one embodiment of the present invention, in Step 3, a phased reward can be given all at once after successfully resisting the first attack; a corresponding reward value can be given after the blue side loses a high-value unit; and a winning reward value can be given after the red side wins.

[0035] This invention is based on a reward function that minimizes resource usage:

[0036]

[0037] In the formula, m represents the number of blue force manned units intercepted, n represents the number of blue force drones intercepted, and i represents the number of missiles launched.

[0038] In a specific embodiment of the present invention, in Step 4, the Red Force base status includes basic information about key locations and the status of key locations under attack; the Red Force radar status and interceptor status include the current configuration of fire units, the working status of radars, the working status of launch vehicles, the status of radars under attack, and information on enemy units that the fire units can strike; the Blue Force unit status includes basic information about Blue Force units and the status of Blue Force units being attacked by Red Force missiles; the status of Blue Force units that can be attacked by the Red Force includes the status of units that can be struck by Red Force fire units.

[0039] In another embodiment of the present invention, in Step 5, the subject of the action is to select which interception units, the predicate of the action is to select which launcher to intercept at what time, and the object of the action is to select which enemy targets.

[0040] This invention proposes a "one strong agent leading N weak agents" multi-agent structure, which reduces system complexity and eliminates the weakness of multi-agent systems prone to interaction conflicts when handling complex problems. Addressing the characteristics of large-scale task allocation problems, an improved Proximal Policy Optimization (PPO-TAGNA) algorithm is proposed. Through multi-head attention and phased reward mechanisms, it effectively improves training efficiency and stability. Finally, experiments in a digital battlefield verify the feasibility and superiority of the multi-agent architecture and the PPO-TAGNA algorithm in solving the ground-to-air combat task allocation problem. Attached Figure Description

[0041] Figure 1 A research framework for a collaborative behavior decision-making model between strong and weak agents is presented.

[0042] Figure 2 Provide the structure of the intelligent agent training network;

[0043] Figure 3 A performance comparison of ablation experiments algorithms is presented;

[0044] Figure 4 Provide a schematic diagram of the experimental scenario;

[0045] Figure 5 Provide a comparison of algorithm performance;

[0046] Figure 6 Provide the agent's performance before training;

[0047] Figure 7 Provide the performance of the agent after training. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings.

[0049] The agent task allocation method based on deep reinforcement learning of this invention specifically includes the following steps:

[0050] Step 1. Strong and Weak Agent Architecture Design

[0051] Large-scale air-to-ground combat mission allocation requires handling numerous concurrent mission allocations and random events, making the entire battlefield situation highly complex and uncertain. Fully distributed multi-agent architectures suffer from poor global coordination for random events, making them unsuitable for the demands of air-to-ground combat mission allocation. Current centralized allocation architectures can achieve globally optimal results, but their practicality is limited by the excessive time cost for solving large-scale, complex problems. To address the distributed collaborative combat command and control problem in air-to-ground combat, this invention combines a deep reinforcement learning development architecture and proposes a strong-weak agent command and control system based on the dual-driven approach of data rules. The strong-weak agent system architecture is as follows: Figure 1As shown, within a certain timeframe, the global situation (represented as "state" in the diagram) serves as the input for a strong agent with superior computing power, yielding combat missions. A weak agent, based on tactical rules, decomposes the strong agent's combat mission according to its own situation, forming "actions" to be executed, and then outputs specific instructions for those actions to the simulation environment. After the weak agent makes a decision based on the instructions, the situation of the entire "simulation environment" changes accordingly, generating "environmental data." This "environmental data" is then transformed into "states" according to the data structure required by the strong agent, serving as its input. In other words, the strong agent allocates tasks to the weak agent based on the global situation, and the weak agent decomposes the combat mission into instructions (such as intercepting a target at a specific time). Using simulated adversarial data to drive the strong agent and combat rules to drive the weak agent aims to improve the speed at which multi-agent systems solve complex tasks and reduce system complexity, eliminating the shortcomings of multi-agent systems in handling complex problems. Strong agents are superior to weak agents. Each weak agent corresponds to a combat unit. Weak agents execute specific actions in parallel and in coordination. The goal is to use multiple agents with strong computing power to acquire situational information and quickly allocate tasks in a short period of time. Then, multiple weak agents select appropriate times and interceptors to intercept enemy targets based on specific tasks and their own status, so as to save resources as much as possible while achieving tactical objectives.

[0052] Step 2. Design of Markov Decision Processes for Collaborative Behavior

[0053] Traditional research on multi-agent cooperative decision-making primarily focuses on model-based approaches, specifically the study of rational agents. However, traditional task allocation research suffers from drawbacks such as over-reliance on the accuracy of the underlying models, focusing only on the design from model to executor while neglecting the model generation process. In intelligent adversarial environments, the variety of agents makes it difficult to obtain accurate decision-making models for multiple agents. Furthermore, the complex task environment and situational perturbations result in environmental models exhibiting a degree of randomness and time-varying characteristics. Therefore, research is needed on control methods for agent models under conditions of information deficiency.

[0054] The essence of this model framework is to solve large-scale task allocation problems by combining the idea of ​​an optimal allocation strategy algorithm with deep reinforcement learning methods.

[0055] Let a Markov process (MDP) have four elements (S, A, r, p): state S, action A, reward function r, and transition probability p; Markov property: p(s) t+1 |s0, a0, ..., s t , a t )=p(s t+1 |s t , a t ); Policy function π: S→A. tLet a be the state at time t. t Let t be the action at time t.

[0056] Optimization objective: To find the optimal policy function π. * Maximize the expected cumulative reward value:

[0057]

[0058] sts t+1 ~p(·|s t , a t ), a t ~π(·|s t ), t=0,...T-1 (1)

[0059] In the formula, γ, r t T represents the reward decay coefficient, the reward at time t, and the total time, respectively, and the superscript t represents the power exponent.

[0060] In p(s) t+1 |s t , a t Solving Markov decision processes under unknown conditions involves using a time-difference algorithm to estimate the action-value function Q. π (s, a):

[0061]

[0062] Q π (s t , a t )←Q π (s t , a t )+α[r(s t , a t )+γmax a Q π (s t+1 ,a)-Q π (s t , a t (3)

[0063] π′(s)=arg max a Q π (s, a) (4)

[0064] In the formula, s refers to the state in general, a refers to the action in general, and r(s) t , a t ) represents the state s at time t. t Execute action a under the following circumstances t The reward obtained, π′(s) represents the optimal policy in state s, and argmax is the maximum reward. aLet represent the set of independent variable points that represent the maximum value of action 'a'.

[0065] Step 3. Design of a phased reward mechanism

[0066] The design of the reward function is crucial for applying deep reinforcement learning to task allocation, and the design must be tailored to the specific problem. For the air-to-ground combat task allocation problem, the common approach is to assign a reward value to each type of unit. If a unit is lost, a corresponding reward value is given, and the reward values ​​at the end of each round are summed to obtain the final reward value. However, in practice, the reward values ​​for unit losses cancel each other out at each step, resulting in a small final reward value and low learning efficiency. On the other hand, if a reward value is only given in the last step of each round for victory or defeat, with all other steps having zero reward values, it's equivalent to not adding any prior knowledge, maximizing the learning space for the neural network. However, this leads to overly sparse reward values, making it difficult for the neural network to discover the winning state and learn the strategy. Therefore, the ideal reward value should be neither too sparse nor too dense, clearly guiding the agent to learn in the direction of victory.

[0067] The phased reward mechanism breaks down task objectives and awards rewards in stages, guiding the neural network to find a winning strategy. For example, a one-time reward can be given after successfully repelling the first attack; a corresponding reward can be given after the blue team loses a high-value unit; and a winning reward can be given after the red team wins. Based on this, the reward function is optimized according to different objectives in the actual task, and minimizing interception resource consumption is also included as a reward metric, increasing the impact of maximizing global gains on the agent's profitability.

[0068] For example, the reward function of this invention is based on the principle of minimizing resource usage:

[0069]

[0070] In the formula, m represents the number of manned blue team units intercepted, n represents the number of blue team drones intercepted, and i represents the number of missiles launched. The reward value is 50 points for victory, 5 points for intercepting manned targets such as blue team fighters, 1 point for intercepting drones, and no points for other targets. Each missile launched deducts 0.05 points. Since each stage that awards a reward value represents a necessary objective for the red team to achieve victory, it can guide the agent to learn gradually and in stages.

[0071] Step 4. Agent Network Structure Design

[0072] The neural network structure of the multi-agent command and control model is as follows: Figure 2As shown, the situational input data is divided into four categories: The first category is the Red Force base status, including basic information about key locations and the status of key locations under attack; the second category is the Red Force radar and interceptor status, including the current configuration of fire units, the operational status of radars and launch vehicles, the status of radars under attack, and information on enemy units that fire units can engage; the third category is the Blue Force unit status, including basic information about Blue Force units and the status of units being attacked by Red Force missiles; the fourth category is the status of Blue Force units that can be attacked by the Red Force, including the status of units that can be engaged by Red Force fire units. The number of units in each category is not fixed and varies with the battlefield situation.

[0073] Each type of situational data undergoes feature extraction using two layers of fully connected linear rectified logic (FC-ReLU). All feature vectors are then merged. These feature vectors are predefined; for example, radar power-on is represented by "1" and power-off by "0"; missile launch is represented by "1" and no launch by "0". Multiple feature vectors are then joined and merged using the `concat` function (a commonly used function in programming software). The `concat` method is equivalent to a full connection (UNION ALL) in databases, allowing multiple feature vectors to be connected along a specified axis to form new feature vectors (e.g., horizontally connecting multiple arrays to create a new array). A single FC-ReLU layer and a gated recurrent unit (GRU) are then used to generate global features. GRU introduces the concepts of reset and update gates, modifying the calculation of hidden states in the network and addressing the gradient vanishing problem inherent in neural networks. The details are as follows.

[0074] Given the input X at time t t Hidden state H at the previous time step t-1 After calculation by the fc-ReLU layer with the sigmoid activation function, the reset gate R can be obtained. t Update Gate Z t Output:

[0075] R t =σ(X) t W xr +H t-1 W hr +b r )

[0076] Z t =σ(X) t W xz +H t-1 W hz +b z )

[0077] Among them, W xr 、Whr and W xz 、W hz All are weight parameters, b r 、b z σ is the bias parameter, and σ is the sigmoid function. The sigmoid function can transform the value of an element to between 0 and 1, so the value range of each element in the reset gate and update gate is [0, 1].

[0078] Candidate hidden state at time t and hidden state H t The calculation is as follows:

[0079]

[0080]

[0081] Among them, W xh and W hh It is the weight parameter, b h is the bias parameter, and · represents element-wise multiplication. Reset gate R t The function of `Z` is to determine how much information from the previous hidden state needs to be reset. When an element's value is close to 0, it means that the corresponding hidden state element is reset to 0, that is, the entire hidden state from the previous moment is reset to the current moment's input; `Update Gate Z` t The function of `reset` is to determine whether information from the previous time step is discarded. The smaller the value, the more information contained in the hidden node from the previous time step is discarded. The reason why the GRU network model has low complexity is that it ignores some useless information. The reset gate can capture short-term dependencies in the time series, while the update gate can capture long-term dependencies in the time series.

[0082] Neural networks must consider both the current situation and historical information when making decisions. This requires continuous interaction with the global situation via GRU, selecting which information to retain or forget. Global features and the feature vectors of selectable blue-side units are processed through an attention mechanism to choose the interception unit. Each interception unit then selects the interception timing and enemy unit based on its own state and a pre-designed rule base through attention operations. This part is the architecture design; the attention mechanism and interception unit selection are performed by continuously adjusting probabilities using training parameters. Details are as follows:

[0083] Attention processing is a mechanism that allows a strong agent to focus on certain information at a specific point in time while ignoring other information. It enables the strong agent to make better decisions more quickly and accurately in local areas.

[0084] When faced with a large amount of input contextual information, neural networks can borrow from human attention mechanisms, selecting only key information for processing. This attention mechanism can also be called an attention network. Taking a long text as an example: given an article, extract its content and propose a certain number of questions. These questions are only relevant to a portion of the text and irrelevant to the rest. To reduce computational burden, only the relevant content needs to be selected for the neural network to process.

[0085] Define X = [x1, ..., x2] N Given an array of N input information, first calculate the attention distribution a for selecting the i-th input information, given q and X. i Let i = 1, ..., M, where the value of i is related to the number of environmental targets, and q be the query vector. Then, define a... i for

[0086]

[0087] In the formula, s(x i q) is the attention scoring function, and the calculation model is as follows:

[0088] Additive model: s(x) i ,q)=v T tanh(Wx i +Uq);

[0089] Dot product model:

[0090] Scaling the click model:

[0091] Bilinear model:

[0092] In the formula, W, U, and v are the learnable network parameters; d is the dimension of the input information, and T represents the transpose. The bilinear model can be viewed as a generalized dot product model. Assume that in the bilinear model, W = U... T v, then the model can be written as That is, calculate the dot product after performing linear transformations on x and q respectively.

[0093] Step 5. Multi-head attention mechanism and goal selection design

[0094] like Figure 2 As shown, this invention performs multi-head processing of decision-making actions as the output of the network, that is, the actions are divided into action subject (which interception units to select), action predicate (which launcher to select and at what time to intercept), and action object (which enemy targets to select).

[0095] When selecting interception targets, the neural network needs to focus on some important targets in a local area. This invention uses an additive model to implement the attention mechanism operation for the feature vectors of each fire unit state and the incoming target. The "feature vectors of each fire unit state and the incoming target" are obtained from the "state" data, which is transformed from the "simulation environment data". The "transformation" is to concatenate and merge the "simulation environment data" into the "state" using the concat function.

[0096] Define X = [x1, ..., x2] N [x] is a vector formed by N input information. i Let i be the i-th incoming target that can be selected, i = 1, 2, ..., N. First, calculate the probability a of selecting the i-th target given the query vectors q and X. i , a i This is called the attention distribution, then we define a i for

[0097]

[0098] In the formula, s(x i Let q) be the attention scoring function, representing the score for selecting the i-th incoming target given the query vector q. The higher the score, the greater the probability that the i-th incoming target will be selected. An additive model is chosen for the calculation.

[0099] s(x i ,q)=v T tanh(Wx i +Uq) (7)

[0100] In the formula, the query vector q is the feature vector of each fire unit, W and U are the first and second trainable neural network parameters, v is the global situation feature vector, which is the conditional attention mechanism that allows global situation information to participate in the calculation, and tanh is the hyperbolic tangent function. The attention score of each fire unit for each target is obtained, and sigmoid sampling is performed on each bit of the score vector to finally generate the overall decision. Specific Implementation

[0102] To investigate the impact of the two mechanisms on algorithm performance, this invention designs an ablation experiment. Four different algorithms are set up by adding or removing the two mechanisms from the basic proximal policy optimization (PPO) algorithm, and the differences in performance are compared. The experimental setup is shown in the table below:

[0103] Table 1 Comparison of different algorithms

[0104]

[0105] All algorithms were trained iteratively 1,000,000 times under the same scenario settings. The experimental results are as follows: Figure 3 As shown, both the staged reward mechanism and the multi-head attention mechanism can improve the performance of the basic PPO algorithm, increasing the average reward from 10 to between 30 and 40. The staged reward mechanism has a slightly greater effect and is more stable. When both mechanisms are added simultaneously, the algorithm's performance is significantly improved, with the average reward value increasing to around 65. This demonstrates that the PPO-TAGNA method proposed in this invention is effectively applicable to task allocation problems within the framework of strong and weak agents.

[0106] This invention's neural network training environment takes place in a virtual digital battlefield. Within a hypothetical combat area, facing a certain number of Blue Force attacking troops, and with the Red Force having key locations to protect and limited forces, the Red Force agent needs to make real-time decisions based on the battlefield situation. Task allocation is performed according to factors such as the enemy's threat level, aiming to preserve its own strength while protecting key locations from destruction. This invention trains the Red Force's task allocation strategy using deep reinforcement learning methods. The hypothetical scenario involves Red and Blue Forces opposing each other. Figure 4 As shown.

[0107] Red Force Force Deployment and Capability Indicators: Defend 2 key locations: command post and airfield; 1 early warning aircraft: detection range 400km; Long-range firepower unit composition: 1 long-range fire control radar vehicle (capable of simultaneously tracking 8 Blue Force targets, guiding 16 anti-aircraft missiles, maximum detection range 200km, sector 120°), 8 long-range missile launchers (compatible with long-range / intermediate anti-aircraft missiles, each launcher carries 3 long-range anti-aircraft missiles and 4 short-range anti-aircraft missiles); Short-range firepower unit composition: 1 short-range fire control radar vehicle ( It can simultaneously track 4 enemy targets, guide 8 anti-aircraft missiles, with a maximum detection range of 60km and a 360° sector; it also supports 3 short-range missile launchers (each carrying 4 short-range anti-aircraft missiles); if the fire control radar is destroyed, the fire unit loses its combat capability; the radar needs to be powered on throughout the guidance process; when powered on, it radiates electromagnetic waves, which can be detected by the enemy and reveal its position; the fire control radar is limited by the curvature of the earth and the obstruction of terrain objects, resulting in blind spots; considering the refraction of electromagnetic waves by the atmosphere, the radar's maximum line-of-sight range is... H T H represents the altitude of the target. R The altitude of the radar antenna, H in this invention RThe range is set to 4m. The anti-aircraft missile's flight trajectory is the optimal energy trajectory, with interception limits of 160km (long range) and 40km (short range). The high kill probability within the kill zone against UAVs, fighter jets, bombers, anti-radiation missiles, and air-to-ground missiles is 75%, and the low kill probability is 55%. Against cruise missiles, the high kill probability within the kill zone is 45%, and the low kill probability is 35%. A fan-shaped deployment of 4 long-range fire units and 3 short-range fire units protects the Red Force command post, and a fan-shaped deployment of 2 long-range fire units and 3 short-range fire units protects the Red Force airfield, for a total of 12 fire units.

[0108] Blue Force Force Deployment and Capabilities: 18 cruise missiles; 20 UAVs, each carrying 2 anti-radiation missiles and 1 air-to-ground missile; 12 fighter jets, each carrying 6 anti-radiation missiles and 2 air-to-ground missiles; 4 bombers. Anti-radiation missiles have a range of 110km and a hit probability of 80%; air-to-ground missiles have a range of 60km and a hit probability of 80%; 2 jamming aircraft provide long-range standoff jamming support, with a jamming sector of 15°. After the Red Force radar is jammed, the probability of kill is reduced accordingly based on the jamming level.

[0109] The training hardware configuration is as follows: CPU running the simulation environment, model Intel Xeon E5-2678V3, 88 cores, 256GB RAM; GPU*2 running neural network training, model Nvidia GeForce 2080Ti, 72 cores, 11GB VRAM. The hyperparameter ε in PPO is 0.2, and the learning rate is 10. -4 The batch size is 5120, and the number of hidden layer units in the neural network is 128 and 256, respectively.

[0110] The proposed "One Strong Agent with N Weak Agents (OGMN)" and Alpha C2 agent architectures were iterated 1,000,000 times in a digital battlefield using the PPO algorithm, and the results were compared with a rule base based on expert decision-making criteria. Figure 5 As shown:

[0111] In the simulation of digital battlefields, this invention can give rise to certain strategies and tactics. For example... Figure 6 The image shows the performance of the Red Team's agent before training. At this point, only the unit closest to the target is allowed to defend, without any awareness of sharing the defensive pressure, and no distinction is made between the value of the targets. Ultimately, when high-value targets attack, the resources of the units that can intercept them are exhausted, resulting in failure.

[0112] Figure 7 For the red team's decision-making after training, the agent can now distinguish high-threat units from the blue team, share the defensive burden, utilize resources more rationally, defend key locations more efficiently, and ultimately proactively attack high-value targets of the blue team to achieve victory. Please combine the following two parts into one paragraph.

[0113] To address the shortcomings of existing ground-to-air combat task allocation methods, such as insufficient game-theoretic adversarial representation, low efficiency of multi-agent systems in handling complex tasks, and interaction conflicts, this invention proposes a "one strong agent with N weak agents (OGMN)" multi-agent architecture to reduce problem complexity and minimize allocation conflicts. Furthermore, to address the limitations of traditional dynamic task allocation algorithms, such as high computational cost and slow solution speed, this invention proposes a proximal policy optimization (PPO for TaskAssignment General and Narrow Agent, PPO-TAGNA) algorithm for task allocation between strong and weak agents.

[0114] This invention proposes a "One Strong Agent with N Weak Agents (OGMN)" multi-agent architecture, which reduces system complexity and eliminates the weakness of multi-agent systems prone to interaction conflicts when handling complex problems. Addressing the characteristics of large-scale task allocation problems, the PPO-TAGNA algorithm is proposed, which effectively improves training efficiency and stability through multi-head attention and phased reward mechanisms. Finally, experiments in a digital battlefield verify the feasibility and superiority of the OGMN agent architecture and the PPO-TAGNA algorithm in solving the ground-to-air combat task allocation problem.

Claims

1. A task allocation method for intelligent agents based on deep reinforcement learning, characterized in that, Specifically, the following steps are included: Step 1. Strong and Weak Agent Architecture Design Within a certain timeframe, the global situation, referred to simply as "state," serves as the input for a strong agent with superior computing power, enabling it to obtain combat missions. A weak agent, based on tactical rules, decomposes the strong agent's combat missions according to its own situation, forming "actions" to be executed, and then outputs specific instructions for those actions to the simulation environment. After the weak agent makes a decision based on the instructions, the situation of the entire "simulation environment" changes accordingly, generating "environmental data." This "environmental data" is then transformed into "states" according to the data structure required by the strong agent, serving as the input for the strong agent. Step 2. Design of Markov Decision Processes for Collaborative Behavior Let the Markov process MDP have four elements (S, A, r, p): state S, action A, reward function r, and transition probability p; Markov property: p(s) tj |s0, a0, ..., s t , a t )=p(s tj |s t , a t ); Policy function π: S→A; s t Let a be the state at time t. t The action at time t; Optimization objective: To find the optimal policy function π. * Maximize the expected cumulative reward value: In the formula, γ, r t T represents the reward decay coefficient, the reward at time t, and the total time, respectively, and the superscript t indicates the power exponent; In p(s) tj |s t , a t Solving Markov decision processes under unknown conditions involves using a time-difference algorithm to estimate the action-value function Q. π (s, a): Q π (s t ,a t )←Q π (s t ,a t )+α[r(s t ,a t )+γ max a Q π (s t+1 ,a)-Q π (s t ,a t )] (3) π′(s)=arg max a Q π (s, a) (4) In the formula, s refers to the state in general, a refers to the action in general, and r(s) t , a t ) represents the state s at time t. t Execute action a under the following circumstances t The reward obtained, π′(s) represents the optimal policy in state s, and argmax is the maximum reward. a The set of independent variable points representing the maximum value of action a; Step 3. Design of a phased reward mechanism The phased reward mechanism adopts the method of breaking down task objectives and giving reward values ​​in stages to guide the neural network to find a winning strategy. On this basis, the reward function is optimized according to different objectives in actual tasks, and minimizing interception resource consumption is also used as a reward indicator to increase the effect of maximizing global benefits on the agent's benefits. Step 4. Agent Network Structure Design The situational input data is divided into four categories: the first category is the status of the Red Team's base; the second category is the status of the Red Team's radar and interceptors; the third category is the status of the Blue Team's units; and the fourth category is the status of Blue Team units that can be attacked by the Red Team. The number of units in each category is not fixed and varies with the battlefield situation. Each type of situational data is extracted using two fully connected linear rectified loop (FC-ReLU) layers. Then, all feature vectors are merged, and the feature vectors are predefined. Multiple feature vectors are connected and merged using the concat function to form an array. Then, a global feature is generated using an FC-ReLU layer and a gated recurrent unit (GRU). The GRU introduces the concepts of reset gate and update gate to modify the calculation method of hidden states in the network, which can solve the gradient vanishing problem in neural networks. When making decisions, neural networks need to consider both the current situation and historical information. They need to continuously interact with the global situation through GRU to choose to retain or forget information. The global features and the feature vectors of the selectable blue team units are processed through an attention mechanism to select the interception unit. Each interception unit then selects the interception timing and enemy unit based on its own state and a pre-designed rule base through attention operations. Step 5. Multi-head attention mechanism and goal selection design The decision-making actions are processed by multiple heads as the output of the network, that is, the actions are divided into action subject, action predicate, and action object; When selecting interception targets, the neural network needs to focus on some important targets in a local area; the feature vectors of each fire unit state and the incoming target are calculated using an additive model to implement the attention mechanism; the "feature vectors of each fire unit state and the incoming target" are obtained from the "state" data, which is converted from the "simulation environment data"; Define X = [x1, ..., x2] N [x] is a vector formed by N input information. i Let i be the i-th incoming target that can be selected, i = 1, 2, ..., N. First, calculate the probability a of selecting the i-th target given the query vectors q and X. t , a i This is called the attention distribution, then we define a i for In the formula, s(x i Let q) be the attention scoring function, representing the score for selecting the i-th incoming target given the query vector q. The higher the score, the greater the probability that the i-th incoming target will be selected. An additive model is chosen for the calculation. s(x i ,q)=v T tanh(Wx i +Uq) (7) In the formula, the query vector q is the feature vector of each fire unit, W and U are the first and second trainable neural network parameters, v is the global situation feature vector, which is the conditional attention mechanism, so that the global situation information participates in the calculation, and tanh is the hyperbolic tangent function; the attention score of each fire unit with respect to each target is obtained, and sigmoid sampling is performed on each bit of the score vector, and finally the overall decision is generated.

2. The agent task allocation method based on deep reinforcement learning as described in claim 1, characterized in that, In Step 3, a one-time stage reward can be given after successfully repelling the first attack; a corresponding reward value can be given after the blue side loses a high-value unit; and a victory reward value can be given after the red side wins. This invention is based on a reward function that minimizes resource usage: In the formula, m represents the number of blue force manned units intercepted, n represents the number of blue force drones intercepted, and D represents the number of missiles launched.

3. The agent task allocation method based on deep reinforcement learning as described in claim 1, characterized in that, In Step 4, the Red Team's base status includes basic information about key locations and the status of key locations under attack; the Red Team's radar and interceptor status includes the current configuration of fire units, the operational status of radars and launch vehicles, the status of radars under attack, and information on enemy units that fire units can engage; the Blue Team's unit status includes basic information about Blue Team units and the status of units being attacked by Red Team missiles; the status of Blue Team units that can be attacked by the Red Team includes the status of units that can be engaged by Red Team fire units.

4. The agent task allocation method based on deep reinforcement learning as described in claim 1, characterized in that, In Step 5, the action subject is which interception units to select, the action predicate is which launcher to select at what time to intercept, and the action object is which enemy targets to select.

Images