An intelligent decision-making method based on hierarchical A3C

Abstract

The application discloses an intelligent decision-making method based on layered A3C, and relates to the field of intelligent decision-making of air defense combat, and comprises the following method steps: S01: a mathematical model is established by taking the optimization targets of minimizing the probability of missing high-value targets, minimizing the loss of defending a key point, maximizing the probability of effective killing, and minimizing the resource consumption of an interception missile; S02: the state, action, reward and target in air defense combat are determined according to step S01; S03: air situation feature extraction based on a Bi-GRU network; S04: calculation of a multi-head attention mechanism; S05: parallel calculation and training are performed by using an A3C algorithm; the air situation information is subjected to feature extraction by the Bi-GRU network, and then the H-A3C algorithm is adopted for training; in the training process, the multi-head attention mechanism and the event-based reward mechanism are introduced, thereby helping the agent to train. Finally, the proposed H-A3C algorithm is verified in a digital battlefield environment, and is faster and better than a traditional algorithm.

Description

Technical Field

[0001] This invention belongs to the field of intelligent decision-making in air defense operations, and specifically relates to an intelligent decision-making method based on hierarchical A3C. Background Technology

[0002] In modern warfare, decision-making is the most crucial element of command and control. It refers to the efficient use of diverse, multi-platform weapon resources based on battlefield situational awareness, according to certain criteria and constraints. This involves the rational allocation of resources to engage multiple incoming targets, avoiding the omission of key targets and repeated firing, in order to achieve optimal combat results. Efficient weapon-target allocation, compared to free firing, increases combat effectiveness by more than three times, acting as a "force multiplier."

[0003] In recent years, deep reinforcement learning has shone brightly in the field of intelligent decision-making for complex problems. Works such as AlphaGo, Alpha Zero, AlphaStar, and Alpha Fold2 by the DeepMind team exemplify this. Reinforcement learning is a trial-and-error approach that continuously learns and acquires better strategies through interaction between the agent and its environment, achieving an end-to-end process from perception to decision-making. This provides a new approach to solving information game problems under incomplete conditions. Deep reinforcement learning methods have achieved good results in areas such as Go, real-time strategy games, autonomous driving, resource allocation, and drone control, gaining widespread attention.

[0004] Intelligent decision-making in air defense operations has attracted the attention of many scholars. However, when faced with complex battlefield environments, existing decision-making methods have failed to design targeted decision-making methods that address the characteristics of hierarchical decision-making in air defense command and control. Furthermore, they also face problems such as the curse of dimensionality and insufficient real-time performance during the solution process. Summary of the Invention

[0005] This invention provides an intelligent decision-making method based on hierarchical A3C, which has attracted much attention from scholars in addressing intelligent decision-making in air defense operations. However, in the face of complex battlefield environments, existing decision-making methods have failed to design targeted decision-making methods that address the characteristics of hierarchical decision-making in air defense command and control. Furthermore, they also face technical problems such as the curse of dimensionality and insufficient real-time performance during the solution process.

[0006] To address the aforementioned technical problems, this invention provides an intelligent decision-making method based on hierarchical A3C, comprising the following steps:

[0007] S01: A mathematical model is established with the optimization objectives of minimizing the probability of missing high-value targets, minimizing the loss of key areas, maximizing the effective kill probability, and minimizing the resource consumption of interceptor missiles.

[0008] S02: Step S01 clarifies the status, actions, rewards, and objectives in air defense operations;

[0009] S03: Air situation feature extraction based on Bi-GRU network;

[0010] S04: Calculation of multi-head attention mechanism;

[0011] S05: Use the A3C algorithm for parallel computation and training.

[0012] Preferably, in step S01, the agent's goal is to maximize the cumulative reward function.

[0013]

[0014] Where γ is the discount factor, T is the time range, and r t This is the reward obtained at step t. The state value function, Rt, can be obtained from the agent's policy π. t Expectations, V π (s), calculation method V π (s)=E[R t |s t =s,π], and the action state value function Q π (s,a), calculation method Q π (s,a)=E[R t |s t =s,a t =a,π], dominant function A π (s t ,a t )=Q π (s,a)-V π (s), representing the current state s. t Next, take action a t The reward obtained is the average reward under strategy π, i.e., V π (s) The good and bad compared to each other.

[0015] Preferably, in step S02, the status aspects include the status of our defended key locations, the status of our firepower units, the status of detected enemy targets, and the status of attackable enemy targets. The status information is maintained by the digital battlefield environment. The status of defended key locations includes the key location number, location, type, and attack status. The status of our firepower units includes the firepower unit target number, location, number of remaining missiles, availability, target number that this unit can attack, and attack status. The status of detected enemy targets includes the target number, location, type, movement status, and attack status. The action space aspects include target selection, firepower unit selection, and launch quantity selection.

[0016] Preferably, in step S03, the Bi-GRU method is used to analyze the input state. As a simplified version of LSTM, GRU uses an update gate instead of the input gate and forget gate in LSTM. The update gate determines the retention of historical information, and the reset gate determines the combination of historical information and current information. The main parameter calculation formulas are as follows:

[0017] z t =σ(w z [h t-1 ,x t ]+b z )

[0018] r t =σ(w r [h t-1 ,x t ]+b r )

[0019]

[0020]

[0021] Where, x t Given the input at time t, h t For the output at time t, r t To reset the door, z t To update the door, To generate information based on the update gate, σ is the sigmoid activation function, tanh is the hyperbolic tangent activation function, and w and b are the weights and biases, respectively.

[0022] In a Bi-GRU network, the forward unit analyzes the forward state sequence pattern, and the backward unit analyzes the reverse state sequence pattern. The main calculation formulas are as follows:

[0023]

[0024]

[0025]

[0026] in, Indicates the state of the forward hidden layer. Indicates the reverse hidden layer state, w t1 w t2 b represents the output weights of the hidden layers during forward and backward propagation. t f is the bias, and f is the activation function.

[0027] Preferably, in step S04, for single-head self-attention, the processed situational information is converted into three vectors: query, key, and value. The formula for the linear transformation is as follows:

[0028] P = W P x,P∈(Q,K,V)

[0029] Among them, W P Let be the linear transformation matrix of the corresponding vector;

[0030] Attention is calculated using Scaled Dot-Product Attention, and the formula is as follows:

[0031]

[0032] Multiple self-attention modules are stacked in parallel (the multi-head attention mechanism is calculated as follows):

[0033] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0034] where head = Attention(QW i Q ,KW i K VW i V )

[0035] Among them W O W i Q W i K The learnable parameter matrix in the data projection, where h is the number of heads, is selected as h=4 in this paper.

[0036] Preferably, in step S05, during the training process, both the upper and lower layer controllers use the A3C algorithm for parallel computation and training to accelerate the training speed. An entropy regularization term is added to enhance the controller's exploration process and encourage the controller to explore. The expected reward for regularization is:

[0037]

[0038] and Let represent the strategies of the upper-level controller and the lower-level controller, respectively. The exploration intensity of the entropy regularization term can be adjusted by the hyperparameter β. The parameters of the actor-critic algorithm consist of two parts: The actor's parameters are The parameters and gradients for the commentator are as follows:

[0039]

[0040]

[0041] As a critic function, R is the policy reward baseline, which can reduce the variance of the policy gradient. H It's a discount reward;

[0042] The objective function of the upper-level controller is to maximize the external reward and, as far as possible, ensure that the proposed sub-objectives can be executed by the lower-level controller. The reward function of the upper-level controller is:

[0043] r H (t)=c1·1 {subgoal reached} +r ex (t)

[0044] The first term is the reward for the lower-level controller achieving the sub-goal proposed by the upper-level controller. The weight of this term is adjusted by the coefficient c1. The second term is the cumulative reward. This term encourages the upper-level policy to obtain more external rewards. The external cumulative reward is obtained by the lower-level controller performing actions in the environment.

[0045] The cumulative reward for the upper-level controller is:

[0046]

[0047] The lower-level controller interacts directly with the environment, allowing for finer-grained actions. During these actions, it attempts to achieve the sub-goals generated by the upper-level controller; and during the actions, it adjusts the sub-goals according to their values. T and state s t Take action to reach g T Stop at the predetermined number of steps, and wait for the upper-level controller to provide a new sub-target g. T+1 A training method combining external and internal rewards is employed, with parameter α adjusting the weights of internal and external rewards. The reward function is:

[0048] r L (t)=α*[c2·1 {subgoal reached} -c2·1 {exceed step limit} -c3]+(1-α)·min(max(r ex (t),-c2),c2)

[0049] The reward function includes a reward for achieving the goal and a penalty for exceeding the specified number of steps. Through parameter c2, the internal reward and external reward scales are controlled within the same range. Through parameter c3, the controller receives a small penalty for each action taken (c2>0, c3≥0).

[0050] The cumulative reward for the lower-level controller is:

[0051]

[0052] The lower-level controller is also trained using the A3C algorithm, and the regularization expected reward and parameter gradient are similar to those of the upper-level controller.

[0053] The present invention has the following advantages over the prior art:

[0054] 1. The present invention provides an intelligent decision-making method based on hierarchical A3C, which proposes a novel hierarchical asynchronous advantage actor-commentator algorithm, enabling the agent to learn better strategies during training and to generate new tactics and strategies.

[0055] 2. The present invention provides an intelligent decision-making method based on hierarchical A3C, in which a hierarchical air defense combat decision-making framework is proposed. It adopts a two-layer control approach, which is closer to the decision-making process of human commanders in air defense combat. Specifically, the attack target is selected first, followed by the selection of fire units and missile launch, which can effectively reduce the ambiguity of the decision.

[0056] 3. The present invention provides an intelligent decision-making method based on hierarchical A3C, which proposes an event-based reward mechanism to improve the training speed in the initial stage of reinforcement learning, and proposes an air situation processing perception method based on Bi-GRU and Multi-Head Attention mechanism to improve the agent's ability to understand complex air situation environments. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the Bi-GRU structure of an intelligent decision-making method based on hierarchical A3C according to the present invention.

[0058] Figure 2 This is a schematic diagram of the multi-head attention structure in an intelligent decision-making method based on hierarchical A3C according to the present invention;

[0059] Figure 3 This is a structural diagram of the intelligent decision-making module for air defense operations in an intelligent decision-making method based on hierarchical A3C according to the present invention;

[0060] Figure 4 This is a structural diagram of the training reward comparison in a hierarchical A3C-based intelligent decision-making method of the present invention; Detailed Implementation

[0061] Please see Figures 1-4The present invention provides a technical solution: an intelligent decision-making method based on hierarchical A3C, including the following steps: S01: establishing a mathematical model with minimizing the probability of missing high-value targets, minimizing the loss of key areas, maximizing the effective kill probability, and minimizing the resource consumption of interceptor missiles as optimization objectives;

[0062] S02: Step S01 clarifies the status, actions, rewards, and objectives in air defense operations;

[0063] S03: Air situation feature extraction based on Bi-GRU network;

[0064] S04: Calculation of multi-head attention mechanism;

[0065] S05: Use the A3C algorithm for parallel computation and training.

[0066] The intelligent decision-making problem in air defense operations usually establishes a mathematical model with optimization objectives such as minimizing the probability of missing high-value targets, minimizing the loss of key areas, maximizing the effective kill probability, and minimizing the consumption of interceptor missile resources. This technical solution expresses the intelligent decision-making problem in air defense operations as a Markov decision process.

[0067] Intelligent agents in battlefield situation i Take action a at that time i To intercept enemy targets, a Markov process model includes a state set S = [s1, s2, ..., s...]. n ] and action set A = [a1, a2, ..., a n The agent, according to the policy π: S×A → [0,1], in state s i Select action a from the action set i The battlefield environment transitions to the next state according to the state transition function P: S×A×S→[0,1]. The agent's goal is to maximize the cumulative reward function.

[0068]

[0069] Where γ is the discount factor, T is the time range, and r t This is the reward obtained at step t. The state value function, Rt, can be obtained from the agent's policy π. t Expectations, V π (s), calculation method V π (s)=E[R t |s t =s,π], and the action state value function Q π (s,a), calculation method Q π (s,a)=E[R t |s t =s,a t=a,π], dominant function A π (s t ,a t )=Q π (s,a)-V π (s), representing the current state s. t Next, take action a t The reward obtained is the average reward under strategy π, i.e., V π (s) Comparison of good and bad;

[0070] Please refer to Tables 1 and 2. In this technical solution, the intelligent decision-making problem in air defense operations is mapped to a Markov decision problem by clarifying the state, actions, rewards, and objectives in air defense operations.

[0071] In terms of status, this includes the status of our defended key locations, the status of our fire units, the status of detected enemy targets, and the status of attackable enemy targets. The status information is maintained by the digital battlefield environment. The status of defended key locations includes the key location number, location, type, and attack status. The status of our fire units includes the fire unit target number, location, number of remaining missiles, availability, target number that the unit can attack, and attack status. The status of detected enemy targets includes the target number, location, type, movement status, and attack status. In terms of action space, this includes target selection, fire unit selection, and launch quantity selection.

[0072] Please refer to Table 3. A reward mechanism is also designed. Regarding reward design, due to the large number of units on both sides, the state space and action space are both large. If the reward value for victory or defeat is only given once after each round of combat, the rewards will be very sparse, and the possibility of the agent independently exploring the winning state will be very low. Therefore, it is necessary to guide the agent's learning by reasonably designing the reward function. The idea of ​​this technical solution is to set corresponding reward values ​​for each type of unit on both the red and blue sides. If there is unit loss during the battle, the corresponding unit's reward value is given, i.e., our loss is negative, and the enemy's loss is positive. The agent learns to reduce our losses, increase the enemy's losses, and ultimately achieve victory. Since each round of training takes a long time, only giving rewards for victory and defeat would result in long reward delays and excessive sparseness. Therefore, in addition to giving the final reward, EpisodicReward, a more intensive reward based on key events is also given to accelerate the agent's training process.

[0073] Please see Figure 1In air defense missions, air situation information is a time-series event. The decision-making process needs to consider the state input of a period of time before the current moment. This technical solution adopts the Bi-GRU method to analyze the input state. Compared with the traditional GRU method, the Bi-GRU method analyzes the pattern of state information from two directions, past to present and present to past, to enhance the model's ability to learn air situation features. Bi-GRU consists of a forward GRU and a backward GRU.

[0074] The Bi-GRU method is used to analyze the input state. As a simplified version of LSTM, GRU uses an update gate instead of the input gate and forget gate in LSTM. The update gate determines the retention of historical information, and the reset gate determines the combination of historical and current information. The main parameter calculation formulas are as follows:

[0075] z t =σ(w z [h t-1 ,x t ]+b z )

[0076] r t =σ(w r [h t-1 ,x t ]+b r )

[0077]

[0078]

[0079] Where, x t Given the input at time t, h t For the output at time t, r t To reset the door, z t To update the door, To generate information based on the update gate, σ is the sigmoid activation function, tanh is the hyperbolic tangent activation function, and w and b are the weights and biases, respectively.

[0080] In a Bi-GRU network, the forward unit analyzes the forward state sequence pattern, and the backward unit analyzes the reverse state sequence pattern. The main calculation formulas are as follows:

[0081]

[0082]

[0083]

[0084] in, Indicates the state of the forward hidden layer. Indicates the reverse hidden layer state, w t1 w t2 b represents the output weights of the hidden layers during forward and backward propagation. t f is the bias, and f is the activation function.

[0085] Please see Figure 2 In the digital battlefield environment, the state space has a high dimension and the network needs to process a large amount of information. The multi-head attention mechanism enables the neural network to focus on important information and prioritize the processing of information with high attention. Multi-head attention splices multiple self-attention layers, and multiple attention layers perform linear transformations on the same input from different angles, thereby obtaining information from different subspaces.

[0086] For single-head self-attention, the processed situational information is converted into three vectors: query, key, and value. The formula for the linear transformation is as follows:

[0087] P = W P x,P∈(Q,K,V)

[0088] Among them, W P Let be the linear transformation matrix of the corresponding vector;

[0089] Attention is calculated using Scaled Dot-Product Attention, and the formula is as follows:

[0090]

[0091] Multiple self-attention modules are stacked in parallel (the multi-head attention mechanism is calculated as follows):

[0092] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0093] where head = Attention(QW i Q ,KW i K VW i V )

[0094] Among them W O W i Q W i K The learnable parameter matrix in the data projection, where h is the number of heads, is selected as h=4 in this paper.

[0095] Please seeFigure 3 The proposed hierarchical Actor-Critic algorithm structure in this technical solution divides the agent into an upper-layer controller and a lower-layer controller. The upper-layer controller employs the Actor-Critic algorithm, comprising two modules: Critic and Actor. The Actor module receives the state and reward output by the digital battlefield environment (referred to as external reward here to distinguish it from internal reward). Based on the current state, it provides sub-objectives for the lower-layer controller to execute. The controller adjusts its strategy based on the reward provided by the Critic module. The Critic module, based on the current environmental state, determines whether the sub-objectives provided by the Actor can be executed by the lower-layer strategy. The lower-layer controller then... The ller provides a positive internal reward when the action reaches the subgoal within a specified number of steps, and a negative internal reward when the action fails to reach the subgoal. The lower-level controller also uses the Actor-Critic algorithm, which includes two modules: Critic and Actor. The Critic module receives the subgoal from the controller and determines whether the Actor's action meets the subgoal. It provides a positive internal reward when the action reaches the subgoal and a negative internal reward when the action fails to reach the subgoal. The Actor module performs actions based on the current state and adjusts its strategy based on the reward provided by the Critic module.

[0096] To better analyze the digital battlefield environment, both the upper-layer controller and the lower-layer controller adopt the Bi-GRU feature extraction method and the Multi-Head Attention mechanism. The parameters of these two modules are shared between the two layers of controllers.

[0097] During training, both the upper and lower layer controllers use the A3C algorithm for parallel computation and training to accelerate the training speed. An entropy regularization term is added to enhance the controller's exploration process and encourage it to explore. The expected reward for regularization is:

[0098]

[0099] and Let represent the strategies of the upper-level controller and the lower-level controller, respectively. The exploration intensity of the entropy regularization term can be adjusted by the hyperparameter β. The parameters of the actor-critic algorithm consist of two parts: The actor's parameters are The parameters and gradients for the commentator are as follows:

[0100]

[0101]

[0102] As a critic function, R is the policy reward baseline, which can reduce the variance of the policy gradient. H It's a discount reward;

[0103] The objective function of the upper-level controller is to maximize the external reward and, as far as possible, ensure that the proposed sub-objectives can be executed by the lower-level controller. The reward function of the upper-level controller is:

[0104] r H (t)=c1·1 {subgoal reached} +r ex (t)

[0105] The first term is the reward for the lower-level controller achieving the sub-goal proposed by the upper-level controller. The weight of this term is adjusted by the coefficient c1. The second term is the cumulative reward. This term encourages the upper-level policy to obtain more external rewards. The external cumulative reward is obtained by the lower-level controller performing actions in the environment.

[0106] The cumulative reward for the upper-level controller is:

[0107]

[0108] The lower-level controller interacts directly with the environment, allowing for finer-grained actions. During these actions, it attempts to achieve the sub-goals generated by the upper-level controller; and during the actions, it adjusts the sub-goals according to their values. T and state s t Take action to reach g T Stop at the predetermined number of steps, and wait for the upper-level controller to provide a new sub-target g. T+1 A training method combining external and internal rewards is employed, with parameter α adjusting the weights of internal and external rewards. The reward function is:

[0109] r L (t)=α*[c2·1 {subgoal reached} -c2·1 {exceed step limit} -c3]+(1-α)·min(max(r ex (t),-c2),c2)

[0110] The reward function includes a reward for achieving the goal and a penalty for exceeding the specified number of steps. Through parameter c2, the internal reward and external reward scales are controlled within the same range. Through parameter c3, the controller receives a small penalty for each action taken (c2>0, c3≥0).

[0111] The cumulative reward for the lower-level controller is:

[0112]

[0113] The lower-level controller is also trained using the A3C algorithm, and the regularization expected reward and parameter gradient are similar to those of the upper-level controller.

[0114] Please see Figure 4 In this embodiment, the training reward effect is shown in the figure. The shaded area is the error band of the five experiments. During the training process, the reward value of all algorithms increased, which shows that all algorithms can gradually learn the rules and characteristics of combat during the training process and improve the benefits. As can be seen from the figure, the method of using a hierarchical architecture is better than the method of not using a hierarchical architecture in terms of convergence speed and final reward. This shows that the proposed hierarchical method can help the agent find better strategies faster in air defense combat missions.

[0115] Among the three algorithms employing a hierarchical architecture, the proposed H-A3C method converges significantly faster than the others during training, while H-PPO converges slightly faster than H-DDPG. In terms of final results, H-A3C and H-PPO algorithms achieve similar final rewards, both outperforming H-DDPG. Among the three algorithms not employing a hierarchical architecture, A3C converges faster than PPO and DDPG, while PPO converges slightly faster than DDPG. The final rewards obtained by A3C and PPO algorithms are similar, both outperforming DDPG. This indicates that in air defense missions, A3C and PPO algorithms achieve similar final results, both outperforming DDPG, with A3C converging faster than the other two algorithms.

[0116] Table 1. Definition of Air Defense Operational Status Space

[0117]

[0118] Table 2 Definition of Air Defense Operational Space

[0119]

[0120] Table 3 Definition of Rewards for Air Defense Operations

[0121]

[0122]

Claims

1. A hierarchical A3C-based intelligent decision-making method, characterized in that: The following methods and steps are included: S01: A mathematical model is established with the optimization objectives of minimizing the probability of missing high-value targets, minimizing the loss of key areas, maximizing the effective kill probability, and minimizing the consumption of interceptor missile resources. S02: Step S01 clarifies the status, actions, rewards, and objectives in air defense operations; S03: Air situation feature extraction based on Bi-GRU network; S04: Calculation of multi-head attention mechanism; S05: Use the A3C algorithm for parallel computing and training; In step S05, during the training process, both the upper and lower layer controllers use the A3C algorithm for parallel computation and training to accelerate the training speed. An entropy regularization term is added to enhance the controller's exploration process and encourage it to explore. The expected reward for regularization is: ； and Let these represent the policies of the upper-level controller and the lower-level controller, respectively. The exploration intensity of the entropy regularization term can be determined using hyperparameters. To adjust, the parameters of the actor-critic algorithm consist of two parts: The actor's parameters are The parameters and gradients for the commentator are as follows: ； As a critic function, it serves as the policy's reward baseline and can reduce the variance of the policy gradient. It's a discount reward; The objective function of the upper-level controller is to maximize the external reward and, as far as possible, ensure that the proposed sub-objectives can be executed by the lower-level controller. The reward function of the upper-level controller is: ； The first term is the reward for the lower-level controller achieving the sub-goal proposed by the upper-level controller; the weight of this term is determined by a coefficient. The first item is adjustment; the second item is cumulative reward; this item encourages the upper-level strategy to obtain more external rewards; the acquisition of external cumulative rewards is obtained by the lower-level controller performing actions in the environment; The cumulative reward for the upper-level controller is: ； The lower-level controller interacts directly with the environment, allowing for more precise actions. During these actions, it attempts to achieve sub-goals generated by the upper-level controller; and during the actions, it adjusts its actions based on these sub-goals. and state Take action to achieve Stop at the predetermined number of steps, and wait for the upper-level controller to provide a new sub-target. A training method combining external and internal rewards was adopted, with parameters... Adjusting the weights of internal and external rewards, the reward function is as follows: ； The reward function includes a reward for achieving the goal and a penalty for exceeding the specified number of steps, quantified by parameters. Internal and external rewards are controlled within the same range, through parameters. The controller receives a small penalty for each action it takes. ; The cumulative reward for the lower-level controller is: ； The lower-level controller is also trained using the A3C algorithm, and the regularization expected reward and parameter gradient are similar to those of the upper-level controller.

2. The intelligent decision-making method based on hierarchical A3C according to claim 1, characterized in that, In step S01, the agent's goal is to maximize the cumulative reward function. ； in, It is a discount factor. It refers to a time range. The reward is obtained at step t; from the agent's policy The state value function can be obtained, i.e. Expectations Calculation method and action state value function Calculation method Advantage function , indicating that it is used to characterize the current state Next, take action Rewards and Strategies Lower average reward, i.e. The good and the bad compared.

3. The intelligent decision-making method based on hierarchical A3C according to claim 1, characterized in that, In step S02, the status includes the status of our defended key locations, the status of our firepower units, the status of detected enemy targets, and the status of attackable enemy targets. The status information is maintained by the digital battlefield environment. The status of defended key locations includes the key location number, location, type, and attack status. The status of our firepower units includes the target number, location, number of remaining missiles, availability, target number that the unit can attack, and attack status. The status of detected enemy targets includes the target number, location, type, movement status, and attack status. The action space includes target selection, firepower unit selection, and launch quantity selection.

4. The intelligent decision-making method based on hierarchical A3C according to claim 1, characterized in that, In step S03, the Bi-GRU method is used to analyze the input state. As a simplified version of LSTM, GRU uses an update gate instead of the input gate and forget gate in LSTM. The update gate determines the retention of historical information, and the reset gate determines the combination of historical information and current information. The main parameter calculation formulas are as follows: ； in, The input at time t, The output at time t, To reset the door, To update the door, To update the information generated by the gate, It is the sigmoid activation function. It is a hyperbolic tangent activation function. and These are the weights and bias terms, respectively. In a Bi-GRU network, the forward unit analyzes the forward state sequence pattern, and the backward unit analyzes the reverse state sequence pattern. The main calculation formulas are as follows: ； in, Indicates the state of the forward hidden layer. Indicates the reverse hidden layer state. , This represents the output weights of the hidden layers during forward and backward propagation. For bias, This is the activation function.

5. The intelligent decision-making method based on hierarchical A3C according to claim 1, characterized in that, In step S04, for single-head self-attention, the processed situational information is converted into three vectors: query, key, and value. The formula for the linear transformation is as follows: ； in, Let be the linear transformation matrix of the corresponding vector; Attention is calculated using Scaled Dot-Product Attention, and the formula is as follows: ； Multiple self-attention modules are stacked in parallel, and the multi-head attention mechanism is calculated as follows: ； in , , The learnable parameter matrix in the data projection, where h is the number of heads, is selected as h=4 in this paper.

Images