Multi-agent-based unmanned aerial vehicle formation defense penetration method and application thereof

Abstract

The invention relates to a multi-agent-based unmanned aerial vehicle formation penetration method and application thereof, and the method comprises the steps: constructing a multi-agent confrontation scene model which comprises an unmanned aerial vehicle formation and a to-be-penetration party; performing role distinguishing on the intelligent agents, improving a control network of part of the intelligent agents, enabling the unmanned aerial vehicle formation to decompose a defense penetration task into a plurality of sub-targets, and obtaining an unmanned aerial vehicle formation defense penetration strategy; the method is applied to a confrontation scene for coordinating and guiding an intelligent body to perform unmanned aerial vehicle formation defense penetration. The method improves the convergence speed of the intelligent agent in a complex environment, enhances the exploration capability of the intelligent agent, avoids falling into a local optimal solution, enhances the perception capability of the intelligent agent for the dynamic change of the environment, improves the decision-making effect, and improves the learning efficiency of the intelligent agent.

Description

Technical Field

[0001] The present invention relates to the technical field of calculation, extrapolation or counting, and in particular to a multi-agent based UAV formation penetration method in a complex environment in the field of game confrontation and its application. Background Art

[0002] Drone penetration refers to the process in which drones break through enemy defense systems (such as radar detection, interception weapons, and electronic jamming) when performing military missions, and successfully infiltrate the target area to complete reconnaissance, strike, or other combat missions. With the development of technologies such as airborne radar systems and medium- and long-range air-to-air defense missiles, defense capabilities are constantly increasing, which poses a great challenge to our own fighters in breaking through defenses and striking targets. Early fighters usually used preset maneuvers such as snake maneuvers and spiral maneuvers to penetrate, but these modes were easily detected and identified by the enemy and lacked flexibility.

[0003] In recent years, with the rapid development of artificial intelligence (AI) technology, researchers have begun to apply deep reinforcement learning theory to the field of drone penetration. At present, drone penetration mostly uses traditional AI based on behavior trees. However, this type of AI often shows insufficient adaptability in complex battlefield environments. This is because the manually designed fixed behavior trees are difficult to adjust according to actual conditions or generate new behavior trees that match the current state. In addition, traditional behavior tree AI usually only designs a set of behavior trees for a single type of combat entity, resulting in limited flexibility, making it difficult to reasonably respond to flexible and changeable action decisions, and unable to accurately give rewards or penalties. More importantly, traditional behavior tree AI usually only focuses on the drone entity it controls. Even if some excellent behavior trees can consider other combat entities, their joint action capabilities are limited to relatively simple scenarios, which makes it difficult to collaborate efficiently with teammates in complex multi-agent scenarios.

[0004] The same-policy algorithm always uses the data generated by the current policy when updating, which can avoid inconsistencies and estimation errors caused by outdated experience. Therefore, it is suitable for tasks that require policy consistency and stability. Multi-Agent Proximal Policy Optimization (MAPPO) is a same-policy algorithm. However, the advantages of the same-policy algorithm also make it usually less efficient in sample efficiency. After updating the model, the used sample experience can only be used through the probability distribution of the old policy in the importance sampling weight. The application of this sample experience has limitations and cannot effectively learn the excellent sample experience in the old policy. In addition, in some observable environments, MAPPO lacks a memory mechanism and only relies on current observations for decision-making, resulting in insufficient adaptation to environmental ambiguity, difficulty in inferring hidden states, and inability to capture long-term dependent features in time series. Therefore, it performs poorly in scenarios that require planning based on historical information or responding to dynamic changes. In addition, the algorithm's exploration ability is also insufficient. Its conservative policy update mechanism is limited to the neighborhood of the current policy distribution. Although it improves training stability, it limits the global exploration ability and is prone to falling into local optimal solutions. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a multi-agent based UAV formation penetration method and its application.

[0006] The technical solution adopted by the present invention is a multi-agent-based UAV formation penetration method, which constructs a multi-agent confrontation scenario model, including a UAV formation and a party to be penetrated; the roles of the agents are distinguished, and the control network of some agents is improved. The UAV formation decomposes the penetration task into several sub-goals to obtain the UAV formation penetration strategy.

[0007] Preferably, based on the multi-agent confrontation scenario model, an interactive network is established for the drone formation, including a decision-making layer and an execution layer;

[0008] For each agent in the decision layer and the execution layer, respectively configure the state space and action space and set the reward function;

[0009] Build an improved control network for each agent in the execution layer.

[0010] Preferably, the state space of the agent in the decision layer is the observation of all agents in the multi-agent confrontation scenario model, and the action space is 0 or 1, which is a discrete value;

[0011] The state space of each agent in the execution layer is the local observation value of the current agent, and the action space is a continuous value between -1 and 1.

[0012] Preferably, the reward function r of each agent in the execution layer is tot satisfy,

[0013]

[0014] Among them, r eps It is represented by the reward at the end of each game, N is the number of categories of additional rewards, r n The additional reward obtained for the nth classification; the additional reward is associated with the distance between the intelligent agent of the execution layer and the party to be penetrated, and the survival time of the intelligent agent of the execution layer;

[0015] In practical applications, the reward of the decision-making layer is the sum of the reward values ​​of all agents in the current execution layer.

[0016] Preferably, the improved control network includes a policy network corresponding to an agent at each execution layer;

[0017] The output of each policy network is added with a noise disturbance signal and then output to the action processing module. A feature extraction module is provided between the input of the policy network and the state space of the corresponding intelligent agent.

[0018] The sampling module and the evaluation network are arranged in sequence after the policy network of the intelligent agents in all execution layers.

[0019] Preferably, the trajectory of the interaction between the intelligent agent of each execution layer and the environment is input into the entire experience pool of the sampling module, and high-quality intelligent agents are selected based on preset conditions and placed into the high-quality intelligent agent pool; the trajectory of the interaction between the intelligent agent of the execution layer and the environment is a local observation value;

[0020] The entire experience pool and the high-quality agent pool are sampled separately, the sampled data are spliced ​​and input into the global sampling pool, and the sampled data in the global sampling pool is output as the input of the evaluation network.

[0021] Preferably, the UAV formation penetration strategy is obtained and executed, and after obtaining a preset amount of interactive information, the networks corresponding to the decision layer and the execution layer are updated based on the objective function.

[0022] Preferably, the objective function of the evaluation network of the decision layer is

[0023]

[0024] The objective function of the policy network at the decision layer is

[0025]

[0026] Among them, r t+e is the agent’s reward at time t+e, N tis the time step used to execute a sub-goal, e is the e-th time step when executing a sub-goal, s is the state, γ represents the discount rate, which ranges from 0 to 1. The closer the discount rate is to 1, the greater the impact of future rewards, and the closer it is to 0, the opposite is true; ω is the evaluation network parameter, θ is the policy network parameter, π θ and The probability of selecting an action for the policy network determines the possibility of the agent choosing a certain action, g t is the sub-goal at time t; E is the expectation, Q(s t+e ,g t ; ω) means adopting sub-goal g at time t+e t The action value function, Q(s t ,g t ; ω) means adopting sub-goal g at time t t The action-value function of .

[0027] Preferably, the objective function of the evaluation network of the execution layer is

[0028]

[0029] The objective function of the policy network of the execution layer is

[0030]

[0031] Among them, t is the number of steps, T is the trajectory length, that is, the number of steps in a complete interaction process (including multiple sub-goals), is a scaling factor used for the calculation of the evaluation network in reinforcement learning, which helps to moderate the size of the gradient used in network training and avoid numerical instability during training; γ represents the discount rate, ω is the evaluation network parameter, θ is the policy network parameter, and a tot represents the joint action of all agents in the execution layer at time t, Indicates that the sampling strategy network selects action a at time t t The probability of θ (a t |s t ) represents the target policy network selecting action a at time t t probability.

[0032] An application of the multi-agent-based UAV formation penetration method is applied to a confrontation scenario of coordinating and guiding intelligent agents to conduct UAV formation penetration.

[0033] The present invention relates to a multi-agent-based UAV formation penetration method and its application. The method constructs a multi-agent confrontation scenario model, including a UAV formation and a party to be penetrated; the roles of the agents are distinguished, and the control network of some agents is improved. The UAV formation decomposes the penetration task into a number of sub-goals, and the UAV formation penetration strategy is obtained; and the method is applied to a confrontation scenario in which the UAV formation is coordinated and guided to penetrate.

[0034] The beneficial effects of the present invention are:

[0035] (1) Improve the convergence speed of intelligent agents in complex environments. The entire experience pool and the high-quality intelligent agent pool are used to sample and train the network in a certain proportion. Through the setting of the experience pool, the evaluation network can effectively learn high-quality sample experience, so that the intelligent agent can adapt to the complex environment more quickly and improve the convergence speed of the intelligent agent in complex environments.

[0036] (2) Enhance the agent's exploration ability and avoid falling into the local optimal solution; add noise perturbation to the output of the MAPPO algorithm's policy network to improve the agent's exploration ability in complex environments and improve the agent's training effect;

[0037] (3) Enhance the agent's ability to perceive dynamic changes in the environment and improve decision-making results; the feature extraction module makes up for the lack of observation by memorizing historical information, thereby more accurately estimating the current state. Through feature extraction, the agent can perceive the state of both units and environmental changes, thereby improving decision-making results;

[0038] (4) Improve the learning efficiency of the intelligent agent; by layering, tasks can be decomposed into multiple smaller, controllable subtasks, which can reduce the complexity of high-dimensional space and improve the learning efficiency of the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 It is the overall framework diagram of the present invention;

[0041] Figure 3 This is a diagram of the execution layer network architecture of the present invention;

[0042] Figure 4 It is a comparison diagram of the application effects of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific implementation described herein is only used to explain the present invention and is not intended to limit the present invention.

[0044] The present invention relates to a multi-agent-based UAV formation penetration method. The method constructs a multi-agent confrontation scenario model, including a UAV formation and a party to be penetrated; roles of the agents are differentiated, and the control network of some agents is improved. The UAV formation decomposes the penetration task into a number of sub-goals, and the UAV formation penetration strategy is obtained.

[0045] The method of the present invention mainly comprises the following steps:

[0046] (1) Construct a multi-agent confrontation scenario model;

[0047] (2) Differentiate the roles of intelligent agents and improve the control networks of some intelligent agents;

[0048] (3) The UAV formation decomposes the penetration mission into several sub-goals and obtains the UAV formation penetration strategy;

[0049] (4)Update strategy.

[0050] The method is described in detail below with reference to embodiments.

[0051] (1) Construct a multi-agent confrontation scenario model;

[0052] In this embodiment, a 4-to-6 red-blue asymmetric game air-to-ground scenario is constructed, in which the red side has 4 drones (aircraft), each of which carries ground-to-ground missiles and can attack enemy units within the detection range. The blue side has 5 missile platoons and 1 base (a total of 6 targets), each missile platoon carries air-to-air missiles, and both sides obtain points based on the number of enemy units destroyed and the distance from the central strategic area. When training one side, each intelligent agent represents a combat unit.

[0053] (2) Differentiate the roles of intelligent agents and improve the control networks of some intelligent agents;

[0054] Based on the multi-agent confrontation scenario model, an interactive network is established for the drone formation, including a decision-making layer and an execution layer;

[0055] Among them, the decision-making layer is equivalent to a manager, issuing goals to all intelligent agents in the execution layer. Since it only needs to issue goals to the lower layer, the network structure of the decision-making layer adopts the single-agent PPO algorithm; the execution layer is equivalent to an employee, executing the sub-goals issued by the decision-making layer, making decisions through local observations of the environment and sub-goals to generate actions and execute them in the environment. Since the execution layer is the actual implementer of the action and there are more than two intelligent agents in the adversarial environment, the MAPPO algorithm is adopted.

[0056] The state space and action space are configured for each agent in the decision layer and the execution layer respectively; specifically, the state space of the agent in the decision layer is the observation of all agents in the multi-agent confrontation scenario model, including but not limited to the position information, weapon information, unit status, etc. of all our agents and the position information, weapon information, unit status, etc. of the enemy agents, and the action space is a discrete value of 0 or 1; and the state space of each agent in the execution layer is the local observation value of the current agent, including but not limited to the position information, weapon information, unit status, etc. of the current agent and the observed position information, weapon information, unit status, etc. of the enemy agents, and the action space is a continuous value between -1 and 1; in fact, the tanh activation function is used here.

[0057] Then, a reward function is set for each agent. Considering that in a multi-agent confrontation environment, if based on the traditional reward function, the agent can only be rewarded when certain conditions are met (such as the agent destroys the enemy or is destroyed by the enemy), this overly sparse reward state will lead to the problem that the algorithm converges slowly or even fails to converge. Therefore, the reward function is modified so that the reward function of each agent in the execution layer is r tot satisfy,

[0058]

[0059] Among them, r eps It is expressed as the reward at the end of each round, including the reward for destroying enemy units and the penalty for destroying our units. This reward is calculated after each round. N is the number of categories of additional rewards, r n is the extra reward obtained for the nth classification; let the extra reward here be associated with the distance between the agent at the execution layer and the party to be penetrated, and the survival time of the agent at the execution layer; for the former, r d =dis ago -dis now ,dis ago Indicates the distance between the agent and the attack target at the previous moment, dis now Represents the distance between the agent and the attack target at the current moment. For the latter, define step is the number of steps in this round, max_step is the maximum time step size of the environment; in practical applications, d and r time Add up and get the extra reward value.

[0060] Based on the above, an improved control network is constructed for each agent in the execution layer.

[0061] The improved control network includes a strategy network corresponding to each execution layer agent;

[0062] The output of each policy network is added with a noise disturbance signal and then output to the action processing module. A feature extraction module is provided between the input of the policy network and the state space of the corresponding intelligent agent.

[0063] Here, a noise disturbance signal is added to the output of the strategy network to enhance the agent's ability to explore the environment, prevent it from falling into the local optimum, and improve the flexibility and adaptability of the strategy. The final output of the agent is:

[0064] a t =π(s t |θ)+ε,ε~N(0,σ 2 )

[0065] Among them, π(s t |θ) is the output of the agent’s policy network, and ε is Gaussian noise;

[0066] The feature extraction module uses LSTM to extract features from the local observations of each agent, and improves the accuracy of decision-making by temporally modeling the states of both units and environmental changes. Specifically, the agent inputs current observation data and historical information into the LSTM network, because UAV penetration missions usually involve complex dynamic environments, time series dependencies, partially observable states, and the need to consider historical decisions and predictions of enemy behavior. Through LSTM, the long-term dependency problem in time series can be solved based on its characteristics of processing long-term dependencies and memorizing historical information.

[0067] The sampling module and the evaluation network are arranged in sequence after the policy network of the intelligent agents in all execution layers.

[0068] The trajectory of the interaction between the agent and the environment of each execution layer is input into the entire experience pool of the sampling module, and high-quality agents are selected based on preset conditions and placed into the high-quality agent pool;

[0069] In this embodiment, the preset condition is that each agent calculates the reward G of each round (n) after each round. n , G n =r1+r2+…+r Q , where r1 to r Q is the reward of this agent for each step in this round, and Q is the total number of steps;

[0070] If the return of this round is greater than the average return Right now The round trajectory is put into the high-quality agent pool, and i represents the total number of rounds in one training.

[0071] The entire experience pool and the high-quality agent pool are sampled in a certain proportion, and the sampled data are spliced ​​and input into the global sampling pool. The sampled data in the global sampling pool is output as the input of the evaluation network, and the evaluation network is updated through the data in the global sampling pool;

[0072] The sampling method is,

[0073]

[0074] Among them, T λ Represents the type of sampled experience pool, λ values ​​0 and 1 represent the entire experience pool and high-quality agents respectively; n is the total number of experience data that the network needs to sample; σ is the coefficient weight, σ∈(0,1).

[0075] The experience selected for each neural network update is:

[0076] T={T0,T1}

[0077] The network parameters initially set include weights and bias items, which are generally randomly assigned to very small random values ​​ranging from 0 to 1 to prevent problems such as gradient explosion or gradient disappearance in the early stages of model training. As training progresses, these parameters will be adjusted accordingly based on performance optimization goals in order to enhance the agent's task execution capabilities.

[0078] (3) The UAV formation decomposes the penetration mission into several sub-goals and obtains the UAV formation penetration strategy;

[0079] In this embodiment, the policy network of the decision layer performs action interaction at a fixed period e, and sets the sub-goal g t Input to the execution layer, the agent in the execution layer takes corresponding actions according to the sub-goals and interacts with the environment, and the sub-goal g output by the policy network in the decision layer t The collection is,

[0080] goal = {attack,circumvent}

[0081] Among them, attack means executing the task of attacking the enemy's high-value target, and circumvent means avoiding the enemy's attack task. The corresponding policy network output values ​​are [0, 0.5) and [0.5, 1] ​​respectively.

[0082] The execution layer agent receives the current action output by the execution layer, i.e., the target g t , by superimposing the value corresponding to the target with the original input of the strategy network of the execution layer, a vector (o t ,g t ), and then sends the vector to the policy network of each agent in the execution layer.

[0083] The UAV formation penetration strategy is obtained and executed, and after obtaining a preset amount of interactive information, the networks corresponding to the decision layer and the execution layer are updated based on the objective function.

[0084] (4) Update strategy;

[0085] The decision layer interacts with the execution layer at a fixed period e and uses the advantage function to update the evaluation network;

[0086] The objective function (advantage function) of the evaluation network of the decision layer is

[0087]

[0088] The objective function of the policy network at the decision layer is

[0089]

[0090] Among them, r t+e is the agent’s reward at time t+e, N t is the time step used to execute a sub-goal, e is the e-th time step when executing a sub-goal, s is the state, γ represents the discount rate, which ranges from 0 to 1. The closer the discount rate is to 1, the greater the impact of future rewards, and the closer it is to 0, the opposite is true; ω is the evaluation network parameter, θ is the policy network parameter, π θ and The probability of selecting an action for the policy network determines the possibility of the agent choosing a certain action, g t is the sub-goal at time t; E is the expectation, Q(s t+e ,g t ; ω) means adopting sub-goal g at time t+e t The action value function, Q(s t ,g t ; ω) means adopting sub-goal g at time t t The action-value function of .

[0091] The objective function of the evaluation network of the execution layer is

[0092]

[0093] The objective function of the policy network of the execution layer is

[0094]

[0095] Among them, t is the number of steps, T is the trajectory length, that is, the number of steps in a complete interaction process, is a scaling factor used for the calculation of the evaluation network in reinforcement learning, which helps to moderate the size of the gradient used in network training and avoid numerical instability during training; γ represents the discount rate, ω is the evaluation network parameter, θ is the policy network parameter, and a tot represents the joint action of all agents in the execution layer at time t, A t (ω) represents the advantage function of evaluating the network at time t, Indicates that the sampling strategy network selects action a at time t t The probability of θ (a t |s t ) represents the target policy network selecting action a at time t t probability.

[0096] The present invention also relates to a computer-readable storage medium, on which a multi-agent-based UAV formation penetration program is stored. When the program is executed by a processor, the multi-agent-based UAV formation penetration method is implemented.

[0097] The present invention also relates to a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multi-agent-based UAV formation penetration method is implemented.

[0098] With reference to the aforementioned embodiment, the present invention also relates to an application of the multi-agent-based UAV formation penetration method, which is applied to a confrontation scenario of coordinating and guiding the intelligent agents to conduct UAV formation penetration. However, the method of the present invention is not limited to this in practical applications. The 4-to-6 red-blue asymmetric game air-ground scenario of this embodiment was tested, and the method of the present invention (MPE-MAPPO) was compared with the traditional method (MAPPO). The test results are as follows: Figure 4 As shown, it can be seen that compared with the traditional method, the training effect of the present invention is significantly improved.

[0099] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0103] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0104] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-agent based UAV formation penetration method, characterized by: The method constructs a multi-agent confrontation scenario model, including a drone formation and a party to be penetrated; The roles of intelligent agents are differentiated, and the control networks of some intelligent agents are improved. The UAV formation decomposes the penetration task into several sub-goals and obtains the UAV formation penetration strategy.

2. The multi-agent based UAV formation penetration method according to claim 1, characterized in that: Based on the multi-agent confrontation scenario model, an interactive network is established for the drone formation, including a decision-making layer and an execution layer; For each agent in the decision layer and the execution layer, respectively configure the state space and action space and set the reward function; Build an improved control network for each agent in the execution layer.

3. The multi-agent based UAV formation penetration method according to claim 2, characterized in that: The state space of the agent in the decision layer is the observation of all agents in the multi-agent confrontation scenario model, and the action space is 0 or 1; The state space of each agent in the execution layer is the local observation value of the current agent, and the action space is a continuous value between -1 and 1.

4. The multi-agent based UAV formation penetration method according to claim 2, characterized in that: The reward function r of each agent in the execution layer tot satisfy, Among them, r eps It is represented by the reward at the end of each game, N is the number of categories of additional rewards, r n The additional reward obtained for the nth classification; the additional reward is associated with the distance between the intelligent agent of the execution layer and the party to be penetrated, and the survival time of the intelligent agent of the execution layer.

5. The multi-agent based UAV formation penetration method according to claim 2, characterized in that: The improved control network includes a strategy network corresponding to each execution layer agent; The output of each policy network is added with a noise disturbance signal and then output to the action processing module. A feature extraction module is provided between the input of the policy network and the state space of the corresponding intelligent agent. The sampling module and the evaluation network are arranged in sequence after the policy network of the intelligent agents in all execution layers.

6. The multi-agent based UAV formation penetration method according to claim 5, characterized in that: The trajectory of the interaction between the agent and the environment of each execution layer is input into the entire experience pool of the sampling module, and high-quality agents are selected based on preset conditions and placed into the high-quality agent pool; The entire experience pool and the high-quality agent pool are sampled separately, the sampled data are spliced ​​and input into the global sampling pool, and the sampled data in the global sampling pool is output as the input of the evaluation network.

7. The multi-agent based UAV formation penetration method according to claim 5, characterized in that: The UAV formation penetration strategy is obtained and executed, and after obtaining a preset amount of interactive information, the networks corresponding to the decision layer and the execution layer are updated based on the objective function.

8. The multi-agent based UAV formation penetration method according to claim 7, characterized in that: The objective function of the evaluation network of the decision layer is The objective function of the policy network at the decision layer is Among them, N t is the time step used to execute a sub-goal, e is the e-th time step when executing a sub-goal, s is the state, γ represents the discount rate, which ranges from 0 to 1; ω is the evaluation network parameter, θ is the policy network parameter, π θ and The probability of the policy network choosing an action, g t is the sub-goal at time t, E is the expectation, Q(s t+e ,g t ; ω) means adopting sub-goal g at time t+e t The action value function, Q(s t ,g t ; ω) means adopting sub-goal g at time t t The action-value function of .

9. The multi-agent based UAV formation penetration method according to claim 7, characterized in that: The objective function of the evaluation network of the execution layer is The objective function of the policy network of the execution layer is Among them, t is the number of steps, T is the length of the trajectory, is the scaling factor; γ represents the discount rate, ω is the evaluation network parameter, θ is the policy network parameter, and a tot represents the joint action of all agents in the execution layer at time t, Indicates that the sampling strategy network selects action a at time t t The probability of θ (a t |s t ) represents the target policy network selecting action a at time t t probability.

10. An application of the multi-agent based UAV formation penetration method according to any one of claims 1 to 9, characterized in that: It is used to coordinate and guide intelligent agents to conduct confrontation scenarios where drone formations penetrate.

Images