Multi-agent based method for formation penetration of unmanned aerial vehicles and application thereof

Abstract

The application relates to a multi-agent-based unmanned aerial vehicle formation penetration method and application thereof, constructs a multi-agent confrontation scene model, including an unmanned aerial vehicle formation and a to-be-penetrated side; roles of the agents are distinguished, a control network of part of the agents is improved, the unmanned aerial vehicle formation decomposes a penetration task into a plurality of sub-targets, and an unmanned aerial vehicle formation penetration strategy is obtained; and the application is applied to a confrontation scene for coordinating and guiding the agents to penetrate the unmanned aerial vehicle formation. The application improves convergence speed of the agents in a complex environment, enhances exploration ability of the agents, avoids falling into a local optimal solution, enhances perception ability of the agents to dynamic changes of the environment, improves decision-making effect, and improves learning efficiency of the agents.

Description

Technical Field

[0001] The present invention relates to the technical field of calculation, extrapolation or counting, and in particular to a multi-agent-based UAV formation penetration method in a complex environment in a game confrontation field and its application. Background Art

[0002] Drone penetration refers to the process by which drones, while performing military missions, circumvent enemy defenses (such as radar detection, interception weapons, and electronic jamming) and successfully infiltrate the target area to complete reconnaissance, strike, or other combat missions. With the advancement of technologies such as airborne radar systems and medium- and long-range air-to-air defense missiles, defense capabilities have continued to grow, posing significant challenges for friendly aircraft in penetrating defenses and striking targets. Early fighter jets typically employed pre-defined maneuvers such as serpentine and spiral maneuvers to penetrate defenses, but these patterns were easily detected and identified by the enemy and lacked flexibility.

[0003] In recent years, with the rapid development of artificial intelligence (AI) technology, researchers have begun to apply deep reinforcement learning theory to the field of drone penetration. Currently, drone penetration mostly uses traditional AI based on behavior trees. However, this type of AI often exhibits insufficient adaptability in complex battlefield environments. This is because manually designed fixed behavior trees are difficult to adjust according to actual conditions or generate new behavior trees that match the current state. In addition, traditional behavior tree AI usually only designs one set of behavior trees for a single type of combat entity, resulting in limited flexibility, making it difficult to reasonably respond to flexible and changing action decisions, and unable to accurately assign rewards or penalties. More importantly, traditional behavior tree AI usually only focuses on the drone entity it controls. Even if some excellent behavior trees can consider other combat entities, their joint action capabilities are limited to relatively simple scenarios, making it difficult to effectively collaborate with teammates in complex multi-agent scenarios.

[0004] On-policy algorithms consistently use data generated by the current policy during updates, avoiding inconsistencies and estimation errors caused by outdated experience. Therefore, they are suitable for tasks requiring policy consistency and stability. Multi-Agent Proximal Policy Optimization (MAPPO) is an example of an on-policy algorithm. However, these advantages also often result in low sample efficiency. After updating the model, the used sample experience can only be utilized through the old policy's probability distribution in the importance sampling weights. This limited application of sample experience prevents effective learning of the excellent sample experience from the old policy. Furthermore, in partially observable environments, MAPPO lacks a memory mechanism and relies solely on current observations for decision-making. This leads to insufficient adaptation to environmental ambiguity, difficulty in inferring hidden states, and an inability to capture long-term dependencies in time series. Consequently, it performs poorly in scenarios requiring planning based on historical information or responding to dynamic changes. Furthermore, the algorithm's exploration capabilities are limited. Its conservative policy update mechanism is restricted to the neighborhood of the current policy distribution. While this improves training stability, it limits global exploration and makes it prone to falling into local optima. Summary of the Invention

[0005] In response to the problems existing in the prior art, the present invention provides a multi-agent based UAV formation penetration method and its application.

[0006] The technical solution adopted by the present invention is a multi-agent-based UAV formation penetration method. The method constructs a multi-agent confrontation scenario model, including a UAV formation and a party to be penetrated; the roles of the agents are distinguished, and the control network of some agents is improved. The UAV formation decomposes the penetration task into several sub-goals to obtain the UAV formation penetration strategy.

[0007] Preferably, based on the multi-agent confrontation scenario model, an interactive network is established for the drone formation, including a decision-making layer and an execution layer;

[0008] Configuring the state space, action space and reward function for each agent in the decision layer and execution layer respectively;

[0009] Build an improved control network for each agent in the execution layer.

[0010] Preferably, the state space of the agent in the decision layer is the observation of all agents in the multi-agent confrontation scenario model, and the action space is 0 or 1, which is a discrete value;

[0011] The state space of each agent in the execution layer is the local observation value of the current agent, and the action space is a continuous value between -1 and 1.

[0012] Preferably, the reward function r of each agent in the execution layer is tot satisfy,

[0013]

[0014] Among them, r eps It is expressed as the reward at the end of each round, N is the number of categories of additional rewards, r n The additional reward obtained for the nth classification; the additional reward is associated with the distance between the execution layer agent and the target party, and the survival time of the execution layer agent;

[0015] In practical applications, the reward of the decision-making layer is the sum of the reward values ​​of all agents in the current execution layer.

[0016] Preferably, the improved control network includes a policy network corresponding to an agent at each execution layer;

[0017] The output of each policy network is added with a noise perturbation signal and then output to the action processing module. A feature extraction module is set between the input of the policy network and the state space of the corresponding intelligent agent.

[0018] The sampling module and evaluation network are sequentially arranged after the strategy network of the intelligent agents in all execution layers.

[0019] Preferably, the trajectory of the interaction between the agent of each execution layer and the environment is input into the entire experience pool of the sampling module, and high-quality agents are selected based on preset conditions and placed into the high-quality agent pool; the trajectory of the interaction between the agent of the execution layer and the environment is a local observation value;

[0020] The entire experience pool and the high-quality agent pool are sampled separately, the sampled data are spliced ​​and input into the global sampling pool, and the sampled data in the global sampling pool is output as the input of the evaluation network.

[0021] Preferably, the UAV formation penetration strategy is obtained and executed, and after obtaining a preset amount of interactive information, the networks corresponding to the decision layer and the execution layer are updated based on the objective function.

[0022] Preferably, the objective function of the evaluation network of the decision layer is

[0023]

[0024] The objective function of the policy network of the decision layer is

[0025]

[0026] Among them, r t+e is the agent’s reward at time t+e, N tis the time step used to execute a sub-goal, e is the e-th time step when executing a sub-goal, s is the state, γ represents the discount rate, which ranges from 0 to 1. The closer the discount rate is to 1, the greater the impact of future rewards, and the closer it is to 0, the opposite is true; ω is the evaluation network parameter, θ is the policy network parameter, π θ and The probability of selecting an action for the policy network determines the possibility of the agent choosing a certain action, g t is the sub-goal at time t; E is the expectation, Q(s t+e ,g t ;ω) means adopting sub-goal g at time t+e t The action value function, Q(s t ,g t ;ω) represents the adoption of sub-goal g at time t t The action-value function of .

[0027] Preferably, the objective function of the evaluation network of the execution layer is

[0028]

[0029] The objective function of the policy network of the execution layer is

[0030]

[0031] Where t is the number of steps, T is the trajectory length, that is, the number of steps in a complete interaction process (including multiple sub-goals), is a scaling factor used for the calculation of the evaluation network in reinforcement learning, which helps to moderate the size of the gradient used in network training and avoid numerical instability during training; γ represents the discount rate, ω is the evaluation network parameter, θ is the policy network parameter, and a tot represents the joint action of all agents in the execution layer at time t, Indicates that the sampling strategy network selects action a at time t t The probability of π θ (a t |s t ) represents the target policy network’s choice of action a at time t t probability.

[0032] An application of the multi-agent-based UAV formation penetration method is applied to a confrontation scenario in which the agents are coordinated and guided to conduct UAV formation penetration.

[0033] The present invention relates to a multi-agent-based UAV formation penetration method and its application. The method constructs a multi-agent confrontation scenario model, including a UAV formation and a target penetration party; differentiates the roles of the agents, improves the control network of some agents, and the UAV formation decomposes the penetration task into several sub-goals to obtain a UAV formation penetration strategy; and is applied to a confrontation scenario in which the UAV formation is coordinated and guided to conduct a UAV formation penetration.

[0034] The beneficial effects of the present invention are:

[0035] (1) Improve the convergence speed of intelligent agents in complex environments; use the entire experience pool and the high-quality intelligent agent pool to sample according to a certain ratio to train the network. Through the setting of the experience pool, the evaluation network can effectively learn high-quality sample experience, so that the intelligent agent can adapt to the complex environment more quickly and improve the convergence speed of the intelligent agent in complex environments;

[0036] (2) Enhance the agent's exploration ability and avoid falling into local optimal solutions; add noise perturbations to the output of the MAPPO algorithm's policy network to improve the agent's exploration ability in complex environments and improve the agent's training effect;

[0037] (3) Enhance the agent's ability to perceive dynamic changes in the environment and improve decision-making effectiveness; the feature extraction module compensates for the lack of observation by memorizing historical information, thereby more accurately estimating the current state. Through feature extraction, the agent can perceive the state of both units and environmental changes, thereby improving decision-making effectiveness;

[0038] (4) Improve the learning efficiency of the intelligent agent; by layering, the task is decomposed into multiple smaller, controllable subtasks, which can reduce the complexity of the high-dimensional space and improve the learning efficiency of the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 It is the overall framework diagram of the present invention;

[0041] Figure 3 This is a diagram of the execution layer network architecture of the present invention;

[0042] Figure 4 This is a comparison diagram of the application effects of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific implementations described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] The present invention relates to a multi-agent-based UAV formation penetration method. The method constructs a multi-agent confrontation scenario model, including a UAV formation and a target penetration party. The roles of the agents are differentiated, and the control network of some agents is improved. The UAV formation decomposes the penetration task into several sub-goals to obtain the UAV formation penetration strategy.

[0045] The method of the present invention mainly comprises the following steps:

[0046] (1) Construct a multi-agent confrontation scenario model;

[0047] (2) Differentiate the roles of intelligent agents and improve the control network of some intelligent agents;

[0048] (3) The UAV formation decomposes the penetration mission into several sub-goals and obtains the UAV formation penetration strategy;

[0049] (4) Update strategy.

[0050] The method is described in detail below with reference to examples.

[0051] (1) Construct a multi-agent confrontation scenario model;

[0052] This example constructs a 4-on-6 red-blue asymmetric air-to-ground game scenario, in which the red team has 4 drones (aircraft), each carrying ground-to-ground missiles that can attack enemy units within detection range. The blue team has 5 missile platoons and 1 base (a total of 6 targets), each carrying air-to-air missiles. Both teams earn points based on the number of enemy units destroyed and the distance from the central strategic area. When training one side, each intelligent agent represents a combat unit.

[0053] (2) Differentiate the roles of intelligent agents and improve the control network of some intelligent agents;

[0054] Based on the multi-agent confrontation scenario model, an interactive network is established for the drone formation, including a decision-making layer and an execution layer;

[0055] Among them, the decision-making layer is equivalent to a manager, issuing goals to all intelligent agents in the execution layer. Since it is only necessary to issue goals to the lower layer, the network structure of the decision-making layer adopts the single-agent PPO algorithm; the execution layer is equivalent to an employee, executing the sub-goals issued by the decision-making layer, making decisions based on local observations of the environment and sub-goals to generate actions and execute them in the environment. Since the execution layer is the actual implementer of the action and there are more than two intelligent agents in the adversarial environment, the MAPPO algorithm is adopted.

[0056] A state space and an action space are configured for each agent in the decision layer and the execution layer respectively; specifically, the state space of the agent in the decision layer is the observation of all agents in the multi-agent confrontation scenario model, including but not limited to the position information, weapon information, unit status, etc. of all our agents and the position information, weapon information, unit status, etc. of the enemy agents, and the action space is a discrete value of 0 or 1; and the state space of each agent in the execution layer is the local observation value of the current agent, including but not limited to the position information, weapon information, unit status, etc. of the current agent and the position information, weapon information, unit status, etc. of the observed enemy agents, and the action space is a continuous value between -1 and 1; in fact, the tanh activation function is used here.

[0057] Then, a reward function is set for each agent. Considering that in a multi-agent confrontation environment, if based on the traditional reward function, the agent can only be rewarded when it reaches certain conditions (such as the agent destroys the enemy or is destroyed by the enemy), this overly sparse reward state will lead to slow convergence of the algorithm or even complete non-convergence. Therefore, the reward function is modified so that the reward function of each agent in the execution layer is r tot satisfy,

[0058]

[0059] Among them, r eps It is expressed as the reward at the end of each round, including the reward for destroying enemy units and the penalty for destroying our units. This reward is calculated after the end of each round; N is the number of categories of additional rewards, r n is the additional reward obtained for the nth classification; let the additional reward here be associated with the distance between the execution layer agent and the party to be penetrated, and the survival time of the execution layer agent; for the former, r d =dis ago -dis now ,dis ago Indicates the distance between the agent and the attack target at the previous moment, dis now Represents the distance between the agent and the attack target at the current moment. For the latter, define step is the step number of this round, max_step is the maximum time step size of the environment; in practical applications, d and r time Sum them up to get the extra reward value.

[0060] Based on the above, an improved control network is constructed for each agent in the execution layer.

[0061] The improved control network includes a policy network corresponding to each execution layer agent;

[0062] The output of each policy network is added with a noise perturbation signal and then output to the action processing module. A feature extraction module is set between the input of the policy network and the state space of the corresponding intelligent agent.

[0063] Here, we add a noise disturbance signal to the output of the policy network to enhance the agent's ability to explore the environment, prevent it from falling into the local optimum, and improve the flexibility and adaptability of the strategy. The final output of the agent is,

[0064] a t =π(s t |θ)+ε,ε~N(0,σ 2 )

[0065] Among them, π(s t |θ) is the output of the agent’s policy network, and ε is Gaussian noise;

[0066] The feature extraction module uses LSTM to extract features from the local observations of each agent, and improves the accuracy of decision-making by temporally modeling the status of both units and environmental changes. Specifically, the agent inputs current observation data and historical information into the LSTM network. Because UAV penetration missions usually involve complex dynamic environments, time series dependencies, partially observable states, and the need to consider historical decisions and predictions of enemy behavior, LSTM can solve the long-term dependency problem in time series based on its characteristics of processing long-term dependencies and memorizing historical information.

[0067] The sampling module and evaluation network are sequentially arranged after the strategy network of the intelligent agents in all execution layers.

[0068] The trajectory of the interaction between the agent and the environment of each execution layer is input into the entire experience pool of the sampling module, and high-quality agents are selected based on preset conditions and placed into the high-quality agent pool;

[0069] In this embodiment, the preset condition is that each agent calculates the reward G of each round (n) after each round. n , G n =r1+r2+…+r Q , where r1 to r Q is the reward of the agent for each step in this round, and Q is the total number of steps;

[0070] If the return of this round is greater than the average return Right now The trajectory of this round is placed into the high-quality agent pool, and i represents the total number of rounds of training.

[0071] The entire experience pool and the high-quality agent pool are sampled according to a certain ratio, and the sampled data are spliced ​​and input into the global sampling pool. The sampled data in the global sampling pool is output as the input of the evaluation network, and the evaluation network is updated with the data in the global sampling pool.

[0072] The sampling method is,

[0073]

[0074] Among them, T λ Represents the type of sampled experience pool, λ values ​​0 and 1 represent the entire experience pool and high-quality intelligent agents respectively; n is the total number of experience data that the network needs to sample; σ is the coefficient weight, σ∈(0,1).

[0075] The experience selected for each neural network update is:

[0076] T={T0,T1}

[0077] The network parameters initially set include weights and biases, which are generally randomly assigned to small random values ​​ranging from 0 to 1 to prevent problems such as gradient explosion or gradient vanishing in the early stages of model training. As training progresses, these parameters will be adjusted accordingly based on performance optimization goals to enhance the agent's task execution capabilities.

[0078] (3) The UAV formation decomposes the penetration mission into several sub-goals and obtains the UAV formation penetration strategy;

[0079] In this embodiment, the policy network of the decision layer performs action interaction at a fixed period e, and sets the sub-goal g t Input to the execution layer, the agent in the execution layer takes corresponding actions according to the sub-goals and interacts with the environment, and the sub-goal g output by the policy network in the decision layer t The collection is,

[0080] goal = {attack,circumvent}

[0081] Among them, attack means executing the task of attacking the enemy's high-value target, and circumvent means avoiding the enemy's attack task. The corresponding policy network output values ​​are [0, 0.5) and [0.5, 1] ​​respectively.

[0082] The execution layer agent receives the current action output by the execution layer, that is, the target g t , by superimposing the value corresponding to the target with the original input of the strategy network of the execution layer, a vector (o t ,g t ), and then sends the vector to the policy network of each agent in the execution layer.

[0083] The UAV formation penetration strategy is obtained and executed, and after obtaining a preset amount of interactive information, the networks corresponding to the decision layer and the execution layer are updated based on the objective function.

[0084] (4) Update strategy;

[0085] The decision layer interacts with the execution layer at a fixed period e and uses the advantage function to update the evaluation network;

[0086] The objective function (advantage function) of the evaluation network of the decision layer is

[0087]

[0088] The objective function of the policy network of the decision layer is

[0089]

[0090] Among them, r t+e is the agent’s reward at time t+e, N t is the time step used to execute a sub-goal, e is the e-th time step when executing a sub-goal, s is the state, γ represents the discount rate, which ranges from 0 to 1. The closer the discount rate is to 1, the greater the impact of future rewards, and the closer it is to 0, the opposite is true; ω is the evaluation network parameter, θ is the policy network parameter, π θ and The probability of selecting an action for the policy network determines the possibility of the agent choosing a certain action, g t is the sub-goal at time t; E is the expectation, Q(s t+e ,g t ;ω) means adopting sub-goal g at time t+e t The action value function, Q(s t ,g t ;ω) represents the adoption of sub-goal g at time t t The action-value function of .

[0091] The objective function of the evaluation network of the execution layer is

[0092]

[0093] The objective function of the policy network of the execution layer is

[0094]

[0095] Among them, t is the number of steps, T is the trajectory length, that is, the number of steps in a complete interaction process, is a scaling factor used for the calculation of the evaluation network in reinforcement learning, which helps to moderate the size of the gradient used in network training and avoid numerical instability during training; γ represents the discount rate, ω is the evaluation network parameter, θ is the policy network parameter, and a tot A represents the joint action of all agents in the execution layer at time t, t (ω) represents the advantage function of evaluating the network at time t, Indicates that the sampling strategy network selects action a at time t t The probability of π θ (a t |s t ) represents the target policy network’s choice of action a at time t t probability.

[0096] The present invention also relates to a computer-readable storage medium on which a multi-agent-based UAV formation penetration program is stored. When the program is executed by a processor, the multi-agent-based UAV formation penetration method is implemented.

[0097] The present invention also relates to a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multi-agent-based UAV formation penetration method is implemented.

[0098] With reference to the aforementioned embodiment, the present invention also relates to an application of the multi-agent-based UAV formation penetration method, which is applied to a confrontation scenario in which the agents are coordinated to guide the UAV formation penetration. However, the method of the present invention is not limited thereto in practical applications. The 4-to-6 red-blue asymmetric game air-ground scenario of this embodiment is tested, and the method of the present invention (MPE-MAPPO) is compared with the traditional method (MAPPO). The test results are as follows: Figure 4 As shown, it can be seen that compared with the traditional method, the training effect of the present invention is significantly improved.

[0099] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0103] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0104] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A multi-agent-based UAV formation penetration method, characterized by: The method constructs a multi-agent confrontation scenario model, including a UAV formation and a target to be penetrated; By distinguishing the roles of intelligent agents and improving the control networks of some intelligent agents, the UAV formation decomposes the penetration task into several sub-goals and obtains the UAV formation penetration strategy; Based on the multi-agent confrontation scenario model, an interactive network is established for the drone formation, including a decision-making layer and an execution layer; Configuring a state space, an action space, and a reward function for each agent in the decision layer and the execution layer; and constructing an improved control network for each agent in the execution layer; Obtain and execute the UAV formation penetration strategy, obtain a preset amount of interactive information, and update the corresponding networks of the decision layer and execution layer based on the objective function; The objective function of the evaluation network of the decision layer is ， The objective function of the policy network of the decision layer is ， in, The time step used to execute a subgoal, The first time steps, s is the state, Indicates the discount rate, with a value between 0 and 1; To evaluate the network parameters, are policy network parameters, and The probability of selecting an action for the policy network, is the sub-goal at time t, E is the expectation, Indicates Always use subgoals The action-value function, Indicates that the sub-goal is adopted at time t The action-value function of .

2. The multi-agent-based UAV formation penetration method according to claim 1, characterized in that: The state space of the agent in the decision layer is the observation of all agents in the multi-agent confrontation scenario model, and the action space is 0 or 1; The state space of each agent in the execution layer is the local observation value of the current agent, and the action space is a continuous value between -1 and 1.

3. The multi-agent-based UAV formation penetration method according to claim 1, characterized in that: Reward function for each agent in the execution layer satisfy, ， in, It is expressed as the reward at the end of each round, N is the number of categories of additional rewards, The additional reward obtained for the nth classification; the additional reward is associated with the distance between the intelligent agent of the execution layer and the party to be penetrated, and the survival time of the intelligent agent of the execution layer.

4. The multi-agent-based UAV formation penetration method according to claim 1, characterized in that: The improved control network includes a policy network corresponding to each execution layer agent; The output of each policy network is added with a noise perturbation signal and then output to the action processing module. A feature extraction module is set between the input of the policy network and the state space of the corresponding intelligent agent. The sampling module and evaluation network are sequentially arranged after the strategy network of the intelligent agents in all execution layers.

5. The multi-agent-based UAV formation penetration method according to claim 4, characterized in that: The trajectory of the interaction between the agent and the environment of each execution layer is input into the entire experience pool of the sampling module, and high-quality agents are selected based on preset conditions and placed into the high-quality agent pool; The entire experience pool and the high-quality agent pool are sampled separately, the sampled data are spliced ​​and input into the global sampling pool, and the sampled data in the global sampling pool is output as the input of the evaluation network.

6. The multi-agent-based UAV formation penetration method according to claim 1, characterized in that: The objective function of the evaluation network of the execution layer is ， The objective function of the policy network of the execution layer is ， Among them, t is the number of steps, T is the length of the trajectory, is the scaling factor; represents the discount rate, To evaluate the network parameters, are policy network parameters, represents the joint action of all agents in the execution layer at time t, Indicates the sampling strategy network selection action at time t The probability of Indicates the target policy network selection action at time t probability.

7. An application of the multi-agent based UAV formation penetration method according to any one of claims 1 to 6, characterized in that: It is used in confrontation scenarios where intelligent agents are coordinated to guide drone formations to penetrate defenses.

Images