Heterogeneous unmanned aerial vehicle cluster target area arrival game scene and strategy generation method

Abstract

The invention discloses a heterogeneous unmanned aerial vehicle cluster target area arrival game scene and strategy generation method, and the method comprises the steps: 1) setting a scene area of a heterogeneous unmanned aerial vehicle cluster target area arrival game task, and initializing the initial ranges of a breakthrough unmanned aerial vehicle and a defense unmanned aerial vehicle; setting a target area in the initial range of the defense unmanned aerial vehicle; 2) constructing a distributed partially observable Markov model corresponding to a heterogeneous unmanned aerial vehicle cluster target area arrival game task by taking a breakthrough unmanned aerial vehicle cluster as a core perspective; each breakthrough-party unmanned aerial vehicle stores a strategy network, and the optimal action of each breakthrough-party unmanned aerial vehicle is calculated according to observation of each breakthrough-party unmanned aerial vehicle by using a distributed partial observable Markov model; summarizing observations and actions of unmanned aerial vehicles of all breakthrough parties to form a batch of data sets to train and update the strategy network; and 3) when the heterogeneous unmanned aerial vehicle cluster target area arrives at the game task and reaches a set termination condition, obtaining a strategy network of the unmanned aerial vehicle of the breakthrough party.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a method for generating strategies for a game mission scenario in which a cluster of heterogeneous unmanned aerial vehicles (UAVs) reaches a target area. Background Art

[0002] A heterogeneous drone cluster refers to a drone cluster that includes multiple different types of drones and different number configurations. It needs to use its own characteristics and advantages, cooperate with each other, and jointly complete a certain cluster task.

[0003] The target area arrival game involves a scenario in which two drone swarms attempt to break through and defend a specific target area. The drone swarm that attempts to break through the defended target area is designated as the breakout team, while the drone swarm that defends the target area is designated as the defender. The defending team's drones defend the target area based on a predetermined strategy. The breakout team is the primary research target. An intelligent algorithm is developed to dynamically plan and adjust the strategy using real-time situational information, ultimately achieving a breakthrough and reaching the target area.

[0004] System simulation verification refers to testing the effectiveness and adaptability of the algorithm under different input conditions and parameters by setting task assumptions, conducting in-the-loop simulation and deduction of intelligent decision-making technology software / hardware, and promoting the iterative upgrade of the algorithm.

[0005] The target area arrival game for drone swarms is a current focus of intelligent systems research. Its core objective is to achieve the mission objective of reaching the designated target area through the coordinated operation of multiple drones, overcoming external defenses within the target area. Drone swarm collaborative simulation, a key step in translating the concept of drone swarm game into reality, systematically tests the effectiveness of strategies under varying input and parameter conditions and their adaptability to environmental changes by constructing diverse scenarios and tasks. However, traditional collaborative methods currently suffer from poor real-time performance and low mission success rates, making them inadequate for intelligent swarm collaboration. Furthermore, existing research has largely focused on the performance optimization of single drones or simple swarm collaboration strategies, lacking in-depth research on complex, heterogeneous drone swarms. To advance drone swarms from the experimental stage to practical application, comprehensive research based on simulation and deduction of software and hardware environments, combined with autonomous intelligent decision-making technologies, is urgently needed. This will help improve the collaborative capabilities and mission execution efficiency of drone swarms, laying the foundation for future practical applications.

[0006] Currently, methods for automated control of unmanned swarm systems based on multi-agent reinforcement learning are developing rapidly. This method has attracted widespread attention due to its wide versatility and end-to-end fully automated control features. Multi-agent reinforcement learning algorithms require a large amount of simulation data for network-level training and optimization. The training process first establishes a simulation scenario based on actual mission scenarios to perform deductions and obtain simulation data. Then, a comprehensive reward and punishment mechanism based on mission requirements is established to guide the iterative optimization of the network model and converge to the expected results. The cluster collaboration scenarios in existing technologies are relatively simple, and do not consider the task scenarios of cluster games involving heterogeneous clusters. Moreover, the scenarios involved are relatively fixed, resulting in poor generalization of the subsequent generation algorithms. Summary of the Invention

[0007] In response to the problems existing in the prior art, the purpose of the present invention is to provide a method for generating strategy scenarios for a heterogeneous drone cluster target area arrival game mission, which supports simulation deduction based on scenario settings, supports the generation, recording, display and interaction of real-time data related to the execution of tasks by each drone, and supports the iterative optimization and performance verification of intelligent cluster decision-making algorithms.

[0008] This patent is based on a game scenario design for a heterogeneous drone swarm reaching a target area. It provides a fixed-strategy method for defending a target area with a drone swarm. Furthermore, it proposes a heterogeneous multi-agent reinforcement learning algorithm for a breakthrough drone swarm, building an observation, action, and reward design related to the mission objectives.

[0009] The technical solution of the present invention is:

[0010] A heterogeneous UAV cluster target area arrival game scenario and strategy generation method, the steps of which include:

[0011] 1) Set the scene area where the target area of ​​the heterogeneous drone cluster reaches the game task, initialize the initial range of the breakthrough drone and the initial range of the defending drone in the scene area; set a target area within the initial range of the defending drone; where the number of breakthrough drones is N R , including N RL A breakthrough large drone and N RS Small drones of the breakthrough side; the number of the defending side's drones is N B Initialize the position of each UAV of the breakthrough party within the initial range of the breakthrough party's UAV, and initialize the position of each UAV of the defense party within the initial range of the defense party's UAV; initialize the flight speed, detection range and damage tolerance of the UAV. The attack capability of the breakthrough party's small UAV is greater than that of the breakthrough party's large UAV, and the detection range of the breakthrough party's large UAV is greater than that of the breakthrough party's small UAV;

[0012] 2) Taking the breakthrough drone cluster as the core perspective, a distributed partially observable Markov model corresponding to the heterogeneous drone cluster target area arrival game task is constructed; each breakthrough drone saves a strategy network π θ , using the distributed partially observable Markov model based on the observation of each breakthrough party's drone t Calculate the optimal action a of the corresponding breakthrough drone in the local state it currently detects t , complete the interaction with the environment and obtain new observations o t+1 After a fixed time interval, all observations and actions of the breakthrough drones are aggregated to form a batch of data sets as the experience buffer pool, and a strategy network π is performed by maximizing the advantage function J(θ) θ The parameters of the new policy network θ are updated * The information is transmitted to each UAV of the breaching party; the actions of the UAV of the defending party are controlled by a predetermined strategy based on the information recorded by the UAV of the defending party;

[0013] 3) When the heterogeneous UAV cluster reaches the target area and the game task reaches the set termination condition, the strategy network θ of the breakthrough UAV is obtained * .

[0014] Furthermore, the information recorded by each drone includes its own position information Pos self , own orientation angle information ori self 、The amount of damage you can withstand AM self , expected arrival location information TPos self , the location information of the i-th friendly drone Relative Euclidean distance to the i-th friendly neighbor drone The relative angle with the i-th friendly drone Angle information of the i-th friendly drone The amount of damage that the i-th friendly drone can withstand The position information of the jth adversarial drone Relative Euclidean distance to the jth adversarial drone The relative angle with the jth adversarial drone Angle information of the jth adversarial drone The amount of damage that the i-th opposing drone can withstand

[0015] Furthermore, the predetermined strategies include a fixed-point patrol joint defense strategy, a threat target interception and tracking strategy, a gradient configuration and return defense strategy; wherein, the fixed-point patrol joint defense strategy is: when no target is detected, go to the default defense point and patrol; the threat target interception and tracking strategy is: after detecting the target, evaluate the distance from each detected breakthrough drone to the target area and the distance to itself, and select the detected breakthrough drone with the smallest weighted sum of the two distances as the target for interception and tracking; the gradient configuration and return defense strategy is: in the process of tracking the threat target, set the odd-numbered defender drone to continue tracking, and return to defense when the even-numbered defender drone breaks through the set maximum defense area.

[0016] Furthermore, the tuple information recorded by the distributed partially observable Markov model is in, represents the global state space, represents the action space of each drone, Indicates the drone is in the global state The local detection information is obtained from the scene, R represents the reward function under the current state of the scene, P represents the state transition probability of the global state s transferring to the new global state s′ under the joint action A of n breakthrough drones, n represents the total number of breakthrough drones in the current scene, and γ represents the discount factor for calculating the reward.

[0017] Furthermore, the breakthrough drone adopts the reinforcement learning intelligent algorithm MAPPO based on centralized training and distributed execution to obtain the real-time strategy of the breakthrough drone; the damage reward Rwd of the breakthrough small drone and the breakthrough large drone AD 、Target distance reward Rwd GD , time penalty Pnt T , out of map penalty Pnt OM , Failure penalty Pnt D The final reward of the breakthrough party's small drone is Rwd F The final reward of the large drone of the breakthrough party is Rwd′ F different.

[0018] Furthermore, for the breakthrough drone R r , set its maximum tolerable damage to The amount of damage that can be tolerated at time t is AM r (t), the amount of damage that can be tolerated at time t-1 is AM r (t-1), then the damage bonus in, Indicates the initial value of the damage reward; for the breakthrough drone R r, set the position at time t to The position at time t-1 is The target area is POS DZC , calculate the distance from the drone to the target area at time t and time t-1 respectively Target distance reward Among them, R DZ Indicates the radius of the target area, k1 and k2 represent two distance threshold parameters respectively; the final reward for the breakthrough party's small drone Final reward for the breakthrough of the large drone Satisfaction|w F |+|w′ F |=1 and |w F |≤|w′ F |, The initial final reward.

[0019] Furthermore, the scene area is long Width The target area is a rectangular area with POS DZC The center is R and the radius is DZ circular area.

[0020] Furthermore, the termination conditions set include: a) if any breakthrough party's drone reaches the target area within the specified time, or all breakthrough party's drones fail within the specified time, the breakthrough party's drone wins; b) when the specified time is reached, at least one drone in the breakthrough party's drone cluster is operating normally, and the number of breakthrough drones that have arrived at the target area is 0, the breakthrough party's drone fails.

[0021] Furthermore, if the opponent wins, the final strategy network of the breakthrough drone is retained and the reservation strategy of the defender drone is adjusted; if the opponent loses, the reservation strategy of the defender drone is retained and the final strategy network of the breakthrough drone is obtained by retraining.

[0022] The advantages of the present invention are as follows:

[0023] Addressing the shortcomings of existing drone swarm collaborative mission scenarios, this paper constructs a collaborative breakthrough scenario closer to actual missions based on two heterogeneous drone types. This not only fills a gap in the relevant technical field but also provides a more targeted and practical mission context for algorithm design.

[0024] Existing reinforcement learning methods lack consideration of the impact of initialization positions in mission scenarios, resulting in weak generalization performance. This paper proposes a random initialization method for drones. By increasing the diversity of mission scenarios, it improves the robustness of reinforcement learning algorithms in heterogeneous drone collaborative breakthrough missions.

[0025] This invention optimizes the strategic design of the defending drone, overcoming the limitations of overly simplistic strategies in existing solutions. The proposed strategy integrates fixed-point patrol and joint defense, threat target interception and tracking, gradient configuration, and return defense strategies, significantly improving the effectiveness and strategic diversity of the defending drone in collaborative missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the simulation scenario.

[0027] Figure 2 This is a schematic diagram of random scene initialization.

[0028] Figure 3 This is a schematic diagram of the initial random area of ​​the defending drone.

[0029] Figure 4 This is a schematic diagram of the random initialization of the center position and radius of the target area.

[0030] Figure 5 This is a schematic diagram of the defending drone patrol strategy.

[0031] Figure 6 This is a schematic diagram of the target distance reward for the breakthrough party’s drone. DETAILED DESCRIPTION

[0032] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0033] 1.1 Design of the game scenario for reaching the target area of ​​a heterogeneous drone swarm

[0034] The goal of the game mission scenario of heterogeneous drone cluster reaching the target area is to make any drone in the breakthrough party's drone cluster break through the defender and reach the designated target area. The breakthrough party's drone will be continuously intercepted by the defender's drone cluster during the breakthrough process. The breakthrough party's cluster should avoid the defender's drone cluster as much as possible and reduce the time consumed to complete the task. The breakthrough party's drone cluster needs to autonomously coordinate based on real-time observation information, complete distributed dynamic planning, and finally break through the target area defense. The breakthrough party's drone cluster setting includes two types of heterogeneous drones, namely large drones with a wider detection range and small drones with stronger flight performance, hereinafter referred to as large drones and small drones. The defender's drone cluster setting includes one type of drone, and the defender's drone is set to have the same performance as the breakthrough party's small drone. The schematic diagram of the multi-machine collaborative breakthrough mission scenario is as follows Figure 1 shown.

[0035] The target area arrival game task scenario of heterogeneous UAV cluster is set as a rectangular area, and the scene boundary is set. The length and width of the scene boundary are set as All drones in the scene are set to fly only within the set scene boundaries to complete the mission. The default scene boundary length and width are set as follows:

[0036]

[0037] The scene takes the upper left vertex as the origin and sets the coordinates to [0,0]. The midpoint of the scene is represented as [x,y], satisfying:

[0038]

[0039] The target area is set as a circular area. The position and radius of the target area have a great influence on the training process of the intelligent collaborative algorithm. When initializing the target area, the center position and radius need to be set. Use POS DZC 、R DZ To ensure the robustness of the algorithm, the center coordinates and radius of the circle need to be randomly initialized. The initialized center position is closer to the defending drone, and both the center position and the radius of the target area are set within a random range.

[0040] The schematic diagram of random initialization of the scenario in which the breakthrough team uses two large drones and four small drones, and the defense team uses six defense drones is as follows: Figure 2 shown.

[0041] like Figure 2 As shown, the red translucent area is the random initialization range of the breakthrough drone, the blue translucent area is the random initialization range of the defending drone, and the green translucent area is the random initialization range of the target area.

[0042] Denote the number of breakthrough drones as N R , where the number of large drones on the breakthrough side is represented by N RL , the number of small drones on the breakthrough side is expressed as N RS , denote the number of defending drones as N B The initial random area diagram of the defender's drone is as follows. Figure 3 shown.

[0043] The random initial ranges of the breakthrough drones and the defender drones in the vertical direction are set with upper and lower limits, represented by Ly1 and Ly2, and L′y1 and L′y2 respectively. The ranges outside the upper and lower limits are evenly distributed according to the number of drones, and the random initial position size of each drone is set as Ly and L′y, which are expressed as follows:

[0044]

[0045] The distance between the breakthrough drone and the defender drone to the area boundary is set to Lx1 and L′x1 respectively, and the horizontal width of the random area is L x and L′ x Therefore, for the breakthrough small drone numbered i The random initial range of its initial coordinates (x, y) is x∈[Lx1,Lx1+Lx], y∈[Ly1,Ly1+i·Ly]. i , the random initial range of its initial coordinates (x, y) is The initial random position of the breakthrough party's large drone is set to be closer to the left edge of the area and closer to the center, so that the breakthrough party's small drones are evenly distributed around the large drone to protect it. For the breakthrough party's large drone, the vertical upper and lower limits Ly1 and Ly2 are shared with the breakthrough party's small drones. RL Evenly distributed, the horizontal distance limit to the area boundary is set to half of the breakthrough party's small drone, and the maximum change range of the area is set to L L x <L L x / 2. To make the initial position of the large drone closer to the center, add a maximum longitudinal length L L y is restricted. For the large drone of the breakthrough party numbered i The random initial range of its initial coordinates (x, y) is:

[0046] x∈[Lx1 / 2,Lx1 / 2+L L x]

[0047]

[0048] The target area center position and radius are randomly initialized as shown in the following figure: Figure 4 shown.

[0049] Set the horizontal width of the random area in the center of the target area to L * x, the distance to the right region boundary is L * x1, the vertical distance to the upper and lower boundaries of the region are L * y1 and L * y2, where L * x and L * x1 satisfies:

[0050]

[0051] Where k is a hyperparameter used to adjust the horizontal random range of the target area, so that the target area is closer to the defender's drone. The default setting is k = 0.65. The target area center coordinate POS DZC The random range is:

[0052]

[0053] Target area radius R DZ The upper and lower limits of the random range are represented as minR DZ and maxR DZ ,Right now:

[0054] R DZ ∈[minR DZ ,maxR DZ ]R DZ ≥k′·maxR DZ

[0055] Where k′ is a hyperparameter used to adjust the radius of the target area. The default setting is k′=0.75, maxR DZ =100.

[0056] The termination conditions of the game task of reaching the target area of ​​the heterogeneous UAV cluster are set as shown in Table 1:

[0057] Table 1 shows the termination conditions for the game scenario in which a cluster of heterogeneous UAVs reaches the target area.

[0058]

[0059] Among them, drone failure means that the drone's tolerable damage is less than 0, and the drone is considered unable to work normally.

[0060] 1.2 Task Modeling Based on Distributed Partially Observable Markov Model

[0061] In the scenario, each drone has limited detection capabilities. It is necessary to analyze the detected information of some friendly and adversary forces and calculate the optimal action under the current local detectable state. Taking the breakthrough drone cluster as the core perspective, each breakthrough drone is regarded as an intelligent agent. The cluster mission scenario is modeled as a distributed partially observable Markov model DEC-POMDP. The recorded tuple information is recorded in the following form:

[0062]

[0063] in, represents the global state space, represents the action space of each drone, Indicates the drone is in the global state The local detection information, R represents the reward function under the current state of the scene, and P represents the joint action of n breakthrough drones. The state transition probability of the global state s to the new global state s′ under the action of , n represents the total number of breakthrough drones in the current scene, and γ represents the discount factor for calculating the return. The algorithm uses a centralized training distributed execution method to update the network parameters, that is, each breakthrough drone saves a copy of the strategy network π θ , as the task is executed, the observations available from its own perspective are obtained t , generate the corresponding action a t , complete the interaction with the environment and obtain new observations o t+1 After a fixed time interval, the observations and actions of all breakthrough drones are summarized to form a batch of data sets as the experience buffer pool. By maximizing the advantage function J(θ), a policy network parameter π is performed. θ The overall update of the new policy network parameters θ * , the expression is:

[0064]

[0065] After the update is complete, the new policy network Transmitted to each intelligent agent, that is, each breakthrough drone, for a new round of network update process.

[0066] In a swarm mission scenario, the breacher's actions are controlled by the aforementioned intelligent algorithm, while the defender's actions are governed by a predetermined strategy based on information recorded by the defending drone. Both drones engage in a confrontation within a defined rectangular area within the scenario. The breacher and defender drones are configured to acquire observation information using the same data structure, but the models used to process this information and generate corresponding control actions differ. After processing each, they generate action control commands with the same structure, driving further updates to the scenario and gradually forming new state information. This cycle repeats until the scenario's termination condition is triggered.

[0067] 1.2.1 Basic settings of drones on both sides

[0068] Both drones possess certain detection and damage capabilities. Two heterogeneous drone types are used: small drones and large drones. Compared to large detection drones, small drones possess greater attack capabilities, while large detection drones have a wider detection range. When setting up the mission scenario, both drones establish initial information such as flight speed, detection range, and damage tolerance.

[0069] During the simulation scenario, both drones can obtain observation data in real time, including their own position, direction, damage tolerance, target area position, position, direction, and damage tolerance of the breakthrough party within their own detection range, as well as the position, direction, and damage tolerance of the opponent's drone. The local observation data that can be obtained by the breakthrough party drone numbered r is represented as The local observation data that can be obtained by the breakthrough drone numbered b is expressed as The specific data tables are as follows:

[0070] Table 2 is a summary of drone observation data

[0071]

[0072] Here, i is uniformly used to represent the number of a friendly drone, and j is the number of an adversary drone. Neighboring friendly drones and neighboring adversary drones refer to those within the detection range of the own drone. Friendly adversary drones within the own detection range can be shared, while information about drones outside the own detection range cannot be observed. The relative Euclidean distances and relative angles of the aforementioned neighboring friendly drones and adversary drones are further calculated from the position information. The formula for calculating relative Euclidean distance is as follows:

[0073]

[0074] The formula for calculating the relative angle is as follows:

[0075]

[0076] 1.2.2 Action Space Settings of Both UAVs

[0077] Considering the consistency and transferability of subsequent algorithm training, the breakthrough drone and the defending drone in the scene use the same action space. Both parties use angular acceleration to achieve flight control. Both drones are fixed-wing drones, and there is a range limit on angular velocity during flight. This paper considers that the flight speed of both drones is consistent, and the angular velocity control is directly affected by the flight speed, so the angular velocity control of the drones is consistent. The angular velocity of the drone is expressed as:

[0078]

[0079] where R Turn Indicates the absolute value of the maximum variable range of angular velocity, r represents the number of the breakthrough drone, and b represents the number of the opposing drone. The action of the breakthrough drone is set to angular velocity The defender's drone's movement is set to angular velocity The action space is set to [-R Turn ,R Turn ].

[0080] 1.2.3 Update of the position status of both drones

[0081] In the environment stepping process, each drone takes its own action and Update status.

[0082] The update formula of the breakthrough party's drone action is expressed as follows:

[0083]

[0084] The update formula of the adversary drone action is expressed as follows:

[0085]

[0086] First, the angle is updated, and then the updated angle is adjusted to [-π,π]. Based on this angle, the aircraft position information is updated. The update formula for the breakthrough drone action is as follows:

[0087]

[0088]

[0089] The update formula for the defender's drone action is as follows:

[0090]

[0091] Where Spd represents the speed of the drone. The speed of each drone is the same, which is Spd. The updated aircraft position information also needs to be limited in range. Set the maximum horizontal and vertical range of the scene, which are recorded as and The range limit expression of the breakthrough party's drone is as follows:

[0092]

[0093] The range limit expression of the defender's drone is as follows:

[0094]

[0095] 1.2.4 Defender UAV Strategy Settings

[0096] The defense strategy for drones is as follows:

[0097] (1) Fixed-point patrol joint defense: When no target is detected, the patrol goes to the default defense point. The default defense point is set as a point with equal angles on the circumference of a concentric circle with a radius slightly larger than the target area. Based on the defender's drone ID, it is represented as where b∈[0,N B ], b represents the number of the defending drone, N B Indicates the total number of defending drones.

[0098] Target area center position POS DZC , and the target area radius R DZ Set the default defense radius to twice the target area radius, expressed as R df =2×R DZ Based on the defending drone's number b, the default defense points are assigned as follows:

[0099]

[0100] A schematic diagram of the patrol strategy is shown in Figure 5 shown.

[0101] (2) Threat target interception and tracking: After detecting the target, the distance between each detected breakthrough UAV and the target area and the distance to itself is evaluated, and the detected breakthrough UAV with the smallest weighted sum of the two distances is selected as the target for interception and tracking.

[0102] For the defender drone B numbered b b ,set up is the number of detected breakthrough drones, where N R is the total number of UAVs on the breakthrough side. Set the number of UAVs on the defense side B b The set of breakthrough drones within the detection range is If no breakthrough drone is detected, then If any breaching drone is detected, the expression is as follows:

[0103]

[0104] Set Defense Drone B b Your own location information is The position information of the UAV that breaks through the detection range can be observed and expressed as:

[0105]

[0106] Based on the location information POS of the target area DZC , calculate the distance to the detected breakthrough drone And the distance from the detected breakthrough drone to the target area The weighted sum of:

[0107]

[0108] Based on the weighted sum of the above distances {WDist r}, the defender's drone B b The tracking target is defined as the weighted distance and the minimum breakthrough drone, and the number expression of the intercepted target aircraft is as follows:

[0109]

[0110] (3) Gradient configuration and return defense: Set the maximum defense area for the defending drone. In the process of tracking the threat target, set the odd-numbered defending drone to continue tracking, and the even-numbered defending drone to return defense when it breaks through the maximum defense area. Set the radius R of the maximum defense area dfMax is four times the radius of the target area, that is, R dfMax =4×R DZ .

[0111] For the defender drone B numbered b b , set the distance to the target area to:

[0112]

[0113] For even-numbered defender drones, that is, those that satisfy b mod 2 = 0, they will return to defense, while odd-numbered defender drones will not be affected. When the defender's drone tracks the target T b Set as the default defense point in step (1)

[0114] 1.2.5 Breakthrough UAV Reward Function Setting

[0115] The Breakthrough UAV uses the MAPPO intelligent reinforcement learning algorithm based on centralized training distributed execution (CTDE) to train the real-time strategies of two types of Breakthrough UAVs. The basic reward function settings of the two types of Breakthrough UAVs are the same, and only the final reward R F The reward function setting table for the breakthrough drone is as follows:

[0116] Table 3 is the reward function table for the breakthrough drone MAPPO training

[0117]

[0118]

[0119] For the breakthrough drone R r , set its maximum tolerable damage to The amount of damage that can be tolerated at time t is AM r (t), the amount of damage that can be tolerated at time t-1 is AM r (t-1), then the damage bonus is Rwd AD The expression is as follows:

[0120]

[0121] where k AD Indicates the proportional change in the current amount of damage that can be tolerated compared to the previous moment and the maximum amount of damage that can be tolerated. Represents the initial value of the damage reward. The damage reward at each moment is determined by the parameter k AD Real-time adjustment. r (t) with AM r The value range of (t-1) is Then k AD The value range of Rwd is [-1,1]. AD The value range is Rewards are given when the amount of damage the drone can withstand increases, and penalties are given otherwise.

[0122] For the breakthrough drone R r , set the position at time t to The position at time t-1 is The target area is POS DZC Calculate the distance from the drone to the target area at time t and time t-1 respectively:

[0123]

[0124] Target distance bonus Rwd GD The expression is as follows:

[0125]

[0126] where R DZ Indicates the radius of the target area, k1 and k2 represent two distance threshold parameters, respectively. Rewards are given in stages for behaviors that bring the agent closer to the target area than the previous moment.

[0127] The diagram of target distance reward is as follows Figure 6 .

[0128] At each time step, a certain time penalty is given to the breakthrough agent, expressed as Pnt T , so that each agent can complete the collaborative breakthrough task as quickly as possible. When the breakthrough agent fails or the red agent reaches the map boundary, it is considered that the red agent has lost the possibility of completing the task and is punished, represented by Pnt D and Pnt OM .

[0129] The Breakthrough Agent calculates the final reward after triggering the task termination condition. The initial final reward is expressed as A positive reward is given when the mission is successful, and a negative penalty is given when the mission is unsuccessful. The final reward weight parameters are set for the two heterogeneous models of the red team to adjust the reward. The final reward of the breakthrough team's small drone is expressed as The final reward for the breakthrough party's large drone is expressed as Satisfaction|w F |+|w′ F |=1 and |w F |≤|w′ F |.

[0130] 1.2.6 Breakthrough UAV Agent Training Framework

[0131] The Breakthrough Party UAV agent training framework utilizes a centralized training and distributed execution architecture. Learning is accelerated by sharing parameters across all Breakthrough Party agents, allowing only a common evaluator (Critic) and a few types of action policies (Actors) to be learned for agent training. All UAV agents of the same type share action policy parameters. Based on the different local observations of each UAV, the action policy network (Actor) will output different actions. The network parameter learning process for each UAV agent remains independent, with each action policy evaluator (Critic) independently estimating a local value function based on the interaction data of different UAV agents.

[0132] In traditional methods, the independent learning and training process of drone action strategies lacks information sharing, making it difficult to learn coordinated strategies that rely on interactions between multiple drone agents. It is also difficult for a single drone agent to estimate the contribution of its behavior to the overall reward obtained by the entire drone cluster. To accelerate the convergence of the drone agent network, a centralized evaluator, counterfactual baseline, and cluster total value decomposition strategy are added to the cluster adversarial training process. The cluster adopts a centralized training distributed execution training framework based on the multi-agent proximal policy optimization algorithm MAPPO to complete the breakthrough agent training and optimization process. The agent training and optimization process is as follows:

[0133]

[0134]

[0135] The advantage function A of each agent r r (s t ,a t ) is obtained by the evaluation network for each action based on comparison while ensuring that other agents remain unchanged. The Q-value function estimate and its counterfactual baseline b(s t ,a t -r ) is obtained, and the calculation formula is as follows:

[0136]

[0137] The value function estimate for each agent is the original value function The calculation is further based on the truncated TD(λ) method, and the calculation formula is as follows:

[0138]

[0139] where δ t Indicates the timing difference error TD error.

[0140] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A heterogeneous UAV swarm target area arrival game scenario and strategy generation method, the steps of which include: 1) Set the scene area where the target area of ​​the heterogeneous drone cluster reaches the game task, initialize the initial range of the breakthrough drone and the initial range of the defending drone in the scene area; set a target area within the initial range of the defending drone; where the number of breakthrough drones is N R , including N RL A breakthrough large drone and N RS Small drones of the breakthrough side; the number of the defending side's drones is N B Initialize the position of each UAV of the breakthrough party within the initial range of the breakthrough party's UAV, and initialize the position of each UAV of the defense party within the initial range of the defense party's UAV; initialize the flight speed, detection range and damage tolerance of the UAV. The attack capability of the breakthrough party's small UAV is greater than that of the breakthrough party's large UAV, and the detection range of the breakthrough party's large UAV is greater than that of the breakthrough party's small UAV; 2) Taking the breakthrough drone cluster as the core perspective, a distributed partially observable Markov model corresponding to the heterogeneous drone cluster target area arrival game task is constructed; each breakthrough drone saves a strategy network π θ , using the distributed partially observable Markov model based on the observation of each breakthrough party's drone t Calculate the optimal action a of the corresponding breakthrough drone in the local state it currently detects t , complete the interaction with the environment and obtain new observations o t+1 After a fixed time interval, all observations and actions of the breakthrough drones are aggregated to form a batch of data sets as the experience buffer pool, and a strategy network π is performed by maximizing the advantage function J(θ) θ The parameters of the new policy network θ are updated * The information is transmitted to each UAV of the breaching party; the actions of the UAV of the defending party are controlled by a predetermined strategy based on the information recorded by the UAV of the defending party; 3) When the heterogeneous UAV cluster reaches the target area and the game task reaches the set termination condition, the strategy network θ of the breakthrough UAV is obtained * .

2. The method according to claim 1, characterized in that The information recorded by each drone includes its own location information Pos self , own orientation angle information ori self 、The amount of damage you can withstand AM self , expected arrival location information TPos self , the location information of the i-th friendly drone Relative Euclidean distance to the i-th friendly neighbor drone The relative angle with the i-th friendly drone Angle information of the i-th friendly drone The amount of damage that the i-th friendly drone can withstand The position information of the jth adversarial drone Relative Euclidean distance to the jth adversarial drone The relative angle with the jth adversarial drone Angle information of the jth adversarial drone The amount of damage that the i-th opposing drone can withstand 3. The method according to claim 2, characterized in that The predetermined strategies include a fixed-point patrol joint defense strategy, a threat target interception and tracking strategy, and a gradient configuration and return defense strategy; wherein the fixed-point patrol joint defense strategy is: when no target is detected, go to the default defense point and patrol; the threat target interception and tracking strategy is: after detecting the target, evaluate the distance from each detected breakthrough drone to the target area and the distance to itself, and select the detected breakthrough drone with the smallest weighted sum of the two distances as the target for interception and tracking; the gradient configuration and return defense strategy is: in the process of tracking the threat target, set the odd-numbered defender drone to continue tracking, and return to defense when the even-numbered defender drone breaks through the set maximum defense area.

4. The method according to claim 1, 2 or 3, characterized in that: The tuple information recorded by the distributed partially observable Markov model is: in, represents the global state space, represents the action space of each drone, Indicates the drone is in the global state Under the local detection information, R represents the reward function under the current state of the scene, and P represents the global state s transferred to the new global state s under the joint action A of n breakthrough drones. ′ The state transition probability of n is n, n represents the total number of breakthrough drones in the current scene, and γ represents the discount factor for calculating the return.

5. The method according to claim 1, 2 or 3, characterized in that: The breakthrough drone adopts the reinforcement learning intelligent algorithm MAPPO based on centralized training distributed execution to obtain the real-time strategy of the breakthrough drone; the damage reward Rwd of the breakthrough small drone and the breakthrough large drone AD 、Target distance reward Rwd GD , time penalty Pnt T , out of map penalty Pnt OM , Failure penalty Pnt D The final reward of the breakthrough party's small drone is Rwd F The final reward of the large drone with the breakthrough party Rwd ′F different.

6. The method according to claim 5, characterized in that For the breakthrough drone R r , set its maximum tolerable damage to The amount of damage that can be tolerated at time t is AM r (t), the amount of damage that can be tolerated at time t-1 is AM r (t-1), then the damage bonus in, Indicates the initial value of the damage reward; for the breakthrough drone R r , set the position at time t to The position at time t-1 is The target area is POS DZC , calculate the distance from the drone to the target area at time t and time t-1 respectively Target distance reward Among them, R DZ Indicates the radius of the target area, k1 and k2 represent two distance threshold parameters respectively; the final reward for the breakthrough party's small drone Final reward for the breakthrough of the large drone Satisfaction|w F |+|w ′F |=1 and |w F |≤|w ′F |, The initial final reward.

7. The method according to claim 1, 2 or 3, characterized in that: The scene area is long Width The target area is a rectangular area with POS DZC The center is R and the radius is DZ circular area.

8. The method according to claim 1, 2 or 3, characterized in that: The set termination conditions include: a) If any breakthrough drone reaches the target area within the specified time, or all breakthrough drones fail within the specified time, the breakthrough drone wins; b) When the specified time is reached, at least one drone in the breakthrough drone cluster is working normally, and the number of breakthrough drones that have arrived at the target area is 0, the breakthrough drone fails.

9. The method according to claim 8, characterized in that If the opponent wins, the final strategy network of the breakthrough drone is retained and the reservation strategy of the defense drone is adjusted; if the opponent loses, the reservation strategy of the defense drone is retained and the final strategy network of the breakthrough drone is obtained by retraining.

Images