Multi-UAV cooperative strategy learning method and device based on temporal hierarchical decision-making

Abstract

The present application relates to a multi-UAV collaborative strategy learning method and device based on temporal hierarchical decision-making. This method overcomes the problems of high complexity and large decision space of multi-UAV sequential decision-making by performing temporal abstraction on the original sequential decision sequence, thereby improving the optimization efficiency of multi-UAV collaborative strategy. Temporal abstraction is performed on the original sequential decision sequence to construct a drone macro-action space capable of reaching a positive reward state. A temporal hierarchical decision-making method including macro-strategies and micro-strategies is introduced. The macro-strategy makes top-level decisions based on macro-actions and updates them using the intrinsic rewards of equivalent state novelty values combined with extrinsic rewards. The micro-strategy makes bottom-level decisions based on the original actions and is generated through rule-based definition or pre-training. In order to overcome the weakening of collaborative strategy performance caused by hierarchical decision-making, a strategy parallel optimization mechanism based on experience reuse is designed. This method improves the efficiency of strategy optimization and enhances the performance of UAV collaborative strategies.

Description

Technical Field

[0001] The present application relates to the technical field of multi-UAV collaborative strategy learning methods, and in particular to a multi-UAV collaborative strategy learning method and device based on temporal hierarchical decision-making. Background Art

[0002] In recent years, multi-UAV collaborative tasks (such as area search, formation control, and disaster relief) have become a research hotspot in the field of intelligent unmanned systems due to their flexibility and efficiency. To achieve autonomous collaboration in complex environments, multi-agent reinforcement learning has been widely used to learn collaborative strategies for UAVs. However, in real-world scenarios, task objectives often correspond to extremely sparse extrinsic reward signals (e.g., only a one-time reward upon task completion). This results in inefficient exploration of the agents within the vast collaborative decision space and difficulty in achieving convergence, becoming a key bottleneck hindering the implementation of this technology.

[0003] Under sparse reward conditions, intrinsic rewards drive agents to search for positive (extrinsic) rewards in state space. Essentially, they drive agents to search for valid decision sequences that lead to positively rewarded states in a collaborative sequential decision space. While intrinsic rewards provide an effective decision sequence search strategy, they do not optimize the sequential decision space itself. The size of the collaborative sequential decision space grows exponentially with the number of agents and the length of the decision sequences. In long-term collaborative tasks, agents must complete tasks through continuous coordinated action sequences, posing significant challenges to the search and utilization of sparse positive rewards.

[0004] Furthermore, multi-UAV collaboration requires addressing real-world constraints such as distributed decision-making under limited communication, coordinating heterogeneous capabilities, and addressing dynamic environmental disturbances, further complicating policy learning under sparse reward conditions. Existing approaches often suffer from divergent exploration directions and disordered collaborative behavior during long-term missions, making it difficult to meet practical requirements for mission success rates and algorithm robustness. Summary of the Invention

[0005] Based on this, it is necessary to provide a multi-UAV collaborative strategy learning method and device based on temporal hierarchical decision-making to address the above technical problems.

[0006] A multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making, the method comprising:

[0007] The temporal hierarchical decision mechanism is used to perform temporal abstraction on the original sequential decision sequence, and a macro action space of the drone with the ability to achieve a positive reward state is constructed; each macro action is represented by It consists of a set of primitive actions.

[0008] According to the current observation, macro action and original action, the pre-built hierarchical strategy is trained; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, in which each drone A macro action is selected at each step; a micro strategy is used to select an original action at each step based on the current observation and the most recently selected macro action.

[0009] The UAV conducts hierarchical decision search in the macro action space through a hierarchical strategy, and stores the experience samples generated by the UAV's hierarchical decision into the experience cache.

[0010] After the drones use the hierarchical strategy to search for positive extrinsic rewards, they use the experience samples containing the original actions sampled from the experience cache to train the pre-built parallel strategy offline. Then, the parallel strategy is gradually used to replace the hierarchical strategy to obtain a multi-drone collaborative strategy. The parallel strategy is used to output the original actions based on the current observations, and only extrinsic rewards are used during training.

[0011] In one embodiment, a temporal hierarchical decision-making mechanism is used to perform temporal abstraction on the original sequential decision sequence to construct a macroscopic action space of a drone capable of achieving a positive reward state, including:

[0012] Based on the analysis and induction of known successful collaborative behaviors, the original action sequences coexisting in multiple collaborative modes are extracted as macro-actions.

[0013] The multi-UAV collaborative action sequence of a successful mission is decomposed into macro-action sequences, and a reasonable macro-action space is abstracted from a variety of spatiotemporal coordination modes.

[0014] A multi-UAV cooperative strategy learning device based on temporal hierarchical decision-making, comprising:

[0015] The UAV macro action space construction module is used to use the temporal hierarchical decision-making mechanism to perform temporal abstraction on the original sequential decision sequence and construct the UAV macro action space capable of achieving a positive reward state; each macro action is composed of It consists of a set of primitive actions.

[0016] The hierarchical strategy training module is used to train the pre-built hierarchical strategy based on the current observation, macro action and original action; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, in which each drone A macro action is selected at each step; a micro strategy is used to select an original action at each step based on the current observation and the most recently selected macro action.

[0017] The hierarchical decision module is used for the UAV to perform hierarchical decision search in the macro action space through a hierarchical strategy, and store the experience samples generated by the UAV's hierarchical decision into the experience cache.

[0018] The multi-UAV collaborative strategy determination module is used to train a pre-built parallel strategy offline using experience samples containing primitive actions sampled from the experience cache after the UAVs use the hierarchical strategy to search for positive extrinsic rewards. The parallel strategy is then used to gradually replace the hierarchical strategy to obtain a multi-UAV collaborative strategy. The parallel strategy is used to output primitive actions based on the current observation, and only extrinsic rewards are used during training.

[0019] The aforementioned multi-UAV collaborative strategy learning method and device based on temporal hierarchical decision-making overcomes the high complexity and large decision space of multi-UAV sequential decision-making by performing temporal abstraction on the original sequential decision sequence, thereby improving the optimization efficiency of multi-UAV collaborative strategies. Temporal abstraction is performed on the original sequential decision sequence to construct a drone macro-action space capable of achieving a positive reward state. A temporal hierarchical decision-making method comprising macro- and micro-strategies is introduced. The macro-strategy makes top-level decisions based on macro-actions and is updated using intrinsic rewards combined with extrinsic rewards based on equivalent state novelty values. The micro-strategy makes bottom-level decisions based on the original actions and is generated through rule-based definition or pre-training. To overcome the weakening of collaborative strategy performance caused by hierarchical decision-making, a strategy parallel optimization mechanism based on experience reuse is designed. This method improves strategy optimization efficiency and effectively enhances the performance of UAV collaborative strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 1 is a flow chart of a multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making in one embodiment;

[0021] Figure 2 Schematic diagram of a multi-UAV mission scenario in another embodiment, wherein Figure 2 (a) is a schematic diagram of scenario 2 UAV-1ADS-2, Figure 2 (b) is a schematic diagram of UAV-1ADS in scenario 3. Figure 2 (c) is a schematic diagram of UAV-2ADS in scenario 3. Figure 2 (d) is a schematic diagram of UAV-2ADS in scenario 5;

[0022] Figure 3 This is a graph showing the cumulative extrinsic rewards and mission success rate for scenario 2 UAV-1ADS-2 in another embodiment, where Figure 3 (a) is the cumulative external reward curve. Figure 3 (b) is the task success rate curve;

[0023] Figure 4This is a curve diagram of the cumulative external rewards and mission success rate of UAV-1ADS in scenario 3 in another embodiment, where Figure 4 (a) is the cumulative external reward curve. Figure 4 (b) is the task success rate curve;

[0024] Figure 5 This is a curve diagram of the cumulative extrinsic rewards and mission success rate of UAV-2ADS in scenario 3 in another embodiment, where Figure 5 (a) is the cumulative external reward curve. Figure 5 (b) is the task success rate curve;

[0025] Figure 6 This is a graph showing the cumulative extrinsic rewards and mission success rate of UAV-2ADS in scenario 5 in another embodiment, where Figure 6 (a) is the cumulative external reward curve. Figure 6 (b) is the task success rate curve;

[0026] Figure 7 A bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in another embodiment, wherein Figure 7 (a) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 2UAV-1ADS-2. Figure 7 (b) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-1ADS. Figure 7 (c) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-2ADS. Figure 7 (d) A bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 5 UAV-2ADS. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] In one embodiment, Figure 1 As shown, a multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making is provided, which includes the following steps:

[0029] Step 100: Use the temporal hierarchical decision mechanism to perform temporal abstraction on the original sequential decision sequence and construct a macro action space of the drone capable of achieving a positive reward state; each macro action is represented by It consists of a set of primitive actions.

[0030] Specifically, the multi-UAV system is modeled as a distributed partially observable Markov decision process (Dec-POMDP) model, and the size of the multi-UAV collaborative sequential decision space is ,in is the size of the UAV’s individual action space, is the number of drones, is the decision sequence length, that is, the maximum decision step length of each round.

[0031] Multi-UAV cooperative strategy learning aims to search and optimize the state that can reach positive rewards from the collaborative sequential decision space as quickly as possible. Valid action sequence ,in For multi-UAV joint action, From the initial state To the positive reward state The number of state transitions, .

[0032] For long-term collaborative tasks, the state containing positive rewards is far away from the initial state, that is, Larger, maximum number of decision steps It is usually set to a large value, which leads to two problems that affect strategy optimization:

[0033] First, the size of the multi-UAV collaborative sequential decision space varies with the length of the decision sequence. The increase is exponential, driven only by the novelty value of equivalent states, for the effective decision sequence The search efficiency still needs to be improved;

[0034] The second is the long feedback delay of the positive reward signal, combined with the time discount factor When used for policy updating, the guiding effect increases with Increase and decrease.

[0035] The temporal hierarchical decision-making mechanism is introduced to narrow the collaborative sequential decision space, reduce the decision complexity, and improve the efficiency of multi-UAV collaborative strategy optimization. The temporal hierarchical decision-making mechanism performs temporal abstraction on the original decision sequence and constructs the UAV macro action space. Each macro action Depend on primitive actions, namely , , . An integer greater than zero.

[0036] Step 102: Based on the current observation, macro action and original action, the pre-built hierarchical strategy is trained; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, wherein each drone A macro action is selected at each step; a micro strategy is used to select an original action at each step based on the current observation and the most recently selected macro action.

[0037] Specifically, based on the macro action space, this application proposes a Temporal-Hierarchical decision making-based Multi-UAV cooperative Policy Learning method (TH-MUPL), which trains the UAV hierarchical cooperative strategy and further optimizes it through parallel strategy learning based on offline training. and micro-strategies It is the basis for hierarchical decision-making. Macro strategy The input is the observation at the current moment , the output is macro action , every Then, reselect the macro action. Receive the most recently selected macro action and the current moment observation As input, the output is the original action .

[0038] In the multi-UAV cooperative strategy learning method based on temporal layering, the macro strategy It is responsible for determining the timing of each macro action according to the changing situation, and micro strategy Responsible for selecting the appropriate primitive action To complete the currently specified macro action , thereby decoupling the collaborative decision-making behavior from the time sequence and reducing the complexity of decision-making. In addition to macro strategies and micro strategies, drones also include parallel strategies , the input of the parallel strategy is the current observation , the output is the original action .

[0039] In order to improve the stability and efficiency of training, the macro strategy of the drone in this method and micro-strategies Optimize independently.

[0040] Step 104: The UAV performs hierarchical decision search in the macro action space through a hierarchical strategy, and stores the experience samples generated by the UAV hierarchical decision in the experience cache.

[0041] Step 106: After the UAVs use the hierarchical strategy to search for positive extrinsic rewards, they use the experience samples containing the original actions sampled from the experience cache to offline train the pre-built parallel strategy, and then gradually use the parallel strategy to replace the hierarchical strategy to obtain the multi-UAV collaborative strategy; the parallel strategy is used to output the original action based on the current observation, and only uses the extrinsic rewards during training.

[0042] Specifically, since the macro-actions defined are primitive action sequences of fixed length, the macro-actions switched at fixed intervals under the temporal hierarchical decision-making mechanism lead to reduced flexibility in state transitions, and the performance of the multi-UAV hierarchical collaborative strategy generated by training is subsequently reduced. Therefore, a parallel strategy offline training mechanism is proposed. After searching for positive extrinsic rewards, the parallel strategy is trained offline using experience samples containing primitive actions. , and gradually replace the hierarchical strategy, thereby further improving the optimization of the multi-UAV collaboration strategy.

[0043] The aforementioned multi-UAV collaborative strategy learning method based on temporal hierarchical decision-making overcomes the high complexity and large decision space of multi-UAV sequential decision-making by performing temporal abstraction on the original sequential decision sequence, thereby improving the optimization efficiency of multi-UAV collaborative strategies. Temporal abstraction of the original sequential decision sequence constructs a macro-action space for UAVs capable of achieving a positive reward state. A temporal hierarchical decision-making method comprising macro- and micro-strategies is introduced. The macro-strategy makes top-level decisions based on macro-actions and is updated using intrinsic rewards combined with extrinsic rewards based on equivalent state novelty values. The micro-strategy makes bottom-level decisions based on the original actions and is generated through rule-based definition or pre-training. To overcome the weakening of collaborative strategy performance caused by hierarchical decision-making, a policy parallel optimization mechanism based on experience reuse is designed. This method improves strategy optimization efficiency and effectively enhances the performance of UAV collaborative strategies.

[0044] In one embodiment, step 100 includes: analyzing and summarizing known successful collaborative behaviors, extracting original action sequences coexisting in multiple collaborative modes as macro-actions; decomposing the multi-UAV collaborative action sequences of successful missions into macro-action sequences, and abstracting a reasonable macro-action space from multiple spatiotemporal coordination modes; wherein the macro-action space is:

[0045] ;

[0046] in, is the macro action space.

[0047] Specifically, under the temporal hierarchical decision-making mechanism, the drone The decision-making process consists of: Shift to hierarchical decision making:

[0048] ;

[0049] Each drone Each step selects a macro action; each step is based on the current observation and the most recently selected macro action Select the original action . Multiple drones The size of the macro decision space of step sequential decision is ,in is the size of the individual macro action space. Therefore, if the underlying decision is a predefined deterministic mapping, and the macro action satisfies , which can realize the compression of sequential decision space, namely The construction of the macro action space must meet the reachability requirement from the initial state to the positive reward state, that is, for any initial state , there are macro decision sequences , , , corresponding to the micro decision sequence ,satisfy , , , so that the state can be changed from Transfer to positive reward state , that is, there is a state transition sequence ,satisfy:

[0050] ;

[0051] in, represent the transition probabilities of the two states respectively.

[0052] To construct a macro-action space that meets these requirements, we analyze and summarize known successful collaborative behaviors, extracting essential primitive action sequences that coexist across multiple collaborative modes as macro-actions. We decompose the multi-UAV collaborative action sequence for a successful penetration assault into macro-action sequences, and abstract a reasonable macro-action space from various spatiotemporal coordination modes.

[0053] The individual behaviors of multiple drones in the collaborative execution of tasks are abstracted as: approaching the target, moving away from the target, flying around the target clockwise, flying around the target counterclockwise, and circling in place. The individual behaviors of the above drones must meet certain time and space constraints and be executed in a specific time sequence to achieve the collaborative effect. Therefore, a macro action space based on time sequence abstraction is defined. .

[0054] In one embodiment, step 102 includes: the micro-strategy is defined by pre-training or rule-based definition, and the macro-strategy is obtained by pre-training using a reinforcement learning method; the current observation and macro-action are input into the micro-strategy, and the original action is output; the macro-strategy of the drone is updated using the ACER method, and the macro-strategy includes a drone macro-value network and a macro-strategy network.

[0055] In one embodiment, the current observation and macro action are input into the micro policy, and the original action is output, including: when the macro action chooses to approach the target, the drone micro policy expression is:

[0056] ;

[0057] ;

[0058] in, is the drone’s micro-strategy when the macro-action is to approach the target, Indicates the current observation of drone i, Indicates proximity to the target. Indicates drone Select the heading angle unit vector after each primitive action, Indicates that from the drone The unit vector pointing to the protected target, Indicates the k A primitive action; represents the original action space, They respectively represent the location coordinates of the protected targets.

[0059] When macro actions choose to approach threats k When , the drone micro-strategy expression is:

[0060] ;

[0061] ;

[0062] in, represents the drone’s micro-strategy when the macro action is to approach the threat k, Indicates drone Point to the threat area The unit vector of Indicates approaching threat k, Express threats respectively k The location coordinates of Respectively represent drones i exist tThe position coordinates at the moment.

[0063] When macro actions choose to stay away from the threat area When , the drone micro-strategy expression is:

[0064] ;

[0065] in, Indicates that when the macro action is to move away from the threat area k Drone micro-strategy at the time, Stay away from the threat area .

[0066] When the macro action is to fly clockwise around the nearest threat area, the micro-strategy expression of the drone is:

[0067] ;

[0068] ;

[0069] in, It represents the drone’s micro-strategy when the macro-action is to fly clockwise around the nearest threat area. Indicates flying clockwise around the nearest threat area. Representation and from drone A unit vector that is perpendicular to the vector closest to the threat area and points in a clockwise direction. Indicates distance from drone The location of the nearest threat area.

[0070] When the macro action is to fly counterclockwise around the nearest threat area, the micro-strategy expression of the drone is:

[0071] ;

[0072] in, It represents the drone's micro-strategy when the macro-action is to fly counterclockwise around the nearest threat area; Indicates the nearest threat area when flying counterclockwise;

[0073] When the macro action is to hover in place, the micro-strategy expression of the drone is:

[0074] ;

[0075] in, represents the drone's micro-strategy when the macro-action is circling in place; The center of the circle is the center of the macro action. The time is the reference, the closest threat area at that time and the drone at that time The line extending outwards to the position point of 0.5km, Indicates the reference moment of the macro action.

[0076] Circling Center for:

[0077] ;

[0078] in, To perform the original action Next moment drone The predicted location of Indicates the drone at this moment i The angle between the flight and the horizontal plane, Indicates the time interval, Represents the macro value network parameters.

[0079] In one embodiment, the drone macro value network is updated as follows:

[0080] ;

[0081] ;

[0082] ;

[0083] in, represents the UAV macro value network parameters, represents the update hyperparameters of the UAV macro value network parameters, represents the gradient of the mean square error of the macro value network, Represents the value estimation of macro strategy based on Retrace method, Indicates drone i Current observations, express Moment Drone i The macro action, represents the total reward, express The truncated importance sampling coefficient value at time , represents the value network, Indicates the total number of threat areas, represents the hyperparameter, Indicates observation The value function of Respectively represent the drone at that moment i Observation and macro-action.

[0084] In one embodiment, the drone macro policy network is updated as follows:

[0085] ; ;

[0086] in, represents the parameters of the drone macro strategy network, represents the update hyperparameters of the drone macro-strategy network parameters, represents the ACER policy gradient, Indicates the total number of threat areas, Indicates observation The value function of Respectively t Two different values of truncated importance sampling coefficients at time t, represents the drone macro value network, represents the drone macro strategy network, Indicates the i A macro action, Indicates drone i Current observations, Represents the value estimation of the macro strategy based on the Retrace method.

[0087] For the sake of simplicity in this article, some and Replacing the drone macro strategy network and the drone macro value network .

[0088] In one embodiment, a concurrent experience replay mechanism is used to store and extract drone experience samples; the concurrent experience replay mechanism includes: the experience cache capacity of each drone is equal, the storage of experience samples follows the first-in-first-out principle, and each experience cache is guaranteed to be located at the same index position. The experience tuples of are derived from the same state transition process; when sampling the experience cache, the indices of the batch samples sampled by all drones are the same.

[0089] Specifically, the micro-strategy is pre-trained or defined based on rules, and then the macro-strategy is learned based on the stable and effective micro-strategy, so as to continuously optimize the switching timing of each macro-action. Since the macro-actions of the drone are relatively simple and there is no complex decision-making, the micro-strategy is defined in a rule-based way. When the macro action setting is more complex and difficult to complete through rule-based methods, reinforcement learning methods can be used for pre-training. The original individual action space of the drone:

[0090] ;

[0091] in, Indicates that the angular acceleration of the UAV is gradually adjusted to , the speed is gradually adjusted to .

[0092] Micro-strategy is a deterministic strategy, that is, based on the current observation and macro actions Output specific actions , rather than the probability distribution of actions, so the micro-strategy Expressed as .

[0093] When the macro action selects "approach target", that is, When , the drone’s micro-strategy is as shown above. When the macro action is to move closer to the target, the drone’s micro-strategy expression is shown above.

[0094] When the macro action is chosen to be close to the threat k, that is, When , the drone’s micro-strategy is as shown above: When the macro action is to approach the threat, the drone’s micro-strategy expression is shown above.

[0095] When the macro action is chosen to be far away from the threat area k, that is, When , the drone’s micro-strategy is as shown above when the macro action chooses to stay away from the threat area.

[0096] When the macro action is "fly clockwise around the nearest threat area", When the macro action is to fly clockwise around the nearest threat area, the drone's micro-strategy is as shown in the expression.

[0097] When the macro action is "fly counterclockwise around the nearest threat area", When the macro action is to fly counterclockwise around the nearest threat area, the drone's micro-strategy is as shown above.

[0098] When the macro action selects "hover in place", When the macro action is to hover in place, the drone's micro-strategy is as shown above.

[0099] Drone micro-strategy Choose to make drone The predicted location Distance from the center of the circle The original action closest to the 0.5km circling radius The macro strategy of drones is updated using the ACER method, involving the drone macro value network. and macro strategy network .

[0100] Macro value network parameters By minimizing the mean square error To update:

[0101] ;

[0102] in, Represents the value estimation of macro strategy based on Retrace method, , used to update the macro value and reward of the policy function For drones Macro Action The corresponding original action sequence The sum of the rewards generated, ,award Include extrinsic rewards and the equivalent state novelty value intrinsic reward Respectively t Two different values of truncated importance sampling coefficients at time t, , For drones According to observation Select Macro Action Historical macro-strategy at the time.

[0103] Parameters of the drone macro strategy network Based on ACER policy gradient renew:

[0104] ;

[0105] In the process of multi-UAV collaborative strategy learning, after the macro action is completed, the experience tuple generated will be Store to macro experience cache After each decision round, the previous experience samples are randomly sampled from the experience cache to train the current strategy. UAV for sampling i Historical decision-making trajectory as follows:

[0106] ;

[0107] Leveraging historical decision trajectories Computing Macro Value Network Gradient of parameters; value network parameters and policy network parameters Both are updated based on gradients.

[0108] In one embodiment, after the drone searches for a positive extrinsic reward using a hierarchical strategy, it uses a cached experience sample containing the original action to offline train a pre-built parallel strategy, including: after the drone stably searches for a positive extrinsic reward using the hierarchical strategy, it begins training the parallel strategy using offline experience samples and optimizes the parallel strategy using D3QN; the parallel strategy includes an action value network and a target value network; the action value network and the target value network have the same structure but different parameters; the action value network parameters are updated in the following manner:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] in, represents the action-value network parameters, represents the action-value network, represents the loss function, For external rewards, Indicates value, represents the learning rate, express The first truncated importance sampling coefficient value at time, represents the observation at the next moment, 、 Respectively represent drones i The current observation and the original action, represents the advantage function network parameters, represents the advantage function network, represents the state value function network parameters, Represents the state-value function network.

[0114] The target value network parameter update method is:

[0115] ;

[0116] in, represents the target value network parameter, Represents the learning rate.

[0117] In one embodiment, a parallel strategy is adopted to gradually replace the hierarchical strategy, including: Adjustment strategy replacement process, , is a positive integer; after searching for a positive reward, start training the parallel strategy; when When , the parallel strategy is used instead of the hierarchical strategy to make decisions, and the number of decision rounds is recorded as ;in It represents the success rate of UAVs completing tasks through hierarchical collaboration strategy; The parallel strategy completely replaces the hierarchical strategy in the transition round. The parallel strategy of the drone is not presented in an independent parameterized form, but is defined according to the action value function. All drones are based on Method for action selection:

[0118] ;

[0119] in , , drones The probability of selecting the action corresponding to the maximum action value function is All candidate actions are randomly selected with probability Take the smaller value between 0 and 1, represents the original action space, represents the action-value network, k Indicates the action sequence number, represents the action-value network parameters, 、 Respectively represent drones i The current observation and the original action, Indicates the k A primitive action, Indicates drone i parallel strategy.

[0120] Specifically, the introduction of a temporal hierarchical decision-making mechanism effectively narrows the collaborative sequential decision space. By rationally setting the macro-action space, redundant decision branches in long-term collaborative tasks are avoided, effectively improving the efficiency of collaborative strategy learning under sparse reward conditions. However, the hierarchical strategy based on the macro-action space has the following two limitations: first, the macro-action is a fixed-length primitive action sequence that switches only at certain intervals, and all drones switch simultaneously; second, the macro-action space is artificially defined. Although it is based on the analysis of successful collaborative behaviors that have emerged autonomously from drones in previous work, it is difficult to accurately split the optimal collaborative mode into a fixed-length primitive action sequence. Therefore, while the setting of macro-actions reduces decision complexity, the fixed and limited action sequence setting also reduces the flexibility of state transitions, resulting in redundant actions in the trained hierarchical collaborative strategy when completing tasks, and the strategy performance needs to be improved.

[0121] To solve this problem, a parallel strategy optimization mechanism based on offline training is proposed. That is, when the UAV searches for positive external rewards through hierarchical strategies, it uses the generated experience samples to train the parallel strategy offline. Unlike macro strategies, parallel strategies According to current observations , output the original action , only extrinsic rewards are used during training Considering that the drone passes and Hierarchical selection actions , and micro-strategy Based on the rule definition, the probability distribution of each action in the historical action strategy Difficult to obtain and record directly, so choose not to use The offline learning method D3QN (Dueling Double DeepQ-Network) optimizes the parallel strategy. Each drone maintains its own value network , based on the Dueling structure, the action value function consists of two parts: state value and advantage function:

[0122] ;

[0123] After the drone stably searches for positive extrinsic rewards through the hierarchical strategy, the parallel strategy (i.e. D3QN network ) starts training using offline experience samples. The experience samples generated by the drone’s hierarchical decision making Store to experience cache In order to alleviate the instability caused by the simultaneous updating of multiple drone strategies, the concurrent experience playback mechanism is used to store and extract drone experience:

[0124] (1) Experience cache of each drone The storage of experience samples follows the "first in, first out" principle, ensuring that each experience cache is located at the same index position. Experience tuple Originated from the same state transfer process;

[0125] (2) When sampling the experience cache, all drones sample the index of the batch sample are all the same, ,Right now After each round, the experience cache Sampling in , by minimizing the loss function Optimizing action-value network parameters : .

[0126] To avoid the phenomenon of "overestimation", the D3QN method estimates the value based on the temporal difference method. When , a target value network with the same structure as the value network but delayed parameter update is used. : .

[0127] Action-value network parameters and target value network parameters The update method is as follows:

[0128] ;

[0129] ;

[0130] .

[0131] As the parallel strategy is continuously optimized, it will gradually replace the macro strategy and micro strategy, specifically through the hyperparameters Adjustment strategy replacement process, , is a positive integer, It represents the success rate of UAVs completing tasks through hierarchical collaboration strategy.

[0132] After searching for positive rewards, start training the parallel strategy; when When , the parallel strategy is used instead of the hierarchical strategy to make decisions, and the number of decision rounds is recorded as ;go through The parallel strategy completely replaced the hierarchical strategy. It is not presented in a separate parameterized form, but is based on the value function Definition. All drones are based on Method for action selection.

[0133] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0134] In a verification example, the following four scenarios are set for algorithm verification: 2UAV-1ADS-2, 3UAV-1ADS, 3UAV-2ADS, and 5UAV-2ADS, with the UAV mission team arriving at the periphery of the mission area and identifying protected targets in multiple threat areas. Figure 2 As shown, Figure 2 (a) is a schematic diagram of scenario 2 UAV-1ADS-2, Figure 2 (b) is a schematic diagram of UAV-1ADS in scenario 3. Figure 2 (c) is a schematic diagram of UAV-2ADS in scenario 3. Figure 2 (d) is a schematic diagram of Scenario 5 UAV-2ADS. The scenario name includes the number of drones and threats in the mission area. The parameter settings for the multi-drone mission scenario are shown in Table 1.

[0135] Table 1 Multi-UAV mission scenario parameter settings

[0136]

[0137] Three algorithms are tested and compared. The three algorithms are:

[0138] TH-MUPL, a multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making;

[0139] TH-MUPL-np, a multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making without parallel strategy optimization mechanism;

[0140] Multi-UAV collaborative policy learning method using only extrinsic rewards, No-Intrinsic-Reward.

[0141] The drone's macro-strategy network, from input to output, consists of a fully connected layer (1024 layers), a linear rectifier unit (LRU), a fully connected layer (512 layers), a linear rectifier unit (LRU), a fully connected layer (6 layers), and a normalized exponential function (Softmax). The drone's macro-value network, from input to output, consists of a fully connected layer (1024 layers), a linear rectifier unit (LRU), a fully connected layer (512 layers), a linear rectifier unit (LRU), and a fully connected layer (6 layers). The drone's parallel value network, from input to output, consists of a fully connected layer (1024 layers), a linear rectifier unit (LRU), a fully connected layer (512 layers), a linear rectifier unit (LRU), and a fully connected layer (7 layers).

[0142] Scenario 2: UAV-1ADS-2 cumulative extrinsic rewards and mission success rate curves are shown below: Figure 3As shown, Figure 3 (a) is the cumulative external reward curve. Figure 3 (b) is the mission success rate curve. In scenario 2, UAV-1ADS-2, the TH-MUPL algorithm's cumulative extrinsic rewards and mission completion rate increase rapidly with the number of training rounds. The TH-MUPL algorithm requires approximately 4.6×10 training cycles to achieve a mission completion rate of 60%, 80%, and 90%, respectively. 3 , 5.5×10 3 , 7.5×10 3 Compared to the No-Intrinsic-Reward method, the training cost is significantly reduced. The TH-MUPL algorithm's cumulative extrinsic reward curve initially increases rapidly, but converges to 0.4 after approximately 15,000 training rounds. This phenomenon occurs because after the TH-MUPL algorithm's hierarchical decision-making searches for positive rewards, it optimizes the macro-strategy to maximize the cumulative extrinsic reward. Once the macro-strategy achieves a certain task success rate, it transitions to a parallel strategy based on the original action space. The macro-cooperative strategy generated by TH-MUPL's early training contains some redundant decision steps, resulting in slightly lower cumulative extrinsic rewards. However, as the transition to the parallel strategy progresses, the performance of the TH-MUPL collaborative strategy further improves, and the cumulative extrinsic reward also increases.

[0143] Figure 4 The cumulative extrinsic rewards and mission success rate curves of each test algorithm in scenario 3UAV-1ADS are shown. Figure 4 (a) is the cumulative external reward curve. Figure 4 (b) is the task success rate curve. The cumulative extrinsic rewards of TH-MUPL converge to around 0.25, and the task success rate converges to around 95%. The training rounds required for TH-MUPL to reach 60%, 80%, and 90% task completion rates are approximately 8.0×10 3 , 1.0×10 4 and 1.2×10 4 Compared to the No-Intrinsic-Reward method, the training cost is significantly reduced. Furthermore, the TH-MUPL algorithm exhibited a rapid growth in the mission success rate curve for Scenario 3 (UAV-1ADS), but the cumulative extrinsic reward metric did not demonstrate an advantage. The reason is similar to that for Scenario 2 (UAV-1ADS-2): the TH-MUPL algorithm was the first to achieve a high success rate through a macro-collaboration strategy, and therefore did not achieve an advantage in the cumulative extrinsic reward metric, which comprehensively reflects the mission success rate and collaborative strategy performance.

[0144] Scenario 3: UAV-2ADS cumulative extrinsic rewards and mission success rate curve, where Figure 5 (a) is the cumulative external reward curve. Figure 5(b) is the mission success rate curve. In scenario 3, UAV-2ADS, the TH-MUPL algorithm performs well in terms of both cumulative extrinsic rewards and mission success rate. The TH-MUPL algorithm requires approximately 6.1×10 3 , 7.8×10 3 , 9.0×10 3 , compared with the No-Intrinsic-Reward method, the training cost is greatly reduced.

[0145] The curve of UAV-2ADS cumulative external rewards and mission success rate in scenario 5 is as follows: Figure 6 As shown, Figure 6 (a) is the cumulative external reward curve. Figure 6 (b) is the task success rate curve. The bar chart of the number of training rounds required for each algorithm strategy to achieve a specified success rate is shown in the figure below. Figure 7 As shown in Figure 7, 7(a) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 2UAV-1ADS-2. Figure 7 (b) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-1ADS. Figure 7 (c) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 3 UAV-2ADS. Figure 7 (d) is a bar chart showing the number of training rounds required for each algorithm strategy to achieve a specified success rate in scenario 5UAV-2ADS. In scenario 5UAV-2ADS, which is more difficult, only TH-MUPL can achieve a success rate of 5.0×10 4 The TH-MUPL algorithm learns an effective multi-UAV collaborative penetration and assault strategy within the training round. 4 After the round, the cumulative extrinsic reward reaches about -0.75, and the task success rate increases to about 80%. The number of training rounds required for the TH-MUPL algorithm to achieve a success rate of 60% and 80% is about 4.0×10 4 and 4.8×10 4 As shown in the simulation results above, after the maximum number of training rounds in each scenario, the TH-MUPL algorithm achieved a success rate of over 95% in scenarios 2UAV-1ADS-2, 3UAV-1ADS, and 3UAV-2ADS, and over 80% in scenario 5UAV-2ADS, demonstrating that the TH-MUPL algorithm can successfully train an effective multi-UAV collaborative penetration and assault strategy. The number of training rounds required for the TH-MUPL algorithm to achieve success rates of 60%, 80%, and 90% was significantly reduced, indicating that the temporal hierarchical decision structure effectively improves the optimization efficiency of multi-UAV collaborative strategies under sparse reward conditions.

[0146] In one embodiment, a multi-UAV cooperative strategy learning device based on temporal hierarchical decision-making is provided, comprising: a UAV macro-action space construction module, a hierarchical strategy training module, a hierarchical decision module, and a multi-UAV cooperative strategy determination module, wherein:

[0147] The UAV macro action space construction module is used to use the temporal hierarchical decision-making mechanism to perform temporal abstraction on the original sequential decision sequence and construct the UAV macro action space capable of achieving a positive reward state; each macro action is composed of It consists of a set of primitive actions.

[0148] The hierarchical strategy training module is used to train the pre-built hierarchical strategy based on the current observation, macro action and original action; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, in which each drone A macro action is selected at each step; a micro strategy is used to select an original action at each step based on the current observation and the most recently selected macro action.

[0149] The hierarchical decision module is used for the UAV to perform hierarchical decision search in the macro action space through a hierarchical strategy, and store the experience samples generated by the UAV's hierarchical decision into the experience cache.

[0150] The multi-UAV collaborative strategy determination module is used to train a pre-built parallel strategy offline using experience samples containing primitive actions sampled from the experience cache after the UAVs use the hierarchical strategy to search for positive extrinsic rewards. The parallel strategy is then used to gradually replace the hierarchical strategy to obtain a multi-UAV collaborative strategy. The parallel strategy is used to output primitive actions based on the current observation, and only extrinsic rewards are used during training.

[0151] In one embodiment, the drone macro-action space construction module is further used to analyze and summarize known successful collaborative behaviors, extracting original action sequences coexisting in multiple collaborative modes as macro-actions; decomposing the multi-drone collaborative action sequences of successful missions into macro-action sequences, and abstracting a reasonable macro-action space from multiple spatiotemporal coordination modes; wherein the macro-action space is:

[0152] ;

[0153] in, is the macro action space.

[0154] In one embodiment, the hierarchical strategy training module is also used to define micro-strategies using pre-training or rule-based definition, and macro-strategies are obtained by pre-training using reinforcement learning methods; current observations and macro-actions are input into the micro-strategies, and the original actions are output; the ACER method is used to update the drone's macro-strategy, and the macro-strategy includes the drone's macro-value network and the macro-strategy network.

[0155] In one embodiment, the hierarchical policy training module is also used when the macro action selection is close to the target, i.e. , the drone micro-strategy is as shown above when the macro action is to approach the target; when the macro action is to approach the threat k, that is, When , the drone micro-strategy is as shown above. When the macro action is to move closer to the threat, the drone micro-strategy expression is shown above. When the macro action is to move away from the threat area k, that is, When , the drone's micro-strategy is as shown above. When the macro action is to stay away from the threat area, the drone's micro-strategy expression is shown above. When the macro action is to "fly clockwise around the nearest threat area", that is, When the UAV micro-strategy is as shown above, when the macro action is to fly clockwise around the nearest threat area, the UAV micro-strategy expression is shown; when the macro action is to fly counterclockwise around the nearest threat area, that is, When the macro action is to fly counterclockwise around the nearest threat area, the drone's micro-strategy is as shown above; when the macro action is to "hover in place", that is, When the macro action is to hover in place, the drone's micro-strategy is as shown above.

[0156] In one embodiment, the drone macro value network is updated using the above-mentioned drone macro value network update method.

[0157] In one embodiment, the drone macro-strategy network is updated using the above-mentioned drone macro-strategy network update method.

[0158] In one embodiment, the hierarchical strategy training module is also used to store and extract drone experience samples using a concurrent experience replay mechanism; the concurrent experience replay mechanism includes: the experience cache capacity of each drone is equal, the storage of experience samples consistently follows the first-in-first-out principle, and each experience cache is ensured to be located at the same index position The experience tuples of are derived from the same state transition process; when sampling the experience cache, the indices of the batch samples sampled by all drones are the same.

[0159] In one embodiment, the hierarchical strategy training module is also used to start training the parallel strategy using offline experience samples after the drone stably searches for positive extrinsic rewards through the hierarchical strategy, and optimize the parallel strategy through D3QN; the parallel strategy includes an action value network and a target value network; the action value network and the target value network have the same structure but different parameters; the action value network is updated using the above-mentioned action value network parameter updating method; and the target value network is updated using the above-mentioned target value network parameter updating method.

[0160] In one embodiment, the hierarchical strategy training module adopts a parallel strategy to gradually replace the hierarchical strategy, specifically including: Adjustment strategy replacement process, is a positive integer, ; After searching for positive rewards, start training the parallel strategy; when When , the parallel strategy is used instead of the hierarchical strategy to make decisions, and the number of decision rounds is recorded as ;in It represents the success rate of UAVs completing tasks through hierarchical collaboration strategy; The parallel strategy completely replaces the hierarchical strategy in the transition round; the parallel strategy of the drone is not presented in an independent parameterized form, but is based on the value function Definition; All drones are based on Method for action selection.

[0161] The specific limitations of the multi-UAV cooperative strategy learning device based on temporal hierarchical decision-making can be found in the limitations of the multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making above and will not be repeated here. The various modules in the aforementioned multi-UAV cooperative strategy learning device based on temporal hierarchical decision-making can be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each of these modules.

[0162] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multi-UAV cooperative strategy learning method based on temporal hierarchical decision making, characterized by: The method comprises: The temporal hierarchical decision mechanism is used to perform temporal abstraction on the original sequential decision sequence, and a macro action space of the drone with the ability to achieve a positive reward state is constructed; each macro action is represented by primitive actions; According to the current observation, macro action and original action, the pre-built hierarchical strategy is trained; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, wherein each drone The micro-strategy is used to select the original action at each step based on the current observation and the most recently selected macro-action; The UAV performs hierarchical decision search in the macro action space through the hierarchical strategy, and stores the experience samples generated by the UAV hierarchical decision in the experience cache; After the drones use the hierarchical strategy to search for positive extrinsic rewards, they use experience samples containing primitive actions sampled from the experience cache to offline train a pre-built parallel strategy, and then gradually use the parallel strategy to replace the hierarchical strategy to obtain a multi-drone collaborative strategy; the parallel strategy is used to output primitive actions based on current observations, and only uses extrinsic rewards during training; The parallel strategy is gradually adopted to replace the hierarchical strategy, including: Through hyperparameters Adjustment strategy replacement process, , is a positive integer; After searching for positive rewards, start training the parallel strategy; when When , the parallel strategy is used instead of the hierarchical strategy to make decisions, and the number of decision rounds is recorded as ;in represents the success rate of UAVs completing tasks through hierarchical collaboration strategies; go through In a transition round, the parallel strategy completely replaces the hierarchical strategy.

2. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 1 is characterized in that: A temporal hierarchical decision-making mechanism is used to perform temporal abstraction on the original sequential decision sequence, and a macroscopic action space of the drone capable of achieving a positive reward state is constructed, including: Based on the analysis and induction of known successful collaborative behaviors, the original action sequences coexisting in multiple collaborative modes are extracted as macro-actions; The multi-UAV collaborative action sequence of a successful mission is decomposed into a macro action sequence, and a reasonable macro action space is abstracted from a variety of spatiotemporal coordination modes; the macro action space is: in, is the macro action space.

3. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 1 is characterized in that: According to the current observation, macro action and original action, the pre-built hierarchical strategy is trained; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, wherein each drone Step 1: Select a macro action; The micro-strategy is used to select the original action at each step based on the current observation and the most recently selected macro-action, including: The micro-strategy is defined by pre-training or rule-based definition, and the macro-strategy is obtained by pre-training using reinforcement learning methods; Input the current observation and macro action into the micro strategy and output the original action; The ACER method is used to update the macro-strategy of the drone, and the macro-strategy includes the drone macro-value network and the macro-strategy network.

4. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 3 is characterized in that: Input the current observation and macro action into the micro strategy and output the original action, including: When the macro action is to approach the target, the drone's micro strategy is: in, is the drone’s micro-strategy when the macro-action is to approach the target, Indicates drone i Current observations, Indicates proximity to the target. Indicates drone Select the heading angle unit vector after each primitive action, Indicates that from the drone The unit vector pointing to the protected target, Indicates the k A primitive action; represents the original action space; When macro actions choose to approach threats k When , the drone’s micro-strategy is: in, represents the drone’s micro-strategy when the macro action is to approach the threat k, Indicates drone Point to the threat area The unit vector of Indicates approaching threat k ; When macro actions choose to stay away from the threat area When , the drone’s micro-strategy is: in, Indicates that when the macro action is to move away from the threat area k Drone micro-strategy at the time, Stay away from the threat area ; When the macro action is to fly clockwise around the nearest threat area, the drone's micro strategy is: in, It represents the drone's micro-strategy when the macro-action is to fly clockwise around the nearest threat area; Indicates flying clockwise around the nearest threat area; Representation and from drone A unit vector that is orthogonal to the vector closest to the threat area and points in a clockwise direction; When the macro action is to fly counterclockwise around the nearest threat area, the drone's micro strategy is: in, It represents the drone's micro-strategy when the macro-action is to fly counterclockwise around the nearest threat area; Indicates flying counterclockwise around the nearest threat area. When the macro action is to hover in place, the drone's micro strategy is: in, represents the drone's micro-strategy when the macro-action is circling in place; Indicates the center of the circle; Indicates the execution of the original action Next moment drone The predicted position of Indicates circling in place.

5. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 3 is characterized in that: The update method of the drone macro value network is: in, represents the UAV macro value network parameters, represents the update hyperparameters of the UAV macro value network parameters, represents the gradient of the mean square error of the macro value network, Represents the value estimation of macro strategy based on Retrace method, Indicates drone i Current observations, express Moment Drone i The macro action, represents the total reward, yes The truncated importance sampling coefficient value at time , represents the value network, Indicates the total number of threat areas, represents the hyperparameter, Indicates observation The value function of Respectively Moment Drone i Observation and macro-action.

6. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 3 is characterized in that: The update method of the drone macro strategy network is: in, represents the parameters of the drone macro strategy network, represents the update hyperparameters of the drone macro-strategy network parameters, represents the ACER policy gradient, Indicates the total number of threat areas, Indicates observation The value function of 、 Respectively t Two different values of truncated importance sampling coefficients at time t, represents the drone macro value network, represents the drone macro strategy network, Indicates the i A macro action, Indicates drone i Current observations, Represents the value estimation of the macro strategy based on the Retrace method.

7. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 1 is characterized in that: Use concurrent experience playback mechanism to store and extract drone experience samples; The concurrent experience playback mechanism includes: the experience cache capacity of each drone is equal, the storage of experience samples follows the first-in-first-out principle, and each experience cache is guaranteed to be located at the same index position. The experience tuples of come from the same state transition process; When sampling the experience cache, all drones sample the same batch of samples.

8. The multi-UAV cooperative strategy learning method based on temporal hierarchical decision-making according to claim 1 is characterized in that: The parallel strategy of drones is defined based on the action-value function; all drones are based on Method for action selection: in , , drones The probability of selecting the action corresponding to the maximum action value function is All candidate actions are randomly selected with probability, represents the original action space, represents the action-value network, k Indicates the action sequence number, represents the action-value network parameters, 、 Respectively represent drones i The current observation and the original action, Indicates the k A primitive action, Indicates drone i parallel strategy.

9. A multi-UAV cooperative strategy learning device based on temporal hierarchical decision making, characterized in that: The device comprises: The UAV macro action space construction module is used to use the temporal hierarchical decision-making mechanism to perform temporal abstraction on the original sequential decision sequence and construct the UAV macro action space capable of achieving a positive reward state; each macro action is composed of primitive actions; The hierarchical strategy training module is used to train the pre-built hierarchical strategy based on the current observation, macro action and original action; the hierarchical strategy includes macro strategy and micro strategy; the macro strategy is used to select macro action according to the current observation, wherein each drone The micro-strategy is used to select the original action at each step based on the current observation and the most recently selected macro-action; A hierarchical decision module, configured for the UAV to perform hierarchical decision search in the macro action space using the hierarchical strategy, and store experience samples generated by the UAV hierarchical decision in an experience cache; The multi-UAV cooperation strategy determination module is used to, after the UAV uses the hierarchical strategy to search for positive extrinsic rewards, use the experience samples containing the original actions sampled from the experience cache to offline train the pre-built parallel strategy, and then use the parallel strategy to gradually replace the hierarchical strategy to obtain the multi-UAV cooperation strategy; the parallel strategy is used to output the original action according to the current observation, and only uses the extrinsic rewards during training; wherein, using the parallel strategy to gradually replace the hierarchical strategy includes: through the hyperparameter Adjustment strategy replacement process, , is a positive integer; After searching for positive rewards, start training the parallel strategy; when When , the parallel strategy is used instead of the hierarchical strategy to make decisions, and the number of decision rounds is recorded as ;in represents the success rate of UAVs completing tasks through hierarchical collaboration strategies; go through In a transition round, the parallel strategy completely replaces the hierarchical strategy.

Images