Unmanned aerial vehicle cluster collaborative search method based on multi-agent multi-task reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster collaborative search method based on multi-agent multi-task reinforcement learning. The method comprises the following steps: decomposing a collaborative search task of an unmanned aerial vehicle cluster into a collaborative exploration sub-task and an independent positioning sub-task; the collaborative exploration sub-task realizes collaborative coverage of a search environment and preliminary search of a target signal; the independent positioning sub-task is used for accurately positioning a target signal in a specified search area; executing different constraint conditions on the collaborative exploration sub-task and the independent positioning sub-task respectively; constructing a Markov decision process model for the unmanned aerial vehicle cluster collaborative search task according to the collaborative exploration sub-task, the independent positioning sub-task and the constraint condition; and according to the Markov decision process model, generating a shared decision network solving framework for obtaining an unmanned aerial vehicle cluster collaborative search decision based on a multi-agent multi-task reinforcement learning algorithm. According to the method, the exploration insufficiency in the reinforcement learning training process is effectively relieved, and the target discovery rate and the unmanned aerial vehicle cluster collaborative search efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-UAV multi-target cooperative search, and in particular to a multi-agent cooperative target search method based on multi-task reinforcement learning, which is suitable for multi-target search and cooperative optimization of a UAV cluster system in an unknown environment. BACKGROUND

[0002] In the cooperative search task of a UAV cluster, how to efficiently coordinate the behaviors of each UAV to achieve rapid search and accurate positioning of the target, and maintain the ability to continuously explore the unknown environment, is a highly challenging problem.

[0003] Traditional centralized control methods are limited by high computational complexity, communication delay, single-point failure and other problems, and are difficult to meet the real-time control requirements of complex and variable environments and large-scale clusters. Although the model-based distributed control method improves the scalability and robustness of the system to some extent, its performance is heavily dependent on the accuracy and generalization ability of the model, and it often performs poorly when the environment is highly uncertain and the interaction is complex. In recent years, deep learning technology has provided a new way to solve the above problems due to its powerful feature extraction and end-to-end decision-making capabilities. However, its direct application in UAV cluster control still faces many challenges such as the scarcity of training samples, the strong dynamics of the environment, and the limited onboard computing resources.

[0004] To address the above challenges, multi-agent reinforcement learning technology has emerged. Multi-agent reinforcement learning introduces competition and cooperation mechanisms among multiple agents, providing a new approach to solving the cooperative control problem of UAV clusters. Under the multi-agent reinforcement learning framework, each UAV is modeled as an agent that learns the optimal strategy by interacting with the environment to achieve coordination among individuals and completion of the overall task. However, in the cluster search task, existing multi-agent reinforcement learning algorithms mostly focus on the coordination of strategies while ignoring the structured decomposition of the search task, which may constrain the further improvement of cluster coordination efficiency in actual cluster control. SUMMARY

[0005] The present application aims to address the shortcomings of the prior art by providing a UAV cluster cooperative search method based on multi-agent multi-task reinforcement learning, which effectively alleviates the problem of insufficient exploration in the reinforcement learning training process through multi-task decomposition and shared strategy cooperative optimization, thereby improving the target discovery rate and the efficiency of UAV cluster cooperative search.

[0006] To achieve the above objectives, the technical solutions adopted by the present application are as follows:

[0007] The UAV cluster cooperative search method based on multi-agent multi-task reinforcement learning comprises the following steps:

[0008] Step 1, decomposing a cooperative search task of a UAV cluster into a cooperative exploration subtask and an independent positioning subtask;

[0009] The cooperative exploration subtask realizes cooperative coverage of the search environment and preliminary search of the target signal;

[0010] The independent positioning subtask realizes accurate positioning of the target signal in the specified search area;

[0011] Step 2, different constraint conditions are performed on the cooperative exploration subtask and the independent positioning subtask respectively;

[0012] Step 3, constructing a Markov decision process model for the cooperative search task of the UAV cluster according to the cooperative exploration subtask, the independent positioning subtask and the constraint conditions;

[0013] Step 4, according to the Markov decision process model, a shared decision network solving framework for obtaining cooperative search decisions of the UAV cluster is generated based on a multi-agent multi-task reinforcement learning algorithm.

[0014] As a preferred, the step 3 comprises:

[0015] Respectively setting the observation space of the cooperative exploration subtask and the independent positioning subtask;

[0016] Respectively setting the state space of the cooperative exploration subtask and the independent positioning subtask;

[0017] Setting the same action space of the cooperative exploration subtask and the independent positioning subtask;

[0018] Setting the state transition probability;

[0019] Respectively setting the reward function of the cooperative exploration subtask and the independent positioning subtask.

[0020] As a preferred, in the step 4, according to the Markov decision process model, a shared decision network solving framework is constructed based on a multi-agent multi-task reinforcement learning algorithm;

[0021] In the shared decision network solving framework, the decision network of each agent includes a series of feature extraction layers and a shared decision layer; the feature extraction layer is used to extract a shared feature representation across tasks; and the shared decision layer generates a final policy output according to the shared feature representation.

[0022] As a preferred, each agent further includes a value network isomorphic to the decision network, which is used to update the final policy output after evaluating the value of the current state.

[0023] As a preferred, the constraint conditions comprise:

[0024] Region constraint, the cooperative exploration subtask must be carried out in the whole search region, and the exploration behavior beyond the search region is considered invalid; the search of the independent positioning subtask must be carried out in the specified range;

[0025] Safety distance constraint, when the distance between unmanned aerial vehicles is less than the preset safety distance, it is determined that the unmanned aerial vehicles have collided;

[0026] Signal strength constraint, the unmanned aerial vehicle receives multiple target signals at the same time, but can only obtain the signal strength value of the target with the highest signal strength.

[0027] Preferably, the observation space of the cooperative exploration subtask is:

[0028]

[0029] Wherein, respectively represent the position sequence and the detected signal strength sequence of N unmanned aerial vehicles at time t, respectively represent the position sequence and the detected signal strength sequence of unmanned aerial vehicle n from time t-M+1 to time t, represents the speed of unmanned aerial vehicle n at time t;

[0030] The unmanned aerial vehicle n obtains the positions and the detected signal strengths of other unmanned aerial vehicles at time t through communication, wherein the detected signal strength of unmanned aerial vehicle m at time t is

[0031]

[0032] is the Euclidean distance between the unmanned aerial vehicle and the target at time t, D max is the maximum detection radius of the onboard sensor of the unmanned aerial vehicle, τ max is the maximum signal strength that the target can emit, and the present application does not consider the influence of the signal field of the target on the cooperative search performance of the unmanned aerial vehicle cluster;

[0033] The observation space of the independent positioning subtask is:

[0034]

[0035] Wherein, represents the center position of the search region of unmanned aerial vehicle n at time t, represents the radius of the search region of unmanned aerial vehicle n at time t.

[0036] Preferably, the state space of the cooperative exploration subtask is:

[0037]

[0038] wherein, denote the sequence of position sequences of N UAVs from t-M+1 to t and the sequence of detected signal strength sequences, respectively, denote the sequence of velocity of each UAV n at t;

[0039] The state space of the independent positioning sub-task is:

[0040]

[0041] wherein, denote the sequence of position sequences of UAV n from t-M+1 to t and the sequence of detected signal strength sequences, denote the velocity of UAV n at t, denote the center position of the search area of UAV n at t, denote the radius of the search area of UAV n at t.

[0042] As preferred, the action space of the cooperative exploration sub-task and the independent positioning sub-task is:

[0043] A t ={a1,a2,...,a n},

[0044] wherein a i ={ξ i ,ω i}, ξ i denotes the acceleration of UAV i, and ω i denotes the angular velocity of UAV i;

[0045] The state transition probability is:

[0046] Based on the joint action A t of the agent at t, the state of the environment is transferred to with a probability of .

[0047] As preferred, the reward function of the cooperative exploration sub-task is as follows:

[0048]

[0049] wherein γ1, γ2, γ3 denote the weighting coefficients of each reward, denotes the reward of guiding UAV i to approach the target at t, denotes the punishment of collision of UAV i with other agents, wherein ||*||2 denotes the Euclidean distance, d safe denotes the safety distance for avoiding collision between UAVs, This indicates a penalty for a drone moving outside the designated environment.

[0050] The reward function for the independent positioning subtask is as follows:

[0051]

[0052] in, The reward item τ represents the successful location of drone i at the target location. find This is a signal threshold obtained based on the characteristics of the UAV's onboard sensors. At time t, when the signal strength detected by the UAV exceeds this threshold, it is considered that the UAV is... The target has been successfully located.

[0053] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0054] 1. This invention significantly improves the collaborative efficiency and task adaptability of UAV swarms by rationally decomposing the collaborative search task.

[0055] 2. The multi-agent, multi-task reinforcement learning framework proposed in this invention can effectively address the problems of low exploration efficiency and training convergence difficulties caused by high-dimensional state spaces, thereby improving the stability and generalization performance of the algorithm.

[0056] 3. The present invention demonstrates a higher target detection rate and shorter task completion time in multiple experimental scenarios, and outperforms the comparative algorithms in terms of robustness and scalability. Attached Figure Description

[0057] Figure 1 This is a flowchart of the scheme in Embodiment 1 of the present invention;

[0058] Figure 2 This is a schematic diagram of a multi-target scenario according to Embodiment 1 of the present invention;

[0059] Figure 3 This is a flowchart of the algorithm in Embodiment 1 of the present invention;

[0060] Figure 4 The graph shows a comparison of the results of the algorithm of this invention with other algorithms in scenarios involving eight drones and twenty targets. Detailed Implementation

[0061] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0062] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0063] Example 1:

[0064] like Figure 1 The method for cooperative search of UAV swarms based on multi-agent, multi-task reinforcement learning, shown below, includes the following steps:

[0065] Step 1: Decompose the collaborative search task of the drone swarm into collaborative exploration sub-tasks and independent localization sub-tasks;

[0066] like Figure 2 As shown in the figure, three drones and six targets illustrate a scenario of drone swarm cooperative search for multiple targets. This invention, combining the characteristics of drone onboard sensors and the characteristics of cooperative search tasks, decomposes the cooperative search task into two sub-tasks: a cooperative exploration sub-task and an independent localization sub-task.

[0067] Collaborative exploration subtask: The UAV collaboratively explores the given search environment, explores the entire search environment based on the detected signal strength and historical sequence data, estimates the potential distribution area of ​​the target, and moves towards the area to achieve coverage and exploration of the global search area;

[0068] Independent positioning subtask: When the signal strength detected by the UAV is higher than the specified threshold, the UAV enters the independent positioning subtask. At this time, the UAV takes its own position at the time of task switching as the center and performs a local fine search within the search range determined by the characteristics of the UAV's onboard sensors and the signal threshold in order to achieve accurate positioning of the target.

[0069] Step 2, as attached Figure 2 As shown, this invention proposes multiple constraints based on the characteristics of the decomposed subtasks and the requirements of UAV swarm collaborative search tasks:

[0070] Constraint 1: Search Area Constraint. Collaborative exploration subtasks must be conducted within the entire search area; exploration activities outside the search area are considered invalid. Independent location subtasks must be searched within a specified area.

[0071] Constraint two: safety distance constraint. In order to ensure the completion of the search task of the UAV cluster, it is necessary to avoid the collision between the UAVs, so the UAVs need to maintain a certain safety distance, and when the distance between the UAVs is less than the safety distance, it is determined that the UAVs have collided.

[0072] Constraint three: the UAV can receive multiple target signals at the same time, but only the signal strength value of the target with the highest signal strength can be obtained.

[0073] Step 3: According to the task decomposed in step one and the constraints given in step two, the multi-target search problem is modeled by using the Markov decision process method.

[0074] The observation space of the cooperative exploration subtask is:

[0075]

[0076] Wherein, Respectively represent the position sequence and the detected signal strength sequence of the N UAVs at time t, Respectively represent the position sequence and the detected signal strength sequence of the UAV n from time t-M+1 to time t, Represent the speed of the UAV n at time t.

[0077] The UAV n obtains the positions of other UAVs at time t through communication and the detected signal strength, wherein the detected signal strength of the UAV m at time t is

[0078]

[0079] The Euclidean distance between the UAV and the target at time t is D max The maximum detection radius of the UAV on-board sensor is τ max The maximum signal strength that the target can emit is considered in the present application, and the influence of the signal field of the target on the cooperative search performance of the UAV cluster is not considered.

[0080] The state space of the cooperative exploration subtask is:

[0081]

[0082] Wherein, Respectively represent the sequence formed by the position sequence of the N UAVs from time t-M+1 to time t and the sequence formed by the detected signal strength sequence, Represent the sequence formed by the speed of each UAV of the N UAVs at time t.

[0083] The observation space of the independent positioning subtask is:

[0084]

[0085] where, denotes the position of UAV m at time t, denotes the position sequence and the detected signal strength sequence of UAV n from time t-M+1 to time t, respectively, denotes the velocity of UAV n at time t, denotes the center position of the search area of UAV n at time t, denotes the radius of the search area of UAV n at time t.

[0086] The state space of the independent positioning sub-task is:

[0087]

[0088] where, denotes the position sequence and the detected signal strength sequence of UAV n from time t-M+1 to time t, denotes the velocity of UAV n at time t, denotes the center position of the search area of UAV n at time t, denotes the radius of the search area of UAV n at time t.

[0089] The cooperative exploration sub-task and the independent positioning sub-task have the same action space. Set the action space as A t = {a1, a2,..., a n}, where a i = {ξ i , ω i}, where ξ i denotes the acceleration of UAV i, and ω i denotes the angular velocity of UAV i.

[0090] State transition probability: Based on the joint action A t of the agent at time t, the state of the environment is transferred from to with a probability.

[0091] Construct the reward function:

[0092] The reward function of the cooperative exploration sub-task is:

[0093]

[0094] where γ1, γ2, γ3 denote the weighting coefficients of each reward, denotes the reward for guiding UAV i to approach the target at time t, represents the penalty of collision between UAV i and other agents, where ||*||2 represents the Euclidean distance, d safe represents the safety distance of collision avoidance between UAVs, represents the out-of-bound penalty of UAV moving out of the designated environment.

[0095] Reward function of independent positioning sub-task:

[0096]

[0097] where γ1, γ2, γ3, γ4 represent the weighting coefficients of each reward, represents the reward of guiding UAV i to approach the target at time t, represents the penalty of collision between UAV i and other agents, where ||*||2 represents the Euclidean distance, d safe represents the safety distance of collision avoidance between UAVs, represents the out-of-bound penalty of UAV moving out of the designated environment, represents the reward item of UAV i successfully positioning to the target position, where τ find is the signal threshold value based on the characteristics of the onboard sensors of the UAV, and when the signal strength detected by the UAV at time t is higher than the threshold value, it is considered that the UAV has successfully positioned to a target at .

[0098] Step 4: Based on the Markov decision process model obtained in step 3, a target search decision solving framework is proposed based on the multi-agent multi-task reinforcement learning algorithm.

[0099] The complexity of the multi-target cooperative search problem of the UAV cluster leads to too large dimensions of the observation space and the state space in the Markov decision process model, and using the typical multi-agent reinforcement learning algorithm will have the problems of difficult convergence and instability, therefore, the multi-agent multi-task reinforcement learning algorithm is proposed based on task decomposition.

[0100] As shown in Figure 3 , the present application adopts the MAPPO algorithm based on multi-task reinforcement learning, the MAPPO algorithm contains N agents, and the decision network of each agent includes a task feature extraction layer and a shared decision layer. The decision network is composed of a task feature extraction layer and a shared decision layer in series.

[0101] The task feature extraction layer is responsible for extracting a cross-task shared feature representation from a high-dimensional observation input: the input of this layer is the local observation information of the current agent corresponding to the task, and the feature is transformed and abstracted through an MLP to capture the potential patterns shared between different tasks, thereby reducing redundancy and reducing the feature dimension. The shared decision layer generates the final policy output based on the shared features extracted above. The value network and the decision network of each agent use isomorphic design, i.e., have similar hierarchical structure and parameter sharing mechanism, for evaluating the value of the current state to update the policy, thereby improving the stability and convergence efficiency of multi-agent training.

[0102] The specific process of the MAPPO algorithm of multi-task reinforcement learning is as follows:

[0103] (1) initialize the environment, initialize the decision network and value network of each agent;

[0104] (2) loop 3-8, each loop training round plus one, until the maximum training round is reached;

[0105] (3) reset the environment and obtain the initial state;

[0106] (4) loop 5-6 until the round is over;

[0107] (5) input the initial observation, and each decision network of MAPPO generates an action;

[0108] (6) execute the action and update the environment;

[0109] (7) obtain the reward, and store the observation, state, action, new observation, new state and reward in the experience pool;

[0110] (8) if the experience pool is full, the MAPPO algorithm samples data from the respective experience pools for network update.

[0111] Based on the MPE framework, a multi-target cooperative search scene of an unmanned aerial vehicle cluster with 8 unmanned aerial vehicles and 20 target numbers is built. The target coordinates are fixed, the coordinate range is in the [0, 100] interval, the maximum detection radius of the on-board sensor of each unmanned aerial vehicle is 10, the coordinate range of the unmanned aerial vehicle is in the [0, 100] interval, and each round is executed in the search environment at the maximum speed v max = 5. The algorithm of the application, the MAPPO algorithm and the IPPO algorithm are selected for comparison. The parameter settings of the simulation training of the algorithm of the application are shown in Table 1, and the comparison results are shown in Figure 4 .

[0112] From Figure 4It can be seen that the algorithm proposed in the application converges at 20000 rounds, and the fluctuation of the reward after stabilization is very small, while the other comparative algorithms still have a large fluctuation after convergence, and the reward convergence value of the algorithm proposed in the application is obviously higher than that of the comparative algorithms.

[0113] Table 1: Parameter setting when the algorithm of the application is simulated and trained

[0114] Training epochs 100000 Batch process number 32 Step of round 400 Experience pool capacity 51200 Batch size 8 Discount rate 0.99 Generalized advantage estimation parameter 0.95 Truncation parameter 0.2 Decision network learning rate 1 x 10 -5 ]]> Value network learning rate 5 x 10 -6 ]]

Claims

1. A collaborative search method for UAV swarms based on multi-agent, multi-task reinforcement learning, characterized in that, Includes the following steps: Step 1: Decompose the collaborative search task of the drone swarm into collaborative exploration sub-tasks and independent localization sub-tasks; The collaborative exploration subtask achieves collaborative coverage of the search environment and preliminary search for target signals; The independent positioning subtask enables precise positioning of the target signal within a specified search area; Step 2: Apply different constraints to the collaborative exploration subtask and the independent positioning subtask respectively; Step 3: Construct a Markov decision process model for the UAV swarm cooperative search task based on the cooperative exploration sub-task, the independent positioning sub-task, and the constraints. Step 4: Based on the Markov decision process model, generate a shared decision network solution framework for obtaining collaborative search decisions of UAV swarms using a multi-agent multi-task reinforcement learning algorithm.

2. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 1, characterized in that, Step 3 includes: The observation spaces for the collaborative exploration subtask and the independent positioning subtask are established respectively; Establish separate state spaces for the collaborative exploration subtask and the independent positioning subtask; Establish the same action space for the collaborative exploration subtask and the independent positioning subtask; Establish state transition probabilities; Reward functions are established for the collaborative exploration subtask and the independent positioning subtask, respectively.

3. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 1, characterized in that, In step 4, based on the Markov decision process model, a shared decision network solution framework is constructed using a multi-agent multi-task reinforcement learning algorithm. In the shared decision network solution framework, the decision network of each agent includes a serial feature extraction layer and a shared decision layer; the feature extraction layer is used to extract shared feature representations across tasks; the shared decision layer generates the final policy output based on the shared feature representations.

4. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 3, characterized in that, Each agent also includes a value network isomorphic to the decision network, used to evaluate the value of the current state and update the final policy output.

5. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 1, characterized in that, The constraints include: Regional constraints: the collaborative exploration subtasks must be conducted within the entire search area, and exploration activities outside the search area are considered invalid; the independent positioning subtasks must be searched within a specified range. A safe distance constraint is applied, determining a collision between drones when the distance between them is less than a preset safe distance. Signal strength constraint: the drone receives signals from multiple targets at the same time, but can only obtain the signal strength value of the target with the highest signal strength.

6. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 2, characterized in that, The observation space of the collaborative exploration sub-task is: in, Let N represent the position sequence of N drones at time t and the detected signal strength sequence, respectively. Let these represent the position sequence of UAV n from time t-M+1 to time t and the detected signal strength sequence, respectively. This represents the velocity of drone n at time t; UAV n obtains the positions of other UAVs and the detected signal strength at time t through communication, where the signal strength detected by UAV m at time t is... D is the Euclidean distance between the UAV and the target at time t. max It is the maximum detection radius of the UAV's onboard sensor, τ max It is the maximum signal strength that the target can emit. This invention does not consider the impact of the target's signal field on the collaborative search performance of the UAV swarm. The observation space of the independent positioning subtask is: in, This indicates the center position of the search area for drone n at time t. Let represent the radius of the search area of ​​UAV n at time t.

7. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 2, characterized in that, The state space of the collaborative exploration subtask is: in, Let represent the sequence consisting of the position sequence of N UAVs from time t-M+1 to time t, and the sequence consisting of the detected signal intensity sequence, respectively. This represents the sequence of velocities of N drones at time t; The state space of the independent positioning subtask is: in, This represents the position sequence and detected signal strength sequence of UAV n from time t-M+1 to time t. This represents the velocity of drone n at time t. This indicates the center position of the search area for drone n at time t. Let represent the radius of the search area of ​​UAV n at time t.

8. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 2, characterized in that, The action space for the collaborative exploration subtask and the independent positioning subtask is: A t ={a1,a2,...,a n }, Where a i ={ξ i ,ω i },ξ i ω represents the acceleration of drone i. i This represents the angular velocity of drone i; The state transition probability is: Based on the joint action A of the agents at time t t The environmental state is changed from Transferred to The probability of.

9. The UAV swarm cooperative search method based on multi-agent multi-task reinforcement learning according to claim 2, characterized in that, The reward function for the collaborative exploration sub-task is as follows: Where γ1, γ2, and γ3 represent the weighting coefficients for each reward. This represents the reward for guiding drone i closer to the target at time t. Let ||*||2 represent the penalty for drone i colliding with other intelligent agents, where ||*||2 represents the Euclidean distance, and d safe This indicates the safe distance between drones to avoid collisions. This indicates a penalty for a drone moving outside the designated environment. The reward function for the independent positioning subtask is as follows: in, The reward item τ represents the successful location of drone i at the target location. find This is a signal threshold obtained based on the characteristics of the UAV's onboard sensors. At time t, when the signal strength detected by the UAV exceeds this threshold, it is considered that the UAV is... The target has been successfully located.

Images