A Multi-UAV Task Allocation and Fusion Method Based on Improved Reinforcement Learning

Abstract

This invention discloses a multi-UAV task allocation and fusion method based on improved reinforcement learning, comprising: determining an initial parameter set for multi-UAV task allocation; fragmenting and decomposing multi-task objectives to determine cluster centers; using an adaptive clustering strategy based on center points to screen candidate clustering strategies and determine the optimal cluster set; determining a dynamic decay gradient strategy for the improved Q-learning network based on the action value function of basic reinforcement learning; determining a spiral cross-assignment mechanism for task redistribution based on a task redistribution mechanism with weighted coefficients; and determining a multi-UAV task allocation strategy based on the optimal cluster set of task points, the dynamic decay gradient strategy of the improved Q-learning network, and the task redistribution mechanism to complete the multi-UAV task allocation. This invention can effectively avoid unnecessary redundant allocation, improve the time consumption and range of task allocation, and increase the efficiency of task allocation.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) mission planning technology, specifically involving a multi-UAV mission allocation and fusion method based on improved reinforcement learning. Background Technology

[0002] Multi-UAV task allocation is a type of optimization problem under multiple constraints. As the complexity of task scenarios continues to increase, the multi-UAV task allocation problem becomes more challenging, requiring effective allocation and decision-making methods to achieve efficient and intelligent collaborative task execution.

[0003] Currently, research on multi-UAV task allocation mainly falls into two categories: centralized and distributed task allocation. However, existing multi-UAV task planning methods still have some limitations. For example, in large-scale tasks and dynamic environments, the running time of centralized algorithms such as integer programming, multiple traveling salesman algorithms, and dynamic network flow optimization algorithms increases dramatically, limiting real-time performance. Heuristic methods such as genetic algorithms and particle swarm optimization may neglect the optimal allocation of key resources such as energy consumption. In contrast, reinforcement learning algorithms can autonomously adjust task allocation strategies in complex task environments, improving task completion efficiency and reducing operational costs. They exhibit strong adaptability and robustness, making them a popular research topic with broad applicability and scalability.

[0004] Reinforcement learning algorithms, as a method capable of autonomously learning and optimizing strategies, are based on the idea that through the interaction between an agent and its environment, the agent learns to take a series of actions to maximize cumulative rewards. However, in multi-machine task allocation, reinforcement learning faces the challenge that the complexity of cooperative task planning leads to a dramatic increase in the state and action space, resulting in redundant allocation, which traditional algorithms struggle to handle. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a multi-UAV task allocation and fusion method based on improved reinforcement learning, which can effectively avoid unnecessary redundant allocation, improve the time consumption and range of task allocation, and improve the efficiency of task allocation.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0007] A multi-UAV task assignment and fusion method based on improved reinforcement learning includes:

[0008] Step 1: Determine the initial parameter set for multi-UAV task allocation, decompose the multi-task objectives into fragments, and determine the cluster center points of the tasks;

[0009] Step 2: Based on the cluster centers of the task, use the adaptive clustering strategy of centers to screen clustering candidate strategies and determine the optimal cluster set;

[0010] Step 3: Determine the dynamic decay gradient strategy of the improved Q-learning network based on the action value function of basic reinforcement learning;

[0011] Step 4: Determine the spiral cross-assignment mechanism for task redistribution based on the spiral search mechanism with weighted coefficients;

[0012] Step 5: Determine the multi-UAV task allocation strategy based on the optimal cluster set of task points, the dynamic decay gradient strategy of the improved Q-learning network, and the spiral cross-allocation mechanism for task redistribution. Complete the multi-UAV task allocation based on the multi-UAV task allocation strategy.

[0013] To optimize the above technical solution, the specific measures also include:

[0014] After detecting the drone and the task point in step 1 above, the initial set of task allocation parameters is determined based on the relative geometric relationship between the current state of the drone and the task point; the initial set of task allocation parameters is as follows:

[0015]

[0016] Φ n ={T k |T k ∈T∩Φ nk =1}

[0017] Among them, T k To represent the assigned task, the variable Φ nk U-shaped drone n Assigned task T k T represents the total task set, Φ n Indicates multiple drones U n Task allocation set;

[0018] If both of the above formulas are satisfied, then the drone task allocation parameter set has been initialized and multi-drone task allocation can be performed; otherwise, multi-drone task allocation will not be performed.

[0019] The objective function for task clustering in step 1 above is... for:

[0020]

[0021] In the formula, i represents the current cluster center, j represents the number of cluster center points, and E is the sum of the deviations between all target points and their respective cluster centers;

[0022] X is a given target data point in a known region Ω; C i It is a cluster of clustering algorithms; Vi It is cluster C i The average value;

[0023] ‖XV i || P Representing X and V i The p-order metric between them is the square of the Euclidean distance, i.e., p = 2.

[0024] The centroid adaptive clustering strategy described in step 2 above is used to measure whether the clustering result is close to the optimal one. Only when the loss of a certain clustering is less than a predetermined proportion of the previous optimal clustering loss is the clustering result considered a candidate solution. It introduces a k-centroid randomization strategy, using different initial cluster centroids for each clustering iteration, and selecting the cluster with the smallest sum of squared distances as the optimal clustering result for the current iteration, expressed as:

[0025]

[0026] The action value function based on basic reinforcement learning described in step 3 above is:

[0027]

[0028] Where ∈ is the discount factor, r is the reward function, s is the environment state function, a is the action function, and Q is the value of Q. π (s,a) is the action value function, which is used to measure the quality of a specific action taken by an agent.

[0029] The improved Q-learning network's dynamic decay gradient strategy described in step 3 above is as follows: based on the optimal cluster set in step 2, the task closest to the current UAV is found for initial task allocation. Then, the exploration rate is gradually reduced according to the dynamic decay gradient strategy to allocate tasks, resulting in a task allocation set. Finally, the task is spirally redistributed by combining the task redistribution mechanism to obtain the optimal task allocation set.

[0030] The exploration rate mentioned above is specifically expressed as follows:

[0031]

[0032] Where ε0 is the initial exploration rate, ε end For the final exploration rate, ε1 = ε0 - ε end α, β, δ are the periodic iteration coefficients, and T is the iteration period.

[0033] The allocation mechanism of the spiral cross-assignment mechanism for task redistribution described in step 4 above is as follows:

[0034]

[0035] in For the current U drone n The position, A is the weighting coefficient, ω n Weighting factors for individual n:

[0036] ω n =0.75+p n

[0037] P n For the exploration capabilities of a single drone:

[0038]

[0039]

[0040] Where fitness is the current fitness value of the drone, rank(fitness) is the individual fitness ranking, and ε t This represents the current exploration rate in reinforcement learning. When μ n When μ > 1, enhance the drone mission exploration capability, expand the allocation range, and ensure that all missions are assigned; when μ n When μ < 1, the task exploration capability is reduced, and the convergence of the algorithm is improved. n When = 1, the drone mission exploration capability remains unchanged.

[0041] The present invention has the following beneficial effects:

[0042] This invention incorporates a clustering algorithm into multi-UAV task allocation, grouping similar states or actions and adding a spiral search strategy to increase the UAV search range. This maintains the flexibility of task allocation while reducing computational complexity, thereby improving the overall system performance and enabling the agent to learn and make decisions more efficiently. It addresses the issue that in multi-UAV task allocation, as the number of task points increases, the required state and action space increases dramatically, and the cumulative reward for reinforcement learning also increases dramatically, easily leading to incomplete or duplicate task allocation, causing task allocation to get stuck in a local optimum loop.

[0043] This invention aims at the desired state of multi-UAV task allocation, which reduces the complexity of the state space and action space, and enables task allocation to reach the desired state more quickly, thereby improving the allocation efficiency of multi-UAV task allocation. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the improved reinforcement learning multi-UAV task allocation and fusion method of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating the interaction between the agent and the environment in the basic reinforcement learning algorithm of this invention.

[0046] Figure 3This is a schematic diagram of the improved centroid adaptive clustering strategy of the present invention;

[0047] Figure 4 This is a gradient decay curve diagram of the dynamic decay gradient strategy of this invention;

[0048] Figure 5 This is a schematic diagram of the task allocation process for the dynamic decay gradient strategy of this invention;

[0049] Figure 6 This is a schematic diagram of task clustering in this invention;

[0050] Figure 7 Comparison of effects before and after the reallocation of multiple drone tasks;

[0051] Figure 8 Assign performance evaluations to clustering algorithm tasks;

[0052] Figure 9 Assigning performance to basic reinforcement learning algorithm tasks;

[0053] Figure 10 To improve the effectiveness of reinforcement learning-based multi-UAV task allocation fusion methods. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0056] Example 1: A multi-UAV task allocation and fusion method based on improved reinforcement learning, such as... Figure 1 As shown, it includes:

[0057] Step 1: Determine the initial parameter set for multi-UAV task allocation, and decompose the multi-task objectives into fragments based on task point density and task execution sequence to determine the cluster center points of the tasks;

[0058] 1) Determine the initial parameter set for multi-UAV task allocation. The specific implementation process is as follows:

[0059] After detecting the drone and the task point, an initial set of task allocation parameters is determined based on the relative geometric relationship between the drone's current state and the task point. This initial set of task allocation parameters is expressed as follows:

[0060]

[0061] Φ n ={T k |T k ∈T∩Φ nk =1}

[0062] If both of the above formulas are satisfied, then the UAV task allocation parameter set is determined to be initialized, and multi-UAV task allocation can be performed; otherwise, multi-UAV task allocation will not be performed. Where T... k To represent the assigned task, the variable Φ nk U n Assigned task T k Φ n Indicates multiple drones U n The task allocation set.

[0063] Furthermore, the sum of the Euclidean distances between the drone and the mission point is expressed as:

[0064]

[0065] in, N represents the flight distance from the i-th drone performing the k-th task to the (k+1)-th task. t Indicates the number of task points.

[0066] 2) Determine the cluster center of the task.

[0067] Select j center locations V within the known region Ω. i (i = 1, 2, ..., j), such that N t If the sum of the squared distances from each point to its nearest center is minimized, then the objective function of the clustering algorithm is defined as follows:

[0068]

[0069] In the formula, i represents the current cluster center, and j represents the number of cluster center points. It is the sum of the deviations between all target points and the center of their respective clusters, where X is a given target data point in the known region Ω; C i It is a cluster of clustering algorithms, V i It is cluster C i The average value; ||XV i || P Representing X and V iThis is a p-order metric between distances, typically using the square of the Euclidean distance (p=2) in distance spaces. The goal is to minimize this objective function to make the generated result set as compact as possible.

[0070] Step 2: Based on the cluster centers of the task, use the adaptive clustering strategy of centers to screen clustering candidate strategies and determine the optimal cluster set;

[0071] The core idea of ​​the centroid adaptive clustering strategy is to repeatedly try different numbers of clusters using traditional clustering strategies without prior knowledge, and then select the optimal solution based on the quality of the clustering results.

[0072] like Figure 3 As shown, this invention introduces a k-centroid randomization strategy, using different initial cluster centers each time to improve the flexibility of the clustering algorithm. The cluster with the smallest sum of squared distances among the clustering results is selected as the optimal clustering result for the current iteration, expressed as:

[0073]

[0074] To further enhance the algorithm's adaptability, this invention introduces an adaptive clustering candidate strategy to measure whether the clustering result is close to the optimal one. A clustering result is only considered a candidate solution if the loss of a given clustering iteration is less than a certain proportion of the loss of the previous optimal clustering iteration. This mechanism ensures the algorithm's robustness to different data distributions and prevents over-refinement of clustering.

[0075] Step 3: Determine the dynamic decay gradient strategy of the improved Q-learning network based on the action value function of basic reinforcement learning;

[0076] 1) Reinforcement learning:

[0077] At time t, the agent determines the current environmental state s. t Choose action a t After the action is performed, the environment obtains the next state s. t+1 And feed back a reward signal r to the agent. t+1 At time t+1, the agent determines the state s based on the state. t+1 Select action a t+1 , obtain the environmental state s t+2 and reward signal r t+2 ,like Figure 2 As shown. Then, at the end of the cycle, the AI ​​will have collected a learning sample for one cycle T, represented as:

[0078] {s0,a0,r1,s1,a1,r2,…,s T-1 ,a T-1 ,r T ,sT}

[0079] Define the "reward" R at time t. t The sum of the "rewards" from the start of this period to the end of the cycle is expressed as:

[0080] R t =r t +εr t+1 +ε 2 r t+2 +…+ε T-t-1 r T-1

[0081] Where ε represents the discount factor, it can be seen that the formula weights the reward, and the weight decays at a rate of ε with each time step. The farther the time step is, the smaller the weight becomes.

[0082] Use strategy π from (s) t ,a t The interaction with the environment begins and continues until the end of the cycle, resulting in a complete interaction sequence, which is then used to calculate (s). t ,a t The corresponding R t The value function Q of (s,a) under policy π. π (s,a) measures the average reward obtained at the end of the cycle for the action chosen according to policy π starting from this state, expressed as:

[0083]

[0084] Q π (s,a) is the action value function, used to measure the quality of a specific action taken by the agent. The quality of the agent's state is measured by the state value function, which is equivalent to the quality of Q. π Taking the expectation of (s,a) in terms of action, it can be expressed as:

[0085]

[0086] 2) Determine the dynamic decay gradient principle of the improved Q-learning network.

[0087] Dynamic gradient decay is an adaptive learning method based on gradient descent, aiming to dynamically adjust the learning rate according to the model's performance during the learning process. Specifically, in Q-Learning networks, this invention adaptively reduces the exploration rate based on training progress and the learned Q-value, allowing for greater reliance on the existing Q-value in later training stages, thus enhancing the algorithm's learning ability. Figure 4 As shown, the dynamic gradient decay strategy is specifically expressed as follows:

[0088]

[0089] Where ε0 is the initial exploration rate, ε end For the final exploration rate, ε1 = ε0 - ε end Let α, β, and δ be the periodic iteration coefficients, and T be the iteration period. Then, the exploration capability of a single UAV can be expressed as:

[0090]

[0091] 3) Determine the dynamic gradient decay strategy for the improved Q-learning network.

[0092] The principle of task allocation using a dynamic gradient decay strategy is as follows: Figure 5 As shown in the diagram, based on the initial task allocation set, the task closest to the current UAV is found for initial task allocation. Subsequently, the exploration rate is gradually reduced according to the dynamic decay gradient to allocate tasks, resulting in a task allocation set. Finally, a task reassignment mechanism is used to perform spiral cross-reassignment of tasks to obtain the optimal task allocation set.

[0093] Step 4: Determine the spiral cross-assignment mechanism for task redistribution based on the spiral search mechanism with weighted coefficients;

[0094] The task redistribution mechanism, by introducing spiral crossover motion, enhances the global exploration capability of reinforcement learning algorithms. This helps the algorithms escape local optima, find the global optimum earlier, and improve convergence. A weight factor ω is added to control the range of crossover assignment when introducing the task redistribution mechanism. The task redistribution mechanism is expressed by the following formula:

[0095]

[0096] Where A is the weighting coefficient, and ω is the weighting factor, as shown below:

[0097] ω n =0.75+p n

[0098] Where the weight factor μ of individual n n When ω n When ω > 1, enhance the drone mission exploration capability, expand the allocation range, and ensure that all missions are assigned; when ω n When ω < 1, the task exploration capability is reduced, and the convergence of the algorithm is improved. n When = 1, the drone mission exploration capability remains unchanged.

[0099] Step 5: Determine the multi-UAV task allocation strategy based on the optimal cluster set of task points, the dynamic decay gradient strategy of the improved Q-learning network, and the task reallocation mechanism, and complete the multi-UAV task allocation based on the multi-UAV task allocation strategy.

[0100] Using a point mass as an example, the UAV and mission point models are used, and the clustering algorithm parameters are set as follows: cluster K = [1 13], maximum number of iterations N. KM =100, initial center point running times N K =20, Reinforcement learning parameters: learning rate α = 0.1, discount factor g = 0.9, initial exploration rate ε0 = 0.9, final exploration rate ε end =0.05, exploration rate decay step number E p =500, effective obstacle avoidance distance R=1, weighting factor ω n = [0.75 1.25].

[0101] Figure 6 The clustering diagram obtained in the above simulation environment is obtained using the centroid adaptive clustering strategy. It is clear that the improved method mainly optimizes the following: Traditional clustering algorithms require the number of clusters to be determined in advance. When calculating the average value within a cluster, they are easily affected by outliers, which increases the time complexity of searching for cluster centroids. In contrast, the centroid adaptive clustering strategy introduces a k-centroid random strategy without prior knowledge. It uses different initial cluster centroids each time and repeatedly tries different numbers of clusters using the traditional clustering strategy. Then, it selects the optimal solution based on the quality of the clustering results, which improves the flexibility of the algorithm in clustering.

[0102] Figure 7 (a) and (b) are comparison diagrams showing the effects of multi-UAV task reallocation before and after in the same simulation environment according to the embodiments of the present invention. Figure 7 (a) shows that before task allocation, the improved reinforcement learning algorithm is still affected by isolated points after task allocation, and the UAV's maneuverability is relatively large; from Figure 7 (b) It can be seen that after task redistribution, the number of isolated points in the task environment is significantly reduced, and the task concentration of UAVs is significantly improved.

[0103] Figure 8-10 This section compares the performance of the improved reinforcement learning-based multi-UAV task allocation and fusion method of this invention with other conventional methods under the same simulation environment, as obtained in an embodiment of this invention. Figure 8 It is evident that using only clustering algorithms for multi-UAV task allocation, selecting all UAVs to participate in task execution based on clusters, results in the shortest total UAV flight distance. However, the algorithm's iteration time is too long, and the task planning speed is too slow, failing to meet the timeliness requirements of large-scale task planning. Figure 9It can be seen that when the basic reinforcement learning algorithm is used for multi-UAV task allocation, the timeliness of task allocation and the total flight range are good in the early stage. However, as the number of tasks increases and the exploration range of task allocation continues to expand, the flight range continues to increase, and it is easy to get trapped in local optima. From Figure 10 It can be seen that the improved reinforcement learning multi-UAV task allocation fusion method combines the short range of clustering algorithms with the timeliness of reinforcement learning algorithms, effectively improving the algorithm's adaptability to multi-task environments. It introduces a task reallocation mechanism to achieve a balance between global exploration and local optimization, ensuring the dynamic unity of task allocation and execution.

[0104] Therefore, the improved reinforcement learning-based multi-UAV task allocation fusion method has better advantages in allocation efficiency and convergence speed, and can effectively solve the problem of multi-UAV task allocation.

[0105] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0106] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-UAV task allocation and fusion method based on improved reinforcement learning, characterized in that, include: Step 1: Determine the initial parameter set for multi-UAV task allocation, decompose the multi-task objectives into fragments, and determine the cluster center points of the tasks; Step 2: Based on the cluster centers of the task, use the adaptive clustering strategy of centers to screen clustering candidate strategies and determine the optimal cluster set; Step 3: Determine the dynamic decay gradient strategy of the improved Q-learning network based on the action value function of basic reinforcement learning; the dynamic decay gradient strategy of the improved Q-learning network is specifically as follows: based on the optimal cluster set in Step 2, find the task closest to the current UAV for initial task allocation, then gradually reduce the exploration rate according to the dynamic decay gradient strategy to allocate tasks, obtain a task allocation set, and then combine the task reallocation mechanism to perform spiral cross-allocation of tasks to obtain the optimal task allocation set; the exploration rate is specifically expressed as: in This is the initial exploration rate. For the final exploration rate, , For periodic iteration coefficients, The iteration period; Step 4: Determine the spiral cross-assignment mechanism for task redistribution based on the spiral search mechanism with weighted coefficients; Step 5: Determine the multi-UAV task allocation strategy based on the optimal cluster set of task points, the dynamic decay gradient strategy of the improved Q-learning network, and the spiral cross-allocation mechanism for task redistribution. Complete the multi-UAV task allocation based on the multi-UAV task allocation strategy.

2. The multi-UAV task allocation and fusion method based on improved reinforcement learning according to claim 1, characterized in that, After detecting the drone and the task point in step 1, the initial set of task allocation parameters is determined based on the relative geometric relationship between the current state of the drone and the task point; the initial set of task allocation parameters is as follows: in, Represents the assigned task, variable Indicates drone Assigned tasks Represents the total set of tasks. Indicates multiple drones Task allocation set; If both of the above formulas are satisfied, then the drone task allocation parameter set has been initialized and multi-drone task allocation can be performed; otherwise, multi-drone task allocation will not be performed.

3. The multi-UAV task allocation and fusion method based on improved reinforcement learning according to claim 1, characterized in that, The objective function of task clustering in step 1 is... for: In the formula, Indicates the current cluster center. It is the number of cluster centers. It is the sum of the deviations between all target points and the center of their respective clusters; It is a known region The target data points are given in the text; It is a cluster of clustering algorithms; It is a cluster The average value; express and Between The order metric is the square of the Euclidean distance. To measure.

4. The multi-UAV task allocation and fusion method based on improved reinforcement learning according to claim 3, characterized in that, The centroid adaptive clustering strategy described in step 2 is used to measure whether the clustering result is close to the optimal one. Only when the loss of a clustering iteration is less than a predetermined proportion of the previous optimal clustering loss is the clustering result considered a candidate solution. It introduces a k-centroid randomization strategy, using different initial cluster centroids for each iteration, and selecting the cluster with the smallest sum of squared distances as the optimal clustering result for the current iteration, expressed as: 。 5. The multi-UAV task allocation and fusion method based on improved reinforcement learning according to claim 1, characterized in that, The action value function based on basic reinforcement learning described in step 3 is: in, As a discount factor, For the reward function, For the environment state function, For action functions, It is an action value function, used to measure the quality of a specific action taken by an agent.

6. The multi-UAV task allocation and fusion method based on improved reinforcement learning according to claim 1, characterized in that, The allocation mechanism of the spiral cross-assignment mechanism for task reallocation described in step 4 is as follows: in For current drones Location, These are the weighting coefficients. For individuals Weighting factors: For the exploration capabilities of a single drone: in This represents the current fitness value of the drone. Rank individuals based on fitness. This represents the current exploration rate of reinforcement learning; when At the same time, enhance the drone mission exploration capabilities, expand the allocation range, and ensure that all missions are assigned; when At that time, reduce the task exploration capability to improve the convergence of the algorithm. At the same time, the drone mission exploration capability remains unchanged.

Images