Large-scale multi-UAV task scheduling method based on two-layer reinforcement learning

Abstract

The present invention discloses a large-scale multi-UAV task scheduling method based on double-layer reinforcement learning, comprising the following steps: establishing a multi-UAV task path planning mathematical model, and establishing a Markov decision process based on the multi-UAV task path planning mathematical model; constructing a divide-and-conquer framework including an upper model and a lower model, wherein the upper model is a deep reinforcement learning model based on a Transformer model, which is used to process multi-UAV task allocation; the lower model is a deep reinforcement learning model based on an attention mechanism, which is used to plan a feasible task path for each UAV; an interactive training method is used to train the upper model and the lower model; and large-scale multi-UAV tasks are scheduled based on the trained double-layer reinforcement learning model. The present invention is applied to the field of UAV scheduling, can be effectively applied to large-scale multi-UAV task scheduling problems, and has good generalization.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle scheduling, and in particular to a large-scale multi-unmanned aerial vehicle task scheduling method based on double-layer reinforcement learning. Background Art

[0002] Drones are widely used in logistics, search and patrol, target tracking, disaster relief and other fields due to their small size, high flexibility and strong maneuverability. As the scale and complexity of tasks increase, a single drone is difficult to meet the task requirements. Therefore, multi-drone collaborative task execution gradually replaces single drone as the main way to perform specific tasks in complex scenarios. As the basis for the collaborative task execution of multi-drone systems, task scheduling becomes the key to ensure the safe and efficient execution of multi-drone tasks.

[0003] As a variant of the Multiple Traveling Salesman Problem (M-TSP), the multi-UAV task scheduling problem is a type of combinatorial optimization problem that needs to be solved under constraints such as travel distance, time window, and platform capabilities. Researchers usually construct this problem as a mixed integer programming model and design exact algorithms and heuristic algorithms to solve it.

[0004] Exact algorithms, such as branch and bound, branch and bound, column generation, and dynamic programming, can obtain the optimal solution to small-scale problems, but due to the exponential growth of computational time, they are difficult to solve large-scale problems due to exhaustive search. Heuristic algorithms, such as genetic algorithms (GA), ant colony optimization (ACO), particle swarm optimization (PSO), simulated annealing algorithms (SA), etc., can effectively reduce the search space through carefully designed heuristic rules, so that problems can be solved in a shorter time, and can meet the needs of solving large-scale problems. For example, Ye et al. developed a genetic algorithm with a multi-type chromosome encoding scheme and adaptive operations for effective search of the solution space. Shang et al. proposed a GA-ACO hybrid algorithm combining GA and ACP, in which the worse individuals in the GA population are replaced by the better individuals in the ACO during population evolution. Chen et al. designed a transposition and extension operation, and then proposed an improved two-part wolf pack search algorithm. However, heuristic algorithms require manually designing heuristic rules based on prior knowledge, which may make it difficult to obtain high-quality solutions. In addition, these methods directly generate task scheduling solutions as a whole, that is, the order in which each drone executes its tasks, which further exacerbates the difficulty of these methods in solving large-scale problems.

[0005] At present, in order to solve the large-scale multi-UAV task scheduling problem, some studies decompose the original problem into multiple sub-problems and solve the sub-problems through traditional heuristic methods. Through this problem decomposition method, the complexity of the problem is effectively reduced, and large-scale problems can be better solved. However, the heuristic rules that need to be designed manually are still the main factor limiting these methods. With the vigorous development of deep learning (DL) and reinforcement learning (RL), deep reinforcement learning (DRL) has been widely used in games, robots, and natural language processing. In recent years, DRL has also been used to solve combinatorial optimization problems (COP), such as vehicle routing problem (VRP), traveling salesman problem (TSP), and orientation problem (OP). DRL can autonomously learn the corresponding decision-making strategy and quickly generate high-quality solutions to the problem without human intervention. However, when the scale of the problem increases, its decision space increases rapidly, making it difficult for the model to converge stably during training. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a large-scale multi-UAV task scheduling method based on double-layer reinforcement learning, which can be effectively applied to the large-scale multi-UAV task scheduling problem and has good generalization.

[0007] To achieve the above object, the present invention provides a large-scale multi-UAV task scheduling method based on double-layer reinforcement learning, comprising the following steps:

[0008] Step 1, establishing a multi-UAV mission path planning mathematical model, and establishing a Markov decision process consisting of four tuples based on the multi-UAV mission path planning mathematical model;

[0009] Step 2: Build a divide-and-conquer framework including upper-level models and lower-level models, where:

[0010] The upper model is a deep reinforcement learning model based on the Transformer model, which is used to handle multi-UAV task allocation, that is, to select the appropriate UAV for each task;

[0011] The lower layer model is a deep reinforcement learning model based on the attention mechanism, which is used to plan the feasible mission path for each drone;

[0012] Step 3, training the upper layer model and the lower layer model using an interactive training method;

[0013] Step 4: Schedule large-scale multi-UAV tasks based on the trained two-layer reinforcement learning model.

[0014] The present invention provides a large-scale multi-UAV task scheduling method based on double-layer reinforcement learning, which designs a divide-and-conquer framework to decompose the original problem into task allocation and UAV task planning sub-problems, and solves them in the upper and lower layers respectively. Based on the divide-and-conquer framework, the decision space of each sub-problem is effectively reduced compared with the original problem. Therefore, the present invention solves the large-scale UAV task scheduling problem based on double-layer reinforcement learning (Double-layer Deep Reinforcement Learning Approach, DL-DRL). Among them, DL-DRL includes two reinforcement learning models built based on the encoder-decoder framework, namely the upper model and the lower model. The upper model is used to plan the task allocation sub-problem, in which a UAV selection decoder is designed; the most advanced DRL-based attention model (AM) is used as the lower model to solve the UAV path planning sub-problem. Different from the hierarchical reinforcement learning method that needs to learn the target / option selection through trial and error, the target of the DL-DRL proposed in the present invention is predetermined by the divide-and-conquer framework, so only the action selection strategy needs to be learned, thereby reducing the difficulty of model strategy learning. In addition, the upper and lower models used in DL-DRL influence each other, that is, during the model training process, the input data distribution of the lower model is determined by the upper model, and the reward of the upper model needs to be calculated by the lower model. Therefore, the present invention adopts an interactive training strategy, and the entire training process includes three processes: pre-training, intensive training, and alternating training, so that the present invention can quickly and effectively converge to a better effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0016] Figure 1 Flow chart of a large-scale multi-UAV task scheduling method according to an embodiment of the present invention;

[0017] Figure 2 A structural diagram of a divide-and-conquer framework in an embodiment of the present invention;

[0018] Figure 3A schematic diagram of a deep reinforcement learning network structure for multi-UAV task allocation in an embodiment of the present invention;

[0019] Figure 4 This is a diagram of the model training results in the example of the present invention;

[0020] Figure 5 It is a model convergence curve diagram in the example of the present invention;

[0021] Figure 6 This is a diagram of ablation experiment results in an example of the present invention;

[0022] Figure 7 This is a schematic diagram of the generalization performance of the model trained in a scenario where the number of drones is 4 in the example of the present invention;

[0023] Figure 8 This is a schematic diagram of the generalization performance of the model trained in the scenario where the number of drones is 6 in the example of the present invention.

[0024] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0027] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0028] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0030] like Figure 1 The present embodiment discloses a large-scale multi-UAV task scheduling method based on double-layer reinforcement learning, which includes the following steps:

[0031] Step 1, establishing a multi-UAV mission path planning mathematical model, and establishing a Markov decision process consisting of four tuples based on the multi-UAV mission path planning mathematical model;

[0032] Step 2: Build a divide-and-conquer framework including upper-level models and lower-level models, where:

[0033] The upper model is a deep reinforcement learning model based on the Transformer model, which is used to handle multi-UAV task allocation, that is, to select the appropriate UAV for each task;

[0034] The lower layer model is a deep reinforcement learning model based on the attention mechanism, which is used to plan the feasible mission path for each drone;

[0035] Step 3, training the upper layer model and the lower layer model using an interactive training method;

[0036] Step 4: Schedule large-scale multi-UAV tasks based on the trained two-layer reinforcement learning model.

[0037] In the specific implementation process, the process of establishing a mathematical model for multi-UAV mission path planning is as follows:

[0038] Get the current drone mission set T = {0, 1, ..., N} and the drone set U = {1, 2, ..., V}, where 0 and N in the set T are drone bases, all drones depart from the base and return to the base, and V in the set U is the number of drones;

[0039] Let x ijk is a 0-1 variable, and its value is 1, indicating that the kth UAV departs from mission point i to point j to perform the mission;

[0040] Let y ik is a 0-1 variable, and its value is 1, indicating that the kth UAV has performed mission i;

[0041] Let z ik is a continuous variable, indicating the remaining range of the kth UAV when it reaches the mission point i;

[0042] Aiming at the problem of multi-UAV mission path planning, on the basis that the UAV cannot exceed its maximum range during the mission, with the goal of maximizing the number of tasks performed by all UAVs, a mathematical model for multi-UAV mission path planning is established, which is:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] z ik -z jk +D·x ijk ≤Dd ij ,i≠j∈{0,1…,N},k∈{1,2…,V} (7)

[0050] 0≤z ik ≤Dd i0 ,i∈{1,2,…,N},k∈{1,2,…,V} (8)

[0051] Where D is the maximum range of the UAV, formula (1) indicates that the objective function is to maximize the number of tasks performed by all UAVs; formulas (2), (3) and (4) indicate that all tasks can only be performed once at most; formula (5) indicates that all UAVs depart from the base and return to the base; formula (6) indicates that the total range of each UAV must be less than its maximum range; formulas (7) and (8) are used to avoid the generation of subpaths.

[0052] In order to reduce the complexity of the large-scale multi-UAV task scheduling problem, this embodiment divides the original problem into two sub-problems: task allocation and UAV path planning based on the divide-and-conquer principle. Figure 2As shown in the figure, a divide-and-conquer framework consisting of an upper model and a lower model is constructed, wherein the upper model is a deep reinforcement learning model based on the Transformer model, which is used to handle multi-UAV task allocation, that is, to select a suitable UAV for each task; the lower model is a deep reinforcement learning model based on the attention mechanism, which is used to plan the feasible task path for each UAV. Then, based on the initial scenario, a task subset is generated in the upper model and assigned to the corresponding UAV. After that, each UAV executes the UAV path planning process in the lower model according to its assigned task.

[0053] Based on the divide and conquer framework, this embodiment proposes a double-layer deep reinforcement learning (DL-DRL) method that effectively solves the multi-UAV task scheduling problem, in which the upper model and the lower model use two different DRL models respectively. An upper-layer DRL model for task allocation is constructed in the upper layer to select a suitable UAV for each task. In the lower layer, a lower DRL model for path planning is used to plan the feasible task path for each UAV. Through the combination of the upper and lower layers, the multi-UAV task scheduling problem can be effectively solved, and the framework can be used for other combinatorial optimization problems such as M-TSP and OP.

[0054] In the process of solving the multi-UAV task path planning problem by using reinforcement learning, it is necessary to construct a Markov decision process consisting of a four-tuple, that is, the Markov decision process M = (S, A, P, R), where S is the state space, A is the action space, P is the state transition rule, and R is the reward function. The Markov decision process is as follows:

[0055] state It is divided into two parts, among which, is the set of tasks that have been assigned at time step t, is the task subset assigned to the i-th UAV at time step t; is the two-dimensional coordinate of the task point that needs to be assigned at time step t, corresponding to the i-th element in the task set, that is, x i ;

[0056] action Indicates that the corresponding drone is assigned to the current task point. Assigned to drone u i to execute;

[0057] The state transition rule indicates that in the current state s t Next, perform action a t After that, it transfers to the state s at the next moment according to certain rules.t+1 ,Right now V t+1 Internal elements and The update is:

[0058]

[0059]

[0060] In the formula, [,] is the vector connection, for The last element in is the UAV u at time step t. j The last task point assigned to it can ensure that the task subset dimensions of all drones are equal, which is convenient for subsequent batch processing;

[0061] The reward is for the entire multi-UAV mission planning problem, and its objective function is to maximize the number of tasks executed. Therefore, in this embodiment, the reward function is defined as the total number of tasks executed by all UAVs, that is, Where T i The number of missions performed by the i-th UAV.

[0062] Compared with traditional LSTM, GRU and other models, the Transformer model can better model and process sequence data, so it is widely used in natural language processing, computer vision, recommendation algorithms, and combinatorial optimization problems. The Transformer model is based on the encoder-decoder framework and uses the attention mechanism to reduce the distance between any two positions in the sequence to a constant, thereby having better parallelism. Therefore, the upper model in this embodiment is a deep reinforcement learning network structure based on the Transformer model, which is used to process the task allocation of multiple drones, that is, Figure 3 shown.

[0063] The deep reinforcement learning model of the Transformer model includes an encoder and a decoder.

[0064] In the encoder, the input node information (i.e. the coordinates of the drone base and all mission points) is mapped to a high-dimensional space and extracted through multiple attention layers. For the input data, it is first mapped to a high-dimensional space (128 dimensions in this embodiment) through a linear layer and integrated to obtain the node embedding h 0 , and then through L attention layers to h 0 Processing is performed to obtain the feature representation of the decoder input data, where each attention layer contains a multi-head attention (MHA) layer and a feed-forward (FF) layer. The process of obtaining the feature representation of the decoder input data is as follows:

[0065] Let h l is the output of the lth attention layer. In the lth attention layer, the MHA layer is based on the multi-head attention mechanism and uses the embedded information of the input Generate query vector, key vector and value vector, the dimensions of query vector, key vector and value vector Where M = 8 is the number of attention heads in MHA; then the weight information is calculated through softmax based on the query vector, key vector and value vector to obtain the output Y of each attention head lm Finally, the outputs of each attention head are spliced ​​to obtain the fused and improved node embeddings, and the preliminary node embeddings are obtained after residual network and batch normalization (BN). The specific calculation process is:

[0066]

[0067]

[0068]

[0069]

[0070] In the formula, q lm is the query vector, k lm is the key vector, v lm is the value vector, are the independent network training parameters for the query vector, key vector, and value vector of the mth head in the lth attention layer, respectively. l O are the independent network training parameters in the lth attention layer;

[0071] The node embedding after the MHA layer is passed to the feed-forward (FF) layer, passes through two linear layers and the Relu activation function, and also passes through the residual network (skip-connection) and batch normalization (batchnormalization, BN) to obtain the final node embedding h l ,Right now:

[0072]

[0073] After L attention layers, the final node embedding h is obtained. L , which is the feature representation of the decoder input data.

[0074] In the decoder, the main task is to select the corresponding drone for the current task node and add the task node to its task subset. Specifically, the selection of drones is achieved by embedding the assigned task plan and combining it with the current node embedding. The assigned task plan refers to the current assigned task node set of all drones. For each drone, its corresponding feature is represented as in For drone u i The jth task node is embedded in the node h L The corresponding value in ; then the feature representation of the assigned task node set of all drones is processed by max-pooling to obtain the maximum feature representation Then concatenate to get the task subset context vector of all drones After a linear layer and then a FF layer, the final task subset embedding is obtained. The specific calculation process is:

[0075]

[0076]

[0077]

[0078] In order to obtain the drone number corresponding to the current task node, With the current task node in h L The corresponding embedding The selection probability of each drone under the currently assigned task node is obtained through the linear layer and softmax whose weight matrix and residual are W2 and b2 respectively, and the drone number is selected through the greedy or sampling strategy, where the selection probability is:

[0079]

[0080] In the formula, p t is the selection probability, h L is the encoder output, is the embedding of the node processed at time step t.

[0081] In this embodiment, for the lower layer model, the deep reinforcement learning model (AM) based on the attention mechanism proposed in the document W. Kool, H. van Hoof, and M. Welling, “ATTENTION, LEARN TO SOLVE ROUTING PROBLEMS!” presented at the International Conference on Learning Representations, 2019 is directly adopted. The specific implementation process is all existing technical means, so it will not be repeated in this embodiment.

[0082] In the specific training of the two deep reinforcement learning models of the upper layer and the lower layer, this embodiment adopts the REINFORCE policy gradient method with a baseline that performs well on the DRL model based on the Transformer framework. Since the upper and lower layers of deep reinforcement learning models constructed based on the divide and conquer framework in this embodiment have a strong dependence on each other, that is, the input of the lower layer model is the output of the upper layer model, and at the same time, the final result (i.e., the reward value) obtained by the task allocation scheme in the upper layer model is obtained by the lower layer model by executing the allocation scheme. Therefore, when training the upper and lower layer models, this embodiment proposes an interactive training method to train the upper and lower layer models, so that each model can learn a more targeted strategy. The gradient calculation formula is:

[0083]

[0084] in, is the loss function gradient, θ is the policy network parameter, s is the state, is the mean value in the following [] based on the training model strategy, p θ (π|s) is the strategy for training the model, L(π), L(π BL ) are the cost functions of the training model and the baseline model respectively. In this embodiment, the baseline adopts the greedy rollout policy, and compares the training model with the baseline model after each epoch training. If the trained model has significant improvement, the network parameters in the baseline model are replaced with the parameters of the current training network, thereby reducing the variance of the gradient and speeding up the training speed.

[0085] When training a hierarchical model, the lower-layer model is first pre-trained so that it can learn certain path planning strategies to avoid large fluctuations in subsequent training; then the upper-layer model is trained based on the model trained by the current lower layer. At this time, the lower-layer model is fixed and only the upper-layer model is trained; when the upper-layer model has been trained for a certain number of cycles, the current upper-layer model generates training data for the lower-layer model for training, and the upper and lower layer models are trained alternately until the training is completed.

[0086] In this embodiment, the interactive training of the upper model and the lower model has two stages. First, when the upper model just starts to be trained, intensive training is adopted to enable it to quickly learn a certain solution strategy. c After one cycle, the lower model is trained; second, after the upper and lower models have learned a certain solution strategy, single-step alternating training is performed to further improve the model solution effect. At the same time, when generating the training data for the lower model according to the upper model, since the data generation is generated according to a certain drone order, it is necessary to shuffle the samples in the generated data set and pass them to the lower layer for lower model training.

[0087] The present invention is further described below with reference to specific examples.

[0088] In order to verify that the DL-DRL method proposed in this invention can effectively learn an efficient strategy to solve the multi-UAV task scheduling problem, it is compared with an exact solver (Gurobi), a heuristic algorithm with superior performance (OR-tools), a method based on a divide-and-conquer framework (K-means+VND, K-means+AM), and a method with the same DL-DRL framework but independently trained (DL-DRL-I). The effectiveness of the proposed interactive training strategy is verified by ablation experiments. In addition, in order to explore the generalization performance of the proposed method, the model trained in the existing scenario is applied to large-scale instances, and comparative experiments are also conducted to judge the quality of its generalization performance.

[0089] For the multi-UAV task path planning problem, this example randomly generates UAV bases and customer points in a uniform distribution within a 1×1 area, and considers the cases where the number of UAVs is 4 and 6 and the number of customer points is 80, 100, 150, 200, 300 and 500 as comparative experimental scenarios. Among them, 80 and 100 customer points are small-scale scenarios, 150 and 200 customer points are medium-scale scenarios, and 300 and 500 customer points are large-scale scenarios, so as to show the solution effect of each comparison algorithm under different task point scales. At the same time, the maximum range of the UAV in each scenario is taken as 2.0, and the number of multi-head attention layers in the encoder is taken as 3.

[0090] The DL-DRL method proposed in the present invention includes two different deep reinforcement learning models: task allocation and UAV path planning. The upper model of the task allocation process adopts the model proposed in this embodiment, and is trained for 100 epochs. The training instance size of each epoch is 1,280,000, and the gradient norm is clipped to less than 3.0, and the attenuation coefficient is taken as 0.995; the lower model of the UAV path planning process adopts the deep reinforcement learning model (AM) based on the attention mechanism proposed by Kool et al. in the document W. Kool, H. van Hoof, and M. Welling, "ATTENTION, LEARN TO SOLVE ROUTING PROBLEMS!" presented at the International Conference on Learning Representations, 2019, and the gradient norm is clipped to less than 1.0, and the attenuation coefficient is taken as 1.0; the learning rate is updated using the Adam optimizer and the initial learning rate is set to 10-4. During the interactive training process, the lower model is first pre-trained for 5 epochs based on uniformly distributed randomly generated data, that is, E p =5, and then return to the upper layer for cyclic alternating training, where intensive training epochs E t Take it as 10. For the two cases where the number of drones is 4 and 6, the number of continuous training epochs of the upper model is E c They are set to 4 and 6 respectively, which can ensure that the amount of training data for the upper and lower models remains balanced and avoid excessive model oscillation caused by a large difference in the amount of training data between the two models.

[0091] According to the experimental settings, the DL-DRL model is trained for different scenarios. For model training in small and medium scale cases, a single RTX 3090 GPU with 24GB is used, while two GPUs are used for model training in large scale cases. For each training cycle, the upper model, lower model, and data generation time of the lower model are shown in Table 1, where "U4-80" means that the model is trained for the scenario with 4 drones and 80 missions. After the model training process, the results of the trained model are displayed in Figure 4 , where all problem instances are randomly generated. Figure 4 It can be seen that the training model can obtain reasonable solutions in the corresponding situations.

[0092] Table 1 Training duration of each cycle

[0093]

[0094]

[0095] In addition, Figure 5 The learning curves of the upper and lower models during training are plotted in Figure 1. The ordinate is the cost value, which is the negative of the total number of tasks performed. Therefore, a lower cost value means better performance. Figure 5 Two phenomena can be observed:

[0096] 1) For the lower-level model, it converges quickly during pre-training, while the cost value increases sharply when interactive training begins. This phenomenon is caused by the different distribution of training data for pre-training and interactive training, that is, the former is uniformly distributed, while the latter is generated by the upper-level model;

[0097] 2) For the upper model, the cost value has several rapid declines in the early training stage. This is because the lower model learns better strategies based on the training data generated by the upper model during intensive training, and the upper model quickly obtains good performance.

[0098] In addition, due to the expansion of the state and action space, the learning curve fluctuates more obviously in large-scale cases, but the DL-DRL model eventually converges stably, indicating that the model of the present invention has learned an effective strategy.

[0099] In order to prove that the DL-DRL proposed in the present invention can effectively solve the multi-UAV task path planning problem, this example compares the model obtained by training with other algorithms. First, the method proposed in the present invention is compared with the precise algorithm solver Gurobi and the heuristic algorithm OR-tools with superior performance to test the effect of the framework method proposed in the present invention in solving the multi-UAV task path planning problem; then it is compared with the task clustering + path planning method based on the divide and conquer framework, namely K-means + VND and K-means + AM, where AM is the lower-level model, to test whether the method proposed in the present invention has better performance under the same framework. Finally, the model DL-DRL-I using independent training is compared with the model using interactive training to test the improvement of the model effect by the interactive training proposed in the present invention. In the comparative experiment, 500 instances were generated for different customer point numbers for testing, and the average value of the results was taken as the comparative experimental result of each algorithm. For the Gurobi solver, due to the large scale of the problem, it is difficult to solve the optimal solution within an acceptable time. Therefore, when comparing, the solution time is limited to 1800s, 30 instances are solved, and the average value is taken as the comparative experimental result. The model training and experimental testing of the present invention are all on Intel Xeon (20 cores and 40 threads) and RTX3090 (24GB), and multi-core and GPU acceleration algorithms are used to run and train the model. The comparative experimental results of multi-UAV task path planning in different scenarios are shown in Tables 2 and 3 (in Tables 2 and 3, *Greedy means that the reinforcement learning model adopts a greedy strategy in the decoding process; Sampling means that the reinforcement learning model adopts 128 decodings and takes the optimal value as the output of the final result in the decoding process.). These tables give the average target value (Obj), gap (Gap) and average calculation time of all methods in different UAV and task number scenarios.

[0100] Table 2 Comparative experimental results when the number of drones is 4

[0101]

[0102] Table 3 Comparative experimental results when the number of drones is 6

[0103]

[0104]

[0105] It can be seen from Tables 2 and 3 that even for the scenario with only 80 tasks, Gurobi cannot obtain the optimal solution. At the same time, no matter which decoding strategy is used, the DL-DRL proposed in the present invention can always obtain a better solution than Gurobi.

[0106] For the scenario with 4 UAVs in Table 2, it is obvious that DL-DRL (Greedy) outperforms K-means + VND and K-means + AM in terms of solution effect and computation time, and this advantage becomes more significant as the task size increases. For DL-DRL and DL-DRL-I with the same network architecture, the sampling strategy always generates better solutions than the greedy strategy, and the computation time only increases slightly. DL-DRL (Greedy) outperforms DL-DRL-I (Greedy) and DL-DRL-I (Sampling), which shows that the interactive training strategy can effectively improve the model performance. Compared with the most advanced heuristic algorithm (i.e., OR-Tools), the DL-DRL (Sampling) of the present invention is slightly better in terms of solution effect when the task size is the least. However, as the task size increases, DL-DRL (Sampling) outperforms OR-Tools in terms of solution effect and computation time, which shows the powerful ability of DL-DRL in solving large-scale multi-UAV task scheduling problems.

[0107] Similar results can also be found in Table 3. DL-DRL (Greedy and Sampling) outperforms K-means+VND, K-means+AM, DL-DRL-I (Greedy), and DL-DRL-I (Sampling). At the same time, DL-DRL (Sampling) can produce competitive results with OR-Tools. When there are no more than 150 tasks, DL-DRL (Sampling) is slightly inferior to OR-Tools in terms of solution effect, but its calculation time is much shorter. When the number of tasks increases, DL-DRL (Sampling) outperforms OR-Tools in both solution effect and calculation time. In summary, DL-DRL performs better than Gurobi, K-means+VND, K-means+AM, and DL-DRL-I. At the same time, it outperforms OR-Tools in large-scale situations, and can produce competitive results with OR-Tools in a shorter calculation time when the problem scale is small.

[0108] In order to study the impact of different processes in the interactive training strategy, an ablation experiment was constructed in a scenario with 4 drones and 80 tasks. The model training adopts different training strategies and records the cost value during the training process. Figure 6 The cost value curves of the entire training process are shown, where "ITS / X" represents a training strategy that removes the "X" process from the interactive training strategy. For example, "ITS / Dense" represents a training strategy that only includes pre-training and alternating training processes.

[0109] from Figure 6 It can be seen that the pre-trained model performs better at the beginning of interactive training (for example, "ITS" is compared with "ITS / pre-training"). The intensive training process reduces the convergence speed by several cycles (for example, "ITS / pre-training" is compared with "ITS / pre-training and reinforcement"). However, for each training cycle, according to the data in Table 1, the time-consuming generation of training data for the lower-level model is much longer than the time-consuming training of the lower-level model. The intensive training process can skip the time-consuming generation process of training data for the lower-level model, thereby effectively reducing the entire training time. In terms of the final performance of the model, the interactive training strategy (ITS) proposed in the present invention is similar to the performance of ITS / intensive, while the entire training time of ITS is much shorter than that of ITS / intensive because the lower-level training data generation process is skipped. It can be seen from Table 1 that in large-scale cases, the generation of training data for the lower-level model may take several hours. Therefore, the present invention uses ITS with an intensive training process to speed up the entire training process without any performance degradation.

[0110] The generalization performance of the proposed DL-DRL method is verified by using the trained model to solve a larger scale problem and comparing it with several methods. The sampling strategy can obtain a better solution with only a slight increase in computation time, so DL-DRL (Sampling) is used to evaluate the generalization performance. First, the trained DL-DRL model is used to solve a larger task scale problem. The results are shown in Figure 2. Figure 7 and Figure 8 As shown in the figure. The horizontal axis represents the problem to be solved, and the label represents a trained model. For example, N-100 means that the task scale of the problem is 100, and U4-80 means the DL-DRL model trained for the scenario with 4 drones and a task scale of 80.

[0111] from Figure 7 It can be seen that the models trained for a specific task scale perform best on problems with the corresponding task scale compared to the models trained for other task scales. However, the other models still outperform K-means+VND, K-means+AM, and DL-DRL-I, except for U4-80 in solving problems with the number of tasks greater than or equal to 200 and U4-100 in solving problems with a task scale of 500. In addition, models trained for similar task scales perform better than models trained for farther task scales (for example, U4-200 and U4-300 perform better than U4-80, U4-100, and U4-150 when solving problems with 500 tasks). This phenomenon may be caused by the difference in data distribution, because similar task scales may lead to similar distributions of task locations. Figure 8Similar observations can be found in , where models trained on other task scales outperform K-means+VND and K-means+AM and are able to produce competitive results with DL-DRL-I.

[0112] For larger-scale multi-UAV task scheduling problems with task sizes greater than 1000, training models from scratch is a tricky task, which makes the generalization performance of the trained models for larger-scale problems increasingly important. To further study the generalization performance of the proposed DL-DRL, the DL-DRL model trained for 500 tasks is compared with the comparison algorithms OR-Tools, K-means+AM, and DL-DRL-I in scenarios with 600, 700, 800, 900, and 1000 tasks. Table 4 shows the test results of DL-DRL and the comparison algorithms, including solution effect (Obj) and average computation time. Compared with K-means+AM and DL-DRL-I, DL-DRL has a significant advantage in solution effect, but the computation time increases slightly. For OR-Tools, its computation time increases rapidly with the growth of task size, and DL-DRL outperforms it in both solution effect and computation time. In all scenarios, the proposed DL-DRL method achieves the best overall performance in all scenarios, which demonstrates its excellent generalization performance.

[0113] Table 4. Results of the model generalization experiment test

[0114]

[0115]

[0116] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A large-scale multi-UAV task scheduling method based on double-layer reinforcement learning, characterized in that: The steps include: Step 1, establishing a multi-UAV mission path planning mathematical model, and establishing a Markov decision process consisting of four tuples based on the multi-UAV mission path planning mathematical model; Step 2: Build a divide-and-conquer framework including upper-level models and lower-level models, where: The upper model is a deep reinforcement learning model based on the Transformer model, which is used to handle multi-UAV task allocation, that is, to select the appropriate UAV for each task; The lower layer model is a deep reinforcement learning model based on the attention mechanism, which is used to plan the feasible mission path for each drone; Step 3, training the upper layer model and the lower layer model using an interactive training method; Step 4: Scheduling large-scale multi-UAV tasks based on the trained two-layer reinforcement learning model; In step 1, the mathematical model for multi-UAV mission path planning is established, specifically: Get the current drone mission collection , drone collection , where the set T 0 and N The drone base, all drones depart from the base and return to the base, gather U In V is the number of drones; make x ijk is a 0-1 variable, and its value is 1, indicating k A drone from the mission point i Departure j Click to execute the task; make y ik is a 0-1 variable, and its value is 1, indicating k UAVs have performed missions i ; make z ik is a continuous variable, indicating the k The drone arrived at the mission point i The remaining distance at the time On the basis that the UAV cannot exceed its maximum range during the mission, and with the goal of maximizing the number of missions performed by all UAVs, a mathematical model for multi-UAV mission path planning is established: (1) .(2) (3) (4) (5) (6) (7) (8) in, D is the maximum range of the UAV, formula (1) indicates that the objective function is to maximize the number of tasks performed by all UAVs; formulas (2), (3) and (4) indicate that all tasks can be performed at most once; formula (5) indicates that all UAVs depart from the base and return to the base; formula (6) indicates that the total range of each UAV must be less than its maximum range; formulas (7) and (8) are used to avoid the generation of subpaths.

2. The large-scale multi-UAV task scheduling method based on double-layer reinforcement learning according to claim 1 is characterized in that: In step 1, the Markov decision process consisting of four tuples is established based on the multi-UAV mission path planning mathematical model, specifically: The Markov decision process consisting of four tuples is established as M = ( S , A , P , R ), in, S is the state space, A For action space, P is the state transition rule, R is the reward function, and the Markov decision process is as follows: state It is divided into two parts, among which, is the time step t The set of tasks that have been assigned, For the i The number of drones in time step t The subset of tasks assigned at the time; is the time step t The two-dimensional coordinates of the task points that need to be assigned at the time correspond to the first i elements, namely x i ; action , indicating that the corresponding drone is assigned to the current task point. Assign to drone u i to execute; The state transition rule indicates that in the current state s t Next action a t After that, it transfers to the state of the next moment according to certain rules. s t+1 ,Right now , Internal elements and The update is: (9) (10) In the formula, For vector connection, for The last element in is the time step t Time Drone u j The last task point assigned to it can ensure that the task subset dimensions of all drones are equal, which is convenient for subsequent batch processing; The reward function defines the total number of tasks performed by all drones, i.e. ,in T i For the i The number of missions performed by the drones.

3. The large-scale multi-UAV task scheduling method based on double-layer reinforcement learning according to claim 2 is characterized in that: In step 2, the deep reinforcement learning model of the Transformer model includes an encoder and a decoder.

4. The large-scale multi-UAV task scheduling method based on double-layer reinforcement learning according to claim 3 is characterized in that: In the encoder: The coordinates of the drone base and all mission points are used as the input node information, mapped to a high-dimensional space through a linear layer and integrated to obtain the node embedding ; then through L Attention layer pair Processing is performed to obtain the feature representation of the decoder input data, where each attention layer contains an MHA layer and an FF layer.

5. The large-scale multi-UAV task scheduling method based on double-layer reinforcement learning according to claim 4 is characterized in that: The feature representation of the decoder input data is obtained as follows: make For the l The output of the attention layer is l In the attention layer, the MHA layer is based on the multi-head attention mechanism, according to the input embedding information produce query vector, key Vector and value vector; then according to the obtained query vector, key Vector and value Vector by Calculate the weight information to get the output of each attention ; Finally, the outputs of each attention head are spliced ​​to obtain the fused and improved node embeddings, and the preliminary node embeddings are obtained after the residual network and batch normalization BN. , the specific calculation process is: (11) (12) (13) (14) In the formula, for query vector, for key vector, for value vector, , , Respectively l In the attention layer, m Head query vector, key vector, value vector of independent network training parameters, For the l Independent network training parameters in each attention layer; The node embedding after the MHA layer is passed to the FF layer, passes through two linear layers and the Relu activation function, and also passes through the residual network and batch normalization to obtain the final node embedding ,Right now: (15) go through L After processing by the attention layer, the final node embedding is obtained , which is the feature representation of the decoder input data.

6. The large-scale multi-UAV task scheduling method based on double-layer reinforcement learning according to any one of claims 3 to 5, characterized in that: In the decoder: The selection of drones is achieved by embedding the assigned task plan and combining it with the current node embedding. The assigned task plan refers to the set of currently assigned task nodes of all drones. For each drone, its corresponding feature is expressed as ,in For drones u i No. j Task nodes are embedded in the node The corresponding value in ; then all the UAVs’ assigned task node set feature representations are processed by maximizing the pool to obtain the maximum feature representation Then concatenate to get the task subset context vector of all drones , and then processed by a linear layer and then by the FF layer to obtain the final task subset embedding , the specific calculation process is: (16) (17) (18) In order to obtain the drone number corresponding to the current task node, With the current task node The corresponding embedding Connect them and pass the weight matrix and residual respectively and The linear layer and The selection probability of each drone under the currently assigned task node is obtained, and the drone number is selected through the greedy or sampling strategy, where the selection probability is: (19) In the formula, p t is the selection probability, is the output of the encoder, is the time step t The embedding of the node being processed.

7. The large-scale multi-UAV task scheduling method based on double-layer reinforcement learning according to any one of claims 1 to 5, characterized in that: In step 3, the upper layer model and the lower layer model are trained by an interactive training method, specifically: First, the lower-level model is pre-trained so that it can learn certain path planning strategies to avoid large fluctuations in subsequent training; then the upper-level model is trained based on the pre-trained lower-level model. At this time, the lower-level model is fixed and only the upper-level model is trained; when the upper-level model has been trained for a certain number of cycles, the current upper-level model generates training data for the lower-level model for training, and the upper-level model and the lower-level model are trained alternately until the training is completed.

Images