UAV swarm task chain scheduling method and product based on deep reinforcement learning

Abstract

This invention discloses a method and product for scheduling unmanned aerial vehicle (UAV) swarm task chains based on deep reinforcement learning. This method relates to the field of swarm task chain scheduling. The method includes integrating a joint policy-value network into a Monte Carlo search tree algorithm to generate a training sample set. The training sample set is then used to train the joint policy-value network. Ultimately, the trained joint policy-value network can be used to determine the optimal solution for task chain scheduling, or the trained joint policy-value network can be combined with a Monte Carlo search tree algorithm to obtain the optimal solution. In this invention, the combined use of the joint policy-value network and the Monte Carlo search tree algorithm to determine the optimal task chain improves the efficiency and reliability of swarm task execution.

Description

Technical Field

[0001] The present invention relates to the field of cluster task chain scheduling, and in particular to a method and product for scheduling unmanned aerial vehicle (UAV) cluster task chains based on deep reinforcement learning. Background Art

[0002] In recent years, drones have attracted increasing attention from both academia and industry. Due to their high maneuverability, drones can be used to provide a wide range of applications such as service delivery, pollution mitigation, agriculture, and rescue operations.

[0003] Swarming is a future trend in drone development. Through swarming, drones can complement each other's capabilities, reducing costs while improving efficiency. Several existing technologies have summarized the applications of drone swarm networks in dense urban environments and introduced use case scenarios. Several existing technologies have studied the application of drone swarms in multi-target tracking, noting the significant potential of drone multi-target tracking systems for performing tedious, dangerous, and critical tasks in frontier defense and security. Several existing technologies have proposed a novel air-to-ground air quality sensing framework based on federated learning. This framework uses a trained global model to monitor air quality indices, expanding the scope of drone swarm monitoring. Several existing technologies have summarized the technological hotspots and development trends in the drone field from multiple perspectives, including new development paths for drone systems, practical applications, autonomous swarm control, and innovation-driven industries.

[0004] As mission conditions become increasingly complex and swarms scale, the complexity and uncertainty of drone swarms are rapidly increasing, severely impacting the efficiency and reliability of swarm mission execution. The key to drone swarm technology is how to assign tasks to individual drones through scheduling technology to maximize overall mission benefits. Summary of the Invention

[0005] The purpose of this invention is to provide a UAV cluster task chain scheduling method and product based on deep reinforcement learning, which combines the Monte Carlo search tree algorithm and the joint strategy-value network to determine the optimal task chain for UAV cluster task chain scheduling, thereby improving the efficiency and reliability of cluster task execution.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for scheduling a task chain of a drone swarm based on deep reinforcement learning, the method comprising:

[0008] According to the task loads and task objectives of different functions in the UAV swarm mission, the OODA loop theory is applied to construct a task laminar flow network; the task laminar flow network includes several task chains; a task chain is used to complete a task objective; each task chain includes a perception node, a decision node, an execution node and a target node; there is an edge relationship between the perception node and the decision node, there is an edge relationship between the decision node and the execution node, and there is an edge relationship between the execution node and the target node; the perception node, the decision node and the execution node have perception capability, decision capability and execution capability respectively; the target node has the task capability required by the task objective; the edges between nodes have capability flow;

[0009] Determine an optimization constraint condition for the task chain; the optimization constraint condition means that according to the flow direction of the task capability from the self-sensing node to the target node, the maximum task capability flow to the target node is greater than or equal to the task capability flow required by the target node;

[0010] Constructing an optimization goal for accomplishing all of the stated mission objectives based on mission time, mission reliability, and mission cost of performing the mission;

[0011] Based on the optimization constraints and the optimization objective, an improved Monte Carlo search tree algorithm is applied to output a training sample set of a joint policy-value network, and the joint policy-value network is trained using the training sample set to obtain a trained joint policy-value network; the improved Monte Carlo search tree algorithm uses the policy output of the joint policy-value network to determine the selection of an action in a selection step, and uses the value output of the joint policy-value network to evaluate the state value of the expanded child node in an estimation step;

[0012] The trained joint strategy-value network is used to determine the optimal task chain for completing all task objectives in the task laminar flow network; or the Monte Carlo search tree algorithm combined with the trained joint strategy-value network is used to determine the optimal task chain for completing all task objectives in the task laminar flow network; one task objective corresponds to one optimal task chain.

[0013] A computer program product, including a computer program, is characterized in that when the computer program is executed by a processor, it implements the steps of the above-mentioned method for scheduling a drone cluster task chain based on deep reinforcement learning.

[0014] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0015] This invention provides a method and product for scheduling unmanned aerial vehicle (UAV) swarm task chains based on deep reinforcement learning. This method integrates a joint policy-value network into a Monte Carlo search tree algorithm to generate a training sample set. The training sample set is then used to train the joint policy-value network. Ultimately, the trained joint policy-value network can be used to determine the optimal solution for task chain scheduling, or the trained joint policy-value network can be combined with a Monte Carlo search tree algorithm to obtain the optimal solution. In this invention, the optimal task chain is determined by combining the joint policy-value network with the Monte Carlo search tree algorithm, improving the efficiency and reliability of swarm task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A flowchart of a method for scheduling a drone cluster task chain based on deep reinforcement learning is provided in Example 1 of the present invention.

[0018] Figure 2 A schematic diagram of the concept of a drone cluster task chain scheduling method based on deep reinforcement learning provided in Example 1 of the present invention.

[0019] Figure 3 This is a schematic diagram of the task chain format provided in Example 1 of the present invention.

[0020] Figure 4 This is a diagram of a task laminar flow network provided in Example 1 of the present invention.

[0021] Figure 5 This is the traditional reliability calculation process provided in Example 1 of the present invention.

[0022] Figure 6 This is the node cost calculation process provided in Example 1 of the present invention.

[0023] Figure 7 This is the link cost calculation process provided in Example 1 of the present invention.

[0024] Figure 8 This is a schematic diagram of a node set corresponding to all nodes in the task laminar flow network provided in Example 1 of the present invention.

[0025] Figure 9 This is a schematic diagram of a task chain after task capabilities are assigned according to Example 1 of the present invention.

[0026] Figure 10 Schematic diagram of the task chain after the node flow limit provided in Example 1 of the present invention is converted into the edge flow limit.

[0027] Figure 11 A diagram of a task chain network of an active sink provided in Example 1 of the present invention.

[0028] Figure 12 This is a diagram of a task chain network after the lower bound of traffic volume is flipped according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] Based on the idea of ​​system engineering, the solution to the task scheduling problem is divided into two steps: modeling and optimization.

[0031] In the modeling step, drone clusters belong to complex systems. Complex system modeling is an important means and prerequisite for studying complex systems. It is an abstraction made for understanding complex systems. The rise of complex network research is the most important step in the development of complex system modeling. The present invention chooses to build a model for drone cluster task scheduling based on complex network technology. Its core is to build a task chain that can describe the task execution process, and on this basis, build a task laminar flow network. In terms of single-layer networks, there have been relatively complete studies at home and abroad. Typical single-layer complex network models include small-world network models, random network models, and scale-free network models. Among them, the network's node degree, average degree, degree distribution, clustering coefficient and other indicators measure the basic topological properties.

[0032] According to the idea of ​​this step, the task scheduling problem in this paper can also be called the task chain scheduling problem.

[0033] In the optimization step, the three elements of optimization need to be clarified: optimization constraints, optimization objectives, and optimization algorithms.

[0034] Optimization constraints are auxiliary conditions for forming scheduling strategies. The flow of task capacity of the cluster during the task process directly determines whether the task is successful or not. The task capacity flow to the task target is not less than the task capacity flow required by the task target as the optimization constraint.

[0035] The optimization objective is the core of the scheduling strategy. Consider constructing a value function for the task chain as the optimization objective function. This value function should be able to balance the relationships between various indicators related to task requirements.

[0036] Optimization algorithms are a key factor influencing scheduling strategies. The cluster task chain scheduling problem is a typical NP-hard problem. For NP-hard problems, the solution space is typically very large. The time complexity of traversal algorithms for solving such problems is often too high to meet practical requirements. Therefore, heuristic algorithms, characterized by high solution efficiency and strong optimization capabilities for complex problems, have been proposed and widely used to solve this problem. However, with the increase in computing power, higher requirements have been placed on optimization algorithms, requiring them to exhibit a certain degree of adaptability, such as adapting to changes in optimization constraints or objectives within a certain range. In this case, heuristic algorithms are no longer suitable because they excel at searching but not learning. After running, they only record the solution found. Changes to the optimization constraints or objectives require a complete rerun of the algorithm, which fails to meet the adaptability requirements. A highly adaptable algorithm is needed to solve the task chain problem. Reinforcement learning is a field of machine learning that deals with how intelligent agents act in an environment to maximize cumulative rewards. Deep reinforcement learning (DRL) is a combination of deep learning (DL) and reinforcement learning (RL). In recent years, its ability to solve high-dimensional, nonlinear complex problems has been demonstrated in multiple fields, such as robotics, board games (such as Go), and video games. Cao D et al. conducted a comprehensive literature review of reinforcement learning from the perspectives of its basic ideas, various algorithms, and its applications in power and energy systems, and discussed the challenges and further work. Mosavi A et al. pointed out that the application of deep reinforcement learning (DRL) in economics has grown exponentially, providing a wealth of opportunities for dealing with complex dynamic economic systems. Vignon C et al. focused on the application of DRL in active flow control and emphasized the current limitations of DRL algorithms in this field. Chu T et al. first proposed a fully scalable and decentralized MARL algorithm in the context of adaptive traffic signal control, demonstrating that the algorithm has optimality, robustness, and high sampling efficiency. Therefore, the present invention selects a deep reinforcement learning (DRL) algorithm with high adaptability and self-learning capabilities to optimize task chain scheduling.

[0037] The purpose of this invention is to provide a UAV cluster task chain scheduling method and product based on deep reinforcement learning, which combines the Monte Carlo search tree algorithm and the joint strategy-value network to determine the optimal task chain for UAV cluster task chain scheduling, thereby improving the efficiency and reliability of cluster task execution.

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example 1

[0040] like Figure 1 and Figure 2 As shown, in this embodiment, a method for scheduling a drone cluster task chain based on deep reinforcement learning includes:

[0041] S1: According to the task loads and task objectives of different functions in the drone cluster mission, the OODA loop theory is applied to construct a task laminar flow network; the task laminar flow network includes several task chains; a task chain is used to complete a task objective; each task chain includes a perception node, a decision node, an execution node and a target node; there is an edge relationship between the perception node and the decision node, there is an edge relationship between the decision node and the execution node, and there is an edge relationship between the execution node and the target node; the perception node, the decision node and the execution node have perception capability, decision capability and execution capability respectively; the target node has the task capability required by the task objective; the edges between the nodes have capability flow.

[0042] Based on complex network technology, cluster task chain scheduling modeling is performed. The model is constructed based on system entities. Generally speaking, the basic components of a drone swarm system include the drone carrier, the mission payload, and the communication data link. Based on the drone swarm structure, a three-layer complex network can be established for the cluster, comprising a communication layer network, an aircraft layer network, and a task layer network. Since the research object of this invention is the task chain scheduling problem, this paper only focuses on the task layer network, constructing a task chain and task layer flow network model to reflect the task chain scheduling situation.

[0043] Building a task chain requires a clear understanding of the task process. This cluster task process can be described using the OODA loop theory. The OODA loop, a full cycle of observation (Observe), orientation (Orient), decision (Decision), and action (Act), was originally proposed by American pilot Colonel John Boyd to describe the process of air combat.

[0044] Based on the OODA cycle theory, the task layer network nodes can be divided into perception, decision-making, and execution nodes, represented as S, D, and I respectively, representing the task loads of different functions, and the relationship between the nodes is analyzed and defined to determine the existence of two edge relationships, SD and DI. In order to make the research process more in line with the actual situation of task execution, an additional target node is defined, represented as T, representing the task goal, and the edge relationship of IT is analyzed. Based on this, the form of the task chain is determined to be: SDIT. Figure 3 After determining the number of nodes of each type, all possible task links can be identified according to the form of task chains, thereby constructing a task laminar flow network.

[0045] Constructing a UAV swarm mission-layer flow network: Based on the establishment of swarm mission nodes and the construction of mission edges, the node and edge sets of the swarm mission-layer network can be determined. Taking mission capabilities into account, these capabilities are assigned to nodes, allowing for a preliminary construction of a mission-layer network model. However, for networks with only node attributes, directly applying graph theory algorithms to computations is infeasible. Simply assigning the maximum mission capability that the mission payload can achieve to task-layer nodes would not satisfy the requirements for subsequent analysis. Therefore, consideration was given to flow-based processing of the task-layer network by adding flow attributes to the edges.

[0046] The task layer network is transformed into a task layer flow network, represented as G c '={V c ,E c ,C,F}. Among them, V c ={V c1 ,V c2 ...V cn} corresponds to the node set in the task laminar flow network, representing the mission load and mission objectives of the UAV cluster, which is divided into perception nodes, decision nodes, execution nodes and target nodes, and n represents the number of nodes. c ={E c1 ,E c2 ...E cm} corresponds to the edge set in the task layer flow network, representing the edges abstracted from the task capability flow, and m represents the number of edges. Nodes are connected based on the OODA process to form a task chain, and at the same time, edges are generated between nodes. C={C1,C2...C n} corresponds to the task capability set in the task laminar flow network, representing the task capability of the cluster task load and the task capability required by the task target. In the task laminar flow network, the task capability is expressed as the attribute of the node, and there is a mapping relationship between the elements in the task capability set and the elements in the node set. According to the function of the task load, the task capability can be divided into perception capability, decision-making capability, and execution capability. The key lies in the addition of F, which is the edge flow function set of the task laminar flow network. The flow function is defined in the edge set E. c Each edge in the task layer flow network may have a capability flow in the task process. At the same time, the task process is sequential and the capability flow is directed. m}.

[0047] The flow function must meet the following constraints: the flow in and out of a node cannot exceed the maximum flow value allowed to pass through the node. In the task laminar flow network model, the task capability of a node is the maximum flow value allowed to pass through the node.

[0048] Therefore, step S1 specifically includes:

[0049] (1-1) The mission payloads and mission objectives of different functions in the UAV swarm mission are abstracted into nodes respectively.

[0050] (1-2) Apply the OODA loop theory to define the form of the task chain.

[0051] (1-3) Connect the nodes according to the form of the task chain to obtain edges between the nodes.

[0052] (1-4) Assign the task capability of the task payload to the corresponding node to obtain the node attributes.

[0053] (1-5) Construct a flow function for each edge, and determine the flow function constraint for each edge based on the task capability of each node; the flow function constraint is that the flow inflow and outflow of a node cannot exceed the maximum flow value allowed by the node; the maximum flow value allowed by the node refers to the task capability of the node.

[0054] (1-6) The capacity flow range of each edge is determined based on the flow function of each edge and the corresponding flow function constraints.

[0055] (1-7) The task laminar flow network is derived based on the range of the capacity flow of each node, the edges between nodes, the attributes of the nodes and the edges. Figure 4 , a schematic diagram of a task laminar flow network is given.

[0056] S2: Determine the optimization constraint conditions of the task chain; the optimization constraint conditions refer to the task capability flow direction from the self-sensing node to the target node, and the maximum task capability flow flowing to the target node is greater than or equal to the task capability flow required by the target node.

[0057] Optimization constraint determination method: Optimization constraints can be expressed by the following inequality:

[0058] f Ti >=f Timin .

[0059] Among them, f Ti is the task capacity flow to the i-th target node; f Timin The task capability required for the i-th target node is a constant.

[0060] In a task laminar flow network, task capabilities flow from perception nodes to decision nodes, from decision nodes to execution nodes, and from execution nodes to target nodes. This process, from perception to decision, execution, and finally to the target, constitutes a basic task chain. Based on the constraints of the task laminar flow network's flow function, the maximum task capability flow to the target node in this task chain can be easily calculated. This can be compared with the target node's required task capability flow to verify whether the task chain satisfies the optimization constraints.

[0061] However, in the process of executing tasks, the cluster often has multiple task targets. The number of task chains increases with the increase in the number of task targets, and is the same as the number of task targets. When the same node is reused in different task chains, the task chains will intersect, forming a complex network structure. At this time, the task capacity flow flowing to each target node will affect each other, making it difficult to verify whether the task chain can meet the optimization constraints. The process of determining whether the task chain can meet the optimization constraints in the case of multiple task targets is described in detail in steps (4-9) in step S4 below.

[0062] S3: Construct an optimization goal for completing all of the described mission objectives based on mission time, mission reliability, and mission cost of performing the mission.

[0063] Wherein, step S3 specifically includes:

[0064] (3-1) For each task target, calculate the ratio of the execution time in the task chain to the expected execution time.

[0065] (3-2) The ratios corresponding to all task targets form a task time array, and the maximum value in the task time array is taken to construct the task time function.

[0066] The task time function can be divided into two parts. For the task goal, there is an expected execution time that reflects the time urgency of processing the task goal, which is set as T ta For the process of executing tasks and processing task targets, there is also an execution time, which can be calculated by dividing the distance from the execution node to the target node by the speed of the execution node, and is set as T act .

[0067] Combine the two types of time. For a task, the shorter the execution time, the longer the expected processing time of the task target, and the easier it is to complete the task. Therefore, consider dividing the execution time corresponding to each task target in the task chain by its expected processing time to construct the task time array T ar The task time function takes the maximum value in the task time array. Let the task time function be T, then the calculation formula of T is as follows:

[0068]

[0069] T=max{T ar}.

[0070] (3-3) The traditional reliability of a set of task chains is calculated based on the average failure-to-work time of each drone in the drone swarm system. A vulnerability index is used to represent the reliability of the network formed by the set of task chains, namely, network reliability. A set of task chains includes the task chains of all task objectives, and each task chain in the set of task chains corresponds one-to-one to each task objective. The vulnerability index is obtained by weighting the weighted degree and weighted betweenness. The weighted degree and weighted betweenness are derived from the node degree, node task capability, and number of task chains in the network formed by the set of task chains.

[0071] A set of task chains may intersect to form a network. The greater the number of links and the more complex the intersections, the more complex the network. In this case, traditional reliability cannot fully describe the reliability of a set of task chains. Therefore, we consider network reliability as part of describing task chain reliability. We calculate traditional reliability and network reliability separately and combine them in a certain way to obtain the task reliability function for the task chain.

[0072] Among them, such as Figure 5 This is the traditional reliability calculation process. For traditional reliability, the following assumption is made: any failure of any drone in a task chain will cause the task chain to fail to complete the task. Based on this assumption, the traditional reliability of the link can be calculated using a series model.

[0073] Therefore, the specific calculation process of traditional reliability is:

[0074] (1) The reliability of each UAV in the UAV cluster system is calculated based on the average failure working time of each UAV.

[0075] (2) The reliability of each UAV is assigned to the corresponding node as a node attribute, and the reliability of each node in a set of task chains is extracted.

[0076] (3) Multiply the reliability of each node in a set of task chains to obtain the traditional reliability of a set of task chains. The calculation formula of traditional reliability is:

[0077]

[0078] Where n is the number of nodes in a task chain.

[0079] Research on network reliability has made considerable progress both domestically and internationally. This article analyzes network reliability using vulnerability indicators. Based on existing research, vulnerability indicators can be derived by weighting the weighted degree and weighted betweenness.

[0080] Therefore, the specific calculation process of network reliability is:

[0081] (1) Calculate the weight degree and weight betweenness based on the characteristics of the task laminar flow network.

[0082] (2) Weighting is performed according to the weight degree and the weight betweenness to obtain the vulnerability index.

[0083] (3) The network reliability of the task laminar flow network is represented by the vulnerability index. Let the vulnerability index be P, then the network reliability is:

[0084] R net =1-P.

[0085] Based on the calculation of traditional reliability and network reliability, the task reliability function of the task chain can be obtained by multiplying traditional reliability and network reliability. Let the task reliability function be R, and its calculation formula is as follows:

[0086] R=R t R net .

[0087] During mission execution, the operation of the mission payload carried by the drone incurs costs. These costs correspond to nodes in the mission layer network model and can be considered node costs. Furthermore, communication between drones also incurs costs, corresponding to the links between nodes in the communication layer network. Since the links in the mission layer network roughly correspond to the links between nodes in the communication layer network, these costs can be considered link costs. Based on this assumption, we divide costs into node costs and link costs, calculate them separately, and combine them in a specific way to derive the cost function for the mission chain.

[0088] The node cost and link cost of a set of task chains are calculated based on the operating cost of the mission payload carried by the UAVs and the communication conditions between the UAVs.

[0089] For node cost, refer to the idea of ​​assigning task capability to node attributes. The flowchart for calculating node cost is as follows: Figure 6 shown.

[0090] The calculation process of node cost is:

[0091] (1) The operating cost of the mission payload carried by the UAV is regarded as the node cost.

[0092] (2) Extracting the node cost of each node in a set of task chains, adding up the node costs of each node in a set of task chains, and obtaining the node cost of a set of task chains.

[0093] Regarding link cost, the following assumptions are made here: the communication capability of a single drone is good, medium, and poor; the better the communication capability between two drones, the lower the link cost; the communication capability between two drones is determined by the communication capability of the two drones; link reuse increases the cost of the link. Based on this, a flowchart for calculating link cost is given, as shown below: Figure 7 shown.

[0094] The link cost calculation process is as follows:

[0095] (1) The communication capability between drones is assigned to the corresponding nodes as a node attribute.

[0096] (2) Calculate the reuse of the communication link between two nodes with an edge relationship.

[0097] (3) Calculate the link cost of two nodes with an edge relationship based on the communication capabilities of the two nodes with an edge relationship and the reuse of the communication link between the nodes.

[0098] (4) Add up the cost of each communication link in a set of task chains to obtain the link cost of a set of task chains.

[0099] The task cost function is constructed by adding the node cost and the link cost of a set of task chains. The formula is as follows:

[0100] P=P node +P edge .

[0101] The final structure-value trade-off function is obtained by adding together each function with a certain weight. That is, the optimization objective of the task chain is constructed based on the task time function, the task reliability function, and the task cost function. The above three functions are added together according to the weights w1, w2, and w3 respectively, and the formula is as follows:

[0102] B=w1T+w2R+w3P.

[0103] S4: According to the optimization constraints and the optimization objectives, an improved Monte Carlo search tree algorithm is applied to output a training sample set of the joint strategy-value network, and the joint strategy-value network is trained using the training sample set to obtain a trained joint strategy-value network; the improved Monte Carlo search tree algorithm uses the strategy output end of the joint strategy-value network to determine the choice of action in the selection step, and uses the value output end of the joint strategy-value network to evaluate the state value of the extended sub-node in the estimation step.

[0104] The Monte Carlo search tree algorithm and the imitation learning algorithm (supervised learning algorithm is used in this invention) combined with a deep neural network (joint policy-value network) are selected as the optimization algorithm. The Monte Carlo search tree algorithm (MCTS) is a reinforcement learning algorithm that uses Monte Carlo simulation to estimate the value of each state in the search tree. As more and more simulations are performed, the search tree becomes larger and larger, and the relevant values ​​become more and more accurate. The strategy used to select actions in the search process is also improved over time, ultimately achieving the goal of strategy optimization.

[0105] The algorithm flow of the MCTS part combined with the joint strategy-value network is as follows:

[0106] 1) Iteration. Specifically including:

[0107] 1. Select

[0108] Starting from the root node, select an optional action until a node that is not in the search tree is found or the termination state is reached. Taking the case where the number of task targets is 2 as an example, there should be 2 task chains at this time. When the task chain is built, each of the 2 task chains should contain 1 perception, decision, execution, and target node (except the target node, the other types of nodes can be reused). Assuming that the initial state is the case where no node has been selected, there are 8 optional actions, namely: select the perception node of task chain 1, select the decision node of task chain 1, select the execution node of task chain 1, select the target node of task chain 1, select the perception node of task chain 2, select the decision node of task chain 2, select the execution node of task chain 2, and select the target node of task chain 2. When an action is selected, the state changes, and the optional actions are reduced by 1, and so on, until there are no optional actions. At this time, the task chain is built and the termination state is reached. The action selection is based on the following formula:

[0109]

[0110]

[0111] Among them, Q(s t ,a) represents the value of the action of node st executing a, and the calculation method will be given in the back propagation; u(s t ,a) represents exploratory, where P(s t ,a) is determined by the strategy output of the joint strategy-value network; N(s t ,a) is the child node st after node st executes a +1 The number of times selected in the search, initially 0. c0 is a parameter that does not control exploration.

[0112] S tRepresents a node in the search tree, and a represents the action of jumping from one node to the next. Since there is a one-to-one correspondence between the nodes in the search tree and the state of the task chain, such as the initial root node corresponds to the state where no node is added to the task chain, S t Can represent a task chain state, and a can represent the transition from one task chain state to the next task chain state. a and a t The difference is that the optional range of a is all feasible actions in the current state, and at is obtained by the formula, taking the a that maximizes Q+u among all a.

[0113] 2. Extension

[0114] In the selection step, if the selected action points to a node that is not in the search tree, the node is added to the search tree to generate a new child node.

[0115] 3. Estimation

[0116] Starting from the expanded child node, use the random strategy to simulate until the terminal state is reached, return the result, and evaluate the state value of the child node. The state value is evaluated according to the following formula:

[0117] V(s L )=(1-λ)v θ (s L )+λf iff≠0;

[0118] V(s L )=-1iff=0.

[0119] Among them, s L Indicates expanding the newly added leaf node; v θ (s L ) is the value output of the joint strategy-value network to node s L evaluation result; f is the result obtained from the simulation data, which is the task chain value function here. When the task chain task capability does not meet the requirements, f = 0; λ is the parameter that controls the weight of the simulation result and the neural network estimation result.

[0120] The task chain value is calculated by the optimization target calculation formula in step S3, which is the optimization target value. It can be compared to the value of 1 when winning in chess. In the "estimation" step, it is represented by "f". The value output of the joint strategy-value network outputs the state value of the current task chain state, which is an estimated value used to measure the quality of the current task chain state. In the "estimation" step, it is represented by "v θ (s L )”. V(s L) is the state value of the child node, which is also the state value of the current task chain state (because the nodes in the search tree correspond to the task chain states one by one). The formula in the "Estimate" step is equivalent to correcting the state value of the current task chain state with the task chain value.

[0121] 4. Backpropagation

[0122] Update the action value Q(s) of each node t ,a), according to the following formula:

[0123]

[0124] Here, i represents the i-th iteration of MCTS. A complete process from selection to backpropagation is one iteration in MCTS.

[0125] I(s,a,i) indicates whether to visit child(s) during the i-th search. t ,a);

[0126] 2) When the number of iterations reaches the set value (3-4 times the total number of task chain nodes), the iteration ends. Based on the results of the MCTS iteration, the number of selections for each optional action under the root node is determined. Let the state corresponding to the root node be s0, and then the probability π corresponding to each optional action is calculated according to the following formula:

[0127]

[0128] Select an action based on the probability, that is, select a new node.

[0129] The current task chain status and probability output by the MCTS algorithm are obtained here.

[0130] 3) Output a data set containing the current state value, the current task chain state, and the corresponding next action selection probability.

[0131] The process of combining the strategy-value network and the Monte Carlo search tree algorithm is as follows:

[0132] 1) Construct a randomly initialized joint policy-value network, where the policy network is used to select the next node and the value network is used to evaluate the current state. The initial value of the joint policy-value network is generated randomly.

[0133] 2) Start the loop.

[0134] 3) Substitute the joint strategy-value network into the MCTS algorithm, and the MCTS algorithm generates a large number of simulation results, that is, a large number of sample data constructed by task chains.

[0135] 4) Based on the labeled data obtained from the simulation results, the joint strategy-value network is trained using the imitation learning method.

[0136] 5) When the training result of the joint strategy-value network meets the expectation, the loop ends; otherwise, it returns to step 3).

[0137] Therefore, the specific process of step S4 is:

[0138] (4-1) Construct a node set for each task target; the node set includes all nodes in the task laminar flow network. Figure 8 shown.

[0139] (4-2) Construct the initial joint strategy-value network through random initialization.

[0140] (4-3) The current joint strategy-value network is introduced into the Monte Carlo search tree algorithm to obtain the current improved Monte Carlo search tree algorithm.

[0141] (4-4) The task chain state in which no node is selected is the initial task chain state.

[0142] (4-5) Taking the current task chain state as input, the current improved Monte Carlo search tree algorithm is used to output the current task chain state and the next action selection probability, and record the updated task chain state after adding the newly added node; the next action selection probability refers to the selection probability of the currently unselected node in each node set; the updated task chain state refers to the task chain state composed of all the nodes selected in the previous step plus the currently added node; the newly added node is selected by the current improved Monte Carlo search tree algorithm itself; the newly added node is a node in the node set corresponding to any selected task target.

[0143] When a node in the node set is selected, that is, a position in a row is marked, such as filling in a value of 1, it means adding the node at the corresponding position to the task chain. In this case, the corresponding node in this row will be deleted from the optional nodes.

[0144] (4-6) Determine whether all nodes included in the updated task chain state constitute a complete set of task chains, and obtain a first judgment result; a complete set of task chains includes N×4 nodes; N×4 nodes constitute N task chains; N task chains correspond to N task targets respectively; N represents the number of task targets.

[0145] (4-7) If the first judgment result is no, the updated task chain state is used as the current task chain state, and the process returns to step "using the current improved Monte Carlo search tree algorithm to output the updated task chain state after adding the new node and the probability of the next action selection."

[0146] (4-8) If the first judgment result is yes, the task chain value corresponding to a complete set of task chains is calculated using the optimization objective as the value function, and the currently calculated task chain value is combined with the current task chain state and the next action selection probability in each iteration to obtain N samples; the N samples are added to the training sample set; each sample includes the task chain value, the current task chain state and the corresponding next action selection probability.

[0147] (4-9) Determine whether each task chain in the current complete set of task chains satisfies the optimization constraint condition, and obtain a second determination result.

[0148] Wherein, steps (4-9) specifically include:

[0149] (4-91) For each task chain in a complete set of task chains, the node's perception capability, decision-making capability, and execution capability are used as the upper bound of the flow of the corresponding node, and the task capability required by the task target is used as the lower bound of the flow of the target node. Figure 9 Two quest chains are given for two quest objectives.

[0150] (4-92) Split each node in each task chain in a complete set of task chains into corresponding inbound nodes and outbound nodes, and add edges between the inbound nodes and outbound nodes to obtain the task chains after node splitting. Figure 10 shown.

[0151] (4-93) The upper bounds of the flow of the perception node, decision node, and execution node are converted into the upper bounds of the flow of the corresponding edges in the task chain after the node is split, and the lower bound of the flow of the target node is converted into the lower bound of the flow of the corresponding edges in the task chain after the node is split. Figure 10 shown.

[0152] (4-94) Add source points and sink points at both ends of the task chain after the node splitting to obtain a source-sink network. Figure 11 shown.

[0153] (4-95) In the active sink network, the lower bound of the flow of the edge at the target child node is flipped to the upper bound of the flow, and the flipped network is obtained; in the flipped network, the lower bound of the flow of the edge is 0. Figure 12 shown.

[0154] (4-96) The maximum flow algorithm is used to solve the flow between the source and sink in the flipped network, and the flow between the source and sink in each task chain is obtained.

[0155] (4-97) For each task chain in a complete set of task chains, if the flow of the edge at the target child node is solved to reach the upper limit of the flow, it means that the flow of the edge at the target child node can reach the lower limit of the edge flow before the flow is reversed, that is, the task capacity flow flowing to the target node is not lower than the task capacity flow required by the task target, and the optimization constraint is satisfied.

[0156] (4-98) If each task chain in a complete set of task chains satisfies the optimization constraint, it means that the complete set of task chains satisfies the optimization constraint.

[0157] (4-10) If the second judgment result is no, the task chain value to be modified is modified to the preset value (the task chain value will be normalized, so the preset value can be set to -1, or a value less than -1), and return to the step "taking the task chain state without selecting any node as the initial task chain state"; each iteration refers to each iteration process in the complete iterative process with the initial task chain state as input until a complete set of task chains is obtained; the task chain value to be modified refers to the task chain value calculated based on a set of task chains that do not meet the optimization constraints; the preset value is set to reduce the probability of the Monte Carlo search tree algorithm searching for a set of task chains that do not meet the optimization constraints.

[0158] (4-11) If the second judgment result is yes, determine whether the number of samples in the current training sample set reaches the preset sample size to obtain a third judgment result.

[0159] (4-12) If the third judgment result is no, return to the step of "taking the task chain state in which no node is selected as the initial task chain state".

[0160] (4-13) If the third judgment result is yes, the current task chain state is used as input, the task chain value and the next action selection probability are used as labels, and the current training sample set is used to train the current joint strategy-value network to obtain an updated joint strategy-value network.

[0161] (4-14) Determine whether an update stop condition has been met, and obtain a fourth determination result; the update stop condition is that the number of updates to the joint strategy-value network reaches a preset number of updates or the loss error of the currently updated joint strategy-value network converges to a preset loss error. The update stop condition may also be determining the difference between the optimal solution for the task chain output based on the currently updated joint strategy-value network and the optimal solution for the task chain output by the Monte Carlo search tree algorithm in combination with the currently updated joint strategy-value network. If the difference is greater than a preset value, it indicates that the update needs to continue; otherwise, the update is considered terminated.

[0162] (4-15) If the fourth judgment result is yes, a trained joint strategy-value network is obtained.

[0163] (4-16) If the fourth judgment result is no, the currently updated joint strategy-value network is used as the current joint strategy-value network, and the process returns to the step of "introducing the current joint strategy-value network into the Monte Carlo search tree algorithm to obtain the current improved Monte Carlo search tree algorithm".

[0164] S5: Using the trained joint strategy-value network, determine the optimal task chain that completes all task objectives in the task laminar flow network.

[0165] Wherein, step S5 specifically includes:

[0166] (5-1) Inputting the task chain state corresponding to each target node into the trained joint strategy-value network, outputting the predicted value of the state value of the current task chain state and the predicted value of the next action selection probability.

[0167] (5-2) Select the node with the largest probability value from the predicted value of the next action selection probability, record it as the selected node, input the task chain state composed of each target node and all selected nodes into the trained joint strategy-value network, and output the new predicted value of the state value of the current task chain state and the new next action selection probability.

[0168] (5-3) The new next-step action selection probability is used as the current next-step action selection probability, and the process returns to the step of "selecting the node with the largest probability value from the predicted value of the current next-step action selection probability" until a complete set of task chains is selected, i.e., the optimal task chain corresponding to each task goal is obtained.

[0169] S6: Determine the optimal task chain for completing all task objectives in the task laminar flow network using a Monte Carlo search tree algorithm combined with the trained joint strategy-value network; one task objective corresponds to one optimal task chain.

[0170] Wherein, step S6 specifically includes:

[0171] (6-1) Inputting the task chain state corresponding to each target node into the Monte Carlo search tree algorithm that introduces the trained joint strategy-value network, and obtaining a new updated task chain state and the next action selection probability output by the algorithm.

[0172] (6-2) Replace the task chain state corresponding to each target node with the new updated task chain state, and return to the step of "inputting the task chain state corresponding to each target node into the Monte Carlo search tree algorithm that introduces the trained joint strategy-value network" until the new updated task chain state contains all the nodes that constitute a complete set of task chains, that is, the optimal task chain corresponding to each task target is obtained.

[0173] In the present invention, there are two parallel approaches to obtaining the optimal solution: one can directly utilize the trained joint strategy-value network to obtain the optimal solution, and then select the process of step S5; the other can utilize the Monte Carlo search tree algorithm combined with the trained joint strategy-value network to obtain the optimal solution, and then select the process of step S6. Either step S5 or step S6 can be selected.

[0174] The present invention's method for scheduling drone swarm task chains applies algorithms such as Monte Carlo search trees from reinforcement learning, combined with deep neural networks, to find the optimal task chain scheduling result for drone swarms. Task chains are constructed to describe the execution process of tasks and serve as a model for cluster tasks. The ultimate goal of task chain scheduling is to maximize the benefits of cluster tasks.

[0175] The drone cluster task chain scheduling method of the present invention mainly includes two steps, modeling and optimization. In the modeling step, the cluster task chain and task laminar flow network model are constructed based on complex network technology and OODA theory. In the optimization step, the optimization target is constructed based on time, cost, and reliability. A method for judging whether the task chain meets the optimization constraints is given based on graph theory technology, and a detailed process for solving the task chain scheduling problem using the Monte Carlo search tree algorithm and imitation learning algorithm combined with a deep neural network as the optimization algorithm is given. Compared with traditional scheduling methods, the method of the present invention has high adaptability and self-learning ability. The high adaptability and self-learning ability come from the neural network technology used in both the Monte Carlo search tree and the imitation learning algorithm.

[0176] Example 2

[0177] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a method for scheduling a drone cluster task chain based on deep reinforcement learning in Example 1.

[0178] Example 3

[0179] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for scheduling a drone cluster task chain based on deep reinforcement learning in Example 1.

[0180] Example 4

[0181] A computer program product includes a computer program, which, when executed by a processor, implements the steps of a method for scheduling a drone cluster task chain based on deep reinforcement learning in Example 1.

[0182] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for scheduling UAV cluster task chains based on deep reinforcement learning, characterized in that: include: According to the task loads and task objectives of different functions in the UAV swarm mission, the OODA loop theory is applied to construct a task laminar flow network; the task laminar flow network includes several task chains; a task chain is used to complete a task objective; each task chain includes a perception node, a decision node, an execution node and a target node; there is an edge relationship between the perception node and the decision node, there is an edge relationship between the decision node and the execution node, and there is an edge relationship between the execution node and the target node; the perception node, the decision node and the execution node have perception capability, decision capability and execution capability respectively; the target node has the task capability required by the task objective; the edges between nodes have capability flow; Determine an optimization constraint condition for the task chain; the optimization constraint condition means that according to the flow direction of the task capability from the self-sensing node to the target node, the maximum task capability flow to the target node is greater than or equal to the task capability flow required by the target node; Constructing an optimization goal for accomplishing all of the stated mission objectives based on mission time, mission reliability, and mission cost of performing the mission; Based on the optimization constraints and the optimization objective, an improved Monte Carlo search tree algorithm is applied to output a training sample set of a joint policy-value network, and the joint policy-value network is trained using the training sample set to obtain a trained joint policy-value network; the improved Monte Carlo search tree algorithm uses the policy output of the joint policy-value network to determine the selection of an action in a selection step, and uses the value output of the joint policy-value network to evaluate the state value of the expanded child node in an estimation step; The trained joint strategy-value network is used to determine the optimal task chain for completing all task objectives in the task laminar flow network; or the Monte Carlo search tree algorithm combined with the trained joint strategy-value network is used to determine the optimal task chain for completing all task objectives in the task laminar flow network; one task objective corresponds to one optimal task chain.

2. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 1 is characterized in that: According to the mission load and mission objectives of different functions in the UAV swarm mission, the OODA loop theory is applied to construct a task laminar flow network, including: Abstract the mission payloads and mission objectives of different functions in the UAV swarm mission into nodes respectively; Apply the OODA loop theory to define the form of the task chain; Connecting the nodes according to the form of the task chain to obtain edges between the nodes; Assign the task capability of the task payload to the corresponding node to obtain the node attributes; Construct a flow function for each edge and determine the flow function constraint for each edge based on the task capability of each node; the flow function constraint is that the flow in and out of a node cannot exceed the maximum flow value allowed by the node; the maximum flow value allowed by the node refers to the task capability of the node; Determine the capacity flow range of each edge based on the flow function of each edge and the corresponding flow function constraint; The task laminar flow network is derived based on the range of the capacity flow of each node, the edges between the nodes, the attributes of the nodes and the edges.

3. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 1 is characterized in that: Construct an optimization goal to achieve all the stated mission objectives based on mission time, mission reliability, and mission cost of executing the mission, specifically including: For each task goal, calculate the ratio of the execution time in the task chain to the expected execution time; The ratios corresponding to all task targets are used to form a task time array, and the maximum value in the task time array is taken to construct a task time function; The traditional reliability of a set of task chains is calculated based on the average failure working time of each drone in the drone swarm system, and the network reliability of the task laminar flow network is represented by a vulnerability index. A set of task chains includes the task chains of all task objectives, and each task chain in the set of task chains corresponds to each task objective one-to-one. The vulnerability index is obtained by weighting the weight degree and weight betweenness. The weight degree and weight betweenness are derived from the node degree, node task capability, and number of task chains in the network formed by the set of task chains. constructing a task reliability function according to the traditional reliability and the network reliability; Calculate the node cost and link cost of a set of task chains based on the operating cost of the mission payload carried by the UAVs and the communication conditions between the UAVs; constructing a task cost function based on the node cost and the link cost of a set of task chains; The optimization objectives of the set of task chains are constructed according to the task time function, the task reliability function and the task cost function.

4. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 3 is characterized in that: The traditional reliability of a set of task chains is calculated based on the average failure working time of each drone in the drone cluster system. The network reliability of the task laminar flow network is represented by vulnerability indicators, including: The reliability of each drone is calculated based on the average failure working time of each drone in the drone cluster system; The reliability of each UAV is assigned to the corresponding node as a node attribute, and the reliability of each node in a set of task chains is extracted; Multiplying the reliability of each node in a set of task chains gives the traditional reliability of a set of task chains; Calculate weighted degree and weighted betweenness based on the characteristics of the task laminar flow network; Perform weighting according to the weight degree and the weight betweenness to obtain the vulnerability index; The vulnerability index is used to represent the network reliability of the task laminar flow network.

5. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 3 is characterized in that: The node cost and link cost of a set of task chains are calculated based on the operating cost of the mission payload carried by the UAVs and the communication conditions between the UAVs. Specifically, the cost includes: The operating cost of the mission payload carried by the UAV is regarded as the node cost; Extract the node cost of each node in a set of task chains, add up the node costs of each node in a set of task chains, and obtain the node cost of a set of task chains; Assign the communication capability between drones to the corresponding nodes as a node attribute; Calculate the reuse of the communication link between two nodes with an edge relationship; Calculate the link cost of two nodes with an edge relationship based on the communication capabilities of the two nodes and the reuse of the communication links between the nodes; The link cost of a set of task chains is obtained by adding up the cost of each communication link in the set of task chains.

6. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 3 is characterized in that: According to the optimization constraints and the optimization goal, an improved Monte Carlo search tree algorithm is applied to output a training sample set of a joint strategy-value network, and the training sample set is used to train the joint strategy-value network to obtain a trained joint strategy-value network, specifically including: Constructing a node set for each task target; the node set includes all nodes in the task laminar flow network; Construct the initial joint strategy-value network through random initialization; The current joint policy-value network is introduced into the Monte Carlo search tree algorithm to obtain the current improved Monte Carlo search tree algorithm; The task chain state in which no node is selected is regarded as the initial task chain state; Taking the current task chain state as input, the current improved Monte Carlo search tree algorithm is used to output the current task chain state and the next action selection probability, and record the updated task chain state after adding the newly added node; the next action selection probability refers to the selection probability of the currently unselected node in each node set; the updated task chain state refers to the task chain state composed of all nodes selected in the previous step plus the currently added node; the newly added node is selected by the current improved Monte Carlo search tree algorithm itself; the newly added node is a node in the node set corresponding to any selected task target; Determining whether all nodes included in the updated task chain state constitute a complete set of task chains, obtaining a first determination result; a complete set of task chains includes N×4 nodes; N×4 nodes constitute N task chains; the N task chains correspond to N task objectives respectively; N represents the number of task objectives; If the first judgment result is no, the updated task chain state is used as the current task chain state, and the process returns to step "using the current improved Monte Carlo search tree algorithm to output the current task chain state and the next action selection probability"; If the first judgment result is yes, using the optimization objective as the value function, calculate the task chain value corresponding to a complete set of task chains, combine the currently calculated task chain value with the current task chain state and the next action selection probability in each iteration, and obtain N samples; add the N samples to the training sample set; each sample includes the task chain value, the current task chain state, and the corresponding next action selection probability; Determine whether each task chain in the current complete set of task chains satisfies the optimization constraint condition, and obtain a second determination result; If the result of the second judgment is no, the task chain value to be modified is modified to the preset value, and the process returns to step "taking the task chain state without selecting any nodes as the initial task chain state"; each iteration refers to each iteration process in the complete iterative process using the initial task chain state as input until a complete set of task chains is obtained; the task chain value to be modified refers to the task chain value calculated based on the set of task chains that do not meet the optimization constraints; the preset value is set to reduce the probability of the Monte Carlo search tree algorithm searching for a set of task chains that do not meet the optimization constraints; If the second judgment result is yes, determine whether the number of samples in the current training sample set reaches a preset sample size, and obtain a third judgment result; If the result of the third judgment is no, then return to step "taking the task chain state without selecting any node as the initial task chain state"; If the third judgment result is yes, then the current task chain state is used as input, the task chain value and the next action selection probability are used as labels, and the current training sample set is used to train the current joint strategy-value network to obtain an updated joint strategy-value network; Determine whether an update stop condition is met to obtain a fourth judgment result; the update stop condition is that the number of updates of the joint strategy-value network reaches a preset number of updates or the loss error of the joint strategy-value network after the current update converges to a preset loss error; If the fourth judgment result is yes, a trained joint strategy-value network is obtained; If the result of the fourth judgment is no, the currently updated joint strategy-value network is used as the current joint strategy-value network, and the process returns to step "introducing the current joint strategy-value network into the Monte Carlo search tree algorithm to obtain the current improved Monte Carlo search tree algorithm".

7. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 6 is characterized in that: Determining whether each task chain in the current complete set of task chains satisfies the optimization constraint condition specifically includes: For each task chain in a complete set of task chains, the node's perception capability, decision-making capability, and execution capability are used as the upper bound of the corresponding node's traffic flow, and the task capability required by the task target is used as the lower bound of the target node's traffic flow; Split each node in each task chain in a complete set of task chains into corresponding inbound nodes and outbound nodes, and add edges between the inbound nodes and outbound nodes to obtain the task chain after node splitting; Convert the upper bounds of traffic flow at the perception nodes, decision nodes, and execution nodes into the upper bounds of traffic flow at the corresponding edges in the task chain after node splitting. Convert the lower bound of traffic flow at the target node into the lower bound of traffic flow at the corresponding edges in the task chain after node splitting. Adding source points and sink points at both ends of the task chain after the node splitting, respectively, to obtain a source-sink network; In the active sink network, the flow lower bound of the edge at the target child node is flipped to the flow upper bound, thereby obtaining a flipped network; in the flipped network, the flow lower bounds of the edges are all 0; The maximum flow algorithm is used to solve the flow between the source and sink in the flipped network, and the flow between the source and sink in each task chain is obtained; For each task chain in a complete set of task chains, if the flow of the edge at the target subnode reaches the upper bound of the flow, it means that the flow of the edge at the target subnode can reach the lower bound of the edge flow before the flow is reversed, that is, the task capacity flow flowing to the target node is not less than the task capacity flow required by the task target, and the optimization constraint condition is met; If each task chain in a complete set of task chains satisfies the optimization constraint condition, it means that the complete set of task chains satisfies the optimization constraint condition.

8. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 6 is characterized in that: The trained joint strategy-value network is used to determine the optimal task chain that completes all task objectives in the task laminar flow network, specifically including: Inputting the task chain state corresponding to each target node into the trained joint strategy-value network, and outputting a predicted value of the state value of the current task chain state and a predicted value of the next action selection probability; Selecting the node with the largest probability value from the predicted value of the next action selection probability, and recording it as the selected node; inputting the task chain state composed of each target node and all selected nodes into the trained joint strategy-value network; and outputting the predicted value of the state value of the new current task chain state and the new next action selection probability; The new next-step action selection probability is used as the current next-step action selection probability, and the process returns to step "selecting the node with the largest probability value from the predicted value of the current next-step action selection probability" until a complete set of task chains is selected, that is, the optimal task chain corresponding to each task goal is obtained.

9. The method for scheduling UAV cluster task chains based on deep reinforcement learning according to claim 6, characterized in that: The optimal task chain that completes all task objectives in the task laminar flow network is determined using a Monte Carlo search tree algorithm combined with the trained joint policy-value network, specifically including: Inputting the task chain state corresponding to each target node into the Monte Carlo search tree algorithm that introduces the trained joint strategy-value network to obtain a new updated task chain state and the next action selection probability output by the algorithm; The task chain state corresponding to each target node is replaced by the new updated task chain state, and the process returns to step "inputting the task chain state corresponding to each target node into the Monte Carlo search tree algorithm that introduces the trained joint strategy-value network" until the new updated task chain state obtained contains all the nodes that constitute a complete set of task chains, that is, the optimal task chain corresponding to each task target is obtained.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the drone cluster task chain scheduling method based on deep reinforcement learning according to any one of claims 1 to 7.

Images