A method and system for allocating material delivery tasks of drone clusters

Abstract

The present invention discloses a method and system for allocating material delivery tasks of a drone swarm, and uses a KM-RL algorithm to solve the best match. Specifically, the method comprises a value evaluation model based on reinforcement learning and a task allocation model based on bipartite graph matching. The value evaluation model evaluates the value evaluation value of each current candidate state transfer action according to historical data, and the task allocation model constructs a bipartite graph according to the value evaluation value calculated by the former and the initial reward value of the order. The Hungarian algorithm is used to obtain a complete match, so that the points on one side of the bipartite graph are completely matched, and all complete matches are traversed to find a solution with the largest sum of edge weights in the matching solution as the task allocation solution.

Description

Technical Field

[0001] The present invention relates to a method and system for allocating material delivery tasks of a cluster of unmanned aerial vehicles based on graph matching and reinforcement learning. Background Art

[0002] The coordinated delivery of materials by drone swarms is an important part of automated logistics and an important application scenario for drone swarms. In some congested areas of cities, traffic-blocked roads, and some remote rural areas, the delivery of goods or packages takes too long and is too costly due to the difficulty of people and vehicles passing through. The use of drones to deliver materials can effectively reduce the burden of ground transportation, improve traffic safety, reduce labor costs, and save resources and construction costs by making rational use of idle low-altitude resources. Figure 1 The scenario of drone swarm coordinated material delivery was demonstrated. In this application scenario, ground users are the demanders of material delivery services, and the flying ad hoc network composed of many drones is the service provider. The ground control station plays a core role in this scenario and has two main capabilities: one is to manage the massive tasks submitted by ground users and make intelligent task allocation decisions, and the other is to send task allocation instructions to the flying ad hoc network.

[0003] Task allocation is a key link in the formation of collaborative capabilities of drone clusters. It determines the mapping relationship between the member drones in the cluster and multiple task objectives, as well as the timing relationship of task execution. It is a discrete space combinatorial optimization problem under multiple constraints. The key to its implementation is to establish a mathematically expressible objective function and effectively handle various constraints. The effect of task allocation determines the benefits of the overall system and individual drones, and also directly affects the user experience. Reasonable task allocation can ensure that drones arrive at the mission target area in a timely manner and complete the task with maximum efficiency. At the same time, it can reduce resource waste and ensure the flexibility and robustness of the cluster.

[0004] Task allocation in the drone swarm material delivery scenario is to find the right drone to meet user needs. This is not a simple sequential execution process of a single task sequence. Since the task requirements appear randomly in time and are scattered in space, online dynamic task allocation decisions are required.

[0005] At present, the common task decision architecture suitable for drone clusters can be summarized into centralized, distributed, and decentralized types. Among them, centralized is the most mature and widely used mode. Traditional centralized task scheduling generally uses a "first come, first served" queuing strategy, or adopts a greedy algorithm based on time or location, but from the perspective of time and space, such methods are far from optimal for complex problems. In order to solve this problem, many centralized task allocation solution methods have been proposed, which can be mainly summarized into two categories: optimization methods and heuristic methods. Optimization methods include depth-first, breadth-first search algorithms, dynamic programming, combinatorial optimization, etc. Heuristic methods include ant colony algorithms, genetic algorithms, annealing algorithms, etc. These traditional allocation methods mainly focus on solving problems such as maximizing the benefits, number of tasks, efficiency, and minimizing delays in task allocation.

[0006] However, optimization algorithms such as combinatorial optimization have the disadvantages of low efficiency and poor results in large-scale and long-term tasks, so they are more suitable for offline and limited-scale tasks. Heuristic algorithms such as ant colony algorithms perform well in long-term tasks, but they often lack concise and profound analysis, and have a low convergence speed, making them unsuitable for time-sensitive tasks. In addition, these two types of methods also have the following limitations:

[0007] (1) It is impossible to effectively learn knowledge and find patterns from historical data. In order to obtain mathematically approximate optimal solutions, most traditional combinatorial optimization solutions inevitably simplify the spatial distribution of task requirements, service supply, and environmental variables. Dynamic online task allocation problems generally only know the local spatiotemporal information of task requirements and service supply. Traditional optimization solutions cannot effectively mine and utilize the profound patterns contained in historical data during the solution process.

[0008] (2) It is impossible to solve the time dependency problem between sequential decisions. Since dynamic online tasks are continuous in time, the task allocation decision we make is a multi-stage sequential decision problem, which means that the current decision is likely to affect the results and effectiveness of subsequent decisions. However, traditional optimization schemes can often only find the optimal solution for a certain period of time, thus falling into the local optimum and failing to find the global optimal solution in multi-stage decisions. Summary of the invention

[0009] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method for allocating material delivery tasks of drone swarms, which can improve the rationality of task allocation and the overall utility of the system on a long time scale while ensuring the timeliness of online decision-making. Another purpose of the present invention is to provide a drone swarm material delivery task allocation system that implements the above method.

[0010] To achieve the above-mentioned purpose, the present invention provides a method for allocating material delivery tasks of a drone cluster, which solves the best match based on the KM-RL algorithm as a task allocation scheme; specifically, it includes:

[0011] Step 1) Bipartite graph construction: Get the delivery task orders within the task area within a certain time window , as well as the starting point and destination of the order, determine the grid location of the starting point and destination, and obtain the initial reward value of the order; take the order starting point as the center, and Search for idle drones in a circular area with a radius of As a candidate worker, order and drones Construct a bipartite graph and use the initial reward value of the order as the initial edge weight;

[0012] Step 2) Weight transformation: Calculate the drone’s weight according to the estimated value function Execution of Orders The estimated value of the drone is obtained by adding the initial order reward and the estimated value using the bipartite graph edge weight algorithm. Execution of Orders The overall value of is used to update the edge weights in the bipartite graph to obtain a new bipartite graph;

[0013] Step 3) Matching solution: Use the KM-RL algorithm to calculate the optimal allocation solution for the new bipartite graph ;

[0014] Step 4) Value Update: Drone Fly to the designated location to execute the order After completing the task, the actual reward is obtained. According to the matching scheme and the actual reward, the state value evaluation is updated with the value evaluation function; in the task allocation in the next time window, the new state value evaluation will be used to guide the task allocation decision.

[0015] Furthermore, the task allocation model based on bipartite graph matching is as follows:

[0016] 1) Time window: The online task allocation decision process is divided into a sequence Divide into evenly sized time segments;

[0017] 2) Order: A material delivery task order uses a tuple Indicates that and Respectively indicate orders The origin, destination, initial order reward and estimated completion time;

[0018] 3) Worker: Send task worker drones Using a tuple It indicates that, express exist The position within the time window, express In the time window The state of the device; the states include idle and busy, represented by 0 and 1 respectively. When the system can Assign tasks, not vice versa;

[0019] 4) Flight distance: When two points are directly reachable, the flight distance is equal to the length of the straight line segment between the two points, that is, ; When it is not directly accessible, the flight distance is three times the length of the straight line segment between the two points, that is, ;

[0020] 5) Matching relationship: in the time window When allocating orders, the "order-drone" matching relationship can be represented by a bipartite graph. To indicate; among them, It is a time window The collection of all orders in It is a time window The collection of all idle and available drone workers in is the set of edges;

[0021] 6) In the time window In the allocation algorithm In the bipartite graph Select the allocation scheme ; yes A subset of the middle edges;

[0022] 7) Actual order rewards: is the matching result output by the platform, which includes several "order-drone" pairs. After receiving the order, rush to the designated location to execute Order, the actual order reward obtained after completing the order is :

[0023]

[0024] in, is the initial order reward, is the reward decay coefficient, which is a constant, It is the total time window from when the order is released to when it is accepted, which is composed of the allocation time and the time it takes for the drone to fly to the starting point;

[0025] 8) Utility Rating:

[0026] Allocation plan The utility score of is the sum of all matching actual order rewards in the scheme:

[0027] (1)

[0028] exist In a time window, the total utility is Order allocation for all time windows The sum of the utilities of

[0029] (2)

[0030] Optimization goal: Find a suitable allocation plan in each time window , so that Maximize the total utility within a time window.

[0031] Furthermore, in the value assessment model based on reinforcement learning, a Markov decision process is used to model a single worker drone in the time window. In the video, a drone The finite set of actions express:

[0032] (3)

[0033] Finite state set To express:

[0034] (6)

[0035] In order to reduce errors, the map is segmented into squares and regular hexagons, and the vertical height is segmented only by contour lines. They are the square and regular hexagonal space segmentation and positioning functions, which are used to map the position to a grid in the space; Indicates the current time window Internal drone The location.

[0036] Furthermore, each drone is regarded as an intelligent agent, all agents share the same state-value function, and all drones evaluate the executed commands through the shared state-value function.

[0037] Further, drones In the time window Status , get the estimated state value function of the cumulative reward It means that for different map segmentation methods, the state value function is expressed as:

[0038] (9)

[0039] (10)

[0040] State value function Our Responsibility for Drones In the time window Status The estimated cumulative rewards that may be obtained in the future; from formula (6), we can see that It can be expressed as , are the spatial partitioning and positioning functions of square and regular hexagon, respectively. represents the reward obtained by the action, and E represents the expected accumulation; formulas (9) and (10) represent the state value functions under the two grid segmentation methods respectively.

[0041] Furthermore, the state value function is calculated in Tile encoding:

[0042] (11)

[0043] Among them, V represents the state value evaluation function, Indicates The tiled grid area extending from the center, area represents the extension radius, is the standardization factor, To express During the time window, the drone In Location The subscripts S and H refer to two map segmentation methods.

[0044] Furthermore, the state value function is learned through time difference learning TD Learning , specifically adopting the online TD(0) learning method, according to the online TD(0) learning algorithm, if in the time window Drones in decision making Select an order , then the value evaluation function and The following updates will be made:

[0045] (12)

[0046] (13)

[0047] in is the learning rate, , .

[0048] Furthermore, the KM-RL algorithm is specifically:

[0049] in, is a bipartite graph, It is a time window The collection of all orders in It is a time window The collection of all idle and available drone workers in is the set of edges, It is the matching result output by the platform.

[0050] Furthermore, the value assessment model based on reinforcement learning and the task allocation model based on bipartite graph matching are interactively executed in the decision-making process of each time window. The interactive process is as follows:

[0051] Re-express the bipartite graph edge weights in the matching relation as:

[0052] (14)

[0053] in, is the old edge weight, equal to the initial order reward; The estimated value calculated by the valuation model represents the impact of the decision on the future; the estimated value function is defined as follows:

[0054] (15)

[0055] in, Indicates the value assessment of the current position of the drone, Indicates the value assessment of the destination location of the order. is the estimated time it will take to complete the order. is the discount factor in reinforcement learning, Indicates when the drone moves from the current position Move to The expected cumulative reward income at that time is the estimated value.

[0056] A drone cluster material delivery task allocation system is used to implement the above-mentioned drone cluster material delivery task allocation method.

[0057] The present invention is based on the KM-RL algorithm and is used to solve the online UAV cluster material delivery task allocation decision problem. The algorithm can improve the rationality of task allocation and the total system utility on a long time scale on the basis of ensuring the timeliness of online decision-making, and is suitable for large-scale and long-term UAV cluster material delivery task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of drone cluster coordinated material delivery;

[0059] Figure 2 is the "order-drone" bipartite graph;

[0060] Figure 3 This is a schematic diagram of using regular hexagonal division on the map plane;

[0061] Figure 4 This is a schematic diagram of using a square division method on a map plane;

[0062] Figure 5 A flowchart for value assessment based on reinforcement learning;

[0063] Figure 6 This is the framework diagram of the KM-RL algorithm;

[0064] Figure 7 This is a schematic diagram showing an example of the problem-solving process based on the KM-RL algorithm;

[0065] Figure 8 It is a schematic diagram of the obstacle representation method in the map data;

[0066] Fig. 9 This is a comparison chart of the total utility of the KM-RL algorithm and the other three algorithms;

[0067] Fig.10 This is a comparison chart of the average order-taking time of the KM-RL algorithm and the other three algorithms. DETAILED DESCRIPTION

[0068] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0070] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] The following combination Figure 1-Figure 10 The specific embodiments of the present invention are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0072] The invention discloses a method for allocating material delivery tasks of a drone cluster, which uses a KM-RL algorithm to solve the best match. The method specifically comprises a value evaluation model based on reinforcement learning and a task allocation model based on bipartite graph matching. The value evaluation model evaluates the value evaluation value of each current candidate state transfer action according to historical data, and the task allocation model constructs a bipartite graph according to the value evaluation value calculated by the former and the initial reward value of an order. The Hungarian algorithm is used to obtain a complete match, so that the points on one side of the bipartite graph are completely matched, and all complete matches are traversed to find a solution with the largest sum of edge weights in the matching solution as the task allocation solution.

[0073] Considering that if each drone greedily takes action based on all its candidate orders, conflicts will inevitably occur, resulting in the possibility that the same order may be assigned to multiple drones. Therefore, the present invention adopts a centralized architecture to make task allocation decisions. The optimization goal is to optimize order scheduling over a long period of time (for example, a few hours or a day) to maximize the overall benefits of the drone cluster while ensuring the smooth completion of the delivery task. This relies on the detailed modeling of user needs and drone material delivery characteristics in time and space, as well as rapid perception and control in a real-time environment.

[0074] 1. Task allocation model based on bipartite graph matching

[0075] The present invention uses graph theory to model and solve the problem of unmanned aerial vehicle material delivery task allocation. Graph theory represents the object characteristics of the target and the receiving task by graphical method, and the bipartite graph matching model is a classic graph theory task allocation model.

[0076] The present invention uses a bipartite graph matching model to establish a matching relationship between the members of the drone cluster system and the target task, and then uses the KM (Kuh-Munkras algorithm) algorithm to solve the best match as a task allocation solution. The KM algorithm is a classic algorithm for solving the best match of a bipartite graph model. The best match is the maximum weight match in a weighted bipartite graph. To solve the best match using the KM algorithm, it is necessary to first use the Hungarian algorithm to find a complete match (that is, the points on one side of the bipartite graph have been completely matched), and then traverse all complete matches to find the solution with the largest sum of edge weights in the matching solution.

[0077] In order to better model and solve the problem of drone material delivery, the following assumptions are made:

[0078] (1) Material delivery tasks appear randomly at various locations in space and are discrete in time and space. Each task is independent and unrelated to each other.

[0079] (2) Drones are task executors. Drones have two states: idle and busy. They can only accept orders when they are idle. An order can only be assigned to one drone, and a drone can only accept one order at a time.

[0080] (3) After the drone mission is completed, corresponding rewards can be obtained. Before the task is assigned, the mission starting point, target point and drone location are all known, and the distance can be calculated. The longer the distance and the longer the task takes, the higher the reward.

[0081] (4) All order tasks must be completed. In order to encourage drones to accept orders as early as possible, order rewards are linked to completion time. The later the order is completed, the lower the reward the drone receives.

[0082] Here we formally define the drone material delivery mission.

[0083] Definition 1: Time Window

[0084] The present invention transforms the online task allocation decision process into a sequence The sequence is divided into evenly sized time segments, i.e., time windows. For each task order accumulated in a time window, an allocation decision is executed. Each number in the sequence corresponds to a specific time window, thus establishing an index for the time window.

[0085] By setting the time window, the present invention normalizes the online dynamic and online task allocation problem into a serialized decision problem. In order to achieve the effect of online decision-making, the setting of the time window should not be too large.

[0086] Definition 2: Order

[0087] A material delivery task order (order) uses a tuple Indicates that and Respectively indicate orders The origin, destination, initial order reward and estimated completion time.

[0088] The departure and destination are set by the user when submitting the request, expressed in latitude, longitude and altitude. The initial order reward and estimated completion time are calculated by the platform after receiving the user's request based on factors such as flight distance, climb altitude and route information.

[0089] Definition 3: Worker

[0090] Here we will use the task worker drone (worker) Using a tuple Indicates. Among them, express exist The location within the time window (location), express In the time window The state (state) in which the device is located. The states include idle and busy, which are represented by 0 and 1 respectively. When the system can Assign tasks, but not vice versa. When a drone completes a task, it changes its state to idle and updates its position to the current position. The present invention assumes that the state and position remain unchanged within a time window, that is, the position and state at the beginning of this time window (also at the end of the previous time window) are used as decision data.

[0091] Definition 4: Flight distance

[0092] Flight distance is different from straight-line distance. It is the actual distance that a drone flies when performing a material delivery mission. When there are obstacles or no-fly zones on the drone's route (between the drone's location and the order's starting point, or between the order's starting point and the end point), the drone needs to take a detour. Therefore, points with similar straight-line distances on the map may not be similar for the drone. In fact, flight distance depends on path planning.

[0093] Since path planning is beyond the scope of this invention, this invention simply summarizes the possible situations into two types: directly reachable and indirect reachable. When two points are directly reachable, the flight distance is equal to the length of the straight line segment between the two points, that is, When it is not directly accessible, the flight distance is three times the length of the straight line segment between the two points, that is, .

[0094] Definition 5: Matching Relationship

[0095] like Figure 2 As shown, in the time window When allocating orders, the "order-drone" matching relationship can be represented by a bipartite graph To express.

[0096] in, It is a time window The collection of all orders in It is a time window The collection of all idle and available drone workers in is a set of edges. Each edge Can be used To indicate that , and Represent the weights of orders, workers and edges respectively.

[0097] When assigning tasks, drones that are far from the starting point are obviously not suitable workers. Therefore, the present invention sets a distance threshold (the distance here refers to the flight distance rather than the straight-line distance) with the starting point as the center. , only drones within the range can become candidates. If there are no idle drones near a certain order, the search range will be gradually expanded in the task allocation of subsequent time windows.

[0098] Definition 6: Allocation algorithm and allocation scheme

[0099] In the time window In the allocation algorithm (Here we use the KM algorithm) in a bipartite graph with certain rules Choose the appropriate allocation scheme . yes The subset of middle edges (i.e., the "order-drone" matching relationship) is also the final solution to the bipartite graph matching problem.

[0100] Definition 7: Actual Order Reward

[0101] is the matching result output by the platform, which includes several "order-drone" pairs. After receiving the order, rush to the designated location to execute Order, the actual order reward obtained after completing the order is In order to encourage drones to accept orders and perform tasks as soon as possible, the present invention sets Decreasing over time: in, is the initial order reward, is the reward decay coefficient, which is a constant, It is the total time window from when the order is issued to when it is accepted, which is composed of the allocation time and the time it takes for the drone to fly to the starting point. Therefore, the present invention encourages the platform to allocate orders to nearby drones.

[0102] Definition 8: Utility Rating

[0103] Allocation plan The utility score of is the sum of all matching actual order rewards in the scheme.

[0104] (1)

[0105] exist In a time window, the total utility is Order allocation for all time windows The sum of the utilities.

[0106] (2)

[0107] Optimization goal: Find a suitable allocation plan in each time window , so that Maximize the total utility within a time window.

[0108] 2. Value Assessment Model Based on Reinforcement Learning

[0109] Reinforcement Learning (RL) has significant advantages in solving these two problems. Reinforcement learning can often learn knowledge from a large amount of historical data autonomously and effectively, avoiding pre-assumptions and simplifications, and effectively revealing the dependencies between sequential decisions in continuous time.

[0110] The present invention introduces reinforcement learning to quantitatively understand the spatiotemporal pattern of worker drones and user task requirements in the entire region, with the goal of conducting value assessments for each point in time and space, generating state value assessments, and further utilizing them in the subsequent drone cluster collaborative material delivery task allocation decision-making process. In addition, the present invention uses multi-agent reinforcement learning, treating each drone as an agent.

[0111] (1) Markov decision process

[0112] The present invention uses Markov decision process (MDP) to model a single worker drone, using five-tuple To define. Among them, (State) is a finite set of states, representing different states of the agent. (Action) is an action, which is the set of all possible actions of the agent. (Reward) is the reward, which represents the feedback from the environment to the actions taken by the agent. (Transformer) is the action transfer function, which represents the probability of transitioning from one state to another. is the discount factor, which is the coefficient at which rewards accumulate over time.

[0113] Specifically, in the time window In the game, a certain intelligent agent - drone The finite set of actions The finite state set is represented by To express.

[0114] (3)

[0115] Formula (3) represents the UAV All possible actions, including those from the bipartite graph Select an order from the given candidate matches, or remain idle. When the actions selected by multiple drones conflict (for example, two drones respond to the same order), the platform makes a centralized decision.

[0116] (4)

[0117] In formula (4), and Represent the current time window and drone respectively To avoid the curse of dimensionality, the present invention grids the map, such as Figure 3 As shown, the entire three-dimensional space area is divided into many small grids, and the position of the drone is mapped to a grid in the three-dimensional space, as shown in the figure.

[0118] (5)

[0119] For ease of processing, the present invention ignores the slight changes in the position of the drone within a time window, thereby eliminating the time parameter. Then formula (4) can be approximated as (5), where is a function that maps positions to a spatial grid.

[0120] (6)

[0121] At the same time, in order to reduce the error, the present invention uses two segmentation methods, square and regular hexagon, on the map plane (vertical height is only segmented by contour lines), such as Figure 3 , Figure 4 As shown. Therefore, formula (5) can be further transformed into formula (6). Then, the drone Execute task orders The position transfer can be expressed as Transform to .

[0122] Discount Factor It reflects the expectation of future rewards. ,if As it approaches 0, it places more emphasis on immediate benefits. When it approaches 1, it places more emphasis on future benefits. In formula (4), This means that the current value assessment of a location will decay exponentially over time, which reflects the important role of time in value assessment.

[0123] (2) State value function

[0124] State Value Function The invention represents the invention of unmanned aerial vehicle In the time window Status An estimate of the cumulative rewards that may be obtained in the future. The expected value of the cumulative rewards brought by multiple decisions in the future:

[0125] (7)

[0126] Worker drones are in the same position in the task allocation decision, so different drones in the same position at the same time will have the same value evaluation. Therefore, in the reinforcement learning process, the present invention sets all agents to share the same state-value function. This technique enables all agents to work together. For example, a drone is working in the east of the city, and now there is an order to go to the west. Since the drone has not been to the west and has no state-value function, it cannot make an accurate assessment of the destination. Drones working in the west can be evaluated through their state-value functions. By sharing the state-value function, drones in the east can use this information to evaluate destinations it has never been to. The expected value of the cumulative rewards brought by multiple decisions in the future:

[0127] (8)

[0128] The present invention omits the subscript To represent the shared state value function. Therefore, the state value function of formula (7) can be written as formula (8).

[0129] (9)

[0130] (10)

[0131] In addition, for different map segmentation methods, the state value function can be expressed in two forms: (9) and (10).

[0132] (11)

[0133] The present invention combines the two state value evaluation methods of formula (9) and (10) to calculate the state value function in a tile coding manner, as shown in formula (11). Indicates The tiled grid area extending from the center (such as Figure 3 ), is the normalization factor. This state value evaluation method allows richer features to be expressed. For convenience, the present invention omits the superscript and subscript ,use To express During the time window, the drone In Location Status value assessment.

[0134] (3) Value Assessment and Learning Process

[0135] The present invention uses temporal-difference learning (TD Learning) to learn the state value function TD learning is a model-free method in reinforcement learning. It learns the value function only by combining the immediate reward and the bootstrap approximation of the future reward. In the scenario of drone material delivery mission studied by the present invention, the task orders are sudden and the location is unpredictable. Therefore, the present invention selects the online TD(0) (immediate reward within one step) learning algorithm. The value evaluation and learning process are as follows Figure 5 shown.

[0136] According to the online TD(0) learning algorithm, if Drones in decision making Select an order , then the value evaluation function and The following updates will be made:

[0137] (12)

[0138] (13)

[0139] in is the learning rate, , Same.

[0140] 3. KM-RL algorithm framework and problem-solving process

[0141] The KM-RL algorithm uses reinforcement learning to analyze the spatiotemporal pattern of material delivery mission requirements, and uses the bipartite graph matching model and the KM algorithm to perform task allocation decision calculations, combining the advantages of both.

[0142] (1) Algorithm framework

[0143] The KM-RL algorithm framework is as follows Figure 6 As shown in Figure 1, it consists of two parts: a value evaluation model based on reinforcement learning and a task allocation model based on bipartite graph matching. The value evaluation model aims to evaluate the potential value of each candidate state transition action based on historical data. The task allocation model constructs a bipartite graph based on the value evaluation value calculated by the former and the initial reward value of the order, and then uses the KM algorithm to solve it and make task allocation decisions.

[0144]

[0145] These two models interact in the decision-making process of each time window. Algorithm 1 shows this interaction process, which will be explained in detail below. First, the edge weight of the bipartite graph in Definition 5 is re-expressed as:

[0146] (14)

[0147] in, is the old edge weight, equal to the initial order reward, The estimated value calculated by the valuation model represents the impact of the decision on the future. The estimated value function is defined as follows:

[0148] (15)

[0149] in, Indicates the value assessment of the current position of the drone, Indicates the value assessment of the destination location of the order. is the estimated time it will take to complete the order. is the discount factor in reinforcement learning, which will be introduced later. It can be seen intuitively that Indicates when the drone moves from the current position Move to The expected cumulative reward income at that time is the estimated value.

[0150] Relative to the old edge weight , that is, the current order The initial order reward, the new edge weight More forward-looking because it considers what would happen if drones Assign to order , How much expected benefit will be obtained in the future. For example, if The destination is in the order hotspot area, which means that the drone There may be more opportunities to accept orders with high reward values, then the weight of the edge will be larger, and vice versa.

[0151] The problem-solving process based on the KM-RL algorithm can be summarized into four steps: bipartite graph construction, weight transformation, matching solution and value update. Figure 7 The solution steps are explained in a simple example. To simplify the process description, only the square map segmentation method is used as an example, and the mobilization of idle drones is not considered for the time being. Figure 4 ), discount factor Set to 1 (discount factor It reflects the expectation of future rewards. ,if As it approaches 0, it places more emphasis on immediate benefits. Approaching 1 means more emphasis on future gains), learning rate Set to 0.1.

[0152] Step 1. Bipartite graph construction. In a certain time window, there are two material delivery task orders in the task area. The departure point is in grid 10 and the destination is in grid 16. The starting point is in grid 7, and the destination is in grid 1. The initial reward values ​​of the two orders are 3 and 4 respectively (i.e., the initial edge weight ). The centralized decision-making platform is centered on the two order starting points and Search for idle drones as candidate workers in a circular area with a radius of There are two candidate workers and ,Order There is a candidate Then the platform will order , and drones , Construct a bipartite graph and use the initial order rewards 3 and 4 as the initial edge weights .

[0153] Step 2. Weight transformation. The present invention uses formula (15) The method is used to calculate the drone Execution of Orders , The estimated values ​​are -0.5 and 0.1 respectively, and the drone Execution of Orders The estimated value of is 0.6. The present invention uses formula (14) (3-0.5=2.5; 4+0.1=4.1) method, add the initial order reward and the estimated value to get the drone Execution of Orders , The overall values ​​of drones are 2.5 and 4.1 respectively. Execution of Orders The total value is 3.6 (3+0.6=3.6). And use this to update the edge weights in the bipartite graph to get a new bipartite graph.

[0154] Step 3. Matching solution. The present invention uses the KM algorithm to calculate the optimal allocation solution of the new bipartite graph, namely , Order 1 is assigned to Drone 2 for execution, and Order 2 is assigned to Drone 1 for execution.

[0155] Step 4. Update value. Drone and Fly to designated locations for execution and After completing the order task, you will get the actual reward. The platform will use the formula (12) based on the matching plan and the actual reward. The state value evaluation is updated in the way of reinforcement learning. Figure 5 , which will not be described here. In the task allocation of the next time window, the platform will use the new state value evaluation to guide the task allocation decision.

[0156] The present invention also provides a UAV cluster material delivery task allocation system, which is used to implement the above-mentioned UAV cluster material delivery task allocation method.

[0157] Example:

[0158] 1. Experimental environment and settings

[0159] (1) Experimental environment

[0160] The configuration of the equipment used in the experiment of the present invention is as follows:

[0161] CPU: Intel Core I9-10980XE (18 cores, 3.0GHz);

[0162] Memory: 4 DDR4 memory sticks (16G each, main frequency 3600 MHz);

[0163] Graphics card: 1 GeForce RTX3090, video memory 24G;

[0164] System: Ubuntu 20.04.

[0165] The present invention is experimented on a UAV material delivery order scheduling simulator, and the simulator and all algorithms are implemented in Python3.

[0166] The simulator uses a time window setting when running, and performs a task allocation once in each time window. Each time an allocation is made, a bipartite graph is constructed based on the position of the current idle agent (drone) and the existing task orders (including newly appeared orders and orders that have not been allocated in the previous period). Then, the simulator executes the task allocation algorithm to find the allocation plan, and calculates the actual order reward and total utility according to Definition 7 and Definition 8 to evaluate the allocation plan. It is worth noting that, considering that the actual flight distance and actual completion time of drones in the process of performing tasks are usually close to but not equal to the estimated situation, the present invention adds a random disturbance on the basis of the estimated completion time as the actual completion time.

[0167] (2) Parameter settings

[0168] The present invention sets the time window size to 2 seconds, the side lengths of the map square grid and hexagonal grid are set to 1100 meters and 645 meters respectively, and the drone search range threshold is Set to 3km, the learning rate in formulas (12) and (13) Set to 0.03, the discount factor Set to 0.9.

[0169] (3) Comparison of algorithms

[0170] This paper compares the KM-RL algorithm with the following three scheduling algorithms:

[0171] a) KM algorithm: The KM algorithm is executed on the bipartite graph each time a task is assigned, and the best matching result is directly used as the task assignment solution.

[0172] b) Greedy algorithm. The greedy algorithm is a classic task allocation solution. When allocating tasks, it first sorts all the edges on the bipartite graph in descending order according to the weights, then selects the edges with the largest weights in turn as the task allocation options, and deletes the related nodes from the graph, finally forming a complete allocation plan.

[0173] c) Nearest neighbor priority: It converts the bipartite graph matching problem into a minimum cost maximum flow problem, where the cost is the distance between the order and the drone. The algorithm searches for the allocation solution with the largest matching cardinality and the smallest total cost in each calculation.

[0174] 3. Evaluation indicators

[0175] Total utility: the sum of all actual order rewards for completing orders within a period of time (see formula (2) for details).

[0176] Average order-taking time: The average time it takes for an order to be posted and for the drone to arrive at the mission starting point to pick it up. This reflects the rationality of the allocation algorithm and determines the quality of the user experience.

[0177] 2. Experimental data

[0178] Since there is no real data set for drone swarm material delivery tasks, in order to simulate real scenarios, the present invention draws on public map data and taxi order data to construct the learning and test data of the drone material delivery tasks of the present invention. The specific situation is as follows:

[0179] (1) Map data

[0180] The original data comes from OpenStreetMap, and the present invention has made the following modifications to it:

[0181] a) Added height data and changed the two-dimensional data into three-dimensional.

[0182] b) After the map is gridded, the present invention adds data on whether or not a road can be driven (marked with 0 or 1) to each grid, and sets up obstacles such as no-fly zones and urban buildings. Figure 8 shown.

[0183] (2) Order data

[0184] The original data comes from Didi Chuxing's Gaia Data Open Plan. This paper reduces its data density by random sampling and draws on the spatial distribution of real orders. The main contents of each order data include:

[0185] a) The starting point and end point of the order. The starting point and end point of the order are indicated by longitude, latitude, and altitude data.

[0186] b) Initial order reward. The order reward is related to factors such as flight distance, material weight, route conditions, etc., and is obtained through comprehensive calculation. The present invention uses the order reward value of real taxi order data to simulate the initial order reward.

[0187] c) Estimated completion time. The present invention assumes that all drones have the same flight speed, and the estimated completion time is calculated by the flight distance in Definition 4.

[0188] 3. Experimental results and analysis

[0189] The results are as follows Fig. 9 , Fig.10 shown. Fig. 9 The figure shows the comparison of the total utility of the four algorithms in different test sets for a total of seven days. It can be seen that the utility performance of the KM-RL algorithm has always been better than that of the other three algorithms. Among them, the KM-RL algorithm has a 33% better utility on the day with the largest gap compared to the greedy algorithm, and a 9% better utility on the day with the smallest gap. Fig.10 The figure shows the comparison of the average order-taking time of the four algorithms in different test sets for a total of seven days. It can be seen that the average order-taking time of the KM-RL algorithm is less than that of the other three algorithms. Among them, the average order-taking time of the KM-RL algorithm is 7.8% to 21.6% less than that of the greedy algorithm. Such results show the superior performance of the KM-RL algorithm in terms of total utility and rationality of task allocation.

Claims

1. A method for allocating material delivery tasks of drone swarms, characterized in that: This method solves the best match based on the KM-RL algorithm as a task allocation solution; specifically includes: Step 1) Bipartite graph construction: within a certain time window, obtain the delivery task order o in the task area, as well as the starting point and destination of the order, determine the grid position of the starting point and destination, and obtain the initial reward value of the order; with the order starting point as the center, search for idle drone w as a candidate worker in the circular area with a radius of threshold θ, construct a bipartite graph with order o and drone w, and use the initial reward value of the order as the initial edge weight; Step 2) Weight transformation: Calculate the estimated value of drone w executing order o according to the estimated value function, add the initial order reward and the estimated value using the bipartite graph edge weight algorithm to obtain the overall value of drone w executing order o, and use this to update the edge weights in the bipartite graph to obtain a new bipartite graph; Step 3) Matching solution: Use the KM-RL algorithm to calculate the optimal allocation solution for the new bipartite graph<o,w> ; Step 4) Value update: Drone w flies to the designated location to perform the task of order o. After completing the task, it obtains the actual reward. According to the matching scheme and the actual reward, the state value evaluation function is used to update the state value evaluation. In the task allocation in the next time window, the new state value evaluation will be used to guide the task allocation decision. Among them, the estimated value function is defined as follows: Among them, V(l w ) represents the value assessment of the current position of the UAV, V(d o ) represents the value assessment of the order destination location, τ o is the estimated completion time of the order, γ∈[0,1] is the discount factor in reinforcement learning, and μ o,w It means when the drone moves from the current position l w Move to d o The expected cumulative reward income when , that is, the estimated value; The state of drone w in time window t The estimated cumulative reward is expressed by the state value evaluation function V. For different map segmentation methods, the state value evaluation function is expressed as: δ S ,δ H are the space partitioning and positioning functions of square and regular hexagon, respectively, V S ,V H are the state value evaluation functions corresponding to the square and regular hexagonal map segmentation methods, R (t) represents the reward obtained by the action, E represents the expected accumulation, represents the position of UAV w in the current time window t; The KM-RL algorithm is specifically: Input: G (t) = <O (t) ,W (t) ,E (t) >, algorithm KM; Output: M (t) ; Step 1) If t=1: V S ←0 V H ←0 Step 2) Calculate V using the state value evaluation function formula calculated in a tile encoding manner; Step 3) Update and calculate edge weights; Step 4) M (t) ←A1(G (t) ); Step 5) Loop through <o,w,ω o,w >∈M (t) implement: Update V S Update V H Step 6) Return to M (t) ; Among them, G (t) is a bipartite graph, O (t) is the set of all orders in time window t, W (t) is the set of all available UAVs in time window t, E (t) is the set of edges, M (t) It is the matching result output by the platform; V S ,V H The two state value evaluation methods are combined to calculate the state value evaluation function in Tile encoding: V(l)←δ·(∑ area∈AREA (V(δ S (l+area))+V(δ H (l+area))) (11) Among them, V represents the state value evaluation function, AREA represents is the tiled grid area extending from the center, area represents the extension radius, δ is the normalization factor, and V(l) is used to represent the state value assessment of UAV w at position l within the time window t.

2. The method for allocating materials delivery tasks of drone swarms as claimed in claim 1, characterized in that: The task allocation model based on bipartite graph matching is as follows: 1) Time window: The online task allocation decision process is divided into time segments of uniform size using a sequence <1,2,…,T>; 2) Order: A material delivery task order uses a tuple <m o ,d o ,r o ,τ o > indicates that m o ,d o ,r o and τ o They represent the departure place, destination, initial order reward and estimated completion time of order o respectively; 3) Worker: Use a tuple to represent the task worker drone w It indicates that, represents the position of w in the time window t, Indicates the state of UAV w in time window t; the states include idle and busy, represented by 0 and 1 respectively. When , the system assigns tasks to UAV w, but not vice versa; 4) Flight distance: When two points are directly reachable, the flight distance is equal to the length of the straight line segment between the two points, that is, dist(a,b) = length(a,b); when not directly reachable, the flight distance is three times the length of the straight line segment between the two points, that is, dist(a,b) = 3 · length(a,b); 5) Matching relationship: When allocating orders in time window t, the "order-drone" matching relationship is represented by a bipartite graph G (t) = <O (t) ,W (t) ,E (t) > to indicate; where O (t) is the set of all orders in time window t, W (t) is the set of all available drone workers in time window t, E (t) is the set of edges; 6) In time window t, assign algorithm A (t) In the bipartite graph G (t) Select allocation scheme M (t) ;M (t) It's G (t) A subset of the middle edge; 7) Actual order reward: M (t) is the matching result output by the platform, which includes several "order-drone" pairs. After receiving the order, drone w rushes to the designated location to execute order o. The actual order reward obtained after completing the order is r′ o,w : r′ o,w ＝α Δt ·r o Among them, r o is the initial order reward, α is the reward decay coefficient, which is a constant, and Δt is the total number of time windows from the order being released to the order being accepted, which consists of the allocation time and the time it takes for the drone to fly to the starting point; 8) Utility Rating: Allocation Scheme M (t) The utility score of is the sum of all matching actual order rewards in the scheme: U(M (t) )=∑ (o,w)∈M(t) r′ o,w (1) In T time windows, the total utility is the order allocation M made by all time windows <1,2,…,T> (t) The sum of the utilities of Optimization goal: Find the appropriate allocation solution M in each time window (t) , so that the total utility in T time windows is maximized.

3. The method for allocating materials delivery tasks of drone clusters as claimed in claim 1, characterized in that: In the value assessment model based on reinforcement learning, a single worker drone is modeled using the Markov decision process. In the time window t, the finite action set of a drone w is represented by Represents, including from the bipartite graph G (t) Select an order from the given candidate matches, or remain idle: Finite state set To express: The map is segmented into two ways: square and regular hexagon. The vertical height is segmented only by contour lines. S ,δ H They are the square and regular hexagonal space segmentation and positioning functions, which are used to map the position to a grid in the space; represents the position of UAV w in the current time window t; G (t) represents a bipartite graph; o represents a task order; A represents an action; {idle} represents an idle state.

4. The method for allocating materials delivery tasks of drone clusters as claimed in claim 3, characterized in that: Each drone is regarded as an intelligent agent. All agents share the same state value evaluation function. All drones evaluate the executed commands through the shared state value evaluation function.

5. The method for allocating materials delivery tasks of drone clusters as claimed in claim 4, characterized in that: The state value evaluation function V is learned through time difference learning TD Learning. Specifically, the online TD(0) learning method is adopted. According to the online TD(0) learning algorithm, if drone w selects an order o in the time window t decision, then the state value evaluation function V S and V H The following updates will be made: Where α is the learning rate, τ o is the estimated completion time of the order, γ∈[0,1] is the discount factor in reinforcement learning, and d o represents the position of the drone after moving, r′ o Indicates the actual reward value obtained by the task order.

6. The method for allocating materials delivery tasks of drone clusters as claimed in claim 2, characterized in that: The value assessment model based on reinforcement learning and the task allocation model based on bipartite graph matching are interactively executed in the decision-making process of each time window. The interactive process is as follows: Re-express the bipartite graph edge weights in the matching relation as: oh' o,w ←oh o,w +m o,w (14) Among them, ω o,w is the old edge weight, equal to the initial order reward; μ o,w The estimated value calculated for the valuation model.

7. A drone cluster material delivery task allocation system, characterized in that: The system is used to implement the drone cluster material delivery task allocation method as described in any one of claims 1-6.

Images