Joint resource allocation and trajectory optimization method for cooperation of unmanned aerial vehicle and vehicle

Abstract

The invention relates to an unmanned aerial vehicle and vehicle cooperation joint resource allocation and trajectory optimization method, and the method comprises the steps: obtaining a current network topological graph and a new customer distribution demand, employing a customer allocation algorithm, and achieving the optimal resource allocation based on the minimization of the number of vehicles; the method comprises the following steps: under a dual-objective optimization model which is constructed by taking the unmanned aerial vehicle flight distance as a target and maximizing the ratio of the unmanned aerial vehicle flight distance to the vehicle travel distance as a target, preliminarily updating a current network topological graph according to a new customer delivery demand to obtain an initial network topological graph, and optimizing the initial network topological graph by adopting a maximum iterative algorithm and reinforcement learning to obtain a new customer delivery demand; and obtaining an optimized network topological graph, and taking the optimized network topological graph as a current network topological graph, so that the unmanned aerial vehicle and the vehicle carry out distribution service according to the current network topological graph. By adopting the method, the service quality can be ensured while the distribution service can be completed by using the minimum service resource.

Description

Technical Field

[0001] The present application relates to the field of automated logistics technology, and in particular to a joint resource allocation and trajectory optimization method for collaboration between drones and vehicles. Background Art

[0002] In recent years, drones have played an important role in public safety, logistics and distribution, etc. Under the impact of this technological and industrial revolution, drones have brought significant development to the transportation network with their advantages of labor saving, light weight, flexible flight and environmental protection, so as to achieve energy conservation and emission reduction, cost-effectiveness and fast delivery of packages.

[0003] At present, there is an obvious mismatch between the exponential development of online shopping and the lag in the rapid delivery of data packets, which has become the biggest bottleneck restricting the development of e-commerce. In urban distribution business, the "last mile" is the main body of vehicle dispatch. Whether it can meet the personalized needs of customers depends on the quality and efficiency of trajectory optimization and resource allocation. UAV and vehicle collaborative services have outstanding advantages over vehicle services. For example, UAVs are not affected by complex ground conditions, such as road congestion. Ground-to-air collaborative delivery can significantly reduce the cost of service providers and the time of packet delivery. Summary of the invention

[0004] Based on this, it is necessary to provide a joint resource allocation and trajectory optimization method for UAV-vehicle collaboration that can reduce service costs while ensuring service quality in response to the above technical problems.

[0005] A joint resource allocation and trajectory optimization method for unmanned aerial vehicle and vehicle collaboration, the method comprising:

[0006] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0007] Based on the dual-objective optimization model constructed with the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle, the dual-objective optimization model is converted into a single-objective optimization model, and under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the delivery demand of the new customer to obtain an initial network topology map;

[0008] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0009] In one embodiment, the current network topology diagram includes multiple trips, each of which includes a ground warehouse node, a take-off node and a landing node of a drone, and different customer nodes. In each of the trips:

[0010] The vehicle carrying the drone and the package to be delivered departs from the ground warehouse node. After arriving at the drone take-off node, the drone and the vehicle go to different customer nodes in the itinerary for delivery services. After completing the delivery service, the drone lands on the vehicle to charge and prepare for the next delivery service or the vehicle carries the drone back to the ground warehouse node.

[0011] In one embodiment, the new customer delivery demand includes a delivery destination and a delivery arrival time;

[0012] When the current network topology map is updated and optimized, the delivery destination is added to the current network topology map as a new customer node, and a vehicle or drone is selected for delivery service.

[0013] In one embodiment, when the delivery destination is added as a new customer node to the current network topology graph, a customer allocation algorithm is used to solve the problem under six constraints, wherein the six constraints include:

[0014] The first constraint: at least one vehicle equipped with a drone provides delivery service to the customer node;

[0015] The second constraint: During the delivery service of a drone or vehicle to a customer node, each customer node can only be served by a vehicle or drone;

[0016] The third constraint: the sum of all packages to be distributed cannot exceed the maximum load of the vehicle;

[0017] The fourth constraint: the mutual waiting time between the matched UAV and vehicle at the landing node is less than or equal to the maximum waiting time;

[0018] The fifth constraint: the weight of the package that the drone needs to carry is less than the weight of its maximum payload;

[0019] The sixth constraint: When the drone is completing a service, its energy at the take-off node should be sufficient to support it to complete the service and return to the landing point.

[0020] In one embodiment, when the delivery destination is added as a new customer node to the current network topology map:

[0021] For each trip, determine whether the new client node in the current network topology graph satisfies the first constraint, the second constraint and the third constraint, and put the client nodes that satisfy the three constraints into the to-be-added set;

[0022] When the weight of the package at a certain node is less than the maximum payload of the vehicle, a customer node is randomly selected from the set to be added as the next node to be added to the current trip, and it is determined whether the customer node satisfies the fourth constraint, the fifth constraint and the sixth constraint.

[0023] In one embodiment, when the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology graph:

[0024] The initial network topology graph is optimized by combining reinforcement learning with an improved controller, a disturbance controller and a threshold controller, wherein the improved controller and the disturbance controller correspond to an improved operator and a disturbance operator respectively, and the threshold controller is used to select whether to use the improved or the disturbance controller;

[0025] Through reinforcement learning, operators are mapped to actions, different actions are weighted in continuous iterations, different operators are selected according to different node sequences in continuous training, and the trajectory and resource allocation are continuously optimized in a cycle.

[0026] The present application also proposes a joint resource allocation and trajectory optimization device for cooperation between a drone and a vehicle, the device comprising:

[0027] A data acquisition module, used to acquire a current network topology diagram and new customer delivery requirements, wherein the current network topology diagram includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0028] An initial network topology map obtaining module is used to convert a dual-objective optimization model constructed based on the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle into a single-objective optimization model, and to preliminarily update the current network topology map according to the delivery demand of the new customer using a customer allocation algorithm under the single-objective optimization model to obtain an initial network topology map;

[0029] The network topology optimization module is used to optimize the initial network topology based on the single-objective optimization model by using the maximum iteration algorithm and reinforcement learning to obtain an optimized network topology, and use the optimized network topology as the current network topology so that drones and vehicles can perform delivery services according to the current network topology.

[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0032] Based on the dual-objective optimization model constructed with the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle, the dual-objective optimization model is converted into a single-objective optimization model, and under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the delivery demand of the new customer to obtain an initial network topology map;

[0033] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0035] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0036] Based on the dual-objective optimization model constructed with the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle, the dual-objective optimization model is converted into a single-objective optimization model, and under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the delivery demand of the new customer to obtain an initial network topology map;

[0037] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0038] The above-mentioned joint resource allocation and trajectory optimization method for cooperation between drones and vehicles obtains the current network topology map and the new customer delivery demand, adopts the customer allocation algorithm, and based on the dual-objective optimization model constructed with the goal of minimizing the number of vehicles and maximizing the ratio of the drone flight distance to the vehicle travel distance, preliminarily updates the current network topology map according to the new customer delivery demand to obtain the initial network topology map, optimizes the initial network topology map using the maximum iteration algorithm and reinforcement learning, obtains the optimized network topology map, and uses the optimized network topology map as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map. The use of this method can complete the delivery service with the minimum service resources while ensuring the service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a joint resource allocation and trajectory optimization method for cooperation between a UAV and a vehicle in one embodiment;

[0040] Figure 2 A schematic diagram of drone and vehicle trajectory scheduling and resource allocation in one embodiment;

[0041] Figure 3 A schematic diagram of cooperation between a drone and a vehicle during a trip in one embodiment;

[0042] Figure 4 A flowchart of an initial solution optimization process based on reinforcement learning in one embodiment;

[0043] Figure 5 A structural block diagram of a joint resource allocation and trajectory optimization device for cooperation between a UAV and a vehicle in one embodiment;

[0044] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] Existing research on UAV and vehicle collaboration focuses on the trajectory optimization of UAVs, but there are still challenges in ensuring service reliability by obtaining a large amount of ground and air information, such as the effectiveness of collaborative services, the complexity of collaborative resource allocation, and the reliability of collaborative services.

[0047] Specifically, the effectiveness of collaborative services refers to the fact that in the ground-air collaborative delivery network, the acquisition and coordination of a large amount of ground-air environment information imposes a huge burden on resource allocation decisions. It requires status information of drones and vehicles, including their locations, service capabilities, and waiting times. How to coordinate the service process of drones and vehicles based on a large amount of information and ensure the effectiveness of collaborative services is challenging.

[0048] Specifically, the complexity of collaborative resource allocation means that, as an extension of CVRP (Capacitated Vehicle Routing Problem), the collaborative service problem of drones and vehicles is NP-hard (Non-deterministic Polynomial-time hard). The trade-offs between multiple conflicting objectives are difficult to solve using traditional optimization techniques such as dynamic programming. Although the rich operators in traditional algorithms have a good effect on improving feasible solutions, the process of exploring the search space is relatively difficult.

[0049] Specifically, the reliability of collaborative services refers to the reliability of resource allocation in order to ensure the quality of service (QoS) for customers and maximize the profits of service providers. It is generally affected by three aspects. In the service preparation stage, whether the drone or vehicle has the ability to serve customers. In the service stage, whether the landing node is within the flight range of the drone and whether the drone has enough energy to support its return. At the end of a service, whether the vehicle is within the landing point range of the drone and whether this operation can support the next customer served by the drone. The reliability of resource allocation is particularly important for logistics efficiency and customer service satisfaction.

[0050] In view of the challenges of the above UAV and vehicle collaboration problems, in this application, Figure 1 As shown, a joint resource allocation and trajectory optimization method for UAV and vehicle collaboration is provided, which specifically includes the following steps:

[0051] Step S100, obtain the current network topology and new customer delivery requirements, where the current network topology includes multiple nodes and edges connecting the nodes, where each node includes a ground warehouse node, multiple customer nodes, and a take-off node and a charging node for a drone, and the edges between the nodes represent drone routes or vehicle routes.

[0052] Step S110, based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, it is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery needs to obtain an initial network topology map.

[0053] Step S120, based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0054] In this application, a multi-UAV and multi-vehicle collaborative framework is proposed, which focuses on resource allocation and trajectory optimization of UAVs and vehicles. Vehicles serve as take-off and landing platforms for UAVs and cooperate to complete package delivery services, thereby achieving much-needed energy conservation and reducing service costs.

[0055] In step S100, the current network topology diagram includes multiple planned trips based on the current time point, each trip includes a ground warehouse node, a take-off node and a landing node of a drone, and different customer nodes. In each trip: the vehicle carrying the drone and the package to be delivered departs from the ground warehouse node, and after arriving at the drone take-off node, the drone and the vehicle go to different customer nodes in the trip for delivery services. After completing the delivery service, the drone lands on the vehicle to charge, prepares for the next delivery service, or the vehicle carries the drone back to the ground warehouse node.

[0056] Specifically, in the current network topology, different vehicles are planned to carry drones and packages to be delivered from ground warehouse nodes to provide services to customers. The vehicle route is the primary route, while the UAV flight route is the secondary route. When the drone follows the aircraft to the take-off node, it takes off and provides services to the designated customer, and then returns to the landing node to charge or continue its next service. After the vehicle launches the drone, it can serve other customers or directly reach the landing node of the drone. When the drone completes the package delivery, there are two types of landing nodes. One is the node served by the vehicle, and the other is the range within which the vehicle can reach the flight range of the drone. The location of these nodes depends on the relevant vehicle and UAV status information in the current network topology.

[0057] like Figure 2As shown, trip 1 consists of nodes 1 to 16, and trip 2 consists of nodes 17 to 20. For example, the vehicle transports the drone from the ground warehouse node to node 1, which is the take-off node of the drone. After serving node 3, the drone returns to node 2 for landing. The vehicle launches the drone from node 1 and the aircraft to node 2. After serving each service node in sequence, the drone and the vehicle return to the warehouse.

[0058] Specifically, multiple service nodes that need to deliver packages are taken as a service node set, denoted as C = {1, ..., c}, and the package set to be delivered is denoted as G = {G1, ..., Gc}. The ground warehouse node is copied into two duplicate nodes (0 and c+1) as the starting node and return node of a trip. The set Nd = {0, 1, ..., c} is used to represent the set of starting nodes and customer nodes, which is equivalent to the possible departure node set of vehicles or drones, and Nr = {1, ..., c, c+1} represents the set of return nodes and customer nodes.

[0059] Furthermore, a trip consists of a drone and a number of vehicles. In the network topology, multiple trips are allocated to serve all customer nodes. The process of a vehicle carrying a drone departing from the warehouse to serve a customer node and returning to the warehouse is defined as a trip, denoted by s∈S. Multiple trips in the network topology are denoted by a set S, and Φ(s) represents the set of nodes on the trip s.<i,j> is the distance between nodes i and j, e<i,j,k> is the total distance that the drone takes off from node i, serves node j and returns to node k. Each trip starts from node 0 and returns to node c+1. i,k (s) indicates whether the vehicle is transferred from node i to node k. i,j,k (s) indicates whether the drone takes off from node i to serve node j and returns to node k in trip s, such as Figure 3 shown.

[0060] Furthermore, for each UAV, κ is the maximum load, Indicates its maximum flight range. After delivering the package, the drone can land on the vehicle and charge using the onboard charger.

[0061] In this embodiment, the new customer delivery demand includes the delivery destination and the delivery arrival time. When the current network topology map is updated and optimized, the delivery destination is added to the current network topology map as a new customer node, and the delivery service is selected by vehicle or drone.

[0062] Furthermore, in the current network topology and the updated network topology, the route distance and carrying capacity between nodes, the energy consumption of the drone, and the reliability of the drone can all be calculated and obtained according to existing relevant methods.

[0063] In step S110, with the goal of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, a customer allocation algorithm is used to add the delivery destination as a new customer node to the current network topology map.

[0064] Specifically, the first objective of the dual-objective model is to minimize the number of vehicles in the entire network topology, denoted by Nt, while the second objective is to maximize the flight distance Φ of the UAV. u Distance from vehicle Φ v The ratio value is expressed as:

[0065] minNt(1)

[0066]

[0067] According to the dual-objective model expressed by formula (1) and formula (2), it can be noted that the total distance of the route is equal to the sum of the travel distance of each trip, and the distance of a single trip is composed of the distance traveled by the drone and the vehicle. Therefore, the optimization of the total trip distance can be converted into the optimization of the single trip distance. The relaxed single-objective problem is compared with the customer allocation algorithm for reliable delivery and converted into a traditional NP-hard problem. The dual-objective problem (i.e., minimizing the number of aircraft and maximizing the ratio of the drone flight distance to the aircraft travel distance) is equivalent to minimizing the network overhead, which converts the dual-objective problem into a simpler single-objective problem.

[0068] Therefore, the optimization of the total travel distance can be converted into the optimization of a single travel distance, that is, the dual-objective model is converted into a single-objective model and expressed as:

[0069]

[0070] In formula (3), Φ v (s) / Φ u (s) is the distance Φ traveled by a vehicle in a trip v Flight distance from the droneΦ u The ratio value of . Among them, the variable δ i,k (s) indicates whether the vehicle is transferred from node i to node k. i,j,k (s) indicates whether the drone takes off from node i to serve node j and returns to node k in trip s. Both variables are binary variables. They can be used to determine which route the drones and vehicles are assigned to and which client node they can serve.

[0071] In this embodiment, after the dual-objective model is converted into a single-objective model, six constraints are proposed in the process of solving the single-objective model, i.e., formula (3). That is, under the six constraints, the new customer node is added to the current network topology diagram while satisfying formula (3).

[0072] Specifically, the six constraints include: the first constraint: at least one vehicle carrying a drone provides delivery service for the customer node, the second constraint: during the delivery service provided by the drone or vehicle to the customer node, each customer node can only be served by a vehicle or a drone, the third constraint: the sum of all packages to be distributed cannot exceed the maximum load of the vehicle, the fourth constraint, the mutual waiting time of the matched drone and vehicle at the landing node is less than or equal to the maximum waiting time, the fifth constraint: the weight of the package that the drone needs to carry is less than the weight of the package of its maximum payload and the sixth constraint: when the drone is completing a service, its energy at the take-off node should be sufficient to support it to complete the service and return to the landing point.

[0073] Furthermore, the above six constraints can be expressed as formula (4) to formula (9) respectively:

[0074]

[0075] i,j,k∈Φ(s)

[0076]

[0077] Among them, i,j,k,l,m,n∈Φ(s),s∈S

[0078] In formula (4) to formula (9), Nt represents the number of vehicles, s represents a trip, S represents all trips in the network topology, and N d and N r Respectively represent the starting node set and the ending node set in each trip, C represents the nodes that need to be served, that is, the customer node set, and the variable δ i,k (s) indicates whether the vehicle is transferred from node i to node k, and the variable δ' i,j,k (s) indicates whether the drone takes off from node i to serve node j and returns to node k in trip s. Φ(s) indicates the set of nodes on trip s. C indicates the set of nodes that need to be served, i.e., the set of client nodes. g j Indicates the weight of the package. represents the maximum payload of the vehicle, t' k (s) represents the landing time of the UAV at node k, t k (s) represents the arrival time of the vehicle at node k, represents the maximum delay time, κ represents the maximum payload of the UAV, It represents the maximum flight distance of the UAV starting from node i and serving node j and returning to node k, and ξ represents the power consumption of the UAV in one service.

[0079] In this embodiment, when the delivery destination is added as a new customer node to the current network topology map: for each trip, determine whether the new customer node in the current network topology map satisfies the first constraint, the second constraint and the third constraint, and put the customer nodes that satisfy these three constraints into the to-be-added set; when the package weight at a certain node j is less than the maximum payload of the vehicle, randomly select a customer node from the to-be-added set as the next node to be added to the current trip, and determine whether the customer node satisfies the fourth constraint, the fifth constraint and the sixth constraint.

[0080] Furthermore, in this application, a method for evaluating the service capability of the drone in the initial network topology diagram is also proposed. Since the drone is affected by energy consumption, flight distance, payload and other factors during the delivery of packages, in order to evaluate the reliability of its service, the following formula is used to evaluate its reliability:

[0081]

[0082] In formula (10), α, β, represents the coefficient, It represents the ratio between service energy consumption and take-off power. It represents the ratio between the flight distance in a trip and the maximum flight distance. Indicates the ratio between the weight of the package to be delivered and the maximum payload of the drone.

[0083] Since the initial solution generated by the customer allocation algorithm in step S110 can satisfy all constraints of the optimization problem, but since it is a randomly composed sequence, it is very likely that it cannot achieve the optimal goal. Therefore, it is necessary to design a solution to change the order of the multi-trip sequence.

[0084] In step S120, a method for optimizing the initial network topology map using the maximum iteration algorithm and reinforcement learning is proposed, including optimizing the initial network topology map through an improved controller, a perturbation controller and a threshold controller combined with reinforcement learning, wherein the improved controller and the perturbation controller correspond to the improved operator and the perturbation operator respectively, and the threshold controller is used to select whether to use the improved or perturbation controller, and the operator is mapped to an action through reinforcement learning, and different actions are weighted in continuous iterations. Different operators are selected according to different node sequences in continuous training, and the trajectory and resource allocation are continuously optimized in a cycle.

[0085] In this embodiment, optimization within a trip or optimization between trips is performed through each trip in the initial network topology diagram of the controller and the disturbance controller.

[0086] Further, the improved controller uses the improved operator to optimize within each trip and between trips. When optimizing within a trip, the optimization within the trip can be performed by the deletion operator, the symmetric exchange operator and the insertion operator. Among them, the deletion operator is used to delete the edge between two nodes and reconnect, the symmetric exchange operator is used to symmetrically exchange the positions between two nodes in a trip, and the first insertion operator is used to insert a node in the trip to the new position. When optimizing between trips, the crossover operator, the reverse crossover operator, the symmetric exchange operator, the asymmetric exchange operator, the second insertion operator and the periodic exchange operator can be used. Among them, the crossover operator is used to exchange the trip segments in two trips, the reverse crossover operator is used to reverse a trip once, and then exchange the trip segments in the two trips, the symmetric exchange operator is used to symmetrically exchange the trip segments of preset lengths between the two trips, the asymmetric exchange operator is used to exchange two trip segments of different lengths in two trips, the second insertion operator is used to insert a trip segment in one trip into another trip, and the periodic exchange operator is used to cyclically exchange a node in three trips.

[0087] Furthermore, the disturbance controller uses disturbance operators to optimize between trips, including random reconstruction operators, random exchange operators, and periodic exchange operators, to optimize between two trips. Among them, the random reconstruction operator is used to randomly select a segment of the trip, disturb the order of the reorganization action, the random exchange operator is used to randomly exchange several pairs of adjacent nodes, and the periodic exchange operator is used to periodically exchange nodes in multiple trips.

[0088] In this embodiment, different operators are selected through a threshold controller based on reinforcement learning to optimize the initial solution. The purpose of action selection in this reinforcement learning is to select the most appropriate operator for each input path. By inputting the current state, reinforcement learning uses a policy gradient method to train the network weights to select operators.

[0089] Furthermore, each node served by a drone or vehicle in each trip has a state. Each state consists of three parts: relevant information of the current node, specific information of the current solution, and historical information. The relevant information of the current node includes: the position of node i in the trip, the position of the previous node i-, the position of the subsequent node i+, and the distance between the above nodes. The specific information of the current solution includes: the demand Gi of node i, the remaining capacity of the vehicle serving trip s, and the service reliability of the drone at node i. The historical information includes: the operator selected by the node in the previous h steps under this state and the corresponding reward.

[0090] Furthermore, the action selection in each round is determined by reinforcement learning. The action selection decision in each round is decomposed into selecting an operator as an action from all operators in each iteration.

[0091] Furthermore, reward refers to the design reward function, where A(t) is the advantage function, which represents the gap between the current optimization target value and the initial solution value generated by CARD. The optimization goal of reinforcement learning is to minimize the ratio in formula (3).

[0092] Specifically, in the network strategy, the policy gradient can be directly applied to reinforcement learning in high-dimensional or continuous action spaces. Since the perturbation operator can reconstruct the current solution and make it jump out of the local optimum, the Monte Carlo-based policy gradient method is selected to update the policy gradient. The solution and problem information in the state are the input of the policy network, and the output is the probability of choosing a certain action. After passing through the policy network, the threshold controller selects the operator to change the solution and continuously reach the optimum.

[0093] like Figure 4 As shown, it is a whole flow chart of optimizing the initial network topology map by using the maximum iteration algorithm and reinforcement learning.

[0094] In the above-mentioned joint resource allocation and trajectory optimization method for cooperation between drones and vehicles, by obtaining the current network topology map and the new customer delivery demand, the customer allocation algorithm is adopted, and based on the dual-objective optimization model constructed with the goal of minimizing the number of vehicles and maximizing the ratio of the drone flight distance to the vehicle travel distance, the current network topology map is preliminarily updated according to the new customer delivery demand to obtain the initial network topology map, and the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain the optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles can perform delivery services according to the current network topology map. The use of this method can complete the delivery service with the minimum service resources while ensuring the service quality.

[0095] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In one embodiment, Figure 5 As shown, a joint resource allocation and trajectory optimization device for cooperation between a UAV and a vehicle is provided, comprising: a data acquisition module 200, an initial network topology map acquisition module 210 and a network topology map optimization module 220, wherein:

[0097] The data acquisition module 200 is used to acquire the current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0098] The initial network topology map obtaining module 210 is used to convert the dual-objective optimization model constructed based on the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle into a single-objective optimization model, and to preliminarily update the current network topology map according to the delivery demand of the new customer using the customer allocation algorithm under the single-objective optimization model to obtain the initial network topology map;

[0099] The network topology map optimization module 220 is used to optimize the initial network topology map based on the single-objective optimization model, using the maximum iteration algorithm and reinforcement learning to obtain an optimized network topology map, and use the optimized network topology map as the current network topology map, so that drones and vehicles can perform delivery services according to the current network topology map.

[0100] For the specific definition of the joint resource allocation and trajectory optimization device for cooperation between drones and vehicles, please refer to the definition of the joint resource allocation and trajectory optimization method for cooperation between drones and vehicles above, which will not be repeated here. Each module in the above-mentioned joint resource allocation and trajectory optimization device for cooperation between drones and vehicles can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0101] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a joint resource allocation and trajectory optimization method for cooperation between unmanned aerial vehicles and vehicles is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0102] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0104] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0105] Based on the dual-objective optimization model constructed with the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle, the dual-objective optimization model is converted into a single-objective optimization model, and under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the delivery demand of the new customer to obtain an initial network topology map;

[0106] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0107] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0108] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route;

[0109] Based on the dual-objective optimization model constructed with the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle, the dual-objective optimization model is converted into a single-objective optimization model, and under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the delivery demand of the new customer to obtain an initial network topology map;

[0110] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A joint resource allocation and trajectory optimization method for UAV and vehicle collaboration, characterized in that: The method comprises: Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route; Based on the dual-objective optimization model constructed with the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle, the dual-objective optimization model is converted into a single-objective optimization model, and under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the delivery demand of the new customer to obtain an initial network topology map; Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map, and the optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

2. The joint resource allocation and trajectory optimization method for UAV-vehicle collaboration according to claim 1 is characterized in that: The current network topology diagram includes multiple itineraries, each of which includes a ground warehouse node, a take-off node and a landing node of a drone, and different customer nodes. In each of the itineraries: The vehicle carrying the drone and the package to be delivered departs from the ground warehouse node. After arriving at the drone take-off node, the drone and the vehicle go to different customer nodes in the itinerary for delivery services. After completing the delivery service, the drone lands on the vehicle to charge and prepare for the next delivery service or the vehicle carries the drone back to the ground warehouse node.

3. The joint resource allocation and trajectory optimization method for UAV-vehicle collaboration according to claim 2 is characterized in that: The new customer delivery requirements include the delivery destination and delivery arrival time; When the current network topology map is updated and optimized, the delivery destination is added to the current network topology map as a new customer node, and a vehicle or drone is selected for delivery service.

4. The method for joint resource allocation and trajectory optimization of UAV and vehicle collaboration according to claim 3, characterized in that: When the delivery destination is added as a new customer node to the current network topology graph, a customer allocation algorithm is used to solve the problem under six constraints, wherein the six constraints include: The first constraint: at least one vehicle equipped with a drone provides delivery service to the customer node; The second constraint: During the delivery service of a drone or vehicle to a customer node, each customer node can only be served by a vehicle or drone; The third constraint: the sum of all packages to be distributed cannot exceed the maximum load of the vehicle; The fourth constraint: the mutual waiting time between the matched UAV and vehicle at the landing node is less than or equal to the maximum waiting time; The fifth constraint: the weight of the package that the drone needs to carry is less than the weight of its maximum payload; The sixth constraint: When the drone is completing a service, its energy at the take-off node should be sufficient to support it to complete the service and return to the landing point.

5. The method for joint resource allocation and trajectory optimization of UAV and vehicle collaboration according to claim 4, characterized in that: When the delivery destination is added as a new customer node to the current network topology map: For each trip, determine whether the new client node in the current network topology graph satisfies the first constraint, the second constraint and the third constraint, and put the client nodes that satisfy the three constraints into the to-be-added set; When the weight of the package at a certain node is less than the maximum payload of the vehicle, a customer node is randomly selected from the set to be added as the next node to be added to the current trip, and it is determined whether the customer node satisfies the fourth constraint, the fifth constraint and the sixth constraint.

6. The method for joint resource allocation and trajectory optimization of UAV and vehicle collaboration according to claim 5, characterized in that: When the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology graph: The initial network topology graph is optimized by combining reinforcement learning with an improved controller, a disturbance controller and a threshold controller, wherein the improved controller and the disturbance controller correspond to an improved operator and a disturbance operator respectively, and the threshold controller is used to select whether to use the improved or the disturbance controller; Through reinforcement learning, operators are mapped to actions, different actions are weighted in continuous iterations, different operators are selected according to different node sequences in continuous training, and the trajectory and resource allocation are continuously optimized in a cycle.

7. A joint resource allocation and trajectory optimization device for cooperation between drones and vehicles, characterized in that: The device comprises: A data acquisition module, used to acquire a current network topology diagram and new customer delivery requirements, wherein the current network topology diagram includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a take-off node and a charging node of a drone, and the edges between the nodes represent a drone route or a vehicle route; An initial network topology map obtaining module is used to convert a dual-objective optimization model constructed based on the objectives of minimizing the number of vehicles and maximizing the ratio of the flight distance of the drone to the travel distance of the vehicle into a single-objective optimization model, and to preliminarily update the current network topology map according to the delivery demand of the new customer using a customer allocation algorithm under the single-objective optimization model to obtain an initial network topology map; The network topology optimization module is used to optimize the initial network topology based on the single-objective optimization model by using the maximum iteration algorithm and reinforcement learning to obtain an optimized network topology, and use the optimized network topology as the current network topology so that drones and vehicles can perform delivery services according to the current network topology.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images