Joint resource allocation and trajectory optimization method for unmanned aerial vehicle and vehicle cooperation

Abstract

The application relates to a method for jointly allocating resources and optimizing trajectories of unmanned aerial vehicles and vehicles, which comprises the following steps: acquiring a current network topology graph and new customer distribution requirements; adopting a customer allocation algorithm; constructing a double-target optimization model based on the minimum number of vehicles and the maximum ratio of the flight distance of the unmanned aerial vehicle to the travel distance of the vehicle; preliminarily updating the current network topology graph according to the new customer distribution requirements to obtain an initial network topology graph; optimizing the initial network topology graph by adopting a maximum iteration algorithm and reinforcement learning to obtain an optimized network topology graph; and taking the optimized network topology graph as the current network topology graph, so that the unmanned aerial vehicle and the vehicle can perform distribution services according to the current network topology graph. By adopting the method, the distribution services can be completed by using the minimum service resources while the service quality is ensured.

Description

Technical Field

[0001] The present application relates to the field of automated logistics technology, and in particular to a joint resource allocation and trajectory optimization method for collaboration between drones and vehicles. Background Art

[0002] In recent years, drones have played a vital role in public safety, logistics, and distribution. Driven by this technological and industrial revolution, drones, with their labor-saving, lightweight, flexible, and environmentally friendly advantages, have significantly advanced transportation networks, achieving energy conservation and emission reduction, cost-effectiveness, and rapid package delivery.

[0003] Currently, there's a significant mismatch between the exponential growth of online shopping and the lag in rapid data packet delivery, which has become the biggest bottleneck hindering the development of e-commerce. In urban delivery, the "last mile" is the primary driver of vehicle dispatch. Whether it can meet personalized customer needs depends on the quality and efficiency of trajectory optimization and resource allocation. Drone-vehicle collaborative services offer significant advantages over vehicle-based services. For example, UAVs are unaffected by complex ground conditions, such as congested roads. Ground-to-air collaborative delivery can significantly reduce service provider costs and packet delivery times. Summary of the Invention

[0004] Based on this, it is necessary to provide a joint resource allocation and trajectory optimization method for UAV-vehicle collaboration that can reduce service costs while ensuring service quality to address the above technical problems.

[0005] A joint resource allocation and trajectory optimization method for UAV-vehicle collaboration, the method comprising:

[0006] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0007] Based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, the model is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery demand to obtain an initial network topology map;

[0008] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map. The optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0009] In one embodiment, the current network topology includes multiple itineraries, each of which includes a ground warehouse node, a takeoff node and a landing node of a drone, and different customer nodes. In each of the itineraries:

[0010] The vehicle carrying the drone and the package to be delivered departs from the ground warehouse node. After arriving at the drone's take-off node, the drone and vehicle go to different customer nodes in the itinerary to provide delivery services. After completing the delivery service, the drone lands on the vehicle to charge and prepare for the next delivery service, or the vehicle carries the drone back to the ground warehouse node.

[0011] In one embodiment, the new customer delivery requirement includes a delivery destination and a delivery arrival time;

[0012] When the current network topology is updated and optimized, the delivery destination is added to the current network topology as a new customer node, and a vehicle or drone is selected for delivery service.

[0013] In one embodiment, when the delivery destination is added as a new customer node to the current network topology, a customer allocation algorithm is used to solve the problem under six constraints, wherein the six constraints include:

[0014] The first constraint: at least one vehicle equipped with a drone provides delivery service to the customer node;

[0015] The second constraint: During the delivery service of a drone or vehicle to a customer node, each customer node can only be served by a vehicle or drone;

[0016] The third constraint: the sum of all packages to be distributed cannot exceed the maximum load of the vehicle;

[0017] The fourth constraint: the mutual waiting time between the matched UAV and vehicle at the landing node is less than or equal to the maximum waiting time;

[0018] The fifth constraint: the weight of the package carried by the drone must be less than the weight of its maximum payload;

[0019] The sixth constraint: When a drone is completing a service, its energy at the takeoff node should be sufficient to support it to complete the service and return to the landing point.

[0020] In one embodiment, when the delivery destination is added as a new customer node to the current network topology map:

[0021] For each trip, determining whether a new client node in the current network topology satisfies the first constraint, the second constraint, and the third constraint, and placing the client node that satisfies the three constraints into a to-be-added set;

[0022] When the package weight at a certain node is less than the maximum payload of the vehicle, a customer node is randomly selected from the to-be-added set as the next node to be added to the current trip, and it is determined whether the customer node satisfies the fourth constraint, the fifth constraint, and the sixth constraint.

[0023] In one embodiment, when the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology graph:

[0024] The initial network topology graph is optimized by combining reinforcement learning with an improvement controller, a disturbance controller, and a threshold controller, wherein the improvement controller and the disturbance controller correspond to the improvement operator and the disturbance operator respectively, and the threshold controller is used to select whether to use the improvement controller or the disturbance controller;

[0025] Through reinforcement learning, operators are mapped to actions, different actions are assigned weights in continuous iterations, different operators are selected according to different node sequences in continuous training, and the trajectory and resource allocation are continuously optimized in a cycle.

[0026] This application also proposes a joint resource allocation and trajectory optimization device for UAV and vehicle collaboration, the device comprising:

[0027] A data acquisition module, configured to obtain a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0028] an initial network topology map obtaining module, configured to convert a dual-objective optimization model constructed based on the objectives of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance into a single-objective optimization model, and to perform a preliminary update of the current network topology map based on the delivery requirements of the new customers using a customer allocation algorithm under the single-objective optimization model to obtain an initial network topology map;

[0029] The network topology optimization module is used to optimize the initial network topology based on the single-objective optimization model using the maximum iteration algorithm and reinforcement learning to obtain an optimized network topology, and use the optimized network topology as the current network topology so that drones and vehicles can perform delivery services according to the current network topology.

[0030] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0032] Based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, the model is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery demand to obtain an initial network topology map;

[0033] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map. The optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0035] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0036] Based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, the model is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery demand to obtain an initial network topology map;

[0037] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map. The optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0038] The joint resource allocation and trajectory optimization method for drone-vehicle collaboration obtains the current network topology and new customer delivery demands. Using a customer allocation algorithm, based on a dual-objective optimization model constructed to minimize the number of vehicles and maximize the ratio of drone flight distance to vehicle travel distance, the method performs a preliminary update of the current network topology based on the new customer delivery demands, obtaining an initial network topology. This initial network topology is then optimized using a maximum iteration algorithm and reinforcement learning to obtain an optimized network topology. This optimized network topology serves as the current network topology, enabling drones and vehicles to perform delivery services based on the current network topology. This method can achieve delivery services while ensuring service quality while utilizing minimal service resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 1 is a flow chart of a method for joint resource allocation and trajectory optimization for collaboration between a UAV and a vehicle in one embodiment;

[0040] Figure 2 A schematic diagram of drone and vehicle trajectory scheduling and resource allocation in one embodiment;

[0041] Figure 3 is a schematic diagram of cooperation between a drone and a vehicle during a trip in one embodiment;

[0042] Figure 4 A flowchart of an initial solution optimization process based on reinforcement learning in one embodiment is provided;

[0043] Figure 5 A block diagram of a joint resource allocation and trajectory optimization device for collaboration between a UAV and a vehicle in one embodiment;

[0044] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] Existing research on UAV and vehicle collaboration focuses on UAV trajectory optimization, but there are still challenges in ensuring service reliability by obtaining a large amount of ground and air information, such as the effectiveness of collaborative services, the complexity of collaborative resource allocation, and the reliability of collaborative services.

[0047] Specifically, the effectiveness of collaborative services refers to the significant burden placed on resource allocation decisions by acquiring and coordinating a vast amount of ground-air environment information within a ground-air collaborative delivery network. This requires state information on drones and vehicles, including their locations, service capabilities, and wait times. Coordinating the service processes of drones and vehicles based on this vast amount of information and ensuring the effectiveness of collaborative services is challenging.

[0048] Specifically, the complexity of collaborative resource allocation refers to the fact that, as an extension of the Capacitated Vehicle Routing Problem (CVRP), the collaborative service problem of drones and vehicles is NP-hard (Non-deterministic Polynomial-time hard). The trade-offs between multiple conflicting objectives are difficult to resolve using traditional optimization techniques such as dynamic programming. Although the rich operators in traditional algorithms are effective in improving feasible solutions, exploring the search space is relatively difficult.

[0049] Specifically, the reliability of collaborative services refers to the reliability of resource allocation to ensure customer quality of service (QoS) and maximize service provider profits. This reliability is generally influenced by three factors: During the service preparation phase, the drone or vehicle must be capable of serving the customer. During the service phase, the landing node must be within the drone's flight range and the drone must have sufficient energy to return. At the end of a service session, the vehicle must be within range of the drone's landing point and the operation must be able to support the next customer served by the drone. The reliability of resource allocation is particularly important for logistics efficiency and customer service satisfaction.

[0050] In response to the challenges of the above drone and vehicle collaboration issues, in this application, Figure 1 As shown, a joint resource allocation and trajectory optimization method for UAV and vehicle collaboration is provided, which specifically includes the following steps:

[0051] Step S100, obtain the current network topology and new customer delivery requirements, the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a take-off node and a charging node for the drone, and the edges between the nodes represent drone routes or vehicle routes.

[0052] Step S110, based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, it is converted into a single-objective optimization model. Under this single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery needs to obtain an initial network topology map.

[0053] In step S120, based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology to obtain an optimized network topology. The optimized network topology is used as the current network topology, so that drones and vehicles can provide delivery services according to the current network topology.

[0054] In this application, a collaborative framework for multiple drones and vehicles is proposed, focusing on resource allocation and trajectory optimization for drones and vehicles. Vehicles serve as takeoff and landing platforms for drones, collaborating to complete package delivery services, thereby achieving much-needed energy savings and reducing service costs.

[0055] In step S100, the current network topology includes multiple planned itineraries based on the current time point. Each itinerary includes a ground warehouse node, a drone's takeoff and landing nodes, and various customer nodes. In each itinerary, a vehicle carrying a drone and a package departs from a ground warehouse node. Upon arriving at the drone's takeoff node, the drone and vehicle proceed to different customer nodes within the itinerary to perform delivery services. Upon completion of the delivery service, the drone lands on the vehicle to recharge and prepare for the next delivery, or the vehicle, carrying the drone, returns to the ground warehouse node.

[0056] Specifically, in the current network topology, different vehicles are planned to transport drones and packages to be delivered from ground warehouse nodes to provide customer service. The vehicle route is the primary route, while the UAV flight route is the secondary route. When the drone follows the aircraft to the takeoff node, it takes off and provides service to the designated customer, then returns to the landing node to recharge or continue its next service. After the vehicle launches the drone, it can serve other customers or directly return to the drone's landing node. After completing a package delivery, the drone has two landing nodes: one is the node served by the vehicle, and the other is the range within which the vehicle can reach the drone. The location of these nodes depends on the relevant vehicle and UAV status information in the current network topology.

[0057] like Figure 2As shown, Trip 1 consists of nodes 1 to 16, and Trip 2 consists of nodes 17 to 20. For example, a vehicle transports a drone from a ground warehouse node to Node 1, the drone's takeoff node. After servicing Node 3, the drone returns to Node 2 for landing. The vehicle launches the drone from Node 1 and the aircraft to Node 2. After servicing each service node in sequence, the drone and vehicle return to the warehouse.

[0058] Specifically, multiple service nodes that need to deliver packages are taken as a service node set, denoted as C = {1, ..., c}, and the set of packages to be delivered is denoted as G = {G1, ..., Gc}. The ground warehouse node is copied into two duplicate nodes (0 and c+1) as the starting node and return node of a trip. The set Nd = {0, 1, ..., c} is used to represent the set of starting nodes and customer nodes, which is equivalent to the possible departure node set of vehicles or drones, and Nr = {1, ..., c, c+1} represents the set of return nodes and customer nodes.

[0059] Furthermore, a trip consists of a drone and a number of vehicles. Multiple trips are allocated in the network topology to serve all customer nodes. The process of a vehicle carrying a drone departing from the warehouse to serve a customer node and returning to the warehouse is defined as a trip, denoted by s∈S. Multiple trips in the network topology are represented by a set S, and Φ(s) represents the set of nodes on trip s.<i,j> is the distance between nodes i and j, e<i,j,k> is the total distance the drone travels from node i, serving node j, and returning to node k. Each trip starts at node 0 and returns to node c+1. i,k (s) indicates whether the vehicle is transferred from node i to node k. i,j,k (s) indicates whether the drone takes off from node i to serve node j and returns to node k in trip s, as Figure 3 shown.

[0060] Furthermore, for each UAV, κ is the maximum load, Indicates its maximum flight range. After delivering the package, the drone can land on a vehicle and charge using the onboard charger.

[0061] In this embodiment, the new customer delivery demand includes the delivery destination and the delivery arrival time. When updating and optimizing the current network topology map, the delivery destination is added to the current network topology map as a new customer node, and the delivery service is selected by vehicle or drone.

[0062] Furthermore, in the current network topology and the updated network topology, the route distance and carrying capacity between nodes, the energy consumption of the drone, and the reliability of the drone can all be calculated and obtained according to existing relevant methods.

[0063] In step S110, with the goal of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, a customer allocation algorithm is used to add the delivery destination as a new customer node to the current network topology graph.

[0064] Specifically, the first objective in the dual-objective model is to minimize the number of vehicles in the entire network topology, denoted by Nt, while the second objective is to maximize the flight distance Φ of the UAV. u Distance from vehicle Φ v The ratio value is expressed as:

[0065] minNt(1)

[0066]

[0067] According to the dual-objective model expressed in formulas (1) and (2), it can be noted that the total distance of the route is equal to the sum of the travel distances of each trip, and the distance of a single trip is composed of the distances traveled by the drone and the vehicle. Therefore, the optimization of the total trip distance can be converted to the optimization of the distance of a single trip. The relaxed single-objective problem is compared with the customer assignment algorithm for reliable delivery and converted into a traditional NP-hard problem. The dual-objective problem (i.e., minimizing the number of aircraft and maximizing the ratio of the drone flight distance to the aircraft travel distance) is equivalent to minimizing the network overhead, which converts the dual-objective problem into a simpler single-objective problem.

[0068] Therefore, the optimization of the total travel distance can be converted into the optimization of the single travel distance, that is, the dual-objective model is converted into a single-objective model and expressed as:

[0069]

[0070] In formula (3), Φ v (s) / Φ u (s) is the distance traveled by the vehicle in a trip Φ v Flight distance from the drone Φ u The ratio value of . Among them, the variable δ i,k (s) indicates whether the vehicle is transferred from node i to node k. i,j,k (s) indicates whether the drone takes off from node i to serve node j and returns to node k in trip s. Both variables are binary variables. They can be used to determine which route the drone and vehicle are assigned to and which client node they can serve.

[0071] In this embodiment, after the dual-objective model is converted into a single-objective model, six constraints are proposed in the process of solving the single-objective model, namely, formula (3). That is, under these six constraints, the new customer node is added to the current network topology diagram while satisfying formula (3).

[0072] Specifically, the six constraints include: the first constraint: at least one vehicle carrying a drone provides delivery service for the customer node, the second constraint: during the delivery service of a drone or vehicle to a customer node, each customer node can only be served by a vehicle or a drone, the third constraint: the sum of all packages to be distributed cannot exceed the maximum load of the vehicle, the fourth constraint: the mutual waiting time between the matched drones and vehicles at the landing node is less than or equal to the maximum waiting time, the fifth constraint: the weight of the package that the drone needs to carry is less than the weight of the package of its maximum payload, and the sixth constraint: when the drone is completing a certain service, its energy at the take-off node should be sufficient to support it to complete the service and return to the landing point.

[0073] Furthermore, the above six constraints can be expressed as formula (4) to formula (9):

[0074]

[0075] i,j,k∈Φ(s)

[0076]

[0077] Among them, i,j,k,l,m,n∈Φ(s),s∈S

[0078] In formulas (4) to (9), Nt represents the number of vehicles, s represents a trip, S represents all trips in the network topology, and N d and N r Represent the starting node set and the ending node set in each trip respectively, C represents the nodes that need to be served, that is, the customer node set, and the variable δ i,k (s) indicates whether the vehicle is transferred from node i to node k, and the variable δ' i,j,k (s) indicates whether the drone takes off from node i to serve node j and returns to node k in trip s, Φ(s) indicates the set of nodes on trip s, C indicates the set of nodes that need to be served, i.e., the set of client nodes, g j Indicates the weight of the package. represents the maximum payload of the vehicle, t' k (s) represents the landing time of the UAV at node k, t k (s) represents the arrival time of the vehicle at node k, represents the maximum delay time, κ represents the maximum payload of the UAV, represents the maximum flight distance of the UAV from node i to node j and then returns to node k, and ξ represents the power consumption of the UAV in one service.

[0079] In this embodiment, when the delivery destination is added as a new customer node to the current network topology map: for each trip, the new customer node in the current network topology map is judged whether it satisfies the first constraint, the second constraint, and the third constraint, and the customer nodes that satisfy these three constraints are placed in the to-be-added set; when the package weight at a certain node j is less than the maximum payload of the vehicle, a customer node is randomly selected from the to-be-added set as the next node to be added to the current trip, and it is judged whether the customer node satisfies the fourth constraint, the fifth constraint, and the sixth constraint.

[0080] Furthermore, this application also proposes a method for evaluating the service capabilities of drones in the initial network topology. Since drones are affected by flight factors such as energy consumption, distance, payload, and other factors during package delivery, the following formula is used to evaluate the reliability of their services:

[0081]

[0082] In formula (10), α, β, represents the coefficient, It represents the ratio between service energy consumption and takeoff power, Indicates the ratio between the flight distance in a trip and the maximum flight distance. Indicates the ratio between the weight of the package to be delivered and the maximum payload of the drone.

[0083] Although the initial solution generated by the customer allocation algorithm in step S110 satisfies all constraints of the optimization problem, it is likely that it will not achieve the optimal goal because it is a randomly composed sequence. Therefore, it is necessary to design a solution to change the order of the multi-trip sequence.

[0084] In step S120, a method for optimizing the initial network topology graph using a maximum iteration algorithm and reinforcement learning is proposed, including optimizing the initial network topology graph through an improved controller, a perturbation controller, and a threshold controller combined with reinforcement learning, wherein the improved controller and the perturbation controller correspond to the improved operator and the perturbation operator respectively, and the threshold controller is used to select whether to use the improved or perturbation controller, and the operator is mapped to an action through reinforcement learning, and different actions are assigned weights in continuous iterations. Different operators are selected according to different node sequences in continuous training, and the trajectory and resource allocation are continuously optimized in a cycle.

[0085] In this embodiment, optimization within a trip or optimization between trips is performed on each trip in the initial network topology diagram of the controller and the disturbance controller.

[0086] Furthermore, the improved controller utilizes improved operators for optimization within each trip and between trips. When optimizing within a trip, the optimization can be performed using the deletion operator, the symmetric swap operator, and the insertion operator. The deletion operator is used to delete the edge between two nodes and reconnect them, the symmetric swap operator is used to symmetrically swap the positions of two nodes within a trip, and the first insertion operator is used to insert a node in the trip to the new location. When optimizing between trips, the crossover operator, the reverse crossover operator, the symmetric swap operator, the asymmetric swap operator, the second insertion operator, and the periodic swap operator can be used. The crossover operator swaps trip segments between two trips, the reverse crossover operator reverses a trip and then swaps trip segments between the two trips, the symmetric swap operator symmetrically swaps trip segments of a preset length between two trips, the asymmetric swap operator swaps two trip segments of different lengths between two trips, the second insertion operator inserts a trip segment from one trip into another trip, and the periodic swap operator cyclically swaps a node between three trips.

[0087] Furthermore, the perturbation controller uses perturbation operators to optimize between trips, employing a random reconstruction operator, a random swap operator, and a periodic swap operator. The random reconstruction operator randomly selects a segment and perturbs the order of actions, the random swap operator randomly swaps pairs of adjacent nodes, and the periodic swap operator periodically swaps nodes across multiple trips.

[0088] In this embodiment, reinforcement learning is used to optimize the initial solution by selecting different operators through a threshold controller. The goal of action selection in this reinforcement learning approach is to select the most appropriate operator for each input path. Reinforcement learning uses a policy gradient method to train the network weights based on the current state to select the operator.

[0089] Furthermore, each node served by a drone or vehicle in each trip has a state. Each state consists of three parts: information about the current node, specific information about the current solution, and historical information. The information about the current node includes: the position of node i in the trip, the position of the previous node i-, the position of the subsequent node i+, and the distance between the above nodes. The specific information of the current solution includes: the demand Gi of node i, the remaining capacity of the vehicle serving trip s, and the service reliability of the drone at node i. The historical information includes: the operator selected by the node in the previous h steps in this state and the corresponding reward.

[0090] Furthermore, the action selection in each round is determined by reinforcement learning. The action selection decision in each round is decomposed into selecting an operator as the action from all operators in each iteration.

[0091] Furthermore, reward refers to the design of a reward function, where A(t) is the advantage function, which represents the difference between the current optimization target value and the initial solution value generated by CARD. The optimization goal of reinforcement learning is to minimize the ratio in formula (3).

[0092] Specifically, within the network policy, policy gradients can be directly applied to reinforcement learning in high-dimensional or continuous action spaces. Because perturbation operators can reconstruct the current solution and move it out of local optima, a Monte Carlo-based policy gradient method is used to update the policy gradient. The solution and problem information in the state serve as inputs to the policy network, and the output is the probability of selecting a particular action. After passing through the policy network, a threshold controller selects operators to alter the solution to continuously reach the optimal solution.

[0093] like Figure 4 , which is a flowchart of optimizing the initial network topology using the maximum iteration algorithm and reinforcement learning.

[0094] In this joint resource allocation and trajectory optimization method for drone-vehicle collaboration, the current network topology and new customer delivery demands are obtained. A customer allocation algorithm is then employed. Based on a dual-objective optimization model constructed to minimize the number of vehicles and maximize the ratio of drone flight distance to vehicle travel distance, the current network topology is initially updated based on the new customer delivery demands to obtain an initial network topology. This initial network topology is then optimized using a maximum iteration algorithm and reinforcement learning to obtain an optimized network topology. This optimized network topology serves as the current network topology, enabling drones and vehicles to perform delivery services based on the current network topology. This method allows delivery services to be completed using minimal service resources while ensuring service quality.

[0095] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In one embodiment, Figure 5 As shown, a joint resource allocation and trajectory optimization device for cooperation between drones and vehicles is provided, including: a data acquisition module 200, an initial network topology map acquisition module 210 and a network topology map optimization module 220, wherein:

[0097] Data acquisition module 200, for acquiring a current network topology and new customer delivery requirements, wherein the current network topology includes a plurality of nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, a plurality of customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0098] An initial network topology map obtaining module 210 is configured to convert a dual-objective optimization model constructed based on the objectives of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance into a single-objective optimization model, and to perform a preliminary update of the current network topology map based on the new customer delivery demand using a customer allocation algorithm under the single-objective optimization model to obtain an initial network topology map;

[0099] The network topology optimization module 220 is used to optimize the initial network topology based on the single-objective optimization model using the maximum iteration algorithm and reinforcement learning to obtain an optimized network topology, and use the optimized network topology as the current network topology so that drones and vehicles can perform delivery services according to the current network topology.

[0100] Regarding the specific definition of the joint resource allocation and trajectory optimization device for drone and vehicle collaboration, please refer to the definition of the joint resource allocation and trajectory optimization method for drone and vehicle collaboration above, and will not be repeated here. The various modules in the above-mentioned joint resource allocation and trajectory optimization device for drone and vehicle collaboration can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0101] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a joint resource allocation and trajectory optimization method for collaboration between drones and vehicles is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0102] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0103] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0104] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0105] Based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, the model is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery demand to obtain an initial network topology map;

[0106] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map. The optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0108] Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes;

[0109] Based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, the model is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery demand to obtain an initial network topology map;

[0110] Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology map to obtain an optimized network topology map. The optimized network topology map is used as the current network topology map, so that drones and vehicles perform delivery services according to the current network topology map.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A joint resource allocation and trajectory optimization method for UAV and vehicle collaboration, characterized in that: The method comprises: Obtaining a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and a charging node, and the edges between the nodes represent drone routes or vehicle routes, wherein the new customer delivery requirements include a delivery destination and a delivery arrival time; Based on the dual-objective optimization model constructed with the goals of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance, the model is converted into a single-objective optimization model. Under the single-objective optimization model, a customer allocation algorithm is used to preliminarily update the current network topology map according to the new customer delivery demand to obtain an initial network topology map; Based on the single-objective optimization model, the maximum iteration algorithm and reinforcement learning are used to optimize the initial network topology to obtain an optimized network topology. The optimized network topology is used as the current network topology so that drones and vehicles perform delivery services according to the current network topology. When the current network topology is updated and optimized, the delivery destination is added to the current network topology as a new customer node, and the vehicle or drone is selected for delivery service. After the delivery destination is added to the current network topology as a new customer node, the customer allocation algorithm is used to solve the problem under six constraints. The six constraints include: the first constraint: at least one vehicle carrying a drone provides delivery service for the customer node; the second constraint: during the delivery service of the drone or vehicle for the customer node, each customer node can only be served by a vehicle or a drone; the third constraint: the sum of all packages to be allocated cannot exceed the maximum load of the vehicle; the fourth constraint: the mutual waiting time between the matched drone and the vehicle at the landing node is less than or equal to the maximum waiting time; the fifth constraint: the weight of the package that the drone needs to carry is less than the weight of the package of its maximum payload; the sixth constraint: the drone must complete a certain service. During the delivery process, the energy at the takeoff node should be sufficient to support it to complete the service and return to the landing point. When the delivery destination is added as a new customer node to the current network topology: for each trip, determine whether the new customer node in the current network topology satisfies the first constraint, the second constraint, and the third constraint, and place the customer node that satisfies these three constraints into a to-be-added set. When the package weight at a node is less than the maximum payload of the vehicle, randomly select a customer node from the to-be-added set as the next node to be added to the current trip, and determine whether the customer node satisfies the fourth constraint, the fifth constraint, and the sixth constraint. When optimizing the initial network topology using the maximum iteration algorithm and reinforcement learning: optimize the initial network topology using an improved controller, a perturbation controller, and a threshold controller in combination with reinforcement learning, wherein the improved controller and perturbation controller correspond to an improved operator and a perturbation operator, respectively, and the threshold controller is used to select whether to use the improved or perturbation controller. Through reinforcement learning, operators are mapped to actions, and weights are assigned to different actions in continuous iterations. Different operators are selected according to different node sequences in continuous training, and trajectory and resource allocation are continuously optimized.

2. The joint resource allocation and trajectory optimization method for UAV and vehicle collaboration according to claim 1 is characterized in that: The current network topology includes multiple itineraries, each of which includes a ground warehouse node, a takeoff node and a landing node of a drone, and different customer nodes. In each of the itineraries: The vehicle carrying the drone and the package to be delivered departs from the ground warehouse node. After arriving at the drone's take-off node, the drone and vehicle go to different customer nodes in the itinerary to provide delivery services. After completing the delivery service, the drone lands on the vehicle to charge and prepare for the next delivery service, or the vehicle carries the drone back to the ground warehouse node.

3. A joint resource allocation and trajectory optimization device for UAV and vehicle collaboration, characterized in that: The device implements the joint resource allocation and trajectory optimization method for UAV-vehicle collaboration according to any one of claims 1-2, and the device comprises: A data acquisition module, configured to obtain a current network topology and new customer delivery requirements, wherein the current network topology includes multiple nodes and edges connecting the nodes, wherein each node includes a ground warehouse node, multiple customer nodes, and a drone takeoff node and charging node, and the edges between the nodes represent drone routes or vehicle routes; an initial network topology map obtaining module, configured to convert a dual-objective optimization model constructed based on the objectives of minimizing the number of vehicles and maximizing the ratio of drone flight distance to vehicle travel distance into a single-objective optimization model, and to perform a preliminary update of the current network topology map based on the delivery requirements of the new customers using a customer allocation algorithm under the single-objective optimization model to obtain an initial network topology map; The network topology optimization module is used to optimize the initial network topology based on the single-objective optimization model using the maximum iteration algorithm and reinforcement learning to obtain an optimized network topology, and use the optimized network topology as the current network topology so that drones and vehicles can perform delivery services according to the current network topology.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images