A vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning

Abstract

The present invention relates to a vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning, which belongs to the field of transportation and logistics technology. It solves the problems of high cost, low efficiency and irrational resource allocation of "ground-air integrated" combined transportation in the prior art. The method comprises: step S1, constructing an interactive simulation environment for urban rapid logistics transportation; step S2, constructing a vehicle-UAV dynamic scheduling model based on the vehicle-UAV collaborative scheduling process using r-MDP; step S3, designing a hierarchical reinforcement learning algorithm framework, and solving the vehicle-UAV dynamic scheduling model using the hierarchical reinforcement learning algorithm framework to obtain a collaborative scheduling strategy; and step S4, substituting the obtained collaborative scheduling strategy into the constructed interactive simulation environment for verification to obtain a verification result. If the verification result meets preset requirements, the collaborative scheduling strategy is output for vehicle-UAV collaborative scheduling; otherwise, the method returns to step S3 to modify parameters and rebuild the hierarchical reinforcement learning algorithm framework.

Description

Technical Field

[0001] The present invention relates to the field of transportation and logistics technology, and in particular to a vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning. Background Art

[0002] With the acceleration of urbanization, the strengthening of transportation infrastructure, the rapid changes in consumer behavior, and the development of e-commerce, the demand for rapid urban logistics and delivery in my country is growing. Therefore, the development and construction of efficient urban rapid logistics and delivery service systems is imperative. Traditional logistics and transportation methods rely on ground transportation methods such as trucks. However, urban road congestion, traffic control, and environmental pollution pose challenges to these traditional logistics and transportation methods.

[0003] In recent years, drone technology has developed rapidly and demonstrated tremendous potential in the logistics and transportation sector. Drones, characterized by speed, efficiency, and flexibility, can, to a certain extent, alleviate ground traffic congestion. Drones, in particular, offer a highly efficient delivery method for short-distance, small-package deliveries in cities. Developing a collaborative "vehicle + drone" delivery model can, on the one hand, leverage the advantages of both modes of transportation. Working together, they are expected to provide a strong foundation for reliable, timely, and rapid urban logistics delivery services. On the other hand, it will promote the development of technologies for the coordinated and deeply integrated integration of air and ground transportation modes, accelerating the construction and implementation of a comprehensive, three-dimensional urban transportation network.

[0004] Currently, vehicle-drone collaborative scheduling is still in its early stages of exploration, facing challenges such as complex scheduling algorithms, poor real-time robustness, imperfect resource optimization, and difficulty in controlling transportation costs. Therefore, to address the shortcomings of current technologies, there is an urgent need to develop efficient and real-time vehicle-drone collaborative scheduling methods for rapid urban logistics transportation. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning. This method is a dynamic optimization method for urban rapid logistics transportation, which can realize the dynamic allocation of transportation demand, real-time scheduling of vehicles and drones, and route planning, thereby achieving the purpose of optimizing resource allocation, controlling transportation costs, and reducing environmental pollution.

[0006] According to one embodiment of the present invention, a vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning is provided, comprising the following steps:

[0007] Step S1, constructing an interactive simulation environment for urban rapid logistics transportation;

[0008] Step S2, using r-MDP to build a vehicle-UAV dynamic scheduling model based on the vehicle-UAV collaborative scheduling process;

[0009] Step S3, designing a hierarchical reinforcement learning algorithm framework, and solving the vehicle-UAV dynamic scheduling model through the hierarchical reinforcement learning algorithm framework to obtain a collaborative scheduling strategy; and

[0010] Step S4: Substitute the obtained collaborative scheduling strategy into the constructed interactive simulation environment for verification to obtain a verification result. If the verification result meets the preset requirements, the collaborative scheduling strategy is output for vehicle-UAV collaborative scheduling. Otherwise, return to step S3 to modify the parameters and rebuild the hierarchical reinforcement learning algorithm framework.

[0011] Wherein, the step S1 specifically includes the following steps:

[0012] Step S1.1, combining actual needs and historical information, simulate the time distribution of dynamic arrival of transportation demand;

[0013] Step S1.2: Simulate the spatial distribution of transportation demand generation based on the actual customer location distribution pattern and historical related information, using the simulation of the temporal distribution of dynamic arrival of transportation demand and the simulation of the spatial distribution of transportation demand generation as the simulation environment;

[0014] Step S1.3, configuring the performance indicators and driving parameters of the vehicles and drones in the fleet in the simulation environment to obtain an interactive simulation environment;

[0015] Wherein said step S2 specifically comprises the following steps:

[0016] Step S2.1, define the time point to trigger dynamic decision, set t k The time of the kth dynamic decision point is: the arrival of a new customer demand, the return of the vehicle to the distribution center, and the completion of the previous waiting interval Δt. The time when these three events occur is determined as the time point that triggers the dynamic decision;

[0017] Step S2.2, based on the time point of triggering dynamic decision, construct the various components of r-MDP, including the state space S k , action space A k , state transition P k , reward function R k ;

[0018] Step S2.3: Based on the actual fleet operator's optimization goal, the objective function of the dynamic decision-making process of the r-MDP is constructed. The various components of the r-MDP and the objective function constitute the vehicle-drone dynamic scheduling model;

[0019] Wherein said step S3 specifically comprises the following steps:

[0020] A hierarchical reinforcement learning algorithm framework is defined, which includes an upper-level intelligent agent, a lower-level intelligent agent, and a path-inspired algorithm layer. By defining the input, algorithm execution process, and training method, a pseudocode of the algorithm flow is formulated to obtain a hierarchical reinforcement learning algorithm framework. The vehicle-UAV dynamic scheduling model is solved through the hierarchical reinforcement learning algorithm framework to obtain a collaborative scheduling strategy.

[0021] Optionally, the step S1.1 specifically includes the following steps:

[0022] Step S1.1.1, simulate Class I time distribution data: Based on historical information and actual needs in production operations, set the transport demand arrival rate λ1, which represents the number of transport demand arrivals per minute. Keep λ1 constant and generate the dynamic arrival time of transport demand according to the Poisson distribution as the simulated Class I time distribution data;

[0023] Step S1.1.2, simulate Class II time distribution data: based on historical information and actual needs in production operations, set the transport demand arrival rate λ2, which represents the number of transport demands arriving per minute. Set λ2 to change according to the peak and off-peak periods of demand arrival. Generate the dynamic arrival time of transport demand according to the Poisson distribution as simulated Class II time distribution data. Use the simulated Class I time distribution data and the simulated Class II time distribution data as a simulation of the time distribution of the dynamic arrival of transport demand.

[0024] Optionally, the step S1.2 specifically includes the following steps:

[0025] Step S1.2.1, simulate type I spatial distribution data: generate N1 geographical location coordinates that obey the normal distribution, and the jth geographical coordinate is expressed as (x j ,y j ), 0 <j≤N1;

[0026] Step S1.2.2, simulate the spatial distribution data of type II: divide the city into several geographical regions, and obtain the population density in each geographical region. m Inside, generate N m The geographical location coordinates obey the normal distribution, and the geographical location coordinates of all geographical areas of the city are obtained as the simulation of the spatial distribution of transportation demand generation.

[0027] Optionally, step S1.3 specifically includes the following steps:

[0028] Step S1.3.1, set the fleet-related parameters, including: the number of drones |D| and the number of vehicles |V|, as well as the fleet working time T work ;

[0029] Step S1.3.2, setting vehicle performance and driving parameters, including: vehicle speed v e , vehicle loading time T vl and the time T for vehicle to unload cargo vu ;

[0030] Step S1.3.3, set the UAV performance and driving parameters, including: UAV flight speed v d , UAV charging time T charge , UAV loading cargo time T dl and the time T for drone to unload cargo du ;

[0031] In step S1.3.4, the performance indicators and driving parameters of the vehicles and drones of the fleet set in the simulation environment are used as the interactive simulation environment.

[0032] Optionally, step S2.2 specifically includes the following steps:

[0033] Step S2.2.1, construct the current state space S k : time t including the kth dynamic decision point k 、Customer demand information i ={(x i ,y i ),t i ,TL i}, the planned driving route of the vehicle at the kth dynamic decision point and the planned driving route of the UAV at the kth dynamic decision point Among them, t i is the arrival time of the i-th demand, TL i c i The delivery time limit of the fleet is v∈V, where V is the set of all vehicles in the fleet. is the planned route of vehicle v, and the number of drones d∈D in the fleet, where D is the set of all drones in the fleet, The planned route for UAV d;

[0034] Step S2.2.2, construct action space A k : in, Indicates t k The vehicle that is returning to the distribution center or finishing the last waiting interval is dispatched. Indicates t k The allocation of transport modes to the demand points at any given time, Based on the and Get updated planned routes for vehicles and drones;

[0035] Step S2.2.3, describe the state transition process P k :Define the current state space S k Transfer to the post-decision state space and the next decision-making state space S k+1 processes and changes;

[0036] Step S2.2.4, construct the reward function: construct it as a function of action and state, expressed as R k (S k ,A k ), when the transport demand is assigned to the fleet and the delivery is completed successfully, R k >0; otherwise, R k <0.

[0037] Optionally, step S3 specifically includes the following steps:

[0038] Step S3.1, define a demand buffer component, which is used to store customer demands arriving between two adjacent vehicles in the interactive simulation environment, and serves as a link between the upper-level agent and the lower-level agent;

[0039] Step S3.2: Construct the upper-layer agent to obtain the DQN network, which is used to decide whether the vehicle should depart based on the time point when the dynamic decision is triggered, when the vehicle returns to the distribution center and the vehicle has completed the previous waiting interval Δt;

[0040] Step S3.3: construct the lower-level agent and obtain the proximal policy optimization PPO network, which is used to allocate the newly arrived customer demands in the simulation environment according to the time point when the dynamic decision is triggered;

[0041] Step S3.4: construct a path heuristic algorithm layer to update the routes of drones and vehicles in real time;

[0042] In step S3.5, the decision on whether to depart a vehicle obtained through the DQN network, the allocation of newly arrived customer demands in the simulation environment obtained through the PPO network, and the real-time update of the driving routes of drones and vehicles by constructing a path heuristic algorithm layer are used as a collaborative scheduling strategy obtained through the hierarchical reinforcement learning algorithm framework.

[0043] Optionally, step S3.2 specifically includes the following steps:

[0044] Step S3.2.1, based on the state space S in the constructed vehicle-drone dynamic scheduling model k , extract the state features as the input of the DQN network, the state features include the current decision time t k The earliest return time of the drone according to the planned route The earliest return time of the vehicle according to the planned route and the minimum remaining delivery time required for the vehicle's planned route

[0045] Step S3.2.2, construct the action space of the DQN network

[0046] Step S3.2.3, construct the reward function of the DQN network;

[0047] In step S3.2.4, based on the input state characteristics, the dimensions of the action space of the DQN network and the reward function of the DQN network are constructed, and the DQN network structure and parameters are configured: the number of hidden layers, the number of neurons, the learning rate, the exploration rate, and the discount factor are set to obtain the DQN network, and the vehicle is decided whether to depart through the DQN network.

[0048] Optionally, the step S3.3 specifically includes the following steps:

[0049] Step S3.3.1, based on the state space S in the constructed vehicle-drone dynamic scheduling model k , extract the state features as the input of the PPO network, the state features include: current transportation demand c i Arrival time t i , the straight-line distance from the transportation demand to the distribution center The earliest return time of the drone according to the planned route The earliest return time of the vehicle according to the planned route Vehicle delivery demandc i Feasibility V (c i ), drone delivery demand c i Feasibility D (c i ), c i Minimum insertion time cost assigned to a vehicle

[0050] Step S3.3.2, construct the action space of the PPO network and obtain the decision c i Mode of transport;

[0051] Step S3.3.3, construct the reward function of the PPO network;

[0052] In step S3.3.4, according to the input state characteristics and the dimension of the action space of the constructed PPO network, the PPO network structure and parameters are configured: the number of hidden layers, the number of neurons, the learning rate, the weights in the loss function, and the discount factor are set to obtain the PPO network, and the newly arrived customer demands in the simulation environment are distributed through the PPO network.

[0053] Optionally, step S3.4 specifically includes the following steps:

[0054] Step S3.4.1, use the first-come-first-served heuristic algorithm to plan the route of the UAV obtained in step S2.2.1 Planning includes arranging deliveries in the order of arrival of requests, prioritizing drones that are idle and have returned to the distribution center the earliest. Each drone only serves one transport request at a time, and the drone's route is updated in real time.

[0055] Step S3.4.2, use the minimum insertion cost method to calculate the vehicle route obtained in step S2.2.1 Planning is performed, including giving priority to vehicle routes that meet feasibility requirements, that is, routes that can successfully complete delivery before the delivery time limit and minimize the added time cost, as well as selecting insertion points with the smallest added cost, and updating the vehicle's route in real time.

[0056] According to one embodiment of the present invention, a vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning is provided, which includes the following steps:

[0057] (1) Constructing a simulation environment for urban rapid logistics transportation. Aiming at the urban rapid logistics transportation scenario, the dynamic generation process of customer orders is simulated, and an interactive simulation environment of "demand generation-agent decision-making-fleet delivery" is established.

[0058] (2) Construct a dynamic scheduling model for vehicles and drones, and use the route-based Markov Decision Process (r-MDP) to mathematically abstract, define and describe the collaborative scheduling process of vehicles and drones.

[0059] (3) Design a hierarchical reinforcement learning framework that integrates reinforcement learning and search-based heuristic algorithms to achieve demand allocation, fleet scheduling, and route planning functions.

[0060] The present invention constructs an urban rapid logistics transportation simulation environment, generates Class I and Class II transportation demand time distribution data, Class I and Class II spatial distribution data, and provides support for the simulation of urban rapid logistics transportation scheduling of "demand generation-agent decision-making-fleet distribution". A mathematical model for dynamic scheduling of vehicles and drones is constructed, and r-MDP is used to describe and model the dynamic collaborative scheduling process of vehicles and drones, defining the dynamic decision time points and state space (S k ), action space (A k ), state transfer (P k ), reward function (R k) and objective function. A three-layer reinforcement learning framework was designed to solve dynamic fleet scheduling solutions. The upper-layer agent, the Deep Q-network (DQN), decides whether vehicles from the distribution center should depart. The lower-layer agent, the Proximal Policy Optimization (PPO), allocates the required transportation mode. The third layer, the path heuristic algorithm, updates the routes of drones and vehicles in real time using first-come, first-served and minimum insertion cost heuristics.

[0061] Compared with the existing technology, a vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning provided in accordance with an embodiment of the present invention has at least the following advantages: realizing real-time dynamic allocation of transportation demand, real-time scheduling and route planning of vehicles and drones, thereby achieving the purpose of optimizing resource allocation, controlling transportation costs and reducing environmental pollution, and providing a dynamic optimization method for urban rapid logistics transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention can be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 The present invention provides a flowchart of a vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning according to an embodiment of the present invention.

[0064] Figure 2 Schematic diagram of the temporal and spatial distribution of transportation demand for the vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning according to an embodiment of the present invention.

[0065] Figure 3 Schematic diagram of the hierarchical reinforcement learning algorithm framework of the vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0067] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0068] The following describes in detail a vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0069] Explanation of symbols:

[0070] The number of geographical location coordinates in N1Ⅰ category spatial distribution;

[0071] N m Geographical region in type II spatial distribution m The number of geographic location coordinates within (x j ,y j ) jth geographic coordinate;

[0072] |D|Number of drones;

[0073] |V|Number of vehicles;

[0074] T work Fleet working hours;

[0075] v e Vehicle speed;

[0076] T vl Time taken to load cargo on the vehicle;

[0077] T vu The time when the vehicle unloads the cargo;

[0078] v d Drone flight speed;

[0079] T charge Drone charging time;

[0080] T dl The time it takes for drones to load cargo;

[0081] T du The time it takes for the drone to unload the cargo;

[0082] t k The time of the kth dynamic decision point;

[0083] ΔtThe vehicle has finished the last waiting interval;

[0084] V is the set of all vehicles in the fleet;

[0085] v is the vehicle v∈V in the fleet;

[0086] S k Current state space;

[0087] A k Action space;

[0088] P k State transfer;

[0089] R k Reward function;

[0090] c i Customer demand information;

[0091] (x i ,y i ) The i-th demand c i geographical coordinates;

[0092] t i The i-th demand c i Arrival time;

[0093] TL i c i Delivery time limit;

[0094] The planned driving route of the vehicle at the kth dynamic decision point; The planned route of vehicle v;

[0095] The planned driving route of the UAV at the kth dynamic decision point; The planned route of drone d;

[0096] D is the set of all drones in the fleet;

[0097] d is the drone in the fleet, d∈D;

[0098] A k Action space;

[0099] post-decision state;

[0100] Indicates t k Dispatch vehicles that are always returning to the distribution center or completing their last waiting interval;

[0101] Indicates t k The allocation of transport modes to reach the demand point at any time;

[0102] Based on and Get updated planned routes for vehicles and drones;

[0103] S k+1 The state space before the next decision;

[0104] The earliest return time of the drone according to the planned route;

[0105] The earliest return time of the vehicle according to the planned route;

[0106] The minimum remaining delivery time for the demand in the vehicle's planned route;

[0107] The action space of the DQN network;

[0108] The reward function of the DQN network;

[0109] is the reward obtained by the lower-level agent PPO network for allocating demand c;

[0110] RBU Request Buffer (RBU), customer demand information received between the departures of two adjacent vehicles;

[0111] c Lower-level agent PPO network allocation demand, c∈RBU;

[0112] The straight-line distance of the transportation demand from the distribution center;

[0113] Π V (c i ) Vehicle delivery demand c i feasibility;

[0114] Π D (c i ) UAV delivery demand i feasibility;

[0115] c i the minimum insertion time cost assigned to the vehicle;

[0116] Action space of PPO network;

[0117] Reward function of the PPO network.

[0118] like Figure 1 As shown, a vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning is provided according to an embodiment of the present invention, including the following steps.

[0119] Step S1: Construct an interactive simulation environment for urban rapid logistics. Aiming at the urban rapid logistics scenario, simulate the dynamic generation of customer orders and establish an interactive simulation environment for "demand generation - agent decision-making - fleet delivery."

[0120] In step S2, the vehicle-UAV collaborative scheduling process is constructed into a vehicle-UAV dynamic scheduling model using a route-based Markov decision process (hereinafter referred to as r-MDP).

[0121] Step S3: Design a hierarchical reinforcement learning algorithm framework and use it to solve the vehicle-drone dynamic scheduling model to obtain a collaborative scheduling strategy. This integrates reinforcement learning and search-based heuristic algorithms to implement demand allocation, fleet scheduling, and route planning.

[0122] In step S4, the obtained collaborative scheduling strategy is substituted into the constructed interactive simulation environment for verification to obtain the verification result. If the verification result meets the preset requirements, the collaborative scheduling strategy is output for vehicle-UAV collaborative scheduling. Otherwise, return to step S3 to modify the parameters and rebuild the hierarchical reinforcement learning algorithm framework.

[0123] Optionally, step S1 includes the following sub-steps:

[0124] Step S1.1, combining actual needs and historical information, simulate the time distribution of dynamic arrival of transportation demand;

[0125] Step S1.2: Simulate the spatial distribution of transportation demand generation based on the actual customer location distribution pattern and historical related information, using the simulation of the temporal distribution of dynamic arrival of transportation demand and the simulation of the spatial distribution of transportation demand generation as the simulation environment;

[0126] Step S1.3, configure the performance indicators and driving parameters of the vehicle-drone fleet in the simulation environment to obtain an interactive simulation environment.

[0127] The simulated transportation demand time distribution is shown as follows Figure 2 As shown, the step S1.1 specifically includes the following sub-steps.

[0128] Step S1.1.1, simulate Class I time distribution data: Based on historical information and actual needs in production operations, set the transport demand arrival rate λ1, which represents the number of transport demand arrivals per minute. Keep λ1 constant and generate the dynamic arrival time of transport demand according to the Poisson distribution as the simulated Class I time distribution data;

[0129] Step S1.1.2, simulating type-II time distribution data: According to historical information and actual needs in production and operation, set the transportation demand arrival rate λ2, which represents the number of transportation demands arriving per minute. Set λ2 to change according to the peak and off-peak periods of demand arrival. Generate the dynamic arrival time of transportation demands according to the Poisson distribution as the type-II time distribution data of the simulation. Use the type-I time distribution data and type-II time distribution data of this simulation as the simulation of the time distribution of the dynamic arrival of transportation demands.

[0130] The schematic diagram of the simulated transportation demand spatial distribution is as Figure 2 shown, and the specific steps of step S1.2 include the following sub-steps.

[0131] Step S1.2.1, simulating type-I spatial distribution data: Generate N1 geographical location coordinates that follow a normal distribution. The jth geographical coordinate is represented as (x j , y j ), where 0 < j ≤ N1. Among them, the central position (the center of the normal distribution) of these N1 geographical coordinates is the randomly selected location of the distribution center. The distribution rule of geographical coordinates is that the degree of dispersion (standard deviation) of geographical locations is set according to historical data or experience. The amount of geographical location coordinate data can be adjusted according to the scale of the application scenario.

[0132] Step S1.2.2, simulating type-II spatial distribution data: According to the urban administrative divisions and land use functions released by the government and public data platforms, divide the city into several geographical regions, and obtain the population density within each geographical region. Within each geographical region Region m , generate N m geographical location coordinates that follow a normal distribution, where the value of N m is directly proportional to the population density. Use the geographical location coordinates within all geographical regions of this city obtained as the simulation of the spatial distribution of transportation demand generation.

[0133] The specific steps of step S1.3 can include the following sub-steps.

[0134] Step S1.3.1, set the parameters related to the fleet, including: the number of drones |D| and the number of vehicles |V|, and the working time T work of the fleet. Among them, the number of drones |D| and the number of vehicles |V| are used for distribution tasks and are set according to the procurement plan, customer demand volume, and the size of the distribution area; the working time T work of the fleet, that is, the number of hours that vehicles and drones work per day, is set according to actual operating conditions and regulatory requirements.

[0135] Step S1.3.2, set the vehicle performance and driving parameters, including: the vehicle driving speed v e , the vehicle loading time Tvl and the time T for vehicle to unload cargo vu Among them, on urban roads, the vehicle speed v e , which can usually be set at 30-50km / h. The specific value can be set according to different traffic conditions. The time for loading cargo on the vehicle is T vl and the time T for vehicle to unload cargo vu The specific value of can be set by measuring or estimating the actual operation time of loading and unloading cargo on the vehicle.

[0136] Step S1.3.3, set the UAV performance and driving parameters, including: UAV flight speed v d , UAV charging time T charge , UAV loading cargo time T dl and the time T for drone to unload cargo du Among them, the UAV flight speed v d The setting is based on the drone's technical specifications, flight tests, and actual operating conditions (taking into account weather conditions such as wind speed); the drone's charging time is T charge , that is, the charging time required after the drone battery is completely consumed, which is usually related to the drone model, battery capacity and charging power, and can be set through actual testing; the drone loading time T dl and the time T for drone to unload cargo du The specific value of can be set by measuring or estimating the actual operation time of loading and unloading cargo by the UAV.

[0137] In step S1.3.4, the performance indicators and driving parameters of the vehicles and drones of the fleet are configured in the simulation environment as an interactive simulation environment.

[0138] Optionally, step S2 includes the following sub-steps.

[0139] Step S2.1, define the time point to trigger dynamic decision, set t k The time of the kth dynamic decision point is: the arrival of a new customer demand, the return of the vehicle to the distribution center, and the completion of the previous waiting interval Δt. The time when the above three events occur is determined to be the time point that triggers the dynamic decision.

[0140] Step S2.2, based on the time point of triggering dynamic decision, construct the various components of r-MDP, including: state space S k , action space A k , state transition P k , reward function R k and a discount factor γ.

[0141] In step S2.3, based on the actual fleet operator's optimization goals, an objective function is constructed in the r-MDP. The various components of the r-MDP and this objective function constitute the vehicle-drone dynamic scheduling model. The dynamic decision-making process of the r-MDP seeks the optimal strategy to maximize the number of customer requests served by the fleet.

[0142] Optionally, step S2.2 includes the following sub-steps:

[0143] Step S2.2.1, construct the state space S k : time t including the kth dynamic decision point k 、Customer demand information i ={(x i ,y i ),t i ,TL i}, the planned driving route of the vehicle at the kth dynamic decision point Among them, vehicle v∈V, V is the set of all vehicles in the fleet; the planned driving route of the UAV at the kth dynamic decision point is Among them, drone d∈D, D is the set of all drones in the fleet. (x i ,y i ) is the geographical coordinate of customer demand, t i is the arrival time of the i-th demand, TL i c i delivery time limit. represents the planned route of vehicle v, including the vehicle departure time, the order of planned delivery demands, the time when the vehicle arrives at the demand point, and the return time of the vehicle. represents the planned driving route of drone d, including the drone’s departure time, the time when the drone arrives at the demand point, and the drone’s return time.

[0144] Step S2.2.2, construct the action space as:

[0145] in, Indicates t k The vehicles that have returned to the distribution center or completed the previous waiting interval are dispatched to maximize the number of customer demands served by the fleet. They are determined to continue waiting at the distribution center for new transportation demands to arrive, or to depart immediately.

[0146]

[0147] In order to maximize the number of customer demands for fleet services, k The allocation of transport modes to the demand point at any given time, including allocation to vehicles or drones, or refusal to provide transport services, specifically defined as

[0148]

[0149] Based on the above and As a result, the updated vehicle and drone planned driving routes are obtained. in, planning a driving route for the updated vehicle; Plan a route for the updated drone.

[0150] Step S2.2.3, describe the state transition process P k : Define the current state S k Transition to post-decision state and the next pre-decision state S k+1 The updated planned driving route after the decision is made k Transfer to Changes in the external environment Transfer to S k+1 .

[0151] Step S2.2.4, construct the reward function: construct it as a function of action and state, expressed as R k (S k ,A k ). When the transport demand is assigned to the fleet and the delivery is completed successfully, R k >0; otherwise, R k <0.

[0152] Optionally, step S3 specifically includes the following steps: Define a hierarchical reinforcement learning algorithm framework to solve the optimal strategy of the vehicle-drone dynamic scheduling model. The hierarchical reinforcement learning algorithm framework is a multi-level algorithm structure based on hierarchical reinforcement learning. Formulate the overall workflow of the hierarchical reinforcement learning algorithm framework, such as Figure 3 As shown, the cascade and collaborative relationship between the modules of each layer of the hierarchical reinforcement learning algorithm framework is clarified, and the functions and processes of each layer of the agent / algorithm of the hierarchical reinforcement learning algorithm framework are defined. The hierarchical reinforcement learning framework includes three hierarchical modules: the upper-layer agent, the lower-layer agent, and the path heuristic algorithm layer. By interacting with the simulation environment, the three layers respectively decide whether the vehicle should depart, how to allocate demand, and the vehicle-drone route, and feed back to the simulation environment. The elements between the upper-layer agent and the lower-layer agent are interrelated. At the same time, the output results of the upper-layer agent and the lower-layer agent affect the output results of the path heuristic algorithm module. Step S3 also includes formulating the pseudo code of the hierarchical reinforcement learning algorithm framework, including input, algorithm execution process, and training method.

[0153] Optionally, step S3 specifically includes the following sub-steps:

[0154] In step S3.1, a request buffer (RBU) component is defined. This component stores customer requests arriving between two adjacent vehicles in the simulation environment and serves as a link between upper and lower-level agents. Customer requests arriving between two adjacent vehicles are generated based on demand information in the simulation environment.

[0155] In step S3.2, the upper-layer agent, the Deep Q-network (DQN), is constructed. Based on the time point for triggering dynamic decision-making defined in step S2.1, when the vehicle returns to the distribution center and completes the previous waiting interval Δt, the DQN decides whether the vehicle should depart.

[0156] Step S3.3: Construct the lower-level agent, namely the Proximal Policy Optimization (PPO). Based on the time point for triggering dynamic decision-making defined in step S2.1, when new customer demands arrive in the simulation environment, PPO allocates them.

[0157] In step S3.4, a path heuristic algorithm layer is constructed to update the routes of UAVs and vehicles in real time.

[0158] In step S3.5, the decision on whether to depart the vehicle is obtained through the DQN network, the allocation of newly arrived customer demands in the simulation environment is obtained through the PPO network; and the real-time update of the driving routes of drones and vehicles is performed by constructing a path heuristic algorithm layer. The vehicle-drone dynamic scheduling model is solved through the hierarchical reinforcement learning algorithm framework to obtain a collaborative scheduling strategy.

[0159] Optionally, step S3.2 includes the following sub-steps:

[0160] Step S3.2.1, based on the state space S constructed in step S2.2.1 k , extract the state features as the input of the DQN network, including the current decision time t k The earliest return time of the drone (according to the planned route) is recorded as The earliest return time of the vehicle (according to the planned route) is recorded as The minimum remaining delivery time for the demand in the vehicle's planned route

[0161] Step S3.2.2, construct the action space of the upper-layer intelligent DQN network, denoted as:

[0162]

[0163] Step S3.2.3: Construct the reward function for the DQN network. When the upper-layer DQN network decides that the vehicle should continue to wait, the reward is 0. However, when the vehicle is decided to depart immediately, the reward is the sum of the rewards for allocating the transportation needs stored by the RBU, expressed as:

[0164]

[0165] in, is the reward obtained by the lower-level agent PPO for allocating demand c.

[0166] In step S3.2.4, based on the input state features (state features in step S3.1.2.1), the dimensions of the action space of the constructed DQN network (the action space in step 3.1.2.2) and the reward function of the DQN network obtained in step 3.1.2.3, configure the DQN network structure and parameters: set the number of hidden layers, the number of neurons, the learning rate, the exploration rate, and the discount factor to obtain the DQN network, and use the DQN network to decide whether the vehicle should depart.

[0167] Optionally, step S3.3 includes the following sub-steps:

[0168] Step S3.3.1, based on the state space S constructed in step S2.2.1 k , extract the state features as the input of the PPO network, including: current customer demand c i Arrival time t i ; The straight-line distance from the transportation demand to the distribution center The earliest return time of the drone according to the planned route The earliest return time of the vehicle according to the planned route Vehicle delivery demandc i The feasibility of V (c i ); drone delivery demand c i The feasibility of D (c i ); c i Minimum insertion time cost assigned to a vehicle

[0169] Step S3.3.2, construct the action space of the lower-level intelligent agent PPO network, that is, decision c i The mode of transport is recorded as:

[0170]

[0171] Step S3.3.3, construct the reward function of the PPO network. When the lower-level agent assigns the transportation demand to the drone or vehicle and successfully delivers it, the reward is 1; when the demand c i When the delivery is rejected, the reward is -1. The reward function of the PPO network is expressed as:

[0172]

[0173] In step S3.3.4, according to the input state characteristics and the dimension of the action space of the constructed PPO network, the PPO network structure and parameters are configured: the number of hidden layers, the number of neurons, the learning rate, the weights and discount factors in the loss function, etc. are set, and the newly arrived customer demands in the simulation environment are distributed through the PPO network.

[0174] Optionally, step S3.4 specifically includes the following sub-steps:

[0175] Step S3.4.1, using the first-in-first-out (FIFO) heuristic algorithm to calculate the planned route of the UAV obtained in step S2.2.1 Planning is carried out, that is, delivery is carried out in the order of arrival time of demand, and priority is given to drones that are idle and return to the distribution center the earliest. Each drone only serves one transportation demand at a time, and the drone's driving route is updated in real time.

[0176] Step S3.4.2, using the minimum insertion cost method to plan the driving route of the vehicle in step S2.2.1 Planning is carried out, giving priority to vehicle routes that meet feasibility, that is, those that can successfully complete delivery before the delivery time limit and with the minimum added time cost, and selecting insertion points with the minimum added cost, and updating the vehicle's driving route in real time.

[0177] The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning provided in accordance with the embodiments of the present invention provides a collaborative scheduling strategy for vehicle-drone collaborative scheduling, which can realize dynamic allocation of transportation demand, real-time scheduling and route planning of vehicles and drones, and provide a fast and efficient logistics method for optimizing resource allocation and controlling transportation costs, thereby improving urban logistics transportation efficiency.

[0178] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0179] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0180] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning, characterized in that: The following steps are involved: Step S1, constructing an interactive simulation environment for urban rapid logistics transportation; Step S2, using r-MDP to build a vehicle-UAV dynamic scheduling model based on the vehicle-UAV collaborative scheduling process; Step S3: designing a hierarchical reinforcement learning algorithm framework, and solving the vehicle-UAV dynamic scheduling model through the hierarchical reinforcement learning algorithm framework to obtain a collaborative scheduling strategy; as well as Step S4: Substitute the obtained collaborative scheduling strategy into the constructed interactive simulation environment for verification to obtain a verification result. If the verification result meets the preset requirements, the collaborative scheduling strategy is output for vehicle-UAV collaborative scheduling. Otherwise, return to step S3 to modify the parameters and rebuild the hierarchical reinforcement learning algorithm framework. Wherein, the step S1 specifically includes the following steps: Step S1.1, combining actual needs and historical information, simulate the time distribution of dynamic arrival of transportation demand; Step S1.2: Simulate the spatial distribution of transportation demand generation based on the actual customer location distribution pattern and historical related information, using the simulation of the temporal distribution of dynamic arrival of transportation demand and the simulation of the spatial distribution of transportation demand generation as the simulation environment; Step S1.3, configuring the performance indicators and driving parameters of the vehicles and drones in the fleet in the simulation environment to obtain an interactive simulation environment; Wherein said step S2 specifically comprises the following steps: Step S2.1, define the time point to trigger dynamic decision, set t k The time of the kth dynamic decision point is: the arrival of a new customer demand, the return of the vehicle to the distribution center, and the completion of the previous waiting interval Δt. The time when these three events occur is determined as the time point that triggers the dynamic decision; Step S2.2, based on the time point of triggering dynamic decision, construct the various components of r-MDP, including the state space S k , action space A k , state transition P k , reward function R k ; Step S2.3: Based on the actual fleet operator's optimization goal, the objective function of the dynamic decision-making process of the r-MDP is constructed. The various components of the r-MDP and the objective function constitute the vehicle-drone dynamic scheduling model; Wherein said step S3 specifically comprises the following steps: A hierarchical reinforcement learning algorithm framework is defined, which includes an upper-level intelligent agent, a lower-level intelligent agent, and a path-inspired algorithm layer. By defining the input, algorithm execution process, and training method, a pseudocode of the algorithm flow is formulated to obtain a hierarchical reinforcement learning algorithm framework. The vehicle-UAV dynamic scheduling model is solved through the hierarchical reinforcement learning algorithm framework to obtain a collaborative scheduling strategy.

2. The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The step S1.1 specifically includes the following steps: Step S1.1.1, simulate Class I time distribution data: Based on historical information and actual needs in production operations, set the transport demand arrival rate λ1, which represents the number of transport demand arrivals per minute. Keep λ1 constant and generate the dynamic arrival time of transport demand according to the Poisson distribution as the simulated Class I time distribution data; Step S1.1.2, simulate Class II time distribution data: based on historical information and actual needs in production operations, set the transport demand arrival rate λ2, which represents the number of transport demands arriving per minute. Set λ2 to change according to the peak and off-peak periods of demand arrival. Generate the dynamic arrival time of transport demand according to the Poisson distribution as simulated Class II time distribution data. Use the simulated Class I time distribution data and the simulated Class II time distribution data as a simulation of the time distribution of the dynamic arrival of transport demand.

3. The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to claim 2 is characterized in that: The step S1.2 specifically includes the following steps: Step S1.2.1, simulate type I spatial distribution data: generate N1 geographical location coordinates that obey the normal distribution, and the jth geographical coordinate is expressed as (x j ,y j ), 0 <j≤N1; Step S1.2.2, simulate the spatial distribution data of type II: divide the city into several geographical regions, and obtain the population density in each geographical region. m Inside, generate N m The geographical location coordinates obey the normal distribution, and the geographical location coordinates of all geographical areas of the city are obtained as the simulation of the spatial distribution of transportation demand generation.

4. The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to claim 3 is characterized in that: The step S1.3 specifically includes the following steps: Step S1.3.1, set the fleet-related parameters, including: the number of drones |D| and the number of vehicles |V|, as well as the fleet working time T work ; Step S1.3.2, setting vehicle performance and driving parameters, including: vehicle speed v e , vehicle loading time T vl and the time T for vehicle to unload cargo vu ; Step S1.3.3, set the UAV performance and driving parameters, including: UAV flight speed v d , UAV charging time T charge , UAV loading cargo time T dl and the time T for drone to unload cargo du ; In step S1.3.4, the performance indicators and driving parameters of the vehicles and drones of the fleet set in the simulation environment are used as the interactive simulation environment.

5. The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to claim 4 is characterized in that: The step S2.2 specifically includes the following steps: Step S2.2.1, construct the current state space S k : time t including the kth dynamic decision point k 、Customer demand information i ={(x i ,y i ),t i ,TL i }, the planned driving route of the vehicle at the kth dynamic decision point and the planned driving route of the UAV at the kth dynamic decision point Among them, t i is the arrival time of the i-th demand, TL i c i The delivery time limit of the fleet is v∈V, where V is the set of all vehicles in the fleet. is the planned route of vehicle v, and the number of drones d∈D in the fleet, where D is the set of all drones in the fleet, The planned route for UAV d; Step S2.2.2, construct action space A k : in, Indicates t k The vehicle that is returning to the distribution center or finishing the last waiting interval is dispatched. Indicates t k The allocation of transport modes to the demand points at any given time, Based on the and Get updated planned routes for vehicles and drones; Step S2.2.3, describe the state transition process P k :Define the current state space S k Transfer to the post-decision state space and the next decision-making state space S k+1 processes and changes; Step S2.2.4, construct the reward function: construct it as a function of action and state, expressed as R k (S k ,A k ), when the transport demand is assigned to the fleet and the delivery is completed successfully, R k >0; otherwise, R k <0.

6. The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to claim 5 is characterized in that: The step S3 specifically includes the following steps: Step S3.1, define a demand buffer component, which is used to store customer demands arriving between two adjacent vehicles in the interactive simulation environment, and serves as a link between the upper-level agent and the lower-level agent; Step S3.2: Construct the upper-layer agent to obtain the DQN network, which is used to decide whether the vehicle should depart based on the time point when the dynamic decision is triggered, when the vehicle returns to the distribution center and the vehicle has completed the previous waiting interval Δt; Step S3.3: construct the lower-level agent and obtain the proximal policy optimization PPO network, which is used to allocate the newly arrived customer demands in the simulation environment according to the time point when the dynamic decision is triggered; Step S3.4: construct a path heuristic algorithm layer to update the routes of drones and vehicles in real time; In step S3.5, the decision on whether to depart a vehicle obtained through the DQN network, the allocation of newly arrived customer demands in the simulation environment obtained through the PPO network, and the real-time update of the driving routes of drones and vehicles by constructing a path heuristic algorithm layer are used as a collaborative scheduling strategy obtained through the hierarchical reinforcement learning algorithm framework.

7. The vehicle-drone collaborative scheduling method based on hierarchical reinforcement learning according to claim 6 is characterized in that: The step S3.2 specifically includes the following steps: Step S3.2.1, based on the state space S in the constructed vehicle-drone dynamic scheduling model k , extract the state features as the input of the DQN network, the state features include the current decision time t k The earliest return time of the drone according to the planned route The earliest return time of the vehicle according to the planned route and the minimum remaining delivery time required for the vehicle's planned route Step S3.2.2, construct the action space of the DQN network Step S3.2.3, construct the reward function of the DQN network; In step S3.2.4, based on the input state characteristics, the dimensions of the action space of the DQN network and the reward function of the DQN network are constructed, and the DQN network structure and parameters are configured: the number of hidden layers, the number of neurons, the learning rate, the exploration rate, and the discount factor are set to obtain the DQN network, and the vehicle is decided whether to depart through the DQN network.

8. The vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning according to claim 7 is characterized in that: The step S3.3 specifically includes the following steps: Step S3.3.1, based on the state space S in the constructed vehicle-drone dynamic scheduling model k , extract the state features as the input of the PPO network, the state features include: current transportation demand c i Arrival time t i , the straight-line distance from the transportation demand to the distribution center The earliest return time of the drone according to the planned route The earliest return time of the vehicle according to the planned route Vehicle delivery demandc i Feasibility V (c i ), drone delivery demand c i Feasibility D (c i ), c i Minimum insertion time cost assigned to a vehicle Step S3.3.2, construct the action space of the PPO network and obtain the decision c i Mode of transport; Step S3.3.3, construct the reward function of the PPO network; In step S3.3.4, according to the input state characteristics and the dimension of the action space of the constructed PPO network, the PPO network structure and parameters are configured: the number of hidden layers, the number of neurons, the learning rate, the weights in the loss function, and the discount factor are set to obtain the PPO network, and the newly arrived customer demands in the simulation environment are distributed through the PPO network.

9. The vehicle-UAV collaborative scheduling method based on hierarchical reinforcement learning according to claim 8 is characterized in that: The step S3.4 specifically includes the following steps: Step S3.4.1, use the first-come-first-served heuristic algorithm to plan the route of the UAV obtained in step S2.2.1 Planning includes arranging deliveries in the order of arrival of requests, prioritizing drones that are idle and have returned to the distribution center the earliest. Each drone only serves one transport request at a time, and the drone's route is updated in real time. Step S3.4.2, use the minimum insertion cost method to calculate the vehicle route obtained in step S2.2.1 Planning is performed, including giving priority to vehicle routes that meet feasibility requirements, that is, routes that can successfully complete delivery before the delivery time limit and minimize the added time cost, as well as selecting insertion points with the smallest added cost, and updating the vehicle's route in real time.

Images