Unmanned aerial vehicle and truck collaborative distribution method and system based on multi-target reinforcement learning

Abstract

The invention belongs to the technical field of logistics distribution, and particularly relates to an unmanned aerial vehicle and truck collaborative distribution method based on multi-target reinforcement learning, and the method comprises the steps: order clustering: a logistics service provider carries out the clustering of orders according to a K-means method, enables the orders with the similar distribution time and address to be divided into one order cluster, and carries out the clustering of the orders according to a K-means method; the unmanned aerial vehicles are delivered by the truck and the unmanned aerial vehicles carried by the truck; according to the NSGA-II algorithm, multiple targets such as path cost, distribution time and carbon emission are considered, and optimal paths of multiple schemes are provided; and planning an optimal path by adopting a reinforcement learning method. According to the cooperative logistics distribution method based on the unmanned aerial vehicle and the truck, the three targets of logistics distribution time, cost and carbon emission are considered, the dynamic conditions of low-altitude traffic restriction of the unmanned aerial vehicle, the road congestion condition of the truck, load constraint and the like are also considered, the optimal path is optimized by adopting a reinforcement learning mode, and the logistics distribution efficiency is improved. And parameter setting of reinforcement learning is automatically updated by using a non-dominated sorting algorithm, so that multi-dimensional optimal path selection can be provided for logistics service providers.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of logistics and distribution technology, and in particular relates to a "drone + truck" collaborative distribution method and system based on multi-objective reinforcement learning. Background Art

[0002] The rapid development of drone technology is driving the low-altitude economy toward intelligent and collaborative development. The Chinese government has also issued various powerful policies to promote its development. Drones offer advantages in cargo delivery, including high speed, geographic independence, environmental friendliness, and energy efficiency. However, they also have limitations, such as limited range and payload. Using drones alone cannot meet the demands of modern logistics. Trucks, while offering large payloads and endurance, are slower, complementing drone delivery methods. The "drone + truck" delivery model can shorten truck-based delivery times while expanding the truck's service range. Trucks can also perform operations such as restocking drones and replacing batteries. The economic and environmental benefits of this delivery model have been well-proven.

[0003] Logistics and delivery must consider not only economic and user satisfaction goals, but also environmental objectives. These three objectives often trade off against each other. For example, to achieve high user satisfaction, short delivery times are crucial. Drone delivery is ideal, but this approach is costly and falls short of the logistics service provider's expectations. The reverse is also true. Existing methods for planning delivery routes consider only one or two objectives, neglecting situations like road congestion or drone bans. How to comprehensively consider economic, user satisfaction, and environmental objectives in logistics and delivery, and plan optimal routes for different scenarios, is a pressing issue.

[0004] Several researchers have studied multi-objective logistics delivery solutions using drones and trucks. Zhang et al. constructed a path planning model that minimizes economic costs and truck energy consumption, and proposed a non-dominated sorting genetic algorithm to solve the optimal Pareto front. Luo et al. incorporated factors such as drone weight, payload, and flight distance into energy consumption calculations. With the goal of minimizing the sum of drone energy consumption and other costs, they proposed an improved artificial bee colony algorithm to solve the optimal path. Zhong Daofeng et al. proposed a two-stage path optimization method for emergency logistics based on truck-drone collaboration. They used K-means clustering to determine the location and regional division of emergency warehouses and distribution centers, and employed a variable neighborhood return simulation method to plan the path. Zhang Lianmin et al. proposed a path planning method for collaborative delivery between trucks and drones, considering the order delivery delay risk index and using graph theory to solve the path. Ma Fei proposed a multi-objective optimization method for collaborative delivery based on a TBL perspective, comprehensively considering delivery cost, delivery time, and carbon emissions, and solving the model using an extended non-dominated sorting genetic algorithm.

[0005] Most existing studies consider one or two objectives, such as minimizing total delivery costs, shortening delivery time, or maximizing user satisfaction, without considering path planning between multiple conflicting objectives. Furthermore, most studies do not consider constraints such as real-time road congestion for trucks and restricted areas for drones when solving path planning problems. Consequently, these methods cannot be applied in real-world scenarios. Summary of the Invention

[0006] In response to the problems existing in the existing technology, the present invention provides a "drone + truck" collaborative delivery method based on multi-objective reinforcement learning.

[0007] The present invention is implemented as follows: a "drone + truck" collaborative delivery method based on multi-objective reinforcement learning, the method comprising:

[0008] S1: Order clustering: The logistics service provider clusters orders using the K-means method, so that orders with similar delivery times and addresses can be grouped into a single order cluster and delivered by trucks and their drones.

[0009] S2: NSGA-II algorithm. The NSGA-II algorithm optimizes the parameter combination, including the path cost weight ω_c, the time consumption weight ω_t, the carbon emission weight ω_e, and the reinforcement learning parameters learning rate α, discount factor γ, and exploration rate ε. The concept of Pareto optimal frontier is introduced to help logistics service providers make trade-offs between different objectives based on their specific needs and preferences.

[0010] S3: Use reinforcement learning method to plan the optimal path.

[0011] Furthermore, the S1 specifically includes:

[0012] S11: Feature Extraction

[0013] The k-means method divides order features into three parts: geographic location, delivery time, and order priority. The feature vector of each order is (x, y, t, p), where x represents the longitude of the order location, y represents the latitude of the order location, t represents the remaining time window, and p represents the order priority. The order priority is determined by the customer's priority and whether the order delivery is time-sensitive. If the goods are blood, medicine, fresh food, etc., the order priority is higher, and if the goods are daily necessities, the order priority is lower.

[0014] S12: Feature Normalization

[0015] The latitude and longitude coordinates of the order delivery location, the remaining time window, and the order priority are normalized using the min-max method to control the eigenvalues ​​within the range of [0,1].

[0016] S13: Distance formula

[0017] The distance formula between two orders is defined as formula (1):

[0018]

[0019] Where d(A,B) represents the distance between order A and order B, d geo (A,B) represents the Euclidean distance between the geographical locations of order A and order B, d t (A, B) represents the delivery time requirements of order A and order B, α represents the weight of time distance, t A With t B Represents the remaining delivery time for order A and order B, respectively. Measuring the distance between order A and order B using this distance formula ensures that orders with close physical locations are grouped together.

[0020] S14: Clustering process of K-means method

[0021] (1) Randomly select K orders as K centers;

[0022] (2) Use formula (1) to calculate the distance between all orders and the order center;

[0023] (3) Determine whether the time difference between each order and the order center is greater than 24 hours;

[0024] (4) If yes, select the order cluster closest to the order center to join; if no, create a new order cluster with the order as the center;

[0025] (5) Recalculate the geographic center and time mean of each order cluster and use this value as the new order center;

[0026] (6) Determine whether the order cluster has converged or reached the maximum number of iterations;

[0027] (7) If yes, then the method ends; if no, then the operations (2)-(7) are repeated.

[0028] Using this method, orders are divided into at least K order clusters, each of which is delivered by one or more trucks and their drones. The specific delivery path is given by the following method, which takes into account the cost, time, and carbon emissions of delivery, ensuring that the logistics company saves costs while meeting customer delivery time requirements.

[0029] Furthermore, the S2 specifically includes:

[0030] S21: Path allocation objectives. The objectives considered for the "drone + truck" path include path cost C, time TI, and carbon emissions E. Path cost C includes the truck's fuel cost, labor cost, and depreciation costs of the drone and truck. Since drones are powered by electricity and have a low unit power consumption cost, this invention does not currently consider the drone's power cost. If the logistics service provider requires it, this cost can be included without changing the process.

[0031] S22: Based on the target characterization of delivery path cost, delivery time and comprehensive carbon emissions, the NSGA-II algorithm is used to tune the parameter combination and output the Pareto frontier.

[0032] Furthermore, the S21 specifically includes:

[0033] (1) Truck t’s fuel cost C1 t The calculation method is formula (2):

[0034]

[0035] Among them, η F represents the price of a unit of fuel, = represents the fuel consumption of truck t from point i to point j. The fuel consumption of a truck is related to the specific model parameters of the truck, its own weight, load, and speed. Therefore, referring to the truck fuel consumption quantitative model constructed by Barth, the fuel consumption of a truck is calculated as formula (3):

[0036]

[0037] in, represents the specific model parameters of the UAV or truck (obtained through the output power of the engine or motor and other characteristics), vt represents the speed of the truck (km / h), W T Indicates the weight of the truck when it is unloaded, W i t represents the load of truck t when it departs from point i, It represents the distance from point i to point j for truck t, which is obtained by the optimized path length obtained by reinforcement learning. T represents the set of trucks, and V represents the set of all destinations.

[0038] (2) The labor cost of the delivery person is fixed, represented by C2, and can also be calculated based on the historical costs of the logistics service provider.

[0039] (3) The depreciation cost of the drone is related to the flight mileage and unit flight cost, expressed as C3 d The specific calculation method is formula (4):

[0040]

[0041] where η d Indicates the flight cost per unit distance of the drone.

[0042] (4) The depreciation cost of a truck is related to its mileage and the cost per unit distance, expressed as C4 t The calculation method is formula (5):

[0043]

[0044] where η t Represents the flight cost per unit distance of the truck.

[0045] Therefore, the path cost of order delivery is C, which is calculated using formula (6):

[0046]

[0047] Time TI, assuming that the drone and the truck are moving at a constant speed, the time cost of using the truck is TI T The calculation method is formula (7):

[0048]

[0049] Where vt is the speed of the truck traveling at a constant speed.

[0050] The time cost of using drones D The calculation method is formula (8):

[0051]

[0052] Among them, vd represents the speed of the drone moving at a constant speed.

[0053] Therefore, the calculation method for the time spent on all orders is formula (9):

[0054] TI=TI T +TI D (9)

[0055] Carbon emissions E, the carbon emissions E of the “drone + truck” logistics distribution path, consists of the carbon emissions of the truck and the carbon emissions of the drone;

[0056] The carbon emissions of a truck are linearly related to fuel consumption. The total carbon emissions are calculated by multiplying the carbon emission rate per unit of fuel consumption by the total fuel consumption of the truck. UAVs use clean energy functions, so their carbon emissions can be ignored. Therefore, the total carbon emissions E are calculated as formula (10):

[0057]

[0058] Furthermore, the S22 specifically includes:

[0059] (1) Obtain the results of order clustering and obtain each order cluster. The order information includes the location of the order and the estimated delivery time;

[0060] (2) Randomly generate N chromosomes, the contents of which are parameters (ω_c, ω_t, ω_e, α, γ, ε), where ω_c+ω_t+ω_e=1, α<1, γ<1, ε<1, the number of initialization iterations g=0, ω_c is the weight coefficient of the cost of the "drone+truck" planning path in reinforcement learning, ω_t is the weight coefficient of the time spent on planning the path, ω_e is the weight coefficient of the carbon emissions of the planning path, α is the learning rate in the reinforcement learning algorithm, γ is the attenuation coefficient in the reinforcement learning algorithm, and ε is the exploration probability in the reinforcement learning algorithm;

[0061] (3) Input the parameters of a chromosome into the reinforcement learning process to obtain an optimal path;

[0062] (4) Select N chromosomes using the tournament selection strategy. When selecting, give priority to chromosomes with lower non-dominated rankings. If two chromosomes are in the same level, the chromosome with higher crowding is selected. If this is the first selection, the chromosome is selected randomly.

[0063] (5) Use the simulated binary crossover (SBX) method to cross the selected chromosomes, generate N-sized daughter chromosomes after crossover, and normalize the first three parameters of each chromosome to ensure that the sum of the weights of the first three objectives is 1;

[0064] (6) The chromosomes of the parent and offspring are merged;

[0065] (7) Calculate the path cost, time consumption, and energy consumption of the optimal path obtained by reinforcement learning under the parameters corresponding to each chromosome;

[0066] (8) Calculate the path cost, delivery order time and comprehensive carbon emissions for all chromosome parameters using formulas (6)(9)(10), and perform a quick dominance ranking of these three objectives;

[0067] (9) Calculate the crowding degree of each chromosome parameter;

[0068] (10) The number of iterations is increased by 1, and it is determined whether the maximum number of iterations has been reached. If so, the result is output; if not, the process continues with step (4).

[0069] Through this method, the Pareto optimal frontier can be calculated, providing logistics service providers with a multi-objective trade-off solution.

[0070] Furthermore, the S3 specifically includes:

[0071] (1) Initialize the aircraft and truck agents for each order cluster, and set the action space, state space, and reward function of the drone and truck respectively:

[0072] The action space of the drone includes [take-off, landing, unloading, loading];

[0073] The action space of the truck is [forward, turn left, turn right, reverse, wait, unload, load, release drone];

[0074] The state space of the drone is [x, y, z, lc, g], where x is the longitude coordinate of the drone, y is the latitude coordinate of the drone, z is the altitude coordinate of the drone, lc is the remaining flight time of the drone, and g is the delivery order of the drone;

[0075] The state space of the truck is [x,y,Ld,Lg], where x is the longitude of the truck's location, y is the latitude of the truck's location, Ld is the list of drones carried by the truck, and Lg is the list of cargo carried by the truck;

[0076] The reward function is formula (11):

[0077]

[0078] in, Indicates that the cargo carried by the drone cannot exceed the rated load, W g is the weight of an order, W d is the weight of a drone;

[0079] The time taken by truck t with coordinates (i, j) to deliver order g cannot exceed the time required by the order. The calculation method is formula (12):

[0080]

[0081] Among them (g i ,g j ) is the location coordinate of the cargo g, vt is the speed of the truck;

[0082] W g >Qd means that the cargo load carried by drone d cannot exceed the rated load of the drone;

[0083] The energy required by the drone t at coordinate (i, j) to deliver order g cannot exceed the remaining energy of the drone;

[0084] The time required for drone t with coordinates (i, j) to deliver order g cannot exceed the time required by the order;

[0085] When the above requirements are met, the reward value of action a of the truck carrying the drone in state s is ω_c*C+ω_t*TI+ω_e*E.

[0086] (2) Initialize the starting positions of the drones and trucks in the order cluster, as well as the positions of all orders in the order cluster, and initialize the values ​​of the state-action table Q(s,a).

[0087] (3) Use the order priority and the ε-greedy method to select the next action based on the state-action table Q(s,a). Sort the order priorities from high to low, and plan the paths for the ones with higher priorities first. The path selection action is completed by the ε-greedy method. The specific method is as follows: use a random number generated. If the random number is less than ε or the value of the Q(s,a) table is empty, then a random action is selected. Otherwise, the action a' with the largest value in the current Q(s,a) table is selected. ε is the value determined by the chromosome in the NSGA-II algorithm.

[0088] (4) Calculate the reward value reward obtained by action a'.

[0089] (5) Update the Q(s,a) table, and calculate it using formula (13):

[0090] Q(s,a)=(1-α)*Q(s,a)+α*(reward+γ*max a Q(s_new,a)) (13)

[0091] Among them, s_new is the state in the Q(s,a) table corresponding to a', and the values ​​of α and γ are the values ​​determined by the chromosome in the NSGA-II algorithm.

[0092] (6) The number of iterations is increased by 1, and it is determined whether the maximum number of iterations has been reached or all orders have been delivered. If so, the algorithm ends and the optimal path corresponding to the current parameters is obtained. Otherwise, step (3) is continued.

[0093] Another object of the present invention is to provide a working system for the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning, the system specifically comprising:

[0094] In the order clustering module, the logistics service provider clusters orders using the K-means method, so that orders with similar delivery times and addresses can be grouped into a single order cluster and then delivered by trucks and their drones.

[0095] The NSGA-II algorithm module optimizes the "drone + truck" route allocation based on a multi-objective reinforcement learning method. The NSGA-II algorithm tunes a combination of parameters, including the path cost weight ω_c, the time weight ω_t, the carbon emission weight ω_e, and the reinforcement learning parameters learning rate α, discount factor γ, and exploration rate ε. The concept of the Pareto optimal frontier is introduced to help logistics service providers make trade-offs between different objectives based on their specific needs and preferences.

[0096] The path planning module uses reinforcement learning methods to plan the optimal path.

[0097] Another object of the present invention is to provide a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the working method of the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning.

[0098] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the working method of the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning.

[0099] Another object of the present invention is to provide an information data processing terminal, which is used to implement the "drone + truck" collaborative distribution method working system based on multi-objective reinforcement learning.

[0100] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0101] First, the "drone + truck" collaborative logistics distribution method proposed in this invention takes into account the three goals of logistics distribution: time, cost, and carbon emissions. It also considers dynamic constraints such as drones' low-altitude restrictions, trucks' road congestion conditions, and load constraints. It uses reinforcement learning to optimize the search for the optimal path and uses a non-dominated sorting algorithm to automatically update the reinforcement learning parameter settings, which can provide logistics service providers with multi-dimensional optimal path selection.

[0102] 1. Path Planning Method Framework. Orders are first clustered using the K-means method to ensure that a cluster of orders is delivered by a "truck + drone" approach. Path planning for a cluster of orders is accomplished using the NSGA-II combined with reinforcement learning method. The NSGA-II algorithm automatically generates multi-objective weights and reinforcement learning algorithm parameters. It also performs non-dominated sorting based on the calculation methods of the three objectives, providing logistics service providers with multiple solutions. Reinforcement learning uses the parameters provided by NSGA-II to plan the optimal path.

[0103] 2. NSGA-II algorithm framework: This invention does not randomly generate paths, but rather performs non-dominated sorting on reinforcement learning parameters and multi-objective weights, ensuring that logistics service providers are provided with multi-dimensional delivery solutions.

[0104] 3. Reinforcement learning framework. This invention considers the no-fly zones for drones and the real-time congestion of roads when initializing the reinforcement learning environment. The load conditions of drones and trucks are used as constraints to update the reward function of reinforcement learning to find the optimal path under specific weights.

[0105] 4. The present invention is extensible. If there are new objective functions or new path constraints in the future, it is only necessary to add calculation methods in specific steps without changing the overall architecture of the method.

[0106] Second, the technical solution of the present invention solves a technical problem that people have long been eager to solve but have never been able to successfully solve:

[0107] 1. This paper considers a multi-objective path planning method in a "truck + drone" delivery environment. This method considers three common conflicting delivery objectives: cost, time, and carbon emissions, performs non-dominated sorting on the planned paths, and provides logistics service providers with multiple delivery solutions.

[0108] 2. This method first clusters orders to ensure that orders with similar delivery locations and times can be divided into one order cluster. When planning the route, only the orders in this order cluster need to be planned, which reduces the algorithm running time and improves the operating efficiency of the method.

[0109] 3. This method uses reinforcement learning method for path planning, and the parameter setting is given by the NSGA-II algorithm, which reduces manual intervention and improves the accuracy of path planning.

[0110] 4. This method is scalable. If a logistics service provider needs to add a delivery target, it only needs to provide a calculation method for the target without having to modify the framework structure of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 It is a multi-objective reinforcement learning path planning algorithm framework provided by an embodiment of the present invention;

[0112] Figure 2 This is the clustering process of the K-means method provided by the embodiment of the present invention;

[0113] Figure 3 This is the NSGA-II algorithm process provided by an embodiment of the present invention;

[0114] Figure 4 is a reinforcement learning flow chart provided by an embodiment of the present invention;

[0115] Figure 5 This is a schematic diagram of multi-path collaborative path planning based on "truck + drone". DETAILED DESCRIPTION

[0116] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0117] like Figure 1 As shown, an embodiment of the present invention provides a "drone + truck" collaborative delivery method based on multi-objective reinforcement learning, the method comprising:

[0118] S1: Order clustering: The logistics service provider clusters orders using the K-means method, so that orders with similar delivery times and addresses can be divided into a cluster and delivered by trucks and drones;

[0119] S2: NSGA-II algorithm, based on a multi-objective reinforcement learning method, optimizes the "drone + truck" route allocation. The NSGA-II algorithm tunes a combination of parameters, including the path cost weight ω_c, the time weight ω_t, the carbon emission weight ω_e, and the reinforcement learning parameters learning rate α, discount factor γ, and exploration rate ε. The concept of the Pareto optimal frontier is introduced to help logistics service providers make trade-offs between different objectives based on their specific needs and preferences.

[0120] S3: Use reinforcement learning method to plan the optimal path.

[0121] Furthermore, the S1 specifically includes:

[0122] S11: Feature Extraction

[0123] The k-means method divides order features into three parts: geographic location, delivery time, and order priority. The feature vector of each order is (x, y, t, p), where x represents the longitude of the order location, y represents the latitude of the order location, t represents the remaining time window, and p represents the order priority. The order priority is determined by the customer's priority and whether the order delivery is time-sensitive. If the goods are blood, medicine, fresh food, etc., the order priority is higher, and if the goods are daily necessities, the order priority is lower.

[0124] S12: Feature Normalization

[0125] The latitude and longitude coordinates of the order delivery location, the remaining time window, and the order priority are normalized using the min-max method to control the eigenvalues ​​within the range of [0,1].

[0126] S13: Distance formula

[0127] The distance formula between two orders is defined as:

[0128]

[0129] Where d(A,B) represents the distance between order A and order B, d geo (A,B) represents the Euclidean distance between the geographical locations of order A and order B, d t (A, B) represents the delivery time requirements of order A and order B, α represents the weight of time distance, t A With tB Represents the remaining delivery time for order A and order B, respectively. Measuring the distance between order A and order B using this distance formula ensures that orders with close physical locations are grouped together.

[0130] S14: Clustering process of K-means method, such as Figure 2 As shown;

[0131] (1) Randomly select K orders as K centers;

[0132] (2) Use formula (1) to calculate the distance between all orders and the order center;

[0133] (3) Determine whether the time difference between each order and the order center is greater than 24 hours;

[0134] (4) If yes, select the order cluster closest to the order center to join; if no, create a new order cluster with the order as the center;

[0135] (5) Recalculate the geographic center and time mean of each order cluster and use this value as the new order center;

[0136] (6) Determine whether the order cluster has converged or reached the maximum number of iterations;

[0137] (7) If yes, then the method ends; if no, then the operations (2)-(7) are repeated.

[0138] Using this method, orders are divided into at least K order clusters, each of which is delivered by one or more trucks and their drones. The specific delivery path is given by the following method, which takes into account the cost, time, and carbon emissions of delivery, ensuring that the logistics company saves costs while meeting customer delivery time requirements.

[0139] The S2 specifically includes:

[0140] S21: Path allocation objectives. The objectives considered for the "drone + truck" path include path cost C, time TI, and carbon emissions E. Path cost C includes the truck's fuel cost, labor cost, and depreciation costs of the drone and truck. Since drones use electricity and have a low unit power consumption cost, this invention does not currently consider the drone's power cost. If the logistics service provider requires it, this cost can be included without changing the process.

[0141] S22: Based on the target characterization of delivery path cost, delivery time and comprehensive carbon emissions, logistics service providers can independently weigh the paths of these three target weights, that is, use the NSGA-II algorithm to tune the parameter combination and output the Pareto frontier.

[0142] The S21 specifically includes:

[0143] (1) Truck t’s fuel cost C1 t The calculation method is formula (2):

[0144]

[0145] Among them, η F represents the price of a unit of fuel, The fuel consumption of truck t from point i to point j is expressed as follows. The fuel consumption of the truck is related to the specific model parameters of the truck, its own weight, load, and speed. Therefore, referring to the truck fuel consumption quantitative model constructed by Barth, the fuel consumption of the truck is calculated according to formula (3).

[0146]

[0147] in, represents the specific model parameters of the UAV or truck (obtained through the output power of the engine or motor and other characteristics), vt represents the speed of the truck (km / h), W T Indicates the weight of the truck when it is unloaded, W i t represents the load of truck t when it departs from point i, It represents the distance from point i to point j for truck t, which is obtained by the optimized path length obtained by reinforcement learning. T represents the set of trucks, and V represents the set of all destinations.

[0148] (2) The labor cost of the delivery person is fixed, represented by C2, and can also be calculated based on the historical costs of the logistics service provider.

[0149] (3) The depreciation cost of the drone is related to the flight mileage and unit flight cost, expressed as C3 d The specific calculation method is formula (4):

[0150]

[0151] where η d Indicates the flight cost per unit distance of the drone.

[0152] (4) The depreciation cost of a truck is related to its mileage and the cost per unit distance, expressed as C4 t The specific calculation method is formula (5);

[0153]

[0154] where η t Represents the flight cost per unit distance of the truck.

[0155] Therefore, the path cost of order delivery is C calculated using formula (6).

[0156]

[0157] Time TI, assuming that the drone and the truck are moving at a constant speed, the time cost of using the truck is TI T The calculation is formula (7):

[0158]

[0159] Where vt is the speed of the truck traveling at a constant speed.

[0160] The time cost of using drones D The calculation method is formula (8):

[0161]

[0162] Among them, vd represents the speed of the drone moving at a constant speed.

[0163] Therefore, the time taken for all orders is calculated as formula (9):

[0164] TI=TI T +TI D (9)

[0165] Carbon emissions E, the carbon emissions E of the “drone + truck” logistics distribution path, consists of the carbon emissions of the truck and the carbon emissions of the drone;

[0166] The carbon emissions of a truck are linearly related to fuel consumption. The total carbon emissions are calculated by multiplying the carbon emission rate per unit of fuel consumption by the total fuel consumption of the truck. UAVs use clean energy functions, so their carbon emissions can be ignored. Therefore, the calculation formula for the total carbon emissions E is formula (10):

[0167]

[0168] like Figure 3 As shown, the S22 specifically includes:

[0169] (1) Obtain the results of order clustering and obtain each order cluster. The order information includes the location of the order and the estimated delivery time;

[0170] (2) Randomly generate N chromosomes, the contents of which are parameters (ω_c, ω_t, ω_e, α, γ, ε), where ω_c+ω_t+ω_e=1, α<1, γ<1, ε<1, the number of initialization iterations g=0, ω_c is the weight coefficient of the cost of the "drone+truck" planning path in reinforcement learning, ω_t is the weight coefficient of the time spent on planning the path, ω_e is the weight coefficient of the carbon emissions of the planning path, α is the learning rate in the reinforcement learning algorithm, γ is the attenuation coefficient in the reinforcement learning algorithm, and ε is the exploration probability in the reinforcement learning algorithm;

[0171] (3) Input the parameters of a chromosome into the reinforcement learning process to obtain an optimal path.

[0172] (4) Select N chromosomes using the tournament selection strategy. When selecting, give priority to chromosomes with lower non-dominated rankings. If two chromosomes are in the same level, the chromosome with higher crowding is selected. If this is the first selection, the chromosome is selected randomly.

[0173] (5) Use the simulated binary crossover (SBX) method to cross the selected chromosomes, generate N-sized daughter chromosomes after crossover, and normalize the first three parameters of each chromosome to ensure that the sum of the weights of the first three objectives is 1;

[0174] (6) The chromosomes of the parent and offspring are merged;

[0175] (7) Calculate the path cost, time consumption, and energy consumption of the optimal path obtained by reinforcement learning under the parameters corresponding to each chromosome;

[0176] (8) Calculate the path cost, delivery order time and comprehensive carbon emissions for all chromosome parameters using formulas (6)(9)(10), and perform a quick dominance ranking of these three objectives;

[0177] (9) Calculate the crowding degree of each chromosome parameter;

[0178] (10) The number of iterations is increased by 1, and it is determined whether the maximum number of iterations has been reached. If so, the result is output; if not, step (4) is continued.

[0179] Through this method, the Pareto optimal frontier can be calculated, providing logistics service providers with a multi-objective trade-off solution.

[0180] like Figure 4 As shown, the S3 specifically includes:

[0181] (1) Initialize the aircraft and truck agents for each order cluster, and set the action space, state space, and reward function of the drone and truck respectively:

[0182] The action space of the drone includes [take-off, landing, unloading, loading];

[0183] The action space of the truck is [forward, turn left, turn right, reverse, wait, unload, load, release drone];

[0184] The state space of the drone is [x, y, z, lc, g], where x is the longitude coordinate of the drone, y is the latitude coordinate of the drone, z is the altitude coordinate of the drone, lc is the remaining flight time of the drone, and g is the delivery order of the drone;

[0185] The state space of the truck is [x,y,Ld,Lg], where x is the longitude of the truck's location, y is the latitude of the truck's location, Ld is the list of drones carried by the truck, and Lg is the list of cargo carried by the truck.

[0186] The reward function is formula (11):

[0187]

[0188] in, Indicates that the cargo carried by the drone cannot exceed the rated load. g is the weight of an order, W d is the weight of a drone;

[0189] The time taken by truck t with coordinates (i, j) to deliver order g cannot exceed the time required by the order. The calculation method is formula (12).

[0190]

[0191] Among them (g i ,g j ) is the location coordinate of the cargo g, v t is the truck's travel speed;

[0192] wg>Qd means that the cargo load carried by drone d cannot exceed the rated load of the drone;

[0193] The energy required by the drone t at coordinate (i, j) to deliver order g cannot exceed the remaining energy of the drone;

[0194] The time required for drone t with coordinates (i, j) to deliver order g cannot exceed the time required by the order;

[0195] When the above requirements are met, the reward value of action a of the truck carrying the drone in state s is ω_c*C+ω_t*TI+ω_e*E.

[0196] (2) Initialize the starting positions of the drones and trucks in the order cluster, as well as the positions of all orders in the order cluster, and initialize the values ​​of the state-action table Q(s,a).

[0197] (3) Use the order priority and the ε-greedy method to select the next action based on the state-action table Q(s,a). Sort the order priorities from high to low, and plan the paths for the orders with higher priorities first. The action selection for the specific path is completed by the ε-greedy method. The specific method is as follows: use a random number to generate a random number. If the random number is less than ε or the value of the Q(s,a) table is empty, then a random action is selected. Otherwise, the action a' with the largest value in the current Q(s,a) table is selected. ε is the value determined by the chromosome in the NSGA-II algorithm.

[0198] (4) Calculate the reward value reward obtained by action a'.

[0199] (5) Update the Q(s,a) table, and calculate it using formula (13):

[0200] Q(s,a)=(1-α)*Q(s,a)+α*(reward+γ*max a Q(s_new,a)) (13)

[0201] Among them, s_new is the state in the Q(s,a) table corresponding to a', and the values ​​of α and γ are the values ​​determined by the chromosome in the NSGA-II algorithm.

[0202] (6) The number of iterations is increased by 1, and it is determined whether the maximum number of iterations has been reached or all orders have been delivered. If so, the algorithm ends and the optimal path corresponding to the current parameters is obtained. Otherwise, continue to step (3).

[0203] Through this method, the optimal path under specific parameters can be obtained, and the operation efficiency of the algorithm will be improved.

[0204] 1. Specific application fields or related products of the present invention.

[0205] An embodiment of the present invention provides a working system for the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning, which specifically includes:

[0206] In the order clustering module, the logistics service provider clusters orders using the K-means method, so that orders with similar delivery times and addresses can be grouped into a single order cluster and then delivered by trucks and their drones.

[0207] The NSGA-II algorithm module optimizes the "drone + truck" route allocation based on a multi-objective reinforcement learning method. The NSGA-II algorithm tunes a combination of parameters, including the path cost weight ω_c, the time weight ω_t, the carbon emission weight ω_e, and the reinforcement learning parameters learning rate α, discount factor γ, and exploration rate ε. The concept of the Pareto optimal frontier is introduced to help logistics service providers make trade-offs between different objectives based on their specific needs and preferences.

[0208] The path planning module uses reinforcement learning methods to plan the optimal path.

[0209] An embodiment of the present invention provides a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the working method of the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning.

[0210] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the working method of the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning.

[0211] An embodiment of the present invention provides an information data processing terminal, which is used to implement the "drone + truck" collaborative distribution method working system based on multi-objective reinforcement learning.

[0212] The present invention is based on Solomon's example set and runs a "drone + truck" system delivery method based on multi-objective reinforcement learning. The designed parameter table is shown in Table 1.

[0213] Table 1 Parameter values ​​involved in the “drone + truck” collaborative delivery method based on multi-objective reinforcement learning

[0214]

[0215] Run the method proposed by the present invention, and the path diagram of one of the orders is as follows: Figure 5As shown. The red path is the path planned by pure trucks. Since trucks cannot reach places close to the destination directly, this solution takes a long time and has high carbon emissions, but the cost is low. The blue path is the second planned path. This path uses drones for delivery at the coordinates (40,60). Therefore, it costs a lot, but the delivery time is the shortest and the carbon emissions are the smallest. The green path is the third planned path. Compared with the first path, it takes less time, has a higher cost, and has lower carbon emissions. Compared with the first path, it takes longer time, but has a lower cost. Figure 5 shown.

[0216] The calculation results of delivery cost, time and carbon emissions of these three routes are as follows:

[0217] path Cost (yuan) Time (hours) Carbon emissions (kg) Pure Truck 2.89 2.41 0.84 Truck + Drone Path 1 5.23 1.56 0.55 Truck + Drone Path 2 4.12 1.87 0.65

[0218] In order to verify the effectiveness of the method proposed in this invention, the collaborative distribution method based on multi-objective reinforcement learning proposed in this invention is compared with the existing methods as shown in Table 2.

[0219] Table 2 Total cost (yuan), average delivery time (minutes), and total carbon emissions (kg) of different methods under different order sizes.

[0220]

[0221]

[0222] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0223] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A "drone + truck" collaborative delivery method based on multi-objective reinforcement learning, characterized by: The method includes: S1: Order clustering: The logistics service provider clusters orders using the K-means method, so that orders with similar delivery times and addresses can be divided into a cluster and delivered by trucks and drones; S2: The NSGA-II algorithm, based on a multi-objective reinforcement learning approach, optimizes the "drone + truck" route allocation. The NSGA-II algorithm tunes a combination of parameters, including the path cost weight ω_c, the time weight ω_t, the carbon emission weight ω_e, and the reinforcement learning parameters learning rate α, discount factor γ, and exploration rate ε. The concept of the Pareto optimal frontier is introduced to help logistics service providers make trade-offs between different objectives based on their specific needs and preferences. S3: Use reinforcement learning method to plan the optimal path.

2. The "drone + truck" collaborative delivery method based on multi-objective reinforcement learning according to claim 1 is characterized in that: Said S1 specifically includes: S11: Feature Extraction The k-means method divides order features into three parts: geographic location, delivery time, and order priority. The feature vector of each order is (x, y, t, p), where x represents the longitude of the order location, y represents the latitude of the order location, t represents the remaining time window, and p represents the order priority. The order priority is determined by the customer's priority and whether the order delivery is time-sensitive. If the goods are blood, medicine, fresh food, etc., the order priority is higher, and if the goods are daily necessities, the order priority is lower. S12: Feature Normalization The latitude and longitude coordinates of the order delivery location, the remaining time window, and the order priority are normalized using the min-max method to control the eigenvalues ​​within the range of [0,1]. S13: Distance formula The distance formula between two orders is defined as: Where d(A,B) represents the distance between order A and order B, d geo (A,B) represents the Euclidean distance between the geographical locations of order A and order B, d t (A, B) represents the delivery time requirements of order A and order B, α represents the weight of time distance, t A With t B Represent the remaining delivery time of order A and order B respectively. Measuring the distance between order A and order B using this distance formula ensures that orders with close physical locations are grouped together. S14: Clustering process of K-means method (1) Randomly select K orders as K centers; (2) Use formula (1) to calculate the distance between all orders and the order center; (3) Determine whether the time difference between each order and the order center is greater than 24 hours; (4) If yes, select the order cluster closest to the order center to join; if no, create a new order cluster with the order as the center; (5) Recalculate the geographic center and time mean of each order cluster and use this value as the new order center; (6) Determine whether the order cluster has converged or reached the maximum number of iterations; (7) If yes, then the method ends; if no, then the operations (2) to (7) are repeated; Using this method, orders are divided into at least K order clusters, each of which is delivered by one or more trucks and their drones. The specific delivery path is given by the following method, which takes into account the cost, time, and carbon emissions of delivery, ensuring that the logistics company saves costs while meeting customer delivery time requirements.

3. The "drone + truck" collaborative delivery method based on multi-objective reinforcement learning according to claim 1 is characterized in that: The S2 specifically includes: S21: Path allocation objectives. The objectives considered for the "drone + truck" path include path cost C, time TI, and carbon emissions E. Path cost C includes the truck's fuel cost, labor cost, and depreciation costs of the drone and truck. Because drones use electricity and have a low unit power cost, this invention does not currently consider drone power costs. If the logistics service provider requires it, this cost can be included without changing the process. S22: Based on the target characterization of delivery path cost, delivery time and comprehensive carbon emissions, the values ​​of these three targets are minimized, the NSGA-II algorithm is used to tune the parameter combination and output the Pareto frontier.

4. The "drone + truck" collaborative delivery method based on multi-objective reinforcement learning according to claim 3 is characterized in that: The S21 specifically includes: (1) Truck t’s fuel cost C1 t Calculated by formula (2); Among them, η F represents the price of a unit of fuel, Represents the fuel energy consumption of truck t from point i to point j; the fuel quantity of the truck is related to the specific model parameters of the truck, its own weight, load, and speed. Therefore, referring to the truck fuel consumption quantification model constructed by Barth, the fuel quantity of the truck is calculated according to formula (3); in, represents the specific model parameters of the UAV or truck (obtained through the output power of the engine or motor and other characteristics), vt represents the speed of the truck (km / h), W T Indicates the weight of the truck when it is unloaded, W i t represents the load of the drone or truck t when it departs from point i, represents the distance from point i to point j for truck t, obtained by the optimized path length obtained through reinforcement learning, T represents the set of trucks, and V represents the set of all destinations; (2) The labor cost of the delivery personnel is fixed, represented by C2, and can also be calculated based on the historical costs of the logistics service provider; (3) The depreciation cost of the drone is related to the flight mileage and unit flight cost, expressed as C3 d The specific calculation method is formula (4): where η d Indicates the flight cost per unit distance of the drone; (4) The depreciation cost of a truck is related to its mileage and the cost per unit distance, expressed as C4 t The specific calculation method is formula (5): where η t represents the flight cost per unit distance of the truck; Therefore, the path cost of order delivery is C, which is formula (6): Time TI, assuming that the drone and the truck are moving at a constant speed, the time cost of using the truck is TI T The calculation is formula (7): Among them, vt represents the time the truck travels at a constant speed; The time cost of using drones D The calculation method is formula (8): Among them, vd represents the time the UAV travels at a constant speed; Therefore, the time taken for all orders is calculated as formula (9): YOU=YOU T +TI D (9) Carbon emissions E: The carbon emissions E of the "drone + truck" logistics distribution route are composed of the carbon emissions of the truck and the carbon emissions of the drone; The carbon emissions of trucks are linearly related to fuel consumption. The total carbon emissions are calculated by multiplying the carbon emission rate per unit of fuel consumption by the total fuel consumption of the truck. UAVs use clean energy functions, so their carbon emissions can be ignored. The calculation formula for the total carbon emissions E is formula (10):

5. The "drone + truck" collaborative delivery method based on multi-objective reinforcement learning according to claim 3 is characterized in that: The S22 specifically includes: (1) Obtain the results of order clustering and obtain each order cluster. The order information includes the location of the order and the estimated delivery time; (2) Randomly generate N chromosomes, the contents of which are parameters (ω_c, ω_t, ω_e, α, γ, ε), where ω_c+ω_t+ω_e=1, α<1, γ<1, ε<1, the number of initialization iterations g=0, ω_c is the weight coefficient of the cost of the "drone+truck" planning path in reinforcement learning, ω_t is the weight coefficient of the time spent on planning the path, ω_e is the weight coefficient of the carbon emissions of the planning path, α is the learning rate in the reinforcement learning algorithm, γ is the attenuation coefficient in the reinforcement learning algorithm, and ε is the exploration probability in the reinforcement learning algorithm; (3) Input the parameters of a chromosome into the reinforcement learning process to obtain an optimal path; (4) Select N chromosomes using the tournament selection strategy. When selecting, give priority to chromosomes with lower non-dominated rankings. If two chromosomes are in the same level, the chromosome with higher crowding is selected. If this is the first selection, the chromosome is selected randomly. (5) Use the simulated binary crossover (SBX) method to cross the selected chromosomes, generate N-sized daughter chromosomes after crossover, and normalize the first three parameters of each chromosome to ensure that the sum of the weights of the first three objectives is 1; (6) The chromosomes of the parent and offspring are merged; (7) Calculate the path cost, time consumption, and energy consumption of the optimal path obtained by reinforcement learning under the parameters corresponding to each chromosome; (8) Calculate the path cost, delivery order time and comprehensive carbon emissions for all chromosome parameters using formulas (6)(9)(10), and perform a quick dominance ranking of these three objectives; (9) Calculate the crowding degree of each chromosome parameter; (10) The number of iterations is increased by 1, and it is determined whether the maximum number of iterations has been reached. If so, the result is output; if not, the process continues with step (4). Through this method, the Pareto optimal frontier can be calculated, providing logistics service providers with a multi-objective trade-off solution.

6. The multi-objective reinforcement learning-based "drone + truck" collaborative delivery method according to claim 1, characterized in that: The S3 specifically includes: (1) Initialize the drone and truck agents for each order cluster, setting the action space, state space, and reward function for the drone and truck respectively; The action space of the drone includes [take-off, landing, unloading, loading]; The action space of the truck is [forward, turn left, turn right, reverse, wait, unload, load, release drone]; The state space of the drone is [x, y, z, lc, g], where x is the longitude coordinate of the drone, y is the latitude coordinate of the drone, z is the altitude coordinate of the drone, lc is the remaining flight time of the drone, and g is the delivery order of the drone; The state space of the truck is [x,y,Ld,Lg], where x is the longitude of the truck's location, y is the latitude of the truck's location, Ld is the list of drones carried by the truck, and Lg is the list of cargo carried by the truck; The reward function is formula (11): in, Indicates that the cargo carried by the drone cannot exceed the rated load, W g is the weight of an order, W d is the weight of a drone; The time taken by truck t with coordinates (i, j) to deliver order g cannot exceed the time required by the order. The calculation method is formula (12): Among them (g i ,g j ) is the location coordinate of the cargo g, v t is the truck's travel speed; wg>Qd means that the cargo load carried by drone d cannot exceed the rated load of the drone; The energy required by the drone t at coordinate (i, j) to deliver order g cannot exceed the remaining energy of the drone; The time required for drone t with coordinates (i, j) to deliver order g cannot exceed the time required by the order; When the above requirements are met, the reward value of action a of the truck carrying the drone in state s is ω_c*C+ω_t*TI+ω_e*E; (2) Initialize the starting positions of the drone and truck of the order cluster, as well as the positions of all orders in the order cluster, and initialize the values ​​of the state-action table Q(s,a); (3) Use the order priority and the ε-greedy method to select the next action according to the state-action table Q(s,a); sort the order priorities from high to low, and plan the paths with higher priorities first. The specific path selection action is completed by the ε-greedy method. The specific method is: use a random number to generate a random number. If the random number is less than ε or the value of the Q(s,a) table is empty, then randomly select an action. Otherwise, select the action a' with the largest value in the current Q(s,a) table. ε is the value determined by the chromosome in the NSGA-II algorithm. (4) Calculate the reward value reward obtained by action a'; (5) Update the Q(s,a) table, and calculate it using formula (13): Q(s,a)=(1-α)*Q(s,a)+α*(reward+γ*max a Q(s_new,a)) (13) Among them, s_new is the state in the Q(s,a) table corresponding to a', and the values ​​of α and γ are the values ​​determined by the chromosome in the NSGA-II algorithm; (6) The number of iterations is increased by 1, and it is determined whether the maximum number of iterations has been reached or all orders have been delivered. If so, the algorithm ends and the optimal path corresponding to the current parameters is obtained. Otherwise, continue to step (3); Through this method, the optimal path under specific parameters can be obtained, and the algorithm's operating efficiency is also high.

7. A working system for the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning as described in claims 1-6, characterized in that: The system specifically includes: In the order clustering module, the logistics service provider clusters orders using the K-means method, so that orders with similar delivery times and addresses can be grouped into a single order cluster and then delivered by trucks and their drones. The NSGA-II algorithm module uses a multi-objective reinforcement learning approach to optimize the "drone + truck" route allocation. The NSGA-II algorithm tunes a combination of parameters, including the path cost weight ω_c, the time weight ω_t, the carbon emission weight ω_e, and the reinforcement learning parameters α, the discount factor γ, and the exploration rate ε. The concept of the Pareto optimal frontier is introduced to help logistics service providers balance different objectives based on their specific needs and preferences. The path planning module uses reinforcement learning methods to plan the optimal path.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the working method of the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the "drone + truck" collaborative delivery method based on multi-objective reinforcement learning as described in any one of claims 1 to 6.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the "drone + truck" collaborative delivery method working system based on multi-objective reinforcement learning as described in claim 7.

Images