A collaborative delivery path planning method based on deep reinforcement learning

Through the collaborative distribution path planning method based on deep reinforcement learning, the problems of low efficiency and insufficient applicability of the adaptive large neighborhood search algorithm under the needs of multiple customer points are solved, and the rapid optimization of collaborative distribution routes and operator weights are achieved. Improve delivery timeliness and solution efficiency.

CN116523161BActive Publication Date: 2024-11-08NANJING UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310538777.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-08
Estimated Expiration
2043-05-15

AI Technical Summary

Technical Problem

The existing adaptive large neighborhood search algorithm is inefficient when facing the needs of multiple customer points and the algorithm is insufficient for applicability, so it is necessary to frequently resolve the optimal route, resulting in insufficient delivery timeliness and operator weight generalization capabilities.

Method used

The collaborative distribution path planning method based on deep reinforcement learning is adopted, and the initial solution is generated by obtaining customer order information, path damage and repair are carried out, deep reinforcement learning model (DQN) is constructed, material distribution path is optimized, and solution efficiency and applicability are improved.

Benefits of technology

It has realized the rapid planning of global optimal collaborative distribution routes, improved the path optimization effect and solution efficiency, ensured timely delivery of materials, and solved the generalization ability of operator weights, and was suitable for path planning in different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116523161B_ABST
    Figure CN116523161B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of adaptive large neighborhood search, and specifically discloses a collaborative distribution path planning method based on deep reinforcement learning. By obtaining multiple customer order information, multiple customer points are determined, an initial solution is generated, and a VRCD initial path that can connect all customer points in series is constructed; nodes or paths are removed to destroy the current solution in the VRCD initial path, and route repair or route construction is performed to change the structure of the initial solution to form a new solution; according to the new solution, a DQN model is constructed; and the path of material distribution is optimized according to the DQN model. It has a good path optimization effect, and can plan a relatively short collaborative distribution path, thereby ensuring that materials are delivered to all customer points in a timely manner. It not only solves the generalization ability of operator weights, but also does not need to repeatedly train operator weights. It can also be applied to path planning in different scenarios, which greatly improves the solution efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of adaptive large neighborhood search, and in particular to a collaborative delivery path planning method based on deep reinforcement learning. Background Art

[0002] The Adaptive Large Neighborhood Search algorithm is a heuristic method proposed by Ropke and Pisinger in 2006. It adds a measure of the effect of operators on the basis of neighborhood search, so that the algorithm can automatically select good operators to destroy and repair the solution, thereby having a certain probability of obtaining a better solution.

[0003] In view of the large demand for logistics distribution and the difficulty of "last mile" delivery, the existing technology proposes a multi-vehicle and multi-UAV collaborative delivery problem with dynamic energy consumption constraints, and establishes a mixed integer programming model (MIP) with the goal of minimizing delivery time. To solve this problem, a K-means clustering and nearest neighbor collaborative initial solution generation algorithm is designed, and an adaptive large-scale neighborhood search algorithm based on problem neighborhood knowledge is proposed.

[0004] The prior art has the following defects:

[0005] 1. Low efficiency when there are many demand points. When there are many customer points, the ALNS algorithm needs to train and update the operator weights, and its solution speed and efficiency are relatively low, which may affect the timeliness of delivery;

[0006] 2. The algorithm lacks applicability. Since there are new customer demands every time, the ALNS algorithm needs to be used to re-solve the optimal route during collaborative delivery. The weight of the operator selection obtained last time has no guiding role in the next path planning and is only applicable to this path planning. Summary of the invention

[0007] The purpose of an embodiment of the present invention is to provide a collaborative delivery path planning method based on deep reinforcement learning, aiming to solve the problems raised in the background technology.

[0008] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0009] A collaborative delivery path planning method based on deep reinforcement learning, the method specifically comprising the following steps:

[0010] Obtain multiple customer order information, determine multiple customer points, generate initial solutions, and construct a VRCD initial path that can connect all customer points;

[0011] Removing nodes or paths to destroy the current solution in the VRCD initial path, and performing route repair or route construction to change the structure of the initial solution to form a new solution;

[0012] According to the new solution, a DQN model is constructed;

[0013] The path optimization of material distribution is performed according to the DQN model.

[0014] As a further limitation of the technical solution of the embodiment of the present invention, the steps of obtaining multiple customer order information, determining multiple customer points, generating an initial solution, and constructing a VRCD initial path capable of connecting all customer points in series specifically include the following steps:

[0015] Receive multiple customer orders, obtain multiple customer order information, and determine multiple customer points;

[0016] Randomly placing the plurality of customer orders into candidate batches, and generating an order batching plan;

[0017] Adopting a search strategy, analyzing the order batching scheme, and generating an initial solution;

[0018] According to the initial solution, a VRCD initial path capable of connecting all customer points in series is constructed.

[0019] As a further limitation of the technical solution of the embodiment of the present invention, the receiving of multiple customer order information, determining multiple customer points, generating an initial solution, and constructing a VRCD initial path capable of connecting all customer points in series also includes the following steps:

[0020] According to the plurality of customer order information, goods are prepared and equipped into a coordinated fleet having trucks and robots.

[0021] As a further limitation of the technical solution of the embodiment of the present invention, the removing of nodes or paths, destroying the current solution in the VRCD initial path, and performing route repair or route construction, changing the structure of the initial solution, and forming a new solution specifically includes the following steps:

[0022] Destroying the current solution in the VRCD initial path by removing nodes or paths through a plurality of destruction measures;

[0023] The route is repaired or constructed through a variety of repair measures, the structure of the initial solution is changed, and a new solution is formed.

[0024] As a further limitation of the technical solution of the embodiment of the present invention, the various sabotage measures specifically include: customer removal measures, stop removal measures, route closure measures and route sabotage measures.

[0025] As a further limitation of the technical solution of the embodiment of the present invention, in the customer removal measure, a random customer removal operation and a greedy customer removal operation are used to remove customer points; in the stop point removal measure, a robot route connecting a stop point and arriving at and leaving the stop point is randomly removed; in the route closing measure, a random truck / robot route closing operation and a greedy truck / robot route closing operation are used to remove truck / robot routes; in the route destruction measure, a random route destruction operation and a greedy route destruction operation are used to select a VRCD route from the VRCD initial path for destruction.

[0026] As a further limitation of the technical solution of the embodiment of the present invention, the multiple repair measures specifically include: route reconstruction measures, customer insertion measures and division measures.

[0027] As a further limitation of the technical solution of the embodiment of the present invention, in the route reconstruction measure, the VRCD route is reconstructed using customer points through the route reconstruction operation; in the customer insertion measure, a customer point is selected to be inserted into the VRCD route through the random customer insertion operation and the greedy customer insertion operation to reconstruct the new VRCD route; in the division measure, a new truck route is started from the connection stop point through the truck route opening operation, a new robot route is started from the stop point through the robot route opening operation, and a position in the VRCD route is selected to insert the stop point through the random stop point insertion operation and the greedy stop point insertion operation.

[0028] As a further limitation of the technical solution of the embodiment of the present invention, constructing the DQN model according to the new solution specifically includes the following steps:

[0029] According to the new solution, the estimated network weight θ and the target network weight θ are determined.

[0030] Initialize the memory capacity N and input the estimated network weight θ and target network weight

[0031] Train and output the DQN model through the preset model training algorithm.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention can quickly determine the global optimal collaborative distribution route and deliver the materials required by customers to the demand points. It has effectiveness and convergence in solving VRCD under multiple constraints. Compared with ECW and SA algorithms, it has better path optimization effect and can plan a relatively short collaborative distribution route, thereby ensuring that the materials are delivered to all customer points in a timely manner.

[0034] 2. Compared with the ALNS algorithm itself, the operator weights obtained after path planning are only applicable to the current customer demand. The next time a new customer demand is generated, the operator still needs to be re-identified and selected. The present invention uses DQN to not only solve the generalization ability of operator weights, but also make it applicable to path planning in different scenarios, and the solution efficiency is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0036] Figure 1 A flow chart of a method provided by an embodiment of the present invention is shown.

[0037] Figure 2 A schematic diagram of a VRCD initial path in a method provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] It is understandable that the existing technology has the following defects: 1. Low efficiency when there are many demand points. When the number of customer points is large, the ALNS algorithm needs to train and update the operator weights, and its solution speed and efficiency are relatively low, which may affect the timeliness of delivery; 2. The algorithm lacks applicability. Since there are new customer demands every time, the ALNS algorithm needs to be used to re-solve the optimal route during collaborative delivery. The weight of the operator selection obtained last time has no guiding role in the next path planning and is only applicable to this path planning.

[0040] To solve the above problems, the embodiment of the present invention obtains multiple customer order information, determines multiple customer points, generates an initial solution, and constructs a VRCD initial path that can connect all customer points in series; removes nodes or paths, destroys the current solution in the VRCD initial path, and performs route repair or route construction to change the structure of the initial solution to form a new solution; constructs a DQN model based on the new solution; and loads the DQN model for use. It has a good path optimization effect and can plan a relatively short collaborative distribution path, thereby ensuring that materials are delivered to all customer points in a timely manner. It not only solves the generalization ability of operator weights, but also does not require repeated training of operator weights. It can also be applied to path planning in different scenarios, which greatly improves the solution efficiency.

[0041] Figure 1 A flow chart of a method provided by an embodiment of the present invention is shown.

[0042] Specifically, in a preferred embodiment of the present invention, a collaborative delivery path planning method based on deep reinforcement learning comprises the following steps:

[0043] Step S101: Acquire multiple customer order information, determine multiple customer points, generate an initial solution, and construct a VRCD initial path that can connect all customer points in series.

[0044] In the embodiment of the present invention, multiple customer orders are received, multiple customer order information is obtained, and multiple customer points are determined. Under the premise of not violating the batch capacity constraint, the multiple customer orders are randomly placed into candidate batches in turn to generate an order batching plan. Then, according to the multiple customer order information, the corresponding quantity of goods required by the customer is prepared and equipped to the coordinated fleet of trucks-robots. The order batching plan is analyzed by adopting a suitable search strategy to generate an initial solution, and a VRCD initial path that can connect all the customer points in series is constructed, such as Figure 2 A schematic diagram of the VRCD initial path in the method provided by an embodiment of the present invention is shown, wherein triangles represent stop points, squares represent warehouse points, circles represent robot customer points / truck customer points, solid lines represent truck routes, and dotted lines represent robot routes.

[0045] It can be understood that the construction of the VRCD initial path has limited assumptions: (1) Two types of customer points are defined: the first type is the truck customer point, which is a customer point where both trucks and robots can deliver goods, and the other type is the robot customer point, where only robots are allowed to deliver goods; (2) There are specific stops (non-customer points) along the path for trucks to release and recover robots; (3) For cargo replenishment, robots can fill cargo at the warehouse or at the truck, while trucks must load cargo at the warehouse, and both trucks and robots have maximum cargo capacity; (4) The time taken for robots and trucks to interact with each other is not considered; (5) It is assumed that robots are allowed to visit multiple customers during a delivery process, and each customer point must be visited only once by a truck / robot. Customer points and warehouses have time windows, allowing free waiting at all nodes; (6) It is assumed that each truck must and can only carry one robot, and the robot released by the truck must be recovered by the same truck, and the robot cannot leave the warehouse alone to serve customers; (7) The operation time required for release, recovery, cargo replenishment, and charging preparation is not considered.

[0046] Step S102: remove nodes or paths, destroy the current solution in the VRCD initial path, and perform route repair or route construction to change the structure of the initial solution and form a new solution.

[0047] In the embodiment of the present invention, nodes or paths are removed through multiple destruction actions such as customer removal action, stop point removal action, route closing action and route destruction action, so as to destroy the current solution in the VRCD initial path, and then the route is repaired or constructed through multiple repair actions such as route reconstruction action, customer insertion action and division action, so as to change the structure of the initial solution and form a new solution.

[0048] Specifically, the operations of removing nodes or paths by multiple destructive measures are as follows:

[0049] 1. Customer removal measures, using random customer removal operations and greedy customer removal operations. The random customer removal operation removes customer points randomly, and the greedy customer removal operation removes customer points that can minimize the value of the objective function of a given route;

[0050] 2. Stop point removal measures, which remove stops from VRCD routes. Stop point-route removal operations are used to randomly remove a robot route that connects a stop point and reaches and leaves the stop point. After the stop point is removed, the route may be infeasible due to insufficient vehicle power. The infeasible route caused by insufficient vehicle power can be avoided by visiting fewer customer points or visiting more stop points. The operation removes customer points before or after the removed stop points in the truck route, or inserts stop points different from the removed stop points in the truck route to make the route feasible.

[0051] 3. Route Closure Initiative, using random truck / robot route closure operations and greedy truck / robot route closure operations to close a truck or robot route in a VRCD route;

[0052] 4. Route destruction measures, using random route destruction operations and greedy route destruction operations. The random route destruction operation randomly selects a VRCD route for destruction, and the greedy route destruction operation selects a VRCD route with the least customer points for destruction, and puts all customer points in the selected VRCD route into the unassigned customer list.

[0053] Specifically, the operations of various repair measures to repair routes or construct routes are as follows:

[0054] 1. Route reconstruction measures, which use route reconstruction operations to reconstruct the route and reconstruct the VRCD path using customer points to ensure that the algorithm can always obtain a feasible solution;

[0055] 2. Customer insertion action: insert the customer point into the VRCD route. First, use the random / greedy customer insertion operation to insert the customer point. If the customer point cannot be inserted due to insufficient battery power, try to insert a stop point before or after the inserted customer point position to make the route feasible. If all possible positions are tried but the customer point still cannot be inserted, try to construct a new robot route with the same starting and ending points and insert the customer point into the route. Finally, if there are still unassigned customer points, use the repair operation based on route reconstruction to reconstruct a new VRCD route.

[0056] 3. Divisional actions (change route structure first / insert customer actions second). Two route structure change methods are proposed: truck / robot route opening operation and random / greedy stop point insertion operation. For truck / robot route opening operation, a new truck / robot route is first started from a stop point, and then a customer point insertion action is performed. The truck / robot route includes a customer point, and the end stop point can be the same as or different from the starting stop point. For random / greedy stop point insertion operation, the stop point is first inserted into the VRCD route. If the stop point is not inserted into an independent truck route, a route opening operation is performed on this stop point, and then a repair action based on customer insertion is performed.

[0057] Step S103: construct a DQN model according to the new solution.

[0058] In the embodiment of the present invention, the estimated network weight θ and the target network weight θ are determined according to the new solution. Initialize the memory capacity N, and input the estimated network weight θ and the target network weight According to the preset model training algorithm, train and output the DQN model, where the model training algorithm is:

[0059] The number of repeat rounds is epoch = 0 to M do;

[0060] Initialize state s1;

[0061] for the number of steps in one round t = 1 to T do;

[0062] The collaborative delivery route optimization agent chooses a with probability ε t = argmaxQ(s t , a, θ) t , that is, select appropriate destruction and repair operators to perform destruction and repair operations on the initial solution;

[0063] After executing action a t After that, the agent gets the reward r(s t , a t ) and the new state s of the path optimization systemt+1 =s′ t ;

[0064] The memory D stores the previous iteration experience (s t , a t ,r(s t , a t ), s′ t );

[0065] Randomly draw a small batch of samples (s t , a t , r t , s′ t ), network training is performed, and the target network obtains the Q target value Q+(s,a);

[0066] if the termination state is reached;

[0067] Q + (s, a) = r(s t , a t );

[0068]

[0069] Calculate the loss value according to the defined DQN loss function and estimate the weight of the network according to the relationship of stochastic gradient descent;

[0070] At regular intervals, the parameters of the target network are updated with the parameters of the estimated network. Set as

[0071] end

[0072] until the iteration termination condition is reached.

[0073] Step S104: Optimize the material distribution path according to the DQN model.

[0074] In an embodiment of the present invention, the trained DQN model is saved, and when in use, the DQN model is directly loaded, and the path of material distribution is optimized according to the DQN model to obtain the optimal material distribution path.

[0075] It can be understood that in the embodiment of the present invention, the collaborative distribution path planning method based on deep reinforcement learning is mainly divided into two parts: training and path planning. The core of the model training algorithm corresponding to the DQN model is to train the neural network. In order to make the DQN model suitable for the material distribution path optimization scenario, it is necessary to obtain enough data samples of the material distribution system in the scenario, use these data samples to train the DQN model, and continuously update the network parameters through iterative solution to obtain the optimal parameters under the minimum mean square error. For path planning, the DQN model has learned the characteristics of all material distribution system data samples at this time, so the trained DQN model is directly used to optimize the material distribution path to obtain the optimal material distribution path, where the network training related parameters are set as follows:

[0076]

[0077]

[0078] The adaptive moment estimation (ADAptive Moment estimation, ADAM) algorithm and stochastic gradient descent (SGD) method are used to update the parameter set of the neural network. The ADAM algorithm is different from the traditional gradient descent algorithm that uses a fixed learning rate. It can adaptively update the learning rate during the training process. Batch training can be used for training. The training set size is set to 20000 and the validation set size is set to 2000. Random sampling is adopted during testing, and the sample size is set to 500.

[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0080] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A collaborative delivery path planning method based on deep reinforcement learning, characterized in that: The method specifically comprises the following steps: Obtain multiple customer order information, determine multiple customer points, generate initial solutions, and construct a VRCD initial path that can connect all customer points; Removing nodes or paths to destroy the current solution in the VRCD initial path, and performing route repair or route construction to change the structure of the initial solution to form a new solution; According to the new solution, a DQN model is constructed; Optimize the path of material distribution according to the DQN model; The steps of obtaining multiple customer order information, determining multiple customer points, generating an initial solution, and constructing a VRCD initial path that can connect all customer points in series specifically include the following steps: Receive multiple customer orders, obtain multiple customer order information, and determine multiple customer points; Randomly placing the plurality of customer orders into candidate batches, and generating an order batching plan; Adopting a search strategy, analyzing the order batching scheme, and generating an initial solution; According to the initial solution, constructing a VRCD initial path capable of connecting all customer points in series; The receiving of multiple customer order information, determining multiple customer points, generating an initial solution, and constructing a VRCD initial path capable of connecting all customer points in series also includes the following steps: According to the plurality of customer order information, prepare the goods and equip them into a coordinated fleet, wherein the coordinated fleet has trucks and robots; The removing of nodes or paths, destroying the current solution in the VRCD initial path, and performing route repair or route construction to change the structure of the initial solution and form a new solution specifically includes the following steps: Destroying the current solution in the VRCD initial path by removing nodes or paths through a plurality of destruction measures; Repairing the route or constructing the route through a variety of repair measures, changing the structure of the initial solution, and forming a new solution; The various sabotage measures specifically include: customer removal measures, stop removal measures, route closure measures, and route sabotage measures; In the customer removal action, a random customer removal operation and a greedy customer removal operation are used to remove customer points; in the stop point removal action, a robot route connecting a stop point and reaching and leaving the stop point is randomly removed; in the route closing action, a random truck / robot route closing operation and a greedy truck / robot route closing operation are used to remove truck / robot routes; in the route destruction action, a random route destruction operation and a greedy route destruction operation are used to select a VRCD route from the VRCD initial path for destruction; The various repair actions specifically include: route reconstruction actions, customer insertion actions, and division actions; In the route reconstruction action, the VRCD route is reconstructed using the customer points through the route reconstruction operation; in the customer insertion action, a customer point is selected to be inserted into the VRCD route through the random customer insertion operation and the greedy customer insertion operation to reconstruct a new VRCD route; in the division action, a new truck route is started from the connection stop point through the truck route opening operation, a new robot route is started from the stop point through the robot route opening operation, and a position in the VRCD route is selected to insert the stop point through the random stop point insertion operation and the greedy stop point insertion operation; According to the new solution, constructing the DQN model specifically includes the following steps: According to the new solution, the estimated network weight θ and the target network weight θ are determined. Initialize the memory capacity N and input the estimated network weight θ and target network weight Train and output the DQN model through the preset model training algorithm.