A takeaway delivery route planning method based on deep reinforcement learning

Abstract

This invention discloses a method for food delivery route planning based on deep reinforcement learning, in the field of route planning technology. The method comprises the following steps: Step 1: Reading problem input information, defining optimization objectives, and setting constraints; Step 2: Building a basic neural network within an encoder-decoder architecture; Step 3: Constructing actor and critic networks; Step 4: Setting network training parameters; Step 5: Constructing a dataset; Step 6: Forward pass of the actor and critic networks; Step 7: Network training and updating; and Step 8: Termination judgment. This invention offers the advantages of fast solution speed and strong generalization capabilities, with improved adaptability and development prospects. It can simulate and solve more complex problems. The introduction of an attention-pointer network structure accelerates algorithm convergence, and trained network parameters can be saved, eliminating the need for repeated extensive calculations.

Description

Technical Field

[0001] The present invention belongs to the field of path planning technology, and specifically relates to a takeout delivery path planning method based on deep reinforcement learning. Background Art

[0002] As an integral and crucial component of the food delivery industry, route planning for food delivery improves delivery efficiency and quality while reducing delivery costs, which holds considerable research value and potential for application. The food delivery route planning problem belongs to the pickup and delivery problem with time windows (PDPTW), and scholars have conducted extensive research on the PDPTW problem in recent years. In 2003, Li and Lim proposed a heuristic algorithm based on tabu tables and simulated annealing, combining the local loop-breaking properties of tabu tables with simulated annealing to efficiently solve the PDPTW problem. In 2006, Bent proposed the first two-stage hybrid algorithm for solving the PDPTW problem. In the first stage, a simple simulated annealing algorithm was used to reduce the number of routes and optimize solution quality, while in the second stage, a large neighborhood search (LNS) was used to reduce the total travel cost. In 2012, Pan Lijun and Fu Zhuo proposed a time difference insertion strategy, devising fast and optimal time difference insertion methods, and combined them with a genetic algorithm to solve the time window pickup and delivery problem. Conventional heuristic algorithms have poor adaptability and cannot simulate and solve more complex problems. The overall algorithm convergence speed needs to be improved and the generalization ability is not strong. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a takeout delivery path planning method based on deep reinforcement learning to solve the problems raised in the above background technology.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A takeout delivery route planning method based on deep reinforcement learning, the planning method comprising the following steps:

[0006] Step 1: Read the input information of the problem, define the optimization goal, and set the constraints;

[0007] Step 2: Build the basic neural network in the encoder-decoder architecture based on the attention-pointer network mechanism and initialize its parameters such as weights and biases;

[0008] Step 3: Based on the basic neural network built in step 2, combine the actor-critic algorithm to build the actor network and critic network;

[0009] Step 4: Set network training process parameters;

[0010] Step 5: Collect the location information of the delivery nodes, add order constraints to the data, build a data set, and divide it into a training sample set, a validation sample set, and a test data set;

[0011] Step 6: Input the data from the training sample set and use the actor network to generate a predicted trip sequence for the rider, that is, the legal order in which the rider visits each pickup and delivery node, and the corresponding trip distance. The critic network then evaluates the output of the actor network, that is, the actual trip distance.

[0012] Step 7: Train and update the network. Calculate the difference between the results given by the actor network and the critic network, square it and use it as the loss value. Perform backpropagation based on the loss value and use the Adam optimizer to update the parameters of the neural network.

[0013] Step 8: Termination judgment. If the set number of training rounds has been completed or the loss value meets the termination condition, the iteration is terminated, the optimal network parameters are saved, and the actor network is used to give the planning result of the target problem under the parameters. Otherwise, go to step 6, repeat the training process, and evaluate the network training status by observing the change in loss and the performance of the current network on the validation set.

[0014] Preferably, the input information of the problem in step 1 includes the average speed of the rider, the farthest distance traveled by the rider, the maximum amount of cargo carried by the rider, the order time window, the maximum demand of a single node, the total number of nodes, and the location of each node;

[0015] The optimization goal is to minimize the total travel distance when completing all order delivery tasks;

[0016] The constraints are that each order must be completed within the time window and only once, the rider's driving distance cannot exceed the maximum driving distance, and the rider must pick up first and then deliver.

[0017] Preferably, the basic neural network in step 2 includes one or more of a convolutional encoder network, an attention mechanism network, and a pointer network.

[0018] Preferably, the actor-critic algorithm in step three is a reinforcement learning method that combines policy gradient and temporal difference learning. The actor refers to the policy function, which learns a policy to obtain the highest possible reward. The critic refers to the value function, which estimates the value function of the current policy and evaluates the quality of the actor.

[0019] Preferably, the actor network in step 3 uses an encoder-decoder architecture, with a convolutional input layer as an encoder and an attention-pointer network as a decoder. The output of the encoder is processed by the hidden layer and input into the decoder. The attention-pointer network in the decoder points from the previous state to the next state according to the current attention of each node.

[0020] Preferably, in step 1, the steps of reading the problem input information, defining the optimization goal, and setting the constraints are as follows:

[0021] First, set the model parameters;

[0022] Then clarify the model assumptions;

[0023] Then refine the model constraints;

[0024] Finally, determine the overall optimization goal:

[0025] .

[0026] Preferably, the relevant structure of the basic neural network constructed in step 2 is as follows:

[0027] The encoder-decoder architecture was selected as the basic architecture, and a pointer network using an attention mechanism was introduced to complete the seq2seq process in the traditional method. The mathematical description of the pointer network is as follows:

[0028]

[0029]

[0030] In the encoder-decoder structure, the encoder is composed of a one-dimensional convolutional layer and the decoder is composed of an attention-pointer network, where the attention mechanism is used to calculate the attention on the input node given the current state.

[0031] Preferably, the actor-critic algorithm described in step 3 is implemented as follows:

[0032] Step 1: Initialize relevant parameters;

[0033] Step 2: Select an action from the current state space based on the current policy function;

[0034] Step 3: Perform the action and receive an immediate reward;

[0035] Step 4: Calculate the total reward for executing the action based on the instant reward;

[0036] Step 5: Update the relevant parameters of the policy function and value function based on this reward value and related parameters such as the learning rate;

[0037] Step 6: Update the discount rate and current status;

[0038] Step 7: Repeat the above process until the current state is completed and a new cycle begins;

[0039] Step 8: Until the policy function converges, output the policy function;

[0040] Step 9: This policy function is the current best action selection plan.

[0041] Beneficial effects of the present invention:

[0042] 1. This invention has the advantages of fast solution speed and strong generalization ability. It uses the principle of reinforcement learning and realizes the combination of deep learning and reinforcement learning by building a deep neural network and fitting the relevant functions in the reinforcement learning algorithm.

[0043] 2. This invention applies deep reinforcement learning to solve combinatorial optimization problems. Deep reinforcement learning has better adaptability and development prospects, and can simulate and solve more complex problems, which is unmatched by traditional algorithms such as heuristic algorithms;

[0044] 3. The present invention introduces the attention-pointer network structure to accelerate the convergence of the algorithm. The network trained with a large amount of data has a strong generalization ability and is very effective in solving new problems. The trained network parameters can be saved and trained once and used multiple times without the need to perform a large number of calculations again each time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 It is a schematic flow chart of the present invention;

[0047] Figure 2 It is a schematic diagram of the pickup and delivery problem (PDP) of the present invention;

[0048] Figure 3 is the change of the network during the training process of the present invention;

[0049] Figure 4 This is a 10-node path planning graph obtained based on the deep reinforcement learning solution example in the present invention;

[0050] Figure 5This is the 20-node path planning graph obtained based on the deep reinforcement learning solution example in the present invention;

[0051] Figure 6 This is a comparison chart of the results of the present invention and the traditional algorithm. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] See also Figure 1 As shown, the present invention proposes a takeaway delivery route planning method based on deep reinforcement learning, comprising the following steps:

[0054] Step 1: Read the input information of the problem, define the optimization goal, and set the constraints:

[0055] The input information of the problem includes the average speed of the rider, the longest distance traveled by the rider, the maximum amount the rider can carry, the order time window, the maximum demand of a single node, the total number of nodes and the location of each node; the optimization goal is to minimize the total distance of travel when completing all order delivery tasks; the constraints are that each order must be completed within the time window and only completed once, the rider's travel distance cannot exceed the maximum travel distance, and the rider must pick up first and then deliver.

[0056] Step 2: Build the basic neural network in the encoder-decoder architecture:

[0057] Build some basic neural networks in the encoder-decoder architecture based on the attention-pointer network mechanism, such as convolutional encoder network, attention mechanism network, pointer network and other neural networks, and initialize their parameters such as weights and biases.

[0058] Step 3: Build the actor and critic network:

[0059] The actor-critic algorithm is a reinforcement learning method that combines policy gradient and temporal difference learning. The actor is the policy function, which learns a policy to maximize rewards. The critic is the value function, which estimates the value function of the current policy, that is, evaluates the actor's performance.

[0060] Based on the principles of the actor-critic algorithm, we can construct an actor network and a critic network based on the network built in the previous step to fit the policy function and value function in the algorithm. The actor network uses an encoder-decoder architecture, with a convolutional input layer as the encoder and an attention-pointer network as the decoder. The encoder output is processed by the hidden layer and input to the decoder. The attention-pointer network in the decoder points from the previous state to the next state based on the current attention of each node.

[0061] Step 4: Set network training process parameters:

[0062] Set the relevant parameters of network training, such as whether to generate a new test set for this training, the number of training rounds, the termination condition of training, the size of the data set, the learning rate, the address where the network parameters are saved, and the loss and optimizer algorithms used;

[0063] Step 5: Build the dataset:

[0064] A large amount of location information of pickup and delivery nodes is collected, and order constraints are added to the relevant data. That is, pickup nodes and delivery nodes must be paired, and only after the negative demand of the pickup point is met can the delivery point be accessed to meet its positive demand. On this basis, a data set is constructed and divided into a training sample set, a validation sample set, and a test data set.

[0065] Step 6: Forward pass of actor network and critic network:

[0066] The data in the training sample set is input, and the actor network is used to give the rider's predicted travel sequence, that is, the legal order in which the rider visits each pickup and delivery node, and the corresponding travel distance is given. The critic network is then used to evaluate the output of the actor network, that is, the actual travel distance is given.

[0067] Step 7: Network training and updating:

[0068] The difference between the results given by the actor network and the critic network is calculated and squared as the loss value. Backpropagation is performed based on the loss, and the parameters of the neural network are updated using the Adam optimizer. The Adam optimizer is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can iteratively update the neural network weights based on the training data.

[0069] Step 8: Termination of judgment:

[0070] If the set number of training rounds has been completed or the loss value meets the termination condition, the iteration is terminated, the optimal network parameters are saved, and the planning result of the target problem is given using the actor network under these parameters. Otherwise, go to step 6 and repeat the above training process. The network training status can be evaluated by observing the change in loss and the performance of the current network on the validation set.

[0071] In step 1: read the problem input information, define the optimization goal, and set the constraints as follows:

[0072] First, set the model parameters:

[0073] : Rider set, there are k riders in total;

[0074] : A set of nodes, where 0 represents a distribution center;

[0075] : A collection of pickup points, representing n orders;

[0076] : set of delivery points, i∈P, i+n∈D, i and i+n are paired;

[0077] : the distance between nodes i, j∈N;

[0078] : The total cost per unit distance traveled by rider k;

[0079] : average speed of rider k;

[0080] : The number of takeout orders at node i, i∈P is positive when it indicates pick-up, and i∈D is negative when it indicates delivery;

[0081] : The maximum time window allowed for order i∈P;

[0082] : The maximum number of takeouts that rider k can carry;

[0083] : the maximum distance that rider k can travel;

[0084] : Whether rider k has passed through path (i, j) (i, j∈N), 1 represents yes, 0 represents no;

[0085] : Whether rider k has been to node i∈N;

[0086] : the number of takeaways that rider k carries when he arrives at node i∈N;

[0087] : The distance that rider k has traveled when he arrives at node i∈N;

[0088] : The time it takes for rider k to arrive at node i∈N.

[0089] Then clarify the model assumptions:

[0090] (1) All riders have the same average speed, maximum distance traveled, maximum load, and cost per unit distance: (v, s, q, c are constants);

[0091] (2) The rider’s time at the distribution center is 0, and the distance traveled is 0: ;

[0092] (3) Assuming that the rider only picks up / delivers one takeaway at each node: (When i∈D, it is -1);

[0093] (4) The time window for all orders is the same: (t is a constant value);

[0094] Then improve the model constraints:

[0095] (1) Ensure that each order is picked up by a unique rider: ;

[0096] (2) Ensure that the corresponding food is delivered after the food is picked up: ;

[0097] (3) Ensure that meals are picked up before being delivered: ;

[0098] (4) Ensure the matching relationship between orders and riders: ;

[0099] (5) Maximum number of takeout items a rider can carry: ;

[0100] (6) Order time window constraints: ;

[0101] (7) Rider’s maximum driving distance constraint: ;

[0102] (8) Balance constraints on the number of takeouts carried by riders: ;

[0103] (9) Path balance constraint: ;

[0104] (10) Time balance constraint: ;

[0105] (11) Path time constraints: ;

[0106] Finally, determine the overall optimization goal:

[0107] .

[0108] The relevant structure of the basic neural network built in step 2 is as follows:

[0109] (1) This paper selects the encoder-decoder structure as the basic architecture and introduces a pointer network using the attention mechanism to complete a process similar to the seq2seq in traditional methods. The mathematical description of the pointer network is as follows:

[0110]

[0111]

[0112] in Is the encoder encoder in the time series The hidden layer output of The decoder is a time series The hidden state output of Directly calculating the softmax will yield the probability vector of the output dictionary, and the output vector dimension will be consistent with the input. They are all parameters of fixed dimension and can be trained.

[0113] (2) In the above encoder-decoder structure, the encoder consists of a one-dimensional convolutional layer and the decoder consists of an attention-pointer network. The attention mechanism is used to calculate the attention on the input node given the current state.

[0114] The implementation steps of the actor-critic algorithm described in step 3 are:

[0115] (1) Initialize relevant parameters;

[0116] (2) Select an action from the current state space based on the current policy function;

[0117] (3) Perform the action and receive an immediate reward;

[0118] (4) Calculate the total reward obtained by performing the action based on the immediate reward;

[0119] (5) Update the relevant parameters of the policy function and value function based on this reward value and related parameters such as learning rate;

[0120] (6) Update discount rate and current status;

[0121] (7) Repeat the above process until the current state is completed and a new cycle begins;

[0122] (8) Until the policy function converges, output the policy function;

[0123] (9) This policy function is the current best action selection plan.

[0124] Figure 2 The figure below shows a schematic diagram of the pickup and delivery problem, with rectangles and circles representing pickup and delivery points, respectively. The food delivery route planning problem can be modeled as a pickup and delivery problem with time windows (PDPTW), a combinatorial optimization problem. Assume that pickup and delivery points are paired, and that the pickup point must be visited first. To ensure a legitimate route plan, the order sequence must be reasonable, and the rider must meet the delivery point requirements within the order time window.

[0125] Assume that there are several food delivery centers in a city. Each center has a certain number of delivery personnel, and each center is only responsible for orders within a certain area. Now assume that there is an area where all delivery points and pickup points are located. At a certain moment, several orders are generated in this area (each order has a paired pickup point and delivery point, and must first go to the pickup point to meet the negative demand of the node before reaching the delivery point to meet its positive demand and complete the order, as shown in the figure). Figure 2 In the four task completion processes of A, B, C, and D, these orders must be completed within a certain time window. The system will then distribute these orders to the delivery personnel according to certain rules and plan the route for each delivery personnel under the premise of minimizing the total delivery cost. After the delivery personnel have visited all nodes in a reasonable order, they will return to the distribution center, as shown in the figure. Figure 2 As shown in .

[0126] The model assumes that all riders have a uniform speed. When picking up food, riders can pick it up at the store. When delivering food, the food is picked up immediately upon arrival at the delivery point, meaning there is no waiting time for both pickup and delivery. It also assumes that all riders have the same upper limit on the number of food items they can carry, the same maximum distance they can travel, and the same total cost per unit distance. Furthermore, the food picked up at each pickup point can only be delivered to the corresponding delivery point. If a rider's remaining travel distance is insufficient to reach the next node and return to the distribution center, they return directly to the distribution center, and the next rider departs immediately to take over the unfinished task.

[0127] This model uses a 1×1 coordinate map and numerical coordinates to replace actual maps and geographic locations. The points in the coordinate map replace actual pickup and delivery addresses to simplify computation and workload. The coordinate map ignores various restrictions in the actual terrain and assumes that the shortest straight-line distance between any two points is possible. This model focuses on the order in which nodes are visited, generating a corresponding coordinate point visit sequence for a given coordinate point sequence.

[0128] Since it is difficult to collect a large amount of data to build a dataset in real scenarios, the dataset in the demonstration is randomly generated by a program. It can be proved that training on a randomly generated dataset has stronger generalization ability.

[0129] The effect of the present invention can be further illustrated by the following simulation experiments:

[0130] 1. Experimental conditions:

[0131] The simulation was performed using Python 3.9.7 on an Intel(R) Core(TM) i7-11400H CPU, 16G RAM, and Windows 11.

[0132] 2. Experimental content:

[0133] A large amount of node coordinate data is randomly generated to construct a data set, which is divided into a training set, a validation set, and a test set. The network is repeatedly trained to find the network parameters that minimize the planned path distance. The effect of the network during training can be observed on the validation set, and the effect of the network after training is finally completed can be displayed on the test set. The specific process is as follows:

[0134] First, set the initial parameters and test the problem with a maximum load of 10 nodes / 20 riders. Among the initial parameters, the order time window refers to the maximum time limit for completing all orders, the maximum demand per node represents the maximum number of randomly generated demand shares for a node, whether to generate a new test set refers to whether to use a new node dataset different from the previous simulation for route planning, and the number of training times refers to how many times the neural network is trained. A higher number of training times yields better results but also takes longer. If the number of training times is 0, the default is to read the already trained network parameters and directly perform route planning.

[0135] After confirming the model parameters, click Start Random Simulation. If the number of training times is not equal to 0, the network will be trained for the set number of times first. The training process of one of them is intercepted as follows: Figure 3 As shown in the figure, the reward value represents the original movement distance in the 1×1 plane. After all training is completed, the system will save the best network parameters in the corresponding folder. The next time you use the system, you can set the number of training times to 0 and directly use the trained parameters.

[0136] After the network parameters are trained and loaded, the system will automatically select data randomly from the test set to demonstrate the path planning effect. The path planning results of the test set with a maximum load of 10 nodes / 20 riders are shown below. Figure 4 shown.

[0137] To test the robustness of the algorithm and program, we further expanded the scale of the food delivery problem. We reran the program, set the number of nodes to 20, and increased the maximum number of items a rider could carry to 30, and repeated the test using the same process as above.

[0138] Similar to the maximum load of 10 nodes / 20 riders, the path planning results of the maximum load of 20 nodes / 30 riders on the test set are shown as follows: Figure 5 shown.

[0139] 3. Experimental results

[0140] (1) Under the same conditions, the actor network trained in the present invention and the existing algorithms with the minimum path as the optimization goal (such as the ant colony algorithm, simulated annealing algorithm, and genetic algorithm) are used to perform operations on the same multiple given node sequences to solve the path planning on the given 10 / 20 node sequences and the average travel distance of multiple planning. The results are as follows Figure 6 As shown in the figure, whether in the case of 10 nodes or 20 nodes, the average travel distance of the planning results of deep reinforcement learning is the smallest, that is, compared with the other algorithms, its effect is the best;

[0141] (2) The asterisks in the result graph represent distribution centers, S n Points to the send delivery point, C n Point to the carry pickup point. Pickup points and delivery points are paired. This means that you must first pick up your takeaway at the pickup point before you can deliver it to the corresponding delivery point. The absolute value of the demand for the two is the same, but with opposite signs. For example, if there is a demand for 5 takeaways (+5) at the delivery point, you can pick up these 5 takeaways (-5) at the corresponding pickup point (you must ensure that the number of takeaways the rider can carry at this time is greater than the absolute value of the node demand). Riders all depart from the distribution center. If a rider's available distance during a delivery is insufficient, they return to the distribution center. The next rider then departs with the maximum possible distance and inherits the goods carried by the previous rider.

[0142] The order of the riders' departure is shown in the upper right corner. The line corresponding to color 0 is the route of the first rider, and the subsequent 1, 2, and so on correspond to rider number 2, rider number 3, and so on.

[0143] All riders have the same parameters, such as average speed and the number of takeouts they can carry. The system's goal is to keep the total distance traveled by all riders as small as possible.

[0144] When all node requirements are met, the task is completed and path planning concludes. The data shown in the figure shows that with 10 nodes, the average travel distance of all planned paths is 8.06 kilometers (on a 1:2000m floor plan), and with 20 nodes, the average travel distance is 11.75 kilometers, both meeting the order time window requirements.

[0145] As shown Figure 3 As shown, over multiple iterations, the neural network's loss remained essentially stable and approached zero, indicating that the network has converged significantly during repeated training. Since neural network parameters can be saved and read, the model can be used repeatedly and quickly after training. The 10 / 20 node result path diagram produced by the trained neural network is intuitively consistent, with no redundant paths, indicating a good final model performance, as evidenced by the conclusion in section 1.

[0146] Even if the average driving route of a single rider is only slightly reduced, it can lead to a significant reduction in total costs and a reduction in the average waiting time of users. This is beneficial for improving user satisfaction, increasing rider income and reducing company costs.

[0147] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0148] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A takeaway delivery route planning method based on deep reinforcement learning, characterized in that: The planning method comprises the following steps: Step 1: Read the input information of the problem, define the optimization goal, and set the constraints; Step 2: Build the basic neural network in the encoder-decoder architecture based on the attention-pointer network mechanism and initialize their weights and bias parameters; Step 3: Based on the basic neural network built in step 2, combine the actor-critic algorithm to build the actor network and critic network; Step 4: Set the network training process parameters; Step 5: Collect the location information of the delivery nodes, add order constraints to the data, build a data set, and divide it into a training sample set, a validation sample set, and a test data set; Step 6: Input the data from the training sample set and use the actor network to generate a predicted trip sequence for the rider, that is, the legal order in which the rider visits each pickup and delivery node, and the corresponding trip distance. The critic network then evaluates the output of the actor network, that is, the actual trip distance. Step 7: Train and update the network. Calculate the difference between the results given by the actor network and the critic network, square it and use it as the loss value. Perform backpropagation based on the loss value and use the Adam optimizer to update the parameters of the neural network. Step 8: Termination judgment. If the set number of training rounds has been completed or the loss value meets the termination condition, the iteration is terminated, the optimal network parameters are saved, and the planning result of the target problem is given using the actor network under these parameters. Otherwise, go to step 6 and repeat the training process. The network training status is evaluated by observing the change in loss and the performance of the current network on the validation set. The input information for the problem in step 1 includes the average speed of the rider, the maximum distance traveled by the rider, the maximum amount of cargo carried by the rider, the order time window, the maximum demand of a single node, the total number of nodes, and the location of each node; The optimization goal is to minimize the total travel distance when completing all order delivery tasks; The constraints are that each order must be completed within the time window and only once, the rider's driving distance cannot exceed the maximum driving distance, and the rider must pick up the order before delivering it; The relevant structure of the basic neural network built in step 2 is as follows: The encoder-decoder architecture was selected as the basic architecture, and a pointer network using an attention mechanism was introduced to complete the seq2seq process in the traditional method. The mathematical description of the pointer network is as follows: Where, Is the encoder encoder in the time series The hidden layer output of The decoder is a time series The hidden state output of this time is 、 、 is a parameter of fixed dimension; In the encoder-decoder structure, the encoder is composed of a one-dimensional convolutional layer and the decoder is composed of an attention-pointer network, where the attention mechanism is used to calculate the attention on the input node given the current state.

2. A takeaway delivery route planning method based on deep reinforcement learning according to claim 1, characterized in that: The basic neural network in step 2 includes one or more of a convolutional encoder network, an attention mechanism network, and a pointer network.

3. The method for takeout delivery route planning based on deep reinforcement learning according to claim 1, characterized in that: The actor-critic algorithm in step 3 is a reinforcement learning method that combines policy gradient and temporal difference learning. The actor refers to the policy function, which learns a policy to obtain the highest possible reward. The critic refers to the value function, which estimates the value function of the current policy and evaluates the quality of the actor.

4. The method for takeout delivery route planning based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the actor network uses an encoder-decoder architecture, with a convolutional input layer as the encoder and an attention-pointer network as the decoder. The output of the encoder is processed by the hidden layer and input into the decoder. The attention-pointer network in the decoder points from the previous state to the next state based on the current attention of each node.

5. The method for takeout delivery route planning based on deep reinforcement learning according to claim 1, characterized in that: In step 1, the steps of reading the problem input information, defining the optimization goal, and setting the constraints are as follows: First, set the model parameters: in, : Rider set, there are k riders in total; : A set of nodes, where 0 represents a distribution center; : A collection of pickup points, representing n orders; : set of delivery points, i∈P, i+n∈D, i and i+n are paired; : the distance between nodes i, j∈N; : The total cost per unit distance traveled by rider k; : average speed of rider k; : The number of takeout orders at node i, i∈P is positive when it indicates pick-up, and i∈D is negative when it indicates delivery; : The maximum time window allowed for order i∈P; : The maximum number of takeouts that rider k can carry; : the maximum distance that rider k can travel; : Whether rider k has passed through path (i, j), i, j∈N, 1 represents yes, 0 represents no; : Whether rider k has been to node i∈N; : The number of takeaways that rider k carries when he arrives at node i∈N; : The distance that rider k has traveled when he arrives at node i∈N; : The time it takes for rider k to arrive at node i∈N; Then clarify the model assumptions: All riders have the same average speed, maximum distance traveled, maximum load capacity, and cost per unit distance: , where v, s, q, and c are fixed values; The rider's time at the distribution center is 0, and the distance traveled is 0: ; Assume that the driver only picks up / delivers one takeaway at each node: , i∈D takes -1; The time window is the same for all orders: , t is a constant value; Then improve the model constraints: Ensure that each order is picked up by a unique rider: ; Ensure that there is corresponding delivery after picking up the meal: ; Guaranteed food pick-up before delivery: ; Ensure the matching relationship between orders and riders: ; The maximum number of takeouts a rider can carry: ; Order time window constraints: ; Rider's maximum driving distance constraint: ; Balance constraint on the number of takeouts carried by riders: ; Path balance constraints: ; Time balance constraints: ; Path time constraints: ; Finally, determine the overall optimization goal: 。 6. The method for takeout delivery route planning based on deep reinforcement learning according to claim 1, characterized in that: The implementation steps of the actor-critic algorithm in step 3 are: Step 1: Initialize relevant parameters; Step 2: Select an action from the current state space based on the current policy function; Step 3: Perform the action and receive an immediate reward; Step 4: Calculate the total reward for executing the action based on the instant reward; Step 5: Update the relevant parameters of the policy function and value function based on this reward value and the relevant learning rate parameters; Step 6: Update the discount rate and current status; Step 7: Repeat the above process until the current state is completed and a new cycle begins; Step 8: Until the policy function converges, output the policy function; Step 9: This policy function is the current best action selection plan.

Images