Periodic vehicle routing problem solving method based on deep reinforcement learning
By employing a deep reinforcement learning encoder-decoder structure and the REINFORCE algorithm, the problem of coordinating multi-period service frequencies and cross-period routes in the periodic vehicle routing problem is solved, achieving efficient and fast route planning that outperforms traditional methods.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN UNIV OF TECH
- Filing Date
- 2026-04-15
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies struggle to effectively handle periodic vehicle routing problems involving multi-period service frequency constraints and cross-period route coordination. Traditional heuristic methods rely on manual rules and have limited generalization capabilities, while deep reinforcement learning methods are primarily designed for single-period VRPs and lack the ability to model time-dependent factors.
We employ a deep reinforcement learning-based approach to design an encoder-decoder structure, embed a node access frequency matrix, introduce a periodicity-aware masking mechanism and the REINFORCE algorithm, and train a policy network to generate path schemes that meet the needs of multi-period services.
It significantly reduces the total travel distance, improves the solution speed and generalization ability, can quickly generate high-quality solutions, and is applicable to PVRP instances of different sizes and distributions, outperforming traditional methods.
Smart Images
Figure CN122390178A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent optimization and deep reinforcement learning technology, specifically relating to a method for solving periodic vehicle path problems based on deep reinforcement learning. Background Technology
[0002] With the rapid development of e-commerce, intelligent manufacturing, and smart logistics, the Vehicle Routing Problem (VRP), a classic problem in operations research and combinatorial optimization, has wide applications in supply chain management, express delivery, raw material supply, supermarket replenishment, and other practical scenarios. Traditional VRP focuses on single-day or single-cycle vehicle route planning, aiming to minimize total travel distance or transportation costs while meeting vehicle capacity constraints and customer demand. However, in actual logistics and distribution, many customers need to receive multiple services within a certain time period (e.g., weekly, monthly), such as regular supermarket replenishment, periodic community group buying deliveries, and regular industrial raw material supply. These application scenarios require extending single-cycle VRP to a multi-cycle planning problem, namely the Periodic Vehicle Routing Problem (PVRP). PVRP not only needs to determine the number of services and specific service dates for each customer within a cycle, but also needs to plan a route that meets vehicle capacity constraints for each day, making its decision complexity far higher than traditional VRP.
[0003] Since PVRP is essentially an NP-hard combinatorial optimization problem, exact solution algorithms (such as branch and bound and cutting plane methods) exhibit exponentially increasing computational complexity in the worst case, making them unsuitable for handling practical problems of medium to large scale. Traditional heuristic methods and metaheuristic algorithms, such as Genetic Algorithms (GA) and Adaptive Large Neighborhood Search (ALNS), can obtain near-optimal solutions within a reasonable timeframe and are therefore widely used in PVRP solutions. However, these methods typically rely on a large number of manually designed rules and domain expertise. Rule formulation requires repeated debugging by experienced experts, and the generalization ability of heuristic rules is limited for problem instances of different distributions or sizes, often requiring redesign or adjustment. Furthermore, traditional heuristic methods require starting the search from scratch for each new instance, resulting in low computational efficiency and making it difficult to meet the demands of real-time or near-real-time logistics scheduling.
[0004] In recent years, Deep Reinforcement Learning (DRL) has demonstrated great potential in solving combinatorial optimization problems. Unlike traditional heuristic methods that rely on manually generated rules, DRL methods can automatically learn solution strategies from data, forming end-to-end neuroheuristic algorithms. Through an encoder-decoder policy network, DRL methods can capture latent patterns in problem instances and efficiently explore the combinatorial solution space using the parallel computing power of GPUs. Currently, DRL-based methods have achieved performance close to or even surpassing traditional heuristic methods on VRP variants such as the Traveling Salesman Problem (TSP) and the Vehicle Routing Problem with Capacity Constraints (CVRP). However, most existing neuroheuristic solvers are primarily designed for single-cycle VRPs and fail to effectively handle complex decision-making factors in PVRPs, such as multi-cycle service frequency constraints, customer visit date combinations, and cross-cycle route coordination. Specifically, PVRP requires the model to simultaneously determine each customer's visit date (i.e., service plan) and the specific driving route for each day, while traditional DRL models often assume that all customers complete the service once within a single day, lacking the ability to model the time dependency of periodic demand. Therefore, how to effectively apply deep reinforcement learning technology to the periodic vehicle path problem and design a neural solver that can sense the frequency of node visits and cross-period constraints is an important problem that urgently needs to be solved in this field. Summary of the Invention
[0005] In addressing the limitations of existing methods for solving the Periodic Vehicle Routing Problem (PVRP), traditional heuristic algorithms rely heavily on manual rule design and have limited generalization capabilities. Existing deep reinforcement learning methods primarily target single-period PVRP problems and cannot effectively handle technical issues such as multi-period service frequency constraints and cross-period route coordination. This invention proposes a deep reinforcement learning-based method for solving the PVRP, aiming to automatically learn efficient route construction strategies. This method significantly reduces the total travel distance and improves the solution speed while meeting the multi-period service needs of customers.
[0006] This invention is the first to apply the DRL framework to solving PVRP (Physical Vehicle Resource Planning) problems. By designing an encoder-decoder structured policy network, the access frequency matrix of nodes is embedded into the network input during the encoding phase, enabling the model to perceive the service demand of each node throughout the entire cycle. During the decoding phase, a cycle-aware masking mechanism and vehicle capacity constraints are introduced to ensure that the generated route plan meets multi-cycle feasibility requirements. Furthermore, this invention employs the REINFORCE algorithm and a multi-starting-point strategy to train the network, significantly improving solution quality and generalization ability. Experimental results show that the method of this invention can quickly obtain high-quality solutions on PVRP instances of various sizes and distributions, outperforming traditional methods such as genetic algorithms and adaptive large neighborhood search in terms of solution speed and generalization performance, providing a new technical solution for the intelligent solution of periodic logistics and distribution problems.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for solving the periodic vehicle routing problem based on deep reinforcement learning is presented, with the following specific steps: Step 1: Model the periodic vehicle routing problem as a Markov decision process The optimization objective of the periodic vehicle routing problem is to minimize the total travel distance while satisfying constraints on customer visit frequency and vehicle capacity. Its mathematical expression is as follows: in, The objective of this invention is the total distance traveled by the vehicle to complete delivery at each node. It is the total length of the planning period. This refers to the number of nodes where delivery needs to be carried out, for each node where a vehicle needs to provide delivery services. From node To the node The distance is calculated using Euclidean distance in this method; It is a decision variable, representing the period. Are there any vehicles inside from the node? Drive to the node If it exists, then ,otherwise ; It is a vehicle In the cycle internal nodes Load during the time; This is the maximum capacity of each vehicle; It is a node The total number of services required throughout the entire cycle; Formula (1) represents minimizing the total travel distance of all vehicles throughout the entire cycle; Formula (2) ensures that the daily cargo load of each vehicle does not exceed its capacity limit; Formula (3) ensures that the total number of services served by each node throughout the entire cycle is equal to its preset service frequency; The solution process of this problem is constructed as a Markov decision process, defining its state, action, transition function and reward function.
[0008] state :in, For the deadline step The set of visited nodes, used to record partial decomposition; This represents the vehicle's current remaining capacity. To indicate the current progress within the planning cycle, record which stage of the cycle you are currently in.
[0009] action : at time step From the feasible action space Choose one action from the options, where This represents the set of unreachable nodes that the current vehicle can reach. This represents the set of depot nodes that a vehicle can return to; a depot node is the vehicle's garage location. The vehicle can start from or return to this node to end the delivery. If the vehicle's remaining capacity cannot meet the needs of any node, or if returning to the depot helps improve the overall path efficiency within the cycle, then the action of returning to the depot is selected.
[0010] Transfer function: based on the selected action , the current state Transition to the next state If the selected action is used as a node, then the selected node is added to the set of existing nodes. And update the vehicle's remaining capacity to ,in To complete the node The amount of resources required for the delivery task; if the selected action is to return to the depot, then reset the vehicle capacity. and update the cycle progress. Once all nodes have been visited, the state transitions to the final state.
[0011] Reward function: The reward is defined as a negative distance traveled, meaning the reward for each transfer is... To minimize the total driving distance.
[0012] Step 2: Construct a policy network with an encoder-decoder structure 2.1) Input all node information of the periodic vehicle routing problem into the encoder. The initial embedding vector of each node i is composed of its two-dimensional coordinates. With the entire planning cycle Service frequency within Concatenate to form the initial embedding vector The encoder updates the node embeddings through L identical network layers, each containing a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer captures the relationships between nodes, and the feedforward neural network layer performs non-linear transformations on the node features. After L layers of processing, the final embedding vectors of all nodes are obtained. ; 2.2) During the decoding phase, the decoder uses an autoregressive approach to progressively generate vehicle routes for each day. At the start of route construction for each day, vehicles depart from the depot node (numbered 0), and the remaining capacity of the current vehicles is initialized to the maximum capacity. ; in each decoding step The decoder first constructs a context vector. The vector is composed of the following three parts: the global graph embedding from the encoder output. (Mean of all node embeddings), embedding of the last visited node (If the current step is the first node of the path, then the previous node is the depot, and its embedding is a learnable vector.) , The embedding vector of the garage represents the current vehicle not yet departing and the current vehicle's remaining capacity. : Subsequently, the decoder will use the context vector As a query vector, embed all nodes from the encoder output. As key and value vectors, they are input into a multi-head attention layer, which calculates the interaction information between the context and each node to obtain the output of the attention layer. 2.3) Input the output of the multi-head attention layer into the single-head attention layer, calculate the compatibility score of each node in the current state, and mask nodes that do not meet the constraints through a masking mechanism. These constraints include: nodes that have already been visited, nodes that cannot be reached due to insufficient remaining vehicle capacity, and nodes that have no access requirement in the current cycle step; compatibility score The calculation is as follows: in, For scaling hyperparameters, In the encoder stage, each node obtains its corresponding query vector after passing through the attention layer. In the decoder, this vector represents the context-aware hidden state of each node within the currently selected path. The product of the node embedding vector output by the encoder and the key projection matrix. For the hidden layer dimension; The compatibility score is converted into a probability distribution using the Softmax function. The node with the highest probability is selected as the next node to be visited. The probability distribution is calculated as follows: in, This represents the probability of choosing action a in the current state s. After obtaining the node with the highest selection probability, calculate and determine whether the vehicle's remaining capacity can continue delivery after adding the node to the delivery route, and whether the newly added delivery node meets the previous delivery constraints. If the constraints are met, add the node to the delivery route and continue to calculate the selection probability of the remaining nodes. If the constraints are not met, select the depot node to add to the delivery route and end the current delivery task.
[0013] Step 3: Train the policy network using reinforcement learning algorithms. 3.1) Initialize policy network parameters Set the number of training rounds Batch size Planning cycle For each round of training, a batch of... A series of periodic vehicle routing problem instances; for each instance, a multi-starting point strategy is adopted to generate... There are several different delivery routes, each starting from a different node. For each route, within each day of the planning cycle, the following operations are performed cyclically from the initial state until the access demand of all nodes for that day is met or vehicle capacity is insufficient: Based on the current state, the action probability distribution is calculated through a policy network, the next node is selected, the action is executed, a reward is obtained, and the process transitions to the next state; where the reward is defined as the negative travel distance, i.e. ; 3.2) Calculate the total reward for each route over the entire period. ,in Calculate the total number of steps required to complete the route; calculate the shared baseline. Used to stabilize the training process, its value is the average total reward of all routes in the current batch: in, Given an instance of a vehicle routing problem to be solved Next, the The total reward for each route; the policy gradient is calculated using the REINFORCE algorithm, and the network parameters are recalculated through gradient ascent. The gradient calculation formula is as follows: in, Used to measure how well a current action performs relative to average performance. The gradient is the logarithmic probability; iterative training stops when the preset number of training rounds is reached or the model performance converges.
[0014] The beneficial effects of this invention are: This invention is the first to apply deep reinforcement learning to the periodic vehicle routing problem, proposing an end-to-end neuroheuristic solution method. Compared with existing technologies, this invention has the following advantages: By embedding the node access frequency matrix into the encoder input, this invention enables the policy network to perceive the service demand of each node over multiple periods, thereby effectively handling the time-dependent constraints in PVRP, which is impossible for existing DRL methods. Furthermore, a periodic-aware masking mechanism is designed to dynamically mask nodes that do not meet the combination of service frequency, vehicle capacity, and access date during decoding, ensuring that the generated solution satisfies all PVRP constraints while significantly reducing invalid searches. During training, a multi-starting-point REINFORCE training strategy is employed, utilizing a shared baseline to reduce gradient variance, thereby improving training stability and convergence speed. Experiments show that this method outperforms traditional genetic algorithms and adaptive large neighborhood search algorithms in solving PVRP instances. It also exhibits a significant advantage in solution speed: thanks to the advantages of autoregressive decoding, the single-run solution time for PVRP instances with 20 to 300 nodes is only 0.02 to 0.51 seconds, far faster than traditional methods such as LKH-3, GA, and ALNS (which typically require tens of seconds to several hours). This invention demonstrates good generalization ability, stably obtaining high-quality solutions under various node distributions, including uniform, Gaussian, and Beta distributions, and is also applicable to PVRP variants with node access restrictions, exhibiting strong robustness and adaptability.
[0015] Experimental results show that the present invention can quickly obtain high-quality solutions to periodic vehicle routing problems of different scales and distributions, with a solution time much faster than traditional heuristic methods, and has good generalization ability and practicality. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the framework of the present invention.
[0017] Figure 2 This is the encoder-decoder network structure of the present invention. Detailed Implementation
[0018] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.
[0019] The method of this invention can be used to solve periodic vehicle routing problems (PVRP) of different scales and distributions. The method flow of this invention is as follows: Figure 1 As shown, an end-to-end neural solver is constructed using an encoder-decoder structure policy network. Attention mechanisms and masking techniques are used to ensure the feasibility of the solution, and the multi-starting point REINFORCE algorithm is used for training.
[0020] The encoder-decoder network structure used in this invention is as follows: Figure 2 As shown, the encoder consists of stacked multi-head attention layers and feedforward neural network layers, while the decoder includes multi-head attention layers, single-head attention layers, and a masking mechanism.
[0021] The objective of this invention is to address a given set of nodes and a period length. Service frequency of each node and demand Vehicle capacity It can quickly generate multi-cycle vehicle path schemes that satisfy all constraints, minimizing the total travel distance.
[0022] The method of this invention can be divided into three stages: (1) Problem Encoding Phase: Transforming PVRP instances into embedded representations that the encoder can process. For each node Its initial features are determined by two-dimensional coordinates. It is formed by concatenating with a learnable node access frequency matrix, where the node access frequency matrix is used to encode the node throughout the entire cycle. Number of services within and service date distribution. The encoder via Multi-head attention layers and feedforward neural network layers transform node features into high-dimensional embedding vectors. This embedding captures the spatial location of the node, its periodic needs, and its interactions with other nodes.
[0023] (2) Path Decoding Stage: The decoder uses an autoregressive approach to generate the next node to be visited at each time step. The decoding process is carried out daily. Each day, starting from the depot node, nodes are selected sequentially until the termination condition is met, at which point the decoder returns to the depot and begins path construction for the next day. In each decoding step, the decoder first constructs a context vector, fusing the global graph embedding, the previous node embedding, and the current vehicle's remaining capacity; then, it calculates the interaction between the context and all node embeddings through multi-head attention; next, it calculates the compatibility score of each node through single-head attention and applies a masking mechanism to block infeasible nodes; finally, it obtains the selection probability through Softmax and selects the next node.
[0024] (3) Policy Training Phase: The policy network is trained using the multi-starting point REINFORCE algorithm. For each training instance, from... Using 1 different node as the starting node, generate The system calculates the total reward (negative total distance traveled) for each trajectory and uses a shared baseline as the average reward for all trajectories within the batch. It then updates the network parameters using policy gradients to increase the probability of high-reward trajectories and decrease the probability of low-reward trajectories.
[0025] This embodiment demonstrates the training and testing process of the present invention based on PVRP instances of different sizes. Node sizes include... Planning cycle Service frequency of each node exist Internally randomly generated, demand quantity exist Uniform sampling within the vehicle, vehicle capacity Node coordinates at Within the square region, nodes are generated according to different distributions (uniform, Gaussian, and Beta), with the depot node located at the origin. .
[0026] The specific implementation steps are as follows: Step 1: Data Generation and Parameter Initialization 1.1) Generate training instance set: randomly generated PVRP instances, each containing Each node has a set of coordinates, requirements, service frequency, and service dates that are randomly generated.
[0027] 1.2) Initialize policy network parameters: Randomly initialize all learnable parameters of the encoder and decoder using the Kaiming uniform initialization method. Set the number of encoder layers. The number of multi-head attention heads is 8, and the embedding dimension is... The decoder uses the same embedding dimension.
[0028] 1.3) Setting training hyperparameters: batch size Multiple starting points REINFORCE learning rate Using the Adam optimizer. Number of training epochs. Each round contains 1,000 batches.
[0029] 1.4) Set up test instance set: Generate test instances independently, with 128 instances generated for each scale, to evaluate model performance.
[0030] Step 2: Encoder Forward Calculation 2.1) For each training instance, extract the coordinate matrix of all nodes. and frequency embedding matrix (in ), splicing together to obtain the initial embedding , dimension .
[0031] 2.2) Map the initial embedding to a linear projection. dimension: .
[0032] 2.3) For arrive layer: Multi-head attention: computation Feedforward Networks: Computation MHA uses 8-head attention, and FF contains two linear layers. ) and ReLU activation.
[0033] 2.4) Output the final embedded value and global graph embedding .
[0034] Step 3: Decoder Autoregressive Path Generation For each instance, path building is performed daily, and the decoder state is initialized daily.
[0035] 3.1) Initialize the path for the current day: Set the current node as depot (index 0), and the remaining capacity. The visited set is empty; the path sequence for the day. .
[0036] 3.2) When the remaining capacity is greater than 0 and there are nodes that have not completed their service but are reachable, repeat the following steps: 3.2.1) Constructing the context vector: ,in The embedding of the previous node (if it is a depot, use the learnable depot embedding). ).
[0037] 3.2.2) will As a query vector As key and value vectors, they are input into a multi-head attention layer to obtain the attention output.
[0038] 3.2.3) Pass the attention output through a single-head attention layer to compute the attention for each node. The compatibility score.
[0039] 3.2.4) Applying a mask: For nodes in any of the following cases ,set up (1) Node The number of services has been reached within this period. (i.e., the number of times service has been provided equal to) (2) Nodes demand (3) Nodes There is no service requirement for the current day (judged based on the combination of its service dates); (4) Node i has been visited in the path of the current day.
[0040] 3.2.5) Crop and scale the scores of unmasked nodes using tanh: ,in .
[0041] 3.2.6) Calculate the selection probability: .
[0042] 3.2.7) Use a greedy strategy to select the node with the highest probability. .
[0043] 3.2.8) Execute action: Set the node Add the route for the day and update the remaining vehicle capacity. , The demand for nodes, For the updated vehicle capacity, The vehicle capacity before the update is incremented by 1 after the update, and the number of times the node has been served is set. .
[0044] 3.3) When it is unable to continue serving any node, choose to return to depot: add depot to the path, end the path for the day, and store the path for the day in the solution set.
[0045] 3.4) Repeat steps 3.1-3.3 until all nodes have reached the required number of services in all periods. .
[0046] 3.5) Calculate the total travel distance: For all days and all consecutive node pairs along all paths, sum the Euclidean distances. Total reward .
[0047] Step 4: Multi-starting point trajectory generation 4.1) For each training instance, generate 10 different starting nodes (10 are randomly selected from all nodes).
[0048] 4.2) For each starting node, make it the first node of the path for that day (i.e., the first node visited after starting from the depot), and then complete the path construction for all days according to step 3. Each starting node generates a complete solution trajectory.
[0049] 4.3) Calculate the total reward for each trajectory. .
[0050] Step 5: Update policy network parameters 5.1) For all instances and all trajectories within a batch, calculate the shared baseline: 5.2) Calculate the policy gradient loss: 5.3) Using the Adam optimizer, learning rate Gradient descent is applied to the loss function to update the policy network parameters. .
[0051] 5.4) Every 10 training rounds, evaluate the current model on the validation set. If the validation performance is better than the historical best, save the model parameters.
[0052] Step 6: Model Testing and Comparison 6.1) Load the trained policy network parameters. For each test instance, run the decoding process in step 3 (using a greedy strategy) and record the total driving distance and solution time.
[0053] 6.2) Comparison with baseline methods: LKH-3 (running time limit of 1 hour), Genetic Algorithm (GA, population size 100, crossover probability 0.8, mutation probability 0.1, iterations 1000), Adaptive Large Neighborhood Search (ALNS, destruction ratio 0.1-0.4, initial temperature 30, cooling rate 0.9, maximum number of improvements 100).
[0054] 6.3) Record the average objective value, average gap (relative to the optimal value of LKH-3), and average solution time for each method. Regarding solution quality, in the uniform distribution example, when N=20, the method of this invention achieves the optimal objective value (23.72), superior to LKH-3 (23.81); when N=50~200, the gap between the method of this invention and LKH-3 is only 0.92%~3.87%, significantly better than GA (gap 13.95%~120.57%) and ALNS (8.40%~93.97%). Under Gaussian and Beta distributions, the gap between the method of this invention and LKH-3 is controlled between 2% and 22%, and in the Beta distribution with N=200, it surpasses LKH-3 to achieve the optimal result. For PVRP with access restrictions, the gap of the method of this invention is only 0.43%~1.92%, which is much lower than GA (8.78%~54.28%) and ALNS (6.31%~42.84%).
[0055] In terms of solution efficiency, the average solution time of the method of this invention does not exceed 0.5 seconds across all scales, with only 0.02 seconds for N=20 and approximately 0.51 seconds for N=300. In contrast, the solution time for LKH-3 increases from 11 seconds (N=20) to 3600 seconds (N≥150); ALNS reaches 3476 seconds at N=300; and GA also requires tens to hundreds of seconds. The method of this invention has a significant advantage in solution efficiency compared to other methods.
Claims
1. A method for solving the periodic vehicle routing problem based on deep reinforcement learning, characterized in that, The specific steps are as follows: Step 1: Model the periodic vehicle routing problem as a Markov decision process The optimization objective of the periodic vehicle routing problem is to minimize the total travel distance while satisfying the constraints of customer visit frequency and vehicle capacity. The solution process is constructed as a Markov decision process, defining its state, action, transition function and reward function. Step 2: Construct a policy network with an encoder-decoder structure 2.1) Input all node information of the periodic vehicle routing problem into the encoder. The initial embedding vector of each node i is composed of its two-dimensional coordinates. With the entire planning cycle Service frequency within Concatenate to form the initial embedding vector The encoder updates the node embeddings through L identical network layers, each containing a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer is used to capture the relationships between nodes, and the feedforward neural network layer is used to perform non-linear transformations on the node features. After L layers of processing, the final embedding vectors of all nodes are obtained. ; 2.2) During the decoding phase, the decoder uses an autoregressive approach to progressively generate vehicle routes for each day. At the start of route construction for each day, vehicles depart from the depot node, and the remaining capacity of the current vehicle is initialized to the maximum capacity. ; in each decoding step The decoder first constructs a context vector. The vector is composed of the following three parts: the global graph embedding from the encoder output. Embedding of the previously visited node and the current remaining capacity of the vehicle : Subsequently, the decoder will use the context vector As a query vector, embed all nodes from the encoder output. As key and value vectors, they are input into a multi-head attention layer, which calculates the interaction information between the context and each node to obtain the output of the attention layer. 2.3) Input the output of the multi-head attention layer into the single-head attention layer, calculate the compatibility score of each node in the current state, and mask nodes that do not meet the constraints through a masking mechanism. The constraints include: nodes that have been visited, nodes that cannot be reached due to insufficient vehicle capacity, and nodes that have no access requirements in the current cycle step. The compatibility score is converted into a probability distribution using the Softmax function. The node with the highest probability is selected as the next node to be visited. After obtaining the node with the highest selection probability, it is calculated and determined whether the vehicle's remaining capacity can continue delivery after adding the node to the delivery route and whether the newly added delivery node meets the previous delivery constraints. If the constraints are met, the node is added to the delivery route and the selection probability of the remaining nodes is calculated. If the constraints are not met, the depot node is selected to be added to the delivery route and the current delivery task is ended. Step 3: Train the policy network using reinforcement learning algorithms.
2. The method for solving the periodic vehicle routing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 1, the mathematical expression of the optimization objective of the periodic vehicle routing problem is as follows: in, The objective of the problem is the total distance traveled by the vehicle to complete the delivery at each node; It is the total length of the planning period. This refers to the number of nodes where delivery needs to be carried out, for each node where a vehicle needs to provide delivery services. From node To the node The distance is calculated using Euclidean distance in this method; It is a decision variable, representing the period. Are there any vehicles inside from the node? Drive to the node If it exists, then ,otherwise ; It is a vehicle In the cycle internal nodes Load during the time; This is the maximum capacity of each vehicle; It is a node Total number of services required throughout the entire cycle.
3. The method for solving the periodic vehicle pathing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 1, its state, action, transition function, and reward function are defined as follows: state :in, For the deadline step The set of visited nodes, used to record partial decomposition; This represents the vehicle's current remaining capacity. To record the current stage of the planning cycle, indicating which phase of the cycle the current project is in. action : at time step From the feasible action space Choose one action from the options, where This represents the set of unreachable nodes that the current vehicle can reach. This represents the set of depot nodes that a vehicle can return to; a depot node is the vehicle's garage location. A vehicle can start from or return to this node to end its delivery. If the vehicle's remaining capacity cannot meet the needs of any node, or if returning to the depot helps improve the overall path efficiency within the cycle, then the action of returning to the depot is selected. Transfer function: based on the selected action , the current state Transition to the next state If the selected action is used as a node, then the selected node is added to the set of existing nodes. And update the vehicle's remaining capacity to ,in To complete the node The amount of resources required for the delivery task; if the selected action is to return to the depot, then reset the vehicle capacity. and update the cycle progress. Once all nodes have been visited, the state transitions to the final state. Reward function: The reward is defined as a negative distance traveled, meaning the reward for each transfer is... To minimize the total driving distance.
4. The method for solving the periodic vehicle pathing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 2.2), the context vector The expression is: 。 5. The method for solving the periodic vehicle routing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 2.3), the compatibility score... The calculation is as follows: in, For scaling hyperparameters, In the encoder stage, each node obtains its corresponding query vector after passing through the attention layer. In the decoder, this vector represents the context-aware hidden state of each node within the currently selected path. The product of the node embedding vector output by the encoder and the key projection matrix. For the hidden layer dimension.
6. The method for solving the periodic vehicle pathing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 2.3), the probability distribution is calculated as follows: in, This represents the probability of choosing action a in the current state s.
7. The method for solving the periodic vehicle pathing problem based on deep reinforcement learning according to claim 1, characterized in that, Step 3 is as follows: 3.1) Initialize policy network parameters Set the number of training rounds Batch size Planning cycle ; For each round of training, a batch of... A series of periodic vehicle routing problem instances; for each instance, a multi-starting point strategy is adopted to generate... There are several different delivery routes, each starting from a different node. For each route, within each day of the planning cycle, the following operations are performed cyclically from the initial state until the access demand of all nodes for that day is met or vehicle capacity is insufficient: Based on the current state, the action probability distribution is calculated through a policy network, the next node is selected, the action is executed, a reward is obtained, and the process transitions to the next state; where the reward is defined as the negative travel distance, i.e. ; 3.2) Calculate the total reward for each route over the entire period. ,in Calculate the total number of steps required to complete the route; calculate the shared baseline. , used to stabilize the training process, is the average total reward of all routes in the current batch; The REINFORCE algorithm is used to calculate the policy gradient, and the network parameters are recalculated using gradient ascent. .
8. The method for solving the periodic vehicle routing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 3.2), the baseline is shared. The calculation formula is as follows: in, Given an instance of a vehicle routing problem to be solved Next, the Total reward for the route.
9. The method for solving the periodic vehicle pathing problem based on deep reinforcement learning according to claim 1, characterized in that, In step 3.2), the gradient calculation formula is as follows: in, Used to measure how well a current action performs relative to average performance. The gradient is the logarithmic probability; iterative training stops when the preset number of training rounds is reached or the model performance converges.