A hybrid deep reinforcement learning method for vehicle routing optimization with time windows
By combining deep reinforcement learning methods with graph attention networks and long short-term memory networks, the problems of spatiotemporal correlation and long-term temporal dependence in vehicle path optimization are solved, and efficient vehicle path planning is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGCHUN UNIV
- Filing Date
- 2026-02-24
- Publication Date
- 2026-06-05
AI Technical Summary
Existing deep reinforcement learning methods struggle to effectively model the spatiotemporal relationships of customer nodes, long-term temporal dependencies in the path generation process, and policy optimization stability in vehicle path optimization with time windows, resulting in a difficulty in dynamically balancing solution quality and solution time.
A hybrid deep reinforcement learning approach is adopted, utilizing an encoder-decoder architecture and combining a multi-layer graph attention network, an adaptive gating mechanism, a long short-term memory network, and an attention mechanism to generate vehicle path sequences that meet time window and capacity constraints.
It improves the model's ability to model graph topology, enhances the feasibility and adaptability of path planning, and optimizes the efficiency and quality of vehicle path solving.
Smart Images
Figure CN122154820A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, specifically a hybrid deep reinforcement learning method for vehicle path optimization with time windows. Background Technology
[0002] Vehicle routing with time windows (VRPTW) has significant application value in logistics optimization, particularly in parcel delivery, time-sensitive services, and electric vehicle dispatching. By modeling and analyzing data related to customer node location, demand, time window constraints, and vehicle capacity, VRPTW can accurately plan vehicle routes and proactively identify potential delay risks or capacity overload anomalies. This not only aids in vehicle health management and dispatch early warning but also optimizes logistics system operation and maintenance strategies, reduces transportation costs, minimizes delay risks and operational losses, thereby ensuring the safe, stable, and efficient operation of the logistics system.
[0003] There are various methods for solving VRPTW (Vertical Time Point Problem), which can be divided into two categories based on their solution principles: traditional optimization methods and deep reinforcement learning-based methods. Traditional optimization methods often have limited accuracy and are only suitable for small-scale or structurally simple instances. Recently developed deep reinforcement learning-based methods have achieved great success in solving combinatorial optimization problems such as VRPTW. The model architectures used are mostly based on sequence modeling using recurrent neural networks (RNNs), attention mechanisms based on Transformers, and graph neural networks (GNNs). However, these models still face key bottlenecks under complex time window constraints. Specifically, RNN sequence modeling struggles to explicitly characterize the spatial proximity relationships and topological structure between client nodes, resulting in limited ability to express constraint information. While Transformer methods excel in parallel computation and global dependency modeling, their computational complexity increases quadratically with the number of nodes, and they are relatively inadequate in explicitly modeling path feasibility and constraint satisfaction. Although GNN methods can model node-edge relationships, they often employ static attention structures, lacking adaptive modeling of neighborhood noise and dynamic spatiotemporal relationships. Furthermore, there is still room for improvement in the collaborative design between encoding representation and sequential decoding. In summary, existing DRL methods still have shortcomings in fine-grained modeling of the spatiotemporal associations of client nodes, long-term temporal dependency capture during path generation, and policy optimization stability, making it difficult to achieve a dynamic balance between solution quality and solution time.
[0004] Therefore, there is an urgent need for a new technical solution to address this problem. Summary of the Invention
[0005] The purpose of this invention is to provide a hybrid deep reinforcement learning method for vehicle path optimization with time windows, in order to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A hybrid deep reinforcement learning method for vehicle path optimization with time windows, the method comprising:
[0008] Obtain a vehicle routing problem dataset with time windows, represent the vehicle routing problem dataset with a graph structure, and generate a training set and a test set. The graph structure includes node features and edge features, and the node features include at least location, demand and time window information.
[0009] A hybrid deep reinforcement learning framework based on encoder-decoder is constructed. The encoder processes the graph data to obtain node embedding representations, and the decoder generates vehicle path sequences that meet preset constraints in an autoregressive manner based on the node embedding representations.
[0010] The encoder includes a multi-layer graph attention network layer and an adaptive gating mechanism layer. The multi-layer graph attention network layer is used to aggregate information of nodes and their neighboring nodes based on the attention mechanism. The adaptive gating mechanism layer is used to generate a gating value based on the node's own embedding and the aggregated neighbor information, and to dynamically adjust the node embedding.
[0011] The decoder integrates a long short-term memory network layer and an attention mechanism layer to perceive dynamic contextual information during path generation.
[0012] The hybrid deep reinforcement learning framework is trained on the training set to optimize the framework parameters; the trained hybrid deep reinforcement learning framework is then used to optimize the test set to generate vehicle path sequences that satisfy time window and capacity constraints.
[0013] As a further embodiment of the present invention, the graph attention network layer is a GATv2 graph attention mechanism.
[0014] As a further embodiment of the present invention, the encoder further includes an input embedding layer and an output layer;
[0015] The input embedding layer is used to map node features and edge features into an initial embedding vector;
[0016] The output layer is used to output the final embedded representation of the nodes and the global graph embedding.
[0017] As a further embodiment of the present invention, the decoder also includes a dynamic context construction layer, and the attention mechanism layer includes a multi-head attention layer and a single-head attention layer;
[0018] The dynamic context construction layer is used to construct a dynamic context vector containing current state information;
[0019] The long short-term memory network layer is used to process the dynamic context vector to capture the temporal dependencies in the path generation process;
[0020] The multi-head attention layer is used to capture global dependencies between nodes based on the output of the long short-term memory network layer.
[0021] The single-head attention layer is used to generate the action probability distribution for selecting the next node based on the output of the multi-head attention layer.
[0022] As a further embodiment of the present invention, the information fused in the dynamic context vector includes: the node embedding selected in the previous step, the warehouse node embedding, the current vehicle remaining capacity, the current time, and the normalization term of the time window feature.
[0023] As a further embodiment of the present invention, the decoder generates the path sequence in an autoregressive manner, and when selecting the next node at each time step, the decoder applies an action mask to filter nodes that violate capacity constraints or time window constraints.
[0024] As a further embodiment of the present invention, the path planning model is trained using a proximal policy optimization algorithm, and the hybrid deep reinforcement learning framework adopts an Actor-Critic architecture.
[0025] As a further aspect of the present invention, the reward function used during training is constructed based on the total travel distance of the generated path and the penalty term for violating constraints.
[0026] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention uses a graph attention network (GATv2) with an adaptive gating mechanism as an encoder, strengthens the node-edge interaction modeling by dynamically adjusting the information flow, effectively suppresses noise interference, realizes the effective representation of the coupling relationship between customer spatial proximity and time window constraints, and improves the model's ability to model graph topology.
[0027] This invention innovatively integrates a multi-head attention mechanism (MHA) with a dynamic context-aware long short-term memory network (LSTM) for encoder-decoder architectures. The MHA module captures global dependencies between nodes, while the LSTM models the dynamic features of temporal constraints. This integrated design constructs a decoding architecture with dynamic context awareness capabilities, improving the feasibility and adaptability of path planning under complex constraints. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.
[0029] Figure 1 This is a flowchart illustrating a hybrid deep reinforcement learning method for vehicle path optimization with time windows, provided as an embodiment of the present invention.
[0030] Figure 2 This is an overall architecture diagram of the steps for constructing a set of texts to be pushed to a WeChat official account based on content location tags, as provided in an embodiment of the present invention.
[0031] Figure 3 This invention provides an encoder architecture diagram for the step of real-time acquisition of user historical browsing data and generation of current requirement tags based on the historical browsing data.
[0032] Figure 4 This is a decoder architecture diagram illustrating the steps of updating the user's current request tags based on the latest browsing data, as provided in an embodiment of the present invention.
[0033] Figure 5 The coordinate distribution diagram of 20 nodes in the scenario provided in the embodiment of the present invention.
[0034] Figure 6 This is a time window distribution diagram for 20 node scenarios provided in an embodiment of the present invention.
[0035] Figure 7 This is a distribution diagram of edge access time in a 20-node scenario provided in an embodiment of the present invention.
[0036] Figure 8 This is a distribution diagram of customer needs in 20 node scenarios provided in an embodiment of the present invention.
[0037] Figure 9 This is a flowchart illustrating the generation process of a dataset in a real-world scenario, as provided in this embodiment of the invention.
[0038] Figure 10 A comparison chart of the average distances of 20 nodes provided in an embodiment of the present invention.
[0039] Figure 11 Access path diagram of 20 nodes in a random instance provided in this embodiment of the invention
[0040] Figure 12 Access path diagram of 20 nodes under H2O instance provided in this embodiment of the invention Detailed Implementation
[0041] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
[0042] like Figure 1 As shown, Figure 1 This is a flowchart of a hybrid deep reinforcement learning method for vehicle path optimization with time windows. In this embodiment of the invention, a hybrid deep reinforcement learning method for vehicle path optimization with time windows includes:
[0043] Obtain a vehicle routing problem dataset with time windows, represent the vehicle routing problem dataset with a graph structure, and generate a training set and a test set. The graph structure includes node features and edge features, and the node features include at least location, demand and time window information.
[0044] A hybrid deep reinforcement learning framework based on encoder-decoder is constructed. The encoder processes the graph data to obtain node embedding representations, and the decoder generates vehicle path sequences that meet preset constraints in an autoregressive manner based on the node embedding representations.
[0045] The encoder includes a multi-layer graph attention network layer and an adaptive gating mechanism layer. The multi-layer graph attention network layer is used to aggregate information of nodes and their neighboring nodes based on the attention mechanism. The adaptive gating mechanism layer is used to generate a gating value based on the node's own embedding and the aggregated neighbor information, and to dynamically adjust the node embedding.
[0046] The decoder integrates a long short-term memory network layer and an attention mechanism layer to perceive dynamic contextual information during path generation.
[0047] The hybrid deep reinforcement learning framework is trained on the training set to optimize the framework parameters; the trained hybrid deep reinforcement learning framework is then used to optimize the test set to generate vehicle path sequences that satisfy time window and capacity constraints.
[0048] In this embodiment, the VRPTW dataset is acquired, preprocessed, and then divided into training and testing sets. A graph model for the vehicle routing problem with time windows is constructed, representing the delivery scenario as a graph structure containing warehouse nodes and customer nodes. Each customer node includes spatial coordinates, demand, and time window constraints, while edges contain information on travel distance and time between nodes. Based on this graph model, a deep reinforcement learning environment state is constructed. This state includes static node features and dynamically changing information such as the vehicle's current location, remaining capacity, current time, and customer service status, which change with the decision-making process. Corresponding actions are then performed based on the environment state under the constraints. The acquired VRPTW data is preprocessed, including normalizing node coordinates, demand, and time windows, to construct a dataset suitable for vehicle routing optimization with time windows.
[0049] A hybrid deep reinforcement learning network model with an encoder-decoder structure is constructed. The encoder adopts the graph attention network GATv2 with an adaptive gating mechanism to jointly model node features and edge features to obtain node embedding representations. The decoder integrates a multi-head attention mechanism and a dynamic context-aware long short-term memory network to generate vehicle access paths step by step under the conditions of time window and capacity constraints.
[0050] The deep reinforcement learning model is used as a policy network. The model is trained based on an optimized proximal policy optimization algorithm. A constraint-aware reward function and action masking mechanism are used to continuously update the network parameters until the policy converges.
[0051] Using a trained hybrid deep reinforcement learning model, reasoning is performed on a given vehicle routing problem instance with a time window, and a vehicle access route scheme that satisfies the time window and capacity constraints is output.
[0052] The specific steps are as follows:
[0053] The acquired VRPTW data was preprocessed, including normalizing node coordinates, requirements, and time windows to standardize feature distribution, improve the stability of model training, and construct a dataset suitable for vehicle path optimization with time windows.
[0054] The preprocessed dataset is represented using a graph structure. The problem is defined on a bidirectional fully connected graph G={V,E}, where the node set V includes warehouse nodes (node 0) and customer nodes (nodes 1 to n), represented as follows: v i For customer nodes, ;(x i ,y i ) is node v i The coordinates of each node feature include coordinates (x, y, y). i ,y i ), demand qi Time window Service time s i， For node v i The time window, represented by the edge set is , side e ij Features include distance d ij and time t ij d ij For edge e ij The driving distance; t ij For edge e ij The travel time of each vehicle is denoted as T, where the initial capacity of each vehicle is Q>0.
[0055] The mathematical model and constraints of VRPTW are as follows: First constraint: ;
[0056] Second constraint: ;
[0057] Third constraint: ;
[0058] Fourth constraint: ;
[0059] Fifth constraint: ;
[0060] Sixth constraint: ;
[0061] Seventh constraint: ;
[0062] Eighth constraint: ;
[0063] Ninth constraint: ;
[0064] In the formula, r ij The binary decision variable represents whether an edge is in the solution; d indicates whether edge e(i,j) is part of the solution. ij This represents the cost (distance) of using e(i,j). M is the number of vehicles currently in use, and r(P) is the minimum number of vehicles required to serve the customer base. It is a sufficiently large constant to linearize the time continuity constraint. The first constraint in the formula gives a compact mathematical expression of the VRPTW objective. In the DRL framework of this paper, multiple vehicles are implicitly modeled by returning to the warehouse multiple times in a single solution sequence. The second and third constraints ensure that each customer visits only once, the fourth and fifth constraints ensure the number of vehicle paths is satisfied, and the sixth and seventh constraints ensure that the vehicle arrival time must be within the time window and the time the vehicle arrives at the next customer. The eighth constraint ensures that the capacity of each vehicle is less than the maximum vehicle capacity, and the ninth constraint ensures that the needs of all customers are fully met. Therefore, given a solution... In the VRPTW environment G, the goal of the DRL architecture is to minimize the total length. :
[0065] ;
[0066] In the formula, Calculate the L2 distance between node pairs. and Representing the solution The length of the sequence.
[0067] The graph structure is a bidirectional complete graph. Warehouse nodes have earliest departure time and latest return time constraints, while customer nodes have service time window intervals. When the vehicle arrival time exceeds the upper limit of the time window, the corresponding action is determined to be infeasible.
[0068] The basic structure of the entire DRL architecture is as follows:
[0069] State space S: includes static elements (coordinates (x, y)). i ,y i ), demand q i Time window Service time s i Access distance d ij Access time t ij ) and dynamic elements (current node position, remaining capacity, current time, and mask of visited nodes);
[0070] Action Space A: Select the next unvisited node, ensuring that capacity, time window, and reachability constraints are met (an action masking mechanism is used to filter out unmovable actions to comply with the hard time window assumption).
[0071] Reward function R: The negative value of the total travel distance of the complete path, including path penalty, capacity overload penalty and unvisited penalty. The reward function uses the negative value of the total travel distance of the complete path as the base reward, and imposes penalties on the sum of overloads that violate capacity constraints or the existence of unserved customer nodes, so as to guide the generation of feasible paths that meet the constraints.
[0072] ;
[0073] In the formula Total distance traveled. The sum of all overloads, This represents the number of unvisited client nodes. The coefficient 1000 is based on experimental tuning, ensuring that any infeasible solution receives a very strong negative reward, thus quickly eliminating infeasible trajectories in the early stages of training.
[0074] Generate a dataset using a random instance rule, with customer coordinates uniformly distributed in [0,1]. 2 , demand q i Uniformly sample from {0.1, 0.2, ..., 0.9}, time window Based on dynamically generated distances (uniformly distributed center points, with widths extracted from {0.1, 0.2, ..., 0.5}), for node sizes of 20 / 50 / 100, the vehicle capacity Q is set to 4 / 8 / 16, and the maximum time T is set to the maximum possible travel time for the corresponding size, to balance the problem difficulty. Figures 5 to 8 The image shows the distribution of a random instance dataset in a 20-node scenario. The Solomon benchmark dataset is based on standard instances (R / C / RC type), while the Hakata real-world dataset is based on a Japanese road network. Figure 9 The following is its production process.
[0075] The data is normalized and then divided into training and test sets; the normalization function is shown below:
[0076] ;
[0077] In the formula, x i Let i be the feature value of the current i-th node. The average of the entire dataset. Let Z be the standard deviation of the entire dataset, and Z be the standardized score of the i-th node. This normalization is applied to all static features to prevent feature scale differences from affecting model learning.
[0078] In a preferred embodiment of the present invention, the graph attention network layer is the GATv2 graph attention mechanism.
[0079] The encoder also includes an input embedding layer and an output layer;
[0080] The input embedding layer is used to map node features and edge features into an initial embedding vector;
[0081] The output layer is used to output the final embedded representation of the nodes and the global graph embedding.
[0082] In this embodiment, the framework of the present invention is as follows: Figure 2 As shown, it consists of a GATv2 encoder, an MHA+LSTM decoder, and PPO training. The encoder extracts spatiotemporal correlation features of nodes, the decoder generates path sequences that satisfy constraints, and the overall framework achieves end-to-end optimization. The specific construction process involves first constructing the encoder module, as shown... Figure 3 As shown, the encoder comprises an input embedding layer, a multi-layer graph attention network (GATv2) layer, an adaptive gating mechanism layer, and an output layer. First, feature embedding processing is performed on the input VRPTW graph data to capture initial representations of nodes and edges. Specifically, the feature vector of node i (including coordinates (x, y, y)) is processed... i ,y i ), demand q i Time window Service time s i Linear transformation and batch standardization are performed, and a time window feasibility prior is incorporated to obtain the initial node embeddings and edge embeddings. The calculation formula is as follows:
[0083] ;
[0084] ;
[0085] ;
[0086] In the formula, Indicates the initial node embedding, f ij , Let A1 and A2 represent the linear weight matrices, and b1 and b2 represent the bias matrices. Indicates batch normalization, t ij Slack represents the travel time from node i to j. ij The time window slackness refers to the feasible margin of the time window from node i to j, which is defined as the difference between the upper limit of the time window of node j and the earliest possible arrival time from node i to j, and serves as the feasibility prior of the static time window.
[0087] In the 4-layer GATv2 attention layer, attention scores are calculated and normalized to enhance the modeling of interactions between nodes. Residual connections are fused to preserve the original information. To suppress noise interference and enhance the fine modeling of spatiotemporal correlation, this invention introduces an adaptive gating mechanism to dynamically adjust the updated representation. Specifically, the gating vector is obtained by fusing local and global embeddings through the Sigmoid activation function, and then element-wise multiplication is applied for fusion and update.
[0088] ;
[0089] ;
[0090] ;
[0091] In the formula, c i It uses attention weights Weighted summation of neighbor features W represents the l-th level neighbor node of node j. c It is a learnable weight matrix. It is the Sigmoid activation function, W g and b g These are the weights and biases of the gating mechanism. An adaptive gating value is used; this mechanism effectively filters irrelevant noise and improves the representation ability of time window constraints and spatial proximity. Subsequently, multi-head attention fusion is used to capture multi-dimensional associations, such as various interactions between spatial proximity and time windows. After four layers of processing, the final node embedding is obtained, and the global graph embedding is calculated through average pooling.
[0092] In a preferred embodiment of the present invention, the decoder further includes a dynamic context construction layer, and the attention mechanism layer includes a multi-head attention layer and a single-head attention layer;
[0093] The dynamic context construction layer is used to construct a dynamic context vector containing current state information;
[0094] The long short-term memory network layer is used to process the dynamic context vector to capture the temporal dependencies in the path generation process;
[0095] The multi-head attention layer is used to capture global dependencies between nodes based on the output of the long short-term memory network layer.
[0096] The single-head attention layer is used to generate the action probability distribution for selecting the next node based on the output of the multi-head attention layer.
[0097] In this embodiment, as Figure 4As shown, the decoder comprises a dynamic context construction layer, a Long Short-Term Memory (LSTM) layer, a Multi-Head Attention (MHA) layer, and a Single-Head Attention (SHA) layer, aiming to capture long-term temporal dependencies and global constraints in path generation. The decoding process is autoregressive, progressively selecting the next node from the warehouse node until all customers have visited. First, a dynamic context vector is constructed at each time step, incorporating current state information such as the previous node embedding, warehouse global embedding, remaining capacity, current time, and its normalization term. The candidate node embedding is then fused with the residual of the projected dynamic context vector. This dynamic context fusion mechanism employs a low-dimensional constraint injection strategy, enhancing the model's ability to perceive capacity and time window constraints with almost no increase in parameters. This invention uses a dynamically constraint-aware LSTM layer to model temporal dynamic features and process context to capture long-term dependencies. Specifically, through forget gates, input gates, output gates, and memory unit update mechanisms, it ensures the perception of dynamic constraint changes, such as changes in remaining capacity and time windows. The calculation formula is as follows:
[0098] ;
[0099] ;
[0100] ;
[0101] In the formula c t For dynamic context vectors, To embed the client node selected in the previous step, h was retrieved in the first step. depot h depot For embedding the encoder in the repository node, load remain t represents the current remaining capacity of the vehicle. curr For the current time, is the global time normalization constant, and the last term is the average normalized relaxation of all client time windows. W is the context vector of the linear mapping. c b is a learnable weight matrix. c For the bias term, u t,i To incorporate node features into the context, Embedding of nodes output by the encoder. It is a single-layer fully connected network with ReLU activation (output dimension d).
[0102] This dynamic context fusion mechanism employs a low-dimensional constraint injection strategy, which can enhance the model's ability to perceive capacity and time window constraints with almost no increase in parameters.
[0103] Then it is input into an LSTM for processing:
[0104] ;
[0105] ;
[0106] ;
[0107] ;
[0108] ;
[0109] ;
[0110] LSTM consists of an input gate, a forget gate, an output gate, and a memory unit, where... It is the sigmoid activation function, W f W i W o These are the weight matrices for the forget gate, input gate, and output gate, respectively, h. t-1 W is the previous hidden state. c Let b be the candidate state weight matrix. f b i b c and b o For the bias term, f t This is the output of the forget gate, i t It is the output of the input gate. It is a candidate memory unit, C t For updated cell states, o t It is the output gate vector, h t This represents the hidden state at the current time step t. This design allows the decoder to dynamically adapt to VRPTW constraints, ensuring the feasibility of path generation.
[0111] h based on LSTM output t The system generates dynamic query vectors and applies action masks to enforce constraints, limiting them to feasible nodes. The MHA layer captures global dependencies between nodes through query, key-value projection, and Softmax attention computation, and integrates residuals and layer normalization to improve adaptability to complex graph structures. Specifically, it constructs updated context vectors by fusing dynamic information such as global graph embedding, previous visited node embedding, remaining capacity, and current time. , and h t Linear mapping is performed to obtain the query vector, key vector, and value vector. Attention is then calculated by multiplying the dot product similarity between the query and key using multiple attention heads. Finally, residual connections and layer normalization are applied to the output to obtain the updated context vector. This stabilizes the training process and preserves the original information. Finally, the SHA layer generates the action probability distribution, which is updated by the context vector via MHA. The original similarity score is calculated using the key vector, scaled using the tanh activation function, and adjusted by combining action masks. Softmax normalization is then applied to obtain the final selection probability for each feasible node. The decoder generates a complete path sequence through sampling or greedy selection, returning it to the repository to conclude the process. This design ensures that the paths satisfy all VRPTW constraints and optimizes the total cost. Through the aforementioned encoder-decoder architecture, this invention achieves fine-grained modeling of the spatiotemporal associations of client nodes and captures long-term temporal dependencies during path generation, improving the overall stability and efficiency of the framework.
[0112] In a preferred embodiment of the present invention, the decoder generates the path sequence using an autoregressive method. When selecting the next node at each time step, the decoder applies an action mask to filter nodes that violate capacity constraints or time window constraints.
[0113] The path planning model is trained using a proximal policy optimization algorithm, and the hybrid deep reinforcement learning framework adopts an Actor-Critic architecture.
[0114] The reward function used during training is constructed based on the total distance traveled on the generated path and the penalty term for violating constraints.
[0115] In this embodiment, the generated training set and test set are used as framework input to train the constructed hybrid deep reinforcement learning framework. Considering different spatiotemporal correlations and constraint requirements, graph data is input into the encoder for feature extraction, and then combined with the decoder for path generation. The framework is continuously optimized until it converges and reaches performance stability. Performance stability refers to the framework achieving low path cost, i.e., high optimization efficiency.
[0116] In this embodiment, the constructed framework is trained with 100 iterations, a training set to test set ratio of 9:1, a default learning rate of 3e-5, a batch size of 512, 8 MHA heads, 4 GATv2 heads, 1 LSTM layer, MSE loss function, Adam optimizer, and pruning parameters. =0.2, discount factor γ=0.99, and the PPO algorithm is adopted. During training, the actor-critic framework is used to optimize the policy by collecting trajectories, calculating the advantage, and updating parameters. Specifically, the framework input is a preprocessed VRPTW instance, the encoder extracts static graph embeddings, the decoder generates path sequences, and the reward is calculated through immediate feedback (negative distance + constraint penalty). To improve training stability and generalization ability, this invention optimizes the standard PPO, including the calculation of the advantage function and batch normalization to reduce variance and accelerate convergence, and introduces a dynamic value function loss weight adjustment mechanism. When the variance of the advantage function is large, the training intensity of the critic is adaptively enhanced, thereby accelerating the convergence of value estimation and improving the stability of policy updates. The specific training process can be represented as follows:
[0117] ;
[0118] ;
[0119] ;
[0120] ;
[0121] ;
[0122] ;
[0123] ;
[0124] In the formula A t Let R be the dominance function, γ be the discount factor, and R be the dominance function. t+l V(s) represents the immediate reward obtained from the environment at time t+l. t Let t be the critical state value function under state t. , These are the mean and standard deviation of the dominance function, respectively. , These represent the parameters of the policy network as θ and θ, respectively. Choose an action when in state s The probability, r(θ), is the ratio of the probability of the new policy to the probability of the old policy. For trimming parameters, For cutting losses, L E (θ) Entropy regularization term Let be the loss function of the Critic network. For the total loss, c p c v c e They are L CLIP , and L E The weighting coefficient of (θ).
[0125] Among them, the dynamic value function loss weight adjustment mechanism adaptively adjusts c. v Implementation: When the standard deviation of the dominance function When the preset threshold is exceeded, increase c. v To enhance the impact of the critic loss, the specific adjustment formula is as follows:
[0126] ;
[0127] In the formula, The baseline weights are set to 1.0 by default; δ is an adjustment coefficient used to control the augmentation magnitude, with a default value of 0.5; and threshold is the variance threshold, also with a default value of 0.5. This mechanism is dynamically calculated during training iterations to ensure that when the variance... When the value is large (and the value estimation is unstable), c is adaptively increased. v This allows for priority optimization of the critic network, accelerating the convergence of the value function.
[0128] The average path change of the 20-node scenario trained by this invention is compared with the average path change of other models trained as shown below. Figure 10 As shown, the path result is as follows Figure 11 As shown, the training of this framework achieves a good convergence speed, and the paths generated on the test set closely approximate the optimal solution.
[0129] The trained path planning model is used to optimize the test set, generating vehicle path sequences that meet the constraints. The framework performance is then evaluated, including calculating the total path cost, inference time, and comparing it with the baseline model to verify the effectiveness and generalization ability of the framework on the VRPTW problem.
[0130] In this embodiment, the trained and converged framework is used to generate end-to-end paths for test set instances. Specifically, for each test instance (such as R / C / RC type instances in the Solomon benchmark dataset or real road network instances in Hakata), the preprocessed graph data is input into the encoder to extract spatiotemporal correlation features; subsequently, the decoder starts autoregressive generation from repository node 0, selecting the next feasible node (applying action masks to ensure that capacity Q is not overloaded, and time windows are used). Arrive within [timeframe], taking service time into account. i and travel time t ij (This continues until all client nodes v have finished accessing and returned to the repository, generating a path sequence.) The average path length of the solution is calculated, and the constraints are verified to be satisfied (no time window violation, load not exceeding Q, and all nodes covered). To quantify the performance, this invention defines key indicators, including: the average path length, the optimal difference between the solution and the benchmark solution obtained by the optimization solver, and the model inference time.
[0131] The optimal gap is calculated as follows:
[0132] ;
[0133] In the formula D averagge D represents the average length of all path solutions obtained by running the current model on the entire test set. best This indicates the length of the baseline optimal solution of the optimizer, and Gap is the percentage difference between the two.
[0134] Comparative and ablation experiments were set up, and evaluation metrics were predicted. The following methods were used as comparative experiments: OR-Tools, VNS / TS, PtrNet, AM, GCN-NPEC, and E-GAT. The results are shown in Table 1. It can be seen that in multi-scale optimization on random datasets, the framework proposed in this invention has the lowest path length and gap in the 20-node and 50-node scenarios. In the 100-node scenario, the path length and gap are better than other reinforcement learning methods and are close to the optimal solution.
[0135] Table 1. Comparison of this model with classical methods and existing mainstream reinforcement learning methods.
[0136] To further verify the effectiveness of the proposed method, experiments were conducted on the Solomon benchmark dataset to evaluate path distance and time for different problem instances. These included evaluations of R-class, C-class, and RC-class paths in scenarios with 20, 50, and 100 nodes, respectively. The results are shown in Table 2. It can be seen that the proposed method performs excellently in path optimization, achieving higher path solution quality and shorter solution time compared to other reinforcement learning methods in the vast majority of instances, effectively balancing solution quality and speed.
[0137] Table 2 Comparison of different methods on the Solomon dataset
[0138]
[0139] GAT+MHA, GATv2+MHA, GATv2+gating+MHA, and GATv2+gating+MHA+LSTM were used as ablation experiments. The experiments were conducted in two scenarios: one with randomly generated datasets and the other with real datasets, containing 20 and 50 client nodes respectively. The aim was to explore the impact of each improvement on the model's average path distance index. The results are shown in Table 3, which verifies the rationality and effectiveness of the model design of this invention.
[0140] Table 3 Ablation Experiment Verification
[0141]
[0142] Furthermore, to further verify the applicability of the method of this invention, multiple experiments were conducted using real road data from the Hakata area of Fukuoka City, Japan, under different customer node scenarios (named H20, H50, and H100). The results are shown in Table 4. The different scenarios of the model experiments in this paper are visualized as follows: Figure 12 As shown in Table 4, the hybrid deep reinforcement learning method proposed in this invention still outperforms other models.
[0143] Table 4 Comparison of different methods in Hakata data instances
[0144]
[0145] This invention achieves stable optimization through an improved proximal policy optimization (PPO) algorithm and a constraint-aware composite reward function, thereby improving the stability and convergence efficiency of the training process. Evaluations on random instances, the Solomon benchmark dataset, and real logistics datasets show that compared to mainstream DRL methods and classic heuristic algorithms, transportation costs are reduced by 2%–10%, demand satisfaction rate is higher than 99%, and the performance degradation is only 3.2% in cross-distribution tests.
[0146] In summary, the hybrid deep reinforcement learning method for vehicle path optimization with time windows proposed in this invention has better solution efficiency and generalization ability than existing mainstream deep reinforcement learning methods and classical heuristic algorithms.
[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A hybrid deep reinforcement learning method for time windowed vehicle routing optimization, characterized in that, The method includes: Obtain a vehicle routing problem dataset with time windows, represent the vehicle routing problem dataset with a graph structure, and generate a training set and a test set. The graph structure includes node features and edge features, and the node features include at least location, demand and time window information. A hybrid deep reinforcement learning framework based on encoder-decoder is constructed. The encoder processes the graph data to obtain node embedding representations, and the decoder generates vehicle path sequences that meet preset constraints in an autoregressive manner based on the node embedding representations. The encoder includes a multi-layer graph attention network layer and an adaptive gating mechanism layer. The multi-layer graph attention network layer is used to aggregate information of nodes and their neighboring nodes based on the attention mechanism. The adaptive gating mechanism layer is used to generate a gating value based on the node's own embedding and the aggregated neighbor information, and to dynamically adjust the node embedding. The decoder integrates a long short-term memory network layer and an attention mechanism layer to perceive dynamic contextual information during path generation. The hybrid deep reinforcement learning framework is trained on the training set to optimize the framework parameters; the trained hybrid deep reinforcement learning framework is then used to optimize the test set to generate vehicle path sequences that satisfy time window and capacity constraints.
2. The hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 1, characterized in that, The graph attention network layer is based on the GATv2 graph attention mechanism.
3. The hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 2, characterized in that, The encoder also includes an input embedding layer and an output layer; The input embedding layer is used to map node features and edge features into an initial embedding vector; The output layer is used to output the final embedded representation of the nodes and the global graph embedding.
4. The hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 1, characterized in that, The decoder also includes a dynamic context building layer, and the attention mechanism layer includes a multi-head attention layer and a single-head attention layer; The dynamic context construction layer is used to construct a dynamic context vector containing current state information; The long short-term memory network layer is used to process the dynamic context vector to capture the temporal dependencies in the path generation process; The multi-head attention layer is used to capture global dependencies between nodes based on the output of the long short-term memory network layer. The single-head attention layer is used to generate the action probability distribution for selecting the next node based on the output of the multi-head attention layer.
5. A hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 4, characterized in that, The information fused in the dynamic context vector includes: the node embedding selected in the previous step, the warehouse node embedding, the current vehicle remaining capacity, the current time, and the normalization term of the time window feature.
6. A hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 1, characterized in that, The decoder generates path sequences using an autoregressive approach. When selecting the next node at each time step, the decoder applies an action mask to filter nodes that violate capacity constraints or time window constraints.
7. A hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 1, characterized in that, The path planning model is trained using a proximal policy optimization algorithm, and the hybrid deep reinforcement learning framework adopts an Actor-Critic architecture.
8. A hybrid deep reinforcement learning method for vehicle path optimization with time windows according to claim 1, characterized in that, The reward function used during training is constructed based on the total distance traveled on the generated path and the penalty term for violating constraints.