Dynamic vehicle path optimization method based on online deep reinforcement learning

Abstract

The invention discloses a dynamic vehicle path optimization method based on online deep reinforcement learning. The method mainly solves the problems that an existing dynamic vehicle path optimization method is poor in dynamic change adaptive capacity and poor in algorithm real-time performance. The implementation scheme is as follows: 1) generating a data set; 2) constructing a decision model; 3) constructing and training a strategy network model and a value function estimation network model; and 4) solving a dynamic vehicle path problem with a time window by using the trained strategy network model. According to the method, a partially observable Markov decision model is established according to a time sequence of delivery clients of a motorcade, and a corresponding strategy network and a value function estimation network are designed to optimize a vehicle path. The strategy model obtained through training can quickly give the dynamic vehicle optimal path, and can adjust the planned path on line according to the dynamic change of related conditions, so that the solution quality is ensured, the solution speed is higher, and the real-time performance of the algorithm in processing the dynamic vehicle path optimization problem is improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning and combinatorial optimization, and specifically relates to a dynamic vehicle path optimization method based on online deep reinforcement learning. Background Art

[0002] With the rapid development of urbanization and e-commerce, the society's demand for real-time response and flexible scheduling of logistics distribution is increasing, which has attracted widespread attention to the dynamic vehicle routing problem. In a dynamic environment, customer demand location information changes dynamically over time. The dynamic vehicle routing problem requires dynamic optimization of the path under uncertain and real-time information changes, so that the solution is closer to the actual needs. Compared with the traditional static vehicle routing problem, the dynamic model can better reflect the complexity and flexibility of the distribution plan. The traditional methods for solving the dynamic vehicle routing problem mainly include exact algorithms, heuristic algorithms and meta-heuristic algorithms. The computational overhead of these methods increases significantly when the problem scale increases, and the solution speed is relatively slow. With the advancement of computing power and the development of machine learning, many methods have emerged in recent years that use machine learning, especially deep reinforcement learning, to solve the dynamic vehicle routing problem. However, most of these deep reinforcement learning methods are offline, that is, the method of solving static problems is used to perform rolling optimization on the dynamic process. Each optimization is solved when all information is known, and it is assumed that this information will not change. These methods are difficult to adapt to the dynamic changes of the environment and demand, resulting in a decrease in the quality of the solution, and the rolling optimization method is difficult to make timely adjustments to the occurrence of dynamic events, and the real-time performance of the algorithm is not strong. Summary of the invention

[0003] In order to overcome the shortcomings of existing methods for solving dynamic vehicle routing problems, the present invention proposes a dynamic vehicle routing optimization method based on online deep reinforcement learning. A partially observable Markov decision process is used to model the dynamic vehicle routing problem with time windows (DVRPTW) dynamically added by customer requests in a time series manner, and a policy network model and a value function estimation network model for solving DVRPTW are designed. Finally, the network model is trained by an Actor-Critic algorithm based on a greedy baseline. The trained policy network model can dynamically receive new customer requests and adjust the route in time according to the current information to ensure the flexibility of path planning.

[0004] 1. Technical Principle

[0005] Most of the current methods for solving dynamic vehicle routing problems split the dynamic problem into multiple static problems according to time slices, and then solve the static problems one by one. The solution method for static problems models the Markov decision process in a vehicle-by-vehicle manner, that is, when the vehicle has not returned to the warehouse, the current action and the next action are issued by the same vehicle. The Markov decision process does not conform to the time sequence in actual distribution, so it cannot describe the dynamic distribution process and can only solve static problems. According to the characteristics of the DVRPTW problem, the present invention models the time sequence process of fleet distribution as a partially observable Markov decision process (Partially Observable Markov Decision Process, POMDP for short), which describes the process of making decisions under incomplete information conditions. For DVRPTW, at a certain moment in the distribution process, the vehicle can only observe the information of some customers, while the remaining customers appear dynamically as the distribution process proceeds. This corresponds to the characteristics of POMDP, that is, the intelligent agent cannot directly obtain the real state, but can only obtain observations and make decisions under limited information. Based on the modeled POMDP, a strategy network model, a value function estimation network model, and an Actor-Critic algorithm based on a greedy baseline are designed to achieve rapid solutions to dynamic problems. Compared with the traditional method that requires solving multiple static problems, the present invention only needs to perform a single forward propagation of the neural network to solve dynamic problems, which greatly improves the algorithm's solution speed and reduces the computational cost.

[0006] 2. According to the above principle, the present invention is implemented by the following scheme:

[0007] A dynamic vehicle path optimization method based on online deep reinforcement learning includes the following steps:

[0008] (1) Generate a dataset:

[0009] The DVRPTW dataset is randomly generated. The training set contains 300,000 DVRPTW instances, and the test set contains 10,000 DVRPTW instances. Each instance includes warehouse node features and customer node features. Warehouse node features are node coordinates, and customer node features include node coordinates, cargo demand, time window, service time, and request appearance time.

[0010] (2) Constructing a decision model:

[0011] The POMDP model is established to describe the solution process of DVRPTW, which consists of multiple time steps, represented by t∈{t0,t1,...,t T}; A time step refers to each moment of decision making in the Markov decision process; all time steps are arranged in the time order of the service, t Tis the moment to end the overall distribution; a POMDP model describing DVRPTW can be composed of a four-tuple Indicates that each part is constructed as follows:

[0012] (2-a) is a state set, including all system states in POMDP, that is, Each state S t Contains the customer state at time step t and vehicle status That is S t ={(C t ,V t )}, where n and m are both positive integers, n represents the number of nodes, m represents the number of vehicles, and n∈[10,500], m∈[2,125]; represents the quantity of goods demanded by the i-th customer at time step t, Indicates whether the i-th customer has been delivered or is scheduled to be delivered by a certain vehicle at time step t. Indicates whether the i-th client request has appeared at time step t; represents the position of the jth vehicle at time step t, represents the remaining cargo volume of the jth vehicle at time step t, represents the activation time of the jth vehicle at time step t, represents the activation state of the jth vehicle at time step t;

[0013] (2-b) is an action set, including all actions of vehicles in POMDP, that is, Action A t It means that the activated vehicle at time step t selects the next customer to be delivered from the customers who have not been delivered and are not scheduled to be delivered, or chooses to return to the warehouse;

[0014] (2-c) is the observation set, which contains all system observations in POMDP, that is, Observation O t is the observation information that the vehicle can obtain at time step t, including the system status information of this time step minus all the customer information that has not appeared;

[0015] (2-d) R is the reward, which is defined as the reward obtained by performing an action in a given state, including the vehicle's driving time, waiting time, and penalty for exceeding the time window; the vehicle's state S at time step t t Execute action A t , the state is transferred to S t+1 The reward obtained in the process can be expressed as:

[0016] R(St ,A t ,S t+1 )=-z ij -g1w j -g2p j

[0017] where z ij represents the travel time between points i and j, w j is the waiting time for the vehicle to arrive at point j in advance, p j is the time when the vehicle exceeds the time window when it arrives at point j, g1, g2 are coefficients, and g1>0, g2>0;

[0018] (3) Build and train the policy network model and value function estimation network model:

[0019] Construct a policy network and a value function estimation network based on the Transformer structure and attention mechanism; the value function estimation network is composed of a policy network followed by a linear layer; the policy network is composed of an encoder with a Transformer structure and a decoder with a multi-head attention layer; the encoder inputs the warehouse node feature x(v 0 ) and the client node features {x(v i )|i=1,2,..,n}, where n is a positive integer representing the number of nodes, and n∈[10,500]; after removing the non-appearing customer node features, the warehouse node features and the remaining customer node features are up-scaled using a linear layer and then concatenated to obtain {h i |i=0,1,..,n}, input it into the Transformer layer, and repeat the operation N times to get the output of the encoder, that is, the intermediate node feature {f i |i=0,1,..,n}, where N is a positive integer less than 10;

[0020] The decoder consists of two multi-head attention layers, which select the intermediate node features f belonging to the current vehicle from the intermediate node features output by the encoder. cur ; Combine it with all the intermediate node features {f i |i=0,1,..,n} are input into the first multi-head attention layer together, and its output is used as the key vector and value vector of the second multi-head attention layer. The current vehicle feature y cur As the query vector of the second multi-head attention layer; the current vehicle feature y cur It consists of the node coordinates of the vehicle, the remaining load of the vehicle, and the next activation time of the vehicle; the output of the second multi-head attention layer and the intermediate node feature {f i|i=0,1,..,n} after point multiplication, and then processed by softmax function to obtain the selection probability of each node {p(A i )|i=0,1,..,n}, where A i Indicates the action of the current vehicle selecting node i;

[0021] For a DVRPTW instance, the cost of generating a solution with a policy network parameterized by θ is:

[0022] L(θ)=-R(S0,A0,S1)-R(S1,A1,S2)-…-R(S T-1 ,A T-1 ,S T )

[0023] The optimization goal is to minimize L(θ). At time step t, according to the stochastic gradient descent algorithm, the sampling of time step t is used to approximate the expectation. The policy network can be trained by the following formula:

[0024]

[0025] Where α is the step size, π(·) is the strategy, q is the action value, ω is the value function estimating the network parameters, and b(S) is the greedy rollout baseline, which means starting from the current state, a complete solution is generated in a greedy manner according to the strategy, and the cost of the solution is used as the baseline of the action value to reduce the variance of the policy gradient estimate. The action value q(S t ,A t ,ω t ) is obtained through the value function estimation network; in the process of updating the policy network, the value function estimation network is continuously updated at the same time; at time step t, according to the strategy π(·), the next action A is generated t+1 , according to the Bellman formula and the gradient ascent algorithm, the training value function estimates the network:

[0026]

[0027] Where R t+1 Represents the reward at time t+1; during the training process, the Adam algorithm is used to update the model parameters until the maximum number of training rounds is reached, and the trained policy network model and value function estimation network model are obtained;

[0028] (4) Use the trained policy network model to solve DVRPTW:

[0029] After the node features are normalized, they are input into the trained policy network model. The model outputs the customers selected at each time step. The final customer sequence is the solution of DVRPTW.

[0030] In step (2-a), n and m are preferably 100 and 25, respectively;

[0031] In step (2-d), g1 and g2 are preferably 1;

[0032] In step (3), N is preferably 3.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) The present invention introduces a deep reinforcement learning algorithm to solve the dynamic vehicle routing problem, which effectively solves the problem that the traditional heuristic method will experience a sharp increase in computational complexity when the problem scale increases, and the problem that the performance may drop significantly if the environment changes significantly. The deep reinforcement learning algorithm can handle complex high-dimensional state spaces and is suitable for solving problems with a large number of customers and many constraints. At the same time, the present invention gradually improves its robustness through self-learning and exploration during multiple interactions with the environment, and better adapts to changing needs.

[0035] (2) The present invention uses a partially observable Markov decision process to model the dynamic vehicle routing optimization process, and realizes the online solution of dynamic problems. During the execution of the algorithm, the routing plan is given online according to the real-time changes in customer information. While ensuring the quality of the solution, the solution speed is greatly improved. Compared with the traditional rolling optimization method, it has stronger dynamic adaptability and real-time performance.

[0036] (3) The present invention constructs a policy network and a value function estimation network based on the Transformer structure and attention mechanism to solve the dynamic vehicle routing problem. The network can effectively extract node features and efficiently give the node access probability at each time step. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flow chart of a dynamic vehicle path optimization method based on online deep reinforcement learning according to an embodiment of the present invention;

[0038] Figure 2 A structural diagram of a policy network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The specific implementation mode of the present invention is described below:

[0040] Example 1

[0041] Figure 1 The flowchart of the dynamic vehicle path optimization method based on online deep reinforcement learning according to the embodiment of the present invention is shown, and the specific steps are as follows:

[0042] Step 1, generate the dataset:

[0043] Random DVRPTW dataset, the training set contains 300,000 DVRPTW instances, and the test set contains 10,000 DVRPTW instances. Each instance includes warehouse node features and customer node features. Warehouse node features are node coordinates, and customer node features include node coordinates, cargo demand, time window, service time, and request appearance time.

[0044] Step 2: Build a decision model:

[0045] The POMDP model is established to describe the solution process of DVRPTW, which consists of multiple time steps, represented by t∈{t0,t1,...,t T}; A time step refers to each moment of decision making in the Markov decision process; all time steps are arranged in the time order of the service, t T is the moment to end the overall distribution; a POMDP model describing DVRPTW can be composed of a four-tuple Indicates that each part is constructed as follows:

[0046] (2-a) is a state set, including all system states in POMDP, that is, Each state S t Contains the customer state at time step t and vehicle status That is S t ={(C t ,V t )}; represents the quantity of goods demanded by the i-th customer at time step t, Indicates whether the i-th customer has been delivered or is scheduled to be delivered by a certain vehicle at time step t. Indicates whether the i-th client request has appeared at time step t; represents the position of the jth vehicle at time step t, represents the remaining cargo volume of the jth vehicle at time step t, represents the activation time of the jth vehicle at time step t, represents the activation state of the jth vehicle at time step t;

[0047] (2-b) is an action set, including all actions of vehicles in POMDP, that is, Action A t It means that the activated vehicle at time step t selects the next customer to be delivered from the customers who have not been delivered and are not scheduled to be delivered, or chooses to return to the warehouse;

[0048] (2-c) is the observation set, which contains all system observations in POMDP, that is, Observation O t is the observation information that the vehicle can obtain at time step t, including the system status information of this time step minus all the customer information that has not appeared;

[0049] (2-d) R is the reward, which is defined as the reward obtained by performing an action in a given state, including the vehicle's driving time, waiting time, and penalty for exceeding the time window; the vehicle's state S at time step t t Execute action A t , the state is transferred to S t+1 The reward obtained in the process can be expressed as:

[0050] R(S t ,A t ,S t+1 )=-z ij -g1w j -g2p j

[0051] where z ij represents the travel time between points i and j, w j is the waiting time for the vehicle to arrive at point j in advance, p j is the time when the vehicle exceeds the time window when it arrives at point j, g1 and g2 are coefficients, both of which are 1;

[0052] Step 3: Build and train the policy network model and value function estimation network model:

[0053] Figure 2 The structure diagram of the strategy network of the embodiment of the present invention is shown, and the specific construction is as follows:

[0054] In a preferred embodiment including 101 nodes, 25 vehicles, and a batch size of 64, the value function estimation network is composed of a policy network followed by a 101×1 linear layer; the policy network is composed of an encoder including a Transformer structure and a decoder including a multi-head attention layer; the encoder input dimension is 64×2 warehouse node feature x(v 0 ) and the client node features {x(v i )|i=1,2,..,100}; After removing the non-appearing customer node features, the warehouse node features and the remaining customer node features are up-scaled using a linear layer and then concatenated to obtain a 64×101×128-dimensional {h i |i=0,1,..,100}, input it into the Transformer layer with 8 heads and 512 linear layers, repeat the operation 3 times, and get the output of the encoder, that is, the intermediate node feature {f i|i=0,1,..,100};

[0055] The decoder consists of two multi-head attention layers, each with 8 heads. The intermediate node features f belonging to the current vehicle are selected from the intermediate node features output by the encoder. cur , with a dimension of 64×1×128; it is combined with all the intermediate node features {f i |i=0,1,..,100} are input into the first multi-head attention layer together, and the output result of its dimension is 64×1×128 as the key vector and value vector of the second multi-head attention layer, and the current vehicle feature y of 64×1×4 cur As the query vector of the second multi-head attention layer; the current vehicle feature y cur It consists of the node coordinates of the vehicle, the remaining load of the vehicle, and the next activation time of the vehicle; the output of the second multi-head attention layer and the intermediate node feature {f i |i=0,1,..,100} after point multiplication, and then processed by softmax function to obtain the selection probability of each node with a dimension of 64×101 {p(A i )|i=0,1,..,100}, where A i Indicates the action of the current vehicle selecting node i;

[0056] For a DVRPTW instance, the cost of generating a solution with a policy network parameterized by θ is:

[0057] L(θ)=-R(S0,A0,S1)-R(S1,A1,S2)-…-R(S T-1 ,A T-1 ,S T )

[0058] The optimization goal is to minimize L(θ). At time step t, according to the stochastic gradient descent algorithm, the sampling of time step t is used to approximate the expectation. The policy network can be trained by the following formula:

[0059]

[0060] Where α is the step size, π(·) is the strategy, q is the action value, ω is the value function estimating the network parameters, and b(S) is the greedy rollout baseline, which means starting from the current state, a complete solution is generated in a greedy manner according to the strategy, and the cost of the solution is used as the baseline of the action value to reduce the variance of the policy gradient estimate; the action value q(S t ,A t ,ω t ) is obtained through the value function estimation network; in the process of updating the policy network, the value function estimation network is continuously updated at the same time; at time step t, according to the strategy π(·), the next action A is generated t+1, according to the Bellman formula and the gradient ascent algorithm, the training value function estimates the network:

[0061]

[0062] Where R t+1 Represents the reward at time t+1; during the training process, the Adam algorithm is used to update the model parameters until the maximum number of training rounds is reached, and the trained policy network model and value function estimation network model are obtained;

[0063] Step 4: Use the trained policy network model to solve DVRPTW:

[0064] After the node features are normalized, they are input into the trained policy network model. The model outputs the customers selected at each time step. The final customer sequence is the solution of DVRPTW.

[0065] Example 2

[0066] The artificially generated data set was experimented with using the method in Example 1. The experimental operating system was Ubuntu 20.04, the PyTorch 1.12.1 framework based on CUDA 11.3 and cuDNN 8.8.0, and training and testing were performed on a personal computer equipped with an Intel Core i5-13490KF CPU (2.90 GHz) and a single NVIDIA GeForce RTX 3090 (24 GB) hardware.

[0067] This example uses cost, service rate (SR), and solution time to compare five test methods, OR-Tools, PyVRP, MTPOMO, MVMoE, and Routefinder, and the method of the present invention on a data set with 100 customer nodes, 25 vehicles, and dynamic degree (DOD) of 0.3, 0.4, and 0.6, respectively. The service rate is the proportion of successfully served customers to all customers, and is calculated as follows:

[0068]

[0069] where n served is the number of customers successfully served, n totalis the total number of customers. The comparison results are shown in Table 1. It can be found that the cost of the present invention is lower than that of the comparison method on three different dynamic degree test sets, and the service rate is better than that of the comparison method in most cases. This shows that the method of this paper has better dynamic change adaptability and adjustment ability than the comparison method. At the same time, it can be seen that the online optimization algorithm of the present invention is significantly better than the comparison method in solution time. It only takes 14 seconds to solve 10,000 DVRPTW instances. This shows that the optimization speed and real-time decision-making ability of the online algorithm are better than the offline algorithm.

[0070] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made based on the structure and principle of the present invention should be included in the protection scope of the present invention.

[0071] Table 1

[0072]

Claims

1. A dynamic vehicle path optimization method based on online deep reinforcement learning, characterized in that The following steps are involved: (1) Generate a dataset: A dynamic vehicle routing problem with time windows (DVRPTW) dataset is randomly generated. The training set contains 300,000 DVRPTW instances, and the test set contains 10,000 DVRPTW instances. Each instance includes warehouse node features and customer node features. Warehouse node features are node coordinates, and customer node features include node coordinates, cargo demand, time window, service time, and request appearance time. (2) Constructing a decision model: A partially observable Markov decision process (POMDP) ​​model is established to describe the solution process of DVRPTW. The process consists of multiple time steps, represented by t∈{t0,t1,...,t T }; A time step refers to each moment of decision making in the Markov decision process; all time steps are arranged in the time order of the service, t T is the moment to end the overall distribution; a POMDP model describing DVRPTW can be composed of a four-tuple Indicates that each part is constructed as follows: (2-a) is a state set, including all system states in POMDP, that is, Each system state S t Contains the customer status and vehicle status at time step t; the customer status includes the demand for goods, whether the delivery vehicle has been determined, and whether the customer request has occurred; the vehicle status includes the vehicle location, the remaining amount of goods, the vehicle activation time, and the vehicle activation status; vehicle activation means that the vehicle has completed the delivery to the current customer and can be assigned to the next customer; (2-b) is an action set, including all actions of vehicles in POMDP, that is, Action A t It means that the activated vehicle at time step t selects the next customer to be delivered from the customers who have not been delivered and are not scheduled to be delivered, or chooses to return to the warehouse; (2-c) is the observation set, which contains all system observations in POMDP, that is, Observation O t is the observation information that the vehicle can obtain at time step t, including the system status information of this time step minus all the customer information that has not appeared; (2-d) R is the reward, which is defined as the reward obtained by performing an action in a given state, including the vehicle's driving time, waiting time, and penalty for exceeding the time window; the vehicle's state S at time step t t Execute action A t , the state is transferred to S t+1 The reward obtained in the process can be expressed as: R(S t ,A t ,S t+1 )=-z ij -g1w j -g2p j where z ij represents the travel time between points i and j, w j is the waiting time for the vehicle to arrive at point j in advance, p j is the time when the vehicle exceeds the time window when it arrives at point j, g1, g2 are coefficients, and g1>0, g2>0; (3) Build and train the policy network model and value function estimation network model: Construct a policy network and value function estimation network based on the Transformer structure and attention mechanism; The value function estimation network consists of a policy network followed by a linear layer. The policy network consists of an encoder with a Transformer structure and a decoder with a multi-head attention layer. The encoder inputs the warehouse node feature x(v 0 ) and the client node features {x(v i )|i=1,2,..,n}, where n is a positive integer representing the number of nodes, and n∈[10,500]; after removing the non-appearing customer node features, the warehouse node features and the remaining customer node features are up-scaled using a linear layer and then concatenated to obtain {h i |i=0,1,..,n}, input it into the Transformer layer, and repeat the operation N times to get the output of the encoder, that is, the intermediate node feature {f i |i=0,1,..,n}, where N is a positive integer less than 10; The decoder consists of two multi-head attention layers, which select the intermediate node features f belonging to the current vehicle from the intermediate node features output by the encoder. cur ; Combine it with all the intermediate node features {f i |i=0,1,..,n} are input into the first multi-head attention layer together, and its output is used as the key vector and value vector of the second multi-head attention layer. The current vehicle feature y cur As the query vector of the second multi-head attention layer; the current vehicle feature y cur It consists of the node coordinates of the vehicle, the remaining load of the vehicle, and the next activation time of the vehicle; the output of the second multi-head attention layer and the intermediate node feature {f i |i=0,1,..,n} after point multiplication, and then processed by softmax function to obtain the selection probability of each node {p(A i )|i=0,1,..,n}, where A i Indicates the action of the current vehicle selecting node i; For a DVRPTW instance, the cost of generating a solution with a policy network parameterized by θ is: L(θ)=-R(S0,A0,S1)-R(S1,A1,S2)-…-R(S T-1 ,A T-1 ,S T ) The optimization goal is to minimize L(θ). At time step t, according to the stochastic gradient descent algorithm, the sampling of time step t is used to approximate the expectation. The policy network can be trained by the following formula: i t+1 =θ t -a▽ θ lnπ(A t |O t ,i t )(q(S t ,A t ,oh t )-b(S t )) where α is the step size, π(·) is the strategy, q is the action value, ω is the value function estimating the network parameters, and b(S) is the greedy rollout baseline. t ,A t ,ω t ) is obtained through the value function estimation network; in the process of updating the policy network, the value function estimation network is continuously updated at the same time; at time step t, according to the strategy π(·), the next action A is generated t+1 , according to the Bellman formula and the gradient ascent algorithm, the training value function estimates the network: oh t+1 =ω t +α[R t+1 +q(O t+1 ,A t+1 ,oh t )-q(O t ,A t ,oh t )]▽ ω q(O t ,A t ,oh t ) Where R t+1 represents the reward at time t+1; During the training process, the Adam algorithm is used to update the model parameters until the maximum number of training rounds is reached, and the trained policy network model and value function estimation network model are obtained; (4) Use the trained policy network model to solve DVRPTW: After the node features are normalized, they are input into the trained policy network model. The model outputs the customers selected at each time step. The final customer sequence is the solution of DVRPTW.

2. A dynamic vehicle path optimization method based on online deep reinforcement learning as claimed in claim 1, characterized in that: In the step (2-a) is a state set, including all system states in POMDP, that is, Each state S t Contains the customer state at time step t and vehicle status That is S t ={(C t ,V t )}, where n and m are both positive integers, n represents the number of nodes, m represents the number of vehicles, and n∈[10,500], m∈[2,125]; represents the quantity of goods demanded by the i-th customer at time step t, Indicates whether the i-th customer has been delivered or is scheduled to be delivered by a certain vehicle at time step t. Indicates whether the i-th client request has appeared at time step t; represents the position of the jth vehicle at time step t, represents the remaining cargo volume of the jth vehicle at time step t, represents the activation time of the jth vehicle at time step t, represents the activation state of the jth vehicle at time step t.

3. A dynamic vehicle path optimization method based on online deep reinforcement learning as claimed in claim 1, characterized in that: In step (3), when training the policy network, a greedy rollout baseline is used, that is, starting from the current state, a complete solution is generated in a greedy manner according to the strategy, and the cost of the solution is used as the baseline of the action value to reduce the variance of the policy gradient estimate.

Images