A tobacco order vehicle distribution optimization method based on deep reinforcement learning

Abstract

The application discloses a tobacco order vehicle distribution optimization method based on deep reinforcement learning, and the optimization method is as follows: step one: using a data structure of a graph to represent customer orders to be distributed on the day, and obtaining a set of available distribution vehicle models as the input of an order vehicle distribution model; step two: designing a state space and an action space of the order vehicle distribution model; the application has the beneficial effects that: an A3C reinforcement learning algorithm is used to autonomously learn an order vehicle distribution strategy, a double-pointer decoding framework is proposed for generating a distribution order and a distribution vehicle sequence in view of the pain point that a transportation vehicle demand is not clear in actual distribution business, a design mode of a distribution environment state, an action and a reward is described in detail, and finally, through experiments on actual order data provided by a company, the method can generate a distribution distance shorter and a total number of used vehicles less in a very short time compared with a traditional heuristic solver.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to an optimization method for tobacco order vehicle delivery based on deep reinforcement learning. Background Technology

[0002] The tobacco industry occupies an important position in my country's national economy, and tobacco logistics determines the quality and development of the tobacco industry. According to research by Joerss et al., more than 50% of the operating costs of logistics companies are spent on customer order delivery, which is known as the last-mile problem. The core of last-mile delivery is the vehicle routing problem (VRP), which aims to find the optimal delivery route from the warehouse to the customer under relevant constraints to reduce operating costs. In the most basic VRP, given a batch of customer orders and a warehouse, the goal is to find the set of routes with the minimum delivery cost without adding other constraints. Each route starts from the warehouse and returns to the warehouse and must deliver all customer orders.

[0003] However, with the increasing scale and complexity of logistics and transportation operations, more constraints need to be considered when solving delivery solutions. These constraints include delivery time, delivery habits, vehicle capacity, and air pollution caused by the delivery plan. The tobacco order vehicle delivery problem is a vehicle routing problem with multiple vehicle types and multiple customers in a single warehouse and with vehicle capacity constraints (MVCVRP). That is, when delivering customer orders, there are multiple vehicle types to choose from, each with a different capacity. The goal is to select the appropriate vehicle type for each customer order while satisfying the vehicle capacity constraints, so as to minimize the delivery cost.

[0004] Hua et al. used integer programming to plan and search for the shortest path for ocean freighter transportation routes; He et al. found that the number of turns in the route during vehicle delivery has a significant impact on delivery costs and used integer programming to solve this problem. While these exact solution algorithms can find the optimal solution, the solution time increases exponentially with the scale of the problem, making them impractical. Conversely, heuristic algorithms such as tabu search and particle swarm optimization can find approximate solutions in a shorter time. Song et al. considered the driver's familiarity with different customer routes and designed a tabu search algorithm to solve the VRP problem with time window constraints; LE Based on the respective advantages of immune algorithms and particle swarm optimization algorithms in VRP, NG et al. proposed a new concentration-based immune particle swarm optimization algorithm to solve the VRP problem with multiple distribution centers, combining the concept of antibody concentration. Qin et al. treated customer satisfaction and path cost as the objective function of the VRP problem and developed a genetic algorithm to solve it. The results showed that a slight increase in path cost can greatly improve the average customer satisfaction. These heuristic algorithms often require domain experts to manually design corresponding strategies according to different problems. The quality of the solution is limited by the expert's prior knowledge, and the generalization ability of the algorithm is not good. When facing different types of VRP problems, it is necessary to find new solutions.

[0005] With the development of artificial intelligence in recent years, deep reinforcement learning methods have been widely used in the field of operations research and optimization. Compared with traditional algorithms, neural networks can discover hidden features inside the problem, describe constraints that cannot be explicitly written, and learn the optimal policy by maximizing the reward function given the solution objective, with the agent autonomously interacting with the environment. Inspired by AlphaGo Zero, Wang et al. designed a lightweight reinforcement learning framework based on Q-learning algorithm to solve various types of VRP problems. They used GAT and GRU to dynamically encode the environment and used MCTS to select the optimal action. Paul et al. proposed a model-free method, RL SolVeR Pr, to solve VRP problems. First, they clustered customers based on their demand and the angle and distance to the warehouse to determine the customer set that each vehicle needs to transport, reducing the dimensionality of the action space. Then, they used the Q-learning algorithm for training. However, in the above methods, the environmental reward is only obtained at the end of an episode. This sparse reward affects the convergence speed of the model.

[0006] In recent years, an increasing number of scholars have studied CVRP with vehicle volume constraints by exploring reasonable delivery schemes through reinforcement learning and interaction with the environment. Tang et al. proposed an RL-based method to solve VRP with more constraints, using the Lagrange relaxation operator to transform hard constraints into soft constraints as part of the objective function, enabling the agent to autonomously learn the relevant constraints, and applied it to VRP with time window constraints and vehicle volume constraints. HORANONT et al. solved the CVRP problem with time window constraints by proposing an optimization model based on RL and complementary tree regression. Unlike other methods, customer demand may exceed vehicle volume, so each customer may be visited multiple times. In the above two papers, the vehicle volume used for delivery is fixed, i.e., only one type of vehicle is available. However, in actual logistics operations, multiple vehicle types are usually available. Therefore, considering the multi-vehicle delivery constraints, the delivery scheme can be further optimized.

[0007] The existing delivery vehicles for tobacco orders are not conducive to reducing the company's delivery costs. Summary of the Invention

[0008] The purpose of this invention is to provide a method for optimizing tobacco order vehicle delivery based on deep reinforcement learning, thereby reducing the company's delivery costs.

[0009] To achieve the above objectives, this invention provides the following technical solution: a tobacco order vehicle delivery optimization method based on deep reinforcement learning. First, tobacco order delivery instances are represented using a graph, and the set of available delivery vehicles of the company is used as input to a neural network. Then, two sequence decoders generate delivery orders and delivery vehicle sequences respectively. Reinforcement learning is used to adjust the parameters of the neural network based on the reward value obtained at each step. Under the premise of ensuring delivery constraints, the decision model learns the selection strategy for orders and vehicles, striving to minimize the total delivery path and the total number of vehicles used, thereby reducing the company's delivery costs. The optimization method is as follows:

[0010] Step 1: Represent the customer orders that need to be delivered that day using a graph data structure, and at the same time obtain the set of available delivery vehicle models as input for the order vehicle delivery model;

[0011] Step 2: Design the state space and action space of the order vehicle delivery model;

[0012] Step 3: Design the reward function for the order vehicle delivery model to guide the training of the delivery model;

[0013] Step 4: Establish an order vehicle delivery model. This invention mainly uses Graph Attention Network (GAT) and Transformer encoder to fit the delivery model. At each time step, the customer nodes visited in the previous time step and the remaining volume of the delivery vehicle are used as dynamic feature information, and the customer orders and delivery vehicle models are used as static feature information. The model outputs the customer nodes visited in the next time step and the vehicle models used.

[0014] Step 5: After defining the state, action, reward, and delivery model information, use a deep reinforcement learning algorithm to update the parameters of the order vehicle delivery model, enabling the model to select appropriate customer nodes and delivery vehicle types.

[0015] As a preferred technical solution of the present invention, the delivery customer order and the delivery vehicle are described as follows:

[0016] A tobacco order delivery instance is described using a graph G = (V, E), where V = {0, ..., n} represents the set of warehouse and customer nodes, index i = 0 represents a warehouse node, and the remaining indices i ∈ {1, ..., n} represent customer nodes. Each customer node i ∈ V has two features: coordinate features. This indicates the customer's geographical location and demand characteristics. This represents the volume of tobacco goods that the customer needs to deliver. The warehouse node only has coordinate features (x0, y0), and the edge set E∈V*V represents the distance between the warehouse and the customer node, as well as the distance between the customer nodes. ij Let e ​​represent the actual distance between two client nodes i and j, and let e represent the distance between them. ij =e ji Similarly, assuming that all nodes in G are connected, G is an undirected fully connected graph. Based on the existing vehicle data of the company's fleet, we define the vehicle set C = {c1, ..., c2}. m There are a total of m types of vehicle models to choose from, each with different length, width, and height dimensions. Based on business needs, the delivery result only needs to provide the index of the selected delivery vehicle model, meaning that the quantity of each vehicle model is considered sufficient.

[0017] As a preferred technical solution of the present invention, the state space and action space of the single-vehicle delivery model are defined as follows:

[0018] The state space contains all the observations of the delivery model. The quality of the observation space design directly affects the training quality of the delivery model. To ensure that the order vehicle delivery model obtains sufficiently comprehensive customer vehicle information while considering the dynamic changes in customer node characteristics and delivery vehicle volume during the delivery process, the state space of the order vehicle delivery model at time t is designed as follows:

[0019]

[0020] Where X represents the set of warehouse customer node coordinates, D represents the set of customer demand quantities, E represents the node distance matrix, and C represents the set of delivery vehicle types. These states are static information used to better describe delivery customer order instances. t-1 This represents the client node accessed at time t-1, v t This represents the remaining volume of the vehicle at time t. This indicates client nodes that have already been visited. These states represent dynamic information about customer nodes that have not yet been visited, helping the order vehicle delivery model learn the optimal order delivery solution.

[0021] The action space of the order vehicle delivery model is divided into two parts: one part is the generated sequence of delivery customer nodes. in Furthermore, warehouse nodes represented by 0 may appear multiple times, while customer nodes will only appear once; the other part is the delivery vehicle vehicle sequence. Each model The customer node for delivery corresponds to π x Customers between two zeros, where c0 represents an invalid vehicle model index. Note that only customers whose order volumes are all greater than the remaining vehicle volume v are considered invalid. t When this occurs, it indicates that the current delivery vehicle needs to return to the warehouse and a new vehicle model needs to be selected for delivery; therefore, the remaining actions are as follows: t This has no practical significance; it uses a mask to mask all vehicle model indices, leaving only c0 as the selectable option.

[0022]

[0023] As a preferred technical solution of the present invention, a reward function for the order vehicle delivery model is designed: the reward function is responsible for evaluating the quality of the actions taken by the strategy and directly determines the optimization direction of the strategy; in view of the problem that the reward function in the existing work cannot reflect the quality of the actions in a timely manner and has sparse rewards, the following single-step reward function is defined.

[0024]

[0025] in Indicates vehicle model Delivery cost per kilometer Indicates the vehicle model used Fixed costs of delivery This represents the total number of times the i-th vehicle type is used in the delivery vehicle sequence. Let t represent the customer nodes visited at time t, end represent the completion of delivery to all customer nodes, and λ and ω represent the weights of different costs.

[0026] Since the purpose of this delivery model is to select the appropriate vehicle type for each customer order and minimize the total distance traveled by delivery vehicles while using as few vehicles as possible, thereby reducing the tobacco company's logistics costs, the reward function needs to consider both the dynamic costs caused by different delivery routes and the fixed costs incurred by using different vehicle types. When t≠end, the main consideration is the magnitude of the dynamic costs caused by different delivery routes; actions with higher route costs will receive lower reward values, thus reducing the probability of those actions occurring. When t=end, the combined impact of dynamic costs and fixed costs caused by different vehicle types is considered; using more vehicle types will result in lower reward values, prompting the delivery model to learn how to complete customer order deliveries using as few vehicle types as possible. Since the reinforcement learning objective is to maximize the cumulative reward value, but the model aims to find the minimum value, a negative sign is added before the reward function.

[0027] As a preferred technical solution of the present invention, the order vehicle delivery model is constructed as follows:

[0028] The system designed in this invention is based on an encoder-decoder architecture. Its input states include warehouse node coordinates, customer order node coordinates, customer order demand quantity, and node distance matrix. This delivery customer instance information can be represented in the form of a graph; while the delivery vehicle set can be represented by a sequence. Therefore, based on the characteristics of the input data, the encoder of the delivery model is built using a graph attention neural network and a Transformer to extract delivery customer instance features and delivery vehicle features, respectively. Since the output of the delivery model is also serialized, the decoder uses a pointer mechanism to output, determining the output of the current time step based on the output of the previous time step.

[0029] As a preferred technical solution of the present invention, the parameters of the order vehicle delivery model are updated as follows:

[0030] In reinforcement learning, a complete episode is defined as the process starting from an initial state s and continuing until the task ends. During this process, a large number of states s and actions a are stored, forming a sequence τ. The probability of sequence τ occurring is:

[0031]

[0032] The reward obtained from a sequence τ can be represented as R(τ). Then, the expected reward at the end of an episode is:

[0033]

[0034] Since the goal of reinforcement learning is to maximize the reward function, the gradient ascent method can be used to update the network parameters θ. The solution process is as follows:

[0035]

[0036] This invention uses the A3C reinforcement learning algorithm to train the order vehicle delivery model. The A3C algorithm is essentially an Actor-Critic algorithm, therefore requiring the separate construction of Actor and Critic networks. The Actor network is constructed using the encoder-decoder network proposed in step four, and its output is the probability distribution of delivery customer nodes. and the probability distribution of delivery vehicle nodes Where θ x and θ c These represent the network parameters of the delivery customer node and the delivery vehicle decoder, respectively; the Critic network is constructed using a two-layer MLP, outputting the cumulative reward value V(s) obtainable from time t to the end of the episode. t ), used to guide the training of the Actor network;

[0037] A3C employs an asynchronous update approach. Compared to the synchronous update approach of A2C, in each training round, the global network waits for each thread to complete its current episode before synchronously updating its parameters. This asynchronous update strategy improves the update speed of the global network parameters and accelerates model convergence. In other words, in the asynchronous update strategy, each thread corresponds to different network parameters. As soon as a thread completes an episode or reaches the set maximum number of updates, it calculates the gradient and passes it to the global network to update the parameters. The improved single-step update form is as follows:

[0038]

[0039] Since the Actor network uses two decoding networks to output the delivery customer node and the delivery vehicle type respectively, the loss function of the Actor network is as follows:

[0040]

[0041] Because the Critic network is used to estimate the cumulative reward value V(s) t Therefore, the mean squared error loss function is used to update the Critic network parameters, and its loss function is as follows:

[0042]

[0043] During the learning process of delivery strategies, the A3C algorithm adjusts the parameters according to the direction of loss function reduction in the order vehicle delivery model, so that the model can eventually generate a suitable sequence of delivery customer nodes and corresponding sequence of delivery vehicle models.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This paper employs the A3C reinforcement learning algorithm to autonomously learn order vehicle delivery strategies. Addressing the pain point of unclear transportation vehicle demand in actual delivery operations, a dual-pointer decoding framework is proposed to generate delivery orders and delivery vehicle sequences. The design of delivery environment states, actions, and rewards is elaborated in detail. Finally, experiments using actual order data provided by the company are conducted, comparing the proposed method with traditional heuristic solvers. The results show that this method can generate loading schemes with shorter delivery distances and fewer vehicles in a very short time. Attached Figure Description

[0046] Figure 1 This is a diagram illustrating an example of tobacco order delivery according to the present invention;

[0047] Figure 2 This is a schematic diagram of a tobacco order delivery example of the present invention;

[0048] Figure 3 This is a structural diagram of the tobacco order delivery model of the present invention;

[0049] Figure 4 Training curve one for the structured tobacco order delivery model of the present invention;

[0050] Figure 5 The second training curve for the structured tobacco order delivery model of this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] This invention uses the reinforcement learning training environment gym provided by OpenAI to simulate the tobacco order vehicle delivery environment based on real customer order data, vehicle model data, and city latitude and longitude data of a tobacco company. The Actor network and Critic network of the order delivery vehicle model are constructed using the PyTorch neural network library. The specific implementation steps are described in detail below.

[0053] Step 1: Experimental Scenario Design

[0054] The model was trained using actual city latitude and longitude coordinates and vehicle model data provided by the tobacco company. Warehouse and customer coordinates in the training data were randomly generated according to a uniform distribution, ranging from [73.33, 135.05] x [3.51, 53.33]. The fact that warehouse and customer coordinates could be distributed anywhere on the map increased the training difficulty. The tobacco company currently has 31 different vehicle models with varying volumes, all of which can be used for customer order delivery. Customer order volumes were randomly generated according to an exponential distribution. Because customer orders are indivisible in actual business needs, the generated order volume must be less than the maximum vehicle volume in the vehicle model set. An example of tobacco order delivery is shown below. Figure 1 As shown, its graphical model is as follows: Figure 2 As shown;

[0055] Step Two: Obtaining and Generating Delivery Orders

[0056] Analysis of actual delivery order data provided by the tobacco company in June 2020 revealed that the number of customer orders delivered daily ranged from 15 to 30. Therefore, we generated three test datasets of different sizes: 20 customer orders, 30 customer orders, and actual delivery order data. For the first two datasets, warehouse, customer coordinates, and order demand were randomly generated based on constraints, while the actual delivery orders were derived from the June 2020 delivery order data.

[0057] Step 3: Building the Order Vehicle Delivery Model

[0058] The order vehicle delivery model uses a neural network for fitting, and the delivery model structure diagram is as follows: Figure 3 As shown, it comprises two networks: Actor and Critic. The Actor network is designed with an encoder-decoder structure. The encoder uses a 6-layer graph attention network (GAT) and a Transformer encoder to extract delivery customer instance features and delivery vehicle features, respectively. Note that the delivery vehicle features do not have a specific order, so positional encoding is omitted. The decoder consists of a delivery customer sequence decoder and a delivery vehicle sequence decoder, both constructed using pointer networks. The pointer network is a type of attention mechanism that uses the encoder's output features as the query, and the customer node features and delivery vehicle features as the key and value to calculate attention weights. The attention weights are then converted into a probability distribution using softmax to select the corresponding action for output. The Critic network consists of two fully connected layers: one hidden layer and one output layer. Each hidden layer contains 64 neurons and uses the tanh activation function. The output layer contains one neuron that outputs the value of the action, also using the tanh activation function.

[0059] Step 4: Order Vehicle Delivery Model Training

[0060] During the experiment, the order vehicle delivery network needs to use reinforcement learning algorithms to continuously adjust the model parameters so that the final output sequence of delivery customer nodes and the corresponding sequence of delivery vehicle models can complete the order delivery using vehicles with the largest possible capacity, while reducing the total delivery path distance. Since A3C adopts a policy gradient-based method, it has a better convergence speed than value function-based methods. At the same time, A3C uses a multi-threaded asynchronous update method, which can effectively solve the problem of low data update efficiency. Therefore, the reinforcement learning algorithm of this invention is A3C.

[0061]

[0062]

[0063] The loss function of the A3C algorithm is shown in the above formula; where r t γ represents the reward value obtained at time step t, which is determined according to the reward value formula; γ represents the discount rate, which is set to 1 in this paper. L represents the output value of the Critic network at time step t. actor and L critic These are used to update the parameters of the Actor and Critic networks, respectively. Unlike other methods that only calculate the cost based on the delivery sequence for parameter updates when an episode ends, this invention calculates the delivery cost based on the reward value formula at each time step t, thereby calculating the corresponding loss function. Therefore, there is no problem of sparse rewards and the model update efficiency is higher.

[0064] When training the order vehicle delivery model using A3C, at each time t, the model in different threads inputs the observed information into the network of the corresponding thread, and selects the delivery customer based on the output of the Actor network. and the delivery vehicles used At the same time, a new environmental state s is obtained. t+1 When the environment state of a certain thread is at the end of an episode or reaches the maximum number of training iterations, the Critic network uses the currently stored environment state s to obtain the value assessment of the corresponding state, and guides the action output of the Actor network through discounted value assessment and discounted reward. Regarding parameter updates, both the Actor network and the Critic network will use backpropagation to calculate gradients and backfeed them to the global network to update parameters as long as a certain thread meets the update conditions.

[0065] Step 5: Simulation Experiment Results

[0066] The changes in two indicators, average delivery distance and total number of vehicles used, during the training of the order vehicle delivery model in this example are as follows: Figure 4 , Figure 5 As shown, the average delivery distance and the total number of vehicles used are the two metrics, respectively. During the training process, the two metrics of the delivery model decreased rapidly and then stabilized, verifying the effectiveness of the order vehicle delivery model proposed in this invention. In addition, the model proposed in this paper was compared with the traditional genetic algorithm and the OR-tools proposed by Google on three test datasets of different sizes.

[0067] The costs required by three different algorithms to solve CVRP instances of different sizes and vehicle models are shown in the table below:

[0068]

[0069] As can be seen from the table, while the solution time of the delivery model proposed in this invention is slightly higher than that of OR-Tools, its average number of vehicles used is much smaller than that of the OR-Tools solution, and the required path distance is close to or better than that of OR-Tools. Furthermore, while ensuring lower costs, the solution time of the delivery model is 500 times faster than that of the genetic algorithm.

[0070] Although embodiments of the invention have been shown and described in detail above, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing tobacco order vehicle delivery based on deep reinforcement learning, characterized in that: The optimization method is as follows: Step 1: Represent the customer orders that need to be delivered on the same day using a graph data structure, and obtain the set of available delivery vehicle models. The set of delivery vehicle models can be represented by a sequence, which serves as the input to the order vehicle delivery model. The input states include warehouse node coordinates, customer order node coordinates, customer order demand quantity, and node distance matrix. Step 2: Design the state space and action space of the order vehicle delivery model; the state space contains all the observations of the delivery model; The action space of the order vehicle delivery model is divided into two parts: one part is the generated sequence of delivery customer nodes, and the other part is the sequence of delivery vehicle models. Step 3: Design the reward function for the order vehicle delivery model to guide the training of the delivery model; the reward function needs to take into account both the dynamic costs caused by different delivery routes and the fixed costs incurred by using different vehicle models; Step 4: Establish an order vehicle delivery model; The designed model is trained using the A3C algorithm. Based on the encoder-decoder architecture, the encoder of the delivery model is built using a graph attention neural network and a Transformer, according to the features of the input data, to extract the features of delivery customer instances and delivery vehicle models, respectively. The decoder uses a pointer mechanism for output; Step 5: After defining the state, action, reward, and delivery model information, use a deep reinforcement learning algorithm to update the parameters of the order vehicle delivery model, enabling the model to select appropriate customer nodes and delivery vehicle types.

2. The method for optimizing tobacco order vehicle delivery based on deep reinforcement learning according to claim 1, characterized in that: The order vehicle delivery model needs to obtain two types of information: first, customer order delivery instance information; and second, delivery vehicle volume information.

3. The method for optimizing tobacco order vehicle delivery based on deep reinforcement learning according to claim 1, characterized in that: A3C uses an asynchronous update method.

Images