A tobacco logistics scheduling method based on reinforcement learning

Abstract

The present invention relates to a tobacco logistics scheduling method based on reinforcement learning, comprising the following steps: S1, randomly generating logistics information of warehouses and nodes for logistics distribution on a two-dimensional Euclidean graph; S2, constructing a deep attention convolutional network based on an encoding-aggregation-decoding architecture, inputting the logistics information into the deep attention convolutional network, and obtaining a scheduling and sorting probability matrix; S3, setting reinforcement learning parameters according to the scheduling and sorting probability matrix, and training the deep attention convolutional network; S4, obtaining real-time order demand information of warehouses and nodes, and inputting the information into the trained deep attention convolutional network to obtain a real-time scheduling and sorting probability matrix; S5, performing beam search on the real-time scheduling and sorting probability matrix of warehouses and nodes to obtain a logistics and transportation scheduling plan. Compared with the prior art, the present invention has the advantages of providing a logistics scheduling plan online and in real time, meeting the requirements of rapid logistics and transportation scheduling, and improving logistics and transportation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of logistics scheduling and transportation, and in particular to a tobacco logistics scheduling method based on reinforcement learning. Background Art

[0002] Currently, logistics and transportation scheduling for industrial enterprises is a key factor in improving their core competitiveness. Many tobacco logistics and transportation companies still rely on manual scheduling methods, which consumes a significant amount of manpower. Furthermore, this primitive scheduling method is extremely inefficient when faced with a large number of orders, resulting in poor service quality for tobacco logistics and transportation companies. Although some tobacco logistics and transportation companies use traditional, manually designed heuristic methods for logistics scheduling, these methods are unable to generate online, real-time logistics scheduling solutions and fail to meet the requirements for rapid logistics and transportation scheduling.

[0003] Tobacco logistics scheduling falls under the category of vehicle routing optimization (VRP), a core concern for the world's top logistics companies. In operations research, VRP is defined as the vehicle routing problem (VRP). VRP doesn't refer to a specific problem, but rather a general term for a class of combinatorial optimization problems. Generally speaking, VRP involves optimizing logistics transportation plans based on customer needs, available vehicles, and the need to minimize total transportation costs while still meeting customer needs.

[0004] Most tobacco logistics and transportation companies are currently exploring new logistics scheduling methods. Combining methods from related fields with tobacco logistics scheduling to develop a simple, efficient method to meet and optimize the logistics scheduling capabilities of tobacco logistics and transportation companies is one of the core issues that tobacco logistics and transportation companies urgently need to address. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a tobacco logistics scheduling method based on reinforcement learning, so as to quickly obtain logistics transportation scheduling plans online in real time, improve logistics transportation efficiency, and save labor costs.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A tobacco logistics scheduling method based on reinforcement learning uses randomly generated training data to train a network model through a reinforcement learning algorithm. The trained network model is used to generate a scheduling probability matrix for warehouses and each node. Finally, a search algorithm is combined to obtain the final logistics transportation scheduling solution. The specific steps include:

[0008] S1. Randomly generate logistics information of warehouses and nodes of logistics distribution on a two-dimensional Euclidean graph;

[0009] S2. Build a deep attention convolutional network based on the encoding-aggregation-decoding architecture, input the logistics information of the warehouse and each node into the deep attention convolutional network, and obtain the scheduling and sorting probability matrix of the warehouse and each node;

[0010] S3. Set reinforcement learning parameters according to the scheduling sorting probability matrix, and train the deep attention convolutional network according to the reinforcement learning parameters;

[0011] S4. Obtain real-time order demand information from the warehouse and each node, and input it into the trained deep attention convolutional network to obtain the real-time scheduling and sorting probability matrix of the warehouse and each node;

[0012] S5. Perform beam search on the real-time scheduling and sorting probability matrix of the warehouse and each node to obtain the logistics transportation scheduling plan.

[0013] The logistics information of the warehouse and each node includes coordinate data of the warehouse and each node and order demand information of each node.

[0014] The deep attention convolutional network includes an encoding network, an aggregation network and a decoding network.

[0015] Furthermore, the operation process of the encoding network in the deep attention convolutional network includes the following steps:

[0016] S11, initializing the coordinate data of the warehouse and each node respectively to obtain the initial embedding information of the warehouse and each node;

[0017] S12, perform convolution dimensionality reduction on the initial embedded information to obtain deep features of each node;

[0018] S13. Sample the warehouse and neighbor nodes of each node based on deep features to obtain and propagate structured information on the graph;

[0019] S14. Perform linear dimensionality reduction on the propagated information to obtain the final embedding information of the warehouse and each node.

[0020] Furthermore, the operation process of the aggregation network in the deep attention convolutional network is to aggregate the final embedding information of the warehouse and each node to obtain graph embedding information.

[0021] Furthermore, the operation process of the decoding network in the deep attention convolutional network includes the following steps:

[0022] S21, performing a concatenation operation on the final embedding information of the specific node, the vehicle remaining capacity, and the graph embedding information to obtain the context node embedding information;

[0023] S22, sampling each node according to the context node embedding information, and collecting the sampling results of each node to obtain the intermediate node embedding information;

[0024] S23. Compare the intermediate node embedding information with the final embedding information of each node to obtain the output probability of each node, and then obtain the scheduling sorting probability matrix of the warehouse and each node.

[0025] The reinforcement learning parameters in step S3 include the dimension of the initial embedded information, the dimension of the final embedded information, the dimension after convolution dimensionality reduction, the dimension after sampling of the neighboring nodes of each node, the dimension after the first feature propagation, the dimension after the second feature propagation, the number of sampling times of the neighboring nodes of each node, and the number of feature cycle extraction times.

[0026] The training process of the deep attention convolutional network in step S3 includes the following steps:

[0027] S31. Obtain the loss function of the deep attention convolutional network, input the coordinate data of the warehouse and each node and the order demand information of each node, and calculate the total path length in the logistics scheduling plan using the scheduling sorting probability matrix obtained by the deep attention convolutional network;

[0028] S32. Update the parameters of the deep attention convolutional network according to the total path length in the logistics scheduling plan, and record the number of updates;

[0029] S33. Determine whether the number of updates reaches the preset threshold. If so, the deep attention convolutional network training is completed. Otherwise, go to step S32 to continue updating the parameters.

[0030] Furthermore, the formula of the loss function of the deep attention convolutional network in step S31 is as follows:

[0031]

[0032]

[0033] Where s is a sample, the solution is π, i is the moment of the output node of the deep attention convolutional network, π i is the node output by the deep attention convolutional network at time i, that is, the node with the largest output probability, θ is the trainable parameter of the deep attention convolutional network, p θ(π|s) is the defined random strategy, and J(θ|s) is the path length after greedy decoding of the scheduling order probability matrix obtained by the deep attention convolutional network for sample s.

[0034] Furthermore, in step S32, the parameters of the deep attention convolutional network are updated through the gradient back propagation algorithm.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention uses randomly generated logistics information of warehouses and each node, trains a deep attention convolutional network through a reinforcement learning algorithm, uses the trained network model to give a scheduling and sorting probability matrix of warehouses and each node, and finally combines the search algorithm to obtain the final logistics transportation scheduling plan. The logistics scheduling plan can be given online in real time, meeting the requirements of rapid logistics transportation scheduling. Compared with the traditional heuristic method used in the prior art, it effectively improves the logistics transportation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention;

[0038] Figure 2 Schematic diagram of the structure of the deep attention convolutional network of the present invention;

[0039] Figure 3 Schematic diagram of the process of reinforcement learning training of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] Example

[0042] like Figure 1 As shown in the figure, a tobacco logistics scheduling method based on reinforcement learning is proposed. The method uses randomly generated training data to train the network model through the reinforcement learning algorithm. The trained network model is used to generate the scheduling probability matrix of warehouses and each node. Finally, the search algorithm is combined to obtain the final logistics transportation scheduling solution. The specific steps include:

[0043] S1. Randomly generate logistics information of warehouses and nodes of logistics distribution on a two-dimensional Euclidean graph;

[0044] S2. Build a deep attention convolutional network based on the encoding-aggregation-decoding architecture, input the logistics information of the warehouse and each node into the deep attention convolutional network, and obtain the scheduling and sorting probability matrix of the warehouse and each node;

[0045] S3. Set reinforcement learning parameters according to the scheduling sorting probability matrix, and train the deep attention convolutional network according to the reinforcement learning parameters;

[0046] S4. Obtain real-time order demand information from the warehouse and each node, and input it into the trained deep attention convolutional network to obtain the real-time scheduling and sorting probability matrix of the warehouse and each node;

[0047] S5. Perform beam search on the real-time scheduling and sorting probability matrix of the warehouse and each node to obtain the logistics transportation scheduling plan.

[0048] The logistics information of the warehouse and each node includes the coordinate data of the warehouse and each node and the order demand information of each node.

[0049] In this embodiment, the training data corresponding to the logistics information of the warehouse and each node includes a training set, a validation set, and an evaluation data set, wherein the data in the training set is randomly generated; the data in the validation set is generated using a random number seed; and the data in the evaluation data set is randomly generated.

[0050] like Figure 2 As shown, the deep attention convolutional network includes an encoding network, an aggregation network, and a decoding network.

[0051] The operation process of the encoding network in the deep attention convolutional network includes the following steps:

[0052] S11, initializing the coordinate data of the warehouse and each node respectively to obtain the initial embedding information of the warehouse and each node;

[0053] S12, perform convolution dimensionality reduction on the initial embedded information to obtain deep features of each node;

[0054] S13. Sample the warehouse and neighbor nodes of each node based on deep features to obtain and propagate structured information on the graph;

[0055] S14. Perform linear dimensionality reduction on the propagated information to obtain the final embedding information of the warehouse and each node.

[0056] The calculation formula for the initial embedding information is as follows:

[0057]

[0058] Among them, x i It is the coordinate data of the warehouse and each node and the index of the warehouse is 0. is the initial embedding information of the warehouse and each node, δ iis the order demand information of each node, W0, b0, W and b are learnable parameters.

[0059] In this embodiment, in step S12, a one-dimensional convolution operation is performed on the initial embedding of the warehouse and each node respectively to extract the deep features of the warehouse and each node. The size of the convolution kernel is 3×1.

[0060] The formula for information propagation in step S13 is as follows:

[0061]

[0062]

[0063]

[0064]

[0065] Among them, Q a =W q H a , K a =W k H a , V a =W v H a , respectively used to represent H a The query value, key value and information to be transmitted, H a is the node embedding after convolution of the initial embedding of each node, and and is a trainable parameter, d h =128 is the dimension after the initial embedding information convolution is reduced, softmax(·) is the function that normalizes the input to [0,1], and ReLu(·) is the nonlinear ramp function. The number of sampling times used in this embodiment is M=8 times, and the setting The parameters of each sampling are different, and the parameter matrix is ​​used each time Will Mapping to d h dimension, is the value of each node sampling its neighbor node once, It is the value after each node completes sampling of neighbor nodes. is the value after the first feature propagation, is the value after the second feature propagation. f,0 、b f,0 、W f,1 、b f,1 、W 2 and b 2In this embodiment, the neighbor nodes of each node are sampled and the first feature propagation is added with residual connection and batch normalization, while the second feature propagation is only added with batch normalization.

[0066] The calculation formula for linear dimensionality reduction in step S14 is as follows:

[0067]

[0068] in, is the final embedding of the warehouse and each node, W 3 and B 3 are learnable parameters.

[0069] The operation process of the aggregation network in the deep attention convolutional network is to aggregate the final embedding information of the warehouse and each node to obtain the graph embedding information.

[0070] In this embodiment, a one-dimensional convolution operation is performed on the final embedding of the warehouse and each node, rather than on individual node embeddings, to obtain a graph embedding. The convolution kernel size used is the number of nodes × 3, and the warehouse and the final embedding of all nodes are zero-padded with a stride of 1 before the convolution operation.

[0071] The operation process of the decoding network in the deep attention convolutional network includes the following steps:

[0072] S21, performing a concatenation operation on the final embedding information of the specific node, the vehicle remaining capacity, and the graph embedding information to obtain the context node embedding information;

[0073] S22, sampling each node according to the context node embedding information, and collecting the sampling results of each node to obtain the intermediate node embedding information;

[0074] S23. Compare the intermediate node embedding information with the final embedding information of each node to obtain the output probability of each node, and then obtain the scheduling sorting probability matrix of the warehouse and each node.

[0075] In step S21, the context node embedding is composed of the graph embedding, the vehicle remaining capacity at the current moment, and the final embedding of the node output at the previous moment, with a dimension of 3×d h At the initial moment, the warehouse is the first node of the output.

[0076] In this embodiment, the attention mechanism is used to sample the warehouse and each node. The calculation formula is as follows:

[0077]

[0078] Among them, v i is the sampled information of the warehouse and each node, umi is the compatibility between the context node and each node, n is the total number of nodes, h m′ It is the embedding of the warehouse and each node after sampling. After sampling 8 times, the embedding of the intermediate node is obtained.

[0079] In step S23, the compatibility between each node and the intermediate node is calculated with reference to the attention mechanism, and the output probability of each node is calculated based on the compatibility. The calculation formula is as follows:

[0080]

[0081] Among them, q M is the query value embedded in the intermediate node, k i is the key value embedded in each node, parameter C = 10, used to The value of u is normalized to [-10,10], Mi is the compatibility between the intermediate node and each node. The node with the highest output probability at each moment and the node whose order demand is greater than the remaining capacity of the vehicle at the current moment will be masked, and its compatibility with the intermediate node is set to -∞. The output probability p of each node at each moment is calculated. i :

[0082]

[0083] The reinforcement learning parameters in step S3 include the dimension of the initial embedded information, the dimension of the final embedded information, the dimension after convolution dimensionality reduction, the dimension after sampling of the neighboring nodes of each node, the dimension after the first feature propagation, the dimension after the second feature propagation, the number of sampling times of the neighboring nodes of each node, and the number of feature cycle extractions.

[0084] In this embodiment, the dimension of the initial embedded information d f =512, the dimension of the final embedded information d h =128, dimension d after convolution h =128, the dimension d after sampling of the warehouse and neighbor nodes of each node h =128, dimension d after the first feature propagation h =128, dimension d after the second feature propagation f =512, the number of sampling of neighboring nodes of each node M=8 and the number of feature loop extraction N=3; among them, in the encoding network of the deep attention convolutional network, from convolution dimensionality reduction to completing two feature propagations is a feature loop extraction.

[0085] The training process of the deep attention convolutional network in step S3 includes the following steps:

[0086] S31. Obtain the loss function of the deep attention convolutional network, input the coordinate data of the warehouse and each node and the order demand information of each node, and calculate the total path length in the logistics scheduling plan using the scheduling sorting probability matrix obtained by the deep attention convolutional network;

[0087] S32. Update the parameters of the deep attention convolutional network according to the total path length in the logistics scheduling plan, and record the number of updates;

[0088] S33. Determine whether the number of updates reaches the preset threshold. If so, the deep attention convolutional network training is completed. Otherwise, go to step S32 to continue updating the parameters.

[0089] The formula of the loss function of the deep attention convolutional network in step S31 is as follows:

[0090]

[0091]

[0092] Where s is an example, the solution is π, i is the moment of the output node of the deep attention convolutional network, π i is the node output by the deep attention convolutional network at time i, that is, the node with the largest output probability, θ is the trainable parameter of the deep attention convolutional network, p θ (π|s) is the defined random strategy, and J(θ|s) is the path length after greedy decoding of the scheduling order probability matrix obtained by the deep attention convolutional network for sample s.

[0093] In step S32, the parameters of the deep attention convolutional network are updated through the gradient backpropagation algorithm.

[0094] In this embodiment, the width parameter of the beam search is 3, and a beam search is performed on the scheduling order probability matrix to obtain a set of logistics scheduling solutions, from which the optimal logistics scheduling solution is selected as the final logistics scheduling solution.

[0095] When implementing it specifically, Figure 3 As shown, reinforcement learning training includes the following steps:

[0096] Adopt the structure of deep attention convolutional network and define a current network and a baseline network;

[0097] Before each training round, a randomly generated evaluation dataset is used to update the parameters of the current network. The baseline network uses greedy decoding (selecting the node with the highest output probability at each decoding moment) to obtain the solution, while the current network obtains the solution through beam search.

[0098] Update the gradient of the current network. The formula is as follows:

[0099]

[0100] The baseline b(s) is the cost of the solution obtained by the baseline network (the average path length of the training data generated in each batch). After the gradient is determined, the network parameters are optimized using the Adam algorithm.

[0101] After completing one round of training, determine whether to update the parameters of the current network to the baseline network. Calculate the average cost of the two networks after greedy decoding on the evaluation dataset (the average path length on the evaluation dataset). If the average cost of the current network is less than the average cost of the baseline network, calculate their difference. If it is greater than the gain α = 0.01, update the parameters of the current network to the baseline network and start the next round of training.

[0102] In addition, it should be noted that the specific embodiments described in this specification may be named differently, and the above content described in this specification is merely an example of the structure of the present invention. Any equivalent changes or simple changes made based on the structure, features and principles of the present invention are included in the protection scope of the present invention. Those skilled in the art of the present invention may make various modifications or supplements to the specific examples described or adopt similar methods, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A tobacco logistics scheduling method based on reinforcement learning, characterized in that: The specific steps include: S1. Randomly generate logistics information of warehouses and nodes of logistics distribution on a two-dimensional Euclidean graph; S2. Build a deep attention convolutional network based on the encoding-aggregation-decoding architecture, input the logistics information of the warehouse and each node into the deep attention convolutional network, and obtain the scheduling and sorting probability matrix of the warehouse and each node; S3. Set reinforcement learning parameters according to the scheduling sorting probability matrix, and train the deep attention convolutional network according to the reinforcement learning parameters; S4. Obtain real-time order demand information from the warehouse and each node, and input it into the trained deep attention convolutional network to obtain the real-time scheduling and sorting probability matrix of the warehouse and each node; S5. Perform beam search on the real-time scheduling and sorting probability matrix of warehouses and nodes to obtain the logistics transportation scheduling plan; The reinforcement learning parameters in step S3 include the dimension of the initial embedded information, the dimension of the final embedded information, the dimension after convolution dimensionality reduction, the dimension after sampling of the neighboring nodes of each node, the dimension after the first feature propagation, the dimension after the second feature propagation, the number of sampling times of the neighboring nodes of each node, and the number of feature cycle extractions.

2. The tobacco logistics scheduling method based on reinforcement learning according to claim 1, characterized in that: The logistics information of the warehouse and each node includes coordinate data of the warehouse and each node and order demand information of each node.

3. The tobacco logistics scheduling method based on reinforcement learning according to claim 2, characterized in that: The operation process of the encoding network in the deep attention convolutional network includes the following steps: S11, initializing the coordinate data of the warehouse and each node respectively to obtain the initial embedding information of the warehouse and each node; S12, perform convolution dimensionality reduction on the initial embedded information to obtain deep features of each node; S13. Sample the warehouse and neighbor nodes of each node based on deep features to obtain and propagate structured information on the graph; S14. Perform linear dimensionality reduction on the propagated information to obtain the final embedding information of the warehouse and each node.

4. The tobacco logistics scheduling method based on reinforcement learning according to claim 3, characterized in that: The operation process of the aggregation network in the deep attention convolutional network is to aggregate the final embedding information of the warehouse and each node to obtain graph embedding information.

5. The tobacco logistics scheduling method based on reinforcement learning according to claim 4, characterized in that: The operation process of the decoding network in the deep attention convolutional network includes the following steps: S21, performing a concatenation operation on the final embedding information of the specific node, the vehicle remaining capacity, and the graph embedding information to obtain the context node embedding information; S22, sampling each node according to the context node embedding information, and collecting the sampling results of each node to obtain the intermediate node embedding information; S23. Compare the intermediate node embedding information with the final embedding information of each node to obtain the output probability of each node, and then obtain the scheduling sorting probability matrix of the warehouse and each node.

6. The tobacco logistics scheduling method based on reinforcement learning according to claim 1, characterized in that: The training process of the deep attention convolutional network in step S3 includes the following steps: S31. Obtain the loss function of the deep attention convolutional network, input the coordinate data of the warehouse and each node and the order demand information of each node, and calculate the total path length in the logistics scheduling plan using the scheduling sorting probability matrix obtained by the deep attention convolutional network; S32. Update the parameters of the deep attention convolutional network according to the total path length in the logistics scheduling plan, and record the number of updates; S33. Determine whether the number of updates reaches the preset threshold. If so, the deep attention convolutional network training is completed. Otherwise, go to step S32 to continue updating the parameters.

7. The tobacco logistics scheduling method based on reinforcement learning according to claim 6, characterized in that: The formula of the loss function of the deep attention convolutional network in step S31 is as follows: Where s is an example, the solution is π, i is the moment of the output node of the deep attention convolutional network, π i is the node output by the deep attention convolutional network at time i, that is, the node with the largest output probability, θ is the trainable parameter of the deep attention convolutional network, p θ (π|s) is the defined random strategy, and J(θ|s) is the path length after greedy decoding of the scheduling order probability matrix obtained by the deep attention convolutional network for sample s.

8. The tobacco logistics scheduling method based on reinforcement learning according to claim 6, characterized in that: In step S32, the parameters of the deep attention convolutional network are updated through the gradient back propagation algorithm.

Images