An intelligent logistics planning method based on deep reinforcement learning

Abstract

This invention proposes an intelligent logistics planning method based on deep reinforcement learning, relating to the technical field of logistics planning. First, historical vehicle delivery information is collected and a fully connected graph is constructed. Then, a sequence dataset is built using the node set of the fully connected graph. Next, an intelligent logistics delivery planning model is constructed, and the sequence dataset is used to train the intelligent logistics delivery planning model, resulting in a trained model. Finally, the node sequence of the vehicle delivery information to be planned is input into the trained intelligent logistics delivery planning model, and the solution sequence output by the intelligent logistics delivery planning model is used as the planning result. The actual logistics planning system is abstracted into a location-route planning problem, and deep reinforcement learning is used to solve it. This approach enables the acquisition of feasible and optimal solutions to the location-route planning problem in real time and effectively improves the quality of the solution.

Description

Technical Field

[0001] This invention relates to the technical field of logistics planning, and more specifically, to an intelligent logistics planning method based on deep reinforcement learning. Background Technology

[0002] The Location-Routing Problem (LRP) is a core application problem in logistics systems, widely used in numerous industries, such as site selection and service route optimization for electric vehicle charging stations or battery replacement stations, site selection and route planning for emergency equipment and facilities, food distribution network planning, waste recycling, and marine supply chain network design. Solving the Location-Routing Problem ensures efficient supply chain operations. Through scientific and rational site selection and route planning, warehousing and logistics efficiency is improved, operating costs are reduced, and rapid and accurate goods movement is achieved, thereby enhancing enterprise competitiveness and customer satisfaction.

[0003] Currently, algorithms for completing distribution center site selection and route planning fall into two main categories: exact solution methods and heuristic methods. Exact solution methods can obtain the optimal solution to the site selection-route planning problem, but since this is an NP-hard problem, the computational cost becomes unacceptable as the problem size increases. Heuristic methods are approximate methods, and the quality of their solutions cannot be guaranteed. Furthermore, if the problem instance changes even slightly, heuristic methods need to re-search for solutions, making real-time solutions difficult. Summary of the Invention

[0004] To address the issues of slow solution speed and poor solution quality in current location-routing problems, this invention proposes an intelligent logistics planning method based on deep reinforcement learning. This method abstracts the actual logistics planning system into a location-routing planning problem. By constructing an end-to-end intelligent logistics distribution planning model and training the location-routing problem using reinforcement learning, the efficiency and quality of solving the location-routing planning problem are effectively improved.

[0005] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows:

[0006] A method for intelligent logistics planning based on deep reinforcement learning, comprising:

[0007] S1. Collect historical vehicle delivery information and construct a fully connected graph, then use the node set of the fully connected graph to construct a sequence dataset;

[0008] S2. Construct an intelligent logistics distribution planning model, and train the intelligent logistics distribution planning model using a sequence dataset to obtain a trained intelligent logistics distribution planning model;

[0009] S3. Input the node sequence of the vehicle delivery information to be planned into the trained intelligent logistics delivery planning model, and use the solution sequence output by the intelligent logistics delivery planning model as the planning result.

[0010] This technical solution uses the sequence of instances of the location-route planning problem as input to the intelligent logistics distribution planning model, and enables the trained intelligent logistics distribution planning model to output the sequence of solutions to the location-route planning problem in an autoregressive manner, thereby achieving rapid output of feasible and better solutions to the location-route planning problem.

[0011] Preferably, in step S1, the fully connected graph is defined as g(X), where X is the set of nodes consisting of distribution center nodes and customer nodes:

[0012] X={x i =(s i ,d i ,c i ,o i ,P)|i∈1,…,D,…,n}

[0013] Where, x i Let be the original characteristics of node i; D be the number of distribution centers; n be the total number of nodes including both distribution center nodes and customer nodes; s i Let d be the two-dimensional coordinates of node i; i This represents the demand for a customer node; if the node is a distribution center node, this value is set to 0. i and o i These represent the capacity and setup cost of the distribution center, respectively; if the node is a customer node, both are set to 0; P is the vehicle capacity.

[0014] Preferably, the intelligent logistics distribution planning model includes an encoder and a decoder;

[0015] The encoder performs an attention mechanism operation on sequence X to obtain a node embedding vector;

[0016] The decoder performs autoregression based on the current state information of the decoder and the embedding vectors of each node to obtain the final solution sequence π = (π1, ..., π). t ,…,π n ), where π t This indicates the node selected by the model at time step t;

[0017] Calculate the total cost of all solution sequences, and select the sequence with the lowest cost as the final solution output of the intelligent logistics distribution planning model.

[0018] Preferably, the encoder consists of an embedding layer and N layers of multi-head attention modules;

[0019] The embedding layer takes the original features x of each input node i Mapping to node embedding features The mapping process is as follows:

[0020]

[0021]

[0022] Among them, W d and b d W represents the network parameters embedded in the distribution center node. c and b c This represents the network parameters embedded in the client node;

[0023] The multi-head attention module includes a multi-head attention sub-layer, a feedforward layer, and a normalization layer; the multi-head attention module embeds nodes into features h. l-1 Updated to h 2 , to obtain the node embedding vector The update process is as follows:

[0024]

[0025]

[0026] Where l represents the l-th multi-head attention module, l∈1,…N, MHA represents the multi-head attention sub-layer, FF represents the feedforward layer, and BatchNorm represents the batch normalization operation.

[0027] Here, the original features of the input nodes are projected into a high-dimensional space using an embedding layer. A multi-head attention module is used to separately compute the node embeddings of distribution centers and customers with different information to obtain more accurate information about the location-route planning problem instance. Finally, a normalization layer is used to make the model training more stable and accelerate the convergence speed of the model.

[0028] Preferably, the decoder is based on the node embedding vector output by the encoder. Using the information from the already constructed partial solution sequence, the process of calculating the node selection at the next time step t includes:

[0029] SA. Set an indicator vector. Indicates the current state;

[0030] SB. Calculation of the indicator vector through a single-head attention mechanism Update, With the embedding vector contained in all nodes The node information in the data is integrated;

[0031] SC. It divides feasible nodes into feasible and infeasible nodes through masking operations and calculates the selection probability of feasible nodes through an attention mechanism;

[0032] SD. Select the node for the next time step based on the obtained feasible node selection probability and store it in the solution sequence until a complete solution sequence is constructed.

[0033] Preferably, in step SA, the indication vector The expression is as follows:

[0034]

[0035] in, This represents the mean vector of all node embeddings output by the encoder. This represents the embedding vector of the distribution center node currently providing services along this path. ξ represents the embedding vector of the model output node in the previous time step (t-1). d ,ξ v ,ξ g g is a learnable parameter used for filling when no nodes are selected. t For a two-dimensional vector, the expression is as follows:

[0036] g t =(c d,t ,P t ) T

[0037] c d,t P represents the remaining quantity of goods at the distribution center. t c represents the remaining cargo quantity of the vehicle currently in service. d,t and P t The update logic is as follows:

[0038]

[0039]

[0040] in, This represents the set of distribution center nodes.

[0041] Preferably, step SB includes the following processes:

[0042] SB1.Use Generate a query using the embedding vectors of all nodes. Generate key and value using the following formula:

[0043]

[0044]

[0045]

[0046] in, W k and W v For network parameters;

[0047] SB2. Generate an n-dimensional vector and normalize it. The calculation formula is as follows:

[0048]

[0049] Where, Φ t It is the set of infeasible nodes;

[0050] SB3. Through an attention mechanism, the indicator vector is... For updates The updated formula is as follows:

[0051]

[0052] Preferably, using an indicator vector The feasible node selection probability is calculated from the embedding vectors of all nodes output by the encoder. The process includes:

[0053] use The query is updated by generating the key from the embeddings of the nodes produced by the encoder, using the following formula:

[0054]

[0055]

[0056] The probability of selecting a feasible node is calculated using the following formula:

[0057]

[0058]

[0059] Among them, u ci p represents the probability of selecting a feasible node from the current state to add to the partial solution. θ (π t =i|X,π1,…,π t-1 ) represents a random policy.

[0060] Preferably, the intelligent logistics distribution planning model utilizes multiple reasoning techniques to perform reasoning, and outputs the sequence with the lowest cost obtained through reasoning as the final solution. The process includes:

[0061] Given m distinct distribution centers as starting points, generate h solution sequences from each starting point using a sampling method;

[0062] Perform p rotation transformations on all nodes in g(X) at different angles, and sample to obtain p×m×h solution sequences;

[0063] Calculate the total cost of all solution sequences and select the sequence with the lowest cost as the final solution output.

[0064] Here, we use multi-reasoning techniques to reason about problem instances and innovatively propose the equivalence problem of the location-planning problem. The model reasones the equivalence problem of multiple problem instances in parallel, which allows the model to approach the problem from multiple angles and improve the quality of the solution.

[0065] Preferably, in step S2, during the training of the intelligent logistics distribution planning model, the parameters of the intelligent logistics distribution planning model are updated using the REINFORCE reinforcement learning algorithm, and the calculation formula is as follows:

[0066]

[0067]

[0068] Among them, according to the chain rule, p θ (π|s) is the cumulative product of the probabilities of choosing each action, then ln p θ (π|s) is calculated as the sum of the logarithms of the probability of choosing each action step. The gradient value can be obtained by calculating the partial derivative of this value with respect to the parameter θ. (L(π)-b(s)) determines the direction of gradient descent, and b(s) represents the baseline performance of the model. If the current model performs better than the baseline performance, the model is positively stimulated; otherwise, the model is negatively constrained.

[0069] Here, the REINFORCE training method is used to overcome the limitation of supervised learning, which lacks optimal solutions and is difficult to train on the location-path planning problem, thereby improving the performance of the model after training.

[0070] The POMO method is used to estimate b(s), and the process is as follows:

[0071] Specify M different distribution centers As the head node of the solution sequence, M solution sequences {τ} are sampled during training. 1 ,τ 2 ,…,τ 3 Each trajectory is defined as a sequence τ. i :

[0072]

[0073] Using b(s) as the average of the total cost obtained from solutions with all different starting points, and taking the average performance of a set of heterogeneous trajectories as a benchmark:

[0074]

[0075] Where R(·) represents the total cost corresponding to the solution sequence.

[0076] Here, POMO is used as the benchmark for the REINFORCE algorithm. The symmetry of the solution sequence of the location-path planning problem is utilized, and prior knowledge is used as a guide to make the training convergence faster and more stable, while avoiding local minima.

[0077] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0078] This invention proposes an intelligent logistics planning method based on deep reinforcement learning. First, historical vehicle delivery information is collected and a fully connected graph is constructed. Then, a sequence dataset is built using the node set of the fully connected graph. Next, an intelligent logistics delivery planning model is constructed and trained using the sequence dataset. Finally, the node sequence of the vehicle delivery information to be planned is input into the trained intelligent logistics delivery planning model, and the solution sequence output by the model is used as the planning result. The actual logistics planning system is abstracted into a location-route planning problem, and deep reinforcement learning is used to solve it. This method enables the acquisition of feasible and optimal solutions to the location-route planning problem in real time and effectively improves the solution quality. Attached Figure Description

[0079] Figure 1 This is a schematic diagram illustrating the process of the intelligent logistics planning method based on deep reinforcement learning proposed in Embodiment 1 of the present invention.

[0080] Figure 2 This is a schematic diagram of the intelligent logistics planning model structure proposed in Embodiment 2 of the present invention;

[0081] Figure 3 This is a schematic diagram of the encoder structure proposed in Embodiment 2 of the present invention. Detailed Implementation

[0082] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0083] To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions;

[0084] It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0085] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0086] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0087] Example 1

[0088] like Figure 1 As shown, this embodiment proposes an intelligent logistics planning method based on deep reinforcement learning, including:

[0089] S1. Collect historical vehicle delivery information and construct a fully connected graph, then use the node set of the fully connected graph to construct a sequence dataset;

[0090] Define the fully connected graph as g(X), where X is the set of nodes consisting of distribution center nodes and customer nodes:

[0091] X={x i =(s i ,d i ,c i ,o i ,P)|i∈1,…,D,…,n}

[0092] Where, x i Let be the original characteristics of node i; D be the number of distribution centers; n be the total number of nodes including both distribution center nodes and customer nodes; s i Let d be the two-dimensional coordinates of node i; i This represents the demand for a customer node; if the node is a distribution center node, this value is set to 0. i and o i These represent the capacity and setup cost of the distribution center, respectively; if the node is a customer node, both are set to 0; P is the vehicle capacity.

[0093] S2. Construct an intelligent logistics distribution planning model, and train the intelligent logistics distribution planning model using a sequence dataset to obtain a trained intelligent logistics distribution planning model;

[0094] In this embodiment, as Figure 2 As shown, the intelligent logistics distribution planning model includes an encoder and a decoder;

[0095] The encoder performs an attention mechanism operation on sequence X to obtain a node embedding vector;

[0096] The decoder performs autoregression based on the current state information of the decoder and the embedding vectors of each node to obtain the final solution sequence π = (π1, ..., π). t ,…,π n ), where π tThis indicates the node selected by the model at time step t;

[0097] Calculate the total cost of all solution sequences, and select the sequence with the lowest cost as the final solution output of the intelligent logistics distribution planning model.

[0098] S3. Input the node sequence of the vehicle delivery information to be planned into the trained intelligent logistics delivery planning model, and use the solution sequence output by the intelligent logistics delivery planning model as the planning result.

[0099] Example 2

[0100] In this embodiment, based on Embodiment 1, the main function of the encoder is to perform feature abstraction and information fusion on the input problem instance. For example... Figure 3 As shown, the encoder consists of an embedding layer and N layers of multi-head attention modules. In this embodiment, N=3.

[0101] The embedding layer takes the original features x of each input node i Mapping to node embedding features The mapping process is as follows:

[0102]

[0103]

[0104] Among them, W d and b d W represents the network parameters embedded in the distribution center node. c and b c This represents the network parameters embedded in the client node;

[0105] The multi-head attention module includes a multi-head attention sub-layer, a feedforward layer, and a normalization layer. In this embodiment, to avoid gradient vanishing, residual connections are added to the multi-head attention sub-layer, and residual connections are also added to the feedforward layer.

[0106] The multi-head attention module embeds nodes into feature h. l-1 Updated to h l , to obtain the node embedding vector The update process is as follows:

[0107]

[0108]

[0109] Where l represents the l-th multi-head attention module, l∈1,…N, MHA represents the multi-head attention sub-layer, FF represents the feedforward layer, and BatchNorm represents the batch normalization operation.

[0110] The role of multi-head attention sublayers in a network is to improve the model's ability to model complex relationships by simultaneously focusing on different parts of the input—that is, on information from different distribution centers and customer nodes—and thus more effectively capturing information and patterns in problem instances. In this embodiment, eight attention heads are used.

[0111] In practical implementation, the calculation process of the multi-head attention mechanism is as follows:

[0112]

[0113]

[0114]

[0115] Where m represents the calculation performed in the m-th attention head dimension. These are all network parameters; the calculated q m k m v m These are the query, key, and value obtained from the m-th attention head dimension, respectively.

[0116] During the application, the input problem instance is treated as a fully connected network, which includes self-connections, meaning that every node is adjacent to all other nodes.

[0117] q is calculated in each attention head dimension space. i,m With k j,m Compatibility between The compatibility This represents the relationship between node i and node j in the m-th attention head subspace, and uses the softmax operation to... Normalized to attention score

[0118] Attention score With the corresponding v i,m Perform a dot product operation to fuse the information from all nodes into the i-th node, thus obtaining the subspace feature h′ of each attention head. i,m Finally, all features in the attention head dimension space are fused into complete node features.

[0119] The calculation for each attention point is as follows:

[0120]

[0121]

[0122]

[0123]

[0124] in, Network parameters are used to fuse features represented in all attention head dimensions.

[0125] The feedforward layer consists of two linear feedforward layers, which are represented as follows:

[0126]

[0127] in, and These are the parameters of the first linear feedforward layer network. and These are the parameters of the second linear feedforward layer network; during application, the multi-head attention module does not change the dimension of node embedding.

[0128] In this embodiment, the decoder is based on the node embedding vector output by the encoder. Using the information from the already constructed partial solution sequence, the process of calculating the node selection at the next time step t includes:

[0129] SA. Set an indicator vector. Indicates the current state;

[0130] The indicator vector The expression is as follows:

[0131]

[0132] in, This represents the mean vector of all node embeddings output by the encoder. This represents the embedding vector of the distribution center node currently providing services along this path. ξ represents the embedding vector of the model output node in the previous time step (t-1). d ,ξ v ,ξ g g is a learnable parameter used for filling when no nodes are selected. t For a two-dimensional vector, the expression is as follows:

[0133] g t =(c d,t ,P t ) T

[0134] c d,t The remaining quantity of goods in the distribution center is indicated, and the update logic is as follows:

[0135] When the node selected at time (t-1) belongs to the distribution center node, the remaining quantity c of goods in the distribution center will be... d,t Update the inventory of the selected distribution center at time t; when the selected node at time (t-1) belongs to a customer node, update the remaining inventory of the distribution center c. d,t At time t, the update is the remaining quantity of goods at the distribution center currently being served minus the demand for goods at that customer node, thus satisfying the customer node's demand; the update logic is expressed by the following formula:

[0136]

[0137] in, This represents the set of distribution center nodes.

[0138] P t This indicates the remaining cargo quantity of vehicles currently in service, and the update logic is as follows:

[0139] When the node selected at time (t-1) belongs to a distribution center, it means that the vehicle has returned to the distribution center from which it departed, and the next vehicle is ready to depart at the distribution center node selected at time t. At this time, the remaining cargo quantity P of the vehicles currently in service is... t The vehicle load will be updated to the load when fully loaded. When the node selected at time (t-1) is a customer node, but the demand of that customer node exceeds the remaining load capacity of the vehicle currently in service (meaning the vehicle cannot meet the customer node's demand in one go), the vehicle will return to its originating distribution center. A new vehicle should be dispatched from the currently serving distribution center node to meet the node's demand. In this case, the remaining load capacity refers to the load capacity of the newly dispatched vehicle, which is the fully loaded load capacity P. Otherwise, when the selected node at time (t-1) is a customer node, and the current vehicle can meet the selected customer node's demand, the remaining load capacity at time t will be updated to the remaining load capacity at time (t-1) minus the customer node's demand. The formula for this update logic is as follows:

[0140]

[0141] SB. Calculation of the indicator vector through a single-head attention mechanism Update, With the embedding vector contained in all nodes The process of merging node information includes:

[0142] SB1.Use Generate a query using the embedding vectors of all nodes. Generate key and value using the following formula:

[0143]

[0144]

[0145]

[0146] in, W k and W v For network parameters;

[0147] SB2. Generate an n-dimensional vector and normalize it. The calculation formula is as follows:

[0148]

[0149] Where, Φ t It is the set of infeasible nodes;

[0150] SB3. Through an attention mechanism, the indicator vector is... For updates The updated formula is as follows:

[0151]

[0152] SC. Feasible and infeasible nodes are divided through masking operations, and the selection probability of feasible nodes is calculated through an attention mechanism; in this embodiment, an indicator vector is used. The feasible node selection probability is calculated from the embedding vectors of all nodes output by the encoder. The process includes:

[0153] use The query is updated by generating the key from the embeddings of the nodes produced by the encoder, using the following formula:

[0154]

[0155]

[0156] The probability of selecting a feasible node is calculated using the following formula:

[0157]

[0158]

[0159] Among them, u ci p represents the probability of selecting a feasible node from the current state to add to the partial solution. θ (π t =i|X,π1,…,π t-1() represents a random policy. Here, the selection probability is calculated only for feasible nodes, where the probability of infeasible nodes is chosen as -∞, and all u ci After calculation using softmax, the probability of infeasible nodes is set to 0.

[0160] In practical applications, there are two ways to select nodes based on probability: one is a greedy approach, which selects the node with the highest probability and adds it to the partial solution; the other is a sampling approach, which uses p... θ (π t =i|X,π1,…,π t-1 The nodes are selected using a probability distribution. For example, the final generated p... θ Given a probability [0.6, 0.1, 0.1, 0, 0, 0.2], node 1 has a probability of 0.6 when selected for the partial solution, node 2 has a probability of 0.1, and so on. This method does not simply select the node with the highest probability for the partial solution; rather, it involves some exploration, probabilistically exploring other solutions. In this embodiment, a sampling method is used for training.

[0161] In this embodiment, infeasible nodes are excluded from the current node selection range using a masking operation. The division of infeasible nodes follows these rules:

[0162] When decoding at time step t=1, all customer nodes are infeasible, that is, all customer nodes are masked to indicate that the distribution center node must be the head node of the output solution sequence;

[0163] The nodes in the partial solution are infeasible, that is, the nodes selected by all decoding steps before the current decoding step t are masked, indicating that each node can only be visited once.

[0164] Masking is applied to all customer nodes whose demand exceeds the current remaining capacity of the distribution center, thus avoiding infeasible solutions due to insufficient remaining goods in the distribution center.

[0165] SD. Select the node for the next time step based on the obtained feasible node selection probability and store it in the solution sequence until a complete solution sequence is constructed.

[0166] In this embodiment, the intelligent logistics distribution planning model utilizes multiple reasoning techniques to perform reasoning, and outputs the sequence with the lowest cost obtained through reasoning as the final solution. The process includes:

[0167] Given m distinct distribution centers as starting points, generate h solution sequences from each starting point using a sampling method;

[0168] By performing p rotations on all nodes in g(X) at different angles, p×m×h solution sequences are obtained. In this embodiment, rotation angles of 30°, 60°, 90°, 120°, and 150° yield a total of 6×m×h solution sequences. Regardless of the rotation, the optimal solution sequence remains the same. Rotating all nodes of the problem instance is equivalent to performing an equivalent transformation on the problem, presenting different perspectives of the problem to the model.

[0169] Calculate the total cost of all solution sequences, and select the sequence with the minimum cost as the final solution output. Assume that in the feasible solution, d warehouses are opened, v vehicles are dispatched, and the distance of the planned path is r. The formula for calculating the total cost z is as follows:

[0170] z = d × cost d +v×cost v +r×cost r

[0171] Among them, cost d Cost represents the cost of setting up a warehouse. v Cost represents the cost of dispatching one vehicle. r This represents the transportation cost per unit path distance.

[0172] Example 3

[0173] In this embodiment, based on the above embodiments, when training the intelligent logistics distribution planning model, the REINFORCE reinforcement learning algorithm is used to update the parameters of the intelligent logistics distribution planning model. The calculation formula is as follows:

[0174]

[0175]

[0176] Among them, according to the chain rule, p θ (π|s) is the cumulative product of the probabilities of choosing each action, then ln p θ (π|s) is calculated as the sum of the logarithms of the probability of choosing each action step. The gradient value can be obtained by calculating the partial derivative of this value with respect to the parameter θ. (L(π)-b(s)) determines the direction of gradient descent, and b(s) represents the baseline performance of the model. If the current model performs better than the baseline performance, the model is positively stimulated; otherwise, the model is negatively constrained.

[0177] Here, the REINFORCE training method is used to overcome the limitation of supervised learning, which lacks optimal solutions and is difficult to train on the location-path planning problem, thereby improving the performance of the model after training.

[0178] The POMO method is used to estimate b(s), and the process is as follows:

[0179] Specify M different distribution centers As the head node of the solution sequence, M solution sequences {τ} are sampled during training. 1 ,τ 2 ,…,τ 3 Each trajectory is defined as a sequence τ. i :

[0180]

[0181] Using b(s) as the average of the total cost obtained from solutions with all different starting points, and taking the average performance of a set of heterogeneous trajectories as a benchmark:

[0182]

[0183] Where R(·) represents the total cost corresponding to the solution sequence.

[0184] Here, POMO is used as the benchmark for the REINFORCE algorithm. The symmetry of the solution sequence of the location-path planning problem is utilized, and prior knowledge is used as a guide to make the training convergence faster and more stable, while avoiding local minima.

[0185] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An intelligent logistics planning method based on deep reinforcement learning, characterized in that, include: S1. Collect historical vehicle delivery information and construct a fully connected graph, then use the node set of the fully connected graph to construct a sequence dataset; S2. Construct an intelligent logistics distribution planning model, and train the intelligent logistics distribution planning model using a sequence dataset to obtain a trained intelligent logistics distribution planning model; S3. Input the node sequence of the vehicle delivery information to be planned into the trained intelligent logistics delivery planning model, and use the solution sequence output by the intelligent logistics delivery planning model as the planning result; In step S1, the complete connection graph is defined as , a set of nodes consisting of distribution center nodes and customer nodes: in, For nodes The original characteristics; The number of distribution centers; This represents the total number of all nodes, including distribution center nodes and customer nodes. For nodes Two-dimensional coordinates; This represents the demand for the customer node; if the node is a distribution center node, this value is set to 0. and These represent the capacity and setup cost of the distribution center, respectively. If the node is a customer node, both are set to 0. It refers to the vehicle's capacity; The intelligent logistics distribution planning model includes an encoder and a decoder; The encoder performs an attention mechanism operation on the sequence to obtain a node embedding vector. The decoder performs autoregressive solving based on the current state information of the decoder and the embedding vectors of each node to obtain the final solution sequence. ,in, Indicates at time step The node selected by the model at that time; Calculate the total cost of all solution sequences and select the sequence with the minimum cost as the final solution output of the intelligent logistics distribution planning model. The decoder computes the next time step t selection node from the node embedding vector output by the encoder and information about the partially unrolled sequence that has been built so far. SA. Set an indicator vector representing the current state; SB. Calculation of the indicator vector through a single-head attention mechanism. Update, With the embedding vector contained in all nodes The node information in the data is integrated; SC. It divides feasible nodes into feasible and infeasible nodes through masking operations and calculates the selection probability of feasible nodes through an attention mechanism; SD. Select the node for the next time step based on the obtained feasible node selection probability and store it in the solution sequence until a complete solution sequence is constructed; In step SA, the indicator vector The expression of the indicator vector is as follows: in, This represents the mean vector of all node embeddings output by the encoder. This represents the embedding vector of the distribution center node currently providing services along this path. Indicates the previous time step The model outputs the embedding vectors of the nodes. These are the learnable parameters used for population when no nodes are selected. For a two-dimensional vector, the expression is as follows: Indicates the remaining quantity of goods at the distribution center. This indicates the remaining cargo quantity of vehicles currently in service. and The update logic is as follows: wherein, represents a set of distribution center nodes. 2.The intelligent logistics planning method based on deep reinforcement learning according to claim 1, wherein, The encoder consists of an embedding layer and N layers of multi-head attention modules; The embedding layer will incorporate the original features of each input node. Mapping to node embedding features The mapping process is as follows: wherein, and denote network parameters that embed the nodes; The multi-head attention module comprises a multi-head attention sublayer, a feedforward layer and a normalization layer; the multi-head attention module updates the node embedding feature is updated to , to obtain a node embedding vector The updating process is as follows: in, Indicates the first Multi-head attention module MHA stands for Multi-head Attention Sublayer, FF stands for Feedforward Layer, and BatchNorm stands for Batch Normalization Operation. 3.The intelligent logistics planning method based on deep reinforcement learning according to claim 1, characterized in that, Step SB Includes the following processes: SB1. Utilize Generate query, utilize embedding vectors of all nodes Generate key and value, formula as follows: wherein , and are network parameters; SB2. Generate a multidimensional vector and normalize it. The calculation formula is as follows: wherein, is a set of infeasible nodes; SB3. The indication vector is passed through an attention mechanism To update The update formula is as follows: 。 4.The intelligent logistics planning method based on deep reinforcement learning according to claim 3, characterized in that, Utilizing indicator vectors and the embedding vectors of all nodes of the encoder output to compute feasible node selection probabilities, the process comprising: Utilizing The update query, the embedding of the node generated by the encoder generates the key, the formula is as follows: The probability of selecting a feasible node is calculated using the following formula: in, This indicates the probability of selecting a feasible node from the current state and adding it to the partial solution. This represents a random policy. 5.The intelligent logistics planning method based on deep reinforcement learning according to claim 1, wherein, The intelligent logistics distribution planning model utilizes multi-inference technology for reasoning, and outputs the sequence with the lowest cost obtained through reasoning as the final solution. The process includes: Designated to h different distribution centers as starting points, and h solution sequences are generated from each starting point in a sampling manner; All nodes in are rotated by different angles, and deinterleaving sequences are obtained. Calculate the total cost of all solution sequences and select the sequence with the lowest cost as the final solution output.

Images