Supply chain distribution path planning method based on reinforcement learning

Abstract

The invention discloses a supply chain distribution path planning method based on reinforcement learning, and the method comprises the steps: constructing a model based on an encoder and a decoder, converting the features extracted from the historical distribution data of a deliveryman into a high-dimensional feature vector through a Transform encoder, and transmitting the high-dimensional feature vector to the decoder; a VAE-based feature reconstructor performs feature reconstruction on the feature vectors to obtain reconstructed features, and a decoder processes the reconstructed features to obtain a material delivery path; optimizing the pre-trained model based on the encoder and the decoder by using a near-end strategy optimization algorithm to obtain a target path planning model; and performing supply chain material distribution path planning by using the target path planning model. According to the invention, a variational auto-encoder and dynamic context embedding are introduced, so that supply chain material delivery path planning which is economical and efficient and adapts to a complex dynamic environment and individual delivery preferences of deliverymen is realized; and a near-end strategy optimization algorithm is used, so that the decision-making quality and stability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of path planning technology, and in particular to a supply chain distribution path planning method, system, terminal and computer-readable storage medium based on reinforcement learning. Background Art

[0002] The vehicle routing problem (VRP) is defined as a mathematical model that seeks the best path to minimize cost or time. It is an NP (Non-deterministic Polynomial) hard problem. Many VRP variants have been proposed to meet different needs. In the manufacturing industry, the material distribution problem can be regarded as a specific example of the VRP problem. In the supply chain material distribution scenario, materials need to be distributed from warehouses or supply points to multiple receiving points of the production line. Reasonable planning of distribution routes is the key to improving production efficiency and reducing costs for enterprises. Traditional VRP models generally reduce costs and improve efficiency by optimizing material distribution routes.

[0003] However, when faced with the dynamic changes in the production site and the individual delivery preferences of couriers, the traditional VRP model often appears to be inflexible and difficult to adapt to the complex dynamic environment or the individual delivery preferences of couriers, resulting in its limited performance in dealing with uncertainties in practical applications.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a supply chain distribution path planning method, system, terminal and computer-readable storage medium based on reinforcement learning, aiming to solve the problem that the traditional VRP model is difficult to adapt to the complex dynamic environment in the supply chain material distribution scenario, and cannot adapt to the individual distribution preferences of the delivery personnel, resulting in limited performance.

[0006] To achieve the above-mentioned purpose of the invention, the present invention provides a supply chain distribution path planning method based on reinforcement learning, and the supply chain distribution path planning method based on reinforcement learning includes:

[0007] Obtaining historical delivery data of the delivery person, and extracting features of the historical delivery data;

[0008] Constructing a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into a high-dimensional feature vector, a feature reconstructor based on a VAE (Variational Auto-Encoder) for reconstructing the feature vector to obtain a reconstructed feature, and a decoder for processing the reconstructed feature to obtain a material distribution path;

[0009] Pre-training the encoder-decoder based model, using a grid search algorithm to find optimal model parameters, and obtaining a pre-trained model under the optimal model parameters;

[0010] Using a proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model;

[0011] The unfinished task data of the delivery personnel are obtained, and the unfinished task data are input into the target path planning model, and the target path planning model outputs the target material delivery path.

[0012] Optionally, the acquiring of historical delivery data of the delivery person and extracting features of the historical delivery data specifically includes:

[0013] Acquire historical delivery data of the delivery personnel, pre-process the historical delivery data, and extract features of the historical delivery data, wherein the historical delivery data includes the completed task data and historical material delivery paths of the delivery personnel, and the features include the delivery distance and delivery time corresponding to the completed tasks in the completed task data;

[0014] Define the completed task data of deliveryman w as M w ={m1,m2,m3,...,m j ,...,m N}, where N represents the number of completed tasks, m N Indicates the Nth completed task, each completed task m j Including delivery distance and delivery time;

[0015] Define the historical material delivery path of the delivery person w in, Represents the jth completed task m in the historical material distribution path j The delivery location, Indicates the Nth completed task m in the historical material distribution path N The delivery location.

[0016] Optionally, the construction is based on an encoder and a decoder model, the encoder and the decoder model comprising a Transformer encoder for converting the feature into a high-dimensional feature vector, a VAE-based feature reconstructor for reconstructing the feature vector to obtain a reconstructed feature, and a decoder for processing the reconstructed feature to obtain a material distribution path, specifically comprising:

[0017] Constructing an encoder-decoder based model, wherein the encoder-decoder based model includes a Transformer encoder, a VAE-based feature reconstructor and a decoder, wherein the VAE-based feature reconstructor includes a variational autoencoder and a feature reconstructor;

[0018] The Transformer encoder is used to parse and encode the features through a multi-head self-attention mechanism, and the features are converted into a high-dimensional feature vector h;

[0019] The variational autoencoder is used to convert the feature vector into parameters of the latent space through a linear layer, wherein the parameters include the mean μ and the logarithmic variance log(σ 2 ):

[0020] μ=W μ h+b μ ;

[0021] log(σ 2 )=W σ h+b σ ;

[0022] Among them, W μ and W σ The weight matrices representing the mean and log variance, b μ and b σ denote the deviation vectors of mean and logarithmic variance respectively, and σ denotes the standard deviation;

[0023] The latent variable z is extracted from the distribution of the mean and the standard deviation σ obtained from the logarithmic variance by a sampling method:

[0024] z=μ+σ⊙ε;

[0025] Among them, ε represents the random noise vector, ⊙ represents the scaling;

[0026] The feature reconstructor is used to reconstruct the latent variables through a multi-layer fully connected network and batch normalization process to obtain a reconstructed feature

[0027]

[0028] Among them, VAEDecoder(·) represents feature reconstruction;

[0029] The decoder is adopted, dynamic context embedding is introduced, probability distribution is output according to the reconstructed features, a predicted action sequence is generated, and then the material distribution path is obtained.

[0030] Optionally, the feature reconstructor is used to reconstruct the latent variables through a multi-layer fully connected network and a batch normalization process to obtain a reconstructed feature In , only the reconstruction loss is considered, without considering the KL divergence loss:

[0031]

[0032] Among them, L recon represents the reconstruction loss, N represents the number of completed tasks, and h i represents the feature vector corresponding to the i-th task, Represents the reconstructed features corresponding to the i-th task.

[0033] Optionally, the decoder is adopted, dynamic context embedding is introduced, probability distribution is output according to the reconstructed features, a predicted action sequence is generated, and then a material distribution path is obtained, specifically including:

[0034] The decoder uses a long short-term memory network combined with an attention mechanism to calculate the probability distribution of the output of each decoding step;

[0035] Embed the global information of the input data into h' g , the information embedding of the generated node at the current time step The first time step generates the node information embedding and the average value of the information embedding of the nodes generated at all previous time steps The dynamic context embedding h' is obtained by summing:

[0036]

[0037] Dynamically update the hidden state using dynamic context embedding, combine the dynamic context embedding h' with the current hidden state h0 output by the LSTM network as the query input q of the attention mechanism t :

[0038]

[0039] Through the cyclic properties of the long short-term memory network and the attention mechanism, the prediction output of the entire task sequence is gradually established, the predicted action sequence is generated, and the material distribution path is obtained.

[0040] Optionally, the encoder-decoder-based model is pre-trained, and a grid search algorithm is used to find optimal model parameters to obtain a pre-trained model under the optimal model parameters, and then the method further includes:

[0041] The material distribution path planning problem is described as a Markov decision process M:

[0042] M=(S,A,P,R,γ,s0,T);

[0043] Among them, S represents the state set, which is a collection of unfinished tasks and completed tasks; A represents the action space, which refers to the set of actions that can be performed in each state; P represents the state transition probability, which refers to the probability of transitioning from state s to state s' after executing action a; R represents the reward function; γ is the discount factor, γ∈[0,1], which is used to balance the relationship between current rewards and future rewards; s0 represents the initial state distribution; T represents the number of time steps.

[0044] Optionally, the using a proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model specifically includes:

[0045] Loading the weights of the pre-trained model, and iteratively optimizing the pre-trained model using a proximal strategy optimization algorithm;

[0046] Generate actions through the current strategy and calculate the transition probability ratio between the new strategy and the old strategy;

[0047] Using the transition probability ratio and the advantage value obtained from the environment feedback, the strategy loss is calculated:

[0048]

[0049] Among them, L a represents the policy loss; t represents the time step; A t represents the advantage function; r t (θ) represents the transition probability ratio of the new strategy θ' to the old strategy θ, P θ' (a t |s t ) indicates that under the new strategy θ', in state s t Execute action a t The probability of transferring to the next state when θ (a t |s t ) means that under the old strategy θ, in state s t Execute action a t The probability of transitioning to the next state when represents the corrected strategy ratio, the clip operation is introduced through maximum minimization to limit the step of strategy update, ε' is a constant;

[0050] By using state value prediction and target return, the value function loss is calculated:

[0051]

[0052] Among them, L c represents the value function loss, V'(s i ) indicates state s i The corresponding state value prediction, V(s i ) indicates state s i The corresponding state value, R i Indicates state s i The corresponding target return; clip(·) represents the clipping operation, which is used to limit V'(s i ) relative to V(s i ) changes;

[0053] Calculate the entropy loss using the probability output of the policy network:

[0054]

[0055] Among them, L e represents entropy loss, π(a j |s i ) means that in a given state s i Next select action a j The probability output of the policy network;

[0056] The policy loss, the value function loss and the entropy loss are weighted and summed to obtain the total loss:

[0057] L rl =θ a L a +θ c L c +θ e L e ;

[0058] Among them, L rl is the total loss of reinforcement learning, θ a ,θ c and θ e All are variable parameters;

[0059] Performing back propagation to update the pre-trained model according to the total loss, and saving the current best model according to the early stopping condition and the performance improvement condition;

[0060] The current best model is tested, and the current best model that meets the test requirements is used as the target path planning model.

[0061] To achieve the above-mentioned purpose of the invention, the present invention further provides a supply chain distribution path planning system based on reinforcement learning, and the supply chain distribution path planning system based on reinforcement learning includes:

[0062] Data acquisition module: used to obtain the historical delivery data of the delivery personnel and extract the features of the historical delivery data;

[0063] Model construction module: used to construct a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material distribution path;

[0064] Model pre-training module: used to pre-train the model based on the encoder and decoder, use the grid search algorithm to find the optimal model parameters, and obtain the pre-trained model under the optimal model parameters;

[0065] Model optimization training module: used to optimize the pre-trained model using a proximal strategy optimization algorithm to obtain a target path planning model;

[0066] Material distribution path planning module: used to obtain the unfinished task data of the delivery personnel, input the unfinished task data into the target path planning model, and the target path planning model outputs the target material distribution path.

[0067] In order to achieve the above-mentioned purpose of the invention, the present invention also provides a terminal, which includes: a memory, a processor, and a supply chain distribution path planning program based on reinforcement learning stored in the memory and executable on the processor, and the supply chain distribution path planning program based on reinforcement learning, when executed by the processor, implements the steps of the supply chain distribution path planning method based on reinforcement learning as described above.

[0068] In order to achieve the above-mentioned purpose of the invention, the present invention also provides a computer-readable storage medium, which stores a supply chain distribution path planning program based on reinforcement learning. When the supply chain distribution path planning program based on reinforcement learning is executed by a processor, the steps of the supply chain distribution path planning method based on reinforcement learning as described above are implemented.

[0069] In the present invention, historical delivery data of delivery personnel are obtained, and features of the historical delivery data are extracted; a model based on an encoder and a decoder is constructed, wherein the model based on the encoder and the decoder comprises a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material delivery path; the model based on the encoder and the decoder is pre-trained, and the best model parameters are found using a grid search algorithm to obtain a pre-trained model under the best model parameters; the pre-trained model is optimized and trained using a proximal strategy optimization algorithm to obtain a target path planning model; the unfinished task data of the delivery personnel is obtained, and the unfinished task data is input into the target path planning model, and the target path planning model outputs a target material delivery path. The present invention combines Transformer encoder, variational autoencoder and feature reconstructor to extract features reflecting the individual delivery preferences of the delivery personnel from the historical delivery data of the delivery personnel, introduces dynamic context embedding to dynamically update the focus of the decoder, and meticulously reflects the individual delivery preferences of the delivery personnel. At the same time, it improves the adaptability and decision-making quality of the model when facing dynamically changing data, so that the model can adapt to complex dynamic environments and the individual delivery preferences of the delivery personnel, and performs superiorly in dealing with uncertainties in practical applications; uses the proximal strategy optimization algorithm to optimize the model, effectively improving the decision-making quality and stability of the model; and realizes economical and efficient supply chain material distribution path planning that adapts to complex dynamic environments and the individual delivery preferences of the delivery personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of a preferred embodiment of the supply chain distribution path planning method based on reinforcement learning of the present invention;

[0071] Figure 2 is another flow chart of a preferred embodiment of the supply chain distribution path planning method based on reinforcement learning of the present invention;

[0072] Figure 3 It is a structural diagram of a preferred embodiment of the supply chain distribution path planning system based on reinforcement learning of the present invention;

[0073] Figure 4 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0075] With the development of information technology and big data analysis, more and more manufacturing companies and supply chain participants have begun to realize the importance of digitalization. Digitalization can improve the transparency and efficiency of the supply chain, enhance the responsiveness and flexibility of the supply chain. Through digitalization, companies can accurately predict market demand, optimize inventory management, reduce costs, and monitor production and distribution in real time to quickly adapt to market changes. In addition, digitalization has also promoted collaboration and resource integration within the supply chain, enhancing the competitiveness of the entire industry. Therefore, building a digital supply chain is a key strategy to improve corporate efficiency and market adaptability.

[0076] The vehicle routing problem (VRP) is defined as a mathematical model that seeks the optimal path to minimize cost or time. It is an NP-hard problem and also a core problem in the logistics and distribution industry. Subsequently, domestic and foreign scholars have proposed many VRP variants to meet different needs. In the manufacturing industry, the material distribution problem can be regarded as a specific instance of the VRP problem. In the specific supply chain material distribution scenario, materials need to be distributed from warehouses or supply points to multiple receiving points of the production line. Reasonable planning of distribution routes is the key to improving production efficiency and reducing costs for enterprises. The traditional VRP problem reduces costs and improves efficiency by optimizing distribution routes. However, when faced with real-time changes in the production site and the distribution preferences of individual delivery personnel, the traditional VRP model often appears to be inflexible. Therefore, the optimization of distribution routes has become an urgent need. It is crucial to choose more economical and efficient routes while adapting to the preferred routes of delivery personnel.

[0077] Although current path planning methods have achieved effective results to a certain extent, they still have many limitations. First, when processing massive historical data of delivery personnel, they cannot fully capture the key features and dynamic changes in the data. Second, traditional path generation methods often rely on predefined rules and algorithms, which are difficult to adapt to complex dynamic environments, resulting in limited performance when dealing with uncertainties in practical applications.

[0078] In order to solve the above technical problems, the present invention provides a supply chain distribution path planning method based on reinforcement learning, which obtains the historical distribution data of the delivery personnel and extracts the features of the historical distribution data; constructs a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material distribution path; pre-trains the model based on the encoder and the decoder, uses a grid search algorithm to find the optimal model parameters, and obtains a pre-trained model under the optimal model parameters; uses a proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model; obtains the unfinished task data of the delivery personnel, inputs the unfinished task data into the target path planning model, and the target path planning model outputs the target material distribution path. The present invention combines Transformer encoder, variational autoencoder and feature reconstructor to extract features reflecting the individual delivery preferences of the delivery personnel from the historical delivery data of the delivery personnel, introduces dynamic context embedding to dynamically update the focus of the decoder, and meticulously reflects the individual delivery preferences of the delivery personnel. At the same time, it improves the adaptability and decision-making quality of the model when facing dynamically changing data, so that the model can adapt to complex dynamic environments and the individual delivery preferences of the delivery personnel, and performs superiorly in dealing with uncertainties in practical applications; uses the proximal strategy optimization algorithm to optimize the model, effectively improving the decision-making quality and stability of the model; and realizes economical and efficient supply chain material distribution path planning that adapts to complex dynamic environments and the individual delivery preferences of the delivery personnel.

[0079] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0080] A preferred embodiment of the supply chain distribution path planning method based on reinforcement learning of the present invention is as follows: Figure 1 and Figure 2 As shown, specifically including:

[0081] S1. Obtain the historical delivery data of the delivery person and extract the features of the historical delivery data.

[0082] In an implementation of this embodiment, the acquiring of the historical delivery data of the delivery person and the extraction of features of the historical delivery data specifically include:

[0083] Acquire historical delivery data of the delivery personnel, pre-process the historical delivery data, and extract features of the historical delivery data, wherein the historical delivery data includes the completed task data and historical material delivery paths of the delivery personnel, and the features include the delivery distance and delivery time corresponding to the completed tasks in the completed task data;

[0084] Define the completed task data of deliveryman w as M w ={m1,m2,m3,...,m j ,...,m N}, where N represents the number of completed tasks, m N Indicates the Nth completed task, each completed task m j Including delivery distance and delivery time;

[0085] Define the historical material delivery path of the delivery person w in, Represents the jth completed task m in the historical material distribution path j The delivery location, Indicates the Mth completed task m in the historical material distribution path N The delivery location.

[0086] Specifically, the present invention collects historical delivery data of the delivery personnel as training data, and analyzes the individual delivery preferences of the delivery personnel through the extensive historical data of the delivery personnel, so that the generated material delivery path adapts to the individual delivery preferences of the delivery personnel.

[0087] S2. Construct a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material delivery path.

[0088] In an implementation of this embodiment, the model based on an encoder and a decoder is constructed, and the model based on the encoder and the decoder includes a Transformer encoder for converting the feature into a high-dimensional feature vector, a VAE-based feature reconstructor for reconstructing the feature vector to obtain a reconstructed feature, and a decoder for processing the reconstructed feature to obtain a material distribution path, specifically including:

[0089] Constructing an encoder-decoder based model, wherein the encoder-decoder based model includes a Transformer encoder, a VAE-based feature reconstructor and a decoder, wherein the VAE-based feature reconstructor includes a variational autoencoder and a feature reconstructor;

[0090] The Transformer encoder is used to parse and encode the features through a multi-head self-attention mechanism, and the features are converted into a high-dimensional feature vector h;

[0091] The variational autoencoder is used to convert the feature vector into parameters of the latent space through a linear layer, wherein the parameters include the mean μ and the logarithmic variance log(σ 2 ):

[0092] μ=W μ h+b μ ;

[0093] log(σ 2 )=W σ h+b σ ;

[0094] Among them, W μ and W σ The weight matrices representing the mean and log variance, b μ and b σ denote the deviation vectors of mean and logarithmic variance respectively, and σ denotes the standard deviation;

[0095] The latent variable z is extracted from the distribution of the mean and the standard deviation σ obtained from the logarithmic variance by a sampling method:

[0096] z=μ+σ⊙ε;

[0097] Among them, ε represents a random noise vector, which is assumed to come from a standard normal distribution; ⊙ represents scaling, and σ⊙ε represents using the standard deviation σ to scale the random noise vector ε drawn from the standard normal distribution;

[0098] The feature reconstructor is used to reconstruct the latent variables through a multi-layer fully connected network and batch normalization process to obtain a reconstructed feature

[0099]

[0100] Among them, VAEDecoder(·) represents feature reconstruction;

[0101] The decoder is adopted, dynamic context embedding is introduced, probability distribution is output according to the reconstructed features, a predicted action sequence is generated, and then the material distribution path is obtained.

[0102] Specifically, the present invention proposes a special variational deep reinforcement learning framework, which is based on an encoder-decoder structure. The Transformer encoder converts the spatiotemporal information of the delivery task into a comprehensive feature representation (i.e., a high-dimensional context-sensitive feature representation, a high-dimensional feature vector); then, this comprehensive feature representation is further converted into two key parameters of the latent space (i.e., mean and logarithmic variance) through two linear layers of the variational autoencoder, and the output of the Transformer encoder is mapped to the latent space; then, latent variables are extracted from the distribution of the key parameters, and the latent variables are sent to a feature reconstructor, and the feature data is reconstructed through a multi-layer fully connected network and a batch normalization process to obtain reconstructed features; the decoder uses the reconstructed features to gradually form a predicted delivery route, thereby learning from complex input data and generating a delivery route that adapts to the individual delivery preferences of the delivery personnel.

[0103] In one implementation of this embodiment, the feature reconstructor is used to reconstruct the latent variables through a multi-layer fully connected network and a batch normalization process to obtain a reconstructed feature In , only the reconstruction loss is considered, without considering the KL divergence loss:

[0104]

[0105] Among them, L recon represents the reconstruction loss, N represents the number of completed tasks, and h i represents the feature vector corresponding to the i-th task, Represents the reconstructed features corresponding to the i-th task.

[0106] Specifically, the traditional VAE model requires the KL divergence loss to force the distribution of the latent space to approach the standard normal distribution. Since the delivery service needs to adapt to the personal delivery preferences of the delivery personnel, if the latent space is subject to overly strict statistical constraints, the generated routes will be too standardized and lack adaptability in specific environments. Therefore, the present invention does not consider adding KL divergence loss during the training process to maintain the diversity and adaptability of the generated data.

[0107] In an implementation of this embodiment, the decoder is adopted, dynamic context embedding is introduced, probability distribution is output according to the reconstructed features, a predicted action sequence is generated, and then a material distribution path is obtained, which specifically includes:

[0108] The decoder uses a long short-term memory network combined with an attention mechanism to calculate the probability distribution of the output of each decoding step;

[0109] Embed the global information of the input data into h' g , the information embedding of the generated node at the current time step The first time step generates the node information embedding and the average value of the information embedding of the nodes generated at all previous time steps The dynamic context embedding h' is obtained by summing:

[0110]

[0111] Dynamically update the hidden state using dynamic context embedding, combine the dynamic context embedding h' with the current hidden state h0 output by the LSTM network as the query input q of the attention mechanism t :

[0112]

[0113] Through the cyclic properties of the long short-term memory network and the attention mechanism, the prediction output of the entire task sequence is gradually established, the predicted action sequence is generated, and the material distribution path is obtained.

[0114] Specifically, the present invention introduces an effective dynamic context embedding, which embeds the global information of the entire input data, i.e., the aggregate, into h' g , the information embedding of the generated node at the current time step The first time step generates the node information embedding and the average value of the information embedding of the nodes generated at all previous time steps The enhanced embedding h' obtained by addition ensures that the model can fully consider the influence of the starting point, the average characteristics of historical path selection, and the current state information at each time step, dynamically integrate new inputs and existing sequence information, and thus comprehensively consider the contextual information of the entire sequence in each decision step.

[0115] At the same time, this enhanced dynamic context embedding h' is used to dynamically update the hidden state. Since the decoder uses LSTM (Long Short-Term Memory) combined with the attention mechanism to calculate the output probability distribution at each decoding step (i.e., each time step), the most likely next task is selected, where LSTM is responsible for taking the fusion of all previous task embedding vectors, the hidden state of the previous time step, and the input features of the current time step as input, and gradually establishes the predicted output of the entire task sequence through the attention mechanism and the cyclic nature of LSTM. Therefore, the present invention combines h' with the current hidden state h0 output by LSTM as the query input of the attention mechanism in the decoder, i.e. The model can dynamically update its attention focus through dynamic node information at each time step, which enhances the adaptability and decision-making quality of the model when facing dynamically changing data.

[0116] S3. Pre-train the model based on the encoder and decoder, use a grid search algorithm to find the optimal model parameters, and obtain a pre-trained model under the optimal model parameters.

[0117] In an implementation of this embodiment, the model based on the encoder and the decoder is pre-trained, and the optimal model parameters are found using a grid search algorithm to obtain a pre-trained model under the optimal model parameters, and then the following is further included:

[0118] The material distribution path planning problem is described as a Markov decision process M:

[0119] M=(S,A,P,R,γ,s0,T);

[0120] Among them, S represents the state set, which is a collection of unfinished tasks and completed tasks; A represents the action space, which refers to the set of actions that can be performed in each state; P represents the state transition probability, which refers to the probability of transitioning from state s to state s' after executing action a; R represents the reward function; γ is the discount factor, γ∈[0,1], which is used to balance the relationship between current rewards and future rewards; s0 represents the initial state distribution; T represents the number of time steps.

[0121] Specifically, in reinforcement learning, the agent learns how to make decisions in a specific environment by interacting with the environment. Unlike other machine learning methods, during the training process of reinforcement learning, the agent needs to receive feedback from the environment, which is usually expressed as a numerical value of reward or punishment. The entire reinforcement learning process is manifested as the agent interacting with the environment through a series of action strategies (Action) to generate a new state (State), and the environment gives a reward (Reward). After multiple iterations, a strategy is found, that is, a series of behavioral rules, which maximizes the long-term accumulated rewards. In the present invention, the entire encoder-decoder model (i.e., a model based on an encoder and a decoder) is regarded as an agent, and the prediction of the deliveryman's future material distribution route is regarded as a sequential decision-making process. This process is represented as a Markov decision process, in which:

[0122] State: At each decoding step, the state s t Includes the embedding representation H of the unfinished task and the hidden state h of the decoder t , path constraint C and the task route y output in the previous time step 1:t-1 :

[0123] s t =(H,h t ,C,y 1:t-1 );

[0124] Action: Action a t∈A t , defined as the tasks that can be selected given the current state, and each decoding step selects a task, A t is the state space at the tth time step, which changes as the decoding process progresses, and the state and mask are updated after the action selection;

[0125] Transition probability: Transition probability P(s t+1 |s t ,a t ):S×A, defined as in state s t Execute action a t When it moves to the next state s t+1 probability;

[0126] Reward: The reward R is set to the square of the position deviation:

[0127]

[0128] Where m represents the number of tasks in the route, i s is the position index of the current output, i l is the target output y t The reward is the index at the correct position of the sequence; if the last element of the sequence is in the label and the position matches, R0 is returned, if the last element of the sequence is not in the label and the index is out of the valid range, 0 is returned, otherwise the reward is the squared average of the index difference between the sequence and the corresponding position in the label.

[0129] S4. Use a proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model.

[0130] In an implementation of this embodiment, the use of a proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model specifically includes:

[0131] S41, loading the weight of the pre-trained model, and iteratively optimizing the pre-trained model using a proximal strategy optimization algorithm;

[0132] S42, generating actions through the current strategy, and calculating the transition probability ratio between the new strategy and the old strategy;

[0133] S43, using the transition probability ratio and the advantage value obtained from the environmental feedback, calculate the strategy loss:

[0134]

[0135] Among them, L a represents the policy loss; t represents the time step; A t represents the advantage function; r t (θ) represents the transition probability ratio of the new strategy θ' to the old strategy θ, P θ' (a t |s t ) indicates that under the new strategy θ', in state s t Execute action a t The probability of transferring to the next state when θ (a t |s t ) means that under the old strategy θ, in state s t Execute action a t The probability of transitioning to the next state when represents the revised policy ratio. The clipping operation clip is introduced through maximum minimization to limit the step of policy update. ε' is a constant. The difference between the new and old policies is calculated, and the clipping mechanism is applied to limit the size of the update to ensure the stability of the policy update.

[0136] S44. Calculate the value function loss through state value prediction and target return:

[0137]

[0138] Among them, L c represents the value function loss, V'(s i ) indicates state s i The corresponding state value prediction, V(s i ) indicates state s i The corresponding state value, R i Indicates state s i The corresponding target return; clip(·) represents the clipping operation, which is used to limit V'(s i ) relative to V(s i ) changes;

[0139] S45. Calculate the entropy loss through the probability output of the policy network:

[0140]

[0141] Among them, L e represents entropy loss, π(a j |s i ) means that in a given state s i Next select action a j The probability output of the policy network;

[0142] S46. Perform weighted summation of the strategy loss, the value function loss, and the entropy loss to obtain a total loss:

[0143] Lrl =θ a L a +θ c L c +θ e L e ;

[0144] Among them, L rl is the total loss of reinforcement learning, θ a ,θ c and θ e All are variable parameters;

[0145] S47, performing back propagation to update the pre-trained model according to the total loss, and saving the current best model according to the early stopping condition and the performance improvement condition;

[0146] S48: Testing the current best model, and using the current best model that meets the test requirements as the target path planning model.

[0147] Specifically, the present invention uses the actor-critic based PPO algorithm for training. The actor is an intelligent agent based on the encoder-decoder model, which is responsible for directly interacting with the environment and generating actions. The critic needs to access and understand the entire environmental state in order to effectively evaluate the actor's behavior and provide feedback, so the critic and the actor are set to share the same neural network and parameters. Since the expected reward is calculated using sampling, this method will result in a higher variance. Therefore, the present invention uses the generalized advantage estimation GAE to calculate the advantage value representing the relative "goodness" of the action in a certain state, and introduces the decay parameter λ to combine short-term and long-term reward estimates to balance the deviation and variance:

[0148]

[0149] in, represents the advantage value at time step t, λ is the decay parameter. When λ is close to 1, the advantage estimate will consider the longer-term future, thereby increasing the variance and reducing the bias; when λ is close to 0, the advantage estimate will rely more on the current estimate, thereby reducing the variance and increasing the bias; r' t+i represents the immediate reward at time step t+i, V(s t+i ) indicates state s t+i The state value, V(s t+i+1 ) indicates state s t+i+1 The state value of , T is the termination time step of the trajectory, and Tt represents the remaining time steps from the current time step t to the end of the trajectory.

[0150] The actor-critic based PPO algorithm includes Actor network and Critic network, where:

[0151] Actor network: handles the encoding and decoding parts and generates new actions, i.e. routes. In the encoder part, the Transformer encoder first uses the multi-head self-attention mechanism to extract the embedded representation (i.e. feature vector) of the delivery task, and then uses the VAE-based feature reconstructor to reconstruct the data. The reconstructed feature data is input into the decoder, which outputs the probability distribution and generates a predicted action sequence.

[0152] Critic network: evaluates the effect of the current strategy (i.e. action, i.e. route), inputs the action probability distribution output by the Actor network into the critic model to calculate the immediate reward of each action during training, and is used to directly feedback the evaluation of action results. It predicts the expected return of each state or state-action pair through the value function network, and determines which actions perform better than the average strategy and should be selected more frequently by the optimized strategy, that is, provides necessary feedback to guide the training of the Actor network.

[0153] The model is optimized and trained using a proximal policy optimization algorithm, including: loading the pre-trained model weights, and optimizing the model through a series of iterative cycles, with steps S42-S45 executed in each cycle. During the entire training process, all losses are weighted and summed to form the total loss of reinforcement learning. At the end of each cycle, the model is evaluated on the validation set, the current best model is saved based on the early stopping conditions and performance improvements, and finally tested on the test set. The present invention first performs pre-training, searches for the best model parameters through grid search, and then fine-tunes the pre-trained model using the PPO algorithm, and finally uses a variety of performance indicators on the test set to evaluate the model effect.

[0154] S5. Obtain the unfinished task data of the delivery personnel, input the unfinished task data into the target path planning model, and the target path planning model outputs the target material delivery path.

[0155] In an implementation of this embodiment, the obtaining of the unfinished task data of the delivery person, inputting the unfinished task data into the target path planning model, and the target path planning model outputting the target material delivery path specifically includes:

[0156] Define the set of unfinished tasks for the deliveryman U w ={u1,u2,u3,...,u n}, n represents the number of unfinished tasks, where each task u i It consists of a set of features, including the distance between the delivery person's current location and the task location, and the remaining committed time of the task.

[0157] Given a set of unfinished tasks U w, the decision of the predicted delivery person on the unfinished task (i.e., path planning) is defined as Since the deliveryman's task list is dynamically changing and new tasks are constantly added, the number of tasks in the label may be less than the number of tasks in the prediction. In addition, the original delivery order may be affected in this process. For example, using the observation data after time t+1 as the basis for model training may introduce misleading information because these data no longer accurately reflect the decision logic and action sequence at time t. Therefore, the present invention only includes those tasks completed within the original planned time when calculating the loss at a certain moment.

[0158] like Figure 2 As shown, in the present invention, based on a large amount of historical delivery data of the delivery personnel, data preprocessing is first performed to extract key features, such as longitude and latitude, deadline, etc.; an encoder with a multi-layer Transformer structure is used to parse and encode the geographic and time data of the input route. It effectively captures the complex relationships and long-distance dependencies between different locations through a multi-head self-attention mechanism, thereby generating a high-dimensional context-sensitive feature representation; a VAE-based feature reconstructor is used to reconstruct the feature representation, and the decoder focuses on using the reconstructed deep features to gradually generate the final route, and finally the PPO algorithm is used to fine-tune the model to optimize model performance.

[0159] In addition, based on the above-mentioned supply chain distribution path planning method based on reinforcement learning, the present invention also provides a supply chain distribution path planning system based on reinforcement learning, wherein a preferred embodiment of the supply chain distribution path planning system based on reinforcement learning is as follows: Figure 3 As shown, including:

[0160] Data acquisition module 01: used to obtain the historical delivery data of the delivery person and extract the features of the historical delivery data;

[0161] Model construction module 02: used to construct a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material distribution path;

[0162] Model pre-training module 03: used to pre-train the model based on the encoder and decoder, use a grid search algorithm to find the optimal model parameters, and obtain a pre-trained model under the optimal model parameters;

[0163] Model optimization training module 04: used to optimize the pre-trained model using a proximal strategy optimization algorithm to obtain a target path planning model;

[0164] Material delivery path planning module 05: used to obtain the unfinished task data of the delivery personnel, input the unfinished task data into the target path planning model, and the target path planning model outputs the target material delivery path.

[0165] In addition, based on the above-mentioned supply chain distribution path planning method and system based on reinforcement learning, the present invention also provides a terminal accordingly, wherein a preferred embodiment of the terminal is as follows: Figure 4 As shown, it specifically includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0166] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card) equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program codes of the storage terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a supply chain distribution path planning program 40 based on reinforcement learning is stored on the memory 20, and the supply chain distribution path planning program 40 based on reinforcement learning can be executed by the processor 10, thereby implementing the steps of the supply chain distribution path planning method based on reinforcement learning in this application.

[0167] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing a supply chain distribution path planning program 40 based on reinforcement learning.

[0168] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0169] In one embodiment, when the processor 10 executes the reinforcement learning-based supply chain distribution path planning program 40 in the memory 20, the steps of the reinforcement learning-based supply chain distribution path planning method as described above are implemented.

[0170] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a supply chain distribution path planning program based on reinforcement learning, and when the supply chain distribution path planning program based on reinforcement learning is executed by a processor, the steps of the supply chain distribution path planning method based on reinforcement learning are implemented as described above.

[0171] It should be noted that, in the present invention, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0172] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0173] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A supply chain distribution path planning method based on reinforcement learning, characterized in that: The supply chain distribution path planning method based on reinforcement learning includes: Obtaining historical delivery data of the delivery person, and extracting features of the historical delivery data; Constructing a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material distribution path; Pre-training the encoder-decoder based model, using a grid search algorithm to find optimal model parameters, and obtaining a pre-trained model under the optimal model parameters; Using a proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model; The unfinished task data of the delivery personnel are obtained, and the unfinished task data are input into the target path planning model, and the target path planning model outputs the target material delivery path.

2. The supply chain distribution path planning method based on reinforcement learning according to claim 1 is characterized in that: The obtaining of the historical delivery data of the delivery personnel and extracting the features of the historical delivery data specifically includes: Acquire historical delivery data of the delivery personnel, pre-process the historical delivery data, and extract features of the historical delivery data, wherein the historical delivery data includes the completed task data and historical material delivery paths of the delivery personnel, and the features include the delivery distance and delivery time corresponding to the completed tasks in the completed task data; Define the completed task data of deliveryman w as M w ={m1,m2,m3,...,m j ,...,m N }, where N represents the number of completed tasks, m N Indicates the Nth completed task, each completed task m j Including delivery distance and delivery time; Define the historical material delivery path of the delivery person w in, Represents the jth completed task m in the historical material distribution path j The delivery location, Indicates the Nth completed task m in the historical material distribution path N The delivery location.

3. The supply chain distribution path planning method based on reinforcement learning according to claim 1 is characterized in that: The model based on the encoder and decoder is constructed, and the model based on the encoder and decoder includes a Transformer encoder for converting the feature into a high-dimensional feature vector, a VAE-based feature reconstructor for reconstructing the feature vector to obtain a reconstructed feature, and a decoder for processing the reconstructed feature to obtain a material distribution path, specifically including: Constructing an encoder-decoder based model, wherein the encoder-decoder based model includes a Transformer encoder, a VAE-based feature reconstructor and a decoder, wherein the VAE-based feature reconstructor includes a variational autoencoder and a feature reconstructor; The Transformer encoder is used to parse and encode the features through a multi-head self-attention mechanism, and the features are converted into a high-dimensional feature vector h; The variational autoencoder is used to convert the feature vector into parameters of the latent space through a linear layer, wherein the parameters include the mean μ and the logarithmic variance log(σ 2 ): μ=W μ h+b μ ; log(σ 2 )=W σ h+b σ ; Among them, W μ and W σ The weight matrices representing the mean and log variance, b μ and b σ denote the deviation vectors of mean and logarithmic variance respectively, and σ denotes the standard deviation; The latent variable z is extracted from the distribution of the mean and the standard deviation σ obtained from the logarithmic variance by a sampling method: z=μ+σ⊙ε; Among them, ε represents the random noise vector, ⊙ represents the scaling; The feature reconstructor is used to reconstruct the latent variables through a multi-layer fully connected network and batch normalization process to obtain a reconstructed feature Among them, VAEDecoder(·) represents feature reconstruction; The decoder is adopted, dynamic context embedding is introduced, probability distribution is output according to the reconstructed features, a predicted action sequence is generated, and then the material distribution path is obtained.

4. The supply chain distribution path planning method based on reinforcement learning according to claim 3 is characterized in that: The feature reconstructor is used to reconstruct the latent variables through a multi-layer fully connected network and a batch normalization process to obtain a reconstructed feature In , only the reconstruction loss is considered, without considering the KL divergence loss: Among them, L recon represents the reconstruction loss, N represents the number of completed tasks, and h i represents the feature vector corresponding to the i-th task, Represents the reconstructed features corresponding to the i-th task.

5. The supply chain distribution path planning method based on reinforcement learning according to claim 3 is characterized in that: The decoder is adopted, dynamic context embedding is introduced, probability distribution is output according to the reconstructed features, a predicted action sequence is generated, and then a material distribution path is obtained, specifically including: The decoder uses a long short-term memory network combined with an attention mechanism to calculate the probability distribution of the output of each decoding step; Embed the global information of the input data into h' g , the information embedding of the generated node at the current time step The first time step generates the node information embedding and the average value of the information embedding of the nodes generated at all previous time steps The dynamic context embedding h' is obtained by summing: Dynamically update the hidden state using dynamic context embedding, combine the dynamic context embedding h' with the current hidden state h0 output by the LSTM network as the query input q of the attention mechanism t : Through the cyclic properties of the long short-term memory network and the attention mechanism, the prediction output of the entire task sequence is gradually established, the predicted action sequence is generated, and the material distribution path is obtained.

6. The supply chain distribution path planning method based on reinforcement learning according to claim 1 is characterized in that: The encoder-decoder based model is pre-trained, and the grid search algorithm is used to find the optimal model parameters to obtain the pre-trained model under the optimal model parameters, and then the following steps are further included: The material distribution path planning problem is described as a Markov decision process M: M=(S,A,P,R,γ,s0,T); Among them, S represents the state set, which is a collection of unfinished tasks and completed tasks; A represents the action space, which refers to the set of actions that can be performed in each state; P represents the state transition probability, which refers to the probability of transitioning from state s to state s' after executing action a; R represents the reward function; γ is the discount factor, γ∈[0,1], which is used to balance the relationship between current rewards and future rewards; s0 represents the initial state distribution; T represents the number of time steps.

7. The supply chain distribution path planning method based on reinforcement learning according to claim 6 is characterized in that: The use of the proximal strategy optimization algorithm to optimize the pre-trained model to obtain a target path planning model specifically includes: Loading the weights of the pre-trained model, and iteratively optimizing the pre-trained model using a proximal strategy optimization algorithm; Generate actions through the current strategy and calculate the transition probability ratio between the new strategy and the old strategy; Using the transition probability ratio and the advantage value obtained from the environment feedback, the strategy loss is calculated: Among them, L a represents the policy loss; t represents the time step; A t represents the advantage function; r t (θ) represents the transition probability ratio of the new strategy θ' to the old strategy θ, P θ' (a t |s t ) indicates that under the new strategy θ', in state s t Execute action a t The probability of transferring to the next state when θ (a t |s t ) means that under the old strategy θ, in state s t Execute action a t The probability of transitioning to the next state when represents the corrected strategy ratio, the clip operation is introduced through maximum minimization to limit the step of strategy update, ε' is a constant; By using state value prediction and target return, the value function loss is calculated: Among them, L c represents the value function loss, V'(s i ) indicates state s i The corresponding state value prediction, V(s i ) indicates state s i The corresponding state value, R i Indicates state s i The corresponding target return; clip(·) represents the clipping operation, which is used to limit V'(s i ) relative to V(s i ) changes; Calculate the entropy loss using the probability output of the policy network: Among them, L e represents entropy loss, π(a j |s i ) means that in a given state s i Next select action a j The probability output of the policy network; The policy loss, the value function loss and the entropy loss are weighted and summed to obtain the total loss: L rl =θ a L a +θ c L c +θ e L e ; Among them, L rl is the total loss of reinforcement learning, θ a ,θ c and θ e All are variable parameters; Performing back propagation to update the pre-trained model according to the total loss, and saving the current best model according to the early stopping condition and the performance improvement condition; The current best model is tested, and the current best model that meets the test requirements is used as the target path planning model.

8. A supply chain distribution path planning system based on reinforcement learning, characterized in that: The supply chain distribution path planning system based on reinforcement learning includes: Data acquisition module: used to obtain the historical delivery data of the delivery personnel and extract the features of the historical delivery data; Model construction module: used to construct a model based on an encoder and a decoder, wherein the model based on the encoder and the decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material distribution path; Model pre-training module: used to pre-train the model based on the encoder and decoder, use the grid search algorithm to find the optimal model parameters, and obtain the pre-trained model under the optimal model parameters; Model optimization training module: used to optimize the pre-trained model using a proximal strategy optimization algorithm to obtain a target path planning model; Material distribution path planning module: used to obtain the unfinished task data of the delivery personnel, input the unfinished task data into the target path planning model, and the target path planning model outputs the target material distribution path.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a supply chain distribution path planning program based on reinforcement learning stored in the memory and executable on the processor. When the supply chain distribution path planning program based on reinforcement learning is executed by the processor, the steps of the supply chain distribution path planning method based on reinforcement learning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a supply chain distribution path planning program based on reinforcement learning, and when the supply chain distribution path planning program based on reinforcement learning is executed by a processor, the steps of the supply chain distribution path planning method based on reinforcement learning as described in any one of claims 1-7 are implemented.

Images