A supply chain distribution path planning method based on reinforcement learning

Abstract

The application discloses a supply chain distribution path planning method based on reinforcement learning, and the method comprises the following steps: constructing a model based on an encoder and a decoder, wherein a Transformer encoder converts features extracted from historical distribution data of a distributor into a high-dimensional feature vector, a feature reconstructor based on VAE performs feature reconstruction on the feature vector to obtain reconstructed features, and a decoder processes the reconstructed features to obtain a material distribution path; using a proximal policy optimization algorithm to optimize the pre-trained model based on the encoder and the decoder to obtain a target path planning model; and using the target path planning model to plan a supply chain material distribution path. The application introduces a variational autoencoder and dynamic context embedding, realizes economic, efficient and adaptive supply chain material distribution path planning suitable for complex dynamic environments and individual distribution preferences of distributors, and uses a proximal policy optimization algorithm to improve the decision quality and stability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a supply chain distribution path planning method and system based on reinforcement learning, a terminal and a computer readable storage medium. BACKGROUND

[0002] Vehicle routing problem (VRP) is defined as a mathematical model seeking an optimal path to minimize cost or time, which is an NP (Non-deterministic Polynomial) hard problem. Many VRP variants are proposed for different needs. In manufacturing industry, material distribution problem can be regarded as a specific instance of VRP problem. In the scene of supply chain material distribution, materials need to be distributed from warehouses or supply points to multiple receiving points of production lines. Reasonable planning of distribution routes is the key for enterprises to improve production efficiency and reduce costs. Traditional VRP models generally optimize material distribution routes to reduce costs and improve efficiency.

[0003] However, when facing dynamic changes in production sites and individual distribution preferences of distributors, traditional VRP models often lack flexibility and are difficult to adapt to complex dynamic environments and individual distribution preferences of distributors, resulting in limited performance in dealing with uncertainties in actual applications.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a supply chain distribution path planning method and system based on reinforcement learning, a terminal and a computer readable storage medium, which aims to solve the problem that traditional VRP models are difficult to adapt to complex dynamic environments and individual distribution preferences of distributors in the scene of supply chain material distribution, resulting in limited performance.

[0006] To achieve the above-mentioned application purposes, the present application provides a supply chain distribution path planning method based on reinforcement learning, which comprises:

[0007] Obtaining historical distribution data of distributors and extracting features of the historical distribution data;

[0008] construct an encoder-decoder based model, the encoder-decoder based model comprising a Transformer encoder for converting the features into a high-dimensional feature vector, a VAE (Variational Auto-Encoder) based feature reconstructor for feature reconstruction of the feature vector to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material delivery path;

[0009] pre-train the encoder-decoder based model, find optimal model parameters using a grid search algorithm, and obtain a pre-trained model under the optimal model parameters;

[0010] optimize and train the pre-trained model using a proximal policy optimization algorithm to obtain a target path planning model;

[0011] obtain unfinished task data of a delivery person, input the unfinished task data into the target path planning model, and output a target material delivery path from the target path planning model.

[0012] Optionally, the historical delivery data of the delivery person is obtained, and features of the historical delivery data are extracted, specifically including:

[0013] obtain historical delivery data of a delivery person, pre-process the historical delivery data, and extract features of the historical delivery data, wherein the historical delivery data comprises completed task data and historical material delivery paths of the delivery person, and the features comprise delivery distances and delivery times corresponding to completed tasks in the completed task data;

[0014] define the completed task data of the delivery person w as M w ={m1,m2,m3,...,m j ,...,m N}, wherein N represents the number of completed tasks, m N represents the Nth completed task, and each completed task m j comprises a delivery distance and a delivery time;

[0015] define the historical material delivery path of the delivery person w as wherein represents a delivery location of the jth completed task m j in the historical material delivery path, represents a delivery location of the Nth completed task m N in the historical material delivery path.

[0016] Optionally, the constructing is based on an encoder-decoder based model, the encoder-decoder based model comprising a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE based feature reconstructor for feature reconstruction on the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain the material distribution path, and specifically comprising:

[0017] constructing an encoder-decoder based model, the encoder-decoder based model comprising a Transformer encoder, a VAE based feature reconstructor, and a decoder, the VAE based feature reconstructor comprising a variational autoencoder and a feature reconstructor;

[0018] adopting the Transformer encoder, analyzing and encoding the features through a multi-head self-attention mechanism to convert the features into high-dimensional feature vectors h;

[0019] adopting the variational autoencoder, converting the feature vectors into parameters of a latent space through a linear layer, the parameters comprising a mean μ and a log variance log(σ 2 ):

[0020] μ=W μ h+b μ ;

[0021] log(σ 2 )=W σ h+b σ ;

[0022] wherein, W μ and W σ represent weight matrices of the mean and the log variance respectively, b μ and b σ represent bias vectors of the mean and the log variance respectively, and σ represents a standard deviation;

[0023] through a sampling method, extracting a latent variable z from the mean and a distribution of the standard deviation σ obtained according to the log variance:

[0024] z=μ+σ⊙ε;

[0025] wherein, ε represents a random noise vector, and ⊙ represents scaling;

[0026] adopting the feature reconstructor, reconstructing the latent variable through a multi-layer fully connected network and a batch normalization process to obtain reconstructed features

[0027]

[0028] wherein, VAEDecoder(·) represents feature reconstruction;

[0029] The decoder is used to introduce dynamic context embedding, generate a predicted action sequence according to a probability distribution output by the reconstructed feature, and further obtain a material distribution path.

[0030] Optionally, the feature reconstructor is used to reconstruct the latent variable through a multi-layer fully connected network and a batch normalization process to obtain a reconstructed feature In this case, only the reconstruction loss is considered, and the KL divergence loss is not considered:

[0031]

[0032] wherein, L recon represents a reconstruction loss, N represents the number of completed tasks, h i represents a feature vector corresponding to the i-th task, represents a reconstructed feature corresponding to the i-th task.

[0033] Optionally, the decoder is used to introduce dynamic context embedding, generate a predicted action sequence according to a probability distribution output by the reconstructed feature, and further obtain a material distribution path, specifically including:

[0034] The decoder uses a long short-term memory network combined with an attention mechanism to calculate a probability distribution output at each decoding step;

[0035] Embedding global information of input data into h' g , information embedding of the current time step generation node Information embedding of the first time step generation node Average value of information embedding of all previous time step generation nodes The dynamic context embedding h' is obtained by summation:

[0036]

[0037] The dynamic context embedding h' is combined with a current hidden state h0 output by the long short-term memory network and used as a query input q of the attention mechanism, to dynamically update the hidden state. t :

[0038]

[0039] Through the cyclic nature of the long short-term memory network and the attention mechanism, a predicted output of the entire task sequence is gradually established, a predicted action sequence is generated, and a material distribution path is obtained.

[0040] Optionally, the pre-training of the encoder-decoder based model uses a grid search algorithm to find the best model parameters, obtaining a pre-training model under the best model parameters, and then further comprising:

[0041] The material distribution path planning problem is described as a Markov decision process M:

[0042] M=(S,A,P,R,γ,s0,T);

[0043] Wherein, S represents a state set, which is a set of unfinished tasks and completed tasks; A represents an action space, which refers to a set of actions that can be performed in each state; P represents a state transition probability, which refers to the probability of transitioning from state s to state s' after performing action a; R represents a reward function; γ is a discount factor, γ∈[0,1], used to balance the relationship between current rewards and future rewards; s0 represents an initial state distribution; T represents the number of time steps.

[0044] Optionally, the pre-training model is optimized and trained using a proximal policy optimization algorithm to obtain a target path planning model, specifically comprising:

[0045] Load the weights of the pre-training model and use the proximal policy optimization algorithm to iteratively optimize the pre-training model;

[0046] Generate actions through the current policy and calculate the transition probability ratio between the new policy and the old policy;

[0047] Calculate the policy loss using the transition probability ratio and the advantage value obtained from the environment feedback:

[0048]

[0049] Wherein, L a represents the policy loss; t represents the time step; A t represents the advantage function; r t (θ) represents the transition probability ratio of the new policy θ' relative to the old policy θ, P θ' (a t |s t ) represents the probability of transitioning to the next state under the new policy θ' when action a t is performed in state s t , P θ (a t |s t ) represents the probability of transitioning to the next state under the old policy θ when action a t is performed in state s t ; represents the modified policy ratio, clip is a clipping operation to limit the step of policy update by maximum minimization, and ε' is a constant;

[0050] The value function loss is calculated by state value prediction and target return:

[0051]

[0052] where L c represents the value function loss, V'(s i ) represents the state value prediction of state s i , V(s i ) represents the state value corresponding to state s i , and R i represents the target return corresponding to state s i ; clip(·) represents a clipping operation to limit the change of V'(s i ) relative to V(s i ).

[0053] The entropy loss is calculated by the probability output of the policy network:

[0054]

[0055] where L e represents the entropy loss, and π(a j |s i ) represents the probability output of the policy network for selecting action a i under given state s j .

[0056] The total loss is obtained by weighted sum of the policy loss, the value function loss and the entropy loss:

[0057] L rl = θ a L a + θ c L c + θ e L e .

[0058] where L rl is the total loss of reinforcement learning, and θ a , θ c and θ e are variable parameters.

[0059] The pre-trained model is updated by back propagation according to the total loss, and the current best model is saved according to early stopping condition and performance improvement condition.

[0060] Test the current best model, and take the current best model meeting the test requirements as a target path planning model.

[0061] To achieve the above-mentioned purposes, the application further provides a supply chain distribution path planning system based on reinforcement learning, comprising:

[0062] A data acquisition module is configured to acquire historical distribution data of a delivery person and extract features of the historical distribution data.

[0063] A model construction module is configured to construct an encoder-decoder-based model, wherein the encoder-decoder-based model comprises a Transformer encoder configured to convert the features into a high-dimensional feature vector, a VAE-based feature reconstructor configured to perform feature reconstruction on the feature vector to obtain reconstructed features, and a decoder configured to process the reconstructed features to obtain a material distribution path.

[0064] A model pre-training module is configured to pre-train the encoder-decoder-based model, find optimal model parameters using a grid search algorithm, and obtain a pre-trained model under the optimal model parameters.

[0065] A model optimization training module is configured to optimize and train the pre-trained model using a proximal policy optimization algorithm to obtain a target path planning model.

[0066] A material distribution path planning module is configured to acquire uncompleted task data of a delivery person, input the uncompleted task data into the target path planning model, and output a target material distribution path from the target path planning model.

[0067] To achieve the above-mentioned purposes, the application further provides a terminal, comprising a memory, a processor, and a supply chain distribution path planning program based on reinforcement learning stored in the memory and executable on the processor, wherein the supply chain distribution path planning program based on reinforcement learning implements the steps of the supply chain distribution path planning method based on reinforcement learning when executed by the processor.

[0068] To achieve the above-mentioned purposes, the application further provides a computer-readable storage medium storing a supply chain distribution path planning program based on reinforcement learning, wherein the supply chain distribution path planning program based on reinforcement learning implements the steps of the supply chain distribution path planning method based on reinforcement learning when executed by a processor.

[0069] In the present application, the historical delivery data of the delivery personnel is acquired, and the features of the historical delivery data are extracted; a model based on an encoder and a decoder is constructed, the model based on the encoder and the decoder comprising a Transformer encoder for converting the features into a high-dimensional feature vector, a feature reconstructor based on VAE for feature reconstruction of the feature vector to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material delivery path; the model based on the encoder and the decoder is pre-trained, the best model parameters are found using a grid search algorithm, and a pre-trained model under the best model parameters is obtained; the pre-trained model is optimized and trained using a proximal policy optimization algorithm, and a target path planning model is obtained; the unfinished task data of the delivery personnel is acquired, and the unfinished task data is input into the target path planning model, and the target path planning model outputs a target material delivery path. The present application extracts features reflecting the individual delivery preferences of the delivery personnel from the historical delivery data of the delivery personnel in combination with the Transformer encoder, the variational autoencoder and the feature reconstructor, introduces a dynamic context embedding dynamic update decoder attention focus point, carefully reflects the individual delivery preferences of the delivery personnel, improves the adaptability and decision quality of the model when facing dynamic change data, enables the model to adapt to complex dynamic environments and individual delivery preferences of the delivery personnel, and performs well when coping with uncertainties in actual applications; the proximal policy optimization algorithm is used to optimize the model, effectively improves the decision quality and stability of the model, and realizes economic, efficient and adaptive supply chain material delivery path planning that adapts to complex dynamic environments and individual delivery preferences of the delivery personnel. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is a flowchart of a preferred embodiment of the supply chain delivery path planning method based on reinforcement learning of the present application;

[0071] Figure 2 is another flowchart of a preferred embodiment of the supply chain delivery path planning method based on reinforcement learning of the present application;

[0072] Figure 3 is a structure diagram of a preferred embodiment of the supply chain delivery path planning system based on reinforcement learning of the present application;

[0073] Figure 4 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0074] To make the objectives, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0075] With the development of information technology and big data analysis, more and more manufacturing enterprises and supply chain participants have begun to realize the importance of digitization. Digitization can improve the transparency and efficiency of the supply chain, enhance the response speed and flexibility of the supply chain, and through digitization, enterprises can accurately predict market demand, optimize inventory management, reduce costs, and monitor production and distribution in real time to quickly adapt to market changes. In addition, digitization also promotes collaboration and resource integration within the supply chain, enhancing the competitiveness of the entire industry. Therefore, building a digital supply chain is a key strategy to improve enterprise efficiency and market adaptability.

[0076] The Vehicle Routing Problem (VRP) is defined as a mathematical model that seeks the optimal path to minimize cost or time, and is an NP-hard problem. It is also a core problem in the logistics and distribution industry. Subsequently, many VRP variants have been proposed by domestic and foreign scholars to meet different needs. In the manufacturing industry, the material distribution problem can be considered as a specific instance of the VRP problem. In a specific supply chain material distribution scenario, materials need to be distributed from warehouses or supply points to multiple receiving points on the production line. Reasonable planning of the distribution route is the key to improving production efficiency and reducing costs for enterprises. Traditional VRP problems optimize distribution routes to reduce costs and improve efficiency. However, when faced with real-time changes in the production site and individual delivery preferences, traditional VRP models often lack flexibility. Therefore, the optimization of the distribution route is an urgent need, and it is crucial to choose a more economical and efficient route that also adapts to the preferences of the delivery personnel.

[0077] Although current path planning methods have achieved effective results to some extent, there are still many limitations. First, when dealing with the massive historical data of delivery personnel, it is difficult to fully capture the key features and dynamic changes in the data. Second, traditional path generation methods often rely on predefined rules and algorithms, making it difficult to adapt to complex dynamic environments, resulting in limited performance in dealing with uncertainties in actual applications.

[0078] To solve the above technical problems, the application provides a supply chain distribution path planning method based on reinforcement learning, historical distribution data of a delivery person is acquired, and features of the historical distribution data are extracted; a model based on an encoder and a decoder is constructed, the model based on the encoder and the decoder includes a Transformer encoder used for converting the features into a high-dimensional feature vector, a feature reconstructor based on VAE used for performing feature reconstruction on the feature vector to obtain reconstructed features, and a decoder used for processing the reconstructed features to obtain a material distribution path; the model based on the encoder and the decoder is pre-trained, a grid search algorithm is used to find optimal model parameters, and a pre-trained model under the optimal model parameters is obtained; a proximal policy optimization algorithm is used to perform optimization training on the pre-trained model, and a target path planning model is obtained; uncompleted task data of the delivery person is acquired, and the uncompleted task data is input into the target path planning model, and the target path planning model outputs a target material distribution path. The application extracts features reflecting individual distribution preferences of the delivery person from the historical distribution data of the delivery person in combination with the Transformer encoder, the variational autoencoder and the feature reconstructor, introduces dynamic context embedding to dynamically update the attention focus point of the decoder, carefully reflects the individual distribution preferences of the delivery person, improves the adaptability and decision quality of the model when facing dynamic change data, and enables the model to adapt to complex dynamic environments and individual distribution preferences of the delivery person, and the model performs well when coping with uncertainties in actual applications; the proximal policy optimization algorithm is used to optimize the model, and the decision quality and stability of the model are effectively improved; and the supply chain material distribution path planning that is economic, efficient and adaptive to complex dynamic environments and individual distribution preferences of the delivery person is realized.

[0079] The application content is further described through the description of the embodiments in combination with the drawings.

[0080] The preferred embodiment of the supply chain distribution path planning method based on reinforcement learning of the application is specifically shown in Figure 1 and Figure 2 , and specifically includes:

[0081] S1, historical distribution data of a delivery person is acquired, and features of the historical distribution data are extracted.

[0082] In an implementation mode of the embodiment, the historical distribution data of the delivery person is acquired, and the features of the historical distribution data are extracted, specifically including:

[0083] The historical distribution data of the delivery person is acquired, and the historical distribution data is preprocessed to extract the features of the historical distribution data, wherein the historical distribution data includes completed task data and historical material distribution paths of the delivery person, and the features include a distribution distance and a distribution time corresponding to a completed task in the completed task data.

[0084] Define the completed task data of delivery person w as M. w ={m1,m2,m3,...,m j ,...,m N}, where N represents the number of tasks completed, m N This represents the Nth completed task, where m represents each completed task. j Including delivery distance and delivery time;

[0085] Define the historical material delivery route of delivery person w. in, This represents the j-th completed task m in the historical material delivery path. j Delivery location, This indicates the Mth completed task in the historical material delivery path. N Delivery location.

[0086] Specifically, this invention collects historical delivery data of delivery personnel as training data, and analyzes the individual delivery preferences of delivery personnel through extensive historical data, so that the generated material delivery route is adapted to the individual delivery preferences of delivery personnel.

[0087] S2. Construct a model based on an encoder and a decoder. The model based on an encoder and a decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the features from the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain the material delivery path.

[0088] In one implementation of this embodiment, the model constructed based on an encoder and a decoder includes a Transformer encoder for converting the features into high-dimensional feature vectors, a VAE-based feature reconstructor for reconstructing the feature vectors to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain the material delivery path. Specifically, it includes:

[0089] Construct a model based on an encoder and a decoder, wherein the model based on an encoder and a decoder includes a Transformer encoder, a VAE-based feature reconstructor and a decoder, wherein the VAE-based feature reconstructor includes a variational autoencoder and a feature reconstructor;

[0090] The Transformer encoder is used to parse and encode the features through a multi-head self-attention mechanism, transforming the features into a high-dimensional feature vector h.

[0091] The variational autoencoder is used to convert the feature vector into parameters of a latent space through a linear layer, the parameters including a mean mu and a log variance log(sigma 2 ):

[0092] mu = W μ h + b μ ;

[0093] log(sigma 2 ) = W σ h + b σ ;

[0094] where W μ and W σ represent weight matrices of the mean and the log variance respectively, b μ and b σ represent bias vectors of the mean and the log variance respectively, and sigma represents a standard deviation;

[0095] A latent variable z is sampled from the mean and a distribution of the standard deviation sigma obtained according to the log variance by a sampling method:

[0096] z = mu + sigma o e;

[0097] where e represents a random noise vector, which is assumed to come from a standard normal distribution; o represents scaling, and sigma o e represents a random noise vector e sampled from the standard normal distribution scaled using the standard deviation sigma;

[0098] The feature reconstructor is used to reconstruct the latent variable through a multi-layer fully connected network and a batch normalization process to obtain reconstructed features

[0099]

[0100] where VAEDecoder(·) represents feature reconstruction.

[0101] The decoder is used to introduce dynamic context embedding, output a probability distribution according to the reconstructed features, generate a predicted action sequence, and further obtain a material distribution path.

[0102] Specifically, the present application proposes a special variational deep reinforcement learning framework, which is based on an encoder-decoder structure. A Transformer encoder converts the spatio-temporal information of a delivery task into a comprehensive feature representation (i.e., a high-dimensional context-sensitive feature representation, a high-dimensional feature vector). Then, the comprehensive feature representation is further converted into two key parameters (i.e., a mean and a log variance) of a latent space through two linear layers of a variational autoencoder, which maps the output of the Transformer encoder to the latent space. Next, a latent variable is drawn from the distribution of the key parameters, and the latent variable is fed into a feature reconstructor to reconstruct feature data through a multi-layer fully connected network and a batch normalization process, thereby obtaining reconstructed features. A decoder utilizes the reconstructed features to gradually form a predicted delivery route, thereby learning from complex input data and generating a delivery route that adapts to the individual delivery preferences of a delivery person.

[0103] In one implementation mode of the present embodiment, the feature reconstructor is adopted to reconstruct the latent variable through a multi-layer fully connected network and a batch normalization process, thereby obtaining reconstructed features In this case, only the reconstruction loss is considered, and the KL divergence loss is not considered.

[0104]

[0105] wherein L recon represents the reconstruction loss, N represents the number of completed tasks, h i represents the feature vector corresponding to the i-th task, represents the reconstructed features corresponding to the i-th task.

[0106] Specifically, the conventional VAE model requires that the distribution of the latent space tends to approach a standard normal distribution through the KL divergence loss. However, in the delivery service, it is necessary to adapt to the individual delivery preferences of a delivery person. If the latent space is subject to too strict statistical constraints, the generated route will be too standardized and lack adaptability in specific environments. Therefore, the present application does not consider adding the KL divergence loss during the training process, so as to maintain the diversity and adaptability of the generated data.

[0107] In one implementation mode of the present embodiment, the decoder is adopted to introduce dynamic context embedding, output a probability distribution according to the reconstructed features, and generate a predicted action sequence, thereby obtaining a material delivery path. Specifically, the method comprises the following steps:

[0108] The decoder uses a long short-term memory network combined with an attention mechanism to calculate the probability distribution output at each decoding step.

[0109] The global information of the input data is embedded into h' g , and the information embedding of the current time step generation node The information embedding of the first time step generating node The average of the information embedding of all previous time step generating nodes The dynamic context embedding h' is obtained by summation

[0110]

[0111] The hidden state is dynamically updated using the dynamic context embedding h', which is combined with the current hidden state h0 output by the LSTM network as the query input q of the attention mechanism t :

[0112]

[0113] The prediction output of the entire task sequence is gradually established through the recurrent nature of the LSTM network and the attention mechanism, a predicted action sequence is generated, and a material distribution path is obtained.

[0114] Specifically, the application introduces an effective dynamic context embedding, which integrates the global information of the entire input data, i.e., the embedding h' g The information embedding of the current time step generating node The information embedding of the first time step generating node The average of the information embedding of all previous time step generating nodes The enhanced embedding h' is obtained by addition, which ensures that the model can fully consider the influence of the starting point, the average features of the historical path selection and the current state information at each time step, dynamically integrate new input and existing sequence information, and thus comprehensively consider the context information of the entire sequence in each step of decision-making.

[0115] Meanwhile, this enhanced dynamic context embedding h' is used to dynamically update the hidden state. Since the LSTM (Long Short-Term Memory) combined with the attention mechanism is used in the decoder to calculate the output probability distribution at each decoding step (i.e., each time step) to select the most likely next task, wherein the LSTM is responsible for the fusion of all previous task embedding vectors, the hidden state of the previous time step and the input features of the current time step as input, the prediction output of the entire task sequence is gradually established through the recurrent nature of the attention mechanism and the LSTM. Therefore, the application combines h' with the current hidden state h0 output by the LSTM as the query input of the attention mechanism in the decoder, i.e. The model can dynamically update its attention focus point at each time step through the dynamic node information, which enhances the adaptability and decision quality of the model when facing dynamic changing data.

[0116] S3, pre-training the encoder-decoder based model, finding the best model parameters using a grid search algorithm, and obtaining a pre-trained model under the best model parameters.

[0117] In one implementation of the embodiment, the pre-training the encoder-decoder based model, finding the best model parameters using a grid search algorithm, and obtaining a pre-trained model under the best model parameters, further comprises:

[0118] The material distribution path planning problem is described as a Markov decision process M:

[0119] M=(S,A,P,R,γ,s0,T);

[0120] Wherein, S represents a state set, the state set is a set of unfinished tasks and completed tasks; A represents an action space, which refers to a set of actions that can be performed in each state; P represents a state transition probability, which refers to the probability of transitioning from state s to state s' after performing action a; R represents a reward function; γ is a discount factor, γ∈[0,1], used to balance the relationship between current rewards and future rewards; s0 represents an initial state distribution; T represents the number of time steps.

[0121] Specifically, in reinforcement learning, an agent learns how to make decisions in a specific environment by interacting with the environment. Unlike other machine learning methods, during the training process of reinforcement learning, the agent needs to receive feedback from the environment, which is usually represented as a numerical value of reward or punishment. The entire reinforcement learning process is represented as an agent interacting with the environment through a series of action strategies (Action), generating a new state (State), while the environment gives a reward (Reward). After multiple iterations, a strategy is found, i.e. a series of behavior rules, which maximizes the long-term cumulative reward. In this invention, the entire encoder-decoder model (i.e. the encoder-decoder based model) is regarded as an agent, and the prediction of the future material distribution route of the delivery personnel is regarded as a sequential decision-making process, which is represented as a Markov decision process, wherein:

[0122] State: in each decoding step, the state s t includes the embedding representation H of the unfinished task, the hidden state h t of the decoder, the path constraint C and the task route y 1:t-1 output at the previous time step:

[0123] s t =(H,h t ,C,y 1:t-1 );

[0124] Action: action a t∈A t , defined as the tasks that can be chosen given the current state condition, one task is chosen at each decoding step, A t is the state space at the t-th time step, which changes as the decoding process proceeds, the state and the mask are updated after action selection;

[0125] Transition probability: the transition probability P(s t+1 |s t ,a t ):S×A, defined as the probability of transitioning to the next state s t when performing action a t in state s t+1 ;

[0126] Reward: the reward R is set as the square of the position bias:

[0127]

[0128] where m represents the number of tasks in the route, i s is the position index of the current output, i l is the correct position index that the target output y t should be in; if the last element of the sequence is in the label and the position matches, return R0, if the last element of the sequence is not in the label and the index is out of the valid range, return 0, otherwise calculate the square average of the index difference between the sequence and the corresponding position in the label as the reward.

[0129] S4, using a proximal policy optimization algorithm to optimize and train the pre-trained model to obtain a target path planning model.

[0130] In one implementation manner of the embodiment, the using a proximal policy optimization algorithm to optimize and train the pre-trained model to obtain a target path planning model specifically includes:

[0131] S41, loading the weight of the pre-trained model, using a proximal policy optimization algorithm to iteratively optimize the pre-trained model;

[0132] S42, generating an action by a current policy and calculating a transition probability ratio between a new policy and an old policy;

[0133] S43, using the transition probability ratio and an advantage value obtained from environment feedback to calculate a policy loss:

[0134]

[0135] wherein L a represents the policy loss; t represents the time step; A t represents the advantage function; r t (θ) represents the transition probability ratio of the new policy θ' relative to the old policy θ, P θ' (a t |s t ) represents the probability of transitioning to the next state when performing action a t in state s t under the new policy θ'; θ (a t |s t ) represents the probability of transitioning to the next state when performing action a t in state s t under the old policy θ; represents the modified policy ratio, the step size of policy update is limited by introducing a clipping operation clip through max-min minimization, and ε' is a constant; the difference between the new and old policies is calculated, and a clipping mechanism is applied to limit the size of the update, ensuring the stability of the policy update;

[0136] S44, calculate the value function loss by state value prediction and target return:

[0137]

[0138] where L c represents the value function loss, V'(s i ) represents the state value prediction corresponding to state s i , V(s i ) represents the state value corresponding to state s i , and R i represents the target return corresponding to state s i ; clip(·) represents a clipping operation to limit the change of V'(s i ) relative to V(s i );

[0139] S45, calculate the entropy loss by the probability output of the policy network:

[0140]

[0141] where L e represents the entropy loss, and π(a j |s i ) represents the probability output of the policy network for selecting action a i in a given state s j ;

[0142] S46, weight and sum the policy loss, the value function loss and the entropy loss to obtain the total loss:

[0143] Lrl = θ a L a + θ c L c + θ e L e ;

[0144] wherein, L rl is the total loss of reinforcement learning, θ a , θ c and θ e are all variable parameters;

[0145] S47, updating the pre-training model according to the back propagation of the total loss, saving the current best model according to the early stopping condition and performance improvement condition;

[0146] S48, testing the current best model, and taking the current best model meeting the test requirements as the target path planning model.

[0147] Specifically, the present application uses the PPO algorithm based on actor-critic for training, the actor is an agent based on an encoder-decoder model, which is responsible for directly interacting with the environment and generating actions. The critic needs to access and understand the entire environment state in order to effectively evaluate the actor's behavior and provide feedback, so the critic and the actor share the same neural network and parameters. Since the expected reward is calculated by sampling, this method will result in high variance. Therefore, the present application uses GAE to calculate the advantage value representing whether the action is "good" or "bad" in a certain state, and introduces a decay parameter λ to balance the bias and variance by combining short-term and long-term return estimates:

[0148]

[0149] wherein, represents the advantage value at time step t, λ is a decay parameter, when λ is close to 1, the advantage estimate considers a longer future, thereby increasing the variance and reducing the bias; when λ is close to 0, the advantage estimate relies more on the current estimate, thereby reducing the variance and increasing the bias; r' t+i represents the immediate reward at time step t+i, V(s t+i ) represents the state value of state s t+i , V(s t+i+1 ) represents the state value of state s t+i+1 , and T is the termination time step of the trajectory, and T-t represents the number of remaining time steps from the current time step t to the end of the trajectory.

[0150] The PPO algorithm based on actor-critic includes an Actor network and a Critic network, wherein:

[0151] Actor network: process the encoding and decoding part and generate new actions, i.e. routes. In the encoder part, first, the embedding representation (i.e. feature vector) of the delivery task is extracted by the Transformer encoder using the multi-head self-attention mechanism, and then the feature data is reconstructed by the VAE-based feature reconstructor. The reconstructed feature data is input into the decoder, and the probability distribution is output by the decoder to generate the predicted action sequence.

[0152] Critic network: evaluate the effect of the current policy (i.e. action, i.e. route). The action probability distribution output by the Actor network is input into the Critic model to calculate the immediate reward of each action in the training process, which is used to directly feedback the action result. The value function network is used to predict the expected return of each state or state-action pair to determine which actions perform better than the average policy and should be optimized by the policy more frequently, i.e. to provide the necessary feedback to guide the training of the Actor network.

[0153] The model is optimized and trained using the Proximal Policy Optimization algorithm, including: loading the pre-trained model weights, and the model is optimized through a series of iteration cycles, and steps S42-S45 are performed in each cycle. During the entire training process, all losses are weighted and summed to form the total loss of reinforcement learning. At the end of each cycle, the model is evaluated on the validation set, and the current best model is saved according to the early stopping condition and performance improvement. Finally, the model is tested on the test set. The present application first performs pre-training, finds the best model parameters through grid search, and then fine-tunes the pre-trained model using the PPO algorithm. Finally, the model effect is evaluated using multiple performance indicators on the test set.

[0154] S5, obtaining the unfinished task data of the delivery personnel, inputting the unfinished task data into the target path planning model, and the target path planning model outputting a target material delivery path.

[0155] In one implementation of the present embodiment, the unfinished task data of the delivery personnel is obtained, the unfinished task data is input into the target path planning model, and the target path planning model outputs a target material delivery path, specifically including:

[0156] Define the set of unfinished tasks of the delivery personnel U w ={u1,u2,u3,...,u n}, n represents the number of unfinished tasks, wherein each task u i is composed of a set of features, including the distance between the current location of the delivery personnel and the task location, and the remaining commitment time of the task.

[0157] Given the set of unfinished tasks U wThe prediction of the decision (i.e., path planning) of the delivery personnel on the unfinished task is defined as Since the task list of the delivery personnel is dynamically changing, new tasks are constantly added, so the number of tasks in the label may be less than the number of tasks in the prediction. And the original delivery order may be affected in the process, such as using the observation data after t+1 time as the basis for model training, which may introduce misleading information, because these data no longer accurately reflect the decision logic and action sequence at t time, so the loss at a certain time in the present application only includes those tasks completed within the original planned time.

[0158] As shown in Figure 2 In the present application, based on a large amount of historical delivery data of the delivery personnel, first, data preprocessing is performed to extract key features such as latitude, longitude, and deadline; an encoder with a multi-layer Transformer structure is used to parse and encode the geographical and temporal data of the input route, which effectively captures the complex relationships and long-distance dependencies between different locations through a multi-head self-attention mechanism, thereby generating a high-dimensional context-sensitive feature representation; a feature reconstructor based on VAE is used to reconstruct the feature representation, and the decoder focuses on using the reconstructed deep features to gradually generate the final route, and finally the PPO algorithm is used to fine-tune the model to optimize the model performance.

[0159] In addition, based on the above-mentioned supply chain distribution path planning method based on reinforcement learning, the present application also correspondingly provides a supply chain distribution path planning system based on reinforcement learning, wherein the preferred embodiment of the supply chain distribution path planning system based on reinforcement learning, as shown in Figure 3 includes:

[0160] The data acquisition module 01 is used to acquire the historical delivery data of the delivery personnel and extract the features of the historical delivery data;

[0161] The model construction module 02 is used to construct an encoder and decoder-based model, which includes a Transformer encoder for converting the features into a high-dimensional feature vector, a feature reconstructor based on VAE for feature reconstruction of the feature vector to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain the material distribution path;

[0162] The model pre-training module 03 is used to pre-train the encoder and decoder-based model, and uses a grid search algorithm to find the best model parameters to obtain a pre-trained model under the best model parameters;

[0163] The model optimization training module 04 is used to optimize and train the pre-trained model using a proximal policy optimization algorithm to obtain a target path planning model;

[0164] The material distribution path planning module 05 is configured to obtain the unfinished task data of the delivery personnel, input the unfinished task data into the target path planning model, and output the target material distribution path from the target path planning model.

[0165] In addition, based on the above-mentioned supply chain distribution path planning method and system based on reinforcement learning, the application further provides a terminal, wherein the preferred embodiment of the terminal comprises a processor 10, a memory 20 and a display 30, as shown in the figure. Figure 4 Figure 4 Only part of the components of the terminal are shown, but it should be understood that all the components shown are not required to be implemented, and more or less components can be alternatively implemented.

[0166] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card and a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a supply chain distribution path planning program based on reinforcement learning 40, which can be executed by the processor 10, so as to realize the steps of the supply chain distribution path planning method based on reinforcement learning in the application.

[0167] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as executing the supply chain distribution path planning program based on reinforcement learning 40, etc.

[0168] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information on the terminal and to display a visualized user interface.

[0169] ​In an embodiment, the steps of the reinforcement learning based supply chain distribution path planning method as described above are implemented when the processor 10 executes the reinforcement learning based supply chain distribution path planning program 40 in the memory 20.

[0170] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a reinforcement learning based supply chain distribution path planning program, and the reinforcement learning based supply chain distribution path planning program, when executed by a processor, implements the steps of the reinforcement learning based supply chain distribution path planning method as described above.

[0171] It should be noted that in the present application, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or terminals including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles, or terminals. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or terminal including the element.

[0172] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, which can be stored in a computer readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0173] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all such improvements and changes shall fall within the protection scope of the claims of the present application.

Claims

1. A supply chain material distribution path planning method based on reinforcement learning, characterized by, The supply chain material distribution path planning method based on reinforcement learning comprises: obtaining historical distribution data of a delivery person and extracting features of the historical distribution data; building an encoder and decoder-based model, the encoder and decoder-based model comprising a Transformer encoder for converting the features into a high-dimensional feature vector, a VAE-based feature reconstructor for feature reconstruction of the feature vector to obtain reconstructed features, the feature reconstructor only considering a reconstruction loss and not considering a KL divergence loss, and a decoder for processing the reconstructed features to obtain a material distribution path, using the decoder, introducing dynamic context embedding, outputting a probability distribution according to the reconstructed features, generating a predicted action sequence, and further obtaining a material distribution path; pre-training the encoder and decoder-based model, using a grid search algorithm to find optimal model parameters, and obtaining a pre-trained model under the optimal model parameters; using a proximal policy optimization algorithm to optimize and train the pre-trained model to obtain a target path planning model; obtaining uncompleted task data of the delivery person and inputting the uncompleted task data into the target path planning model, the target path planning model outputting a target material distribution path.

2. The reinforcement learning-based supply chain material distribution path planning method according to claim 1, characterized by, The obtaining of the historical distribution data of the delivery person and the extraction of the features of the historical distribution data specifically comprises: obtaining historical distribution data of a delivery person and pre-processing the historical distribution data to extract features of the historical distribution data, wherein the historical distribution data comprises completed task data and historical material distribution paths of the delivery person, and the features comprise delivery distances and delivery times corresponding to completed tasks in the completed task data; Definition of the deliveryman The completed task data of the deliveryman is , wherein, represents the number of completed tasks, represents the completed task, and each completed task includes a delivery distance and a delivery time; Define delivery personnel Historical material delivery routes ,in, Indicates the first item in the historical material delivery path Completed tasks Delivery location, Indicates the first item in the historical material delivery path Completed tasks Delivery location. 3.The reinforcement learning-based supply chain material distribution path planning method according to claim 1, wherein, The construction of the encoder and decoder-based model, the encoder and decoder-based model comprising a Transformer encoder for converting the features into a high-dimensional feature vector, a VAE-based feature reconstructor for feature reconstruction of the feature vector to obtain reconstructed features, and a decoder for processing the reconstructed features to obtain a material distribution path, specifically comprises: building an encoder and decoder-based model, the encoder and decoder-based model comprising a Transformer encoder, a VAE-based feature reconstructor, and a decoder, the VAE-based feature reconstructor comprising a variational autoencoder and a feature reconstructor; Adopting the Transformer encoder, the feature is parsed and encoded through a multi-head self-attention mechanism, and the feature is converted into a high-dimensional feature vector ; With the variational autoencoder, the feature vector is converted into parameters of a latent space by a linear layer, the parameters including a mean and a log variance ​ ； ； wherein and Wm and Wlv denote the weight matrix of the mean and the log-variance, respectively, and bmand blvdenote the bias vector of the mean and the log-variance, respectively, denotes the standard deviation; by sampling method from the distribution of the mean and the standard deviation derived from the log-variance of the latent variables : ； wherein represents a random noise vector, and denotes a scaling; The latent variable is reconstructed by a multi-layer fully connected network and a batch normalization process to obtain reconstructed features by using the feature reconstructor : ; wherein represents feature reconstruction.

4. The reinforcement learning-based supply chain material distribution path planning method according to claim 3, characterized by, The feature reconstructor is configured to reconstruct the latent variable by a multi-layer fully connected network and a batch normalization process to obtain reconstructed features In some embodiments, only the reconstruction loss is considered, and the KL divergence loss is not considered. ； wherein, denotes the reconstruction loss, denotes the number of completed tasks, denotes the feature vector corresponding to the th task, denotes the reconstructed feature corresponding to the th task.

5. The reinforcement learning-based supply chain material distribution path planning method according to claim 3, characterized by, The use of the decoder, the introduction of dynamic context embedding, the output of a probability distribution according to the reconstructed features, the generation of a predicted action sequence, and the further obtaining of a material distribution path specifically comprise: The decoder uses a long short-term memory network combined with an attention mechanism to calculate a probability distribution output at each decoding step; Embedding global information of input data , information embedding of the current time step generating node , information embedding of the first time step generating node and average of information embedding of all previous time step generating nodes , dynamic context embedding by summation : ； using a dynamic context embedding with the current hidden state output by the long short-term memory network in combination as a query input to the attention mechanism : ； Through the cyclic nature of the long short-term memory network and the attention mechanism, a predicted output of the entire task sequence is gradually established to generate a predicted action sequence and obtain a material distribution path.

6. The reinforcement learning-based supply chain material distribution path planning method according to claim 1, characterized by, The pre-training of the encoder and decoder-based model, the use of a grid search algorithm to find optimal model parameters, and the obtaining of a pre-trained model under the optimal model parameters further comprise: The material distribution path planning problem is described as a Markov decision process M: ； wherein, S represents a state set, which is a set of unfinished tasks and completed tasks; A represents an action space, which refers to a set of actions performed at each state; P represents a state transition probability, which refers to a probability of transitioning from a state to a state after performing an action ; R represents a reward function; is a discount factor, used to balance the relationship between current rewards and future rewards; is an initial state distribution; is a time step number.

7. The reinforcement learning-based supply chain material distribution path planning method according to claim 6, characterized by, The pre-trained model is optimized and trained using a proximal policy optimization algorithm to obtain a target path planning model, specifically including: Load the weights of the pre-trained model, and use the proximal policy optimization algorithm to iteratively optimize the pre-trained model; Generate actions through the current policy and calculate the transition probability ratio between the new policy and the old policy; Use the transition probability ratio and the advantage value obtained from the environment feedback to calculate the policy loss: ； in, Indicates the strategy loss; Indicates a time step; Represents the dominance function; , Indicating a new strategy Compared to the old strategy The ratio of transition probabilities, Indicating a new strategy Below, in state Execute action The probability of transitioning to the next state. Indicates the old strategy Below, in state Execute action The probability of transitioning to the next state; , This represents the corrected policy ratio, where the clipping operation is introduced through minimization to limit the policy update step size. It is a constant; Calculate the value function loss through state value prediction and target return: ； wherein, represents the value function loss, represents the state corresponding state value prediction, represents the state corresponding state value, represents the state corresponding target return; represents a clipping operation for limiting variations in relative to Calculate the entropy loss through the probability output of the policy network: ； wherein, denotes the entropy loss, denotes the probability output of the policy network for selecting an action under a given state ; Sum the policy loss, value function loss, and entropy loss to obtain the total loss: ； wherein, is the total loss for reinforcement learning, , and are variable parameters; Update the pre-trained model according to the backpropagation of the total loss, and save the current best model according to the early stopping condition and performance improvement condition; Test the current best model, and use the current best model that meets the test requirements as the target path planning model.

8. A supply chain material distribution path planning system based on reinforcement learning, characterized by, The supply chain material distribution path planning system based on reinforcement learning includes: A data acquisition module for acquiring historical distribution data of a delivery person and extracting features of the historical distribution data; A model construction module for constructing an encoder and decoder-based model, the encoder and decoder-based model including a Transformer encoder for converting the features into a high-dimensional feature vector, a VAE-based feature reconstructor for reconstructing the feature vector to obtain reconstructed features, the feature reconstructor only considering reconstruction loss and not considering KL divergence loss, and a decoder for processing the reconstructed features to obtain a material distribution path, using the decoder, introducing dynamic context embedding, outputting a probability distribution based on the reconstructed features, and generating a predicted action sequence to obtain a material distribution path; A model pre-training module for pre-training the encoder and decoder-based model, using a grid search algorithm to find the best model parameters, and obtaining a pre-trained model under the best model parameters; A model optimization training module for optimizing and training the pre-trained model using a proximal policy optimization algorithm to obtain a target path planning model; A material distribution path planning module for acquiring uncompleted task data of a delivery person, inputting the uncompleted task data into the target path planning model, and outputting a target material distribution path from the target path planning model.

9. A terminal, characterized by comprising: The terminal includes a memory, a processor, and a reinforcement learning-based supply chain material distribution path planning program stored on the memory and executable on the processor, which implements the steps of the reinforcement learning-based supply chain material distribution path planning method according to any one of claims 1-7 when executed by the processor.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a reinforcement learning based supply chain material distribution path planning program, and the reinforcement learning based supply chain material distribution path planning program, when executed by the processor, implements the steps of the reinforcement learning based supply chain material distribution path planning method according to any one of claims 1-7.

Images