Supply chain distribution network multi-level inventory cost optimization method

Abstract

The invention relates to a supply chain distribution network multi-level inventory cost optimization method, which belongs to the field of supply chain inventory, and comprises the following steps: obtaining initial information of a full-process supply chain distribution network multi-level inventory cost optimization problem; establishing a multi-level inventory cost model based on the initialization information, and analyzing multi-level inventory cost components of the cost model; determining an objective function of a multi-level inventory cost optimization problem of the whole-process supply chain distribution network based on the cost component; designing and improving a near-end strategy optimization algorithm; and performing optimization and adaptive adjustment on core parameters of the improved near-end strategy optimization algorithm to obtain an optimal multi-level inventory cost control strategy. The method can adapt to the influence of various factors of a supply chain environment, can avoid emergencies such as extreme and abnormal conditions, adopts a time difference error mode to estimate the optimal base line, increase entropy rewards, improve the convergence speed and the return limit, ensures the stability of strategy updating, and achieves the target of long-term cost reduction and efficiency increase of the supply chain.

Description

Technical Field

[0001] The present invention belongs to the field of supply chain inventory, and in particular relates to a multi-level inventory cost optimization method for a supply chain distribution network. Background Art

[0002] The efficiency and cost control of supply chain management are key factors for enterprises to maintain their sustained growth and market competitiveness. In particular, for complex supply chain systems involving multiple production and distribution levels, the optimization and control of inventory costs are particularly important. The multi-level inventory system covers all links from raw material procurement, processing and manufacturing to final product distribution. The inventory management efficiency at each stage directly affects the response speed, operating costs and customer satisfaction of the entire supply chain. However, the diversification of supply chain distribution network levels, inventory components and control strategies has led to the traditional supply chain multi-level inventory control still facing many problems including multi-level inventory coordination, inventory costs and capital occupation, information asymmetry, etc. For such a complex system as the supply chain distribution network, effectively solving the above problems requires comprehensive consideration of the overall characteristics of the supply chain and the mutual influence between each link, and adopting appropriate strategies and tools for scientific inventory planning and management.

[0003] Compared with the traditional management method of a single inventory point, multi-level inventory control aims to optimize the inventory strategy of the entire supply chain, covering the entire process from suppliers to terminal retailers. Clark et al. pioneered the definition of the concept of "multi-level inventory", which has stimulated the interest of researchers in this field. For example, Hsin Ran et al. proposed a three-level inventory system and established a corresponding model; Ganeshan et al. studied a system with multiple suppliers and retailers and proposed a model based on continuous inventory. For inventory problems, the framework provided by the Markov decision process (MDP) can be used, which provides a mathematical basis for sequential decision problems and has been widely used in inventory optimization. For example, Voelke et al. improved value iteration to adapt to small-scale inventory decision problems; Lin Jie et al. explored optimization strategies based on component degradation status through value iteration, while Xu Changkai et al. used policy iteration to solve the aviation material supply problem. When the state transition probability of MDP is unknown, reinforcement learning becomes a feasible solution. For example, Giannoccaro and Pontrandolfo used semi-Markov reward technology to optimize supply chain inventory decisions; Chaharsooghi et al. optimized multi-intelligence supply chain inventory management through Q-learning algorithm; Jiang, Kara et al. used reinforcement learning algorithm to explore dynamic inventory control and proved its effectiveness in inventory management. The above inventory optimization methods are suitable for small-scale inventory systems. When the state space grows exponentially with the state variable, the emergence of deep reinforcement learning (DRL) solves the large-scale MDP problem. DRL uses deep neural networks to represent value or policy functions, overcoming the limitations of traditional algorithms. Moor et al. used deep Q network (DQN) to improve the digital curse question; Oroojlooyjadid et al. proposed a deep Q network (SRDQN) with reward feedback value to deal with the game problem of beer game; Vanvuchelen et al. used the proximal policy optimization algorithm to introduce the joint replenishment problem, while Gijsbrechts et al. applied the A3C (Asynchronous Advantage ActorCritic, A3C) algorithm to improve the efficiency of multiple supplier models, sales loss models and multi-level inventory models.

[0004] The efficiency and cost control of supply chain management are key factors for enterprises to maintain their sustained growth and market competitiveness. Although DRL has shown promise in inventory strategy optimization, its application in multi-level inventory with large-scale and complex action spaces still faces challenges. In particular, for complex supply chain systems involving multiple production and distribution levels, the optimization and control of inventory costs are particularly important. Among them, the multi-level inventory system covers all links from raw material procurement, processing and manufacturing to final product distribution. The inventory management efficiency at each stage directly affects the convergence speed, operating costs, stability and customer satisfaction of the entire supply chain. Summary of the invention

[0005] The present invention provides a method for optimizing multi-level inventory cost in a supply chain distribution network, so as to solve the problems of slow convergence speed and poor updating stability of multi-level inventory cost optimization in existing supply chain distribution network.

[0006] In order to solve the above technical problems, the technical solution provided by the present invention is: The present invention relates to a method for optimizing multi-level inventory costs in a supply chain distribution network, which comprises the following steps: S1. Obtain the initial information of the multi-level inventory cost optimization problem of the whole process supply chain distribution network; S2. Establish a multi-level inventory cost model based on the initialization information and analyze the multi-level inventory cost components of the cost model; S3. Determine the objective function of the multi-level inventory cost optimization problem of the whole process supply chain distribution network based on cost components; S4. Design a proximal policy optimization algorithm based on the objective function, design the state space, action space and reward function mechanism for the proximal policy optimization algorithm, introduce the clipping function as a constraint, select the optimization function to estimate the baseline to improve the policy gradient algorithm, and form an improved proximal policy optimization algorithm; S5. Based on the objective function, the core parameters of the improved proximal strategy optimization algorithm are optimized and adaptively adjusted to obtain the optimal multi-level inventory cost control strategy.

[0007] Preferably, the specific steps of designing and improving the proximal strategy optimization algorithm in S4 include: S4.1. Design the state space, action space, and reward function for the proximal policy optimization algorithm, using a tuple ( s t , a t , r t , s t+1 ) indicates that, s t represents the set of environmental states at time t, a t represents the action at time t, s t+1 Indicates action a t Causes the environment state to change to a new state, r t Represents the reward received as the environment state changes at time t; S4.2. Construct the policy network and value network of the proximal policy optimization algorithm, and design the policy gradient calculation formula of the proximal policy optimization algorithm, specifically: (1), In the formula, θ represents the neural network parameters, s Indicates the state of the environment. a Indicates action, E （s,a） represents the calculation of expected value, old represents the old policy, θ old represents the neural network parameters of the old policy, π represents the probability distribution, ▽ π represents the gradient of the probability distribution, A represents the advantage function, ▽ R represents the gradient of reward expectation, π θ ( a ∣ s ) indicates that in the state s Select action for the current strategy a The probability of logπ θ represents the logarithmic value of the action probability, A θold ( s , a ) represents the advantage function, which is used to evaluate the s Next action a Relative advantage compared to the average behavioral strategy; Among them, the target algorithm calculation formula of the neural network parameter θ is: (2), In the formula, L Represents the neural network parameters θ Function of S4.3. The clipping operation is introduced to constrain the neural network parameters. The target algorithm calculation formula of the neural network parameter θ after the constraint is: (3), In the formula, represents the calculation of the expected value, and the distribution is determined by the probability distribution of the old policy neural network parameters, that is, for all states under the old policy s and actions a Take the weighted average, clip Represents the clipping function, the value range of the clipping function is: (4), in, ε Indicates parameters related to cropping range, ε ∈[0.1,0.3]; S4.4. Use the advantage function to estimate the baseline method to improve the policy gradient algorithm. Specifically, the time difference error is used to estimate the optimal baseline. The calculation formula of the time difference error is: (5), In the formula, A πθ ( s , a ) indicates that in the current strategy πθ Next, status s and actions a The advantage function is used to measure the difference between the effect of taking action a in state s and the average performance of the current strategy; R πθ ( s , a ) is to adopt a strategy π θ Time Status s Take action a The reward you get immediately after V ( s ) is the status s The expected state value when taking the average strategy under t+1 ) is the expected value of the next state, γ is the discount factor, and its value range is [0,1]; S4.5. Add entropy reward on the basis of the target algorithm of the neural network parameter θ after the constraint. The calculation formula of the target algorithm of the neural network parameter θ after the entropy reward is added is: (6), In the formula, L Fast-PPO represents the comprehensive objective function, L CLIP ( θ ) represents the pruned policy loss function, L VF ( θ ) represents the loss of the value function, S πθ represents the entropy of the policy, c 1 and c 2 are weight coefficients, E ( s , a ) represents the calculation of the expected value, that is, the state s and actions a The weighted average of .

[0008] Preferably, the environmental state set at time t in S4.1 is s tIncluding inventory level, in-transit inventory, demand forecast, order status, logistics status, supply status and market factors, environmental status set s t It is expressed as: (7), in, H t p for t Time slot provider p The inventory volume, H t i for t Time slot maker i The inventory volume, H t m for t Time Distributor m The inventory volume, H t n for t Daytime Retailer n The inventory volume, T t pi for t Time slot provider p To Manufacturer i of in-transit inventory, T t im for t Time slot maker i To distributors m of in-transit inventory, T t mn for t Time Distributor m To retailers n of in-transit inventory, O t for t Product demand forecast for the time period, Q t p for t Time slot provider p The order status, Q t i for t Time slot maker i The order status, Q t n for t Daytime Retailern The order status, W t p for t Time slot provider p The logistics status, W t i for t The logistics status of manufacturer i in time period, W t m for t Time Distributor m The logistics status, S t p for t Time slot provider p Supply status, S t i is the supply status of manufacturer i in period t, S t m is the supply status of distributor m at time t, S t n is the supply status of retailer n in period t, M t is the market trend during period t; The action at time t in S4.1 a t It includes inventory reallocation, ordering decisions, replenishment strategies, price adjustments, and transportation scheduling measures at each node in a multi-level supply chain, expressed as: (8), Among them, △ C t express t Inventory allocation between warehouses in different time periods, △ Q t represents the order quantity of each warehouse or distribution center in the multi-level supply chain during period t, R t express t Replenishment strategy for each node in the time period, △ P t Indicates setting or adjusting the selling price of a product, Tr t It means deciding the mode, route and schedule of transportation of the product; The expression of the reward function in S4.1 is: (9), in, C cvd Represents the comprehensive value loss, J OSR represents the service level deviation penalty,J IS represents the inventory shift penalty, F represents the order fulfillment rate, I R and I D Represents actual inventory and target inventory respectively. R ic represents the inventory turnover rate, R ia Indicates the inventory accuracy rate, CE represents inventory cost efficiency, TO , TC represents the total output and total cost of products in any period, α , β , ω 1. ω 2 represents the weight coefficient.

[0009] Preferably, the step of optimizing and adaptively adjusting the core parameters of the improved proximal strategy optimization algorithm in S5 includes: S5.1. Initialize the environment state and input it into the policy network, output the probability distribution of the action, select the largest action and execute it, obtain the reward and the next state, and store them in a tuple ( s t , a t , r t , s t+1 ), repeat this step until the tuple is full; S5.2. Input the state set and action set in the tuple into the value network, output the advantage function, calculate the discounted reward based on the loss function Critic_loss, and update the parameters of the value network. The calculation formula of the advantage function is: (10), in, Q πθ ( s t , a t ) is the action value function, which means that when taking the strategy π θ Time Status s t Take action a t the value of V ( s t , ω t) is the state value function, which means the evaluator parameter in the value network is ω t Time Status s t value expectations; S5.3. Input the state set and action set in the tuple into the new and old strategy networks respectively to obtain the corresponding probability distribution π θnew and π θold , update the neural network parameters according to formula (3) θ The target algorithm is used to update the parameters of the policy network. θ , while cleaning up the tuples; S5.4. Repeat the process of S5.3 until one of the following conditions is met, and then synchronize the data of the new policy network to the old policy network. The conditions include: (1) The update amplitude of the new strategy network is less than the threshold; (2) Strategy performance in continuous k There is no significant improvement in the iterations, that is, the improvement is less than the set threshold δ ; (3) The maximum number of repetitions reaches the set number.

[0010] Preferably, the calculation formulas for updating the parameters of the value network in S5.2 and the strategy network in S5.3 are respectively: (11), in, ω t , θ t are the parameters of the value network and strategy network in period t, δ t is the timing difference error value in period t, α ω , α θ are the parameter update steps of the value network and the policy network respectively, R t for t Instant rewards for the time period, γ is the discount factor, γ Between 0 and 1, V ( s , ω ) represents the state value function, that is, in a given state s Under this circumstance, the expected value of the cumulative reward that the agent is expected to obtain after executing the current strategy is ∇ V ( s , ω) represents the value function V ( s , ω ) for the parameters ω The gradient of is used to guide the parameter update of the value network. π θt ( a t ∣ s t )express t Always in the state s The probability of selecting action a under the current strategy.

[0011] Preferably, the multi-level inventory cost components in S2 include inventory ordering costs, holding costs and out-of-stock costs.

[0012] Preferably, the ordering cost C o The expression is: (12), in, ζ represents a fixed transaction fee, O is the purchase cost of a unit of goods, Q o is the order quantity, g , l They represent the unit distance transportation cost and transportation distance respectively; Inventory holding costs as stated C h The expression is: (13), in, h Represents the inventory holding cost per unit of goods, Ii Indicates the inventory quantity; The cost of stockouts C s The expression is: (14), in, s represents the out-of-stock cost per unit of commodity, Q s Indicates the out-of-stock quantity.

[0013] Preferably, the decision variables of the objective function in S3 include: supplier p Supply point v p ,supplier p Supply batch V p , the production site of manufacturer i x i , Manufactureri Production batch X i , Distributors m Order point y m , Distributors m Order quantity Y m , Retailers n Order point z n , Retailers n Order quantity Z n ; The expression of the objective function is: (15), in, p , i , m , n Indicates the supplier index number, manufacturer index number, distributor index number, and retailer index number, h Represents the inventory cost per unit of goods, I Indicates the storage capacity. G represents the transportation cost coefficient, h Represents the inventory cost per unit of goods, ζ represents a fixed transaction fee, W Indicates the number of orders. O Indicates the unit price of the ordered goods. Q Indicates the order quantity of the product. L pi Indicates supplier p Transportation distance to manufacturer i, L im Indicates the manufacturer i To distributors m The transport distance, L mn Indicates distributor m To retailers n The transport distance, T Indicates the number of times the product is transported. S represents the unit cost of out-of-stock loss of goods, Q s Indicates out-of-stock quantity.

[0014] Preferably, the objective function also satisfies the following constraints: The production batch of manufacturer i does not exceed the maximum production capacity of manufacturer i per unit time period, which is expressed as: (16), in, X iIndicates the manufacturer i Production batch, e i Indicates the manufacturer's maximum production capacity per unit time period; In the ordering relationship between upper and lower level enterprises, the order quantity of the lower level to the upper level enterprise does not exceed the storage quantity of the upper level, and does not exceed its own maximum inventory. The expression is: (17), in, Q k represents the order quantity of the k-th level enterprise to the previous level enterprise, Q k ≥0 and is a positive integer, I j Indicates j Storage capacity of enterprise k The first-level enterprise j The upper level enterprise of a level enterprise.

[0015] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: 1. The multi-level inventory cost optimization method for supply chain distribution network involved in the present invention designs a state space and an action space for the proximal policy optimization algorithm when designing a proximal policy optimization algorithm, introduces a clipping function as a constraint, and selects a method of estimating a baseline of an optimization function to improve the policy gradient algorithm, thereby improving the proximal policy optimization algorithm. The improved proximal policy optimization algorithm can continuously learn so that inventory demand can continuously adapt to the influence of various factors such as seasonal changes, market trends, and supply chain fluctuations in the supply chain environment, thereby improving the convergence speed and return limit. At the same time, by limiting the strategy update amplitude and maintaining the stability of the learning process, it helps to avoid performance collapse caused by extreme or abnormal situations, and can handle emergencies more reliably. Compared with the one-way and local inventory management method, the improved proximal policy optimization algorithm significantly achieves the goal of reducing costs and increasing efficiency for enterprises.

[0016] 2. The multi-level inventory cost optimization method for the supply chain distribution network involved in the present invention not only designs a proximal strategy optimization algorithm, but also designs a reward mechanism for the proximal strategy optimization algorithm to make decisions by optimizing long-term rewards, which helps to maximize long-term benefits in multi-level inventory management. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a framework diagram of multi-level inventory cost optimization of a supply chain distribution network based on an improved proximal strategy optimization algorithm of the present invention; Figure 2 This is a schematic diagram of the training process of improving the proximal strategy optimization algorithm of the present invention.

[0018] Figure 3Training parameters for different networks γ Reward curve under .

[0019] Figure 4 Reward curves for agents trained with different algorithms.

[0020] Figure 5 This is the fitness curve of the comparative experiment. DETAILED DESCRIPTION

[0021] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with examples. The following examples are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0022] The present invention relates to a method for optimizing multi-level inventory costs in a supply chain distribution network, which comprises the following steps: S1. Obtain the initial information of the multi-level inventory cost optimization problem of the whole process supply chain distribution network, including: the demand of suppliers, manufacturers, distributors and retailers; the warehouse inventory and in-transit inventory level of each level of the supply chain; the unit price, total order quantity and total out-of-stock quantity of goods at each level of the supply chain; the transportation distance, transportation frequency and transportation unit price between each level of the supply chain distribution network.

[0023] S2. Establish a multi-level inventory cost model based on the initialization information and analyze the multi-level inventory cost components of the cost model, including inventory ordering cost, holding cost and out-of-stock cost: The inventory ordering costs mentioned above are mainly composed of fixed transaction costs, purchase costs and transportation costs. Among them, fixed transaction costs are related to the number of orders and are not affected by the order quantity; purchase costs are directly related to the order quantity. Ordering costs refer to the sum of ordering costs and transportation costs. Ordering costs C o The expression is: (12), in, ζ represents a fixed transaction fee, O is the purchase cost of a unit of goods, Q o is the order quantity, g , l They represent the unit distance transportation cost and transportation distance respectively; The inventory holding cost mentioned above is mainly composed of inventory opportunity cost, inventory service cost and storage space cost. The inventory holding cost refers to the unit inventory holding cost multiplied by the inventory volume. C h The expression is: (13), in, hRepresents the inventory holding cost per unit of goods, Ii Indicates the inventory quantity; The out-of-stock costs mentioned above are mainly composed of lost potential sales revenue, downtime losses, delivery delays, and additional costs caused by taking emergency replenishment measures. The out-of-stock cost refers to the unit out-of-stock cost multiplied by the out-of-stock quantity. C s The expression is: (14), in, s represents the out-of-stock cost per unit of commodity, Q s Indicates the out-of-stock quantity.

[0024] S3. Determine the objective function of the multi-level inventory cost optimization problem of the whole process supply chain distribution network based on the cost components. The decision variables of the objective function include: suppliers p Supply point v p ,supplier p Supply batch V p , Manufacturer i Production point x i , Manufacturer i Production batch X i , Distributors m Order point y m , Distributors m Order quantity Y m , Retailers n Order point z n , Retailers n Order quantity Z n ; The expression of the objective function is: (15), in, p , i , m , n Indicates the supplier index number, manufacturer index number, distributor index number, and retailer index number, h Represents the inventory cost per unit of goods, I Indicates the storage capacity. G represents the transportation cost coefficient, h Represents the inventory cost per unit of goods, ζ represents a fixed transaction fee, W Indicates the number of orders.O Indicates the unit price of the ordered goods. Q Indicates the order quantity of the product. L pi Indicates supplier p To Manufacturer i The transport distance, L im Indicates the manufacturer i To distributors m The transport distance, L mn Indicates distributor m To retailers n The transport distance, T Indicates the number of times the product is transported. S represents the unit cost of out-of-stock loss of goods, Q s Indicates out-of-stock quantity; The above objective function also satisfies the following constraints: In any unit production cycle, the manufacturer i The production batch does not exceed the manufacturer i The maximum production capacity per unit time period is expressed as: (16), in, X i Indicates the manufacturer i Production batch, e i Indicates the manufacturer's maximum production capacity per unit time period; In the ordering relationship between upper and lower level enterprises, the order quantity of the lower level to the upper level enterprise should not exceed the storage quantity of the upper level, and should not exceed its own maximum inventory. The expression is: (17), in, Q k Indicates k The order quantity of the first-level enterprise to the upper-level enterprise, Q k ≥0 and is a positive integer, I j Indicates j Storage capacity of enterprise k The first-level enterprise j The upper level enterprise of a level enterprise.

[0025] S4. Design a proximal policy optimization algorithm. Design the state space, action space and reward function mechanism for the proximal policy optimization algorithm. Introduce the clipping function as a constraint. Select the optimization function to estimate the baseline to improve the policy gradient algorithm. Form an improved proximal policy optimization algorithm. The specific steps include: S4.1. The process of optimizing multi-level inventory costs with proximal strategies can usually be described as a Markov decision process (MDP). MDP consists of a state space (State, S), an action space (Action Space, A), and a reward function (Reward Function, R). First, design the state space, action space, and reward function for the proximal strategy optimization algorithm, using a tuple ( s t , a t , r t , s t+1 ) indicates that, s t express t The set of environmental states at a given moment, a t express t Action at all times, s t+1 Indicates action a t Causes the environment state to change to a new state, r t Indicates that with t The environment state at the moment is converted to the reward received, which means: at the time point t , facing the current state of the environment s t , the agent performs an action a t As part of its strategy, this move a t Causes the environment state to change to a new state s t+1 , as the environment changes state, the agent receives a reward r t , as direct feedback on the actions they take; t The set of environmental states at a given moment s t Including inventory level, in-transit inventory, demand forecast, order status, logistics status, supply status and market factors, environmental status set s t It is expressed as: (7), in, H t p for t Time slot provider p The inventory volume, H t i for t Time slot maker i The inventory volume, H t m for t Time Distributor m The inventory volume, H t n for t Daytime Retailer n The inventory volume, T t pi for t Time slot provider p To Manufacturer i of in-transit inventory, T t im for t Time slot maker i To distributors m of in-transit inventory, T t mn for t Time Distributor m To retailers n of in-transit inventory, O t for t Product demand forecast for the time period, Q t p for t Time slot provider p The order status, Q t i for t Time slot maker i The order status, Q t n for t Daytime Retailer n The order status, W t p for t Time slot provider p The logistics status, W ti for t Time slot maker i The logistics status, W t m for t Time Distributor m The logistics status, S t p for t Time slot provider p Supply status, S t i for t Time slot maker i Supply status, S t m for t Time Distributor m Supply status, S t n for t Daytime Retailer n Supply status, M t for t Market trends during the time period; t Action at all times a t It includes inventory reallocation, ordering decisions, replenishment strategies, price adjustments, and transportation scheduling measures at each node in a multi-level supply chain, expressed as: (8), Among them, △ C t express t Inventory allocation between warehouses in different time periods, △ Q t express t The order quantity of each warehouse or distribution center in the multi-stage supply chain during the period, △ R t express t Replenishment strategy for each node in the time period, △ P t Indicates setting or adjusting the selling price of a product, Tr t It means deciding the mode, route and schedule of transportation of the product; The comprehension value depletion (CVD) that defines the multi-echelon inventory cost minimization problem in the supply chain consists of two parts: the total direct loss cost and the total indirect loss cost. The total direct loss cost includes procurement cost, holding cost, transportation cost, and out-of-stock cost; the total indirect loss cost includes the cost of service level loss, operational efficiency loss, and market competitiveness loss. Therefore, the expression of the reward function is: (9), in, C cvd Represents the comprehensive value loss, J OSR represents the service level deviation penalty, J IS represents the inventory shift penalty, F represents the order fulfillment rate, I R and I D Represents actual inventory and target inventory respectively. R ic represents the inventory turnover rate, R ia Indicates the inventory accuracy rate, CE represents inventory cost efficiency, TO , TC represents the total output and total cost of products in any period, α , β , ω 1. ω 2 represents the weight coefficient.

[0026] S4.2. Construct the policy network and value network of the proximal policy optimization algorithm. The core of the proximal policy optimization algorithm is the Actor-Critic architecture, in which the Actor Network is called the policy network, including the policy network to be optimized Actors ( θ new ) and fixed policy network Actors ( θ old ) ; Critic Network is the value network Q ( s , a ). This algorithm uses the policy gradient boosting method to find the optimal neural network parameters θ , in order to maximize the cumulative reward value obtained by the agent in the environment, the strategy gradient calculation formula is: (1), In the formula, θ represents the neural network parameters,s Indicates the state of the environment. a Indicates action, E （s,a） represents the calculation of expected value, old represents the old policy, θ old represents the neural network parameters of the old policy, π represents the probability distribution, ▽ π represents the gradient of the probability distribution, A represents the advantage function, ▽ R represents the gradient of reward expectation, π θ (a|s) means in the state s The current policy (represented by the parameter θ) selects an action a The probability of logπ θ Represents the logarithmic value of the action probability πθ(a|s), A θold (s, a) represents the advantage function, which is used to evaluate the s Next action a Relative advantage compared to the average behavioral strategy; Among them, the neural network parameters θ The target algorithm calculation formula is: (2), In the formula, L Represents the neural network parameters θ Function of S4.3. To ensure the stability of training, the present invention introduces a clipping operation to constrain the neural network parameters, that is, the ratio of the new strategy to the old strategy is used as a constraint to ensure that the new strategy is generated near the old strategy. After the constraint, the neural network parameters θ The target algorithm calculation formula is: (3), In the formula, represents the calculation of the expected value, and its distribution is determined by the probability distribution πθold of the old policy neural network parameters, that is, for all states under the old policy s and actions a Take the weighted average, clip Represents the clipping function, which is used to limit the network update range. The value range of the clipping function is: (4), in, ε Indicates parameters related to cropping range, ε ∈[0.1,0.3], the range of the clipping function can be expressed as: θ and θold The difference is too large, π θ ( a | s ) / π θold ( a | s ) is greater than 1+ ε or less than 1- ε When , the clipping function takes the value 1+ ε or 1- ε ,constraint π θ / π θold The value is in [1- ε ,1+ ε ]between.

[0027] S4.4. Based on the proximal policy optimization algorithm, the advantage function is used to estimate the baseline to improve the policy gradient algorithm, thereby improving the convergence speed and reward limit while ensuring the stability of the policy update. Specifically, the time difference error is used to estimate the optimal baseline. The calculation formula of the time difference error is: (5), In the formula, A πθ ( s , a ) means that under the current strategy πθ, the state s and actions a The advantage function is used to measure the s Take action a The difference between the performance of the strategy and the average performance of the current strategy; R πθ ( s , a ) is to adopt a strategy π θ Time Status s Take action a The reward you get immediately after V ( s ) is the status s The expected state value when taking the average strategy is: V ( s t+1 ) is the expected value of the next state, γ is the discount factor, with a value range of [0,1]; S4.5. Neural Network Parameters after Constraint θThe entropy reward is added on the basis of the target algorithm to reduce the difficulty of operation by using parameter sharing in the policy network and the value network. θ The calculation formula of the target algorithm is: (6), In the formula, L Fast-PPO It represents the comprehensive objective function, which is the final objective function when optimizing the neural network parameter θ. This function comprehensively considers the stability of strategy update, the accuracy of value function and the exploratory nature of strategy. L CLIP ( θ ) represents the pruned policy loss function, which is used to limit the magnitude of policy updates. L VF ( θ ) represents the loss of the value function, S πθ represents the entropy of the policy, c 1 and c 2 are weight coefficients, E (s,a) represents the calculation of the expected value, and the distribution is determined by the current strategy πθ, which is the weighted average of state s and action a; The framework diagram of multi-level inventory cost optimization of the supply chain distribution network based on the improved proximal strategy optimization algorithm according to the above steps is as follows: Figure 1 shown.

[0028] S5. Optimize and adaptively adjust the core parameters of the improved proximal strategy optimization algorithm to obtain the optimal multi-level inventory cost control strategy. The specific steps are as shown in the attached Figure 2 As shown, including: S5.1. Initialize the environment state s And input into the strategy network to be optimized Actors ( θ new ), output the probability distribution of actions, and select the largest action a t And execute, get instant rewards r t and the next state s t+1 , store it in a tuple ( s t , a t , r t , s t+1 ), repeat this step until the tuple is full; S5.2. Set the state in the tuple s ={s 1, s 2,… s n} and action collection a ={ a 1, a 2,… a n} Input into the value network Critic Network and output the advantage function A πθ ( s , a ), and then calculate the discounted reward based on the loss function Critic_loss, and update the parameters of the value network. The calculation formula of the advantage function is: (10), in, Q πθ ( s t , a t ) is the action value function, which means that when taking the strategy π θ Time Status s t Take action a t the value of V ( s t , ω t ) is the state value function, which means the evaluator parameter in the value network is ω t Time Status s t value expectations; S5.3. Set the state in the tuple s ={ s 1, s 2,… s n} and action collection a ={ a 1, a 2,… a n} are input into the new and old strategy networks respectively to obtain the corresponding probability distribution π θnew and π θold , update the neural network parameters according to formula (3) θ The target algorithm is used to update the parameters of the policy network. θ , while cleaning up the tuples; The calculation formulas for updating the parameters of the value network in S5.2 and the strategy network in S5.3 are: (11), in, ω t , θ t They are t Parameters of the value network and strategy network under time period, δ t for t The time series difference error value under the period, α ω 、α θ are the parameter update steps of the value network and the policy network respectively, R t for t Instant rewards for the time period, γ is the discount factor, γ Between 0 and 1, V(s,ω) Represents the state value function (Value Function), that is, in a given state s Under this circumstance, the expected value of the cumulative reward that the agent is expected to obtain after executing the current strategy is ∇ V ( s,ω ) represents the value function V ( s,ω ) The gradient of the parameter ω is used to guide the parameter update of the value network; S5.4. Repeat the process of S5.3 until one of the following conditions is met, and then synchronize the data of the new policy network to the old policy network. The conditions include: (1) The update amplitude of the new strategy network is less than the threshold; (2) Strategy performance in continuous k There is no significant improvement in the iterations, that is, the improvement is less than the set threshold δ ; (3) The maximum number of repetitions reaches the set number.

[0029] Synchronize the data of the new policy network to the old policy network, and the agent outputs the network parameter results, which is the final multi-level inventory management strategy, including: Multi-echelon supply chain status space information: inventory levels, inventory in transit, demand forecasts, order status, logistics status, supply status and market factors; Information on the action space of multi-echelon supply chains: inventory reallocation, ordering decisions, replenishment strategies, price adjustments, and transportation scheduling measures at each level.

[0030] Effect example In order to verify the performance of the improved proximal strategy optimization algorithm in solving the multi-level inventory cost optimization problem of the supply chain, a case study was set up for evaluation. The supply chain system of a certain type of fast-moving consumer goods company in a certain province or city was taken as the research object. The system has 5 suppliers, 1 manufacturer, 9 distributors and 25 retailers. The properties of each level involved in the supply chain of the present invention are described as shown in Table 1-1 below: Table 1-1 Schematic diagram of each level of the supply chain system .

[0031] 1. Data preparation Supply chain simulation uses discrete event models to simulate various activities and events in the supply chain, such as order generation, production completion, and transportation arrival. In addition to simulating the operation of the supply chain system by recording state changes at discrete time points, the model also takes into account the randomness of discrete event models, resource allocation and utilization, performance measurement and analysis, decision evaluation, and strategy optimization. Table 1-2 shows the simulation parameters of the multi-level inventory optimization system of the supply chain, mainly including the environment-related parameters of each node in the supply chain and the parameters related to the product itself. Table 1-3 shows the relevant parameters of the model of each node in the supply chain. Table 1-4 shows the demand between each level of the supply chain.

[0032] Table 1-2 Simulation parameters of supply chain multi-level inventory optimization system .

[0033] Table 1-3 Relevant parameters of each node model of the supply chain .

[0034] Table 1-4 Forecast demand table (partial) Related companies Forecasted demand (units) Manufacturer (processing plant) -> Supplier (grower) 01 760122 Manufacturer (processing plant) -> Supplier (grower) 02 1290283 Manufacturer (processing plant) -> Supplier (material production company) 03 67920123 Manufacturer (processing plant) -> Supplier (material supplier) 04 5378749 Manufacturer (processing plant) -> Supplier (material supplier) 05 5320023 Distributor (Large Distribution Company) 01->Manufacturer (Processing Plant) 15360213 Distributor (Large Distribution Company) 02->Manufacturer (Processing Plant) 14279002 Distributor (Large Distribution Company) 03->Manufacturer (Processing Plant) 16789200 Distributor (regional dealer) 04->Manufacturer (processing plant) 4902893 Distributor (regional dealer) 05->Manufacturer (processing plant) 3684920 Distributor (regional dealer) 06->Manufacturer (processing plant) 5425630 Distributor (regional dealer) 07->Manufacturer (processing plant) 2438900 Distributor (regional dealer) 08->Manufacturer (processing plant) 5890024 Distributor (regional dealer) 09->Manufacturer (processing plant) 2229944 Retailer (specialty store) 01->Distributor (with order transaction) 5818924 Retailer (specialty store) 02->Distributor (with order transaction) 4900935 Retailer (specialty store) 03->Distributor (with order transaction) 6438720 Retailer (specialty store) 04->Distributor (with order transaction) 5190563 Retailer (specialty store) 05->Distributor (with order transaction) 4039228 Retailer (convenience store) 06->Distributor (with order transaction) 728900 Retailer (convenience store) 07->Distributor (with order transaction) 598382 Retailer (convenience store) 08->Distributor (with order transaction) 299325 Retailer (convenience store) 09->Distributor (with order transaction) 738200 Retailer (convenience store) 10->Distributor (with order transaction) 342543 Retailer (convenience store) 11->Distributor (with order transaction) 456680 Retailer (convenience store) 12->Distributor (with order transaction) 903234 Retailer (convenience store) 13->Distributor (with order transaction) 893214 Retailer (convenience store) 14->Distributor (with order transaction) 903524 Retailer (convenience store) 15->Distributor (with order transaction) 355523 Retailer (convenience store) 16->Distributor (with order transaction) 598300 Retailer (convenience store) 17->Distributor (with order transaction) 429340 Retailer (supermarket counter) 18->Distributor (with order transaction) 4028322 Retailer (supermarket counter) 19->Distributor (with order transaction) 4789209 Retailer (supermarket counter) 20->Distributor (with order transaction) 3782900 Retailer (supermarket counter) 21->Distributor (with order transaction) 5637280 Retailer (supermarket counter) 22->Distributor (with order transaction) 2752993 Retailer (supermarket counter) 23->Distributor (with order transaction) 6073528 Retailer (supermarket counter) 24->Distributor (with order transaction) 4572630 Retailer (supermarket counter) 25->Distributor (with order transaction) 5728329 .

[0035] Parameter setting: Table 2-1 shows the relevant parameters for algorithm training, different network training parameters γ The training goal of the model of the present invention is to enable the agent to obtain the maximum reward value. If the model is trained so that the agent's reward value oscillates around a certain value with a low amplitude, it means that the algorithm has good convergence.

[0036] Table 2-1 Related parameters of algorithm training .

[0037] Attached Figure 3 Training parameters for different networks γ The reward value changes with the increase of training cycles. γ=0.05, the reward value first increases sharply and then rises slowly as the training cycle increases, and basically stops rising when the training cycle is 360, but the reward value is always negative; when γ =0.5, the reward value increases with the increase of training cycles, but decreases significantly when the training cycle is 500; γ =0.99, the reward value increases rapidly with the increase of training cycle, and basically stabilizes when the training cycle is 160. γ =0.99, the convergence speed is the fastest, and the network training effect of the improved proximal strategy optimization algorithm is the best. In the next network training process, the discount factor is set to 0.99.

[0038] Results and Analysis Substitute the above-mentioned parameters and prepared data into the improved proximal strategy optimization algorithm, and implement the algorithm in this paper through Python3.6.0 to obtain the company's multi-level inventory management strategy.

[0039] In order to verify the advantages of the improved proximal strategy optimization algorithm proposed in this paper and minimize the multi-level inventory cost of the supply chain, Figure 4 The agent training reward curves are given for the traditional Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Improved Proximal Policy Optimization. For Improved Proximal Policy Optimization, the agent converges after about 120 training times, about 62,400 environmental interactions (520 interactions with the environment per training), PPO converges after about 160 training times, about 83,200 environmental interactions, and DDPG converges after about 400 training times, about 208,000 environmental interactions.

[0040] According to the attached Figure 4 The results show that the training times of the improved proximal strategy optimization algorithm are reduced by 25% and 70% respectively compared with those of the PPO algorithm and the DDPG algorithm, which significantly improves the training efficiency of the multi-level inventory cost minimization model.

[0041] According to the attached Figure 5It can be seen from the results that the improved proximal strategy optimization algorithm proposed in the present invention has better adaptability than the comparison algorithm, that is, the convergence speed is increased by 15.38% and 21.43% compared with the PPO algorithm and the DDPG algorithm, respectively. At the same time, the total inventory cost calculated by the multi-level inventory cost optimization objective function is 72,094,521.43 yuan, which is 18.23% and 24.08% lower than that of the PPO algorithm and the DDPG algorithm, respectively. And the total cost of supply chain operation of the enterprise according to the original inventory ordering strategy is 89,025,321.38 yuan, and the cost is reduced by 19.01%. Therefore, the multi-level inventory cost optimization model established in this paper can effectively reduce the total operating cost of the supply chain, thereby illustrating the feasibility of this algorithm.

[0042] The present invention is described in detail above in conjunction with the embodiments, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for optimizing multi-level inventory costs in a supply chain distribution network, characterized by: It includes the following steps: S1. Obtain the initial information of the multi-level inventory cost optimization problem of the whole process supply chain distribution network; S2. Establish a multi-level inventory cost model based on the initialization information and analyze the multi-level inventory cost components of the cost model; S3. Determine the objective function of the multi-level inventory cost optimization problem of the whole process supply chain distribution network based on cost components; S4. Design a proximal policy optimization algorithm based on the objective function, design the state space, action space and reward function mechanism for the proximal policy optimization algorithm, introduce the clipping function as a constraint, select the optimization function to estimate the baseline to improve the policy gradient algorithm, and form an improved proximal policy optimization algorithm; S5. Based on the objective function, the core parameters of the improved proximal strategy optimization algorithm are optimized and adaptively adjusted to obtain the optimal multi-level inventory cost control strategy.

2. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 1, characterized in that: The specific steps of designing and improving the proximal strategy optimization algorithm in S4 include: S4.

1. Design the state space, action space, and reward function for the proximal policy optimization algorithm, using a tuple ( s t , a t , r t , s t+1 ) indicates that, s t represents the set of environmental states at time t, a t represents the action at time t, s t+1 Indicates action a t Causes the environment state to change to a new state, r t Represents the reward received as the environment state changes at time t; S4.

2. Construct the policy network and value network of the proximal policy optimization algorithm, and design the policy gradient calculation formula of the proximal policy optimization algorithm, specifically: （1）， In the formula, θ represents the neural network parameters, s Indicates the state of the environment. a Indicates action, E （s,a） represents the calculation of expected value, old represents the old policy, θ old represents the neural network parameters of the old policy, π represents the probability distribution, ▽ π represents the gradient of the probability distribution, A represents the advantage function, ▽ R represents the gradient of reward expectation, π θ ( a ∣ s ) indicates that in the state s The probability of selecting action a under the current strategy, logπ θ represents the logarithmic value of the action probability, A θold (s, a) represents the advantage function, which is used to evaluate the relative advantage of executing action a in state s compared to the average behavior strategy; Among them, the target algorithm calculation formula of the neural network parameter θ is: （2）， In the formula, L Represents the neural network parameters θ Function of S4.

3. The clipping operation is introduced to constrain the neural network parameters. The target algorithm calculation formula of the neural network parameter θ after the constraint is: （3）， In the formula, represents the calculation of the expected value, and the distribution is determined by the probability distribution of the old strategy neural network parameters, that is, the weighted average of all states s and actions a under the old strategy, clip Represents the clipping function, the value range of the clipping function is: （4）， in, ε Indicates parameters related to cropping range, ε ∈[0.1,0.3]; S4.

4. Use the advantage function to estimate the baseline method to improve the policy gradient algorithm. Specifically, the time difference error is used to estimate the optimal baseline. The calculation formula of the time difference error is: （5）， In the formula, A πθ (s,a) means in the current strategy π θ Next, status s and actions a The advantage function is used to measure the s Take action a The difference between the performance of the strategy and the average performance of the current strategy; R πθ ( s , a ) is to adopt a strategy π θ Time Status s Take action a The reward you get immediately after V ( s ) is the status s The expected state value when taking the average strategy is: V ( s t+1 ) is the expected value of the next state, γ is the discount factor, with a value range of [0,1]; S4.

5. Add entropy reward on the basis of the target algorithm of the neural network parameter θ after the constraint. The calculation formula of the target algorithm of the neural network parameter θ after the entropy reward is added is: （6）， In the formula, L Fast-PPO represents the comprehensive objective function, L CLIP ( θ ) represents the pruned policy loss function, L VF ( θ ) represents the loss of the value function, S πθ represents the entropy of the policy, c 1 and c 2 are weight coefficients, E ( s , a ) represents the calculation of the expected value, that is, the state s and actions a The weighted average of .

3. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 2, characterized in that: The environmental state set at time t in S4.1 s t Including inventory level, in-transit inventory, demand forecast, order status, logistics status, supply status and market factors, environmental status set s t It is expressed as: （7）， in, H t p for t Time slot provider p The inventory volume, H t i for t The inventory of manufacturer i in the period, H t m for t Time Distributor m The inventory volume, H t n for t Daytime Retailer n The inventory volume, T t pi for t Time slot provider p To Manufacturer i of in-transit inventory, T t im for t Time slot maker i To distributors m of in-transit inventory, T t mn for t Time Distributor m To retailers n of in-transit inventory, O t for t Product demand forecast for the time period, Q t p for t Time slot provider p The order status, Q t i for t Time slot maker i The order status, Q t n for t Daytime Retailer n The order status, W t p for t Time slot provider p The logistics status, W t i for t Time slot maker i The logistics status, W t m for t Time Distributor m The logistics status, S t p for t Time slot provider p Supply status, S t i is the supply status of manufacturer i in period t, S t m is the supply status of distributor m at time t, S t n is the supply status of retailer n in period t, M t is the market trend during period t; The action at time t in S4.1 a t It includes inventory reallocation, ordering decisions, replenishment strategies, price adjustments, and transportation scheduling measures at each node in a multi-level supply chain, expressed as: （8）， Among them, △ C t express t Inventory allocation between warehouses in different time periods, △ Q t express t The order quantity of each warehouse or distribution center in the multi-stage supply chain during the period, △ R t represents the replenishment strategy for each node in time period t, P t Indicates setting or adjusting the selling price of a product, Tr t It means deciding the mode, route and schedule of transportation of the product; The expression of the reward function in S4.1 is: （9）， in, C cvd Represents the comprehensive value loss, J OSR represents the service level deviation penalty, J IS represents the inventory shift penalty, F represents the order fulfillment rate, I R and I D Represents actual inventory and target inventory respectively. R ic represents the inventory turnover rate, R ia Indicates the inventory accuracy rate, CE represents inventory cost efficiency, TO , TC represents the total output and total cost of products in any period, α , β , ω 1. ω 2 represents the weight coefficient.

4. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 2, characterized in that: The step of optimizing and adaptively adjusting the core parameters of the improved proximal strategy optimization algorithm in S5 includes: S5.

1. Initialize the environment state and input it into the policy network, output the probability distribution of the action, select the largest action and execute it, obtain the reward and the next state, and store them in a tuple ( s t , a t , r t , s t+1 ), repeat this step until the tuple is full; S5.

2. Input the state set and action set in the tuple into the value network, output the advantage function, calculate the discounted reward based on the loss function Critic_loss, and update the parameters of the value network. The calculation formula of the advantage function is: （10）， in, Q πθ ( s t , a t ) is the action value function, which means that when taking the strategy π θ Time Status s t Take action a t the value of V ( s t , ω t ) is the state value function, which means the evaluator parameter in the value network is ω t Time Status s t value expectations; S5.

3. Input the state set and action set in the tuple into the new and old strategy networks respectively to obtain the corresponding probability distribution π θnew and π θold ， Update the neural network parameters according to formula (3) θ The target algorithm is used to update the parameters of the policy network. θ , while cleaning up the tuples; S5.

4. Repeat the process of S5.3 until one of the following conditions is met, and then synchronize the data of the new policy network to the old policy network. The conditions include: (1) The update amplitude of the new strategy network is less than the threshold; (2) Strategy performance in continuous k There is no significant improvement in the iterations, that is, the improvement is less than the set threshold δ ; (3) The maximum number of repetitions reaches the set number.

5. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 4, characterized in that: The calculation formulas for updating the parameters of the value network in S5.2 and the strategy network in S5.3 are respectively: （11）， in, ω t , θ t They are t Parameters of the value network and strategy network under time period, δ t is the timing difference error value in period t, α ω , α θ are the parameter update steps of the value network and the policy network respectively, R t is the instant reward in period t, γ is the discount factor, γ Between 0 and 1, V ( s , ω ) represents the state value function, that is, in a given state s Under this circumstance, the expected value of the cumulative reward that the agent is expected to obtain after executing the current strategy is ∇ V ( s , ω ) represents the value function V ( s , ω ) for parameters ω The gradient of is used to guide the parameter update of the value network, π θt (a t ∣s t ) represents the probability of the current strategy selecting action a in state s at time t.

6. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 1, characterized in that: The multi-level inventory cost components in S2 include inventory ordering costs, holding costs and stock-out costs.

7. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 6, characterized in that: Ordering costs stated C o The expression is: （12）， Among them, ζ represents the fixed transaction fee, O is the purchase cost of a unit of goods, Q o is the order quantity, g , l They represent the unit distance transportation cost and transportation distance respectively; Inventory holding costs as stated C h The expression is: （13）， in, h Represents the inventory holding cost per unit of goods, Ii Indicates the inventory quantity; The cost of stockouts C s The expression is: （14）， in, s represents the out-of-stock cost per unit of commodity, Q s Indicates the out-of-stock quantity.

8. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 7, characterized in that: The decision variables of the objective function in S3 include: supplier p Supply point v p , the supply batch of supplier p V p , the production site of manufacturer i x i , production batch of manufacturer i X i , Distributors m Order point y m , Distributors m Order quantity Y m , Retailers n Order point z n , the order quantity of retailer n Z n ; The expression of the objective function is: （15）， in, p , i , m , n Indicates the supplier index number, manufacturer index number, distributor index number, and retailer index number, h Represents the inventory cost per unit of goods, I Indicates the storage capacity. G represents the transportation cost coefficient, h Represents the inventory cost per unit of goods, ζ represents a fixed transaction fee, W Indicates the number of orders. O Indicates the unit price of the ordered goods. Q Indicates the order quantity of the product. L pi Indicates supplier p Transportation distance to manufacturer i, L im represents the transportation distance from manufacturer i to distributor m, L mn Indicates distributor m To retailers n The transport distance, T Indicates the number of times the product is transported. S represents the unit cost of out-of-stock loss of goods, Q s Indicates out-of-stock quantity.

9. The method for optimizing multi-level inventory costs in a supply chain distribution network according to claim 8, characterized in that: The objective function also satisfies the following constraints: Manufacturer i The production batch does not exceed the manufacturer i The maximum production capacity per unit time period is expressed as: （16）， in, X i Indicates the manufacturer i Production batch, e i Indicates the manufacturer's maximum production capacity per unit time period; In the ordering relationship between upper and lower level enterprises, the order quantity of the lower level to the upper level enterprise does not exceed the storage quantity of the upper level, and does not exceed its own maximum inventory. The expression is: （17）， in, Q k represents the order quantity of the k-th level enterprise to the previous level enterprise, Q k ≥0 and is a positive integer, I j Indicates j Storage capacity of enterprise k The j-th level enterprise is the upper level enterprise of the j-th level enterprise.

Images