Data-driven omni-channel retailer inventory replenishment and distribution optimization algorithm

Abstract

The invention discloses a data-driven omni-channel retailer inventory replenishment and distribution optimization algorithm, particularly relates to the technical field of retail inventory management, and aims at efficiently solving the problems of multi-SKU inventory management, demand uncertainty, inventory transfer cost and the like by adopting a PPO algorithm in deep reinforcement learning to optimize inventory replenishment and distribution decision. And automatic and intelligent inventory management is realized in actual operation.

Description

Technical Field

[0001] The present invention relates to the technical field of retail inventory management, and particularly to a data-driven inventory replenishment and allocation optimization algorithm for omnichannel retailers. Background Art

[0002] With the rapid development of the e-commerce and retail industries, more and more retailers have adopted the omnichannel retail model, combining online and offline channels to improve the shopping experience of consumers and increase sales efficiency. In this model, retailers not only need to manage the inventory in physical stores (sales floors), but also need to handle the distribution of online orders. How to effectively manage the inventory online and offline and coordinate the replenishment and allocation between the two has become a major challenge for retailers.

[0003] Existing inventory management methods usually rely on rule-based inventory control strategies, such as the Periodic Review Policy or the Continuous Review Policy. These methods may be effective in simple inventory management scenarios, but in complex omnichannel retail environments, they face many problems. For example, traditional inventory management methods usually assume that demand is known and do not consider the uncertainty and variability of demand, which makes inventory management prone to problems such as inventory backlogs, stockouts, or over-replenishment when facing a dynamically changing market environment. In addition, traditional methods lack flexibility and cannot adjust inventory allocation decisions in real time, especially in terms of inventory flow between the sales floor and the back warehouse, making it difficult to effectively reduce inventory transfer costs or handle the balance between inventory shortages and surpluses.

[0004] To address these issues, some studies have attempted to introduce intelligent inventory management techniques, among which Machine Learning and Reinforcement Learning have become research hotspots in inventory management. Reinforcement Learning has received particular attention because it can continuously optimize the decision-making process through interaction with the environment and can handle the uncertainty and dynamics of demand. Through Reinforcement Learning, retailers can adaptively adjust replenishment decisions and optimize the inventory allocation ratio during the inventory management process, thereby reducing operating costs and improving inventory turnover. However, existing Reinforcement Learning methods still have certain limitations in practical applications. For example, most existing Reinforcement Learning algorithms assume that the demand distribution of each product is known, and the training process relies on accurately modeling the product demand. However, in the actual retail environment, product demand is usually complex and dynamic, affected by multiple factors (such as price, promotion, seasonality, weather, etc.), so it is extremely difficult to accurately predict the demand distribution of products. In addition, traditional deep Reinforcement Learning methods usually require a large amount of historical data during training, and training each individual SKU (Stock Keeping Unit) brings a huge computational burden.

[0005] Therefore, in response to the requirements of omnichannel retail inventory management, there is an urgent need for a new method based on Deep Reinforcement Learning (DRL) that can adaptively optimize replenishment and inventory allocation decisions and effectively address the challenges of multi-SKU, demand uncertainty, and multi-channel inventory management. In addition, existing training methods usually suffer from the problem of overfitting to a single product, and there is a lack of solutions that can be efficiently trained in large-scale SKU management. Summary of the Invention

[0006] For this reason, the present invention provides a data-driven omnichannel retailer inventory replenishment and allocation optimization algorithm, which optimizes inventory replenishment and allocation decisions through the Proximal Policy Optimization (PPO) algorithm. The system utilizes the feedback information of the global policy network and multi-SKU sub-environments, combines the data of multiple SKUs during the training process, and performs large-scale optimization through a deep neural network, overcoming the problems of inaccurate demand prediction, high computational complexity, and lack of flexibility existing in existing methods. Through the present invention, retailers can achieve efficient inventory management, reduce inventory costs, improve inventory turnover, and at the same time better meet customer needs and enhance the shopping experience of consumers.

[0007] To achieve the above objectives, the present invention provides the following technical solutions: A data-driven omnichannel retailer inventory replenishment and allocation optimization algorithm, including:

[0008] Step 1: Based on historical demand data and product characteristics, construct an inventory management environment for omnichannel retailers, including historical demand data for offline and online sales channels, data on external characteristics affecting demand, current inventory status, and in-transit order data;

[0009] Step 2: Based on the Markov decision process, transform the omnichannel inventory management problem into a joint optimization problem of inventory replenishment and allocation and model it; among them, the state S t represents the current environmental information; the actions include the replenishment decision o t and the allocation decision k t ;

[0010] Step 3: Use a deep reinforcement learning algorithm based on proximal policy optimization to train the Actor network and the Critic network;

[0011] Actor network: According to the current environmental information S t output the probability distribution of the inventory replenishment decision o t and the allocation decision k t ;

[0012] Critic network: Used to evaluate the long-term reward V t starting from the current environmental information S π (S t );

[0013] Step 4: Based on the policy of the trained Actor network, generate the optimal replenishment quantity o′ t and the allocation ratio k′ t by jointly optimizing the replenishment o t and the allocation decision k t in each decision period and update the inventory status;

[0014] Step 5: Collect data from the sales history sub-environments of multiple SKUs, and these data are used for batch sampling to train a multi-SKU shared policy neural network to overcome the problem of insufficient historical data for a single SKU.

[0015] Preferably, in Step 1, the historical demand data sets for offline and online channels wherein, and respectively represent the demand of the omnichannel retailer for offline sales inventory and online sales inventory at time t;

[0016] The external characteristic data set affecting demand wherein, and They respectively represent the influencing factors of the omnichannel retailer for offline sales inventory and online sales inventory within time t, mainly including price, discount, advertisement, holiday, day of the week, etc.

[0017] Current inventory status and the on-order O t = o t-L , …, o t-1 ; Among them, and respectively represent the inventory levels of the omnichannel retailer for offline sales inventory and online sales inventory within time t.

[0018] Preferably, the Markov decision process described in step 2 is designed as follows:

[0019] State S t represents the current environmental information;

[0020] The actions include:

[0021] Replenishment decision o t : The quantity of goods to be ordered from the distribution center in period t;

[0022] Allocation decision k t : The allocation ratio of the incoming orders between the sales area and the back warehouse in period t;

[0023] The reward function is defined as:

[0024] r t = p t * min(y t , x t ) - c t ;

[0025] Among them, c t represents the operating cost, including inventory holding cost, backorder penalty cost, inventory allocation cost and product loss cost, and x t is the available inventory.

[0026] Preferably, in step 3, the training process of the Actor network and the Critic network includes:

[0027] Initialize the weights Θ of the Actor network and the weights Φ of the Critic network;

[0028] Sample data in multiple SKU sub - environments, and record the state, action, reward and next state;

[0029] Use the PPO algorithm to update the weights of the Actor network and the Critic network, and optimize the policy loss function and the value loss function.

[0030] Preferably, the inventory status update described in step 4:

[0031]

[0032] where η represents the inventory loss rate.

[0033] The present invention has the following advantages:

[0034] By adopting the PPO algorithm in deep reinforcement learning to optimize inventory replenishment and allocation decisions, the present invention can efficiently handle problems such as multi-SKU inventory management, demand uncertainty, and inventory transfer costs, and achieve automated and intelligent inventory management in actual operations. Compared with the prior art, this algorithm can handle complex decision optimization tasks in high-dimensional state and action spaces and realize the automation and intelligence of inventory management in actual retail environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the omnichannel retail inventory update and allocation provided by the present invention.

[0036] Figure 2 Graph of the comparison of the benefits of the method (DRL) of the present invention and the existing method (optimization after prediction).

[0037] Figure 3 Graph of the comparison of the costs of the method (DRL) of the present invention and the existing method (optimization after prediction). DETAILED DESCRIPTION OF THE INVENTION

[0038] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0039] As Figure 1 shown, the present invention provides a data-driven omnichannel retailer inventory replenishment and allocation optimization algorithm, including:

[0040] Step 1: Initialize the input data, receive and process the basic data: Based on the historical demand data and product characteristics, construct the inventory management environment of the omnichannel retailer, including the historical demand data of offline and online sales channels, the data of external characteristics affecting demand, the current inventory status, and the in-transit order data;

[0041] Historical demand data sets for offline and online channels including the demand for the sales floor and the back warehouse; where and respectively represent the demand of the omni-channel retailer for offline sales inventory and online sales inventory within time t;

[0042] External feature dataset affecting demand Among them, and respectively represent the influencing factors of the omni-channel retailer for offline sales inventory and online sales inventory within time t, such as factors that may affect demand, such as price, promotion, weather, etc.

[0043] Current inventory status represents the inventory in the sales floor and the back warehouse; among them, and respectively represent the inventory levels of the omni-channel retailer for offline sales inventory and online sales inventory within time t. Outstanding order O t = o t-L , …, o t-1 , that is, the outstanding orders from the online channel;

[0044] Step 2: Based on the Markov decision process, transform the omni-channel inventory management problem into a joint optimization problem of inventory replenishment and allocation and model it; among them, state S t represents the current environmental information; actions include replenishment decision o t and allocation decision k t ;

[0045] The Markov decision process is designed as follows:

[0046] State S t represents the current environmental information;

[0047] Actions include:

[0048] Replenishment decision o t : The quantity of goods to be ordered from the distribution center in period t;

[0049] Allocation decision k t : The allocation ratio of the incoming orders in the sales area and the back warehouse in period t;

[0050] The reward function is defined as:

[0051] r t = p t * min(y t , x t ) - c t ;

[0052] Among them, c t represents the operating cost, including inventory holding cost, backorder penalty cost, inventory allocation cost and product loss cost, xt Is available inventory.

[0053] Step 3: Use the deep reinforcement learning algorithm based on proximal policy optimization to train the Actor network and the Critic network;

[0054] Actor network: According to the current environmental information S t Output the probability distribution of the inventory replenishment decision o t and the allocation decision k t ;

[0055] Critic network: Used to evaluate the long-term reward V t starting from the current environmental information S π (S t );

[0056] In Step 3, the training process of the Actor network and the Critic network includes:

[0057] Initialize the weights Θ of the Actor network and the weights Φ of the Critic network;

[0058] Sampling: Sample data in multiple SKU sub-environments, and record the state S t , the action (replenishment quantity o′ t and the allocation ratio k′ t ), the reward r t ) and the next state S t+1 ;

[0059] Advantage estimation: Calculate the advantage value A based on the value estimation of each state by the evaluation network t :

[0060]

[0061] where γ is the discount factor, r t is the immediate reward, and S t is the current state;

[0062] Use the PPO algorithm to update the weights of the Actor network and the Critic network, and optimize the policy loss function and the value loss function.

[0063] Update the parameters of the action network and the evaluation network by calculating and minimizing the following loss functions: Policy loss function:

[0064] L clip (θ k ) = ∑ i,t min(u it A it , clip(u it, 1 - ∈, 1 + ∈)A it );

[0065] Among them, is the ratio of the likelihood value of o it , κ it appearing under the current k-th step policy during the training process to the likelihood value under the previous period's policy, and ∈ is the clipping parameter used to prevent over-updating.

[0066] Value loss function:

[0067]

[0068] Among them, is the future return under the current policy.

[0069] Entropy loss function:

[0070]

[0071] It is used to encourage the policy to explore new action options and prevent the policy from converging prematurely.

[0072] Step 4: Based on the trained Actor network policy, by jointly optimizing the replenishment o t and the allocation decision k t generate the optimal replenishment quantity o′ t and the allocation ratio k′ t in each decision cycle and update the inventory status;

[0073] Update inventory status:

[0074]

[0075] Among them, η represents the inventory loss rate.

[0076] Step 5: Collect data from the sales history sub-environments of multiple SKUs. These data are used for batch sampling to train a policy neural network shared by multiple SKUs to overcome the problem of insufficient historical data for a single SKU.

[0077] Step 6: Real-time decision-making application

[0078] Real-time input data: According to real-time inventory, demand, and order data, the system automatically generates the replenishment quantity and the allocation ratio and makes real-time inventory replenishment and allocation decisions.

[0079] Decision output: Through the trained global policy network, generate the optimal inventory replenishment and allocation decisions and apply them to actual inventory management to optimize inventory configuration and improve operational efficiency.

[0080] The inventory replenishment and allocation optimization model in step 2 includes the following sub-steps:

[0081] State definition: The environmental state of each SKU (defined as including historical demand data, current inventory, environmental characteristics (such as price, promotion, etc.), and in-transit order information;

[0082] Action space: The action space includes the replenishment quantity o′ t and the allocation ratio k′ t , where the replenishment quantity o′ t is obtained by sampling from the Gaussian distribution output by the policy network, and the allocation ratio k′ t is transformed through the sigmoid function;

[0083] Reward design: The reward function considers multiple factors such as inventory shortage, inventory surplus, transfer cost, etc., and the reward is calculated by the following formula:

[0084]

[0085] where c h is the holding cost, c s is the shortage penalty, η is the spoilage rate, c f2b is the cost of transferring inventory from the sales floor to the back warehouse, (x t -y t ) + represents the positive part when there is inventory surplus.

[0086] To better illustrate the present invention, in combination with actual applications, the system for optimizing inventory replenishment and allocation decisions based on Deep Reinforcement Learning (DRL) and its training method will be described in detail below.

[0087] 1. System architecture

[0088] Based on the above content, this embodiment provides an inventory replenishment and allocation optimization system mainly composed of the following modules:

[0089] Environment module (Sub-environment): Each SKU (stock keeping unit) has an independent sub-environment, which is responsible for processing inventory, demand, and order information related to that SKU. Each sub-environment feeds back the execution effect of the replenishment and allocation decisions through interaction with the deep reinforcement learning module.

[0090] Deep reinforcement learning module: It includes an Actor Network and a Critic Network, which are used to generate decisions on replenishment quantity and allocation ratio and evaluate the effect of this decision. Specifically:

[0091] The action network generates the replenishment quantity o′ according to the environmental state t and the allocation ratio k′ t ;

[0092] The evaluation network evaluates the value function of the current policy i.e., the expected discounted reward after executing the current policy

[0093] Training module: Responsible for model training through deep reinforcement learning, optimizing the parameters of the action network and the evaluation network to enable effective replenishment and allocation decisions

[0094] Global policy network: By integrating the feedback information of multiple SKU sub-environments, a global policy network is trained to ensure that the system can make optimized inventory management decisions for multiple SKUs

[0095] 2. Data flow and information transfer

[0096] The flow and processing of data in the system are as follows

[0097] Input data: Each SKU sub-environment receives the following inputs

[0098] Historical demand data Y t , including past online and offline channel demands

[0099] Environmental characteristics Z t , such as factors that may affect demand, such as price, promotion, weather, etc

[0100] Inventory data I t , the current inventory levels in the front and back-end warehouses of sales

[0101] In-transit order data O t , the in-transit orders

[0102] Replenishment and allocation decision generation

[0103] The action network generates the replenishment quantity o′ through the state S t and the allocation ratio k′ t : t

[0104]

[0105] where σ represents the sigmoid function to ensure that the value is within the range of (0, 1), and s is the current state

[0106] Reward calculation and feedback

[0107] The evaluation network estimates the value function of the current policy and calculates it through the Bellman equation​

[0108]

[0109] where r t is the immediate reward, γ is the discount factor, and S t+1 is the next state.

[0110] Optimized decision-making:

[0111] Based on the calculated rewards and advantage values, update the weights of the action network and the evaluation network, and optimize the decisions on the replenishment quantity and allocation ratio.

[0112] 3. Training process

[0113] The training process of the present invention is based on the Proximal Policy Optimization (PPO) algorithm in deep reinforcement learning, and the specific process is as follows:

[0114] Step 1: Data sampling

[0115] Sample actions (replenishment quantity (o′ t ) and allocation ratio (k′ t )) from the sub-environment of each SKU, and record the state S t , reward r t , and the next state S t+1 .

[0116] Step 2: Advantage value calculation

[0117] Calculate the advantage value A based on the value estimation of each state by the critic network t :

[0118]

[0119] Step 3: Policy optimization

[0120] Use Mini-batch Gradient Descent to optimize the weights of the actor network and the critic network, and minimize the following loss functions:

[0121] Policy loss function:

[0122]

[0123] where ∈ is the clipping parameter.

[0124] Value loss function:

[0125]

[0126] Entropy loss function:

[0127]

[0128] Step 4: Global policy network training

[0129] Aggregate feedback information from all SKUs and generate globally optimal inventory replenishment and allocation decisions through global policy network training.

[0130] 4. Inference and application

[0131] After training is completed, the system can use the trained global policy network to make real-time decisions and apply them to actual inventory management. The specific process is as follows:

[0132] Step 1: Real-time data input

[0133] Based on real-time inventory data, demand data, and order data, the system calculates the replenishment quantity o′ t and allocation ratio k′ t for each current SKU and generates an optimal inventory configuration decision.

[0134] Step 2: Decision output

[0135] Apply the decisions generated by the global policy network in real time to achieve automated inventory replenishment and allocation, optimize inventory configuration, improve inventory turnover rate, and reduce inventory shortages and surpluses.

[0136] 5. Actual deployment

[0137] During actual deployment, the system will dynamically adjust the replenishment and allocation strategies according to the historical demand data and environmental characteristic information of each SKU through the trained deep reinforcement learning model to ensure that the inventory is always in an optimal state. At the same time, the system can handle large-scale SKU management and has good scalability and computational efficiency.

[0138] 6. Empirical results

[0139] Select two benchmark models in the current inventory control field: the Base-stock and VecBase strategies for comparison. Randomly select 600 days of online and offline sales data of 122 SKUs in 5 omnichannel retail stores to train the models, and use 28 days of data as the test set for verification.

[0140] From Figure 2 it can be found that when the lead time is greater than zero, DRL achieves significantly higher profits than the two benchmark strategies. In Figure 3 the cost structures of different strategies in the experiment are further decomposed, and when the lead time is greater than zero, DRL always achieves a lower total cost than the benchmark strategies.

[0141] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made thereto based on the present invention, which will be obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of the present invention claimed.

Claims

1. A data-driven omni-channel retailer inventory replenishment and allocation optimization algorithm, characterized by: include: Step 1: Build an inventory management environment for omni-channel retailers based on historical demand data and product characteristics, including historical demand data for offline and online sales channels, data on external characteristics that affect demand, current inventory status, and in-transit order data; Step 2: Based on the Markov decision process, the omni-channel inventory management problem is transformed into a joint optimization problem of inventory replenishment and allocation and modeled; where state S t Represents the current environment information; actions include replenishment decisions o t and the allocation decision k t ; Step 3: Use the deep reinforcement learning algorithm based on proximal policy optimization to train the Actor network and the Critic network; Actor network: Based on the current environment information S t Output inventory replenishment decision t and the allocation decision k t The probability distribution of Critic network: used to evaluate the current strategy from the current environment information S t Starting long-term reward V π (S t ); Step 4: Based on the trained Actor network strategy, replenishment is optimized through joint optimization t and the allocation decision k t Generate the optimal replenishment quantity o′ in each decision cycle t and the distribution ratio k′ t And update the inventory status; Step 5: Collect data from the sales history sub-environments of multiple SKUs. These data are batch sampled and used to train a multi-SKU shared strategy neural network to overcome the problem of insufficient historical data of a single SKU.

2. A data-driven omni-channel retailer inventory replenishment and allocation optimization algorithm according to claim 1, characterized in that: In step 1, the historical demand dataset for offline and online channels in, and They represent the demand of omni-channel retailers for offline sales inventory and online sales inventory within time t respectively; External feature datasets that affect demand in, and They represent the influencing factors of omni-channel retailers’ offline sales inventory and online sales inventory within time t; Current stock status and orders in transit t =o t-L ,…,o t-1 ;in, and They represent the inventory of offline sales inventory and online sales inventory of omni-channel retailers within time t respectively.

3. The data-driven omni-channel retailer inventory replenishment and allocation optimization algorithm according to claim 1, characterized in that: The Markov decision process described in step 2 is designed as follows: Status S t Indicates the current environment information; Actions include: Replenishment Decision t : The quantity of goods that need to be ordered from the distribution center in period t; Allocation decision k t : The distribution ratio of the arrival orders in the sales area and the back warehouse in cycle t; The reward function is defined as: r t =p t *min(y t ,x t )-c t ; Among them, c t represents the operating cost, including inventory holding cost, out-of-stock penalty cost, inventory allocation cost and product loss cost, x t For available stock.

4. The data-driven omni-channel retailer inventory replenishment and allocation optimization algorithm according to claim 1, characterized in that: In step 3, the training process of the Actor network and the Critic network includes: Initialize the Actor network weight Θ and the Critic network weight Φ; Sample data in multiple SKU sub-environments, record status, action, reward and next status; Use the PPO algorithm to update the weights of the Actor network and Critic network and optimize the strategy loss function and value loss function.

5. The data-driven omni-channel retailer inventory replenishment and allocation optimization algorithm according to claim 1, characterized in that: Update inventory status as described in step 4: Among them, η represents the inventory loss rate.

Images