Intelligent replenishment decision method based on reinforcement learning and imitation learning and storage medium

Abstract

This invention relates to an intelligent replenishment decision-making method and storage medium based on reinforcement learning and imitation learning. It includes an MDP design module, an imitation learning modeling module, and an intelligent decision-making module. The imitation learning modeling module includes a behavior cloning module and an adversarial imitation learning module. The intelligent decision-making module includes a basic strategy and an exploration strategy. This invention combines behavior cloning and adversarial imitation learning. Behavior cloning addresses simple business scenarios, enabling the agent's replenishment decision-making ability to reach the average level of experts. Adversarial imitation learning constructs an adversarial reward function to address complex business scenarios, surpassing expert decision-making levels. This invention effectively solves the problems of high dependence on input data, low model stability, and weak generalization ability in existing replenishment systems, and avoids black-box prediction, possessing better commercialization and business expansion capabilities.

Description

Technical Field

[0001] This invention relates to the field of data prediction, specifically to an intelligent replenishment decision-making method and storage medium based on reinforcement learning and imitation learning. Background Technology

[0002] Intelligent replenishment can effectively improve operational indicators. Existing replenishment models include Chinese patent publication number CN113095745A (Replenishment Decision Model Training and Replenishment Decision Method, System, Equipment and Medium) for end-to-end neural network models, Chinese patent publication number CN110348635A (Intelligent Replenishment Method, Storage Medium, System and Device Based on End-to-End Learning) for end-to-end operations planning models, and Chinese patent publication number CN106971249A (A Sales Forecasting and Replenishment Method) for step-by-step models including sales forecasting and replenishment. These methods have the following problems: high dependence on data quality and volume, leading to weak model generalization ability. Simultaneously, high data dependence increases the difficulty of model training and is accompanied by frequent model decay, such as... Figure 1 , Figure 2 As shown, this increases the cost of model iteration. Furthermore, for business personnel, the interpretability of model replenishment decisions is also reduced.

[0003] One improvement to this problem is reinforcement learning algorithms. Reinforcement learning emphasizes how to act based on the environment to maximize expected benefits. Its focus is on finding a balance between exploration (of the unknown) and utilization (of existing knowledge). Basic reinforcement learning is modeled as a Markov decision process, which includes the following elements: state space S, action space A, reward function R, policy, and rules P (transition probability matrix) for state transitions. Markov decision processes effectively reduce the algorithm's dependence on data quality and volume. This invention focuses on using imitation learning algorithms within reinforcement learning. Imitation learning is generally considered to have two main categories: behavioral cloning and adversarial imitation learning. Imitation learning is a method that enables intelligent agents to make intelligent decisions like human experts. On the road to general artificial intelligence, it has been found difficult to manually program agents to think, as this involves a significant amount of artificial engineering. For example, teaching a vehicle to drive autonomously requires considering numerous constraints (safe driving to avoid accidents, smooth driving to increase comfort, etc.), and designing specific supervisory signals to guide the agent in response to these constraints is a challenging task. Conversely, humans can accomplish these tasks relatively easily and provide agents with a wealth of exemplary behaviors. Using these expert examples to teach agents to make intelligent decisions is the primary problem that imitation learning addresses.

[0004] Behavior cloning algorithms attempt to minimize the action differences between the agent's policy and the expert's policy, typically reducing behavior cloning tasks to common regression or classification tasks. Adversarial imitation learning algorithms, on the other hand, construct an adversarial reward function through inverse reinforcement learning (IRL), and then maximize this reward function to guide the agent in generating replenishment policies.

[0005] Considering the bottlenecks of existing big data deep learning technologies and the shortcomings of single replenishment strategies, a new reinforcement learning replenishment technology based on behavioral cloning and adversarial imitation learning, which can be flexibly adjusted, needs to be designed. Summary of the Invention

[0006] The purpose of this application is to address and overcome the shortcomings of existing technologies and applications, providing an intelligent replenishment decision-making method and storage medium based on reinforcement learning and imitation learning. This effectively solves the problems of high dependence on input data, low model stability, weak generalization ability, difficulty in commercial implementation, and limited business scenario expansion capabilities. This invention employs imitation learning, resulting in high model stability, low complexity, low data dependence, strong model generalization ability, and low model decay maintenance costs. It avoids the black-box prediction of existing technologies, abandons blind assumptions, and possesses better commercial implementation and business expansion capabilities.

[0007] The purpose of this application is achieved through the following technical solution: an intelligent replenishment decision-making method based on reinforcement learning and imitation learning, comprising the following steps:

[0008] S1. Abstract the replenishment business scenario through the MDP design module, which specifically includes state space design, action space design, reward function design and strategy design.

[0009] S2. Based on the MDP design module described in S1, construct the imitation learning modeling module, which specifically includes a behavior cloning module and an adversarial imitation learning module.

[0010] S3. Train using the Q-Learning algorithm to obtain the optimal replenishment strategy based on the current data;

[0011] S4. Using the strategy trained by the algorithm described in S3, make intelligent decisions for replenishment scenarios, and continuously interact with the environment through the Dagger algorithm to generate more data and optimize the model;

[0012] Specifically, step S1 includes:

[0013] S11. State space design describes the agent's perception of replenishment business scenario information and its dynamic changes;

[0014] S12. Action space design describes the operations that the agent can perform, and determines whether to perform a replenishment action on the goods and the optimal replenishment quantity of the goods based on step S11;

[0015] S13. The reward function design transforms customer expectations and task objectives into quantifiable reward functions to guide the training of reinforcement learning algorithms;

[0016] S14. The strategy design is based on the state space design, action space design and reward function design. Depending on whether the strategy optimization objective is singular, it can be divided into simple strategies and compound strategies, which use single reward functions and compound reward functions respectively.

[0017] Step S2 specifically includes:

[0018] S21. Based on whether the reward function is known, it is divided into a behavior cloning module with a known reward function and an adversarial imitation learning module with an unknown reward function;

[0019] S22. For simple business scenarios where the reward function is known, the known reward function is directly used to guide the training of the reinforcement learning algorithm, and the trained policy is used to guide the agent to generate a replenishment policy. This process is called behavior cloning.

[0020] S23. For complex business scenarios where the reward function is unknown, the reward function is fitted by inverse reinforcement learning. The fitted reward function is used to guide the training of the reinforcement learning algorithm, and the trained strategy is used to guide the agent to generate a replenishment strategy. This process is adversarial imitation learning.

[0021] S24. Step S23 specifically uses inverse reinforcement learning to fit the composite reward function, and the method includes, but is not limited to, linear regression.

[0022] Step S4 specifically includes:

[0023] S41. Based on the replenishment strategy trained in S3, continuously interact with the environment, constantly collect expert examples, and generate new data;

[0024] S42. On the augmented new dataset, repeat the steps described in S1-S3 to retrain the model.

[0025] S43. Continuously optimize the model through an iterative framework of “training-decision-retraining-re-decision”.

[0026] Preferably, the MDP design module includes state space design, action space design, reward function design, and strategy design.

[0027] Preferably, the agent in step S12 determines whether to replenish stock and the optimal replenishment quantity using the following formula:

[0028]

[0029] In the formula: the theoretical optimal replenishment quantity for each product per day includes the following two possibilities:

[0030] The first method: When the actual inventory of a product on a given day is lower than the trigger inventory of the product, the product is replenished. The optimal replenishment quantity is the difference between the product's "expected inventory" and its "actual inventory".

[0031] The second scenario: When the actual inventory of a product on a given day is greater than or equal to the product's triggered inventory level, the product's inventory level is sufficient to meet daily sales needs, and therefore no replenishment is required.

[0032] At the same time, the model also considers replenishment frequency and selects high-frequency or low-frequency replenishment strategies based on the attributes of various products to ensure the long-term rationality of replenishment behavior.

[0033] Preferably, in step S24, inverse reinforcement learning is used to fit a composite reward function, the formula of which is as follows:

[0034]

[0035] in, R i Let α represent the i-th reward function. i This represents the weighting coefficient corresponding to the i-th reward function, with the weighting coefficients summing to 100%. N indicates that the composite reward function comprehensively considers N reward functions; the specific number of N is determined by business requirements. For example,

[0036] The composite reward function = 0.6 * maximize display fullness rate + 0.4 * minimize scrap rate. This composite reward function takes into account both the maximization of display fullness rate and the minimization of scrap rate, and has a greater preference for maximizing display fullness rate (α1 = 0.6 > α2 = 0.4).

[0037] Preferably, step S3 specifically includes:

[0038] S31. Initialize t = 0;

[0039] S32. Start from the initial state S0;

[0040] S33. At time t, select a replenishment action A based on the Q value. t =argmax a∈A Q(S t a) Here, the ε-greedy method is used;

[0041] S34, Execute Action A t Then, it enters the next state S. t+1(New inventory levels, scrap rates, display rates, etc.), and obtain the reward R through the above composite reward function. t+1 ;

[0042] S35, Q-value update function:

[0043] Q(S t A t )←Q(S t A t )+α(R t+1 +γmax a∈A Q(S t+1 ,a)-Q(S t A t ))

[0044] S36. Set t = t + 1, then return to S33.

[0045] Preferably, step S41 specifically involves: performing imitation learning based on the data of the day to obtain a replenishment strategy;

[0046] S42 specifically involves: interacting with the environment based on the current replenishment strategy, specifically through the ε-greedy algorithm: The ε-greedy algorithm is used to balance exploitation and exploration. This algorithm performs random exploration with a probability of ε and takes the optimal replenishment action with a probability of 1-ε, where ε is a small probability number between 0 and 1, generating a new sequence. The sequence consists of alternating states, actions, and reward values.

[0047] S t A t ,R t ,S t+1 A t+1 ,R t+1 ,…,S t+M A t+M ,R t+M

[0048] Where M is the number of steps explored by the algorithm;

[0049] Specifically, S43 is as follows: The model generates replenishment behavior through the ε-greedy algorithm, obtains an augmented dataset based on business indicators after replenishment or new expert examples collected, and then returns to S41.

[0050] The present invention also provides a computer-readable storage medium storing a computer program that can be executed by a computer processor to implement the intelligent replenishment decision-making method based on reinforcement learning and imitation learning as described in any one of the preceding claims.

[0051] Compared with the prior art, this application has at least the following obvious advantages and effects:

[0052] 1. From the perspective of model stability: This invention is based on the imitation learning architecture, which has less dependence on the quality and volume of input data. At the same time, the model can adjust faster to deal with sudden drift problems and can quickly adapt to business changes. Compared with deep neural network models, it takes less time and has a stronger ability to deal with model decay problems. This invention combines behavioral cloning and inverse reinforcement learning, which can improve the model's weak generalization and strong generalization capabilities at the same time.

[0053] 2. From the perspective of model complexity: This invention can clone expert replenishment behavior by simply performing imitation learning without a clear loss function, which is easy to train; it abandons the multi-layer architecture of deep neural networks and innovatively uses reinforcement learning - imitation learning, which requires less data for training and is easy to train; it uses an explicit MDP design (state space, action space, reward function) to iterate quickly and at low cost in the complex and ever-changing business world.

[0054] 3. From the perspective of interpretability of replenishment decisions: Traditional deep neural networks use historical sales as input and predict sales through multiple hidden layers, resulting in low overall interpretability. This invention, however, features a more granular explicit MDP design (state space, action space, reward function), which better explains the replenishment logic and enhances the interpretability of replenishment decisions. By monitoring the action space, data drift or concept drift can be detected, such as parameters describing the replenishment decision logic like expected inventory, triggered inventory, and replenishment frequency. When the model generates a "strange" replenishment decision, domain experts can analyze the decision logic and judge its rationality by analyzing the corresponding product's action space. Regarding the problem of blind assumptions, this invention avoids black-box prediction and prevents logical loopholes in the solution caused by blind assumptions about data distribution, business logic, and model architecture.

[0055] 4. From the perspective of commercialization and business iteration: By adopting imitation learning, the model exhibits high stability, low complexity, minimal data dependency, strong generalization ability, and low maintenance costs due to model decay. Simultaneously, it avoids the black-box prediction of existing technologies and eliminates blind assumptions. The MDP design enhances the transparency of the decision-making process, facilitating domain experts' understanding of the reasons behind decisions and enabling better commercialization and business expansion capabilities. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of model decay and model retraining.

[0057] Figure 2 This is a graph showing the relationship between model generalization ability, model decay risk, and data dependence.

[0058] Figure 3This is a schematic diagram of the overall architecture connection in this application.

[0059] Figure 4 This is a schematic diagram of the overall process in this application.

[0060] Figure 5 This is a diagram showing the relationship between triggered inventory, expected inventory, and actual inventory in this application.

[0061] Figure 6 This is a comparison chart of imitation learning and neural networks in terms of model quality and data quality in this application.

[0062] Figure 7 This is a schematic diagram illustrating the use of a multilayer perceptron to learn a composite reward function in this application.

[0063] Figure 8 This is a schematic diagram of the Dagger (Dataset Aggregation) algorithm in this application. Detailed Implementation

[0064] Specific embodiments of this application have been described in conjunction with the accompanying drawings and the following description to teach those skilled in the art how to make and use this application. For the purpose of teaching the principles of the application, some conventional aspects have been simplified or omitted. Those skilled in the art should understand that variations derived from these embodiments fall within the scope of this application. Those skilled in the art should understand that the following features can be combined in various ways to form multiple variations of this application. Terms such as “upper,” “lower,” “left,” “right,” “middle,” and “a” used in this application are merely for clarity of description and are not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention. Therefore, this application is not limited to the specific embodiments described below, but only to the claims and their equivalents.

[0065] This invention discloses an intelligent replenishment decision-making method based on reinforcement learning and imitation learning. All other embodiments obtained by those skilled in the art without inventive effort according to the embodiments of this invention are within the protection scope of this invention. To make the objectives, technical solutions, and advantages of this invention clearer, the implementation methods of this invention will be described in detail below with reference to the accompanying drawings.

[0066] like Figure 3 , Figure 4 As shown, this embodiment provides an intelligent replenishment decision-making method based on reinforcement learning and imitation learning, including the following steps:

[0067] S1. Abstract the replenishment business scenario through the MDP design module, which specifically includes state space design, action space design, reward function design and strategy design.

[0068] S2. Based on the MDP design module described in S1, construct the imitation learning modeling module, which specifically includes a behavior cloning module and an adversarial imitation learning module.

[0069] S3. Train using the Q-Learning algorithm to obtain the optimal replenishment strategy based on the current data;

[0070] S4. Using the strategy trained by the algorithm described in S3, make intelligent decisions for replenishment scenarios, and continuously interact with the environment through the Dagger algorithm to generate more data and optimize the model.

[0071] like Figure 3 As shown, this embodiment provides an intelligent replenishment decision-making method based on reinforcement learning and imitation learning, including an MDP design module, an imitation learning modeling module, and an intelligent decision-making module. The imitation learning modeling module includes a behavior cloning module and an adversarial imitation learning module. The behavior cloning module is used to handle simple business scenarios where the reward function is known, or expert examples are already sufficiently good; the model simply needs to minimize the difference in actions between the agent's policy and the expert's policy. Behavior cloning is typically categorized as a common regression or classification problem. The adversarial imitation learning module is used to handle complex business scenarios where the reward function is unknown, requiring fitting the reward function before reinforcement learning. The intelligent decision-making module includes a basic strategy and an exploration strategy. The basic strategy clones the store manager's behavior to reach the store manager's average level, handling simple business scenarios. The exploration decision, based on inverse reinforcement learning, learns a composite reward function to handle complex business scenarios, surpassing the store manager's decision-making level. In this invention, through the interaction between the MDP design module, the imitation learning modeling module, and the intelligent decision-making module, the problems of high input data dependence, low model stability, and weak generalization ability in existing replenishment systems are solved, black-box prediction is avoided, and better commercialization and business expansion capabilities are achieved.

[0072] In the embodiments of this application, such as Figure 3 As shown, the MDP design module includes state space design, action space design, reward function design, and policy design. The following section will elaborate on Markov Decision Process (MDP) in conjunction with specific practical applications:

[0073] S11. State Space Design Description: The state space design describes the agent's perception of replenishment business scenario information and its dynamic changes. It describes the agent's current state in various aspects and is a digital abstraction of business operations. In the intelligent replenishment scenario of this invention, the state space design includes information such as the quantity of goods in stock, the quantity of goods in transit, and the type of store, as well as their dynamic changes.

[0074] S12. Action Space Design describes the operations that the agent can perform, including determining whether to replenish the goods and determining the optimal replenishment quantity based on step S11. This describes the operations that the agent can perform, including but not limited to determining whether to replenish and determining the optimal replenishment quantity. In the intelligent replenishment scenario of this invention, the action space includes whether to replenish, how much to replenish, and the replenishment frequency. The action space design incorporates business knowledge and innovatively introduces several new concepts to more finely characterize replenishment actions. Figure 5 To trigger the relationship diagram of inventory, expected inventory, and actual inventory, in Figure 5 In this context, the following parameters are defined: Triggered Inventory: When the actual inventory of a product falls below the trigger inventory, the store manager is prompted to replenish the stock; Expected Inventory: The ideal quantity of the product the store manager expects to replenish; Replenishment Frequency: The time interval between two replenishment actions. After introducing the above action space design information, the agent uses the following formula to determine whether to replenish stock and to ascertain the optimal replenishment quantity:

[0075]

[0076] In the formula: the theoretical optimal replenishment quantity for each product per day includes the following two possibilities:

[0077] The first method: When the actual inventory of a product on a given day is lower than the trigger inventory of the product, the product is replenished. The optimal replenishment quantity is the difference between the product's "expected inventory" and its "actual inventory".

[0078] The second scenario: When the actual inventory of a product on a given day is greater than or equal to the product's triggered inventory level, the product's inventory level is sufficient to meet daily sales needs, and therefore no replenishment is required.

[0079] At the same time, the model also considers replenishment frequency and selects high-frequency or low-frequency replenishment strategies based on the attributes of various products to ensure the long-term rationality of replenishment behavior.

[0080] S13. Reward Function Design: Customer expectations and task objectives are transformed into quantifiable reward functions to guide reinforcement learning algorithm training. In reinforcement learning tasks, the agent continuously improves its strategy based on feedback from the environment during exploration. These feedback signals are called rewards, and the accumulated rewards are called returns. As a concretization and quantification of task objectives, reward signals serve as a bridge between humans and algorithms, transforming business expectations into reward functions to guide reinforcement learning algorithm training. Figure 3 As shown, in the intelligent replenishment scenario of this invention, the reward function can be designed from the perspective of daily sales and net profit. For example, the higher the store's net profit, the higher the corresponding reward. The reward function can also be designed from the perspective of scrap rate and sell-through rate. For example, the lower the store's scrap rate, the higher the corresponding reward.

[0081] S14. Strategy design is based on the state space design, action space design, and reward function design. Depending on whether the strategy optimization objective is singular, it can be divided into simple strategies and composite strategies, using a single reward function and a composite reward function respectively. The agent strategy is based on the state space design, action space design, and reward function design, and changes according to the variations in these design elements. Strategy design includes mimicking the store manager, maximizing daily sales, maximizing net profit, reducing scrap rate, reducing sell-through rate, and ensuring abundant product displays. The strategy design combines the reward functions to obtain a composite reward function, the formula of which is as follows:

[0082]

[0083] in, R i Let α represent the i-th reward function. i This represents the weight coefficient corresponding to the i-th reward function, and the weight coefficients are summed to 100%. N indicates that the composite reward function comprehensively considers N reward functions. The specific number of N is determined by business requirements. The composite reward function can handle both simple and complex business scenarios.

[0084] Merchants have varying requirements and priorities regarding business needs at different times and in different regions. For example, during daily operations, the replenishment strategy aims to maximize daily sales and net profit targets; when capturing market share, the replenishment strategy maximizes the amount of merchandise displayed in stores and allows for higher spoilage rates. By adjusting the composite reward function, replenishment decisions can be guided to adapt to business needs, achieving the goal of flexibly adjusting the model strategy. This allows the model to accommodate various scenarios, which is more in line with the ever-changing business landscape of today's commercial world.

[0085] It should be noted that the behavior cloning module is used to deal with simple business scenarios where the reward function is known. It trains the model with the expert replenishment strategy as the fitting target and quickly reaches the level of the expert's average replenishment strategy by minimizing the difference between the agent's replenishment strategy and the expert's replenishment strategy.

[0086] The behavior cloning module handles simple business scenarios where the reward function is known, such as when store replenishment needs to maintain a stable target scrap rate, and the reward function only needs to satisfy a single business objective.

[0087] Reward function = R1

[0088] Specifically, the closer the actual scrap rate of a store is to the target scrap rate, the greater the R1 feedback reward; the further the actual scrap rate is from the target scrap rate, the less the R1 feedback reward.

[0089] Taking real-world business scenarios as an example, such as Figure 3As shown, in the intelligent replenishment scenario of this invention, the optimal inventory quantity prediction based on behavioral cloning is as follows: First, it is clear that inventory quantity prediction is different from sales forecasting. After sales forecasting, it is often necessary to consider business issues such as carton size conversion, shelf display, and warehouse inventory to obtain the inventory quantity. This invention directly predicts the inventory quantity by cloning the inventory replenishment behavior of experts, replacing the traditional method of predicting sales and then converting them into inventory quantity, greatly reducing the loss of accuracy. In addition, the optimal daily inventory replenishment prediction based on behavioral cloning is also addressed: Store product ordering needs to consider logistics delays, that is, the lead time varies significantly for different products and different logistics methods. This invention first clusters the products, grouping products with similar logistics attributes into one category, and then learns the daily inventory replenishment lead time for a certain cluster of products by cloning the store manager's advance inventory replenishment habits. Behavioral cloning can significantly reduce the model's dependence on data, as practice has shown. Figure 6 As mentioned above, when the data quality is low, imitation learning has a higher model quality than neural networks. That is, behavioral cloning can obtain a better initial replenishment strategy based solely on expert replenishment strategies when the quality of commodity inventory data is low, while traditional technical solutions make decisions with low data quality that are less reasonable.

[0090] It should be further noted that, in the embodiments of this application, step S2 specifically includes:

[0091] S21. Based on whether the reward function is known, it is divided into a behavior cloning module with a known reward function and an adversarial imitation learning module with an unknown reward function;

[0092] S22. For simple business scenarios where the reward function is known, the known reward function is directly used to guide the training of the reinforcement learning algorithm, and the trained policy is used to guide the agent to generate a replenishment policy. This process is called behavior cloning.

[0093] S23. For complex business scenarios where the reward function is unknown, the reward function is fitted by inverse reinforcement learning. The fitted reward function is used to guide the training of the reinforcement learning algorithm, and the trained strategy is used to guide the agent to generate a replenishment strategy. This process is adversarial imitation learning.

[0094] S24. Step S23 specifically uses inverse reinforcement learning to fit the composite reward function, and the method includes, but is not limited to, linear regression. Step S2 is described in detail below.

[0095] The adversarial imitation learning module is used to deal with complex business scenarios where the reward function is unknown. It guides the agent to generate a replenishment strategy by fitting a composite reward function. That is, although there is example data, the model cannot achieve the best results by simply cloning the example data. At the same time, the reward function is unknown or difficult to design. In this case, inverse reinforcement learning can be used to fit a reward function and maximize the reward function to guide the agent's replenishment strategy.

[0096] The adversarial imitation learning module addresses complex business scenarios where the reward function is unknown. For example, store operations aim to maximize net profit, but also need to control product obsolescence, balance shelf space, and meet new product launch rates. In this case, the composite reward function can be derived from historical data through linear regression fitting.

[0097]

[0098] in, R i Let α represent the i-th reward function. i This represents the weighting coefficient corresponding to the i-th reward function, with the weighting coefficients summing to 100%. N indicates that the composite reward function comprehensively considers N reward functions, and the specific number of N is determined by business requirements.

[0099] Specifically, historical data includes indicators such as scrap rate, shelf space, and new product launch rate under different replenishment strategies. Using a linear regression method, with net profit as the dependent variable and scrap rate, shelf space, and new product launch rate as independent variables, a linear regression equation is constructed:

[0100] The composite reward function is: α1 * scrap rate + α2 * display quantity + α3 * new product rate, where the learned weight coefficients are α1 = 0.7, α2 = 0.2, and α3 = 0.1.

[0101] Furthermore, the method for fitting a composite reward function through inverse reinforcement learning in this invention includes, but is not limited to, linear regression. Other common methods include multilayer perceptrons (MLPs) and simple neural network fitting, such as... Figure 7 The diagram illustrates the use of a multilayer perceptron to learn a composite reward function. The input layer consists of different reward values, and the output layer is a composite reward value. The multilayer perceptron is used to learn the interaction between different rewards to meet the requirements of multiple business metrics in complex business scenarios.

[0102] Composite reward function = MLP(R1,R2,…,R) i )

[0103] or

[0104] Composite reward function = Neural network(R1,R2,…,R) i )

[0105] Where i represents the i-th business metric.

[0106] In the intelligent replenishment scenario of this invention, such as Figure 3As shown, similar scenarios include, but are not limited to, new product replenishment and replacement of old and new products. Since the reward function for complex business operations cannot be determined definitively, a composite reward function can be trained based on historical expert strategies for new product replenishment and replacement of old and new products. For example, the composite reward function training might yield a scrap rate of 15% for new product replenishment and a replacement ratio of 20% for replacement of old and new products, with equal weights. The composite reward function would then be 0.5*R1 + 0.5*R2, where R1 guides the scrap rate for new products and R2 guides the replacement ratio. In this case, even if the model encounters a product it has never seen before, it can still reasonably replenish the new product by obtaining the optimal scrap rate and replacement ratio through inverse reinforcement learning, thus improving the model's generalization ability.

[0107] In this invention, an adversarial imitation learning module is used to fit a reward function, and maximizing this reward function guides the agent to replenish stock. This generates strategies that surpass the average level of store managers and improves the model's extrapolation ability when encountering unknown situations.

[0108] Based on the above description, after completing the MDP design and constructing the key elements of the behavior cloning module and adversarial imitation learning module, policy training begins. In this embodiment, Q-Learning is used for training, and the composite reward function is used as the reward function for training Q-Learning to obtain the replenishment policy. The specific steps of Q-Learning in each round are as follows:

[0109] S31. Initialize t = 0;

[0110] S32. Start from the initial state S0;

[0111] S33. At time t, select a replenishment action A based on the Q value. t =argmax a∈A Q(S t a) Here, the ε-greedy method is used;

[0112] S34, Execute Action A t Then, it enters the next state S. t+1 (New inventory levels, scrap rates, display rates, etc.), and obtain the reward R through the above composite reward function. t+1 ;

[0113] S35, Q-value update function:

[0114] Q(S t A t )←Q(S t A t )+α(R t+1 +γmax a∈AQ(S t+1 ,a)-Q(S t A t ))

[0115] S36. Set t = t + 1, then return to S33.

[0116] The training process based on Q-Learning requires fewer parameters, does not need to model the external environment, and can quickly converge to the optimal policy, simplifying the training process. Furthermore, Q-Learning uses a Q-Table to store the optimal behavior, greatly enhancing the policy's business interpretability.

[0117] After completing the MDP design module and the imitation learning modeling module, this invention enters the intelligent decision-making process. The intelligent decision-making is divided into basic strategy and exploratory strategy. The basic strategy clones the store manager's behavior to reach the average level of most experts and deal with the simple business scenarios mentioned above. The exploratory strategy is based on inverse reinforcement learning and learns the reward function to deal with the complex business scenarios mentioned above. The basic decision focuses on weak generalization ability, while the exploratory strategy targets strong generalization ability.

[0118] In the embodiments of this application, such as Figure 8 As shown, the imitation learning modeling module (including the behavior cloning module and the adversarial imitation learning module) continuously interacts with the environment, constantly collecting expert examples and optimizing the model. This process is the Dagger (Dataset Aggregation) algorithm, which specifically includes:

[0119] S41. Based on the replenishment strategy trained in S3, continuously interact with the environment, constantly collect expert examples, and generate new data;

[0120] S42. On the augmented new dataset, repeat the steps described in S1-S3 to retrain the model.

[0121] S43. Continuously optimize the model through an iterative framework of “training-decision-retraining-re-decision”.

[0122] S41 specifically refers to: using the data of the day to learn from imitation and obtain a replenishment strategy;

[0123] S41 specifically refers to: using the data of the day to learn from imitation and obtain a replenishment strategy;

[0124] S42 specifically involves: interacting with the environment based on the current replenishment strategy, specifically through the ε-greedy algorithm. The ε-greedy algorithm is used to balance exploitation and exploration. This algorithm performs random exploration with a probability of ε and takes the optimal replenishment action with a probability of 1-ε, where ε is a small probability number between 0 and 1. It generates a new sequence, which consists of alternating states, actions, and reward values.

[0125] S t A t ,R t ,S t+1 A t+1 ,R t+1 ,…,S t+M A t+M ,R t+M

[0126] Where M is the number of steps explored by the algorithm;

[0127] Specifically, S43 is as follows: The model generates replenishment behavior through the ε-greedy algorithm, obtains an augmented dataset based on business metrics after replenishment or by collecting new expert examples, and then returns to S41.

[0128] Therefore, imitation learning (behavioral cloning or adversarial imitation learning) and interaction with the environment form a cyclical system. Based on historical expert data, a policy is modeled through imitation learning. This policy continuously interacts with the environment, generating new business metric data and collecting expert feedback examples. Then, on the augmented dataset, imitation learning is retrained, and interaction with the environment is repeated to collect feedback. This process is continuously repeated to optimize the replenishment strategy. Thus, the Dagger algorithm reduces the number of unvisited states, thereby improving the model's "strong" and "weak" generalization capabilities.

[0129] In summary, by using an explicit MDP design module (state space, action space, reward function), this invention enables rapid iteration at low cost in the complex and ever-changing business world; it solves the bottlenecks of existing big data deep learning technologies and the shortcomings of single replenishment strategies.

[0130] It should be noted that a computer-readable storage medium stores a computer program that can be executed by a computer processor to implement the intelligent replenishment decision-making method based on reinforcement learning and imitation learning described in the above embodiments. The computer program can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0131] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0133] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0134] This invention integrates behavioral cloning and adversarial imitation learning. Taking a store manager's historical replenishment decisions as an example, behavioral cloning training enables the agent's replenishment decision-making ability to reach the average level of experts. Building on this, when the agent faces complex business scenarios with unclear reward functions, inverse reinforcement learning is used to construct an adversarial reward function, enabling replenishment decisions that surpass expert levels. This effectively solves the problems of high dependence on input data, low model stability, and weak generalization ability. This invention employs imitation learning, resulting in high model stability, strong generalization ability, low decay maintenance costs, and low data dependence. It avoids black-box prediction, abandons blind assumptions, and possesses better commercialization and business expansion capabilities.

[0135] As those skilled in the art will readily conceive, any modifications, equivalent substitutions, improvements, etc., made using the concept and principles of this application should be included within the scope of the claims of this application.

Claims

1. An intelligent replenishment decision-making method based on reinforcement learning and imitation learning, characterized in that, Includes the following steps: S1. Abstract the replenishment business scenario through the MDP design module, which specifically includes state space design, action space design, reward function design and strategy design. S2. Based on the MDP design module described in S1, construct the imitation learning modeling module, which specifically includes a behavior cloning module and an adversarial imitation learning module. S3. Train using the Q-Learning algorithm to obtain the optimal replenishment strategy based on the current data; S4. Using the strategy trained by the algorithm described in S3, make intelligent decisions for replenishment scenarios, and continuously interact with the environment through the Dagger algorithm to generate more data and optimize the model; Specifically, step S1 includes: S11. State space design describes the agent's perception of replenishment business scenario information and its dynamic changes; S12. Action space design describes the operations that the agent can perform, and determines whether to perform a replenishment action on the goods and the optimal replenishment quantity of the goods based on step S11; S13. Reward function design transforms customer expectations and task objectives into quantifiable reward functions to guide the training of reinforcement learning algorithms; S14. The strategy design is based on the state space design, action space design and reward function design. Depending on whether the strategy optimization objective is singular, it can be divided into simple strategies and compound strategies, which use single reward functions and compound reward functions respectively. Step S2 specifically includes: S21. Based on whether the reward function is known, it is divided into a behavior cloning module with a known reward function and an adversarial imitation learning module with an unknown reward function; S22. For simple business scenarios where the reward function is known, the known reward function is directly used to guide the training of the reinforcement learning algorithm, and the trained policy is used to guide the agent to generate a replenishment policy. This process is called behavior cloning. S23. For complex business scenarios where the reward function is unknown, the reward function is fitted through inverse reinforcement learning. The fitted reward function is then used to guide the training of the reinforcement learning algorithm. The trained strategy is then used to guide the agent to generate a replenishment strategy. This process is adversarial imitation learning. S24. Step S23 specifically uses inverse reinforcement learning to fit the composite reward function, and the method includes, but is not limited to, linear regression. Step S4 specifically includes: S41. Based on the replenishment strategy obtained from training in S3, continuously interact with the environment, constantly collect expert examples, and generate new data. That is, learn by imitation based on the data of the day to obtain a replenishment strategy. S42. On the augmented new dataset, repeat steps S1-S3 to retrain the model, i.e., perform environmental interactions with the current replenishment strategy, wherein the environmental interactions are achieved through... -greedy algorithm complete: The -greedy algorithm is used to balance exploitation and exploration. This algorithm uses Random exploration is performed with a probability of 1- To determine the probability of taking the best replenishment action, It is a small probability number between 0 and 1, generating a new sequence, which consists of alternating states, actions, and reward values: ； Where M is the number of steps explored by the algorithm; S43. Through an iterative framework of "training-decision-retraining-re-decision," the model is continuously optimized, and the model passes... The -greedy algorithm generates replenishment behavior, obtains an augmented dataset based on business metrics after replenishment or by collecting new expert examples, and then returns to S41.

2. The intelligent replenishment decision-making method based on reinforcement learning and imitation learning according to claim 1, characterized in that: In step S12, the agent determines whether to replenish stock and the optimal replenishment quantity using the following formula: ； In the formula: the theoretical optimal replenishment quantity for each product per day includes the following two possibilities: The first method: When the actual inventory of a product on a given day is lower than the trigger inventory of the product, the product is replenished. The optimal replenishment quantity is the difference between the product's "expected inventory" and its "actual inventory". The second scenario: When the actual inventory of a product on a given day is greater than or equal to the product's triggered inventory level, the product's inventory level is sufficient to meet daily sales needs, and therefore no replenishment is required. The intelligent agent selects high-frequency or low-frequency replenishment strategies based on the attributes of various commodities.

3. The intelligent replenishment decision-making method based on reinforcement learning and imitation learning according to claim 1, characterized in that: In step S24, inverse reinforcement learning is used to fit a composite reward function, the formula of which is as follows: ； in, , Let i represent the i-th reward function. The weight coefficients represent the weights corresponding to the i-th reward function. The weight coefficients are summed to 100%. N represents the composite reward function that takes into account N reward functions. The specific number of N is determined by business requirements.

4. The intelligent replenishment decision-making method based on reinforcement learning and imitation learning according to claim 1, characterized in that: Step S3 specifically includes: S31, Initialization ; S32, From the initial state start; S33, at time ,according to Choose one replenishment action , adopt -greedy method; S34, Execution Action Then, proceed to the next state. And the reward is obtained through the above composite reward function. ; S35, Update Value functions: ； S36, Settings Return to S33.

5. The intelligent replenishment decision-making method based on reinforcement learning and imitation learning according to claim 1, characterized in that: The behavior cloning module trains the model with the expert replenishment strategy as the fitting target. By minimizing the difference between the agent's replenishment strategy and the expert's replenishment strategy, the behavior cloning task is reduced to a common regression or classification task.

6. A computer-readable storage medium storing a computer program, characterized in that, The computer program can be executed by a computer processor to implement computer-readable instructions for the intelligent replenishment decision-making method based on reinforcement learning and imitation learning as described in any one of claims 1 to 5.

Images