Inventory model training method and device based on deep reinforcement learning

Abstract

The invention relates to an inventory model training method and device based on deep reinforcement learning. The method comprises the following steps: modeling an inventory problem into a Markov decision process, defining a state space, an action space, a state transition probability and a reward function, and constructing an inventory model for a target service; a deep neural network is adopted as an approximator of a value function and a strategy function, and initialized network parameters of the inventory model are determined; acquiring historical inventory system operation data for the target business, and determining a training sample according to the historical inventory system operation data through interaction between the inventory model and the inventory environment based on an experience putting-back mechanism; and extracting data from the training samples stored in the experience playback buffer area to carry out model processing, and updating network parameters of the inventory model until a preset training ending condition is met, thereby obtaining the trained inventory model. By adopting the method, an inventory model which is adaptive to a complex and changeable market environment and is automatically optimized can be constructed, and the inventory management efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for inventory model training based on deep reinforcement learning. Background Art

[0002] In the enterprise operation system, inventory management, as a core component of supply chain management, is directly related to the enterprise's production efficiency, capital occupation, customer service level and market competitiveness. Inventory management is a key link in enterprise operations. Effective inventory management can improve the enterprise's operating efficiency and reduce costs.

[0003] However, traditional inventory management methods often rely on manual experience and fixed rules, and are difficult to adapt to the complex and changing market environment. In the face of rapidly changing market demands, increasing supply chain uncertainty, shortened product life cycles, and rapidly changing consumer preferences, their limitations are becoming increasingly prominent. Not only are they inefficient, but they are also prone to human errors, increasing the complexity and error rate of data processing, and easily leading to inventory backlogs or stockouts. Summary of the invention

[0004] Based on this, it is necessary to provide an inventory model training method, device, computer equipment, computer-readable storage medium and computer program product based on deep reinforcement learning that can improve inventory management efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides an inventory model training method based on deep reinforcement learning, comprising:

[0006] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0007] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0008] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0009] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0010] In one embodiment, the inventory problem is modeled as a Markov decision process, and a state space, an action space, a state transition probability, and a reward function are defined to construct an inventory model for a target business, including:

[0011] Obtaining inventory problem description information of the business entity to which the target business belongs, and determining a resource gain function of the business entity;

[0012] Based on the resource gain function, the inventory problem is modeled as a Markov decision process, and the state space, action space, state transition probability, and reward function are defined to construct an intelligent agent deep reinforcement learning model as the inventory model.

[0013] In one embodiment, the defining of the state space, action space, state transition, and reward function includes:

[0014] Based on the cycle parameters of the inventory processing task, determining the state variables of the cycle parameters and defining the state space;

[0015] Defining the action space based on the decision variables of the periodic parameters;

[0016] Based on the cycle parameters and the order disappearance condition in the inventory, define the state transition probability;

[0017] The reward function is defined based on the relationship between the cycle parameter, the inventory status and the inventory decision reward.

[0018] In one embodiment, the method of using a deep neural network as an approximator of a value function and a policy function to determine the initialization network parameters of the inventory model includes:

[0019] Using the state variables in the Markov decision process as inputs to the deep neural network and using the value function as outputs of the deep neural network;

[0020] The inventory model is set based on the input and output of the deep neural network, the inventory decision information is generated through a learning strategy, and the initialization network parameters of the inventory model are determined.

[0021] In one embodiment, the experience-based replacement mechanism determines the training samples according to the historical inventory system operation data through the interaction between the inventory model and the inventory environment, including:

[0022] Simulating the operation of the inventory system according to the historical inventory system operation data to generate simulated operation data;

[0023] Execute inventory actions based on the simulation operation data, and return inventory status and rewards;

[0024] The inventory state, the inventory state and the reward, and the next inventory state are collected and stored in the experience replay buffer as the training samples.

[0025] In one embodiment, extracting data from the training samples stored in the experience playback buffer for model processing and updating the network parameters of the inventory model until a preset training end condition is met to obtain a trained inventory model includes:

[0026] Extracting the processing data of the current cycle from the training samples stored in the experience playback buffer;

[0027] Inputting the processed data of the current cycle into the inventory model, and updating the network parameters of the inventory model according to the calculated state action value and advantage value function of the current cycle;

[0028] Based on the resource gain data of the current cycle and the inventory status of the next cycle as feedback, the model training is performed in a loop until the preset training end condition is met to obtain the trained inventory model; the preset training end condition includes the end of the product sales period of the target business.

[0029] In a second aspect, the present application also provides an inventory model training device based on deep reinforcement learning, comprising:

[0030] The inventory model building module is used to build an inventory model for the target business by modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function;

[0031] A parameter initialization module, for determining initialization network parameters of the inventory model using a deep neural network as an approximator of a value function and a policy function;

[0032] An inventory environment interaction module, used to obtain historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0033] The model training module is used to extract data from the training samples stored in the experience playback buffer for model processing, update the network parameters of the inventory model, and obtain a trained inventory model until a preset training end condition is met; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0036] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0037] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0038] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0041] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0042] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0043] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0046] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0047] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0048] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0049] The above-mentioned inventory model training method, device, computer equipment, computer-readable storage medium and computer program product based on deep reinforcement learning, by modeling the inventory problem as a Markov decision process, and defining the state space, action space, state transition probability, and reward function, constructing an inventory model for the target business, using a deep neural network as an approximator of the value function and the policy function, determining the initialization network parameters of the inventory model, and then obtaining the historical inventory system operation data for the target business, based on the experience replay mechanism, through the interaction between the inventory model and the inventory environment, determining the training samples according to the historical inventory system operation data, and then extracting data from the training samples stored in the experience replay buffer for model processing, updating the network parameters of the inventory model until the preset training end conditions are met, and obtaining the trained inventory model, the trained inventory model is used to optimize the inventory plan of the target business, generate inventory decision information based on deep reinforcement learning, and realize the construction of an inventory management system that is adaptive to complex and changing market environments and integrates intelligent prediction, dynamic adjustment, real-time monitoring and automatic optimization, which can effectively improve inventory management efficiency and reduce enterprise operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] FIG1 is a schematic diagram of a flow chart of an inventory model training method based on deep reinforcement learning in one embodiment;

[0052] FIG2 is a schematic diagram of an inventory processing flow in one embodiment;

[0053] FIG3 is a schematic diagram of an inventory planning optimization system in one embodiment;

[0054] FIG4 is a schematic diagram of a flow chart of an inventory model training method based on deep reinforcement learning in another embodiment;

[0055] FIG5 is a structural block diagram of an inventory model training device based on deep reinforcement learning in one embodiment;

[0056] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In an exemplary embodiment, as shown in FIG1 , a method for training an inventory model based on deep reinforcement learning is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 101 to 104. Among them:

[0059] Step 101, constructing an inventory model for the target business by modeling the inventory problem as a Markov decision process and defining a state space, an action space, a state transition probability, and a reward function.

[0060] In practical applications, with the development of artificial intelligence and its related industries, deep reinforcement learning has achieved remarkable results in the field of artificial intelligence, providing new ideas for solving inventory management problems. In the inventory model training process based on deep reinforcement learning, the inventory problem can be modeled as a Markov decision process, defining states, actions, rewards, and transition probabilities to build an inventory model for the target business, such as different types of product sales business. Therefore, by effectively integrating deep reinforcement learning technology into the inventory management system, it is possible to achieve intelligent, automated and precise inventory management, which can meet the needs of enterprises for efficient, flexible and precise inventory management.

[0061] Step 102: Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model.

[0062] In a specific implementation, a deep neural network can be used as an approximator of the value function and the policy function to determine the initialization network parameters of the inventory model. For example, the state variables in the Markov decision process can be used as the input of the deep neural network, and the deep neural network can output the approximated value function.

[0063] Step 103 , obtaining historical inventory system operation data for the target business, and determining training samples according to the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism.

[0064] In one example, the cost, revenue, order quantity and past demand information in the inventory system (i.e., historical inventory system operation data for the target business) can be collected. Optionally, the current income of the enterprise can be calculated as the difference between the revenue and the cost in the following way:

[0065]

[0066] Among them, the inventory system t is the return in period t, c o is the unit purchase cost, c h is the unit inventory cost, c p is the unit shortage cost, p is the unit sales price, represents random demand (obeying a known distribution), Q t is the order quantity in period t, AO t is the available inventory in period t, , indicating the available quantity of the product in the current period.

[0067] Specifically, an experience replay mechanism can be used to collect samples of state, action, reward, and next state through interaction with the inventory environment, and store them in an experience replay buffer to obtain training samples.

[0068] Step 104 , extracting data from the training samples stored in the experience playback buffer for model processing, and updating the network parameters of the inventory model until a preset training end condition is met, thereby obtaining a trained inventory model.

[0069] Among them, the trained inventory model can be used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0070] In practical applications, a batch of samples can be randomly drawn from the experience replay buffer, the model target value and advantage value function can be calculated, and the network parameters can be updated. The training process can be repeated until the preset termination condition is reached or the performance requirements are met, such as through continuous loops until the end of a sales period. Then, the intelligent agent can train an optimized neural network (i.e., the trained inventory model). The trained inventory model can be used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning, so as to achieve the purpose of obtaining optimal benefits through learning strategies.

[0071] In one example, as shown in Figure 2, during the training of an inventory model based on deep reinforcement learning, the intelligent agent can simulate the inventory management process, make actions (such as ordering) through intelligent agent decisions, trigger the supplier's delivery process based on the ordering action, and then update the inventory status based on the delivery of the goods. It can also be realized based on demand, and the revenue and status can be updated by selling products, and then the information obtained from inventory processing can be fed back to the intelligent agent.

[0072] In another example, as shown in FIG3 , inventory model training can be performed through an inventory planning optimization system based on deep reinforcement learning to construct an inventory model, and inventory strategies and plans can be dynamically planned by constructing an intelligent agent deep reinforcement learning model. The system may include an information collection module for obtaining historical inventory system operation data; an inventory environment module for simulating the operation of the inventory system, generating demand and supply data, executing inventory actions, and returning inventory status and rewards; a reinforcement learning module for implementing an inventory model training method, which may include a value network and a policy network, and an experience playback buffer; an inventory control module is constructed for generating inventory decisions (such as order quantity) according to the strategy output by the reinforcement learning module, and sending it to the inventory environment module; and a deployment experience playback mechanism is introduced, which can be used to increase the stability of the neural network to train the neural network, so as to achieve the purpose of obtaining the optimal benefit through the learning strategy, so that by introducing modules and mechanisms, it can be more in line with the enterprise's inventory system optimization and control, and the intelligent agent reinforcement learning strategy helps to achieve the optimal benefit through inventory planning.

[0073] Compared with the traditional inventory management method that relies on fixed rules or formula processing and is difficult to adapt to the rapidly changing market environment and customer needs, the technical solution of this embodiment can automatically adjust the inventory strategy through the continuous interaction between the intelligent agent and the environment through the system based on deep reinforcement learning to better adapt to various changes, and has stronger adaptability; the deep reinforcement learning model can use the powerful representation ability of the deep learning network to learn complex patterns and associations from a large amount of historical data to make more accurate demand forecasts, and the reward mechanism of reinforcement learning enables the system to continuously optimize inventory decisions in the long-term learning process to achieve the goal of minimizing costs or maximizing service levels, and can achieve more accurate predictions and optimizations; compared with the traditional inventory management that needs to rely on regular manual review and adjustment, the technical solution of this embodiment can monitor the inventory status and market dynamics in real time through the system based on deep reinforcement learning, and respond immediately. The good real-time and dynamic performance ensures that the inventory level is always kept in the optimal state, effectively avoiding inventory backlogs or out-of-stock situations; the deep reinforcement learning system has a high degree of automation, which can greatly reduce the need for manual intervention and save labor costs. By optimizing inventory strategies, it can also reduce inventory holding costs, out-of-stock costs, and transportation costs, etc., effectively improving the overall operational capabilities of the enterprise.

[0074] In the above inventory model training method based on deep reinforcement learning, the inventory problem is modeled as a Markov decision process, and the state space, action space, state transition probability, and reward function are defined to construct an inventory model for the target business. A deep neural network is used as an approximator of the value function and the policy function to determine the initialization network parameters of the inventory model. Then, the historical inventory system operation data for the target business is obtained. Based on the experience replay mechanism, the training samples are determined according to the historical inventory system operation data through the interaction between the inventory model and the inventory environment. Then, data is extracted from the training samples stored in the experience replay buffer for model processing, and the network parameters of the inventory model are updated until the preset training end conditions are met. The trained inventory model is obtained, and the construction of an inventory management system that is adaptive to complex and changing market environments and integrates intelligent prediction, dynamic adjustment, real-time monitoring and automatic optimization is realized. It can effectively improve inventory management efficiency and reduce enterprise operating costs.

[0075] In an exemplary embodiment, the inventory problem is modeled as a Markov decision process, and a state space, an action space, a state transition probability, and a reward function are defined to construct an inventory model for a target business, which may include the following steps:

[0076] Obtain inventory problem description information of the business entity to which the target business belongs, and determine the resource gain function of the business entity; based on the resource gain function, model the inventory problem as a Markov decision process, and define the state space, action space, state transition probability, and reward function, and construct an intelligent agent deep reinforcement learning model as the inventory model.

[0077] In the specific implementation, the inventory problem can be characterized and a mathematical model can be constructed to obtain the enterprise profit function (that is, the resource gain function of the business entity). The inventory problem can be modeled as a Markov decision process. By defining the state space, action space, state transition probability, and reward function, an intelligent agent deep reinforcement learning model can be constructed as an inventory model.

[0078] In an exemplary embodiment, the definition of state space, action space, state transition, and reward function may include the following steps:

[0079] Based on the cycle parameters of the inventory processing task, the state variables of the cycle parameters are determined to define the state space; based on the decision variables of the cycle parameters, the action space is defined; based on the cycle parameters and the conditions for disappearance of orders in the inventory, the state transition probability is defined; based on the relationship between the cycle parameters, the inventory state and the inventory decision reward, the reward function is defined.

[0080] In one example, for the state space: before each period t (i.e., period parameter) faces random demand, we can observe s t and make decisions t (a t For the order quantity decision Q t ), and can get the current return r after selling it to the market t , then the state variable in period t is a (l-1)-dimensional vector ,s t =AO t , represents the enterprise's available inventory, if AO t >0 means the current available inventory is greater than zero.

[0081] For the action space: Decision variable a in period t t =Q t , Q t is the order quantity, Q t ≥0.

[0082] Regarding the state transition probability: If the company does not have enough inventory to satisfy a certain order, the order will disappear, such as Sometimes, there are ,when Sometimes, there are .

[0083] For the reward function: observe the current state s obtained by the agent t and make a decision t After that, you can get the reward r t , to measure the value of the action; by the state variable You can know the available inventory AO t , and then the corresponding shortage or holding cost can be calculated. t Calculate the corresponding ordering cost, and the price p and demand Calculate the current income and get the following reward function:

[0084]

[0085] Among them, when the company's available inventory is greater than demand, the sales volume is ; When the company is out of stock, the unfilled orders will not be carried over to the next period, and the sales volume is AO t .

[0086] In an optional embodiment, the sequence of events in period t can be expressed as follows:

[0087] 1. The state of the initial update period t is s t ;

[0088] 2. The agent makes a decision t , determine the order quantity Q t ;

[0089] 3. Demand Realize that, in the event of a company out of stock, unfilled orders disappear, and after the order is delivered, the remaining inventory is carried forward to the next cycle;

[0090] 4. At the end of period t, the income and cost of the period are settled, and the agent receives a reward r t , and update the state to s (t+1) .

[0091] In an exemplary embodiment, the method of using a deep neural network as an approximator of a value function and a policy function to determine the initialization network parameters of the inventory model may include the following steps:

[0092] The state variables in the Markov decision process are used as the input of the deep neural network, and the value function is used as the output of the deep neural network; the inventory model is set based on the input and output of the deep neural network, the inventory decision information is generated through a learning strategy, and the initialization network parameters of the inventory model are determined.

[0093] In practical applications, according to the designed Markov decision process, the state variables can be used as the input of the neural network, and the output value function Q(s, a) is expressed as a function by approximating the value function through the neural network:

[0094]

[0095] For example, the agent can obtain the current state by observing the environment, know the product inventory of the enterprise in the sales period, and approximate the value function Q (s, a) based on the state neural network. Based on this, as shown in Figure 2, the agent can make a decision, give the order quantity for this period, and obtain the current enterprise revenue and the state of the next period as feedback after interacting with the environment. Through continuous loop processing until the end of a sales period, the neural network trained by the agent can achieve the purpose of learning strategies to obtain optimal benefits. In terms of strategy update, it can be used The greedy strategy selects actions to balance the agent's development and exploration behaviors, with a very small positive number The probability of selecting a random action in the action space is , in other cases Then choose the action that maximizes the Q value . Moreover, the state action value (Q value) can be updated based on another deterministic strategy instead of the current strategy, that is, the action that maximizes the Q value is selected.

[0096] In one example, DQN (Deep Q-Network) can estimate the Q target value from the current reward and the Q estimate of the next state, and minimize the difference between the Q estimate and the Q target value to update the neural network parameters. Its loss function uses the mean square error between the estimate and the target value:

[0097]

[0098] In an exemplary embodiment, the experience-based replacement mechanism, through the interaction between the inventory model and the inventory environment, determines the training samples according to the historical inventory system operation data, and may include the following steps:

[0099] The operation of the inventory system is simulated according to the historical inventory system operation data to generate simulated operation data; the inventory action is executed based on the simulated operation data, and the inventory status and reward are returned; the inventory status, the inventory status and reward, and the next inventory status are collected and stored in the experience playback buffer as the training sample.

[0100] Optionally, designing an experience-replacement mechanism in the system can increase the stability of the neural network. By storing the historical data of each step of exploration in the experience pool D , from which samples can be taken and the Q target value can be calculated based on the sampled data.

[0101] In one example, the input vector of the neural network is the state space S, and the dimension is the dimension of the state space; the output vector is the state action value (Q value), which corresponds to the dimension of the action space A. The input vector can represent the state variable in the reinforcement learning quadruple, such as product inventory information, and the dimension is the inventory quantity of the product; the output vector can represent the corresponding state action value after the agent makes a certain ordering decision. The target network and the evaluation network have the same neural network structure, both with an input layer, two hidden layers and an output layer. The decision mechanism of inventory DQN can be expressed as follows:

[0102]

[0103] In an exemplary embodiment, extracting data from the training samples stored in the experience playback buffer for model processing, updating the network parameters of the inventory model until a preset training end condition is met to obtain a trained inventory model may include the following steps:

[0104] Extracting the processing data of the current cycle from the training samples stored in the experience playback buffer; inputting the processing data of the current cycle into the inventory model, and updating the network parameters of the inventory model according to the calculated state action value and advantage value function of the current cycle; and performing model training in a loop based on the resource gain data of the current cycle and the inventory state of the next cycle as feedback until the preset training end condition is met to obtain the trained inventory model; the preset training end condition includes the end of the product sales period of the target business.

[0105] In practical applications, a batch of samples are randomly drawn from the experience replay buffer, the model target value (i.e., state-action value) and advantage value function are calculated, and the network parameters are updated. The training process is repeated until the preset termination condition is reached or the performance requirements are met.

[0106] Specifically, the system can obtain the current state by observing the environment, determine the inventory quantity of products of each shelf life owned by the fresh food platform during the current sales period, and approximate the value function based on the state neural network. The agent can make decisions, give the price discount of the current product and the number of products ordered from the supplier, and obtain the current platform revenue and the next state as feedback after interacting with the environment. Based on the continuous loop processing until the end of a sales period, the agent can train a neural network to achieve the goal of obtaining the optimal profit through learning strategies.

[0107] In an exemplary embodiment, as shown in FIG4 , a flow chart of another inventory model training method based on deep reinforcement learning is provided. In this embodiment, the method includes the following steps:

[0108] In step 401, the inventory problem description information of the business subject to which the target business belongs is obtained, and the resource gain function of the business subject is determined. In step 402, based on the resource gain function, the inventory problem is modeled as a Markov decision process, and the state space, action space, state transition probability, and reward function are defined to construct an intelligent agent deep reinforcement learning model as an inventory model. In step 403, the state variables in the Markov decision process are used as the input of the deep neural network, and the value function is used as the output of the deep neural network. In step 404, the inventory model is set based on the input and output of the deep neural network, the inventory decision information is generated through the learning strategy, and the initialization network parameters of the inventory model are determined. In step 405, the historical inventory system operation data for the target business is obtained, and based on the experience replay mechanism, the training samples are determined according to the historical inventory system operation data through the interaction between the inventory model and the inventory environment. In step 406, the processing data of the current cycle is extracted from the training samples stored in the experience replay buffer, and the processing data of the current cycle is input into the inventory model. According to the calculated state action value and advantage value function of the current cycle, the network parameters of the inventory model are updated. In step 407, based on the resource gain data of the current cycle and the inventory status of the next cycle as feedback, the model training is performed in a loop until the preset training end condition is met to obtain a trained inventory model; the preset training end condition includes the end of the product sales period of the target business. It should be noted that the specific definition of the above steps can be found in the specific definition of an inventory model training method based on deep reinforcement learning above, which will not be repeated here.

[0109] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0110] Based on the same inventive concept, the embodiment of the present application also provides a deep reinforcement learning-based inventory model training device for implementing the deep reinforcement learning-based inventory model training method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more embodiments of the inventory model training device based on deep reinforcement learning provided below can refer to the above limitations on the inventory model training method based on deep reinforcement learning, and will not be repeated here.

[0111] In an exemplary embodiment, as shown in FIG5 , a deep reinforcement learning-based inventory model training device is provided, comprising:

[0112] The inventory model building module 501 is used to build an inventory model for a target business by modeling the inventory problem as a Markov decision process and defining a state space, an action space, a state transition probability, and a reward function;

[0113] A parameter initialization module 502, for determining the initialization network parameters of the inventory model using a deep neural network as an approximator of a value function and a policy function;

[0114] The inventory environment interaction module 503 is used to obtain historical inventory system operation data for the target business, and determine training samples according to the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on the experience replacement mechanism;

[0115] The model training module 504 is used to extract data from the training samples stored in the experience playback buffer for model processing, update the network parameters of the inventory model, and obtain a trained inventory model until a preset training end condition is met; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0116] In one embodiment, the inventory model building module 501 includes:

[0117] A resource gain function determination submodule is used to obtain inventory problem description information of the business subject to which the target business belongs, and determine the resource gain function of the business subject;

[0118] The intelligent agent construction submodule is used to model the inventory problem as a Markov decision process based on the resource gain function, and define the state space, action space, state transition probability, and reward function to construct an intelligent agent deep reinforcement learning model as the inventory model.

[0119] In one embodiment, the inventory model building module 501 includes:

[0120] A state space definition submodule, for determining state variables of the cycle parameters based on the cycle parameters of the inventory processing task, and defining the state space;

[0121] An action space definition submodule, used for defining the action space based on the decision variables of the periodic parameters;

[0122] A state transition probability definition submodule, used to define the state transition probability based on the cycle parameter and the order disappearance condition in the inventory;

[0123] The reward function definition submodule is used to define the reward function based on the relationship between the cycle parameter, the inventory status and the inventory decision reward.

[0124] In one embodiment, the parameter initialization module 502 includes:

[0125] An input-output determination submodule, used to use the state variables in the Markov decision process as the input of the deep neural network and the value function as the output of the deep neural network;

[0126] The setting submodule is used to set the inventory model based on the input and output of the deep neural network, generate the inventory decision information through a learning strategy, and determine the initialization network parameters of the inventory model.

[0127] In one embodiment, the inventory environment interaction module 503 includes:

[0128] A simulation operation submodule, used to simulate the operation of the inventory system according to the historical inventory system operation data, and generate simulation operation data;

[0129] A data return submodule, used to perform inventory actions based on the simulation operation data, and return inventory status and rewards;

[0130] The training sample obtaining submodule is used to collect the inventory status, the inventory status and reward, and the next inventory status, and store them in the experience playback buffer as the training sample.

[0131] In one embodiment, the model training module 504 includes:

[0132] A data extraction submodule, used to extract the processing data of the current cycle from the training samples stored in the experience playback buffer;

[0133] A network parameter updating submodule, used for inputting the processing data of the current cycle into the inventory model, and updating the network parameters of the inventory model according to the state action value and advantage value function calculated for the current cycle;

[0134] A loop processing submodule is used to perform model training based on the resource gain data of the current cycle and the inventory status of the next cycle as feedback, and loop processing until the preset training end condition is met to obtain the trained inventory model; the preset training end condition includes the end of the product sales period of the target business.

[0135] Each module in the above-mentioned inventory model training device based on deep reinforcement learning can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0136] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, a near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for training an inventory model based on deep reinforcement learning is implemented.

[0137] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0139] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0140] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0141] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0142] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0143] In one embodiment, when the processor executes the computer program, it also implements the steps of the inventory model training method based on deep reinforcement learning in other embodiments described above.

[0144] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0145] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0146] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0147] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0148] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0149] In one embodiment, when the computer program is executed by a processor, the steps of the inventory model training method based on deep reinforcement learning in other embodiments described above are also implemented.

[0150] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0151] By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed;

[0152] Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model;

[0153] Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism;

[0154] By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

[0155] In one embodiment, when the computer program is executed by a processor, the steps of the inventory model training method based on deep reinforcement learning in other embodiments described above are also implemented.

[0156] It should be noted that the collection, use and processing of the relevant data involved in this application must comply with relevant regulations.

[0157] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0158] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0159] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training an inventory model based on deep reinforcement learning, characterized in that: The method comprises: By modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function, an inventory model for the target business is constructed; Using a deep neural network as an approximator of a value function and a policy function, determining initialization network parameters of the inventory model; Acquire historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism; By extracting data from the training samples stored in the experience replay buffer for model processing, the network parameters of the inventory model are updated until the preset training end conditions are met, thereby obtaining a trained inventory model; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

2. The method according to claim 1, characterized in that The inventory problem is modeled as a Markov decision process, and the state space, action space, state transition probability, and reward function are defined to construct an inventory model for the target business, including: Obtaining inventory problem description information of the business entity to which the target business belongs, and determining a resource gain function of the business entity; Based on the resource gain function, the inventory problem is modeled as a Markov decision process, and the state space, action space, state transition probability, and reward function are defined to construct an intelligent agent deep reinforcement learning model as the inventory model.

3. The method according to claim 1, characterized in that The definition of state space, action space, state transfer, and reward function includes: Based on the cycle parameters of the inventory processing task, determining the state variables of the cycle parameters and defining the state space; Defining the action space based on the decision variables of the periodic parameters; Based on the cycle parameters and the order disappearance condition in the inventory, define the state transition probability; The reward function is defined based on the relationship between the cycle parameter, the inventory status and the inventory decision reward.

4. The method according to claim 1, characterized in that: The method of using a deep neural network as an approximator of a value function and a policy function to determine the initialization network parameters of the inventory model includes: Using the state variables in the Markov decision process as inputs to the deep neural network and using the value function as outputs of the deep neural network; The inventory model is set based on the input and output of the deep neural network, the inventory decision information is generated through a learning strategy, and the initialization network parameters of the inventory model are determined.

5. The method according to claim 1, characterized in that: The experience-based replacement mechanism determines training samples based on the interaction between the inventory model and the inventory environment and according to historical inventory system operation data, including: Simulating the operation of the inventory system according to the historical inventory system operation data to generate simulated operation data; Execute inventory actions based on the simulation operation data, and return inventory status and rewards; The inventory state, the inventory state and the reward, and the next inventory state are collected and stored in the experience replay buffer as the training samples.

6. The method according to any one of claims 1 to 5, characterized in that: The method extracts data from the training samples stored in the experience playback buffer for model processing, updates the network parameters of the inventory model, and obtains the trained inventory model until a preset training end condition is met, including: Extracting the processing data of the current cycle from the training samples stored in the experience playback buffer; Inputting the processed data of the current cycle into the inventory model, and updating the network parameters of the inventory model according to the calculated state action value and advantage value function of the current cycle; Based on the resource gain data of the current cycle and the inventory status of the next cycle as feedback, the model training is performed in a loop until the preset training end condition is met to obtain the trained inventory model; the preset training end condition includes the end of the product sales period of the target business.

7. A deep reinforcement learning-based inventory model training device, characterized in that: The device comprises: The inventory model building module is used to build an inventory model for the target business by modeling the inventory problem as a Markov decision process and defining the state space, action space, state transition probability, and reward function; A parameter initialization module, for determining initialization network parameters of the inventory model using a deep neural network as an approximator of a value function and a policy function; An inventory environment interaction module, used to obtain historical inventory system operation data for the target business, and determine training samples based on the historical inventory system operation data through the interaction between the inventory model and the inventory environment based on an experience replacement mechanism; The model training module is used to extract data from the training samples stored in the experience playback buffer for model processing, update the network parameters of the inventory model, and obtain a trained inventory model until a preset training end condition is met; the trained inventory model is used to optimize the inventory plan of the target business and generate inventory decision information based on deep reinforcement learning.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images