Self-adaptive production-inventory-distribution hierarchical structure control method and system

Abstract

The invention relates to the technical field of industrial production, and discloses a self-adaptive production-inventory-distribution hierarchical structure control method and system.The method comprises the steps that an upper-layer production-inventory environment and a lower-layer sales environment are constructed, the upper-layer production-inventory environment is used for simulating production capacity decision making and inventory metering, and the lower-layer sales environment is used for simulating the distribution condition; the reinforcement learning intelligent agent is designed based on the preset reinforcement learning algorithm, and the reward mechanism is set, so that the adaptive control of the dynamic environment is realized, and the flexibility of the system is improved. Cost minimization, profit maximization and service level maximization are taken as optimization objectives, conflicts among different objectives are balanced, and the overall service level and customer satisfaction are improved. And then the reward mechanism of the optimized intelligent agent is utilized to calculate the income and cost in the upper-layer production-inventory environment and the lower-layer sales environment, a reward value is obtained, a production capacity control strategy is determined based on the reward value, and the production cost, the inventory holding cost and the stockout cost can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of industrial production technology, and in particular to an adaptive production-inventory-distribution hierarchical structure control method and system. Background Art

[0002] For production, inventory, and distribution decisions, current solutions mostly use static methods to solve problems in different scenarios with the same algorithm or solution. This will lead to unstable performance of the algorithm or solution itself in different environments, resulting in decision-making deviations, thereby causing losses to the company. On the other hand, existing solutions need to be designed based on the solutions of different companies themselves. When the company's decision makers change or the market environment changes, the algorithms and solutions used cannot be implemented, which increases the deployment cost of the company and reduces the efficiency of the company. Finally, the distribution model of traditional industries, that is, the offline distribution model, sales are mostly dependent on the performance of distributors. This type of distribution model greatly increases the randomness of the market, making it more difficult for decision makers and decision systems to make ideal decision results, thereby greatly reducing efficiency.

[0003] Adaptive smart inventory is an important part of realizing Industry 5.0 and has received extensive attention from the manufacturing industry. Existing technologies and solutions only innovate the design of algorithms for production, inventory, and distribution in a specific scenario, thereby achieving the optimization of the entire system. However, in real scenarios, whether it is production, inventory, or distribution, human participation leads to greater uncertainty in the entire system. Although existing solutions have optimized adaptive control in a single link under a specific environment, they cannot achieve a relatively stable effect in a dynamic environmental change. Summary of the invention

[0004] The present application provides an adaptive production-inventory-distribution hierarchical structure control method to solve the problem in the prior art that, whether it is production, inventory, or distribution, human participation leads to greater uncertainty in the entire system, and adaptive control cannot achieve a relatively stable effect in a dynamic environmental change.

[0005] Correspondingly, the present application also provides an adaptive production-inventory-distribution hierarchical structure control system, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above technical problems, the present application discloses an adaptive production-inventory-distribution hierarchical structure control method, the method comprising:

[0007] Construct an upper-level production-inventory environment and a lower-level sales environment. The upper-level production-inventory environment is used to simulate production volume decisions and inventory volume measurement, and the lower-level sales environment is used to simulate distribution conditions.

[0008] Design reinforcement learning agents based on the preset reinforcement learning algorithm and set the reward mechanism for the reinforcement learning agents;

[0009] Adjust the parameters of the reward mechanism based on the optimization objectives to obtain the optimized reinforcement learning agent; the optimization objectives include minimizing costs, maximizing profits, and maximizing service levels;

[0010] The reward mechanism of the optimized reinforcement learning agent is used to calculate the revenue and cost of the upper production-inventory environment and the lower sales environment to obtain the reward value;

[0011] Based on the reward value, the optimized reinforcement learning agent is used to determine the production volume control strategy, and the production volume and inventory volume in the upper-level production-inventory environment and the distribution volume in the lower-level sales environment are adjusted according to the production volume control strategy.

[0012] The present application also discloses an adaptive production-inventory-distribution hierarchical structure control system, the system comprising:

[0013] An environment construction module is used to construct an upper-level production-inventory environment and a lower-level sales environment. The upper-level production-inventory environment is used to simulate production volume decisions and inventory volume measurement, and the lower-level sales environment is used to simulate distribution conditions.

[0014] The agent building module is used to design a reinforcement learning agent based on a preset reinforcement learning algorithm and set the reward mechanism of the reinforcement learning agent;

[0015] The agent optimization module is used to adjust the parameters of the reward mechanism based on the optimization objectives to obtain the optimized reinforcement learning agent; the optimization objectives include minimizing costs, maximizing profits, and maximizing service levels;

[0016] A reward calculation module is used to calculate the revenue and cost of the upper-level production-inventory environment and the lower-level sales environment using the reward mechanism of the optimized reinforcement learning agent to obtain the reward value;

[0017] The strategy generation module is used to determine the production volume control strategy based on the reward value using the optimized reinforcement learning agent, and adjust the production volume and inventory volume in the upper-level production-inventory environment and the distribution volume in the lower-level sales environment according to the production volume control strategy.

[0018] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0019] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0020] In this application, an upper production-inventory environment and a lower sales environment are constructed. The upper production-inventory environment is used to simulate production volume decision-making and inventory volume measurement, and the lower sales environment is used to simulate distribution conditions, thereby realizing dynamic simulation of complex production systems. A reinforcement learning agent is designed based on a preset reinforcement learning algorithm, and a reward mechanism for the reinforcement learning agent is set. The reinforcement learning algorithm is used to realize adaptive control of the dynamic environment, and can be adjusted according to real-time data, thereby improving the flexibility of the system. Afterwards, the parameters of the reward mechanism are adjusted with cost minimization, profit maximization, and service level maximization as the optimization goals, and the optimized reinforcement learning agent is obtained. Through multi-objective optimization, the conflicts between different goals are balanced, and the overall service level and customer satisfaction of the system are improved. Then, the reward mechanism of the optimized agent is used to calculate the income and cost of the upper production-inventory environment and the lower sales environment, and the reward value is obtained. Based on the reward value, the optimized reinforcement learning agent is used to determine the production volume control strategy, and the production volume and inventory volume in the upper production-inventory environment and the distribution volume in the lower sales environment are adjusted according to the production volume control strategy, which can effectively reduce production costs, inventory holding costs, and out-of-stock costs.

[0021] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart of an adaptive production-inventory-distribution hierarchical structure control method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of the structure of a hierarchical control system provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a production-inventory-distribution process for one day in a simulation process provided in an embodiment of the present application;

[0026] Figure 4 A training curve diagram of the reinforcement learning algorithm model provided in the embodiment of the present application;

[0027] Figure 5A loss function convergence curve diagram of the reinforcement learning algorithm model provided in the embodiment of the present application during the training process;

[0028] Figure 6 A box plot of monthly production provided for an embodiment of the present application;

[0029] Figure 7 A box plot of monthly sales provided for an embodiment of the present application;

[0030] Figure 8 A box plot of monthly inventory provided for an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of an adaptive production-inventory-distribution hierarchical structure control system provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0034] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0035] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0036] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. With respect to the technical problems existing in the prior art, the adaptive production-inventory-distribution hierarchical structure control method and system provided in this application are intended to solve at least one of the technical problems of the prior art.

[0037] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0038] The present application embodiment provides a possible implementation method, such as Figure 1 As shown, a flowchart of an adaptive production-inventory-distribution hierarchical structure control method is provided, and the scheme can be executed by any electronic device, and optionally, can be executed on a server side or a terminal device.

[0039] like Figure 1 As shown in , the method may include the following steps:

[0040] Step 101, constructing an upper-level production-inventory environment and a lower-level sales environment, wherein the upper-level production-inventory environment is used to simulate production volume decisions and inventory volume measurement, and the lower-level sales environment is used to simulate distribution situations.

[0041] In the embodiment of the present application, a simulation model is constructed based on the key links in real production enterprises, namely production, inventory and distribution. The simulation model simplifies and simulates the production and distribution process of enterprises with a two-layer structure. The simulation environment is divided into an upper-layer production-inventory environment and a lower-layer sales environment according to the link structure. In the upper-layer production-inventory environment, the model mainly simulates production volume decisions and inventory measurement, and simulates sales at the distribution end in the lower-layer sales environment.

[0042] Step 102, designing a reinforcement learning agent based on a preset reinforcement learning algorithm, and setting a reward mechanism for the reinforcement learning agent.

[0043] In the embodiment of the present application, the reinforcement learning algorithm can be a deep reinforcement learning network (DQN), proximal policy optimization (PPO), asynchronous advantage actor-critic algorithm (A3C) and deep deterministic policy gradient (DDPG), and the optimal algorithm can be selected according to the specific application scenario. The reinforcement learning agent is located in the upper production-inventory environment and is mainly responsible for determining the amount of production in the current time period (such as the day). The reinforcement learning agent affects the inventory in the upper production-inventory environment by adjusting the production volume, thereby affecting the distribution volume in the lower sales environment, and ultimately affecting the entire simulation environment.

[0044] Step 103, adjusting the parameters of the reward mechanism based on the optimization objectives to obtain an optimized reinforcement learning agent; the optimization objectives include minimizing costs, maximizing profits, and maximizing service levels.

[0045] In the same simulation environment, the reinforcement learning agent continuously trains and learns a set of neural network parameters. These parameters can understand the change rules of variables in the simulation environment, such as price changes within a month, and can learn the best production control strategy. Among them, the reward mechanism is used to guide the reinforcement learning agent to learn the optimal strategy and optimize production planning, inventory management and distribution strategy.

[0046] In the embodiment of the present application, the system not only optimizes a single goal (such as minimizing costs), but also considers multiple goals (such as maximizing profits and maximizing service levels) at the same time. By designing a reasonable reward mechanism and adjusting the parameters in the reward mechanism, the simulation environment is made as close to real life as possible, and the reinforcement learning agent can balance the impact of cost, profit and service level as much as possible, make reasonable choices, and thus improve the overall performance of the system.

[0047] Step 104, using the reward mechanism of the optimized reinforcement learning agent to calculate the revenue and cost in the upper-level production-inventory environment and the lower-level sales environment to obtain a reward value.

[0048] Step 105, based on the reward value, the optimized reinforcement learning agent is used to determine the production volume control strategy, and the production volume and inventory volume in the upper production-inventory environment and the distribution volume in the lower sales environment are adjusted according to the production volume control strategy.

[0049] Based on the above trained neural network, the reinforcement learning agent can make dynamic decisions based on real-time environmental data. Specifically, the reinforcement learning agent can adjust the decision at the next moment based on the reward at the previous moment. The neural network guides the reinforcement learning agent to make the optimal production volume selection, thereby achieving optimal control of the system.

[0050] In the embodiment of the present application, an upper production-inventory environment and a lower sales environment are constructed. The upper production-inventory environment is used to simulate production volume decision-making and inventory volume measurement, and the lower sales environment is used to simulate distribution conditions, thereby realizing dynamic simulation of complex production systems. A reinforcement learning agent is designed based on a preset reinforcement learning algorithm, and a reward mechanism for the reinforcement learning agent is set. The reinforcement learning algorithm is used to realize adaptive control of the dynamic environment, and can be adjusted according to real-time data, thereby improving the flexibility of the system. Afterwards, the parameters of the reward mechanism are adjusted with cost minimization, profit maximization, and service level maximization as the optimization goals, and the optimized reinforcement learning agent is obtained. Through multi-objective optimization, the conflicts between different goals are balanced, and the overall service level and customer satisfaction of the system are improved. Then, the reward mechanism of the optimized agent is used to calculate the income and cost of the upper production-inventory environment and the lower sales environment, and the reward value is obtained. Based on the reward value, the production volume control strategy is determined using the optimized reinforcement learning agent, and the production volume and inventory volume in the upper production-inventory environment and the distribution volume in the lower sales environment are adjusted according to the production volume control strategy, which can effectively reduce production costs, inventory holding costs, and out-of-stock costs.

[0051] In an optional embodiment, the upper-layer production-inventory environment includes a production volume simulation environment and an inventory volume simulation environment, and the lower-layer sales environment includes a distribution situation simulation environment;

[0052] Construct the upper-level production-inventory environment and the lower-level sales environment, including:

[0053] Construct a production simulation environment based on the key production parameters of each time period; the key production parameters include production volume and production cost;

[0054] Construct an inventory simulation environment based on key inventory parameters; key inventory parameters include production volume, inventory cost and maximum inventory volume;

[0055] A distribution simulation environment is constructed based on key distribution parameters; the key distribution parameters include market price, whether to select corresponding customers for distribution, and the distribution quantity at the customer's location.

[0056] Optionally, the simulation of the upper-level production-inventory environment is based on days, which simplifies the production and warehousing and outbound processes in real life. The simulation of the lower-level sales environment is based on minutes, which simulates door-to-door sales in real life.

[0057] In the upper-level production-inventory environment, a production volume simulation environment is constructed for the production link, which simulates various key production parameters in actual production. Optionally, the key production parameters include daily production volume (tons) and production cost (tons / yuan). The simulation of the production link is based on the values ​​in the real scene, with days as the unit, as the action result of the upper-level intelligent agent, to simulate the production volume.

[0058] Based on the production process, an inventory management inventory simulation environment is constructed. The environment receives the production volume as the inventory volume of the day, and immediately starts to sell it to the lower-level intelligent agent at the next moment after receiving it, taking into account factors such as inventory cost (tons / yuan), maximum inventory, and opportunity cost (i.e., production volume is less than sales volume). Among them, inventory cost is the cost of overstocked products when production volume is greater than sales volume.

[0059] In the lower-level sales environment, a distribution simulation environment for the distribution link is constructed to simulate the sales and distribution process of the product. Specifically, a grid world is simulated in minutes. In the grid world, the speed range is one minute per grid. Each grid intersection is a customer. The process of distributors visiting customers is simulated, including market prices, whether to choose to distribute to the corresponding customers, and the distribution quantity at the customer's place.

[0060] The model takes into account market demand fluctuations, that is, within a month, the market price changes every day, and the agent cannot know whether the price will increase or decrease the next day. At the same time, the lower-level sales environment uses different algorithms for distributors to plan customer access paths to simulate different sales strategies or the performance of different distributors, that is, algorithms with better results correspond to good sales strategies (good distributor performance), and poor algorithms correspond to poor sales strategies (poor distributor performance).

[0061] In an optional embodiment, the production volume simulation environment is used to simulate the production volume decision, and the production volume decision equation in the production volume simulation environment is:

[0062] m k =a k M max

[0063] Among them, m k is the production in k time steps, a k To formulate the parameters, M max For maximum production.

[0064] This equation is the way the reinforcement learning agent makes production volume decisions, that is, production is carried out according to the maximum production volume times the proposed parameter. The selection range of the proposed parameter is between 0 and 1, and there are 10 options discretized in 0.1 units. The reinforcement learning agent makes a decision on the proposed parameter among these ten options, thereby determining the production volume.

[0065] In an optional embodiment, the inventory simulation environment is used to simulate inventory measurement; the inventory measurement equation in the inventory simulation environment is:

[0066]

[0067] Among them, I k (0) represents the inventory before making a production decision in each initial stage, subject to the maximum inventory constraint. is a given value, m k is the production in k time steps, I k-1 (T) represents the inventory remaining in the k-1 time step, and T represents the last distribution time node in the k-1 time step.

[0068] Combining the above upper-level production-inventory environment and the lower-level sales environment, if it is the starting day in the inventory simulation environment, that is, k = 0, the inventory is a given value i; if it is any day after the starting day, the inventory is the upper-level production volume m k The remaining inventory of the previous day k-1 (T), where T represents the last distribution time node of k-1 on the previous day and the last inventory status of k-1 on the previous day.

[0069] In an optional embodiment, the distribution situation simulation environment is used to simulate the distribution situation, and the expression of the distribution situation in the distribution situation simulation environment is:

[0070] F(u kh (t)) = P k u kh (t)q kh (t)

[0071] Among them, u kh (t) is the sales situation to h customers at time k and time t, and the binary decision quantity with a structure of 0,1 corresponds to whether to choose to distribute to the corresponding customer; F is the income from sales; P k is the market price at time step k, q kh (t) is the quantity distributed to the customer;

[0072] For the distribution O at time k and time t k (t) is calculated as:

[0073]

[0074] Where H is the sum of all available customers.

[0075] The parameterized designs of the inventory simulation environment, the inventory simulation environment, and the distribution situation simulation environment in the embodiments of the present application can be adjusted and customized according to the needs of different enterprises.

[0076] In an optional embodiment, the formula of the reward mechanism is:

[0077]

[0078]

[0079] The equation is based on the Markov equation structure, where a k is the action of the agent at time step k, that is, the amount of production, A is the set of all actions, s k0 is the initial state, J is the target, the reward value is within k0 to K time steps, and the reward value r at each moment k The expected sum, E is the calculation of the expected value, Ω is the change rule from the previous time step to the next time step, is the learning rate, and the target J is subject to the maximum inventory The inventory at each time step k is I k The objective equation of the upper-level production-inventory environment is also the reward mechanism for the overall system.

[0080] Among them, the reward value r in each time step k The sales profit b at this time step k , Inventory cost c k and opportunity cost d k Decided:

[0081] r k =b k -c k -d k

[0082] In the embodiment of the present application, the sales profit b in the reward mechanism is adjusted with the optimization goals of minimizing costs, maximizing profits and maximizing service levels. k , Inventory cost c k and opportunity cost d k , and obtain the optimized reinforcement learning agent for subsequent production control decisions.

[0083] In an alternative embodiment, revenue includes sales revenue, and costs include production costs, inventory costs, and opportunity costs;

[0084] The reward mechanism of the optimized reinforcement learning agent is used to calculate the revenue and cost in the production simulation environment, inventory simulation environment, and distribution simulation environment to obtain the reward value, including:

[0085] The reward value is calculated based on sales revenue, production cost, inventory cost and opportunity cost. The calculation formula is:

[0086] r k =b k -c k -d k

[0087] Among them, r k represents the reward value at k time steps; b k =P k O k -λP k m k , represents the total production revenue, that is, the sales revenue minus the production cost at the k time step, and the sales revenue is the market price P k Multiply by total sales volume O k , the production cost is the proposed parameter λ multiplied by the market price P k Multiply by the production volume m k ;c k is the inventory cost, i.e., the storage cost of inventory I in k time steps; d k is the opportunity cost, that is, the cost of unsatisfied customer demand after k time steps are completed.

[0088] As an example, Figure 2 A schematic diagram of a hierarchical control system is shown in FIG. Figure 2 As shown in the figure, in the upper production-inventory environment, production decisions are guided based on inventory costs, opportunity costs, and market information, and production decisions are used to optimize production volume. Specifically, based on the inventory costs and opportunity costs of the current time step, a model-free reinforcement learning algorithm is used to make production decisions. The decision type is production volume, and the decision layer is a global decision. The goal is to maximize overall revenue and optimize the production volume of the next time step.

[0089] In the lower sales environment, based on the production volume and market information received from the upper production-inventory environment, the distribution strategy is guided, and the actual transaction volume is optimized through the distribution strategy. Among them, the actual transaction volume is optimized through the distribution strategy as follows: the decision type is to visit the customer distribution, the decision level is the lower decision, and the goal is to maximize sales revenue, and optimize the actual transaction volume of the next time step. Among them, the actual transaction volume of the current time step determines the inventory cost and opportunity cost of the current time step, which in turn affects the production volume and actual transaction volume of the next time step.

[0090] In the embodiment of the present application, a simulation experiment is designed based on a simulation environment. The experiment simulates the production-inventory-sales process in units of days within a month. The reinforcement learning agent is given a fixed amount of initial inventory on the first day and starts the bottom-level sales. From the second day, the agent decides how much to produce, and the environment performs a new simulation based on the production volume until the last day of the month. Figure 3 The following is a schematic diagram of the production-inventory-distribution process for one day during the simulation. Figure 3 As shown in the figure, in the upper-level intelligent agent part, production decisions are made based on the remaining inventory. The products produced are immediately stored in inventory for sale after pre-processing, and then wait for the start of the sales day. In the lower-level intelligent agent part, the main thing is to distribute the products that have been produced. The lower area is set as a grid world, and each grid intersection is a customer. In the simulation environment, a single distributor is set in the grid world to sell to customers at the grid intersection. The materials sold come from the products produced by the upper-level intelligent agent part on that day until the end of the sales day.

[0091] Figure 4 The training curve diagram of the reinforcement learning algorithm model in the embodiment of the present application is shown. From the curve, it can be seen that the production strategy generated by the reinforcement learning algorithm model in the embodiment of the present application performs under different lower-level distribution strategies, such as Figure 4 As shown, the total reward value is arranged in high and low according to different distribution strategies.

[0092] Figure 5 The figure shows the loss function convergence curve of the reinforcement learning algorithm model in the training process in the embodiment of the present application. The horizontal axis is the training set, the vertical axis is the loss value, DQN represents the deep reinforcement learning algorithm, and the low performance distribution team (LST), medium performance distribution team (MST) and high performance distribution team (HST) are three different distribution strategies. Figure 5 As shown, after 75 episodes of training, the network based on the deep reinforcement learning algorithm used in the embodiment of the present application has obvious convergence effect under different distribution strategies, which verifies the effectiveness of the deep reinforcement learning algorithm in the embodiment of the present application.

[0093] Table 1 below shows the performance comparison between the system based on the deep reinforcement learning algorithm proposed in the embodiment of this application and the traditional production system in the simulation environment test. LST is low performance distribution, MST is medium performance distribution, HST is high performance distribution; OUT is the traditional production mode, and DQN is the deep reinforcement learning algorithm proposed in the embodiment of this application.

[0094] Table 1

[0095]

[0096]

[0097] As shown in Table 1, the present application embodiment compares the traditional maximum inventory principle, that is, when the inventory falls below a certain warning line, full capacity production is carried out to the maximum amount. * , opportunity cost d * The deep reinforcement learning algorithm proposed in the embodiment of the present application is superior to the principle of maximum inventory production in all aspects. It shows that the method in the embodiment of the present application shows good adaptability and robustness in different scenarios.

[0098] Figure 6-8 The comprehensive performance of the trained neural network under the hierarchical control structure in the simulation environment is demonstrated. Figure 6 It reflects the comparison between the hierarchical production control structure proposed in the embodiment of the present application and the traditional fixed production structure in terms of monthly production volume. Figure 7 It reflects the comparison of monthly sales volume. Figure 8 It reflects the comparison of monthly inventory. Figure 6-8 It can be clearly reflected that the comprehensive performance of the embodiment of the present application under the three indicators is far superior to that of the traditional fixed production structure.

[0099] Optionally, the method in the embodiments of the present application can also be applied to logistics management, supply chain optimization, intelligent manufacturing and other fields, and can achieve wider application.

[0100] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides an adaptive production-inventory-distribution hierarchical structure control system, such as Fig. 9 As shown, the system comprises:

[0101] An environment construction module 901 is used to construct an upper-level production-inventory environment and a lower-level sales environment. The upper-level production-inventory environment is used to simulate production volume decision-making and inventory volume measurement, and the lower-level sales environment is used to simulate distribution conditions.

[0102] An agent construction module 902 is used to design a reinforcement learning agent based on a preset reinforcement learning algorithm and set a reward mechanism for the reinforcement learning agent;

[0103] Agent optimization module 903, used to adjust the parameters of the reward mechanism based on the optimization objectives to obtain an optimized reinforcement learning agent; the optimization objectives include minimizing costs, maximizing profits, and maximizing service levels;

[0104] Reward calculation module 904, used to calculate the income and cost in the upper production-inventory environment and the lower sales environment using the reward mechanism of the optimized reinforcement learning agent to obtain a reward value;

[0105] The strategy generation module 905 is used to determine the production volume control strategy based on the reward value using the optimized reinforcement learning agent, and adjust the production volume and inventory volume in the upper-level production-inventory environment and the distribution volume in the lower-level sales environment according to the production volume control strategy.

[0106] In the embodiment of the present application, an upper production-inventory environment and a lower sales environment are constructed. The upper production-inventory environment is used to simulate production volume decision-making and inventory volume measurement, and the lower sales environment is used to simulate distribution conditions, thereby realizing dynamic simulation of complex production systems. A reinforcement learning agent is designed based on a preset reinforcement learning algorithm, and a reward mechanism for the reinforcement learning agent is set. The reinforcement learning algorithm is used to realize adaptive control of the dynamic environment, and can be adjusted according to real-time data, thereby improving the flexibility of the system. Afterwards, the parameters of the reward mechanism are adjusted with cost minimization, profit maximization, and service level maximization as the optimization goals, and the optimized reinforcement learning agent is obtained. Through multi-objective optimization, the conflicts between different goals are balanced, and the overall service level and customer satisfaction of the system are improved. Then, the reward mechanism of the optimized agent is used to calculate the income and cost of the upper production-inventory environment and the lower sales environment, and the reward value is obtained. Based on the reward value, the production volume control strategy is determined using the optimized reinforcement learning agent, and the production volume and inventory volume in the upper production-inventory environment and the distribution volume in the lower sales environment are adjusted according to the production volume control strategy, which can effectively reduce production costs, inventory holding costs, and out-of-stock costs.

[0107] The adaptive production-inventory-distribution hierarchical structure control system provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0108] The adaptive production-inventory-distribution hierarchical structure control system of the embodiment of the present application can execute the adaptive production-inventory-distribution hierarchical structure control method provided in the embodiment of the present application, and the implementation principle thereof is similar. The actions performed by each module and unit in the adaptive production-inventory-distribution hierarchical structure control system in each embodiment of the present application correspond to the steps in the adaptive production-inventory-distribution hierarchical structure control method in each embodiment of the present application. For the detailed functional description of each module of the adaptive production-inventory-distribution hierarchical structure control system, please refer to the description in the corresponding adaptive production-inventory-distribution hierarchical structure control method shown in the previous text, which will not be repeated here.

[0109] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the adaptive production-inventory-distribution hierarchical structure control method shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the adaptive production-inventory-distribution hierarchical structure control method provided by the present application constructs an upper-level production-inventory environment and a lower-level sales environment. The upper-level production-inventory environment is used to simulate production volume decisions and inventory volume measurement, and the lower-level sales environment is used to simulate distribution conditions, thereby realizing dynamic simulation of complex production systems. A reinforcement learning agent is designed based on a preset reinforcement learning algorithm, and a reward mechanism for the reinforcement learning agent is set. The reinforcement learning algorithm can be used to realize adaptive control of a dynamic environment, and can be adjusted according to real-time data, thereby improving the flexibility of the system. Afterwards, with cost minimization, profit maximization, and service level maximization as the optimization goals, the parameters of the reward mechanism are adjusted to obtain an optimized reinforcement learning agent. Through multi-objective optimization, the conflicts between different goals are balanced, and the overall service level and customer satisfaction of the system are improved. Then, the reward mechanism of the optimized agent is used to calculate the income and cost of the upper-level production-inventory environment and the lower-level sales environment to obtain the reward value. Based on the reward value, the optimized reinforcement learning agent is used to determine the production volume control strategy, and the production volume and inventory volume in the upper-level production-inventory environment and the distribution volume in the lower-level sales environment are adjusted according to the production volume control strategy, which can effectively reduce production costs, inventory holding costs and out-of-stock costs.

[0110] In an optional embodiment, an electronic device is also provided, such as Fig.10 As shown, Fig.10 The electronic device 1000 shown may be a server, including: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the electronic device 1000 may further include a transceiver 1004. It should be noted that in actual applications, the transceiver 1004 is not limited to one, and the structure of the electronic device 1000 does not constitute a limitation on the embodiments of the present application.

[0111] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0112] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The memory 1003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0114] The memory 1003 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement the contents shown in the above method embodiment.

[0115] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0117] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0118] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0119] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0120] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0121] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0122] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the adaptive production-inventory-distribution hierarchical structure control method and system provided in the above-mentioned various optional implementations.

[0123] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The modules involved in the embodiments described in the present application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the environment construction module can also be described as "an environment construction module for constructing an upper-level production-inventory environment and a lower-level sales environment".

[0126] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. An adaptive production-inventory-distribution hierarchical structure control method, characterized in that: The method comprises: Constructing an upper-layer production-inventory environment and a lower-layer sales environment, wherein the upper-layer production-inventory environment is used to simulate production volume decision-making and inventory volume measurement, and the lower-layer sales environment is used to simulate distribution conditions; Designing a reinforcement learning agent based on a preset reinforcement learning algorithm, and setting a reward mechanism for the reinforcement learning agent; Adjust the parameters of the reward mechanism based on the optimization goal to obtain an optimized reinforcement learning agent; the optimization goal includes minimizing costs, maximizing profits and maximizing service levels; Calculate the revenue and cost of the upper-level production-inventory environment and the lower-level sales environment using the reward mechanism of the optimized reinforcement learning agent to obtain a reward value; Based on the reward value, the optimized reinforcement learning agent is used to determine a production volume control strategy, and the production volume and inventory volume in the upper-level production-inventory environment and the distribution volume in the lower-level sales environment are adjusted according to the production volume control strategy.

2. The adaptive production-inventory-distribution hierarchical structure control method according to claim 1, characterized in that: The upper-layer production-inventory environment includes a production quantity simulation environment and an inventory quantity simulation environment, and the lower-layer sales environment includes a distribution situation simulation environment; The construction of the upper-level production-inventory environment and the lower-level sales environment includes: Constructing the production volume simulation environment based on the key production parameters of each time period; the key production parameters include production volume and production cost; Constructing the inventory simulation environment based on key inventory parameters; the key inventory parameters include the production volume, inventory cost and maximum inventory; The distribution situation simulation environment is constructed based on key distribution parameters; the key distribution parameters include market price, whether to select corresponding customers for distribution, and the distribution quantity at the customer.

3. The adaptive production-inventory-distribution hierarchical structure control method according to claim 1, characterized in that: The formula for the reward mechanism is: subject to: Among them, a k is the action of the agent at time step k, that is, the amount of production, A is the set of all actions, s k0 is the initial state, J is the target, the reward value is within k0 to K time steps, and the reward value r at each moment k The expected sum, E is the calculation of the expected value, Ω is the change rule from the previous time step to the next time step, is the learning rate, and the target J is subject to the maximum inventory The inventory at each time step k is I k .

4. The adaptive production-inventory-distribution hierarchical structure control method according to claim 3, characterized in that: The revenue includes sales revenue, and the cost includes production cost, inventory cost and opportunity cost; The reward mechanism of the optimized reinforcement learning agent is used to calculate the income and cost of the production simulation environment, the inventory simulation environment, and the distribution simulation environment to obtain the reward value, including: The reward value is calculated based on the sales revenue, production cost, inventory cost and opportunity cost, and the calculation formula is: r k =b k -c k -d k Among them, r k represents the reward value at k time steps; b k =P k O k -λP k m k , represents the total production revenue, that is, the sales revenue minus the production cost at the k time step, and the sales revenue is the market price P k Multiply by total sales volume O k , the production cost is the proposed parameter λ multiplied by the market price P k Multiply by the production volume m k ;c k is the inventory cost; d k For opportunity cost.

5. The adaptive production-inventory-distribution hierarchical structure control method according to claim 2, characterized in that: The production volume simulation environment is used to simulate the production volume decision. The production volume decision equation in the production volume simulation environment is: m k =a k M max Among them, m k is the production in k time steps, a k To formulate the parameters, M max For maximum production.

6. The adaptive production-inventory-distribution hierarchical structure control method according to claim 2, characterized in that: The inventory simulation environment is used to simulate inventory measurement; the inventory measurement equation in the inventory simulation environment is: Among them, I k (0) represents the inventory before making a production decision in each initial stage, subject to the maximum inventory constraint. is a given value, m k is the production in k time steps, I k-1 (T) represents the inventory remaining in the k-1 time step, and T represents the last distribution time node in the k-1 time step.

7. The adaptive production-inventory-distribution hierarchical structure control method according to claim 2, characterized in that: The distribution situation simulation environment is used to simulate the distribution situation. The expression of the distribution situation in the distribution situation simulation environment is: F(u kh (t))=P k u kh (t)q kh (t) Among them, u kh (t) is the sales situation to h customers at time k and time t, and the binary decision quantity with a structure of 0,1 corresponds to whether to choose to distribute to the corresponding customer; F is the income from the sales situation; P k is the market price at time step k, q kh (t) the quantity distributed to the said customer; For the distribution O at time k and time t k (t) is calculated as: Where H is the sum of all available customers.

8. An adaptive production-inventory-distribution hierarchical structure control system, characterized in that: The system comprises: An environment construction module, used to construct an upper-layer production-inventory environment and a lower-layer sales environment, wherein the upper-layer production-inventory environment is used to simulate production volume decision-making and inventory volume measurement, and the lower-layer sales environment is used to simulate distribution conditions; An agent building module, used to design a reinforcement learning agent based on a preset reinforcement learning algorithm and set a reward mechanism for the reinforcement learning agent; An agent optimization module, used to adjust the parameters of the reward mechanism based on optimization objectives to obtain an optimized reinforcement learning agent; the optimization objectives include minimizing costs, maximizing profits, and maximizing service levels; A reward calculation module, used to calculate the income and cost of the upper-level production-inventory environment and the lower-level sales environment using the reward mechanism of the optimized reinforcement learning agent to obtain a reward value; A strategy generation module is used to determine a production volume control strategy based on the reward value using an optimized reinforcement learning agent, and adjust the production volume and inventory volume in the upper-level production-inventory environment and the distribution volume in the lower-level sales environment according to the production volume control strategy.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

Images