Supply chain resource control optimization method based on deep reinforcement learning

Abstract

The invention relates to the technical field of supply chains, and discloses a supply chain resource control optimization method based on deep reinforcement learning, and the method comprises the steps: constructing a multi-stage supply chain environment; constructing a soft actor-commentator agent for action decomposition; training the soft actor-commentator agent by using the multi-stage supply chain environment to obtain a trained soft actor-commentator agent; and deploying the trained soft actor-commentator agent to a target supply chain, and outputting an optimal action decision. The invention provides an intelligent agent based on deep reinforcement learning for the problems of demand fluctuation, complex structure, high-dimensional state and action space, complex reward function and the like. According to the method, the future market trend can be predicted by analyzing historical production order demands and stockout records, and based on real-time supply chain inventory data, an intelligent agent can dynamically adjust an inventory management strategy, so that various problems in supply chain resource cooperative control optimization are effectively solved, the risks of inventory overstock and stockout are effectively reduced, and the system performance is improved. The response speed and flexibility of the supply chain are improved, and finally the overall operation efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of supply chain resource control optimization, and in particular to a supply chain resource control optimization method based on deep reinforcement learning. Background Art

[0002] According to data provided by the China Federation of Logistics and Purchasing, the ratio of China's total social logistics costs to GDP in the first half of 2024 was 14.2%, while the proportion of logistics costs to GDP in the United States was about 7%. Compared with developed countries, China's logistics costs account for a higher proportion of GDP, which shows that China still has room for improvement in supply chain efficiency compared with developed countries. Supply chain resource collaborative control is one of the key technologies for enterprises to gain market competitive advantages. It focuses on exploring the most efficient product production, transportation and distribution paths to ensure that market demand is accurately met within the established planning framework. This technology covers every link from the conversion of raw materials to final products, forming a complete process chain.

[0003] The prior art discloses a multi-cycle inventory optimization management method based on the supply chain. In this method, the downstream node enterprises adopt the (s, Q) production strategy to control production and inventory; at the same time, a corresponding dynamic inventory model is also established for the midstream node enterprises. The upstream and downstream node enterprises work together to minimize the total inventory costs of the entire supply chain. This method is different from the non-cooperative situation where each node enterprise only pursues the lowest inventory cost. This method can also derive the optimal production preparation period, optimal production period and optimal production volume in the entire supply chain within a production cycle. In this prior art, the (s, Q) strategy is used to control production and inventory. In this strategy, when the inventory level drops to the reorder point When the purchase action is triggered, a fixed amount of inventory. These strategies have been widely used in practice because they are simple and easy to understand. However, there are many difficulties in the process of collaborative control and optimization of supply chain resources. First, the supply chain network is full of uncertain factors, such as demand fluctuations and supply disruptions, which greatly increase the complexity of decision-making; second, global economic integration has made the supply chain structure more complex, and it is necessary to comprehensively consider more frequent interactions and collaborations between manufacturers and retailers to achieve the optimization of overall operations. The above difficulties make the traditional rule-based methods perform poorly. Summary of the invention

[0004] The purpose of the present invention is to overcome the problems existing in the prior art and provide a supply chain resource control optimization method based on deep reinforcement learning. The present invention can dynamically adjust the inventory management strategy and effectively deal with various problems in the collaborative control optimization of supply chain resources; it not only reduces inventory backlog and out-of-stock risks, but also improves the response speed and flexibility of the supply chain, and ultimately achieves a significant improvement in overall operational efficiency.

[0005] In order to achieve the above object, the present invention provides a supply chain resource control optimization method based on deep reinforcement learning, the method comprising the following steps: Building a multi-stage supply chain environment; Building an action-decomposed soft actor-critic agent; training the soft actor-critic agent using the multi-stage supply chain environment to obtain a trained soft actor-critic agent; The trained soft actor-critic agent is deployed to the target supply chain to output the optimal action decision.

[0006] Furthermore, the multi-stage supply chain environment includes at least one raw material supplier, at least one manufacturer and multiple retailers, and at least the state space is designed , demand function, action space , state transfer , reward function One of the state space is the set of all possible states in the multi-stage supply chain environment; the demand function is the demand change of each retailer with random and seasonal characteristics; the action space is the set of all possible actions in the multi-stage supply chain environment; the state transition The supply chain environment is the action taken by the agent at the current state, and the multi-stage supply chain environment follows the state transition Update the environment and give the state of the next moment; the reward function Under the current state of the multi-stage supply chain environment, for the actions taken by the agent, the environment should follow the reward function to give the current reward.

[0007] Furthermore, the state space The possible states at the current time t The specific expressions are as follows:

[0008]

[0009]

[0010]

[0011] in, Represents the state at the current time t; represents the inventory level vector of the manufacturer and each retailer at the current time t; represents the out-of-stock level vector of each retailer that fails to meet customer demand at the current time t; represents the demand level vector of each retailer at the first three moments t-3, , And so on; represents the manufacturer’s inventory level at the current time t; represents the inventory level of retailer i at the current time t; represents the out-of-stock level of unmet demand of retailer i at the current time t; represents the customer demand level of retailer i at the current time t.

[0012] Furthermore, the demand function is specifically as follows:

[0013]

[0014] in, represents the demand level of retailer i at the current time t; Indicates the maximum demand level for each retailer; represents the circumference constant of pi; Indicates the offset; It represents the maximum number of steps in the supply chain environment; represents the random disturbance term, which conforms to Uniform distribution .

[0015] Furthermore, the action space Possible actions at the current time t The specific expressions are as follows:

[0016] in, represents the action at the current time t; represents the number of products produced by the manufacturer at the current time t; It represents the quantity of products shipped by the manufacturer to the retailer i at the current time t.

[0017] Furthermore, the state transfer It is deterministic in the described multi-stage supply chain environment, and the state transition is achieved according to the material balance constraints , specifically expressed as follows:

[0018] State Transfer The inventory level vectors of manufacturers and retailers at the current time t are , the out-of-stock level vector of the retailer’s unmet demand at the current time t Update, specifically as follows:

[0019]

[0020]

[0021]

[0022]

[0023] in, Indicates the manufacturer's maximum inventory capacity; represents the maximum inventory capacity of retailer i.

[0024] Furthermore, the multi-stage supply chain environment follows the reward function Give the current reward , specifically expressed as follows

[0025] in, Represents the final reward given by the environment; Indicates the selling price of the product; represents the sales quantity of product i of retailer i; It represents the production cost of the product; represents the storage cost of product i for the manufacturer and retailer; represents the penalty cost when retailers fail to meet demand; represents the transportation cost per truckload delivered to the retailer; Indicates the maximum cargo capacity of each truck; The symbol indicates that the value is rounded up; Specifically, Represents the total sales revenue of the product at the current moment, where Preset parameters for the supply chain environment, It will be calculated based on the sales of each retailer. The specific calculation formula is as follows:

[0026] represents the total production cost of the product at the current moment, where Preset parameters for the supply chain environment, For Action part of; Represents the total storage cost of the product at the current moment, where Preset parameters for the supply chain environment, Status part of represents the total penalty cost for unsatisfied demand products at the current moment, where Preset parameters for the supply chain environment, Status part of; represents the total transportation cost of transporting products to each retailer at the current moment, where and Preset parameters for the supply chain environment, For Action part of.

[0027] Furthermore, the action-decomposed soft actor-critic agent includes a state feature dense network, an action feature dense network, a soft actor network, two soft critic main networks and two soft critic target networks, wherein the state feature dense network is used to extract the features corresponding to the states in the multi-stage supply chain environment, which are recorded as the first features; the soft actor network is used to decide actions based on the first features; the action feature dense network is used to extract the features of the decision-making actions based on the actions, which are recorded as the second features; the soft critic main network is used to calculate the Q value at the current time t based on the first features and the second features ; The soft critic target network is used to calculate the Q value at the next time t+1 , the Q value represents the expected value of the future cumulative reward that can be obtained after selecting a certain action in the current state. The specific formula is as follows:

[0028]

[0029] in, represents the agent's strategy, represents the total discounted reward in the future, Represents the discount factor.

[0030] Furthermore, the soft actor network is used to decide an action based on the first feature, specifically including: inputting the first feature into the soft actor network to obtain a first action, inputting the first action into an intermediate layer of the soft actor network to obtain a second action, concatenating the first action and the second action, and outputting a decision action.

[0031] Further, using the multi-stage supply chain environment to train the soft actor-critic agent to obtain a trained soft actor-critic agent specifically includes: (1) Randomly initialize the parameters of the state feature dense network, the parameters of the action feature dense network, the parameters of the soft actor network, and the parameters of the two soft critic main networks, and copy the parameters of the two soft critic main networks to the parameters of the two soft critic target networks, and use the experience replay pool Capacity Set to 0 to initialize the adaptive temperature coefficient ; (2) Collect training samples. The soft actor-critic agent will calculate the state of the current supply chain environment. Input to Strategy Select the corresponding action , using the action Interact with the multi-stage supply chain environment and observe the immediate rewards returned by the supply chain environment and the state at the next moment , the four-tuple sample Store in experience replay pool and put the experience back into the pool Capacity Set to , determine whether the number of steps in the multi-stage supply chain environment reaches the maximum number of steps set If it is reached, the supply chain environment is reset, otherwise the above sampling process is repeated until the experience replay pool Capacity Reach the number of samples required for training; (3) Use the experience replay pool The samples in update the parameters of the state feature dense network, the parameters of the action feature dense network, the parameters of the soft actor network, and the parameters of the two soft critic main networks; (4) At fixed time intervals, use the parameters of the two soft critic main networks described in step (3) and Update the parameters of the two soft critic target networks and , the specific update method is as follows:

[0032]

[0033] in, Indicates the soft update parameter, which is used to control the update amplitude; (5) Determine whether the number of training times reaches the set number. If so, stop training. Otherwise, go to step (2) to obtain a trained soft actor-critic agent.

[0034] Compared with the prior art, the present invention has the following beneficial effects: Aiming at the challenges in the field of collaborative control and optimization of supply chain resources, such as demand fluctuations, complex structures, high-dimensional state and action spaces, and complex reward functions, the present invention proposes an intelligent agent based on deep reinforcement learning. The intelligent agent uses a feature extraction module and an action decomposition mechanism to enable it to predict future market trends by analyzing historical production order demands and out-of-stock records, and can dynamically adjust inventory management strategies based on real-time supply chain inventory data, effectively addressing various problems in collaborative control and optimization of supply chain resources, not only reducing inventory backlogs and out-of-stock risks, but also improving the response speed and flexibility of the supply chain, ultimately achieving a significant improvement in overall operational efficiency. We also construct a reasonable multi-stage supply chain environment that can provide the agent with sufficient information to guide it to make wise decisions. We also introduce appropriate perturbations, which not only enhances the agent's strategy generalization ability and enables it to adapt to new environments, but also improves the agent's overall robustness and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a supply chain resource control optimization method based on deep reinforcement learning according to Embodiment 1 of the present invention; Figure 2 is the multi-stage supply chain environment of Example 1 of the present invention; Figure 3 is a supply chain environment dynamics flow chart of Example 1 of the present invention; Figure 4 This is the overall structure diagram of the action decomposition soft actor-critic network of Example 1 of the present invention; Figure 5 This is a feature dense network structure diagram of Example 1 of the present invention; Figure 6 This is a diagram of a soft actor network structure of Example 1 of the present invention; Figure 7 This is a diagram of the soft critic network structure of Example 1 of the present invention. DETAILED DESCRIPTION

[0036] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0037] Example 1 like Figure 1As shown, a supply chain resource control optimization method based on deep reinforcement learning in a preferred embodiment of the present invention includes: S1: Building a multi-stage supply chain environment; In a possible embodiment, if Figure 2-3 , the multi-stage supply chain environment constructed for agent learning includes a raw material supplier, a manufacturer, and multiple retailers. Among them, the manufacturer needs to order raw materials from the raw material supplier, and after the manufacturer processes the raw materials into products, it ships the products to multiple retailers. Customers will buy products from retailers, and customers have a demand for products from retailers. At each moment in the supply chain environment, the agent needs to decide how many products the manufacturer should produce and how many products should be shipped to each retailer to ensure that customer demand is met to the greatest extent possible to maximize profits.

[0038] The supply chain environment constructed above conforms to the Markov decision process. Specifically, the constructed supply chain environment also designs the state space , demand function, action space , state transfer , reward function and discount factor, where the state space Refers to all possible states in the supply chain environment The set of possible states at the current time t The specific expressions are as follows:

[0039]

[0040]

[0041]

[0042] in, Represents the state at the current time t; represents the inventory level vector of the manufacturer and each retailer at the current time t; represents the out-of-stock level vector of each retailer that fails to meet customer demand at the current time t; represents the demand level vector of each retailer at the first three moments t-3, , And so on; represents the manufacturer’s inventory level at the current time t; represents the inventory level of retailer i at the current time t; represents the out-of-stock level of unmet demand of retailer i at the current time t; represents the customer demand level of retailer i at the current time t.

[0043] Design a demand function with stochastic and seasonal characteristics for each retailer, which is expressed as follows:

[0044]

[0045] in, represents the demand level of retailer i at the current time t; Indicates the maximum demand level for each retailer; represents the circumference constant of pi; Indicates the offset; It represents the maximum number of steps in the supply chain environment; represents the random disturbance term, which conforms to Uniform distribution The designed demand function can approximately simulate the random and seasonal customer demand in the real supply chain environment. The setting ensures that each time the supply chain environment is reset, the customer demand level will not be exactly the same, further increasing the randomness of the environment to better simulate the real environment.

[0046] Action Space It refers to the set of all possible actions in the supply chain environment, the possible actions at the current time t The specific expressions are as follows:

[0047] in, represents the action at the current time t; represents the number of products produced by the manufacturer at the current time t; It represents the quantity of products shipped by the manufacturer to the retailer i at the current time t.

[0048] The state transfer Refers to the state of the supply chain environment at the current time t Next, for the actions taken by the agent , the environment should follow the state transition Update the environment and give the state at the next moment In the supply chain environment constructed by the present invention, the state transition probability is deterministic, and the state transition is realized according to the material balance constraint. , specifically expressed as follows:

[0049] State Transfer Mainly for the inventory level vector of manufacturers and retailers at the current time t , the out-of-stock level vector of the retailer’s unmet demand at the current time t Update, specifically as follows:

[0050]

[0051]

[0052]

[0053]

[0054] in, Indicates the manufacturer's maximum inventory capacity; represents the maximum inventory capacity of retailer i.

[0055] The reward function Refers to the state of the supply chain environment at the current time t Next, for the actions taken by the agent , the environment should follow the reward function Give the reward for this moment The specific expressions are as follows:

[0056] in, Represents the final reward given by the environment; Indicates the selling price of the product; represents the sales quantity of product i of retailer i; It represents the production cost of the product; represents the storage cost of product i for the manufacturer and retailer; represents the penalty cost when retailers fail to meet demand; represents the transportation cost per truckload delivered to the retailer; Indicates the maximum cargo capacity of each truck; The symbol indicates that the value is rounded up.

[0057] Item 1 Represents the total sales revenue of the product at the current moment, where Preset parameters for the supply chain environment, It will be calculated based on the sales of each retailer. The specific calculation formula is as follows:

[0058] Item 2 represents the total production cost of the product at the current moment, where Preset parameters for the supply chain environment, For Action part of; Item 3 Represents the total storage cost of the product at the current moment, where Preset parameters for the supply chain environment, Status part of; Item 4 represents the total penalty cost for unsatisfied demand products at the current moment, where Preset parameters for the supply chain environment, Status part of; Item 5 represents the total transportation cost of transporting products to each retailer at the current moment, where and Preset parameters for the supply chain environment, For Action part of.

[0059] Discount Factor It is a measure of the importance of future rewards relative to immediate rewards in a supply chain environment. A real number in the interval. Discount factor The core idea is that future rewards are not as valuable as immediate rewards because there is uncertainty and delay in the realization of future rewards. The agent is based on the state of the supply chain environment at the current time t. , decide the action that is suitable for the current state , which determines the number of products that the manufacturer should produce at the current time t , and the quantity of the product that should be shipped to each retailer The agent uses the action it has decided Interact with the supply chain environment.

[0060] like Figure 3 As shown, at time t, the supply chain environment receives the action decided by the agent Finally, the dynamic flow chart of the supply chain environment consists of the following sequence of events: (1) The supply chain environment follows the actions of the agent , extracting the actions that determine the manufacturer’s production of products , that is, at time t, the manufacturer produces products.

[0061] (2) The supply chain environment takes actions based on the agent’s decisions , extract the quantity of products shipped by the decision manufacturer to each retailer , that is, to deliver to retailer 1 at time t Products shipped to Retailer 2 products, and so on.

[0062] (3) In the supply chain environment, after retailers receive products from manufacturers, they meet customers’ demand for products. If the inventory level of each current retailer If all the demand can be met, there will be no shortage level , otherwise calculate the out-of-stock level .

[0063] (4) In the supply chain environment, after retailers meet customer needs, they start calculating profits at that moment. Total Profit Total sales revenue of the product Subtract the total production cost of producing the product , minus the total warehousing costs of storing the product , minus the total penalty cost of the unmet demand product , minus the total shipping costs to ship the product to each retailer .

[0064] (5) The supply chain environment changes according to the above status Perform state transfer updates for the environment.

[0065] S2: Building a soft actor-critic agent for action decomposition; In a feasible embodiment, the action-decomposed soft actor-critic agent includes a state feature dense network, an action feature dense network, a soft actor network, two soft critic master networks and two soft critic target networks. The structure of the action-decomposed soft actor-critic agent is shown in Figure 4 The state feature dense network is used to extract the features corresponding to the states in the multi-stage supply chain environment, which are recorded as the first features; the soft actor network is used to decide actions based on the first features. The structure of the soft actor network is shown in Figure 5 ; The action feature dense network is used to extract the features of the decision action according to the action, and the state (action) feature dense network is used to extract the features of the decision action according to the action. Figure 5 , recorded as the second feature,; the soft critic main network is used to calculate the Q value of the current time t according to the first feature and the second feature ; The soft critic target network is used to calculate the Q value at the next time t+1 , the soft critic network structure can be found in Figure 6 , Q value represents the expected value of future cumulative rewards that can be obtained after selecting a certain action in the current state. The specific formula is as follows:

[0066]

[0067] in, represents the agent's strategy, represents the total discounted reward in the future, Represents the discount factor.

[0068] The above strategy It is the core of the action-decomposed soft actor-critic algorithm agent, which consists of a state feature dense network and a soft actor network. The working principle is as follows: The current state The input state feature dense network extracts features and inputs them into the soft actor network. The soft actor network will first output a part of the final action, then input this part of the action into the middle layer of the network for calculation, and then output the remaining part of the final action. Finally, the two are spliced ​​together to form the action of the final decision of the intelligent agent. .

[0069] S3: training the soft actor-critic agent using the multi-stage supply chain environment to obtain a trained soft actor-critic agent; In a feasible embodiment, the step S3 specifically includes: S3.1: Initialize the network and experience replay pool, and set the parameters of the state feature dense network mentioned above , parameters of the action feature dense network , parameters of the soft actor network , parameters of the two soft critic main networks and Randomly initialize and copy the parameters of the two soft critic main networks and The parameters of the two soft critic target networks are and . Replay experience to the pool Capacity Set to 0 to initialize the adaptive temperature coefficient .

[0070] S3.2: Environmental sampling and sample storage, the agent observes the current state of the supply chain environment , which is input into the agent's strategy Select Action , use actions Interact with the supply chain environment and observe the immediate rewards returned by the supply chain environment and the state at the next moment . The four-tuple sample Store in experience replay pool and put the experience back into the pool Capacity Set to If the supply chain environment reaches the maximum number of steps set , then reset the supply chain environment. Repeat Step 2 until the experience replay pool Capacity When the number of samples required for training is reached, jump to Step 3.

[0071] S3.3: Feature dense network, soft actor network and soft critic main network parameter update: S3.3.1: From the Experience Replay Pool Randomly select a batch of four-tuple samples from ; S3.3.2: Extract the next state using state feature dense network The soft actor network is used to calculate the best possible action at the next moment. and this action The policy entropy , using the action feature dense network to extract the best action that may be taken at the next moment The features of the two soft critic target networks are used to calculate the state-action pair at the next moment. Two Q values and . Choose the smaller of the two Q values , according to the calculated smaller Q value , the policy entropy of the best possible action at the next moment and the current moment reward Calculate target value , the specific formula is as follows:

[0072] S3.3.3: Extract the current state using the state feature dense network The features of the action feature dense network are used to extract the current action The features of the two soft critic main networks are used to calculate the current state-action pair Two Q values and ; S3.3.4: Set the loss function of the soft critic main network for the two Q values and and target value Construct the loss function of the soft critic main network. The specific formula is as follows:

[0073] Apply gradient descent to the above loss function to update the state feature dense network parameters , Action Feature Dense Network Parameters and the two soft critic main network parameters and ; S3.3.5: Extract the current state using the state feature dense network The features of the soft actor network are used to calculate the best possible action at the current moment. and policy entropy , using the action feature dense network to extract the best possible action at the current moment The features of the two soft critic target networks are used to calculate the current state-action pair Two Q values and , choose the smaller Q value ; S3.3.6: Set the loss function of the soft actor network according to the smaller Q value obtained in Step 3.5 and the policy entropy of the best possible action at the current moment Construct the loss function of the soft actor network. The specific formula is as follows:

[0074] Apply gradient descent to the above loss function to update the soft actor network parameters ; S3.3.7: Setting the adaptive temperature coefficient The loss function is based on the policy entropy of the best action that can be taken at the current moment. Building Adaptive Temperature Coefficients The loss function is as follows:

[0075] in, is the target policy entropy, which is a hyperparameter of this algorithm. Apply the gradient descent method to the above loss function to update the adaptive temperature coefficient .

[0076] S3.4: Soft critic target network parameter update: At fixed time intervals, the parameters of the two soft critic main networks are used and Update the parameters of the two soft critic target networks and , the specific update method is as follows:

[0077]

[0078] in, Indicates the soft update parameter, which is used to control the update amplitude.

[0079] S3.5: Training loop and end: If the number of training rounds reaches the set number, the training ends, otherwise jump to S3.2.

[0080] S4: Deploy the trained soft actor-critic agent to the target supply chain and output the optimal action decision.

[0081] In one possible embodiment, after training, the action-decomposed soft actor-critic algorithm agent will obtain an optimal strategy The strategy Accept state output and output the best action under the current state. Deploy the trained agent to the real supply chain environment. The agent receives the state provided by the real supply chain environment in real time and uses the strategy Calculate the optimal action decision and guide the real supply chain to produce and transport products based on the action decision.

[0082] This embodiment aims at challenges in the field of collaborative control and optimization of supply chain resources, such as demand fluctuations, complex structures, high-dimensional state and action spaces, and complex reward functions. A deep reinforcement learning-based intelligent agent is proposed. The intelligent agent uses a feature extraction module and an action decomposition mechanism to enable it to predict future market trends by analyzing historical production order demands and out-of-stock records, and can dynamically adjust inventory management strategies based on real-time supply chain inventory data, effectively addressing various problems in collaborative control and optimization of supply chain resources, not only reducing inventory backlogs and out-of-stock risks, but also improving the response speed and flexibility of the supply chain, and ultimately achieving a significant improvement in overall operational efficiency. We also construct a reasonable multi-stage supply chain environment that can provide the agent with sufficient information to guide it to make wise decisions. We also introduce appropriate perturbations, which not only enhances the agent's strategy generalization ability and enables it to adapt to new environments, but also improves the agent's overall robustness and adaptability.

[0083] Example 2 The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the supply chain resource control optimization method based on deep reinforcement learning described in Example 1 is implemented.

[0084] In summary, the embodiments of the present invention provide a supply chain resource control optimization method and storage medium based on deep reinforcement learning. It aims at the challenges in the field of collaborative control optimization of supply chain resources, such as demand fluctuations, complex structures, high-dimensional state and action spaces, and complex reward functions, and proposes a deep reinforcement learning-based intelligent agent. The intelligent agent uses a feature extraction module and an action decomposition mechanism to enable it to predict future market trends by analyzing historical production order demand and out-of-stock records, and can dynamically adjust inventory management strategies based on real-time supply chain inventory data, effectively respond to various problems in the collaborative control optimization of supply chain resources, not only reduce inventory backlogs and out-of-stock risks, but also improve the response speed and flexibility of the supply chain, and ultimately achieve a significant improvement in overall operational efficiency; it also constructs a reasonable multi-stage supply chain environment that can provide sufficient information for the intelligent agent to guide it to make wise decisions, and introduces appropriate disturbances, which not only enhances the strategy generalization ability of the intelligent agent, enabling it to adapt to new environments, but also improves the overall robustness and adaptability of the intelligent agent.

[0085] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A supply chain resource control optimization method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Building a multi-stage supply chain environment; Building an action-decomposed soft actor-critic agent; training the soft actor-critic agent using the multi-stage supply chain environment to obtain a trained soft actor-critic agent; The trained soft actor-critic agent is deployed to the target supply chain to output the optimal action decision.

2. According to claim 1, a supply chain resource control optimization method based on deep reinforcement learning is characterized in that: The multi-stage supply chain environment includes at least one raw material supplier, at least one manufacturer and multiple retailers, and at least a state space is designed , demand function, action space , state transfer , reward function One of the state space is the set of all possible states in the multi-stage supply chain environment; the demand function is the demand change of each retailer with random and seasonal characteristics; the action space is the set of all possible actions in the multi-stage supply chain environment; the state transition The supply chain environment is the action taken by the agent at the current state, and the multi-stage supply chain environment follows the state transition Update the environment and give the state of the next moment; the reward function Under the current state of the multi-stage supply chain environment, for the actions taken by the agent, the environment should follow the reward function to give the current reward.

3. A supply chain resource control optimization method based on deep reinforcement learning according to claim 2, characterized in that: The state space The possible states at the current time t The specific expressions are as follows: in, Represents the state at the current time t; represents the inventory level vector of the manufacturer and each retailer at the current time t; represents the out-of-stock level vector of each retailer that fails to meet customer demand at the current time t; represents the demand level vector of each retailer at the first three moments t-3, , And so on; represents the manufacturer’s inventory level at the current time t; represents the inventory level of retailer i at the current time t; represents the out-of-stock level of unmet demand of retailer i at the current time t; represents the customer demand level of retailer i at the current time t.

4. A supply chain resource control optimization method based on deep reinforcement learning according to claim 2, characterized in that: The demand function is as follows: in, represents the demand level of retailer i at the current time t; Indicates the maximum demand level for each retailer; represents the circumference constant of pi; Indicates the offset; It represents the maximum number of steps in the supply chain environment; represents the random disturbance term, which conforms to Uniform distribution .

5. The supply chain resource control optimization method based on deep reinforcement learning according to claim 2 is characterized in that: The action space Possible actions at the current time t The specific expressions are as follows: in, represents the action at the current time t; represents the number of products produced by the manufacturer at the current time t; represents the quantity of products shipped by the manufacturer to retailer i at the current time t.

6. A supply chain resource control optimization method based on deep reinforcement learning according to claim 2, characterized in that: The state transfer It is deterministic in the described multi-stage supply chain environment, and the state transition is achieved according to the material balance constraints , specifically expressed as follows: State Transfer The inventory level vectors of manufacturers and retailers at the current time t are , the out-of-stock level vector of the retailer’s unmet demand at the current time t Update, specifically as follows: in, Indicates the manufacturer's maximum inventory capacity; represents the maximum inventory capacity of retailer i.

7. A supply chain resource control optimization method based on deep reinforcement learning according to claim 2, characterized in that: The multi-stage supply chain environment follows the reward function Give the current reward , specifically expressed as follows in, Represents the final reward given by the environment; Indicates the selling price of the product; represents the sales quantity of product i of retailer i; It represents the production cost of the product; represents the storage cost of product i for the manufacturer and retailer; represents the penalty cost when retailers fail to meet demand; represents the transportation cost per truckload delivered to the retailer; Indicates the maximum cargo capacity of each truck; The symbol indicates that the value is rounded up; Specifically, Represents the total sales revenue of the product at the current moment, where Preset parameters for the supply chain environment, It will be calculated based on the sales of each retailer. The specific calculation formula is as follows: represents the total production cost of the product at the current moment, where Preset parameters for the supply chain environment, For Action part of; Represents the total storage cost of the product at the current moment, where Preset parameters for the supply chain environment, Status part of represents the total penalty cost for unsatisfied demand products at the current moment, where Preset parameters for the supply chain environment, Status part of; represents the total transportation cost of transporting products to each retailer at the current moment, where and Preset parameters for the supply chain environment, For Action part of.

8. The supply chain resource control optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The action-decomposed soft actor-critic agent includes a state feature dense network, an action feature dense network, a soft actor network, two soft critic main networks and two soft critic target networks. The state feature dense network is used to extract the features corresponding to the states in the multi-stage supply chain environment, which are recorded as the first features; the soft actor network is used to decide actions based on the first features; the action feature dense network is used to extract the features of the decision-making actions based on the actions, which are recorded as the second features; the soft critic main network is used to calculate the Q value at the current time t based on the first features and the second features ; The soft critic target network is used to calculate the Q value at the next time t+1 , the Q value represents the expected value of the future cumulative reward that can be obtained after selecting a certain action in the current state. The specific formula is as follows: in, represents the strategy of the agent, represents the total discounted reward in the future, Represents the discount factor.

9. The supply chain resource control optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The soft actor network is used to decide an action according to the first feature, specifically including: inputting the first feature into the soft actor network to obtain a first action, inputting the first action into an intermediate layer of the soft actor network to obtain a second action, concatenating the first action and the second action, and outputting a decision action.

10. A supply chain resource control optimization method based on deep reinforcement learning according to any one of claims 1 to 9, characterized in that: Using the multi-stage supply chain environment to train the soft actor-critic agent to obtain a trained soft actor-critic agent specifically includes: (1) Randomly initialize the parameters of the state feature dense network, the parameters of the action feature dense network, the parameters of the soft actor network, and the parameters of the two soft critic main networks, and copy the parameters of the two soft critic main networks to the parameters of the two soft critic target networks, and use the experience replay pool Capacity Set to 0 to initialize the adaptive temperature coefficient ; (2) Collect training samples. The soft actor-critic agent will calculate the state of the current supply chain environment. Input to Strategy Select the corresponding action , using the action Interact with the multi-stage supply chain environment and observe the immediate rewards returned by the supply chain environment and the state at the next moment , the four-tuple sample Store in experience replay pool and put the experience back into the pool Capacity Set to , determine whether the number of steps in the multi-stage supply chain environment reaches the maximum number of steps set If it is reached, the supply chain environment is reset, otherwise the above sampling process is repeated until the experience replay pool Capacity Reach the number of samples required for training; (3) Use the experience replay pool The samples in update the parameters of the state feature dense network, the parameters of the action feature dense network, the parameters of the soft actor network, and the parameters of the two soft critic main networks; (4) At fixed time intervals, use the parameters of the two soft critic main networks described in step (3) and Update the parameters of the two soft critic target networks and , the specific update method is as follows: in, Indicates the soft update parameter, which is used to control the update amplitude; (5) Determine whether the number of training times reaches the set number. If so, stop training. Otherwise, go to step (2) to obtain a trained soft actor-critic agent.

Images