WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning

Abstract

The present invention relates to the field of supply chain management and optimization, and discloses a WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning, which is characterized by comprising the following steps: constructing an interactive WEEE recycling and reuse supply chain virtual system; using a double-layer deep Q network to train an intelligent agent, and the intelligent agent gradually optimizes the strategy in the interactive learning process with the established supply chain virtual system until the supply chain virtual system obtains a stable operating cost; deploying the trained intelligent agent to the WEEE recycling and reuse supply chain real system to perform supply chain management in real time. The method disclosed by the present invention can ensure the normal operation of the supply chain while reducing inventory backlogs, improving the response speed and flexibility of the supply chain, thereby minimizing the total operating cost of the supply chain and promoting the sustainable development of WEEE recycling and reuse enterprises.

Description

Technical Field

[0001] The present invention relates to the field of supply chain management and optimization, and in particular to a WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning. Background Art

[0002] With the rapid growth of waste electrical and electronic equipment (WEEE) worldwide, environmental managers and resource policy makers are facing severe challenges. WEEE mainly includes discarded household appliances, computers, communication equipment and other electronic products, which contain toxic and hazardous substances such as lead, mercury and cadmium. If improperly handled, it will have a serious impact on soil, water and air quality, and thus endanger human health. In order to address this challenge, many companies implement the principle of extended producer responsibility and reuse WEEE products by establishing a reverse supply chain, playing a vital role in the circular economy. In the WEEE recycling and reuse supply chain, inventory control optimization directly affects cost control, resource efficiency and the overall sustainability of the supply chain. An effective inventory control strategy can reduce inventory backlogs, reduce storage costs, and improve the responsiveness and flexibility of the supply chain.

[0003] Similar to conventional reverse supply chains, the main challenges of WEEE recycling and reuse supply chain management and control focus on the uncertainty of recycling volume and demand. Factory production demand is affected by seasonal and random market product demand, showing significant uncertainty. In addition, when products reach the end of their life cycle, it means that they will be transformed into potential recyclable resources, that is, the amount of WEEE to be recycled in the market. Similar to the uncertainty of market product orders, the amount of WEEE to be recycled in the market is affected by multiple factors such as consumer behavior, product durability, policies and regulations, and is also uncertain.

[0004] In order to deal with the above uncertainties, some researchers have used prediction models to predict future recycling volumes, transforming the uncertain inventory control problem into an optimization problem in a deterministic scenario for solution. However, this method relies on scenarios predicted in advance for solution and lacks feedback and adjustment mechanisms for real-time data. Therefore, existing research schemes mostly use reordering policies based on historical data or experience. The reordering strategy determines whether it is necessary to produce and order products or raw materials by comparing the current inventory level with the preset order point. For example, in the prior art, some people have adopted a (T, S) strategy for the inventory control problem of the reusable gas cylinder supply chain, where T represents the review cycle and S represents the target inventory level. The results of the study show that this method is similar to the results obtained by the simulation optimization method, which takes longer.

[0005] Although optimization schemes based on reordering strategies have shown certain effects, these schemes rely on fixed parameters and cannot dynamically adapt to changes in the current market environment. In recent years, researchers have conducted a series of application studies on deep reinforcement learning for two-stage supply chain inventory control systems with stochastic and seasonal demand characteristics. They comprehensively considered key factors such as production constraints, warehouse capacity, and replenishment lead time. Experimental results show that compared with the (s,Q) reordering strategy, reinforcement learning algorithms perform better in coping with the dynamic complexity of the supply chain. However, in complex reverse supply chain scenarios, the application of reinforcement learning is still relatively small, and its potential needs to be further explored. Summary of the invention

[0006] In order to solve the problems of inventory backlog and out-of-stock risks, poor supply chain response speed and flexibility, and low overall operational efficiency in supply chain management and control, the present invention provides a WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning. By training and applying reinforcement learning agents, it is enabled to make decisions on material flow, collection, disassembly and reuse strategies for each role in the system, so as to reduce inventory backlogs and improve the response speed and flexibility of the supply chain while ensuring the normal operation of the supply chain, thereby minimizing the total operating cost of the supply chain and promoting the sustainable development of WEEE recycling and reuse enterprises.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning includes the following steps:

[0009] Step 1: Build an interactive WEEE recycling and reuse supply chain virtual system;

[0010] Step 2: Use a double-layer deep Q network to train the agent. The agent gradually optimizes the strategy through interactive learning with the established supply chain virtual system until the supply chain virtual system achieves a stable operating cost.

[0011] Step 3: Deploy the trained agent into the real WEEE recycling and reuse supply chain system to conduct real-time supply chain management and control.

[0012] In the above scheme, in step 1, the interactive WEEE recycling and reuse supply chain virtual system constructed includes three roles: collection center, pretreatment center and product manufacturing plant; the collection center and pretreatment center adopt the inventory-based production mode, the collection center collects WEEE as planned, puts it into the collection center WEEE warehouse, and waits for subsequent transportation to the pretreatment center WEEE warehouse, the pretreatment center disassembles WEEE and recycles waste plastic into granules, after disassembling and reusing WEEE, the obtained recycled plastic granules are stored in the recycled plastic granule warehouse, waiting for orders from product manufacturing plants; the disassembled scrap metals are sold to metal processing plants; the product manufacturing plant adopts the order-based production mode, places plastic orders to the pretreatment center and external suppliers in advance to reserve raw materials, and then uses the plastic raw material inventory to meet production order needs.

[0013] In the above scheme, in step 2, the two-layer deep Q network includes a main network and a target network, the main network is used to select actions and calculate the Q value in the current state; the target network is used to calculate the maximum Q value of the next state; the Q value represents the expected value of the cumulative reward that can be obtained after selecting a certain action in the current state.

[0014] In the above scheme, the training process of step 2 is as follows:

[0015] Step 1: Initialization:

[0016] First, initialize two main networks and target networks with the same structure, as well as the experience replay pool P , and set the reward discount factor and Exploration rate in policy The initial value of

[0017] Step 2: Environmental interaction:

[0018] The agent observes the current supply chain status , and then by Strategy with a certain probability Next random action After selecting the action, the collection, disassembly and reuse decision corresponding to the action is input into the supply chain virtual system for interaction, and the negative supply chain operation cost returned by the system is obtained, i.e., the reward. R t , and observe the new status of the supply chain , and finally Deposit into experience replay pool P , used to subsequently update the main network parameters ;

[0019] Step 3: Experience playback and main network update:

[0020] During the experience replay process, from the experience replay pool P Randomly select a batch of samples from ; Use the main network to estimate the state Take action The Q value, that is, the expected cumulative return in the future ; Then, the target network is used to calculate the current state Target value ; Finally, calculate the loss function and use the gradient descent method to update the main network parameters ;

[0021] Step 4: Target network update:

[0022] At fixed time intervals, the parameters of the main network are Copy to the target network to update the parameters of the target network ,Right now ;

[0023] Step 5: Exploration Rate attenuation:

[0024] After each training round, gradually reduce The value gradually reduces the exploration behavior and allows the algorithm to make more use of the learned knowledge to select actions;

[0025] Step6: Training convergence:

[0026] As training progresses, the main network gradually learns to accurately estimate the Q value of each state-action pair, thereby selecting the action with the highest Q value and adjusting in each interaction in the direction of maximizing the long-term cumulative reward; when the cumulative reward of a round of length T When the growth of becomes stable, the reinforcement learning agent converges and the training is completed.

[0027] In a further technical solution, the supply chain status It is expressed as follows:

[0028] ;

[0029] in, represents the WEEE warehouse inventory of the collection center at time t, Y t represents the amount of WEEE shipped from the collection center to the pre-processing center but not yet delivered during the time step t-1,…,t-LC, LC represents the delivery period from the collection center to the pre-processing center, represents the WEEE warehouse inventory of the pre-processing center at time t, represents the amount of WEEE dismantled and reused in the pre-processing center at time t-1, It represents the inventory of recycled plastic granules in the pre-processing center at time t. It represents the in-transit inventory of plastic pellets ordered by the product manufacturing plant from the pre-processing center and external suppliers but not yet delivered at time t. It represents the inventory of plastic raw materials in the warehouse of the product manufacturing factory at time t. represents the plastic raw material demand for product production orders at time t-σ, represents the plastic raw material demand for product production orders at time t-1, represents the amount of WEEE to be recycled in the market at time t-σ, It represents the amount of WEEE to be recycled in the market at time t-1, and σ represents the length of time of the observed historical demand and historical amount to be recycled.

[0030] In a further technical solution, the action It is expressed as follows:

[0031] ;

[0032] in, x t represents the WEEE collection plan established at time t, y t represents the amount of WEEE transported from the collection center to the pre-treatment center at time t, z t represents the amount of WEEE disassembled and reused in the pre-processing center at time t, q t represents the number of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t, p t Represents the quantity of plastic particles purchased by the product manufacturing factory from external suppliers at time t.

[0033] In a further technical solution, in Step 2, the specific action selection process is as follows:

[0034] (1) Probabilistic random selection of actions;

[0035] (2) The probability of obtaining the action corresponding to the maximum Q value through the main network.

[0036] In a further technical solution, the reward R t It is expressed as follows:

[0037] ;

[0038] Among them, the first represents the collection cost of WEEE, represents the collection cost per unit of WEEE, represents the actual WEEE collection amount at time t;

[0039] Item 2 represents the variable cost of pre-processing of WEEE, represents the variable cost of pre-processing per unit of WEEE, z t It represents the amount of WEEE dismantled and reused by the pre-processing center at time t;

[0040] Item 3 represents the cost of setting up the pre-processing production line, It represents the startup cost of setting up the unit disassembly line. l t Indicates the number of production lines started at time t;

[0041] Item 4 represents the transportation cost of WEEE, represents the unit WEEE transportation cost, y t It represents the amount of WEEE transported from the collection center to the pre-treatment center at time t;

[0042] Item 5 represents the transportation cost of recycled plastic pellets, represents the transportation cost per unit of recycled plastic pellets, q t It represents the quantity of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t;

[0043] Item 6 represents the cost of purchasing plastic pellets from external suppliers, Represents the unit plastic pellet procurement cost, p t It represents the quantity of plastic pellets purchased by the product manufacturing factory from external suppliers at time t;

[0044] Indicates that during the planning period, At the end of time t, each warehouse needs to pay the inventory holding cost. h i represents the holding cost of each warehouse unit inventory, i is the warehouse index, the collection center WEEE warehouse is represented by index 0, the pre-processing center WEEE warehouse is represented by index 1, the pre-processing center recycled plastic granule warehouse is represented by index 2, and the product manufacturing plant plastic raw material warehouse is represented by index 3; h 3 represents the unit inventory holding cost of the plastic raw material warehouse of the product manufacturing plant, It represents the inventory of each warehouse at the end of time t and the beginning of time t+1. It represents the inventory of plastic raw material warehouse of product manufacturing factory at the end of time t; Represents the cost of delayed production orders due to shortage of plastic raw materials, It represents the cost of production order delay caused by shortage of unit plastic raw materials;

[0045] Indicates the end of the planning period, i.e. When you buy a product, you need to pay additional inventory management costs. represents the unit inventory management cost of each warehouse for the redundant inventory at the end of the planning period, represents the inventory of each warehouse at the end of the planning period T, represents the management cost of redundant inventory in the plastic raw material warehouse, It represents the warehouse inventory of plastic raw materials in the product manufacturing plant at the end of the planning period T; It means that at the end of the planning period, additional production order loss costs need to be paid. represents the cost of production order loss due to unit plastic raw materials at the end of the planning period;

[0046] The last item sj t represents the proceeds from the sale of metal briquettes, Represents the revenue from selling a unit of metal briquettes, j t It represents the number of metal briquettes generated by dismantling and recycling WEEE at time t.

[0047] In a further technical solution, the target value It is expressed as follows:

[0048] ;

[0049] in, R t Representation status Take action The reward obtained immediately after is the discount factor, γ represents the discount factor, whose value is between 0 and 1, which is used to measure the importance of future rewards relative to current rewards; Represents the next state given by the target network Next best move The corresponding Q value.

[0050] In the above scheme, the specific method of step 3 is as follows:

[0051] (1) The status of the supply chain reality system Input into the main network to obtain a set of Q values, each element of which corresponds to a different action A Q value;

[0052] (2) Use the argmax function to select the action with the largest Q value, that is, ;

[0053] (3) Based on Guide the supply chain reality system to carry out recycling plan formulation, transportation, disassembly and reuse operations.

[0054] Through the above technical solution, the WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning provided by the present invention has the following beneficial effects:

[0055] The present invention designs, trains and applies reinforcement learning agents, enabling them to infer future market conditions by observing historical production order demands and recycling volumes, and dynamically adjust inventory control strategies based on real-time supply chain inventory and in-transit inventory, thereby reducing inventory backlogs and out-of-stock risks, and improving supply chain response speed and flexibility, thereby improving overall operational efficiency.

[0056] In addition, the present invention avoids the capital occupation caused by excessive WEEE collection, disassembly and reuse, and procurement by formulating reasonable collection, disassembly and reuse, and raw material procurement management and control strategies, effectively reduces unnecessary WEEE collection, external procurement of raw materials and other operating costs, improves corporate profitability, and achieves sustainable development. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0058] Figure 1 The present invention is a flowchart of a reinforcement learning-based WEEE recycling and reuse supply chain management and optimization method disclosed in an embodiment of the present invention.

[0059] Figure 2 Schematic diagram of the constructed WEEE recycling and reuse supply chain virtual system.

[0060] Figure 3 This is a flow chart of supply chain system dynamics.

[0061] Figure 4 Schematic diagram of the training of a double-layer deep Q network. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0063] The present invention provides a WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning, such as Figure 1As shown, the following steps are included:

[0064] Step 1: Build an interactive WEEE recycling and reuse supply chain virtual system.

[0065] like Figure 2 As shown in the figure, the constructed interactive WEEE recycling and reuse supply chain virtual system includes three roles: collection center, pretreatment center and product manufacturing plant; the collection center and pretreatment center adopt the inventory-based production mode, the collection center collects WEEE as planned and puts it into the WEEE warehouse of the collection center, waiting for subsequent transportation to the WEEE warehouse of the pretreatment center, the pretreatment center disassembles WEEE and recycles and granulates waste plastics, after disassembling and reusing WEEE, the obtained recycled plastic particles are stored in the recycled plastic particle warehouse, waiting for orders from product manufacturing plants; the disassembled scrap metals are sold to metal processing plants; the product manufacturing plant adopts the order-based production mode, places plastic orders to the pretreatment center and external suppliers in advance for raw material reserves, and then uses the plastic raw material inventory to meet production order requirements; the WEEE and plastic raw materials involved in the above tasks will be delivered after a fixed delivery period.

[0066] like Figure 3 As shown in Figure 1, the dynamic flow chart of the above supply chain virtual system at time t consists of the following event sequence:

[0067] (1) Based on the current state of the system at time t, formulate x t The collection program for the quantity and the amount of WEEE shipped to the pre-processing center y t , determine the amount of WEEE to be processed by the pre-treatment center z t and the amount of recycled plastic pellets shipped to product manufacturing plants q t At the same time, the amount of plastic pellets purchased by the product manufacturing plant from external suppliers is planned p t .

[0068] (2) Collection centers according to the planned number x t Carry out WEEE collection and report the actual amount collected The WEEE is stored in the WEEE warehouse and then y t The actual amount of WEEE collected is Plan for collection x t and the market volume to be recycled The smaller value of .

[0069] (3) The pre-processing center receives the transported Amount of WEEE after dismantling and recycling The number of WEEE started to complete the dismantling and recycling task is l t , , the production capacity of each production line is .

[0070] (4) The recycled plastic pellet warehouse receives the plastic pellets that were disassembled and reused at the last moment. Amount of recycled plastic pellets, then transported p t Amount of recycled plastic pellets to product manufacturing plants.

[0071] (4) Product manufacturing factory sends products to external suppliers p t A certain number of plastic purchase orders are placed and external suppliers begin preparing the goods for shipment.

[0072] (5) The plastic raw material warehouse of the product manufacturing plant first receives the amount of recycled plastic particles transported at time t-LQ and the amount of plastic pellets purchased from external sources at time t-LP , and then use existing inventory to meet the plastic raw material needs of product production orders , production orders that do not meet demand for plastic raw materials will be delayed or lost.

[0073] (6) Calculate the operating cost C at time t t :

[0074] ;

[0075] Among them, the first represents the collection cost of WEEE, represents the collection cost per unit of WEEE, represents the actual WEEE collection amount at time t;

[0076] Item 2 represents the variable cost of pre-processing of WEEE, represents the variable cost of pre-processing per unit of WEEE, z t It represents the amount of WEEE dismantled and reused by the pre-processing center at time t;

[0077] Item 3 represents the cost of setting up the pre-processing production line, It represents the startup cost of setting up the unit disassembly line. l tIndicates the number of production lines started at time t;

[0078] Item 4 represents the transportation cost of WEEE, represents the unit WEEE transportation cost, y t It represents the amount of WEEE transported from the collection center to the pre-treatment center at time t;

[0079] Item 5 represents the transportation cost of recycled plastic pellets, represents the transportation cost per unit of recycled plastic pellets, q t It represents the quantity of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t;

[0080] Item 6 represents the cost of purchasing plastic pellets from external suppliers, Represents the unit plastic pellet procurement cost, p t It represents the quantity of plastic pellets purchased by the product manufacturing factory from external suppliers at time t;

[0081] Indicates that during the planning period, At the end of time t, each warehouse needs to pay the inventory holding cost. h i represents the holding cost of each warehouse unit inventory, i is the warehouse index, the collection center WEEE warehouse is represented by index 0, the pre-processing center WEEE warehouse is represented by index 1, the pre-processing center recycled plastic granule warehouse is represented by index 2, and the product manufacturing plant plastic raw material warehouse is represented by index 3; h 3 represents the unit inventory holding cost of the plastic raw material warehouse of the product manufacturing plant, It represents the inventory of each warehouse at the end of time t and the beginning of time t+1. It represents the inventory of plastic raw material warehouse of product manufacturing factory at the end of time t; Represents the cost of delayed production orders due to shortage of plastic raw materials, It represents the cost of production order delay caused by shortage of unit plastic raw materials;

[0082] Indicates the end of the planning period, i.e. When you buy a product, you need to pay additional inventory management costs. represents the unit inventory management cost of each warehouse for the redundant inventory at the end of the planning period, represents the inventory of each warehouse at the end of the planning period T, represents the management cost of redundant inventory in the plastic raw material warehouse, It represents the warehouse inventory of plastic raw materials in the product manufacturing plant at the end of the planning period T; It means that at the end of the planning period, additional production order loss costs need to be paid. Represents the cost of lost production orders per unit of plastic raw material at the end of the planning period.

[0083] (7) Determine the state at the next moment. The inventory transfer status of the system at moment t+1 is expressed as follows:

[0084] ;

[0085] in, It represents the initial WEEE warehouse inventory of the collection center at time t+1. It represents the initial WEEE warehouse inventory of the collection center at time t, represents the actual amount of WEEE collected by the collection center at time t, Indicates the WEEE warehouse capacity of the collection center, represents the amount of WEEE transported from the collection center to the pre-treatment center at time t, It represents the WEEE warehouse inventory of the initial pre-processing center at time t+1. I 1,t It represents the WEEE warehouse inventory of the pre-treatment center at time t. represents the amount of WEEE transported from the collection center to the pre-treatment center at time t-LC, Indicates the WEEE warehouse capacity of the pre-processing center, represents the amount of WEEE dismantled and reused by the pre-processing center at time t, It represents the inventory of recycled plastic granules in the initial pre-processing center at time t+1. It represents the inventory of recycled plastic granules in the initial pre-processing center at time t. It represents the number of recycled plastic particles generated by the dismantling and recycling of WEEE in the pre-processing center at time t-1. Indicates the warehouse capacity of recycled plastic particles in the pre-processing center, represents the number of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t, It represents the warehouse inventory of plastic raw materials at the primary product manufacturing plant at time t+1. It represents the warehouse inventory of plastic raw materials in the primary product manufacturing plant at time t. Indicates the number of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t-LQ, represents the amount of plastic pellets ordered by the product manufacturing plant from external suppliers at time t-LP, Indicates the plastic raw material warehouse capacity of the product manufacturing plant, Represents the plastic raw material demand for production orders at time t.

[0086] The WEEE warehouse inventory at the collection center at the beginning of time t+1 is equal to the inventory at the beginning of time t plus the collection amount, minus the amount shipped to the pre-processing center. The WEEE warehouse inventory at the pre-processing center is equal to the inventory at the beginning of time t, plus the amount shipped from the collection center at time t-LC, minus the amount disassembled and reused. The inventory of the recycled plastic particle warehouse at the pre-processing center is equal to the inventory at the beginning of time t, plus the amount obtained by disassembly and reuse at time t-1, minus the amount shipped to the product manufacturing plant. The plastic raw material inventory of the product manufacturing plant is equal to the inventory at the beginning of time t plus the number of plastic particles received, minus the plastic raw material demand for product production orders. Due to the internal information sharing mechanism of supply chain enterprises, the collection center and the pre-processing center will not have negative inventory, while the product manufacturing plant faces uncertain production raw material demand and may be out of stock.

[0087] Step 2: Use a two-layer deep Q network to train the intelligent agent. The intelligent agent gradually optimizes the strategy through interactive learning with the established supply chain virtual system until the supply chain virtual system achieves stable operating costs.

[0088] In the traditional DQN algorithm, the same neural network is used to estimate the Q value of the current state and the maximum Q value of the next state at the same time, which has certain instability and high variance problems. To solve this problem, the Double DQN introduces two neural networks: the main network ( ) and the target network ( ), the main network is used to select actions and calculate the Q value in the current state; the target network is used to calculate the maximum Q value of the next state; the Q value represents the expected value of the cumulative reward that can be obtained after selecting a certain action in the current state.

[0089] During the training process of the reinforcement learning agent, the agent continuously adjusts its supply chain strategy to maximize the cumulative reward by interacting with the constructed supply chain virtual system, such as Figure 4 As shown, the specific training process of the double-layer deep Q network is as follows:

[0090] Step 1: Initialization:

[0091] First, initialize two main networks and target networks with the same structure, as well as the experience replay pool , and set the reward discount factor and Exploration rate in policy The initial value of .

[0092] Step 2: Environmental interaction:

[0093] The agent observes the current supply chain status , and then by Strategy with a certain probability Next random action After selecting the action, the collection, disassembly and reuse decision corresponding to the action is input into the supply chain virtual system for interaction, and the negative supply chain operation cost returned by the system is obtained, i.e., the reward. R t , and observe the new status of the supply chain , and finally Deposit into experience replay pool P , used to subsequently update the main network parameters .

[0094] The specific action selection process is as follows:

[0095] (1) Probabilistic random selection of actions;

[0096] (2) The probability of obtaining the action corresponding to the maximum Q value through the main network.

[0097] Step 3: Experience playback and main network update:

[0098] During the experience replay process, from the experience replay pool P Randomly select a batch of samples from ; Use the main network to estimate the state Take action The Q value, that is, the expected cumulative return in the future , which can help the agent evaluate the quality of the current state and action and guide its decision-making process; then, the target network is used to calculate the current state Target value ; Target value It is expressed as follows:

[0099] ;

[0100] in, R t Representation status Take action The reward obtained immediately after is the discount factor, γ represents the discount factor, whose value is between 0 and 1, which is used to measure the importance of future rewards relative to current rewards; Represents the next state given by the target network Next best move The corresponding Q value.

[0101] Finally, calculate the loss function , and use the gradient descent method to update the main network parameters .

[0102] Step 4: Target network update:

[0103] Since the parameters of the main network During the training process, the parameters of the main network are continuously updated. In order to stabilize the training, the parameters of the main network are updated at fixed time intervals. Copy to the target network to update the parameters of the target network ,Right now .

[0104] Step 5: Exploration Rate attenuation:

[0105] The strategy is used to balance the trade-off between exploration and exploitation. The larger the value, the more the algorithm tends to explore the environment. After each training round, gradually reduce The value gradually reduces the exploration behavior and allows the algorithm to make more use of the learned knowledge to select actions.

[0106] Step6: Training convergence:

[0107] As training progresses, the main network gradually learns to accurately estimate the Q value of each state-action pair, thereby selecting the action with the highest Q value and adjusting in each interaction in the direction of maximizing the long-term cumulative reward; when the cumulative reward of a round of length T When the growth of becomes stable, the reinforcement learning agent finally converges and the training is completed.

[0108] The supply chain inventory control problem is a random sequential decision problem that satisfies the Markov property, that is, the next state can be obtained from the current state. Therefore, the problem is modeled as a Markov decision process (MDP). MDP consists of three tuples, namely, state space, action space, and reward function.

[0109] 1. State Space

[0110] The state space includes the existing inventory of each warehouse, the number of recycled plastic particles obtained by disassembly and reuse at the last moment, and the in-transit inventory involved in each ordering task. In addition, in order to enable the agent to better cope with the uncertain supply chain environment, the market recycling volume and demand value of the last σ time steps are additionally considered to capture the dynamic characteristics of the environment.

[0111] Specifically, the supply chain status It is expressed as follows:

[0112] ;

[0113] in, represents the WEEE warehouse inventory of the collection center at time t, represents the amount of WEEE shipped from the collection center to the pre-processing center but not yet delivered during the time step t-1,…,t-LC, LC represents the delivery period from the collection center to the pre-processing center; It is expressed as follows:

[0114] ;

[0115] in, represents the amount of WEEE transported from the collection center to the pre-treatment center at time t-1, It represents the amount of WEEE transported from the collection center to the pre-treatment center at time t-LC;

[0116] represents the WEEE warehouse inventory of the pre-processing center at time t, represents the amount of WEEE dismantled and reused at time t-1, It represents the inventory of recycled plastic granules in the pre-processing center at time t. It represents the in-transit inventory of plastic pellets ordered by the product manufacturing plant from the pre-processing center and external suppliers but not yet delivered at time t, expressed as follows:

[0117] ;

[0118] in, It represents the quantity of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t-1. represents the number of plastic pellets ordered by the product manufacturing plant from external suppliers at time t-1, It indicates the quantity of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t-LQ. It represents the quantity of plastic pellets ordered by the product manufacturing plant from external suppliers at time t-LP;

[0119] It represents the inventory of plastic raw materials in the warehouse of the product manufacturing factory at time t. represents the plastic raw material demand for product production orders at time t-σ, represents the plastic raw material demand for product production orders at time t-1;

[0120] In order to simulate the random and seasonal (uncertain) demand for plastic raw materials, a sine function with a random component is considered based on the seasonal demand model of the product, and the coefficient η is introduced to represent the number of plastic particles required to produce a unit of product. The plastic raw material demand for the product production order at time t is It is expressed as follows:

[0121] ;

[0122] Among them, η represents the number of plastic particles required to produce a unit product. Indicates the change in the quantity of product production orders. Indicates the phase, Indicates the product production order change cycle, represents the noise in the production order quantity of the product;

[0123] represents the amount of WEEE to be recycled in the market at time t-σ, It represents the amount of WEEE to be recycled in the market at time t-1, and σ represents the length of time of the observed historical demand and historical amount to be recycled.

[0124] Assume that the product life cycle is L. After L expires, the product enters the terminal state (EOF). In addition, a recycling rate α is introduced to estimate the amount of WEEE to be recycled in the market. Based on this assumption, the amount of WEEE to be recycled in the market at time t is It is expressed as follows:

[0125] ;

[0126] in, Represents the noise in the market's pending recovery volume.

[0127] 2. Action Space

[0128] The present invention designs a continuous action space with independent boundaries. By mapping the continuous action space to discrete actions, the challenge of high-dimensional action space in inventory problems can be effectively solved and the training efficiency can be improved. Specifically, the lower limit of each action value is set to zero. The upper limit of the collection plan and transportation plan is set to the storage capacity of the collection center. The upper limit of the preprocessing quantity is set to the maximum processing level. The upper limit of ordering plastic particles is set to the capacity of the plastic raw material warehouse of the product manufacturing plant. It is worth noting that if the collection, transportation and ordering quantities exceed the remaining capacity of the warehouse, the environment will directly discard the excess quantity, resulting in unnecessary waste of funds. This implicitly punishes the agent and encourages it to learn strategies to prevent excessive collection, transportation and ordering. In addition, in order to ensure that the inventory of the collection center and the preprocessing center is non-negative, when the transportation and preprocessing quantities exceed the existing inventory, they will be limited to the maximum available inventory of the warehouse.

[0129] Specific actions It is expressed as follows:

[0130] ;

[0131] in, x trepresents the WEEE collection plan established at time t, y t represents the amount of WEEE transported from the collection center to the pre-treatment center at time t, z t represents the amount of WEEE disassembled and reused in the pre-processing center at time t, q t represents the number of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t, p t Represents the quantity of plastic particles purchased by the product manufacturing factory from external suppliers at time t.

[0132] 3. Reward Function

[0133] The optimization goal of the present invention is to minimize the total operating cost, while the goal of reinforcement learning is to maximize the cumulative reward. In order to make these two goals consistent, the reward function is defined as a negative cost. This conversion transforms the cost minimization problem into a standard reinforcement learning task, allowing the agent to achieve the optimization goal by learning a strategy that minimizes the cumulative cost over the planning time period.

[0134] Specifically, reward R t It is expressed as follows:

[0135] ;

[0136] Among them, the first represents the collection cost of WEEE, represents the collection cost per unit of WEEE, represents the actual WEEE collection amount at time t;

[0137] Item 2 represents the variable cost of pre-processing of WEEE, represents the variable cost of pre-processing per unit of WEEE, z t It represents the amount of WEEE dismantled and reused by the pre-processing center at time t;

[0138] Item 3 represents the cost of setting up the pre-processing production line, It represents the startup cost of setting up the unit disassembly line. l t Indicates the number of production lines started at time t;

[0139] Item 4 represents the transportation cost of WEEE, represents the unit WEEE transportation cost, y tIt represents the amount of WEEE transported from the collection center to the pre-treatment center at time t;

[0140] Item 5 represents the transportation cost of recycled plastic pellets, represents the transportation cost per unit of recycled plastic pellets, q t It represents the quantity of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t;

[0141] Item 6 represents the cost of purchasing plastic pellets from external suppliers, Represents the unit plastic pellet procurement cost, p t It represents the quantity of plastic pellets purchased by the product manufacturing factory from external suppliers at time t;

[0142] Indicates that during the planning period, At the end of time t, each warehouse needs to pay the inventory holding cost. h i represents the holding cost of each warehouse unit inventory, i is the warehouse index, the collection center WEEE warehouse is represented by index 0, the pre-processing center WEEE warehouse is represented by index 1, the pre-processing center recycled plastic granule warehouse is represented by index 2, and the product manufacturing plant plastic raw material warehouse is represented by index 3; h 3 represents the unit inventory holding cost of the plastic raw material warehouse of the product manufacturing plant, It represents the inventory of each warehouse at the end of time t and the beginning of time t+1. It represents the inventory of plastic raw material warehouse of product manufacturing factory at the end of time t; Represents the cost of delayed production orders due to shortage of plastic raw materials, It represents the cost of production order delay caused by shortage of unit plastic raw materials;

[0143] Indicates the end of the planning period, i.e. When you buy a product, you need to pay additional inventory management costs. represents the unit inventory management cost of each warehouse for the redundant inventory at the end of the planning period, represents the inventory of each warehouse at the end of the planning period T, represents the management cost of redundant inventory in the plastic raw material warehouse, It represents the warehouse inventory of plastic raw materials in the product manufacturing plant at the end of the planning period T; It means that at the end of the planning period, additional production order loss costs need to be paid. represents the cost of production order loss due to unit plastic raw materials at the end of the planning period;

[0144] The last item sj t represents the proceeds from the sale of metal briquettes, Represents the revenue from selling a unit of metal briquettes, j t It represents the number of metal briquettes generated by dismantling and recycling WEEE at time t.

[0145] Step 3: Deploy the trained agent into the real WEEE recycling and reuse supply chain system to conduct real-time supply chain management and control.

[0146] The specific method is as follows:

[0147] (1) The status of the supply chain reality system Input into the main network to obtain a set of Q values, each element of which corresponds to a different action Q value;

[0148] (2) Use the argmax function to select the action with the largest Q value, that is, ;

[0149] (3) Based on Guide the supply chain real system to carry out recycling plan formulation, transportation, disassembly and reuse operations. Finally, observe the operation of the supply chain (operating costs, out-of-stock conditions, etc.) to verify the effectiveness of the management and control strategy.

[0150] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reinforcement learning-based optimization method for WEEE recycling supply chain management and control. It is characterized in that The steps include: Step 1: Build an interactive WEEE recycling and reuse supply chain virtual system; Step 2: Use a double-layer deep Q network to train the agent. The agent gradually optimizes the strategy through interactive learning with the established supply chain virtual system until the supply chain virtual system achieves a stable operating cost. Step 3: Deploy the trained agent to the real WEEE recycling supply chain system to conduct real-time supply chain management and control. In step 1, the interactive WEEE recycling and reuse supply chain virtual system constructed includes three roles: collection center, pre-processing center and product manufacturing plant; the collection center and pre-processing center adopt the inventory production mode, the collection center collects WEEE according to the plan, puts it into the collection center WEEE warehouse, and waits for subsequent transfer to the pre-processing center WEEE warehouse, the pre-processing center disassembles WEEE and recycles waste plastic into granules, after disassembling and reusing WEEE, the obtained recycled plastic granules are stored in the recycled plastic granule warehouse, waiting for orders from the product manufacturing plant; the disassembled scrap metal is sold to the metal processing plant; The product manufacturing plant adopts a production-to-order model, placing plastic orders with pre-processing centers and external suppliers in advance to reserve raw materials, and then using the plastic raw material inventory to meet production order needs; In step 2, the dual-layer deep Q network includes a main network and a target network, wherein the main network is used to select an action and calculate the Q value in the current state; the target network is used to calculate the maximum Q value of the next state; the Q value represents the expected value of the cumulative reward that can be obtained after selecting an action in the current state; The training process of step 2 is as follows: Step 1: Initialization: First, initialize two main networks and target networks with the same structure, as well as the experience replay pool P , and set the reward discount factor and Exploration rate in policy The initial value of Step 2: Environmental interaction: The agent observes the current supply chain status , and then by Strategy with a certain probability Next random action After selecting the action, the collection, disassembly and reuse decision corresponding to the action is input into the supply chain virtual system for interaction, and the negative supply chain operation cost returned by the system is obtained, i.e., the reward. R t , and observe the new status of the supply chain , and finally Deposit into experience replay pool P , used to subsequently update the main network parameters ; Step 3: Experience playback and main network update: During the experience replay process, from the experience replay pool P Randomly select a batch of samples from ; Using the main network to estimate the state Take action The Q value, that is, the expected cumulative return in the future ; Then, the target network is used to calculate the current state Target value ; Finally, calculate the loss function and use the gradient descent method to update the main network parameters ; Step 4: Target network update: At fixed time intervals, the parameters of the main network are Copy to the target network to update the parameters of the target network ,Right now ; Step 5: Exploration Rate attenuation: After each training round, gradually reduce The value gradually reduces the exploration behavior and allows the algorithm to make more use of the learned knowledge to select actions; Step6: Training convergence: As training progresses, the main network gradually learns to accurately estimate the Q value of each state-action pair, thereby selecting the action with the highest Q value and adjusting in each interaction in the direction of maximizing the long-term cumulative reward; when the cumulative reward of a round of length T When the growth of tends to be stable, the reinforcement learning agent converges and the training is completed; The supply chain status It is expressed as follows: ； in, represents the WEEE warehouse inventory of the collection center at time t, Y t represents the amount of WEEE shipped from the collection center to the pre-processing center but not yet delivered during the time step t-1,…,t-LC, LC represents the delivery period from the collection center to the pre-processing center, represents the WEEE warehouse inventory of the pre-processing center at time t, represents the amount of WEEE dismantled and reused in the pre-processing center at time t-1, It represents the inventory of recycled plastic granules in the pre-processing center at time t. It represents the in-transit inventory of plastic pellets ordered by the product manufacturing plant from the pre-processing center and external suppliers but not yet delivered at time t. It represents the inventory of plastic raw materials in the warehouse of the product manufacturing factory at time t. represents the plastic raw material demand for product production orders at time t-σ, represents the plastic raw material demand for product production orders at time t-1, represents the amount of WEEE to be recycled in the market at time t-σ, represents the amount of WEEE to be recycled in the market at time t-1, and σ represents the length of time of the observed historical demand and historical amount to be recycled; The action It is expressed as follows: ； in, x t represents the WEEE collection plan established at time t, y t represents the amount of WEEE transported from the collection center to the pre-treatment center at time t, z t represents the amount of WEEE disassembled and reused in the pre-processing center at time t, q t represents the number of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t, p t It represents the quantity of plastic pellets purchased by the product manufacturing factory from external suppliers at time t; The Reward R t It is expressed as follows: ； Among them, the first represents the collection cost of WEEE, represents the collection cost per unit of WEEE, represents the actual WEEE collection amount at time t; Item 2 represents the variable cost of pre-processing of WEEE, represents the variable cost of pre-processing per unit of WEEE, z t It represents the amount of WEEE dismantled and reused by the pre-processing center at time t; Item 3 represents the cost of setting up the pre-processing production line, It represents the startup cost of setting up the unit disassembly line. l t Indicates the number of production lines started at time t; Item 4 represents the transportation cost of WEEE, represents the unit WEEE transportation cost, y t It represents the amount of WEEE transported from the collection center to the pre-treatment center at time t; Item 5 represents the transportation cost of recycled plastic pellets, represents the transportation cost per unit of recycled plastic pellets, q t It represents the quantity of recycled plastic particles ordered by the product manufacturing plant from the pre-processing center at time t; Item 6 represents the cost of purchasing plastic pellets from external suppliers, Represents the unit plastic pellet procurement cost, p t It represents the quantity of plastic pellets purchased by the product manufacturing factory from external suppliers at time t; Indicates that during the planning period, At the end of time t, each warehouse needs to pay the inventory holding cost. h i represents the holding cost of each warehouse unit inventory, i is the warehouse index, the collection center WEEE warehouse is represented by index 0, the pre-processing center WEEE warehouse is represented by index 1, the pre-processing center recycled plastic granule warehouse is represented by index 2, and the product manufacturing plant plastic raw material warehouse is represented by index 3; h 3 Represents the unit inventory holding cost of the plastic raw material warehouse of the product manufacturing plant, It represents the inventory of each warehouse at the end of time t and the beginning of time t+1. It represents the inventory of plastic raw material warehouse of product manufacturing factory at the end of time t; Represents the cost of delayed production orders due to shortage of plastic raw materials, It represents the cost of production order delay caused by shortage of unit plastic raw materials; Indicates the end of the planning period, i.e. When you buy a product, you need to pay additional inventory management costs. represents the unit inventory management cost of each warehouse for the redundant inventory at the end of the planning period, represents the inventory of each warehouse at the end of the planning period T, represents the management cost of redundant inventory in the plastic raw material warehouse, It represents the warehouse inventory of plastic raw materials in the product manufacturing plant at the end of the planning period T; It means that at the end of the planning period, additional production order loss costs need to be paid. represents the cost of production order loss due to unit plastic raw materials at the end of the planning period; The last item sj t represents the proceeds from the sale of metal briquettes, Represents the revenue from selling a unit of metal briquettes, j t represents the number of metal compacts generated by dismantling and recycling WEEE at time t; The specific method of step 3 is as follows: (1) The status of the supply chain reality system Input into the main network to obtain a set of Q values, each element of which corresponds to a different action A Q value; (2) Use the argmax function to select the action with the largest Q value, that is, ; (3) Based on Guide the supply chain reality system to carry out recycling plan formulation, transportation, disassembly and reuse operations.

2. According to the reinforcement learning-based WEEE recycling and reuse supply chain management and optimization method of claim 1, It is characterized in that In Step 2, the specific action selection process is as follows: (1) Probabilistic random selection of actions; (2) The probability of obtaining the action corresponding to the maximum Q value through the main network.

3. According to the WEEE recycling and reuse supply chain management and optimization method based on reinforcement learning as claimed in claim 1, It is characterized in that Target value It is expressed as follows: ； in, R t Representation status Take action The reward obtained immediately after, γ represents the discount factor, whose value is between 0 and 1, which is used to measure the importance of future rewards relative to current rewards; Represents the next state given by the target network Next best move The corresponding Q value.

Images