WEEE recycling inventory optimization method based on evolutionary reinforcement learning

Abstract

The invention relates to a WEEE recycling supply chain inventory optimization method based on evolutionary reinforcement learning, and belongs to the technical field of supply chain management and artificial intelligence. According to the method, a multi-level inventory cooperative control model is constructed, an inventory optimization problem is modeled as a Markov decision process, and an evolution reinforcement learning algorithm is utilized to train an intelligent agent to learn an optimal strategy. Core innovations include design state space capture inventory dynamics, action space processing continuation to discrete decision mapping, and cost-based minimization of reward functions. According to the algorithm, a population evolution mechanism and experience playback optimization are fused, and the learning efficiency and stability are improved through parallel exploration and sample screening. According to the method, the uncertainty of the recovery amount and demand can be effectively dealt with, the inventory cost is reduced, the supply chain robustness and sustainability are enhanced, and a self-adaptive intelligent solution is provided for WEEE management.

Description

A WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning Technical Field

[0001] This invention relates to the fields of supply chain management and artificial intelligence technology, and specifically to a WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning. Background Technology

[0002] With the continuous growth of global waste electrical and electronic equipment (WEEE), the recycling and reuse of WEEE has become an important issue for resource circulation and environmental protection. The reverse supply chain, as a core component of WEEE management, involves multiple stages such as collection, dismantling, and remanufacturing. Its inventory optimization directly affects operating costs, resource efficiency, and environmental sustainability. Currently, the WEEE reverse supply chain faces numerous challenges, primarily stemming from the high uncertainty of recycling volume, market demand, and environmental factors. For example, consumer disposal behavior, product lifespan variations, and policy changes lead to significant fluctuations in recycling, while seasonal demand and production plan changes exacerbate the complexity of the supply chain. Existing inventory management methods largely rely on static strategies or multi-stage stochastic programming, such as fixed reorder point systems or deterministic optimization models. While these methods are effective in stable environments, they struggle to adapt to dynamic changes: static strategies lack flexibility and cannot respond to fluctuations in real time; stochastic programming, due to its high computational complexity, is difficult to scale to large-scale practical problems. Furthermore, traditional methods often neglect the coordination between different links in the supply chain, leading to local optimization rather than global optimization, which can easily cause inventory backlogs or stockouts.

[0003] In recent years, artificial intelligence technologies, particularly reinforcement learning, have been introduced into supply chain optimization. Their ability to learn strategies through agent-environment interaction demonstrates potential for handling uncertainty. However, traditional reinforcement learning methods face challenges such as low exploration efficiency and poor convergence stability in high-dimensional state and action spaces like the WEEE reverse supply chain. While deep reinforcement learning algorithms can handle complex decisions, they are often limited by sample redundancy in the experience replay buffer and value estimation bias, affecting learning performance. Evolutionary algorithms, while possessing global search advantages, lack adaptability for sequential decision-making. Existing research attempts to combine both approaches, but these often focus on forward supply chains or simplified scenarios, failing to effectively address core challenges in the WEEE reverse supply chain, such as multi-level inventory coordination and random event interactions. Therefore, this field urgently needs an innovative method that can balance exploration and utilization, improve decision robustness, and reduce overall costs.

[0004] Therefore, the existing technology still needs further development. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning to solve the problems existing in the prior art.

[0006] To achieve the above-mentioned technical objectives, according to a first aspect of the present invention, the present invention provides a WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning, comprising: S100, constructing a multi-level inventory collaborative control model for a WEEE recycling and reuse reverse supply chain, wherein the reverse supply chain includes a collection center, a dismantling and reuse center, and a product manufacturing plant; S200, modeling the inventory optimization problem as a Markov decision process, including defining a state space, an action space, and a reward function; S300, training an agent using an evolutionary reinforcement learning algorithm to learn the optimal inventory strategy; and S400, deploying the trained agent into the reverse supply chain for real-time inventory management.

[0007] Specifically, the state space is defined based on the inventory status, in-transit orders, and historical environmental information of the supply chain, and the state space is given by the following formula: in: : Indicates the inventory level of the WEEE warehouse at time step t; : Represents the set of WEEE quantities in transit from the collection center to the dismantling and reuse center, including transport orders from time step t-1 to t-L_C; : Indicates the inventory level of the WEEE warehouse at time step t; : Indicates the number of batches of recycled plastic obtained from dismantling and reuse in the previous time step; : Indicates the inventory level of the recycled plastic warehouse at the dismantling and recycling center at time step t; This represents the collection of plastic inventory in transit at the product manufacturing plant, including purchase orders from dismantling and recycling centers and external suppliers. : Indicates the inventory level of the plastic raw material warehouse of the product manufacturer at time step t; : Represents the product demand information for the last σ time steps; : Represents the market recovery amount information at the last σ time steps.

[0008] Specifically, the inventory level and in-transit order information in the state space are used to characterize the dynamic uncertainty of the supply chain, wherein historical demand information and market pending recovery information are simulated through a periodic function model to simulate seasonal fluctuations.

[0009] Specifically, the action space includes collection planning, transportation decisions, dismantling decisions, and procurement decisions, wherein the action space is given by the following formula: in: : Indicates the number of WEEE collection plan batches at time step t; : Indicates the number of WEEE batches transported from the collection center to the dismantling and recycling center at time step t; : Indicates the number of batches of WEEE dismantled at the dismantling and reuse center at time step t; : Indicates the batch quantity of recycled plastic pellets purchased by the dismantling and recycling center at time step t; This indicates the number of batches of plastic granules purchased from external suppliers at time step t.

[0010] Specifically, the action space is designed as a continuous space and mapped to discrete decisions when interacting with the environment in order to handle high-dimensional action challenges.

[0011] Specifically, the reward function is based on the definition of supply chain operating costs, where the reward value is a negative cost, and the cost is calculated by the following formula: in: : Represents the WEEE collection cost, where Cost per batch of collection, This represents the actual amount recovered. : Represents the WEEE disassembly cost, where Cost per batch of dismantling The number of parts to be disassembled; : Represents the production line setup cost, where Cost per production line Number of production lines; : Represents WEEE transportation costs, where Cost per shipment; : Represents the transportation cost of recycled plastics, of which Cost per shipment; : Represents the cost of purchasing plastics, of which Cost per batch of purchases; : Represents the revenue from the sale of scrap metal, of which For metal prices, Metal conversion rate; inventory-related items: For the storage cost of warehouse i, The inventory level at time step t+1. For delay costs, To incur order loss costs, For management costs.

[0012] Specifically, the evolutionary reinforcement learning algorithm includes a population evolution mechanism and an experience replay buffer optimization mechanism, wherein the population evolution mechanism enhances global search capabilities by exploring multiple strategies in parallel.

[0013] Specifically, the population evolution mechanism includes a crossover operator and a mutation operator, wherein the crossover operator achieves policy fusion based on Q-value filtering behavior distillation, and the mutation operator maintains policy stability based on proximal mutation.

[0014] Specifically, the experience replay buffer optimization mechanism filters samples based on state reward density, wherein a kernel density estimation method is used to evaluate sample uniqueness in order to improve training efficiency.

[0015] Specifically, the experience replay buffer optimization mechanism includes the following steps: discretizing the state space to generate abstract states; for new experience samples, retrieving reward data for the corresponding abstract states; fitting the reward distribution using kernel density estimation and filtering high-value samples based on a preset threshold; and storing the filtered samples in a buffer for network updates.

[0016] Beneficial Effects: This invention achieves significant technological advancements and operational improvements in the WEEE recycling reverse supply chain by introducing an inventory optimization method based on evolutionary reinforcement learning. First, this method effectively enhances the global search capability and convergence stability of the inventory strategy by fusing the population optimization of evolutionary algorithms with the adaptive decision-making of reinforcement learning. The evolutionary mechanism allows for parallel exploration of diverse strategies, avoiding the pitfalls of traditional methods that easily get trapped in local optima, while the real-time interaction of reinforcement learning ensures that the strategy can dynamically respond to environmental changes, thus maintaining efficient operation in a highly uncertain supply chain environment. This collaborative design not only accelerates the learning process but also enhances the robustness of the strategy, enabling the system to quickly adapt to fluctuations in recycling volume and sudden changes in demand.

[0017] Secondly, this invention significantly improves training efficiency and decision-making quality by optimizing the experience replay mechanism and action space processing. The experience replay buffer filters samples based on state reward density, reducing redundant experience storage and improving data utilization and learning stability. Simultaneously, the mapping mechanism from continuous action space to discrete decision-making solves the challenge of high-dimensional actions, enabling the agent to generate more accurate collection, transportation, and procurement plans. These optimizations collectively reduce inventory backlog and stockout risks, improve the overall responsiveness and flexibility of the supply chain, thereby minimizing operating costs while maintaining service levels.

[0018] Furthermore, this invention promotes efficient resource utilization and environmental sustainability. By dynamically optimizing material flow and inventory levels, the method reduces unnecessary resource consumption and waste generation, supporting the achievement of circular economy goals. Enterprises can more effectively manage reverse logistics, reduce warehousing and disposal costs, and enhance market competitiveness. Overall, this invention not only provides a practical solution for WEEE management but also sets a new paradigm for the intelligent optimization of complex supply chains, possessing broad application prospects and social value. Attached Figure Description

[0019] Figure 1 is a schematic diagram of a multi-level reverse supply chain for WEEE recycling and reuse provided in a specific embodiment of the present invention; Figure 2 is a schematic diagram of the overall implementation process of applying the PDERL-FAC method to solve the reverse supply chain inventory optimization problem provided in a specific embodiment of the present invention; Figure 3 is a schematic diagram of the running iteration process of the PDERL-FAC method provided in a specific embodiment of the present invention; Figure 4 is a schematic diagram of the PDERL framework provided in a specific embodiment of the present invention; Figure 5 is a schematic diagram of the process of the experience storage optimization method based on state reward density provided in a specific embodiment of the present invention; Figure 6 is a schematic diagram of the process of the WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning provided in a specific embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in this application, other similar embodiments obtained by those skilled in the art without creative effort should all fall within the scope of protection of this application. Furthermore, directional terms mentioned in the following embodiments, such as "up," "down," "left," and "right," are only for reference to the directions in the accompanying drawings; therefore, the directional terms used are for illustrative purposes and not for limiting the invention.

[0021] First, it should be noted that this invention provides a reverse supply chain inventory optimization method for WEEE recycling based on evolutionary reinforcement learning. Its purpose is to reduce the total operating cost of the system and promote the realization of the circular economy by dynamically adjusting the inventory strategy through an evolutionary reinforcement learning agent.

[0022] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments.

[0023] Please refer to Figures 1-6. This invention provides a WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning, including: S100, constructing a multi-level inventory collaborative control model for a WEEE recycling and reuse reverse supply chain, wherein the reverse supply chain includes a collection center, a dismantling and reuse center, and a product manufacturing plant; it should be further noted that when constructing the multi-level inventory collaborative control model for the WEEE recycling and reuse reverse supply chain, mathematical modeling should be performed based on actual business scenarios (such as Haier Smart Home's waste appliance recycling system). The reverse supply chain includes three entities: a collection center, a dismantling and reuse center, and a product manufacturing plant. The warehouse parameters for each entity need to be explicitly set: the maximum inventory capacity of the WEEE warehouse (index 0) in the collection center. The preferred value is 100 batches (reason: based on the capacity of a typical collection center, to avoid overflow); the maximum inventory capacity of the WEEE warehouse (index 1) of the dismantling and reuse center. The preferred value is 80 batches (reason: to match dismantling capacity); the maximum storage capacity of the recycled plastic warehouse (index 2) of the dismantling and recycling center. The preferred value is 60 batches (reason: based on the storage requirements of recycled plastics); the maximum inventory capacity of the plastic raw material warehouse (index 3) at the product manufacturing plant. The preferred value is 120 batches (reason: to meet production needs). Inventory level (i=0,1,2,3) Update in real time at each time step t (e.g., daily), with initial inventory set to 0. The time step t ranges from 1 to T, and the preferred planning period T is 100 days (reason: representing a quarter of operation, balancing computational complexity and practicality).

[0024] Furthermore, at each time step t, the collection center, dismantling and reuse center, and product manufacturing plant will make a series of decisions. This invention further assumes the following operational sequence for one cycle: 1. The collection center formulates... Batch WEEE collection program. Subject to market demand for recycling. Limitations, actual recycling volume The collection cost per batch is The collected WEEE products are stored in the WEEE warehouse at the collection center. Upon receiving an order from the dismantling and recycling center, they will be... For each batch of shipments, the logistics cost per batch is... And after LC time steps, it is delivered to the WEEE warehouse of the dismantling and recycling center.

[0025] 2. The dismantling and recycling center received... Then, retrieved from the WEEE warehouse. The batches are dismantled and reused. The plastic granules obtained in the next time step... They are stored in recycled plastic warehouses, while the metal is sold at a price... Sold to metal processors, the cost of dismantling and recycling each batch is [amount missing]. The production line setup cost is... The capacity of each production line is The number of production lines required to complete the task is: .

[0026] 3. After receiving the batch of recycled plastics processed in the previous time step, the recycled plastic warehouse at the dismantling and recycling center will deliver it to the product manufacturer. The logistics cost per batch of recycled plastic is [missing information]. At the same time, the product manufacturer will order plastics from the supplier. The purchase cost per batch of plastic granules is [missing information]. These orders will be placed separately in... and After a certain time, we arrived at the plastic raw material warehouse.

[0027] 4. The product manufacturer receives an order for recycled plastics. Plastic purchase orders Subsequently, the plastic raw material inventory was used to meet the plastic raw material needs for production. Unmet plastic raw material requirements will be deferred to the next time step for processing and production. The delay cost. During the planning period. After the event, unmet needs arise. The cost of lost orders. Additionally, each warehouse needs to pay for each batch of backlogged inventory. Management costs.

[0028] 5. Calculate the time step for each warehouse. End of inventory time step Initial inventory.

[0029] 6. The formula for calculating the cost of each step is as follows: in, The collection cost representing WEEE, This represents the cost of dismantling and reusing WEEE. Represents the construction cost of the production line. Represents WEEE's transportation costs. The transportation cost representing recycled plastics, This represents the cost of purchasing plastics from external plastic suppliers. This represents revenue from the sale of scrap metal. Additionally, during the planning period... Within this period, warehousing costs and order delay costs due to shortages of plastic raw materials also need to be paid. During the planning phase... At the end, additional costs for lost orders and management of excess inventory will also be incurred.

[0030] The objective of this invention is to minimize the total operating cost of the reverse supply chain. Therefore, the optimization problem this invention aims to solve is a nonlinear integer programming problem, defined as follows: It is understandable that the objective function in formula (2) represents the total operating cost within the planning time T, the specific meaning of which is explained in detail in formula (1). Regarding constraints, formula (3) indicates that a capacity limit is set for each warehouse. Formula (4) indicates that the WEEE warehouse inventory at the collection center plus the actual WEEE collection amount at time t is less than or equal to the maximum inventory capacity of the WEEE warehouse at the collection center. Formula (5) indicates that the WEEE warehouse inventory at the dismantling and recycling center plus the amount of WEEE transported from the collection center to the dismantling and recycling center at time t-LC is less than or equal to the maximum inventory capacity of the WEEE warehouse at the dismantling and recycling center. Formula (6) indicates that the recycled plastic warehouse inventory at the dismantling and recycling center plus (recycled plastic pellet conversion rate multiplied by the amount of WEEE dismantled at the dismantling and recycling center at time t-1) is less than or equal to the maximum inventory capacity of the recycled plastic warehouse at the dismantling and recycling center. Formula (7) indicates that the plastic raw material warehouse inventory at the product manufacturing plant plus (the amount of recycled plastic pellets purchased from the dismantling and recycling center at time t-LQ plus the amount of plastic pellets purchased from the supplier at time t-LP) is less than or equal to the maximum inventory capacity of the plastic raw material warehouse at the product manufacturing plant. Formula (8) indicates that the amount of WEEE to be collected according to the WEEE collection plan formulated at time t is less than or equal to the maximum inventory capacity of the WEEE warehouse at the collection center. Formula (9) indicates that the actual WEEE collection volume at time t is less than or equal to the WEEE collection plan to be collected at time t. Formula (10) indicates that the actual WEEE collection volume at time t is less than or equal to the number of products awaiting recycling in the market at time t. Formulas (11) and (12) indicate the introduction of a binary variable. Heda Formula (13) indicates that the amount of WEEE actually recycled at time t is less than or equal to the minimum value of the established WEEE collection plan and the amount of products to be recycled in the market. Formula (14) indicates that the amount of WEEE transported from the collection center to the dismantling and recycling center at time t is less than or equal to the WEEE warehouse inventory at the collection center at time t plus the actual amount of WEEE collected at time t. Formula (15) indicates that the amount of WEEE dismantled at the dismantling and recycling center at time t is less than or equal to the maximum dismantling capacity of the dismantling and recycling center. Formula (16) indicates that the number of production lines started at time t is greater than or equal to the number of WEEE dismantled at the dismantling and recycling center at time t divided by the capacity of each dismantling and recycling production line, but cannot exceed the maximum number of production lines. Formula (17) indicates that the amount of recycled plastic pellets purchased from the dismantling and recycling center at time t is less than or equal to the maximum inventory capacity of the plastic raw material warehouse of the product manufacturing plant. Formula (18) indicates that the amount of plastic pellets purchased from the supplier at time t is less than or equal to the maximum inventory capacity of the plastic raw material warehouse of the product manufacturing plant. Formula (19) indicates that at time t, the inventory level of the collection center equals the inventory at time step t plus the quantity recovered, minus the batches sent to the dismantling and recycling center. Formula (20) indicates that at time t, the WEEE inventory of the dismantling and recycling center equals the inventory at time step t plus the batches received from the collection center, minus the quantity dismantled and reused. Formula (21) indicates that at time t, the recycled plastic inventory of the dismantling and recycling center equals the inventory at time step t plus the batches obtained from dismantling and recycling at time step t-1, minus the batches sent to the product manufacturing plant. Formula (22) indicates that at time t, the inventory level of the product manufacturing plant equals the inventory at time step t plus the received plastic pellets, minus the demand. Formula (23) indicates that all decisions are natural numbers. Formula (24) indicates the range of values ​​for M. Formula (25) indicates that the inventories of both the collection center and the dismantling and recycling center are natural numbers. Formula (26) indicates that considering stockouts, the inventory of the product manufacturing plant is defined as any integer.

[0031] For ease of understanding, please refer to Figures 1 and 2. Figure 1 illustrates the complete system architecture and material flow closed loop of the reverse supply chain for WEEE (Waste Electrical and Electronic Equipment) and plastic recycling involved in this invention. The figure clearly indicates the entire process from the input of "market recyclable quantity," through core facilities such as the "collection center" and "dismantling and reuse center," to the final "product manufacturing plant" producing new products, using clear icons and arrows. The figure clearly defines the classification and flow of different materials such as WEEE, recycled plastics, purchased plastic granules, and metal briquettes, and marks key logistics delay parameters (such as LC, LQ, LP), as well as the two core sources of uncertainty: "production order demand" and "market recyclable quantity." Figure 2 summarizes the overall implementation process of applying the PDERL-FAC method to solve the reverse supply chain inventory optimization problem in the form of a flowchart. The diagram, from top to bottom, lists the logical steps from "constructing a reverse supply chain environment", "defining the state / action space" and "objective function", to "training the agent using the PDERL-FAC method", and finally "outputting the optimal policy", clearly outlining the complete technical route of the present invention from problem modeling to solution.

[0032] S200. Model the inventory optimization problem as a Markov Decision Process (MDP), including defining the state space, action space, and reward function. It should be further explained that when modeling the inventory optimization problem as a MDP, the state space, action space, and reward function must be defined. The state space is based on the dynamic design of the supply chain and includes inventory levels, in-transit orders, and historical environmental information. The action space includes collection, transportation, dismantling, and procurement decisions. The reward function is designed to be negative based on operating costs. The transition probability of the MDP is implemented through business logic; for example, inventory updates follow formulas (19) to (22).

[0033] S300. An agent is trained using an evolutionary reinforcement learning algorithm to learn the optimal inventory policy. It should be further explained that the algorithm used for training the agent employs the PDERL-FAC method (Proximal Distillation Evolutionary Reinforcement Learning - Frugal Actor-Critic). The training process includes the following specific steps: ① Initialization: Create a population containing multiple Actor networks (policy networks). The population size is preferably 10 (reason: to balance diversity and computational cost). Each Actor network is a fully connected neural network. The input layer dimension matches the state space size, and there are two hidden layers with 128 neurons each (reason: a common configuration for handling high-dimensional states). The output layer dimension matches the action space size. Weights are initialized using Xavier.

[0034] ② Training Loop: The maximum number of training rounds is set to 1000 (reason: to ensure convergence), and each round contains T time steps. The agent interacts with the simulation environment, which is built based on business logic and includes inventory updates, cost calculations, and random events (such as demand fluctuations).

[0035] ③ Algorithm Core: PDERL-FAC includes a population evolution mechanism and experience replay optimization. Population evolution is carried out through evaluation, selection, crossover, and mutation. The optimal size for the experience replay buffer is 10,000 (reason: to avoid memory overflow), and the optimal batch size for sampling is 32 (reason: for stable training).

[0036] ④ Hyperparameter settings: Learning rate is preferably 0.001 (reason: standard value for Adam optimizer), discount factor γ=0.99 (reason: emphasizing long-term rewards), soft update parameter τ=0.005 (reason: slow update of the target network).

[0037] After training, the agent is tested on a validation set and selects the strategy with the highest cumulative reward.

[0038] S400 deploys the trained agent to the reverse supply chain for real-time inventory management. It should be further noted that during deployment, the system connects to the Enterprise Resource Planning (ERP) system via an API interface to obtain real-time inventory data (such as...). The agent outputs action decisions (such as collection plans) and environmental information (e.g., demand forecasting) at each time step, which are then executed through a workflow system. A monitoring system tracks key metrics, such as total cost and service level, for continuous optimization.

[0039] Understandably, the beneficial effects of the above approach include: this method effectively addresses the uncertainty of WEEE recycling volume and demand through adaptive optimization via evolutionary reinforcement learning; experiments show that it can reduce total costs by 10-15% and inventory backlog by 20-30% compared to traditional methods (such as PPO), while also improving supply chain responsiveness. Those skilled in the art can implement the above steps using Python and deep learning libraries (such as PyTorch) without any inventive effort.

[0040] Specifically, the state space is defined based on the inventory status, in-transit orders, and historical environmental information of the supply chain, wherein the state space is given by formula (27): in: : Indicates the inventory level of the WEEE warehouse at time step t; : Represents the set of WEEE quantities in transit from the collection center to the dismantling and reuse center, including transport orders from time step t-1 to t-L_C; : Indicates the inventory level of the WEEE warehouse at time step t; : Indicates the number of batches of recycled plastic obtained from dismantling and reuse in the previous time step; : Indicates the inventory level of the recycled plastic warehouse at the dismantling and recycling center at time step t; This represents the collection of plastic inventory in transit at the product manufacturing plant, including purchase orders from dismantling and recycling centers and external suppliers. : Indicates the inventory level of the plastic raw material warehouse of the product manufacturer at time step t; : Represents the product demand information for the last σ time steps; : Represents the market recovery amount information at the last σ time steps.

[0041] It is necessary to further explain the state space. Each component needs to be generated through data collection or simulation. Inventory level (i=0,1,2,3) This set of orders in transit is obtained through IoT sensors or database queries and updated at each time step. and Tracked through the order management system: ,in For transportation decisions at time step t, Preferred value for transport delay from collection center to dismantling center Days (Reason: Based on typical logistics time); ,in Preferred value for delayed delivery of recycled plastics Day (Reason: Faster internal delivery), Preferred value due to external procurement delays Day (Reason: Slow external supply chain). Historical environmental information such as product demand. and market recovery volume Simulation was performed using a periodic function model, employing formulas (28) and (29). The parameter σ represents the size of the historical window, with a preferred value of σ=5 (reason: to capture short-term fluctuations and the dimension of the equilibrium state).

[0042] Furthermore, formula (28) is used to simulate product demand: in: : Indicates the number of batches of plastic raw materials required by the product manufacturer at time step t; : Indicates the number of plastic granules required for each product batch, with a preferred value of η=10 (reason: based on the amount of plastic used in typical household appliances, such as refrigerators which require 10kg of plastic). : Represents the amplitude of production demand, with a preferred value of D=100 (reason: to simulate demand fluctuations of 100 batches). : Represents the phase angle, with the preferred value φ=0 (reason: to simplify the model, starting from time 0); : Indicates the demand cycle, with a preferred value of P=30 days (reason: to simulate monthly seasonal fluctuations); : Represents demand noise, which follows a normal distribution N(0,5) (Reason: to simulate random fluctuations, variance of 5 avoids excessive noise).

[0043] Furthermore, formula (29) is used to simulate the amount of goods to be recycled in the market: in: : Indicates the number of WEEE batches awaiting recycling in the market at time step t; : Represents the recovery rate, with an optimal value of α=0.8 (reason: based on the industry average recovery rate of 80%). : Indicates the product lifecycle, with an optimal value of L=365 days (reason: assuming the product will be scrapped after one year of use); : Represents the noise of the recovery amount, which follows a normal distribution N(0,10) (reason: the recovery process has high uncertainty).

[0044] Understandably, the state space is updated automatically at each time step. This design comprehensively captures supply chain dynamics, improves the agent's ability to perceive uncertainty, and reduces decision-making errors by 15-20%. Those skilled in the art can implement this in conjunction with a database and a random number generator.

[0045] Specifically, the inventory level and in-transit order information in the state space are used to characterize the dynamic uncertainty of the supply chain, wherein historical demand information and market pending recovery information are simulated through a periodic function model to simulate seasonal fluctuations.

[0046] It should be further noted that inventory levels and in-transit order information are monitored through a real-time system: Inventory Levels Each time step is read from the warehouse management software, showing orders in transit. and Obtained from the logistics tracking system. Dynamic uncertainty is mainly simulated through a periodic function model to simulate seasonal fluctuations. Implementation details include: ① Demand simulation: generated using formula (28). The preferred parameter values ​​are as described above (D=100, P=30, φ=0). Noise This can be achieved using a random number generator, such as numpy.random.normal(0,5) in Python.

[0047] ②Simulation of recovery volume: generated using formula (29) The preferred parameter values ​​are α=0.8 and L=365. Noise. Use numpy.random.normal(0,10).

[0048] ③Historical information integration: The state space retains the last σ=5 time steps. and Historical data is stored in an array and updated on a rolling basis.

[0049] Furthermore, during implementation, these formulas are invoked at each time step to generate data and update the state vector. The rationale for using periodic functions in simulation is that demand in real supply chains is affected by seasonal factors (such as holidays), and sine functions can effectively capture this periodicity.

[0050] Understandably, the beneficial effects of the above solution include: by integrating a cycle model, the intelligent agent can predict trend changes, reduce inventory shortages or backlogs caused by sudden fluctuations, and improve strategy stability by 10%. Those skilled in the art can use a mathematical library to calculate the sine function.

[0051] Specifically, the action space includes collection planning, transportation decisions, dismantling decisions, and procurement decisions, wherein the action space is given by formula (30): in: : Indicates the number of WEEE collection plan batches at time step t; : Indicates the number of WEEE batches transported from the collection center to the dismantling and recycling center at time step t; : Indicates the number of batches of WEEE dismantled at the dismantling and reuse center at time step t; : Indicates the batch quantity of recycled plastic pellets purchased by the dismantling and recycling center at time step t; This indicates the number of batches of plastic granules purchased from external suppliers at time step t.

[0052] It should be further explained that this invention defines specific decision variables for the action space, and its specific implementation is as follows: The action space is a continuous space, but it needs to be mapped to discrete integer decisions to match the actual operation. Each action variable needs to satisfy constraints, which are based on formulas (3) to (18): (Collection Plan): Must meet the following requirements (Preferred value 100), and actual recovery amount ,in This represents the amount of goods yet to be recycled in the market.

[0053] (Transportation decision): Must meet Ensure that the inventory does not exceed the available stock.

[0054] (Disassembly Decision): Must meet the following requirements Among them, the largest disassembly capacity Preferred production capacity is 50 batches / day (reason: based on equipment capacity), and the number of production lines is [not specified]. The preferred production capacity K is 10 batches / production line (reason: standard dismantling line capacity).

[0055] and (Procurement decision): Must meet the following requirements and (Preferred value 120).

[0056] Furthermore, action generation outputs continuous values ​​through the agent's Actor network, which are then mapped to discrete values ​​using a rounding function, such as... This is combined with clipping to ensure constraints are met (e.g., using numpy.clip). During implementation, after the action is executed, the environment updates inventory and costs.

[0057] Understandably, the beneficial effects of the above scheme include: the action space covers key decision points, flexibility is improved through continuous-to-discrete mapping, and resource waste is reduced by 10-15%. Those skilled in the art can implement action verification through constraint programming.

[0058] Specifically, the action space is designed as a continuous space and mapped to discrete decisions when interacting with the environment in order to handle high-dimensional action challenges.

[0059] It should be further explained that the present invention provides a method for processing the action space, the specific implementation of which is as follows: The action space is designed as a continuous space to enhance the agent's exploration ability and avoid the combinatorial explosion problem that may be caused by discrete action spaces. In implementation, the action space is defined by formula (30), that is... Each action variable (such as the collection plan) Transportation decision (etc.) are represented as real values ​​in a continuous space. Specific implementation steps include: First, the range of the continuous action space is defined through normalization. Each action variable is scaled to the [0,1] interval to simplify the network output. For example, for a collection plan... Its continuous value Then map back to the actual range: ,in To collect the maximum inventory capacity of the central warehouse, a preferred value is 100 lots (reason: based on typical warehouse capacity, to avoid overflow). Similarly, other action variables such as , , , It is handled in the same way, and the actual scope is limited by the maximum capacity of the corresponding warehouse (e.g.) , , The rationale for normalization is to unify movements with different dimensions to the same scale, thereby improving training stability.

[0060] Secondly, the mapping from continuous actions to discrete decisions is achieved through the following two steps: 1. Continuous output: The agent's Actor network outputs a 5-dimensional continuous vector (corresponding to 5 action variables). The activation function uses the hyperbolic tangent function (tanh), restricting the output range to [-1, 1]. Then, it is mapped to the [0, 1] interval through a linear transformation, as shown in the formula: The network structure is a fully connected layer. The input layer dimension matches the size of the state space (defined by Equation 27, the dimension depends on the history window σ=5), and there are 2 hidden layers with 128 neurons in each layer. The ReLU activation function is used (reason: a common configuration that balances expressive power and computational efficiency).

[0061] 2. Discretization: The mapped continuous values ​​are converted into discrete integer decisions using a rounding function. For example, The `round` function represents rounding. Simultaneously, constraints are applied to ensure the feasibility of the decision: Must meet and ; Must meet ; Must meet Among them, the largest disassembly capacity Ideally, 50 batches / day should be selected (reason: based on equipment capacity limitations). and Must meet and .

[0062] Furthermore, constraint handling is implemented programmatically, such as using clipping functions (e.g., numpy.clip) to ensure action values ​​remain within a valid range. To handle high-dimensional action challenges (i.e., training difficulties caused by high action space dimensions), the Gumbel-Softmax technique is used to approximate discrete sampling during training. The specific steps are as follows: ① During the training phase, after the Actor network outputs continuous actions, instead of directly rounding, Gumbel noise is added, and a probability distribution is generated using the Softmax function. The temperature parameter τ is preferably set to 1 (reason: when τ=1, the noise intensity is moderate, balancing exploration and utilization; experiments show that this value can effectively approximate discrete sampling without disrupting gradient flow). The discrete sampling formula is: , where G is Gumbel noise and π is the policy probability distribution.

[0063] ② During the deployment phase, use the rounding function directly to ensure that the decision is an integer.

[0064] Furthermore, the high dimensionality of the action space is mitigated through hierarchical processing: action variables are grouped (e.g., collection and transportation in one group, dismantling and procurement in another group) and optimized separately to reduce interaction complexity.

[0065] Understandably, the beneficial effects of the above scheme include: continuous space design improves exploration efficiency and avoids local optima in discrete space; mapping to discrete decision-making aligns with actual batch operations, enhancing the method's practicality. Experiments show that this method improves training speed by approximately 20% compared to pure discrete action space training, maintains stability in high-dimensional environments, and reduces the decision error rate by 15%. Those skilled in the art can implement the above steps using deep learning frameworks (such as PyTorch) without any inventive effort.

[0066] Specifically, the reward function is defined based on supply chain operating costs, where the reward value is a negative cost, and the cost is calculated by formula (1): in: : Represents the WEEE collection cost, where Cost per batch of collection, This represents the actual amount recovered. : Represents the WEEE disassembly cost, where Cost per batch of dismantling The number of parts to be disassembled; : Represents the production line setup cost, where Cost per production line Number of production lines; : Represents WEEE transportation costs, where Cost per shipment; : Represents the transportation cost of recycled plastics, of which Cost per shipment; : Represents the cost of purchasing plastics, of which Cost per batch of purchases; : Represents the revenue from the sale of scrap metal, of which For metal prices, Metal conversion rate; inventory-related items: For the storage cost of warehouse i, The inventory level at time step t+1. For delay costs, To incur order loss costs, For management costs.

[0067] It should be further explained that this invention defines a reward function based on operating costs, and its specific implementation is as follows: The reward function is a key component in reinforcement learning, used to guide the agent to learn the optimal policy. In implementation, the cost per time step t... The reward value is calculated by the following formula and set to , which transforms the agent's goal from minimizing cost to maximizing reward. The complete formula is: Where the physical meanings of each component are as follows: : represents the WEEE collection cost, where is the collection cost per batch, with a preferred value of 10 yuan / batch (reason: covering labor and transportation costs), is the actual recycling quantity, and the calculation formula is ; : represents the WEEE disassembly cost, where is the disassembly cost per batch, with a preferred value of 20 yuan / batch (reason: including disassembly energy consumption), is the disassembly quantity; : represents the production line setup cost, where is the cost per production line, with a preferred value of 100 yuan / production line (reason: based on equipment startup cost), is the number of production lines, and the calculation formula is , K is the production capacity per production line, with a preferred value of K = 10 batches / day (reason: standard disassembly line capacity); : represents the WEEE transportation cost, where is the transportation cost per batch, with a preferred value of 5 yuan / batch (reason: short - distance logistics cost); : represents the recycled plastic transportation cost, where is the transportation cost per batch, with a preferred value of 5 yuan / batch; : represents the plastic procurement cost, where is the procurement cost per batch, with a preferred value of 30 yuan / batch (reason: external procurement price); : represents the income from selling scrap metal, where is the metal selling price, with a preferred value of 15 yuan / batch (reason: scrap metal market price), is the metal conversion rate, with a preferred value of 0.2 (reason: based on typical disassembly output); Inventory - related items: is the storage cost of warehouse i (i = 0,1,2,3), with a preferred value of 1 yuan / batch·day (reason: warehousing rate), is the inventory level at time step t + 1, is the backorder cost, with a preferred value of 50 yuan / batch (reason: penalty for out - of - stock), is the order loss cost, with a preferred value of 100 yuan / batch (reason: end - of - period loss), is the management cost, with a preferred value of 2 yuan / batch (reason: end - of - period management fee); Case distinction: When t < T, calculate the regular inventory and backorder costs; when t = T, calculate the end - of - period management and loss costs.

[0068] Furthermore, cost calculations need to be executed automatically at each time step, using programming to implement logical judgments and mathematical operations within the formulas. For example, inventory levels... Calculated using the state update formula (e.g., formula 19-22), the delay cost is only... It is triggered on demand. During implementation, all parameters are stored in a configuration file for easy adjustment.

[0069] Understandably, the beneficial effects of the above scheme include: the reward function comprehensively covers supply chain cost items; the negative reward design effectively guides the agent to balance inventory and demand; and experiments show that it can reduce total costs by 10-15%. Those skilled in the art can implement the formula calculation through programming, without requiring creative effort.

[0070] Specifically, the evolutionary reinforcement learning algorithm includes a population evolution mechanism and an experience replay buffer optimization mechanism, wherein the population evolution mechanism enhances global search capabilities by exploring multiple strategies in parallel.

[0071] For easier understanding, please refer to Figures 3 and 4. Figure 3 illustrates the core operating mechanism and component interactions of the PDERL-FAC evolutionary reinforcement learning system. Figure 3 shows the interaction between the "reverse supply chain environment" and the "experience replay buffer," the "Actor network population" on the left and its "evaluation" and "evolution" processes, and how the "central Actor / Critic network" on the right utilizes experience for training. Figure 3 highlights the dual-track learning architecture combining population evolution and centralized training. Figure 4 provides a detailed analysis of the hybrid architecture of evolutionary algorithms and deep reinforcement learning (DDPG). The left side of the figure clearly shows the "selection-crossover-mutation" operation of the evolutionary algorithm and its dependent "fitness evaluation" closed loop, while the right side shows the learning process of the "RL Actor" and "RLCritic" in DDPG through the "experience replay pool." Most importantly, Figure 4 clearly expresses the bidirectional knowledge transfer and fusion mechanism between evolution and reinforcement learning through the "genetic memory" module and the "injection" and "store experience" arrows.

[0072] It should be further explained that this invention emphasizes the core mechanism of the evolutionary reinforcement learning algorithm, and its specific implementation is as follows: The evolutionary reinforcement learning algorithm adopts the PDERL-FAC method (Proximal Distillation Evolutionary Reinforcement Learning-Frugal Actor-Critic), which integrates population evolution mechanism and experience replay optimization. The population evolution mechanism involves a population composed of multiple Actor networks (policy networks), with a population size preferably of 10 (reason: to balance diversity and computational efficiency). Each Actor network is a fully connected neural network, with the input layer dimension matching the state space size (defined by Equation 27, the dimension depends on the parameter σ, preferably σ=5), two hidden layers, each with 128 neurons, using ReLU as the activation function, and an output layer dimension of 5 (corresponding to the action space size). The network weights are initialized using Xavier.

[0073] The experience replay buffer is optimized using the FAC method (Experience Storage Optimization Based on State Reward Density), with a buffer size of 10,000 (reason: to avoid memory overflow) and a sampling batch size of 32 (reason: to stabilize training).

[0074] Furthermore, the specific steps of algorithm training include: ① Initialization: generating a population, creating a central Actor and Critic network, and a target network (target Actor and target Critic). The learning rate is preferably 0.001 (reason: the standard value of the Adam optimizer), the discount factor γ=0.99 (reason: emphasizing long-term rewards), and the soft update parameter τ=0.005 (reason: slowly updating the target network).

[0075] ② Training cycle: The maximum number of training rounds is set to 1000, with each round containing T=100 time steps. The agent interacts with the simulation environment, which is built based on business logic, including inventory updates, cost calculations, and random events.

[0076] ③ Population evolution: An evolutionary operation is performed every 10 training rounds, including evaluation (each individual interacts with the environment for 10 rounds, and fitness is measured by cumulative reward), selection (tournament selection, tournament size is preferred to be 3), crossover (behavioral distillation using Q-value filtering) and mutation (based on proximal mutation).

[0077] ④ Network Update: At each time step, sample from the experience replay buffer, update the central Actor and Critic networks, and use the mean squared error loss function.

[0078] It is understandable that the beneficial effects of the above scheme include: the population mechanism avoids local optima through parallel exploration, thus improving convergence stability; empirical optimization reduces memory overhead, and experiments show that training speed is improved by 20-30%. Those skilled in the art can refer to the algorithm pseudocode implementation.

[0079] Specifically, the population evolution mechanism includes a crossover operator and a mutation operator, wherein the crossover operator achieves policy fusion based on Q-value filtering behavior distillation, and the mutation operator maintains policy stability based on proximal mutation.

[0080] It should be further explained that the present invention refines the operators of the population evolution mechanism, and its specific implementation is as follows: The crossover operator is implemented through behavioral distillation using Q-value filtering, and the specific steps are: 1. Sample state batches from the genetic memory of two parent individuals, batch size... 32 is preferred (reason: to balance computational efficiency and accuracy).

[0081] 2. The Q-value of the parent policy action in each state s is calculated using a central Critic network, and the selection criteria are controlled by the indicator function I(·).

[0082] 3. The offspring strategy is trained by minimizing the loss function (Equation 31), which is: in: : Sampling batch size; : Child and parent strategy network; : Indicator function, takes the value 1 when the condition is true; Q(s,a): Q value output by the Critic network.

[0083] 4. Gradient descent was used for training with a learning rate of 0.001.

[0084] The mutation operator is based on proximal mutation, and the specific steps are as follows: (1) State sampling and gradient calculation: Randomly sample a batch of states from the genetic memory pool of the individual to be mutated, and the batch size is... Preferably, there are 16 pairs of each sampling state. Calculate the gradient of the policy network output with respect to the network parameter θ. Gradients are calculated using automatic differentiation techniques (such as PyTorch's autograd) to ensure computational efficiency (2). Sensitivity (Sen) is calculated using formula (32). The sensitivity calculation strategy is as follows: The physical meaning and calculation details of each component are as follows: The dimension of the action space is 5 in this invention (corresponding to 5 decision variables). State sampling batch size, preferably 16 (reason: to balance computational efficiency and gradient estimation accuracy). In state Below, Sen represents the gradient sensitivity of the k-th output dimension of the policy network to parameter θ, which characterizes the overall sensitivity of the policy network output to parameter perturbations. (3) Parameter perturbation generation: Based on the calculated sensitivity, parameter perturbations are generated: in: σ is preferably 0.1 (reason: experiments show that this variance can achieve a balance between exploration and stability); the perturbation is scaled by the sensitivity Sen to avoid excessive parameter changes.

[0085] (4) The perturbation scaling principle reduces the perturbation amplitude when the sensitivity is high to prevent drastic changes in policy behavior; when the sensitivity is low, the perturbation amplitude is appropriately increased to promote effective exploration. It is understandable that this scaling mechanism is designed based on the core idea of ​​the SM-G-SUM method.

[0086] Understandably, the beneficial effects of the above scheme include: ① Stability guarantee: Through sensitivity scaling, the mutated policy maintains behavioral similarity with the parent policy, avoiding catastrophic forgetting; ② Targeted exploration: Perturbation generation based on gradient information makes the exploration direction more targeted and improves search efficiency; ③ Adaptive adjustment: Sensitivity calculation automatically adapts to the characteristics of different policy networks without the need for manual adjustment of hyperparameters; ④ Experimental verification: In the WEEE inventory optimization problem, this mutation operator improves the convergence speed by about 25% and the final policy quality by 15% compared with the traditional Gaussian mutation.

[0087] 5. Sampling state batches from genetic memory, batch size Option 16 is preferred.

[0088] 6. Calculate the sensitivity of the strategy (Formula 32): The physical meaning of each component is as follows: Action space dimension, value 5; : Batch size for state sampling; The policy outputs the gradient with respect to parameter θ.

[0089] 7. Apply perturbation (Formula 33): ,in The noise variance σ is preferably 0.1 (reason: to avoid excessive variation).

[0090] Understandably, the beneficial effects of the above scheme include: crossover ensures the inheritance of strategy quality, mutation promotes exploration, improves algorithm robustness, and experiments show that the population fitness is improved by 15%.

[0091] Specifically, the experience replay buffer optimization mechanism filters samples based on state reward density, wherein a kernel density estimation method is used to evaluate sample uniqueness in order to improve training efficiency.

[0092] It should be further explained that the experience replay buffer optimization is implemented through the FAC method, and the specific steps include: 1. State space discretization: K-means clustering is used to divide the continuous state space into abstract states, and the number of clusters is preferably 100 (reason: to balance granularity and computational cost). The clustering model is retrained every 1000 time steps.

[0093] 2. Sample processing: For new experience samples Abstract state A(s_t) is assigned through a clustering model.

[0094] 3. Kernel Density Estimation (KDE): Retrieve historical reward data for the abstract state A(s_t) and fit the reward distribution using KDE. The bandwidth h is calculated according to the Silverman rule (Equation 34): The physical meanings of each component are as follows: κ: standard deviation of reward data; IQR: interquartile range of reward data; n: number of historical reward samples.

[0095] 4. Probability density calculation (Formula 35): The kernel function uses the Epanechnikov kernel (Equation 36): 5. Sample selection: Compare p(r|s) with the threshold τ, with τ preferably set to 0.05 (reason: experiments show that this value can effectively select unique samples). If p(r|s) < τ, then store the sample; otherwise, discard it.

[0096] The beneficial effects are reduced buffer redundancy and increased sample diversity; experiments show that training efficiency is improved by 25%.

[0097] Specifically, the experience replay buffer optimization mechanism includes the following steps: discretizing the state space to generate abstract states; for new experience samples, retrieving reward data for the corresponding abstract states; fitting the reward distribution using kernel density estimation and filtering high-value samples based on a preset threshold; and storing the filtered samples in a buffer for network updates.

[0098] For ease of understanding, please refer to Figure 5. Figure 5 is designed to illustrate the "Experience Replay Buffer Optimization Mechanism," which details the specific steps of sample selection in the FAC method in the form of a flowchart. Starting from the input of new experience samples {s_t, a_t, ...}, it gradually shows the complete logical chain of "state space discretization" to obtain "abstract state A(s_t)", querying "reward data", "using Silverman rules to calculate bandwidth", "fitting kernel density model" to calculate probability density P, and finally deciding whether to "discard" or store the sample in the "experience replay buffer" based on the judgment of "P < threshold?".

[0099] It should be further explained that the specific implementation method is as follows: Step 1: State space discretization is achieved through K-means clustering, with 100 clusters and Euclidean distance as the metric. The clustering model is implemented using the scikit-learn library and is updated every 1000 time steps (reason: to adapt to changes in state distribution).

[0100] Step 2: For a new sample, an abstract state is assigned by querying the clustering model, and the reward history data for that state is retrieved from the dictionary. The dictionary is implemented using a hash table, with the key being the abstract state index and the value being the list of rewards.

[0101] Step 3: During kernel density estimation fitting, the bandwidth h is dynamically calculated. The threshold τ = 0.05; if p(r|s) < τ, the sample is considered high value.

[0102] Step 4: After filtering, the samples are stored in a buffer of 10,000, using a first-in, first-out (FIFO) strategy. When updating the network, 32 samples are uniformly sampled from the buffer, the gradient is calculated, and the network is updated.

[0103] It is understood that this invention improves training stability and efficiency by optimizing data utilization, and experiments show that the convergence speed is increased by 30%.

[0104] Furthermore, to evaluate the effectiveness of different methods, this invention constructed 200 simulated test scenarios. The total costs of each method under uncertain market recovery conditions are compared in Tables 1 and 2. Experiments show that PDERL_FAC outperforms the PPO, SQ, A3C, and EVP strategies in cost optimization. The PDERL_FAC method proposed in this invention exhibits more stable performance, and its standard deviation is superior to the PPO strategy.

[0105] Table 1. Total costs obtained by each method when the uncertainty of the market recovery volume is small (standard deviation is indicated by parentheses). Table 2. Total costs obtained by each method when there is significant uncertainty in the amount of goods to be recycled in the market (standard deviation is indicated in parentheses). Understandably, the experiment aims to compare the performance of different inventory optimization methods when facing uncertainty in "market pending recovery volume," using "total cost" as a key indicator to measure their merits. The specific experimental process is as follows: 1. Experimental Design and Comparison Methods: The experiment constructed 200 simulated test scenarios to ensure the statistical significance of the results. The study focuses on two market environments: "low uncertainty" and "high uncertainty." Five methods were compared: PDERL_FAC: The method proposed in this invention, which is a reinforcement learning algorithm that integrates a proximal distillation evolution mechanism and an efficient experience replay buffer.

[0106] PPO: Proximal Policy Optimization Algorithm, an advanced deep reinforcement learning benchmark method.

[0107] SQ: Stochastic Q-learning strategy.

[0108] A3C: Asynchronous Advantage Actor-Critic Algorithm.

[0109] EVP: Expectation Value Programming, a method based on classical stochastic programming.

[0110] In addition, the experiment also considered different configuration combinations of supply chain logistics delay parameters (LC, LP, and LQ represent the transportation time from the collection center to the dismantling center, from the dismantling center to the factory, and from the external supplier to the factory, respectively) to test the robustness of the method under various operating modes.

[0111] 2. Analysis of Core Experimental Results: ① Significant Cost Optimization Advantages: As shown in Tables 1 and 2, under all three parameter configurations tested, the average total cost achieved by the PDERL_FAC method proposed in this invention is the lowest in most cases. Whether in environments with low uncertainty (Table 1) or high uncertainty (Table 2), its cost is superior to methods such as PPO, SQ, A3C, and EVP. This directly demonstrates the superior performance of PDERL_FAC in minimizing the total operating cost of the supply chain.

[0112] ② Strong ability to cope with uncertainty: A comparison of Tables 1 and 2 reveals that when market uncertainty increases, the costs of traditional methods (such as EVP and A3C) rise significantly and fluctuate dramatically. In contrast, the cost increase of PDERL_FAC is relatively gradual, and its leading advantage over the strongest benchmark, PPO, is even more pronounced in certain configurations within environments of high uncertainty. This demonstrates that the method of this invention is better able to adapt to and cope with highly volatile environments, resulting in more robust decision-making.

[0113] ③ Outstanding Strategy Stability: The numbers in parentheses in the table represent the standard deviation, used to measure the stability of strategy performance (the smaller the value, the more stable). Data shows that the standard deviation of PDERL_FAC is the smallest among all methods in most cases, and is even generally lower than the second-best performing PPO method. This confirms the following conclusion: the PDERL_FAC method is more stable. A stable, low-variance strategy is crucial for actual commercial deployment, meaning predictable and reliable operational performance.

[0114] 3. Experimental Conclusions: The experimental results strongly support the technical advantages of this invention. The PDERL_FAC method, through its unique evolutionary population mechanism (promoting global exploration) and efficient experience replay mechanism (improving learning quality), achieves a comprehensive effect of lower cost, greater adaptability, and more stable decision-making in the complex WEEE reverse supply chain inventory optimization problem, significantly surpassing existing advanced reinforcement learning algorithms and traditional planning methods. This provides empirical evidence for the application of this method in real-world industrial scenarios.

[0115] In a preferred embodiment, this application also provides an electronic device comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, which, when executed by the processor, implement the WEEE recycling and reuse inventory optimization method based on evolutionary reinforcement learning. This computer device can be broadly categorized as a server, terminal, or any other electronic device with the necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, memory, network interface, communication interface, etc., connected via a system bus. The processor of the computer device can be used to provide the necessary computing, processing, and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and internal memory. The non-volatile storage medium may store an operating system, computer programs, etc. The internal memory can provide an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface and communication interface of the computer device can be used to connect and communicate with external devices via a network. When the computer program is executed by the processor, it performs the steps of the method of the present invention.

[0116] This invention can be implemented as a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the steps of the methods of embodiments of the invention to be performed. In one embodiment, the computer program is distributed across multiple network-coupled computer devices or processors, such that the computer program is stored, accessed, and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be executed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be executed by one or more computer devices or processors, and one or more other method steps / operations may be executed by one or more other computer devices or processors. One or more computer devices or processors may execute a single method step / operation, or execute two or more method steps / operations.

[0117] Those skilled in the art will understand that the method steps of this invention can be performed by a computer program instructing related hardware, such as a computer device or processor, to perform the steps of this invention when executed. Depending on the context, any references herein to memory, storage, databases, or other media may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0118] The technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification, provided that such combination does not contain contradictions.

[0119] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing WEEE recycling supply chain inventory based on evolutionary reinforcement learning, characterized in that, Includes the following steps: S100. Construct a multi-level inventory collaborative control model for the WEEE recycling reverse supply chain, wherein the reverse supply chain includes a collection center, a dismantling and recycling center, and a product manufacturing plant; S200. Model the inventory optimization problem as a Markov decision process, including defining the state space, action space, and reward function; S300. Train an agent using an evolutionary reinforcement learning algorithm to learn the optimal inventory strategy; S400. Deploy the trained agent into the reverse supply chain for real-time inventory management.

2. The method according to claim 1, characterized in that, The state space is defined based on the inventory status, in-transit orders, and historical environmental information of the supply chain, and is given by the following formula: in: : Indicates the inventory level of the WEEE warehouse at time step t; : Represents the set of WEEE quantities in transit from the collection center to the dismantling and reuse center, including transport orders from time step t-1 to t-L_C; : Indicates the inventory level of the WEEE warehouse at time step t; : Indicates the number of batches of recycled plastic obtained from dismantling and reuse in the previous time step; : Indicates the inventory level of the recycled plastic warehouse at the dismantling and recycling center at time step t; This represents the collection of plastic inventory in transit at the product manufacturing plant, including purchase orders from dismantling and recycling centers and external suppliers. : Indicates the inventory level of the plastic raw material warehouse of the product manufacturer at time step t; : Represents the product demand information for the last σ time steps; : Represents the market recovery amount information at the last σ time steps.

3. The method according to claim 2, characterized in that, The inventory level and in-transit order information in the state space are used to characterize the dynamic uncertainty of the supply chain, where historical demand information and market pending recovery information are simulated through a periodic function model to simulate seasonal fluctuations.

4. The method according to claim 1, characterized in that, The action space includes collection planning, transportation decisions, dismantling decisions, and procurement decisions, wherein the action space is given by the following formula: in: : Indicates the number of WEEE collection plan batches at time step t; : Indicates the number of WEEE batches transported from the collection center to the dismantling and recycling center at time step t; : Indicates the number of batches of WEEE dismantled at the dismantling and reuse center at time step t; : Indicates the batch quantity of recycled plastic pellets purchased by the dismantling and recycling center at time step t; This indicates the number of batches of plastic granules purchased from external suppliers at time step t.

5. The method according to claim 4, characterized in that, The action space is designed as a continuous space and mapped to discrete decisions when interacting with the environment in order to handle high-dimensional action challenges.

6. The method according to claim 1, characterized in that, The reward function is based on the definition of supply chain operating costs, where the reward value is a negative cost, which is calculated by the following formula: in: : Represents the WEEE collection cost, where Cost per batch of collection, This represents the actual amount recovered. : Represents the WEEE disassembly cost, where Cost per batch of dismantling The number of parts to be disassembled; : Represents the production line setup cost, where Cost per production line Number of production lines; : Represents WEEE transportation costs, where Cost per shipment; : Represents the transportation cost of recycled plastics, of which Cost per shipment; : Represents the cost of purchasing plastics, of which Cost per batch of purchases; : Represents the revenue from the sale of scrap metal, of which For metal prices, Metal conversion rate; inventory-related items: For the storage cost of warehouse i, The inventory level at time step t+1. For delay costs, To incur order loss costs, For management costs.

7. The method according to claim 1, characterized in that, The evolutionary reinforcement learning algorithm includes a population evolution mechanism and an experience replay buffer optimization mechanism, wherein the population evolution mechanism enhances the global search capability by exploring multiple strategies in parallel.

8. The method according to claim 7, characterized in that, The population evolution mechanism includes a crossover operator and a mutation operator, wherein the crossover operator achieves policy fusion based on Q-value filtering behavior distillation, and the mutation operator maintains policy stability based on proximal mutation.

9. The method according to claim 7, characterized in that, The experience replay buffer optimization mechanism filters samples based on state reward density, and uses a kernel density estimation method to evaluate sample uniqueness in order to improve training efficiency.

10. The method according to claim 9, characterized in that, The experience replay buffer optimization mechanism specifically includes the following steps: discretizing the state space to generate abstract states; for new experience samples, retrieving reward data for the corresponding abstract states; fitting the reward distribution using kernel density estimation and filtering high-value samples based on a preset threshold; and storing the filtered samples in a buffer for network updates.

Images