Inventory optimization method and system based on evolution-assisted multi-agent reinforcement learning

Abstract

The invention relates to the field of reverse supply chain inventory management, and discloses an inventory optimization method and system based on evolution-assisted multi-agent reinforcement learning, and the method comprises the steps: carrying out the processing of the real-time operation data of a supply chain through a trained evolution-assisted multi-agent reinforcement learning model, the method comprises the steps of obtaining real-time operation data of a supply chain, inputting the real-time operation data of the supply chain into the evolution-assisted multi-agent reinforcement learning model for processing, and then outputting a supply chain inventory optimal strategy. According to the invention, through combination of the evolutionary assisted multi-agent reinforcement learning algorithm EAMARL and the online adaptability of the multi-agent reinforcement learning algorithm MARL and the global search capability of the evolutionary algorithm EA, the strategy has stronger robustness for complex dynamic states such as recovered product quantity fluctuation, customized order demand uncertainty and processing delay in the reverse supply chain; and the inventory management efficiency is obviously improved.

Description

Technical Field

[0001] This application belongs to the field of reverse supply chain inventory management, and specifically relates to an inventory optimization method and system based on evolutionary assisted multi-agent reinforcement learning. Background Technology

[0002] The current recycling and reuse of electronic waste (WEEE) faces systemic challenges, including uncertainties in recycling volume and customized order demand, complex collaboration among multiple decision-making centers, and the inability of traditional inventory management models to cope with nonlinear fluctuations in the supply chain. In recent years, deep reinforcement learning-based methods have been widely applied to supply chain inventory management. These methods can dynamically learn and adapt to environmental changes through continuous interaction with the environment, optimizing control strategies. However, they do not consider that the supply chain typically consists of multiple independent decision-making entities, with complex interaction, collaboration, and information-sharing mechanisms among these entities. Summary of the Invention

[0003] The technical problem to be solved by this application is to overcome the shortcomings of the existing technology. This application provides an inventory optimization method and system based on evolutionary assisted multi-agent reinforcement learning. By combining the evolutionary assisted multi-agent reinforcement learning algorithm EAMARL with the online adaptability of the multi-agent reinforcement learning algorithm MARL and the global search capability of the evolutionary algorithm EA, the strategy has stronger robustness to complex dynamics such as fluctuations in the number of recycled products in the reverse supply chain, uncertainty of customized order demand, and processing delays, thus significantly improving inventory management efficiency.

[0004] To achieve the above objectives, the first aspect of this application provides an inventory optimization method based on evolutionary assisted multi-agent reinforcement learning, which includes applying a trained evolutionary assisted multi-agent reinforcement learning model to process real-time operating data of the supply chain. The method includes: acquiring real-time operating data of the supply chain, inputting the real-time operating data of the supply chain into the evolutionary assisted multi-agent reinforcement learning model for processing, and then outputting the optimal inventory strategy for the supply chain. Training the evolutionary aided multi-agent reinforcement learning model includes the following steps: S1. Environment Setup and Problem Definition: A multi-agent reverse supply chain simulation environment is constructed, in which each factory is modeled as an agent to simulate the forward flow and reverse recycling, dismantling and reuse flow of products and parts in the supply chain; based on the constructed multi-agent reverse supply chain simulation environment, a state space, action space and reward function are defined for each agent; S2. Train all the agents using the Evolutionary Assisted Multi-Agent Reinforcement Learning (EAMARL) algorithm and output the optimal policy: The training process of EAMARL is implemented through a two-layer training architecture, and, The two-layer training architecture includes at least the population evolution process of the evolutionary algorithm EA and the gradient learning process of the multi-agent reinforcement learning MARL. The two-layer training architecture also includes the periodic interaction between the EA and the MARL, forming a two-level optimization closed loop. The periodic interaction includes bidirectional experience transfer and synergy between the population evolution process and the gradient learning process. S3. Repeat S2 until the termination condition is met, and output the optimal policy of all trained agents.

[0005] Optionally, the gradient learning process of the multi-agent reinforcement learning (MARL) described in S2 includes: Based on the state, action and reward defined in S1, all the agents are trained centrally using the multi-agent deep reinforcement learning algorithm MADDPG. The MADDPG is configured to train an Actor network, a Critic network, an experience replay buffer, and a target network for each agent. The policy parameters of the Critic network are optimized by randomly sampling experience in the experience replay buffer and by the target value generated by the target network. The optimized Critic network guides the Actor network to update the policy gradient by maximizing its own evaluated Q value.

[0006] Optionally, the MADDPG is configured to train an Actor network, a Critic network, an experience replay buffer, and a target network for each agent, including: The Actor network outputs deterministic actions based on the agent's local observations; The Critic network outputs a Q-value assessment of the agent's actions based on the global state and the joint actions of all agents. The experience replay buffer stores the joint experience generated by the interaction with the environment for training the Actor network and the Critic network. The policy parameters of the target network are softly updated and synchronized with the policy parameters of the Actor network and the Critic network, respectively, to provide stable target values.

[0007] Optionally, the population evolution process of the evolutionary algorithm EA described in S2 includes: Population evolution is performed using the Particle Swarm Optimization (PSO) algorithm, where the parameters of the population evolution include the hyperparameters of the MADDPG algorithm and / or the policy parameters of the Actor network.

[0008] Optionally, the population evolution process of the evolutionary algorithm EA described in S2 further includes: The PSO initializes a policy population for each agent, which contains several individuals. Each individual represents an agent in an actor network with random weights. The actor network uses a velocity-position update mechanism. The fitness of each individual is calculated as the profit obtained by the agent within a complete cycle. Individuals with high fitness serve as guides for group search. The PSO dynamically balances global optimum and individual optimum by updating the velocity and position of each individual, generating a new generation of candidate policies.

[0009] Optionally, the two-layer training architecture further includes periodic interaction between the EA and the MARL, forming a two-layer optimization closed loop, including: The training process of the MADDPG algorithm is paused at a preset interaction node to evaluate the policy performance of all agents and establish a performance benchmark. By mapping the policy parameter vectors of all current agents to the initial population of PSO, the global search capability of PSO is used to explore and optimize in the policy parameter space. Individuals in the initial population are decoded and evaluated. A distributed evaluation strategy is used during the evaluation. The distributed evaluation strategy does not fix the strategies of other agents in the environment, but only evaluates the performance of the target agent's strategy in the given environment, and quantifies the economic benefit index of the evaluation as fitness. After completing a preset number of PSO iterations, the fitness of the best individual in the initial population is compared with the performance benchmark. If the fitness of the best individual is significantly better than the performance benchmark, a policy injection mechanism is triggered, and the policy parameters of the best individual are written into the Actor network of the corresponding agent. The target policy network is then synchronized through soft updates. If the fitness of the best individual is not significantly better than the performance benchmark, the optimization result of the PSO is abandoned, and the agent continues to retain and use the parameters of the original MADDPG policy. The paused state is lifted, and the training process of the MADDPG algorithm is resumed based on the updated or retained strategy, entering the next optimization cycle.

[0010] Optionally, the training process of the MADDPG algorithm can be paused at a preset interaction node to evaluate the policy performance of all agents and establish a performance benchmark, including: When the training reaches the preset time step trigger point, the training process of the current MADDPG algorithm is paused, and all agents are allowed to run multiple rounds in the environment using the latest policy. The average round reward of all agents in the most recent period is calculated, and the average round reward is used as the benchmark reference value for the performance of the current policy.

[0011] Optionally, the parameter vectors of all current agent policies are mapped to the initial population of PSO individuals, and the global search capability of PSO is used to explore and optimize in the policy parameter space, including: The policy parameters of the Actor network of all current agents are extracted and mapped to an individual in the PSO. Each individual represents a complete set of policy parameters of the Actor network, thereby generating an initial PSO population containing multiple individuals. Each individual in the initial PSO population is decoded into a set of temporary policy parameters and evaluated in a distributed manner in a simulation environment. During the evaluation, the policies of all other agents remain unchanged as the currently frozen MADDPG policy, and only the policy of the target agent is replaced with the temporary policy. The performance of the target agent is recorded, and the economic benefits of the target agent are converted into the fitness value of the individual.

[0012] To achieve the above objectives, a second aspect of this application provides an inventory optimization system based on evolutionary assisted multi-agent reinforcement learning, the inventory optimization system comprising: The data acquisition unit is used to acquire real-time operational data of the supply chain; The processing unit is equipped with a trained evolutionary aided multi-agent reinforcement learning model, the training steps of which are as described above; The processing unit is used to receive the real-time operation data of the supply chain acquired by the data acquisition unit, and input the real-time operation data of the supply chain into the evolutionary assisted multi-agent reinforcement learning model for processing, and output the optimal supply chain inventory strategy. A central control unit is used to execute the optimal inventory strategy for the supply chain output by the processing unit on the supply chain.

[0013] To achieve the above objectives, a third aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning as described above.

[0014] By adopting the above technical solution, this application has the following beneficial effects compared with the prior art: In this application, multi-agent reinforcement learning can dynamically adjust inventory management strategies based on real-time supply chain data and complex environments. Through collaboration among multiple agents, it optimizes the decisions of each agent to achieve global goals. An evolutionary algorithm (EA) is introduced to effectively explore a broad strategy space through population evolution, escape local optima, and discover better collaborative inventory strategies, significantly reducing the risks of inventory shortages and backlogs. The evolution-assisted multi-agent reinforcement learning algorithm EAMARL combines the online adaptability of MARL with the global search capability of EA, making the strategy more robust to complex dynamics such as fluctuations in the quantity of recycled products in the reverse supply chain, uncertainty in customized order demand, and processing delays, significantly improving inventory management efficiency.

[0015] In this application, EA-MARL provides an innovative and effective solution for handling complex inventory collaborative optimization problems with multiple nodes and strong coupling in the reverse supply chain. It creatively combines the advantages of evolutionary computation (global exploration, gradient-free optimization) and multi-agent deep reinforcement learning (online learning, collaborative decision-making) to overcome the limitations of single methods (such as MARL's local optima and EA's sample efficiency), and accelerate the discovery and convergence of high-quality strategies.

[0016] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application. The illustrative embodiments and descriptions of the application are used to explain the application, but do not constitute an undue limitation of the application. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] In the attached diagram: Figure 1 This is a flowchart illustrating the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment. Figure 2 This is a flowchart illustrating the process of training the evolutionary auxiliary multi-agent reinforcement learning model in the inventory optimization method based on evolutionary auxiliary multi-agent reinforcement learning in this specific embodiment. Figure 3 This is a flowchart illustrating the WEEE reverse supply chain environment in the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment. Figure 4 This is a schematic diagram of the EAMARL framework in the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment. Figure 5This is a schematic diagram of the MARL architecture in the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment. Figure 6 This is a schematic diagram of the WEEE reverse supply chain learning process based on MADDPG in the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment; Figure 7 This is a flowchart illustrating the particle swarm optimization algorithm (PSO) in the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment. Figure 8 This is a schematic diagram of the system structure of the inventory optimization system based on evolutionary assisted multi-agent reinforcement learning in this specific embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0020] Currently, methods based on deep reinforcement learning (RL) have been widely applied in Supply Chain Inventory Management (SCIM) problems. However, in terms of uncertainty modeling, most studies suffer from oversimplification. While some methods consider the uncertainty of recovery quantities, they often neglect the unique and increasingly important source of uncertainty: customized order demand. Many studies employ single-agent reinforcement learning (SARL) to model the entire supply chain system, aiming to find the globally optimal strategy. However, real-world supply chains typically consist of multiple independent decision-making entities with complex interaction, collaboration, and information-sharing mechanisms. A single agent interacts with a relatively simple environment whose state and dynamic characteristics are usually fixed or follow known rules. A single agent can only acquire local environmental information and cannot fully understand the state of the entire environment.

[0021] Based on this, please see Figure 1 and Figure 2 This application provides an inventory optimization method based on evolutionary assisted multi-agent reinforcement learning, which includes applying a trained evolutionary assisted multi-agent reinforcement learning model to process real-time operation data of the supply chain. The method includes: inputting the real-time operation data of the supply chain into the evolutionary assisted multi-agent reinforcement learning model for processing, and then outputting the optimal inventory strategy for the supply chain. Training an evolutionary assisted multi-agent reinforcement learning model includes the following steps: S1. Environment Setup and Problem Definition: Construct a multi-agent reverse supply chain simulation environment, in which each factory is modeled as an agent to simulate the forward flow and reverse recycling, dismantling, and reuse of products and parts in the supply chain; based on the constructed multi-agent reverse supply chain simulation environment, define the state space, action space, and reward function for each agent; S2. Train all agents and output the optimal policy using the Evolutionary Assisted Multi-Agent Reinforcement Learning (EAMARL) algorithm: The training process of EAMARL is implemented through a two-layer training architecture, and, The two-layer training architecture includes at least the population evolution process of the evolutionary algorithm EA and the gradient learning process of the multi-agent reinforcement learning MARL. The two-layer training architecture also includes the periodic interaction between EA and MARL, forming a two-level optimization closed loop between the two. The periodic interaction includes bidirectional experience transfer and synergy between the population evolution process and the gradient learning process. S3. Repeat S2 until the termination condition is met, and output the optimal policy of all trained agents.

[0022] This application proposes a hybrid algorithm for WEEE reverse supply chain inventory management: Evolutionary Assisted Multi-Agent Reinforcement Learning Algorithm (EAMARL). The optimal policy is obtained through a two-layer training approach: the gradient learning process of the MARL algorithm and the population evolution process of the EA algorithm. In this two-layer training architecture, the gradient learning process of MARL and the population optimization process of EA work together. On the one hand, the policy of the MARL agent is transferred to the evolutionary population, injecting gradient information to accelerate convergence while maintaining the simplicity of EA. On the other hand, the diversity exploration experience of the evolutionary population is transferred to the learning process, enhancing the robustness and stability of the MARL agent.

[0023] It is worth noting that the Multi-Agent Reinforcement Learning (MARL) algorithm is an extension of reinforcement learning, involving multiple agents interacting and learning within an environment. Each agent has its own policy and optimizes it through interactions with the environment and other agents to maximize cumulative rewards. During MARL, agents can form cooperative relationships, jointly optimizing a shared objective function to solve complex decision-making and optimization problems, gradually determining the optimal management strategy, and achieving efficient supply chain operation.

[0024] This application introduces an evolutionary algorithm (EA) to enhance and optimize the multi-agent reinforcement learning process, maintaining a policy population for each decision-making agent in the supply chain network. Each individual in the population represents a possible policy parameter configuration for that agent. During the multi-agent reinforcement learning training process, evolutionary operations are performed periodically: evaluation and fitness calculation, evolutionary operations, and population updates. Multi-agent reinforcement learning is responsible for online learning and policy fine-tuning in environmental interactions, enabling agents to adjust their decisions based on real-time states; while the evolutionary algorithm provides global search and population-level optimization capabilities. By maintaining a diverse population, evaluating system-level performance, and utilizing evolutionary operations, it effectively explores a broader policy space, avoids getting trapped in local optima, and accelerates convergence to a better cooperative policy. The two work synergistically to drive the policy towards the optimal direction.

[0025] It should be noted that, in this embodiment, the execution subject of the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning is an inventory optimization device based on evolutionary assisted multi-agent reinforcement learning. This device can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, etc., while non-mobile electronic devices can be servers and personal computers, etc., and this application does not make specific limitations. The following description uses a server as the execution subject to illustrate the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning in this embodiment.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0027] Please see Figure 3 S1. Environment Construction and Problem Definition: Construct a multi-agent reverse supply chain simulation environment, in which each factory is modeled as an agent to simulate the forward flow and reverse recycling, dismantling and reuse flow of products and parts in the supply chain; Based on the constructed multi-agent reverse supply chain simulation environment, define the state space, action space and reward function for each agent.

[0028] Please continue reading Figure 3 At each time step , the recycling plant, dismantling plant, reuse plant, product manufacturer, and distributor make a series of decisions. Assume the operation sequence for one cycle is as follows: The recycling plant dynamically collects WEEE according to a Poisson distribution, with a collection volume of [missing information]. Not exceeding the capacity of the collection center warehouse The cost of collecting products in each batch is The collected products are temporarily stored in the collection center warehouse.

[0029] The dismantling plant needs to select the dismantling volume from the collection center warehouse. Dismantling volume This is the minimum of the maximum number of items to be dismantled in a single batch and the minimum inventory status in the collection center warehouse. The dismantling plant will be based on its material conversion rate. The materials are dismantled, the metal is sold, and the recycled plastic pellets are stored in a warehouse.

[0030] Recycling plants must reuse materials according to production requirements, selecting the amount to be reused from the dismantling plant's warehouse. reuse amount This is the minimum of the maximum reuse quantity per batch and the minimum inventory status in the dismantling plant warehouse. , This indicates the inventory status of the dismantling factory. This indicates the maximum amount of plastic that can be reused at the recycling plant, and the recycled plastic is stored in a warehouse.

[0031] The product manufacturing plant is divided into order-based production and regular production. When there are no custom orders, the production volume is... First, select from the reuse factory warehouse. Then consider the inventory status of the reuse factory warehouse. If production requirements are met, then the production volume will be [not specified]. Procurement is made from recycling plants. (Regarding inventory status...) If production requirements are not met, external procurement will be made, along with the inventory status of the reused factory warehouse. Production is carried out to meet production requirements, and the finished products are stored in the warehouse. When there are custom orders, the production volume... First, fulfill the order demand; then utilize the factory warehouse inventory status. If the order requirements are met, then the production volume will be... Procurement is made from recycling plants. (Regarding inventory status...) If order requirements are not met, external procurement will be used, along with the existing inventory status of the reused factory warehouse. Production is carried out to meet production requirements, and customized products are stored in the warehouse.

[0032] Retailers sell products based on market demand and inventory levels in the manufacturing plant's warehouses, while the uncertainty of market demand is generated by the system. Sales volume This represents the minimum value between market demand and the inventory level in the product manufacturer's warehouse. , This indicates the inventory status of the product manufacturer. Customized orders, once completed, are sold directly. The sales AI shares sales figures with the product manufacturer, who then adjusts production based on the sales volume.

[0033] Plastic purchase order The order is received by the product manufacturing plant. The product manufacturing plant receives the product production order, specifying its requirements for plastic raw materials. Delivery is either fulfilled or delayed. If demand cannot be met, the remaining unfulfilled orders are recorded as delayed delivery volume. Due to constraints in raw material supply, the actual production volume was , This indicates the amount of plastic granules required to produce one unit of product. < Then the amount of delayed delivery = - .

[0034] Calculate the inventory of each warehouse at the end of time t. That is, the initial inventory at time t+1; The formula for calculating supply chain profit is: (1); Among them, the first item The second item represents the sales revenue of the products sold. The third item represents revenue from the sale of dismantled metals. The fourth item represents the cost of collecting waste materials. The fifth item represents the cost of dismantling operations. The sixth item represents the cost of reuse operations. This represents the seventh item, indicating production operation costs. This indicates the cost of purchasing plastics from external suppliers, item eight. The ninth item indicates a penalty for delayed delivery. This indicates the cost of storing inventory in each warehouse.

[0035] The inventory transfer status for the next state is defined as follows: (2); (3); (4); (5); (6); in, , , , , These represent the inventory status of the intelligent agents being recycled, dismantled, reused, produced, and sold at time step t+1, respectively. , , , , These represent the upper limits of inventory capacity for recycling, dismantling, reuse, production, and sales of intelligent agents, respectively. This indicates the conversion rate of recycled plastic required to produce one unit of product. This indicates the conversion rate at which recycled plastic pellets can produce recycled plastic. At the start of the next time step t+1, the waste product inventory level at the recycling plant equals the inventory at time step t plus the collection volume, minus the batches shipped to the dismantling plant. The recycled plastic pellet inventory at the dismantling plant equals the inventory at time step t plus the dismantling volume, minus the batches shipped to the reuse plant. The recycled plastic inventory at the reuse plant equals the inventory at time step t plus the reuse volume, minus the batches shipped to the product manufacturing plant. The inventory level at the product manufacturing plant equals the inventory at time step t plus the production volume, minus the batches shipped to the distributors. The distributor's inventory level equals the inventory at time step t minus the sales volume. In reality, no plant will have negative inventory, while distributors will face uncertain market demand and may experience stockouts.

[0036] The objective is to maximize operating profit; therefore, the optimization problem to be solved is a nonlinear integer programming problem, defined as follows: (7); ; (8); (9); (10); (11); (12); (13); (14); (15); (16); (17); (18); (19) (20); (twenty one); (twenty two); (twenty three); (twenty four); (25); In Eq.(7), the objective function represents the total operating profit within time step T, the specific meaning of which is explained in detail in equation (1). Regarding constraints, Eq.(8) to (12) stipulate that the sum of the inventory capacity and actions of each factory warehouse cannot exceed the warehouse capacity limit. Eq. (13) stipulates that the amount collected through the Poisson distribution cannot exceed the current remaining storage capacity of the recycling plant, while Eq. (14) to (17) restrict the actions of each agent at time t to not exceed the plant's inventory capacity. Eq. (18) to (22) describe the inventory dynamics of the recycling plant, dismantling plant, reuse plant, manufacturing plant, and retailer's warehouse, respectively. Eq. (23) indicates that the ordered plastic does not exceed the plastic raw material warehouse capacity of the manufacturing plant. Eq. (24) defines the inventory status of each plant as an integer. Eq. (25) defines the range of values ​​for the variables to ensure that the actions of each plant are natural numbers.

[0037] Please see Figure 5 The supply chain inventory management problem satisfies the Markov property, meaning that the inventory and order status at the next moment depend only on the current decision and state, which conforms to the basic assumptions of the Markov Decision Process. Since each node in the supply chain can only observe its own inventory, order status, and local demand, and cannot obtain the global state, this embodiment models the problem as a Partially Observable Markov Decision Process (POMDP). Because the agents at each stage need to collaboratively optimize the overall profit (e.g., sharing inventory information), a multi-agent reinforcement learning method is used to solve it. The POMDP is characterized by tuples <N, S, O, A, P, r, γ>, where: N is the number of agents, S is the global state space of the environment, O = { } is the set of local observations for each agent, A={ Let} be the joint action set, P be the state transition probability function, r be the reward function, and γ be the discount factor. MARL treats all agent actions as joint actions, and the environment feeds back the joint state S and joint reward R to MARL with the aim of learning an optimal policy set A.

[0038] State space design: The state space includes the existing inventory of each agent, the maximum dismantling capacity, and the maximum reuse capacity, etc., and can be given by the following formula: (26); in, This represents the inventory status of the agent at time t. This indicates the maximum number of intelligent agents that can be disassembled. This indicates the maximum number of reusable smart agents. Indicates the WEEE metal briquette conversion rate. This indicates the amount of plastic granules required to produce one unit of product. To simulate the uncertainty in the collection of waste products at recycling plants, a Poisson distribution was used through parameters. The periodic changes, modeling the dynamic fluctuations of the recovered amount, and the periodic collection rate. The expression is: (27); in, Indicates the base collection value. This indicates the fluctuation range of the collected values. This indicates the fluctuation period of the collected values; The actual collection volume is generated through sampling using a Poisson distribution and is constrained by an inventory ceiling. To simulate the uncertainty of market demand, the quantity of customized orders is generated using a discrete uniform distribution.

[0039] Motion space design: For state Action space The recycling volume, dismantling volume, reuse volume, production volume, and sales volume are defined. The action space can be given by the following formula: (28); in, This represents the amount collected by the factory at time t. This represents the amount of material dismantled at time t in the dismantling factory. This represents the amount of material reused by the reuse plant at time t. This represents the production volume of the manufacturing plant at time t. This represents the sales volume of the seller at time t.

[0040] This embodiment designs a continuous action space with independent boundaries. By mapping the continuous action space to discrete actions, the high-dimensional action space challenge in the inventory problem can be effectively solved, and training efficiency can be improved.

[0041] Specifically, the lower limit of each action value is set to zero. The upper limit of the collection volume of the recycling plant is set to the maximum inventory capacity of the recycling plant. The upper limits of the dismantling and reuse volumes are set to the maximum processing level. The upper limit of the production volume is set to the inventory capacity of the product manufacturing plant. The upper limit of the ordering of plastic pellets is set to the plastic raw material warehouse capacity of the product manufacturing plant. The upper limit of the sales volume is set to the inventory capacity of the distributor. In order to implement the constraints of Eq. (8) to (12), if the batches of recycling, dismantling, reuse and production exceed the remaining capacity of the factory warehouse, the environment will directly discard the excess batches, resulting in unnecessary waste of funds. After exceeding the inventory capacity, the state of the agent will be limited to the maximum capacity of the warehouse. In addition, depending on the actual situation, the state of the distributor cannot be negative.

[0042] Reward function design: The optimization objective of this embodiment is to maximize operating profit, while the objective of reinforcement learning is to maximize cumulative reward. To align these two objectives, this embodiment defines the reward function as positive profit, i.e. ,in Based on Eq. (1). This enables agents to achieve optimization goals by learning strategies that maximize profits within the planning time period. However, to address the issue of each agent focusing only on its own local gains and ignoring the synergistic effects of the entire supply chain system.

[0043] This embodiment proposes a hybrid reward design that encourages agents to not only optimize their own operations but also focus on overall performance, promoting collaborative cooperation and balancing short-term gains with long-term performance. The reward function is shown below: (29).

[0044] Please see Figure 3 The gradient learning process of Multi-Agent Reinforcement Learning (MARL) in S2 includes: Based on the state, action and reward defined in S1, all agents are trained centrally using the multi-agent deep reinforcement learning algorithm MADDPG. MADDPG configures an Actor network, a Critic network, an experience replay buffer, and a target network for each agent for training. It optimizes the policy parameters of the Critic network by randomly sampling experience in the experience replay buffer and the target value generated by the target network. The optimized Critic network guides the Actor network to update the policy gradient by maximizing its own evaluated Q value.

[0045] Specifically, the core feature of the Multi-Agent Reinforcement Learning (MARL) algorithm lies in the fact that multiple agents achieve a joint optimal strategy in a partially observable, non-stationary environment through distributed decision-making, credit allocation, and policy coordination. Based on the modeled partially observable Markov decision process, this embodiment utilizes the Multi-Agent Deep Reinforcement Learning (MADDPG) algorithm to train the optimal strategy for WEEE reverse supply chain inventory management.

[0046] MADDPG is a deep reinforcement learning algorithm for multi-agent continuous action spaces. MADDPG adopts a framework of centralized training and decentralized execution: during the training phase, each agent's critic network can access the state and action information of all agents to learn a more stable policy; during the execution phase, each agent relies only on its own actor network to make decisions, thus achieving decentralized control.

[0047] Actor Network From parameterization Each agent i will observe locally Mapping to deterministic actions Critic Network From parameterization Each agent i will have global state and joint actions Mapped to the Q-value, this predicts the cumulative reward. Experience replay buffer. It is the joint experience generated from the interaction between storage and the environment, used to train the actor network and the critic network. The target network is where each agent i possesses a target actor network. and target critic network It is used for stable training, and its parameters are synchronized through soft updates.

[0048] Please see Figure 4 , Figure 5 , Figure 6 and Figure 7 MADDPG trains each agent by configuring an Actor network, a Critic network, an experience replay buffer, and a target network, including: The Actor network outputs deterministic actions based on the agent's local observations; The Critic network outputs a Q-value assessment of the agent's actions based on the global state and the joint actions of all agents. The experience replay buffer stores the joint experience generated from interactions with the environment, which is used for training the Actor network and the Critic network. The policy parameters of the target network are softly updated and synchronized with the policy parameters of the Actor network and the Critic network, respectively, to provide stable target values.

[0049] The training process is conducted iteratively, including policy evaluation (updating the critic) and policy improvement (updating the actor), using randomly sampled batch experience for optimization. The specific steps are as follows: Strategy Evaluation: For each agent i, optimize the critic network parameters. By minimizing the loss function based on time difference (TD) error, it can be expressed as: (30); in, It is an experience replay buffer; target value Defined as: (31); in, It is a discount factor. It is the target actor network of agent j, used to calculate the target action in the next state x′. It is the target critic network of agent i.

[0050] Strategy Improvement: For each agent i, optimize the actor network parameters. By maximizing the Q-value of the critic evaluation, it can be expressed as: (32); Specifically, when updating the actor of agent i, the actor network of other agents is fixed. (j≠i), generate actions using the current policy. Critic network Provide gradients to guide actor updates.

[0051] Target network update: The policy parameters of the target network are synchronized through soft updates to enhance training stability, as follows: (33); (34); in, It is a smoothing factor, and the update is applied to all agents. =1,…,N.

[0052] Please continue reading Figure 7 The population evolution process of the evolutionary algorithm EA in S2 includes: Population evolution is performed using the Particle Swarm Optimization (PSO) algorithm. The parameters of population evolution include the hyperparameters of the MADDPG algorithm and / or the policy parameters of the Actor network.

[0053] The complexity of reverse supply chain inventory management requires algorithms with global optimization capabilities and multi-node collaborative mechanisms. Evolutionary algorithms (EA) are ideal choices due to their gradient-free optimization and swarm search characteristics, while particle swarm optimization (PSO) is the preferred option in this scenario due to its unique pbest / gbest bidirectional learning mechanism, fast convergence, and swarm intelligence metaphor. The collaborative flight of particles towards their individual historical optimal and the swarm's global optimal is essentially a mathematical abstraction of supply chain nodes dynamically sharing local experience and the global optimal solution in the policy space.

[0054] The specific steps are as follows: Initialization: Randomly generate a group of particles and assign each particle a random initial position. and initial velocity (Usually also random within a certain range). Calculate the fitness of each particle. ,initialization Find the globally optimal position gbest; Iterative optimization: For each time step t (until the termination condition is met), for each dimension d in each particle i, calculate using the velocity update formula. , represented as: (35); in, This represents the velocity of particle i at the current time step t in dimension d; It is the tendency to control the particles to maintain their previous speed; It is the step size that controls the particle to move towards its historical best position (pbest); It is the step size that controls the movement of particles toward the best position in the group's history (gbest); and It is a random number that is uniformly distributed within the interval [0, 1]. This represents the best individual position that particle i has found so far (up to time step t) in dimension d; This represents the globally optimal position found by the entire particle swarm in dimension d up to time step t; This represents the position of particle i at the current time step t in dimension d; Calculate using the location update formula , represented as: (36); in, This represents the position of particle i at the current time step t in dimension d. This represents the new velocity calculated using the velocity update formula; Calculate new position fitness ; Update individual optimal :if Superior Then set = ; Update global optimum gbest: Find all current particles The one with the best fitness is set as the new gbest; Output: When the algorithm terminates, gbest is the approximate optimal solution found, and its fitness is... That is the optimal value.

[0055] Please see Figure 5 and Figure 6 The population evolution process of the evolutionary algorithm EA in S2 also includes: PSO initializes a policy population for each agent, which contains several individuals. Each individual represents an agent in an actor network with random weights. The actor network uses a velocity-position update mechanism. The fitness of each individual is calculated as the profit obtained by the agent within a complete cycle. Individuals with high fitness serve as guides for group search. PSO dynamically balances global optimum and individual optimum by updating the velocity and position of each individual, generating a new generation of candidate policies.

[0056] Please continue reading Figure 5 and Figure 6 The two-layer training architecture also includes periodic interaction between EA and MARL, forming a two-layer optimization loop, including: The training process of the MADDPG algorithm is paused at a preset interaction node to evaluate the policy performance of all agents and establish a performance benchmark. By mapping the policy parameter vectors of all current agents to the initial population of PSO, the global search capability of PSO is used to explore and optimize in the policy parameter space. Individuals in the initial population are decoded and evaluated using a distributed evaluation strategy. This strategy does not fix the policies of other agents in the environment; it only evaluates the performance of the target agent's policy under the given environment and quantifies the economic benefits of the evaluation as fitness. , represented as: ; in, Represents the number of rounds evaluated. Represents the maximum step size in a single round. Representing intelligent agents Evaluate the reward at time step t in round k; After completing a preset number of PSO iterations, the fitness of the best individual in the initial population is compared with the performance benchmark. If the fitness of the best individual is significantly better than the performance benchmark, the policy injection mechanism is triggered, and the policy parameters of the best individual are injected. Write the policy parameters into the Actor network corresponding to the agent. Represented as: ; in, Represents the optimal fitness of the PSO population. This represents the current MADDPG evaluation reward. Represents a performance benchmark. This represents a smoothing parameter to prevent the denominator from being zero. And synchronize the target policy network via soft update: ; in, It is a smoothing factor; If the fitness of the best individual is not significantly better than the performance benchmark, the optimization results of PSO are abandoned. The agent continues to retain and use the parameters of the original MADDPG policy; the pause state is lifted, and the training process of the MADDPG algorithm is resumed based on the updated or retained policy, entering the next optimization cycle; Parameters are dynamically adjusted during PSO iterations to maintain population diversity and avoid premature convergence. ; in, Indicates the diversity ratio, The standard deviation represents fitness; a larger standard deviation indicates higher population diversity, which is still under extensive investigation. A smaller standard deviation indicates lower population diversity. This represents the average fitness level.

[0057] Please see Figure 5 The training process of the MADDPG algorithm is paused at a preset interaction node to evaluate the policy performance of all agents and establish a performance benchmark, including: When the training reaches the preset time step trigger point, the training process of the current MADDPG algorithm is paused, and all agents are allowed to run multiple rounds in the environment using the latest policy. The average round reward of all agents in the most recent period is calculated, and the average round reward is used as the benchmark reference value for the performance of the current policy.

[0058] As a specific implementation method, the parameter vectors of all current agent policies are mapped to the initial population of PSO individuals. The global search capability of PSO is then used to explore and optimize within the policy parameter space, including: Extract the policy parameters of the Actor network of all current agents and map them to an individual in the PSO. Each individual represents a complete set of policy parameters of the Actor network, thereby generating an initial PSO population containing multiple individuals. Each individual in the initial PSO population is decoded into a set of temporary policy parameters and evaluated in a distributed manner in the simulation environment. During the evaluation, the policies of all other agents remain unchanged as the currently frozen MADDPG policies, and only the policy of the target agent is replaced with the temporary policy. The performance of the target agent is recorded, and the economic benefits of the target agent are converted into the fitness value of the individual.

[0059] Please see Figure 8 Based on the same inventive concept, this application also provides an inventory optimization system based on evolutionary assisted multi-agent reinforcement learning, the inventory optimization system comprising: The data acquisition unit is used to acquire real-time operational data of the supply chain; The processing unit is equipped with a trained evolutionary aided multi-agent reinforcement learning model, and the training steps of the evolutionary aided multi-agent reinforcement learning model are as described above. The processing unit is used to receive the real-time operation data of the supply chain acquired by the data acquisition unit, and input the real-time operation data of the supply chain into the evolutionary assisted multi-agent reinforcement learning model for processing, and output the optimal inventory strategy of the supply chain. The central control unit is used to execute the optimal supply chain inventory strategy output by the supply chain processing unit.

[0060] Based on the same inventive concept, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning as described above.

[0061] The program product of this application for implementing the above method may employ a portable compact disk read-only memory and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0062] It should be noted that a computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0063] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-mentioned technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. The implementation schemes in the above embodiments can also be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application shall still fall within the scope of this application.

Claims

1. An inventory optimization method based on evolutionary assisted multi-agent reinforcement learning, characterized in that, The method includes applying a trained evolution-assisted multi-agent reinforcement learning model to process real-time operational data of the supply chain. The method includes: acquiring real-time operational data of the supply chain, inputting the real-time operational data of the supply chain into the evolution-assisted multi-agent reinforcement learning model for processing, and then outputting the optimal inventory strategy for the supply chain. Training the evolutionary aided multi-agent reinforcement learning model includes the following steps: S1. Environment Setup and Problem Definition: A multi-agent reverse supply chain simulation environment is constructed, in which each factory is modeled as an agent to simulate the forward flow and reverse recycling, dismantling and reuse flow of products and parts in the supply chain; based on the constructed multi-agent reverse supply chain simulation environment, a state space, action space and reward function are defined for each agent; S2. Train all the agents using the Evolutionary Assisted Multi-Agent Reinforcement Learning (EAMARL) algorithm and output the optimal policy: The training process of EAMARL is implemented through a two-layer training architecture, and, The two-layer training architecture includes at least the population evolution process of the evolutionary algorithm EA and the gradient learning process of the multi-agent reinforcement learning MARL. The two-layer training architecture also includes the periodic interaction between the EA and the MARL, forming a two-level optimization closed loop. The periodic interaction includes bidirectional experience transfer and synergy between the population evolution process and the gradient learning process. S3. Repeat S2 until the termination condition is met, and output the optimal policy of all trained agents.

2. The method according to claim 1, characterized in that, The gradient learning process of Multi-Agent Reinforcement Learning (MARL) described in S2 includes: Based on the state, action and reward defined in S1, all the agents are trained centrally using the multi-agent deep reinforcement learning algorithm MADDPG. The MADDPG is configured to train an Actor network, a Critic network, an experience replay buffer, and a target network for each agent. The policy parameters of the Critic network are optimized by randomly sampling experience in the experience replay buffer and by the target value generated by the target network. The optimized Critic network guides the Actor network to update the policy gradient by maximizing its own evaluated Q value.

3. The method according to claim 2, characterized in that, The MADDPG is configured to train an Actor network, a Critic network, an experience replay buffer, and a target network for each agent, including: The Actor network outputs deterministic actions based on the agent's local observations; The Critic network outputs a Q-value assessment of the agent's actions based on the global state and the joint actions of all agents. The experience replay buffer stores the joint experience generated by the interaction with the environment for training the Actor network and the Critic network. The policy parameters of the target network are softly updated and synchronized with the policy parameters of the Actor network and the Critic network, respectively, to provide stable target values.

4. The method according to claim 2, characterized in that, The population evolution process of the evolutionary algorithm EA described in S2 includes: Population evolution is performed using the Particle Swarm Optimization (PSO) algorithm, where the parameters of the population evolution include the hyperparameters of the MADDPG algorithm and / or the policy parameters of the Actor network.

5. The method according to claim 4, characterized in that, The population evolution process of the evolutionary algorithm EA described in S2 also includes: The PSO initializes a policy population for each agent, which contains several individuals. Each individual represents an agent in an actor network with random weights. The actor network uses a velocity-position update mechanism. The fitness of each individual is calculated as the profit obtained by the agent within a complete cycle. Individuals with high fitness serve as guides for group search. The PSO dynamically balances global optimum and individual optimum by updating the velocity and position of each individual, generating a new generation of candidate policies.

6. The method according to claim 5, characterized in that, The two-layer training architecture also includes periodic interaction between the EA and the MARL, forming a two-layer optimization closed loop, including: The training process of the MADDPG algorithm is paused at a preset interaction node to evaluate the policy performance of all agents and establish a performance benchmark. By mapping the policy parameter vectors of all current agents to the initial population of PSO, the global search capability of PSO is used to explore and optimize in the policy parameter space. Individuals in the initial population are decoded and evaluated. A distributed evaluation strategy is used during the evaluation. The distributed evaluation strategy does not fix the strategies of other agents in the environment, but only evaluates the performance of the target agent's strategy in the given environment, and quantifies the economic benefit index of the evaluation as fitness. After completing a preset number of PSO iterations, the fitness of the best individual in the initial population is compared with the performance benchmark. If the fitness of the best individual is significantly better than the performance benchmark, a policy injection mechanism is triggered, and the policy parameters of the best individual are written into the Actor network of the corresponding agent. The target policy network is then synchronized through soft updates. If the fitness of the best individual is not significantly better than the performance benchmark, the optimization result of the PSO is abandoned, and the agent continues to retain and use the parameters of the original MADDPG policy. The paused state is lifted, and the training process of the MADDPG algorithm is resumed based on the updated or retained strategy, entering the next optimization cycle.

7. The method according to claim 6, characterized in that, The training process of the MADDPG algorithm is paused at a preset interaction node to evaluate the policy performance of all agents and establish a performance benchmark, including: When the training reaches the preset time step trigger point, the training process of the current MADDPG algorithm is paused, and all agents are allowed to run multiple rounds in the environment using the latest policy. The average round reward of all agents in the most recent period is calculated, and the average round reward is used as the benchmark reference value for the performance of the current policy.

8. The method according to claim 6, characterized in that, By mapping the parameter vectors of all current agent policies to the initial population of PSO individuals, the global search capability of PSO is used to explore and optimize within the policy parameter space, including: The policy parameters of the Actor network of all current agents are extracted and mapped to an individual in the PSO. Each individual represents a complete set of policy parameters of the Actor network, thereby generating an initial PSO population containing multiple individuals. Each individual in the initial PSO population is decoded into a set of temporary policy parameters and evaluated in a distributed manner in a simulation environment. During the evaluation, the policies of all other agents remain unchanged as the currently frozen MADDPG policy, and only the policy of the target agent is replaced with the temporary policy. The performance of the target agent is recorded, and the economic benefits of the target agent are converted into the fitness value of the individual.

9. An inventory optimization system based on evolutionary assisted multi-agent reinforcement learning, characterized in that, The inventory optimization system includes: The data acquisition unit is used to acquire real-time operational data of the supply chain; The processing unit is equipped with a trained evolutionary-assisted multi-agent reinforcement learning model, wherein the training steps of the evolutionary-assisted multi-agent reinforcement learning model are as described in claim 1. The processing unit is used to receive the real-time operation data of the supply chain acquired by the data acquisition unit, and input the real-time operation data of the supply chain into the evolutionary assisted multi-agent reinforcement learning model for processing, and output the optimal supply chain inventory strategy. A central control unit is used to execute the optimal inventory strategy for the supply chain output by the processing unit on the supply chain.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it is used to implement the inventory optimization method based on evolutionary assisted multi-agent reinforcement learning as described in any one of claims 1-8.

Images