Multi-supply chain scheduling method and system based on global Critic multi-agent algorithm

Abstract

The invention relates to the technical field of supply chain scheduling, and discloses a multi-supply chain scheduling method and system based on a global Critic multi-agent algorithm, and the method comprises the steps: constructing a multi-agent supply chain environment; introducing a global Critic network, performing reinforcement learning on the multi-agent supply chain environment by using a multi-agent depth deterministic strategy gradient algorithm, and constructing a multi-agent decision model; and inputting supply chain demand data into the multi-agent decision-making model to carry out optimization scheduling processing, and generating a scheduling strategy for maximizing the overall profit of the supply chain. The global Critic network is introduced to reduce multi-agent competition, and the overall performance is optimized. A decision model is constructed by using a multi-agent depth deterministic strategy gradient algorithm to adapt to a complex supply chain environment. And the problems of low efficiency and income are solved by taking overall profit maximization as a target. The environment has high adaptability, and scheduling problems of different scales can be solved. And through continuous learning optimization decision, the reliability is improved, and the efficiency and income are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of supply chain scheduling, and more specifically, to a multiple supply chain scheduling method and system based on a global Critic multi-agent algorithm. Background Art

[0002] At present, there are still obstacles to information exchange between various links in the supply chain, which leads to problems such as increased product costs, extended work schedules, and product quality risks. In order to break down information barriers and make full use of information sharing, it is urgent to find a way to achieve optimal resource allocation between various links in the supply chain based on multi-level supply chain integration. As deep reinforcement learning develops from single-agent to multi-agent, its application in various industries is becoming increasingly widespread.

[0003] At present, deep reinforcement learning algorithms are mainly divided into two types: deterministic and non-deterministic. Non-deterministic algorithms (such as DQN) need to map the selected optimal action to a value, which is suitable for discrete environments with simple action spaces. Deterministic algorithms (such as DDPG) directly output action values ​​and are more suitable for continuous environments with complex action spaces, and perform better in multi-level supply chain scenarios. However, when single-agent deep reinforcement learning algorithms (such as DDPG) are applied in multi-level supply chain scenarios, due to the large dimensions of the state space and action space, there will be defects such as poor convergence, poor stability, and low and unstable overall reward values, resulting in low efficiency and low returns in multiple supply chain environments. Summary of the invention

[0004] In order to overcome the problems of low efficiency and low benefits in multiple supply chain environments caused by the prior art, the present invention proposes the following technical solutions: In a first aspect, the present invention proposes a multi-supply chain scheduling method based on a global critic multi-agent algorithm, comprising: Building a multi-agent supply chain environment.

[0005] A global critic network is introduced, and a multi-agent deep deterministic policy gradient algorithm is used to perform reinforcement learning on the multi-agent supply chain environment to build a multi-agent decision-making model.

[0006] The supply chain demand data is input into the multi-agent decision model for optimized scheduling processing to generate a scheduling strategy that maximizes the overall profit of the supply chain.

[0007] As a preferred technical solution, a multi-agent supply chain environment is constructed, including: According to the overall business logic of multiple supply chains, the state space, action space and reward function are designed respectively.

[0008] A multi-agent supply chain environment is constructed using the state space, the action space and the reward function.

[0009] As a preferred technical solution, the state space includes t The state space of the agent responsible for factory production at all times , and in t The state space of an agent that is responsible for purchasing goods from retailers at all times , and their expressions are as follows: , ; , , ; In this multi-agent supply chain environment, To distinguish between factory and retailer numbers, When , it indicates that the retailer is being operated on, and is the number of steps per training round, is the factory's inventory level, To meet the factory's order requirements, Indicates j The number of goods that a retailer purchases from a factory, K is the total number of retailers; represents the order demand of all retailers. represents the order demand of retailer 1 at time t, represents the order demand of retailer k at time t, Indicates the maximum order quantity, represents the disturbance coefficient, Indicates rounding down. .

[0010] As a preferred technical solution, the action space includes t The action space of the agent responsible for factory production at all times , and in t Action space of an agent that is responsible for stocking the retailer ,and .

[0011] As a preferred technical solution, the reward function is the sorting profit in a multi-level supply chain environment, and its expression is as follows:

[0012] in, is the price of the factory's products, For Retailers The quantity of goods purchased from the factory, The cost of producing products for the factory, Indicates the quantity of products produced by the factory. is the retailer's product price, For Retailers j exist t The order demand at the moment, is the warehousing cost of retailer j, is the inventory level of retailer j, is the penalty coefficient for lack of goods in the warehouse, is the penalty coefficient for exceeding storage capacity, is the maximum storage capacity of retailer j, is the transportation cost of retailer j, is the maximum cargo capacity of each truck of retailer j.

[0013] As a preferred technical solution, a global critic network is introduced, and a multi-agent deep deterministic policy gradient algorithm is used to perform reinforcement learning on the multi-agent supply chain environment to construct a multi-agent decision model, including: Randomly extract data sample batches of a predetermined size, and integrate the data sample batches to form a data set including the agent's current state, next state, action, reward, and completion flag.

[0014] Based on the current state, use the target Actor network to predict the action set for the next state.

[0015] Using the global critic network, the TD target value is calculated based on the action set of the next state.

[0016] According to the TD target value, the global critic network loss is calculated, and the parameters of the global critic network are updated through back propagation.

[0017] The updated global critic network is used to score the action set of the next state, calculate the Actor network loss of each agent, and update the Actor network parameters of each agent through back propagation based on the Actor network loss.

[0018] Repeat the above steps until the multi-agent supply chain environment converges and obtains the multi-agent decision-making model.

[0019] As a preferred technical solution, the supply chain demand data is input into the multi-agent decision model for optimized scheduling processing to generate a scheduling strategy that maximizes the overall profit of the supply chain, including: Supply chain demand data is input into the multi-agent decision model, and the agent selects the optimal action to respond to demand changes based on the current state and environmental information.

[0020] After the agent takes corresponding actions, the state of the agent is updated using the state transfer function to simulate the new state of the supply chain after each decision cycle.

[0021] A global critic network is used to evaluate the efficiency and effectiveness of the current action selection strategy.

[0022] According to the evaluation results of the global critic network, the action selection strategy of the intelligent agent is adjusted.

[0023] Repeat the above steps until the overall profit of the supply chain is maximized and the final supply chain scheduling strategy is generated.

[0024] As a preferred technical solution, the expression of the state transfer function is as follows:

[0025] in, For t The state space of the agent responsible for factory production at all times, For t The state space of the agent that is responsible for purchasing goods from the retailer at any given moment, represents the order demand of all retailers. is the factory's inventory level, Indicates the quantity of products produced by the factory. For Retailers The quantity of goods purchased from the factory, is the storage capacity of the factory, is the inventory level of retailer 1, The quantity that retailer 1 purchases from the factory, For retailers 1 t The order demand at the moment, is the storage capacity of retailer 1, For Retailers K inventory levels, Retailers K The quantity of goods purchased from the factory, For Retailers K exist t The order demand at the moment, For Retailers K storage capacity, To meet the factory's order requirements, is the retailer’s order demand at time t, is the order demand of the retailer at time t-1.

[0026] As a preferred technical solution, the supply chain demand data includes but is not limited to the revenue from retailer sales of products, factory demand, revenue from factory sales of products, cost of factory production of products, warehousing costs, penalties for overdue goods, penalties for exceeding warehousing limits and transportation costs.

[0027] The agent represents a link in the supply chain, including factories and retailers.

[0028] In a second aspect, the present invention further proposes a multiple supply chain scheduling system based on a global critic multi-agent algorithm, which is applied to the multiple supply chain scheduling method based on a global critic multi-agent algorithm as described in any solution of the first aspect, comprising: The first building block is used to construct a multi-agent supply chain environment.

[0029] The second building module is used to introduce a global critic network, use a multi-agent deep deterministic policy gradient algorithm to perform reinforcement learning on the multi-agent supply chain environment, and build a multi-agent decision-making model.

[0030] The generation module is used to input the supply chain demand data into the multi-agent decision model for optimization and scheduling processing, and generate a scheduling strategy that maximizes the overall profit of the supply chain.

[0031] The beneficial effects of the present invention include at least: The present invention can observe and consider the overall reward value by introducing a global Critic network, effectively reducing the competitive confrontation relationship between multiple agents and optimizing the overall performance. The decision model constructed using the multi-agent deep deterministic policy gradient algorithm can better adapt to the complex multiple supply chain environment and generate a better scheduling strategy. By optimizing the scheduling process, a scheduling strategy is generated with the goal of maximizing the overall profit of the supply chain, which effectively solves the problems of low efficiency and low returns in the multiple supply chain environment. The constructed multi-agent supply chain environment has strong adaptability and can handle multiple supply chain scheduling problems of different scales and types. Through continuous learning, the decision model is continuously optimized and adjusted, which improves the reliability and stability of the decision, significantly improves the efficiency and returns of the multiple supply chain environment, and provides an effective solution for the optimal scheduling of complex supply chain systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the multiple supply chain scheduling method based on the global Critic multi-agent algorithm provided in Example 1.

[0033] Figure 2 This is the overall flow chart of a single factory and a single product provided in Example 1.

[0034] Figure 3 This is the overall flow chart of multiple factories and multiple commodities provided in Example 1.

[0035] Figure 4 This is a neural network model framework diagram of the global critic-based MADDPG provided in Example 1.

[0036] Figure 5 This is the MADDPG update flowchart based on the global critic provided in Example 1.

[0037] Figure 6 This is a comparison chart of the overall profits of various algorithms provided in Example 3 in a single-factory, single-product, multi-level supply chain scenario.

[0038] Figure 7 This is a framework diagram of the parameter-free shared neural network model of MAMT-DDPG based on global critic provided in Example 3.

[0039] Figure 8 This is a framework diagram of the hard parameter sharing neural network model of MAMT-DDPG based on global critic provided in Example 3.

[0040] Fig. 9 This is a comparison chart of the overall profits of various algorithms provided in Example 3 in a multi-factory, multi-product, multi-level supply chain scenario.

[0041] Fig.10 The architecture diagram of the multi-supply chain scheduling system based on the global Critic multi-agent algorithm provided in Example 4 DETAILED DESCRIPTION The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0042] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0043] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0044] Example 1 This embodiment proposes a multi-supply chain scheduling method based on a global critic multi-agent algorithm. Figure 1 As shown, Figure 1 The flowchart of a multiple supply chain scheduling method based on a global Critic multi-agent algorithm provided in this embodiment includes the following steps: S1: Building a multi-agent supply chain environment.

[0045] S2: Introduce a global critic network, use a multi-agent deep deterministic policy gradient algorithm to perform reinforcement learning on the multi-agent supply chain environment, and build a multi-agent decision-making model.

[0046] S3: Input the supply chain demand data into the multi-agent decision model for optimized scheduling processing to generate a scheduling strategy that maximizes the overall profit of the supply chain.

[0047] In the specific implementation process, first, according to the actual supply chain situation, a multi-agent supply chain environment is designed and constructed. The environment includes multiple agents, such as factories and retailers. For each agent, a state space, action space and reward function are defined. The state space may include inventory level, order quantity, production capacity, etc. The action space may include order quantity, production volume, delivery volume, etc. The reward function can be designed based on indicators such as profit, cost, customer satisfaction, etc. Figure 2 and Figure 3 As shown, Figure 2 and Figure 3 They are the overall flow chart of a single factory and a single product and the overall flow chart of multiple factories and multiple products.

[0048] Next, design a global critic network that can receive the state and action information of all agents as input and output a global value estimate. The structure of the global critic network can be a multi-layer perceptron (MLP), which contains multiple hidden layers and an output layer. At the same time, each agent is equipped with an actor network to generate actions based on the current state. The structure of the actor network can also be an MLP, with the input being the local state of the agent and the output being the action. Implement the core steps of the multi-agent deep deterministic policy gradient (MADDPG) algorithm, including the experience replay buffer, target network, policy update, etc.

[0049] It is understandable that this embodiment experiments with a variety of deep reinforcement learning algorithms from shallow to deep in the scenario of a multi-level supply chain, including a non-deterministic single-agent deep reinforcement learning algorithm DQN and a deterministic single-agent deep reinforcement learning algorithm DDPG. After selecting the optimal action, DQN needs to map the action again to output a specific numerical value, while DDPG directly outputs the action value, omitting the action mapping process, and is therefore more efficient. The study further expanded the single-agent deep reinforcement learning algorithm DDPG to a multi-agent deep reinforcement learning algorithm MADDPG. Experiments have shown that in the scenario of a multi-level supply chain, the deterministic deep reinforcement learning algorithm performs better than the non-deterministic deep reinforcement learning algorithm, and the multi-agent deep reinforcement learning algorithm performs better than the single-agent deep reinforcement learning algorithm. Based on these results, the MADDPG algorithm was finally selected as the algorithm model for multi-level supply chain scheduling.

[0050] like Figure 4 and Figure 5 As shown, Figure 4 and Figure 5 The following are the neural network model framework diagram of MADDPG based on global Critic and the update flow chart of MADDPG based on global Critic. During the training process, all network parameters are initialized first. For each training round, the environment is reset and the initial state is obtained. In each time step, each agent selects an action based on the current state and the Actor network, obtains the reward and the next state after executing the action, and stores the experience in the replay buffer. Then a batch of experience is sampled from the replay buffer to update the global Critic network and the Actor network of each agent, as well as the target network. Repeat the training rounds until convergence or the preset number of training times is reached.

[0051] After training is completed, the trained Actor network and the global Critic network are combined to form a multi-agent decision model. In practical applications, actual supply chain demand data is first collected and input into the multi-agent decision model. The Actor network of each agent generates corresponding actions (scheduling decisions) based on the input, and the global Critic network evaluates the overall decision quality. Based on the evaluation results, the decision is adjusted to maximize the overall profit of the supply chain.

[0052] It is understandable that the traditional MADDPG algorithm uses a centralized training and distributed execution mode to train agents. Each agent has an Actor-Critic network framework, in which the Actor network outputs the optimal action value based on the currently observed local environment state, and the Critic network scores its own Actor based on the currently observed global environment state and the action value output by other agents. When updating the network parameters, the Critic network parameters are updated first, and then the Actor network parameters are updated. The Critic network uses TD update, and each agent updates its own Critic network based on the reward value obtained by itself, while the Actor network updates its own parameters based on the score given by each agent's own Critic network. However, the traditional MADDPG algorithm ignores the observation of each other's reward value between agents, which may lead to confrontation between agents and a trend of one rising and the other falling. In order to alleviate this situation, this embodiment introduces a global Critic network to replace the Critic network of each agent. This global Critic network is used to score the Actor network of each agent, and the overall reward value can be observed during the update. This method avoids the situation in which agents in the traditional MADDPG algorithm compete with each other in pursuit of maximizing their own reward values, and can better achieve win-win cooperation in multi-level supply chain scenarios. This improvement enables the algorithm to better capture the interdependencies between various levels of the supply chain and promote the coordinated optimization of the overall system.

[0053] Finally, based on the optimized decision, a detailed scheduling strategy is generated, including specific action plans of each agent, such as order quantity, production plan, delivery route, etc. Through the above steps, this method can effectively utilize the multi-agent deep reinforcement learning algorithm, combined with the global critic network, to achieve intelligent scheduling of multiple supply chains and maximize overall profits.

[0054] It can be understood that this embodiment can observe and consider the overall reward value by introducing a global Critic network, effectively reducing the competitive confrontation relationship between multiple agents and optimizing the overall performance. The decision model constructed using the multi-agent deep deterministic policy gradient algorithm can better adapt to the complex multiple supply chain environment and generate a better scheduling strategy. By optimizing the scheduling process, a scheduling strategy is generated with the goal of maximizing the overall profit of the supply chain, which effectively solves the problems of low efficiency and low returns in the multiple supply chain environment. The constructed multi-agent supply chain environment has strong adaptability and can handle multiple supply chain scheduling problems of different scales and types. Through continuous learning, the decision model is continuously optimized and adjusted, which improves the reliability and stability of the decision, significantly improves the efficiency and benefits of the multiple supply chain environment, and provides an effective solution for the optimal scheduling of complex supply chain systems.

[0055] Example 2 This embodiment makes improvements on the multiple supply chain scheduling method based on the global Critic multi-agent algorithm proposed in Embodiment 1.

[0056] In this embodiment, a multi-agent supply chain environment is constructed, including: According to the overall business logic of multiple supply chains, the state space, action space and reward function are designed respectively.

[0057] A multi-agent supply chain environment is constructed using the state space, the action space and the reward function.

[0058] In the specific implementation process, the multi-agent supply chain environment is driven by the final consumer demand to drive the overall data flow of the multi-level supply chain. The supply chain at each level uses the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) multi-agent reinforcement learning algorithm based on the global Critic to decide what action to take at this time, so that information and materials are updated and flowed in the supply chain in a progressive manner. This method can better capture the complex interactions between the various levels of the supply chain.

[0059] The state space refers to the range of environmental states that the agent can perceive and identify, including key indicators such as inventory levels, order backlogs, and production capacity. The action space refers to the range of actions that the agent can take, such as decision variables such as order quantity, production volume, and delivery volume. The reward function is designed to be the overall profit of the multi-level supply chain, while taking into account factors such as inventory cost, out-of-stock cost, and transportation cost to motivate the agent to make decisions that are beneficial to the entire supply chain system. The variable parameters of the reward function in the multi-level supply chain environment are shown in Table 1.

[0060] Table 1 Composition variables of each step of the reward function

[0061] In this embodiment, using To distinguish between factory and retailer numbers, When the factory is operated, When the retailer is operated, is the number of steps in each round of training. The state space is contained in t The state space of the agent responsible for factory production at all times , and in t The state space of an agent that is responsible for purchasing goods from retailers at all times , and their expressions are as follows: , ; , , ; In this multi-agent supply chain environment, To distinguish between factory and retailer numbers, When , it indicates that the retailer is being operated on, and is the number of steps per training round, is the factory's inventory level, To meet the factory's order requirements, Indicates j The number of goods that a retailer purchases from a factory, K is the total number of retailers; represents the order demand of all retailers. represents the order demand of retailer 1 at time t, represents the order demand of retailer k at time t, Indicates the maximum order quantity, represents the disturbance coefficient, Indicates rounding down. .

[0062] In this embodiment, the action space includes t The action space of the agent responsible for factory production at all times , and in t Action space of an agent that is responsible for stocking the retailer ,and .

[0063] In this embodiment, the reward function is the sorting profit in a multi-level supply chain environment, and its expression is as follows:

[0064] in, is the price of the factory's products, For Retailers The quantity of goods purchased from the factory, The cost of producing products for the factory, Indicates the quantity of products produced by the factory. is the retailer's product price, For Retailers j exist t The order demand at the moment, is the warehousing cost of retailer j, is the inventory level of retailer j, is the penalty coefficient for lack of goods in the warehouse, is the penalty coefficient for exceeding storage capacity, is the maximum storage capacity of retailer j, is the transportation cost of retailer j, is the maximum cargo capacity of each truck of retailer j.

[0065] In this embodiment, a global critic network is introduced, and a multi-agent deep deterministic policy gradient algorithm is used to perform reinforcement learning on the multi-agent supply chain environment to construct a multi-agent decision model, including: Randomly extract data sample batches of a predetermined size, and integrate the data sample batches to form a data set including the agent's current state, next state, action, reward, and completion flag.

[0066] Based on the current state, use the target Actor network to predict the action set for the next state.

[0067] Using the global critic network, the TD target value is calculated based on the action set of the next state.

[0068] According to the TD target value, the global critic network loss is calculated, and the parameters of the global critic network are updated through back propagation.

[0069] The updated global critic network is used to score the action set of the next state, calculate the Actor network loss of each agent, and update the Actor network parameters of each agent through back propagation based on the Actor network loss.

[0070] Repeat the above steps until the multi-agent supply chain environment converges and obtains the multi-agent decision-making model.

[0071] As an example, the code for building and updating a multi-agent decision model is as follows: # Randomly extract batch size tuples from the replay experience pool transitions = self.memory.sample(self.BATCH_SIZE) batch = Transition(*zip(*transitions)) states_batch = T.cat(batch.state) next_states_batch = T.cat(batch.next_state) done = T.cat(batch.done).unsqueeze(1) actions_batch = T.cat(batch.action) rewards_batch = T.cat(batch.reward).sum(1).unsqueeze(1)mult_states_list = [] mult_next_states_list = [] # First, update the critic network with T.no_grad(): # Get the action sets of each agent at the next time step for j in range(1, len(self.agents)): s_ = self.split_state_batch(j, next_states_batch, next_actions_batch) next_actions_batch[:, j:] = self.agents[j].target_actor(s_).squeeze(1) s_ = self.split_state_batch(0, next_states_batch, next_actions_batch) next_actions_batch[:, 0] = self.agents[0].target_actor(s_).squeeze(1) # Get the global state sets for j in range(len(self.agents)): mult_states_list.append(self.split_state_batch(j, states_batch,actions_batch)) mult_next_states_list.append(self.split_state_batch(j, next_states_batch, next_actions_batch)) mult_states_batch = T.cat([x for x in mult_states_list], dim=1) mult_next_states_batch = T.cat([x for x in mult_next_states_list],dim=1) # Get the Target value required for TD update q_ = self.target_total_critic(mult_next_states_batch, next_actions_batch) * done target = q_ * self.gamma + rewards_batch # Get the Q value q = self.total_critic(mult_states_batch, actions_batch) # Calculate the loss value of the global Critic network critic_loss = F.mse_loss(q.float(), target.float().detach()) # Clear the global critic network gradient self.total_critic.optimizer.zero_grad() critic_loss.backward()# Update the parameters of the global critic network self.total_critic.optimizer.step() self.total_critic_scheduler.step(critic_loss) # Get a new set of actions for each agent new_actions_batch = T.tensor(actions_batch).detach() # Update actor network for i in range(len(self.agents)): s = self.split_state_batch(i, states_batch, actions_batch) if i == 0: new_actions_batch[:, i] = self.agents[i].actor(s).squeeze(1) else: new_actions_batch[:, i:] = self.agents[i].actor(s).squeeze(1) # Use the global Critic network to score the current agent's actions score = self.total_critic(mult_states_batch, new_actions_batch) # Calculate the loss value of the Actor network actor_loss = -T.mean(score) for i in range(len(self.agents)): self.agents[i].actor.optimizer.zero_grad() actor_loss.backward() # Update the parameters of the Actor network for i in range(len(self.agents)): self.agents[i].actor.optimizer.step() self.agents[i].actor_loss += actor_loss.item() #Soft update the parameters of the two networks if self.step == self.env.episode_length: self.agents[i].agent_update(self.TAU) self.agents[i].actor_scheduler.step(actor_loss) In this embodiment, the supply chain demand data is input into the multi-agent decision model for optimization and scheduling processing to generate a scheduling strategy that maximizes the overall profit of the supply chain, including: Supply chain demand data is input into the multi-agent decision model, and the agent selects the optimal action to respond to demand changes based on the current state and environmental information.

[0072] After the agent takes corresponding actions, the state of the agent is updated using the state transfer function to simulate the new state of the supply chain after each decision cycle.

[0073] A global critic network is used to evaluate the efficiency and effectiveness of the current action selection strategy.

[0074] According to the evaluation results of the global critic network, the action selection strategy of the intelligent agent is adjusted.

[0075] Repeat the above steps until the overall profit of the supply chain is maximized and the final supply chain scheduling strategy is generated.

[0076] In this embodiment, the expression of the state transfer function is as follows:

[0077] in, For t The state space of the agent responsible for factory production at all times, For t The state space of the agent that is responsible for purchasing goods from the retailer at any given moment, represents the order demand of all retailers. is the factory's inventory level, Indicates the quantity of products produced by the factory. For Retailers The quantity of goods purchased from the factory, is the storage capacity of the factory, is the inventory level of retailer 1, The quantity that retailer 1 purchases from the factory, For retailers 1 t The order demand at the moment, is the storage capacity of retailer 1, For Retailers K inventory levels, Retailers K The quantity of goods purchased from the factory, For Retailers K exist t The order demand at the moment, For Retailers K storage capacity, To meet the factory's order requirements, is the retailer’s order demand at time t, is the order demand of the retailer at time t-1.

[0078] In this embodiment, the supply chain demand data includes but is not limited to the revenue of retailers selling products, the demand of factories, the revenue of factories selling products, the cost of factories producing products, storage costs, penalties for overdue goods, penalties for exceeding storage limits and transportation costs.

[0079] The agent represents a link in the supply chain, including factories and retailers.

[0080] It can be understood that the MAMT-DDPG algorithm based on global Critic proposed in the present invention is compared with the DQN algorithm. As a deterministic deep reinforcement learning algorithm, MAMT-DDPG can directly output action values, omitting the action-to-value mapping step in DQN, and is more efficient in multiple supply chain scenarios. Compared with the MADDPG algorithm, MAMT-DDPG integrates multi-task reinforcement learning mechanisms, including parameter-free sharing and hard parameter sharing modes, so that each agent can learn the different characteristics of each commodity and capture more information. Experiments have shown that in a multi-factory and multi-commodity multi-level supply chain environment, the effect of MAMT-DDPG is better than MADDPG. MAMT-DDPG uses a global Critic network to replace the Critic network of each agent in MADDPG, and performs TD updates through the overall reward value, which reduces the competition between agents and exceeds the original MADDPG in overall profit performance.

[0081] The present invention has conducted in-depth research on cooperative communication among multiple agents, constructed an algorithm model that can reduce competition among multiple agents, and proposed a MAMT-DDPG algorithm model that can observe the overall reward value, effectively improving the overall profit in the multi-level supply chain scenario. Experiments show that the MAMT-DDPG algorithm based on global critic performs better than the MADDPG algorithm in overall profit in a single-factory single-product and multi-factory multi-product multi-level supply chain environment, reflecting the superiority of the present invention.

[0082] Example 3 This embodiment compares the multiple supply chain scheduling method based on the global Critic multi-agent algorithm proposed in Example 2 with the following other methods to verify the effect of the multiple supply chain scheduling method based on the global Critic multi-agent algorithm. The environment used in this embodiment is a single-factory single-product multi-level supply chain environment. The compared methods include the strategy based on (ς, Q), the traditional multi-agent reinforcement learning algorithm MADDPG, and the MADDPG algorithm based on the joint Q value. The (ς, Q)-based strategy is a simple and effective inventory management method. When the current inventory level is lower than the preset threshold ς, the system automatically replenishes Q quantity of goods. This strategy is widely used in practice due to its simplicity and practicality, and is used as a baseline for the performance evaluation of other algorithms in this study.

[0083] In the traditional MADDPG algorithm, each agent is equipped with an Actor-Critic network architecture. This allows each agent to observe global state information and the actions of other agents, thereby making more informed decisions. This approach can capture the complex interactions in multi-level supply chains to a certain extent.

[0084] The MADDPG algorithm based on joint Q-value is an improvement on the traditional MADDPG. In this method, the Q-values ​​output by the critic network of each agent are first calculated, and then these Q-values ​​are added together to obtain a joint Q-value. This joint Q-value is then used to update each critic network. This method aims to better reflect the overall performance of the entire system, rather than just the performance of a single agent.

[0085] like Figure 6 As shown, Figure 6 This is a comparison chart of the overall profits of various algorithms in a single-factory, single-product, multi-level supply chain scenario, where Rewards represents the overall profit value in the multi-level supply chain. The experimental results show that the MADDPG algorithm based on the global critic performs best, and its performance has been significantly improved compared to the traditional MADDPG algorithm. This improvement fully demonstrates the advantages of the global critic in coordinating the behaviors of multiple agents and optimizing the overall system performance.

[0086] In order to fully verify the effectiveness of the global critic-based MAMT-DDPG method proposed in the present invention, this embodiment conducts comparative experiments on the method and the following algorithms. The experimental environment is set as a complex scenario with multiple factories, multiple commodities, and multi-level supply chains.

[0087] MAMT-DDPG algorithm without parameter sharing; in this method, each task has its own independent parameter model in the agent's Actor network and Critic network, and no parameters are shared between tasks. This method allows each task to have a highly specialized model, but may increase computational complexity.

[0088] MAMT-DDPG algorithm with hard parameter sharing; in this method, both the Actor network and the Critic network of the agent consist of a shared backbone network and multiple task-specific output branches. All tasks share the backbone network, and then select the corresponding output branch based on the specific task. This method can reduce the number of parameters and improve learning efficiency.

[0089] Parameter-free shared MAMT-DDPG algorithm based on global critic; This is an improvement on the parameter-free shared MAMT-DDPG algorithm, which introduces a global critic network to replace the independent critic networks of all agents. This approach aims to provide a global perspective to better coordinate the behaviors of different agents. Figure 7 As shown, Figure 7 This is a diagram of the parameter-free shared neural network model framework of MAMT-DDPG based on global critic.

[0090] Hard parameter sharing MAMT-DDPG algorithm based on global critic: This is an improvement on the hard parameter sharing MAMT-DDPG algorithm, which also introduces the global critic network. This method combines the efficiency of parameter sharing and the coordination advantages of global critic. Figure 8 As shown, Figure 8 This is a diagram of the hard parameter sharing neural network model framework of MAMT-DDPG based on global critic.

[0091] In addition, this example also compares the above method with the previously mentioned (ς, Q)-based strategy, the traditional multi-agent reinforcement learning algorithm MADDPG, and the global critic-based MADDPG algorithm. This comprehensive comparative experimental design aims to evaluate the performance of the proposed method from multiple perspectives and deeply understand the performance characteristics of different algorithms in a complex supply chain environment.

[0092] like Fig. 9 As shown, Fig. 9 This is a comparison chart of the overall profits of various algorithms in a multi-factory, multi-product, and multi-level supply chain scenario, where Rewards represents the overall profit value of multiple commodities in the multi-level supply chain. Both the parameter-free sharing method and the hard parameter sharing method of MAMT-DDPG exceed the traditional MADDPG algorithm and the MADDPG algorithm based on global critic. In the global critic-based method, both parameter sharing methods of MAMT-DDPG are better than the method without using global critic.

[0093] It is understandable that in order to better coordinate the operation of each link and each platform of the supply chain and enable each sub-unit to achieve global optimization, the present invention proposes a multi-level supply chain scheduling method based on the MADDPG multi-agent deep reinforcement learning algorithm of the global critic. This method uses the multi-level supply chain as the experimental environment of the multi-agent deep reinforcement learning algorithm, and deeply explores the adaptability of various deep reinforcement learning algorithms in the multi-level supply chain scenario. The research results show that the deterministic multi-agent deep reinforcement learning algorithm is most suitable for this link, so MADDPG is selected for further experiments and optimization.

[0094] Based on the experimental results of MADDPG and the exploration of other deep reinforcement learning algorithms, the traditional MADDPG algorithm was innovated. The innovation is to introduce a global Critic network to replace the independent Critic network in all agents and score all Actor networks. This global Critic network can obtain more global information, which helps to promote win-win cooperation among agents and reduce adversarial behavior between them. The experimental results show that the MADDPG algorithm based on global Critic performs more stably and more effectively in multi-level supply chain scenarios, and has obvious advantages over traditional MADDPG.

[0095] In addition, through the experimental results of MAMT-DDPG (Multi-Agent Multi-Task DDPG), it was found that multi-agent multi-task depth is achieved in a more complex multi-factory, multi-commodity, and multi-level supply chain scenario.

[0096] Example 4 like Fig.10 As shown, this embodiment proposes a multiple supply chain scheduling system based on a global critic multi-agent algorithm, which is applied to the multiple supply chain scheduling method based on a global critic multi-agent algorithm as described in the above embodiment, including: a first building module 100, a second building module 200 and a generation module 300.

[0097] The first construction module 100 is used to construct a multi-agent supply chain environment. The second construction module 200 is used to introduce a global critic network, use a multi-agent deep deterministic policy gradient algorithm to perform reinforcement learning on the multi-agent supply chain environment, and construct a multi-agent decision model. The generation module 300 is used to input the supply chain demand data into the multi-agent decision model for optimization and scheduling processing, and generate a scheduling strategy that maximizes the overall profit of the supply chain.

[0098] It should be noted that the aforementioned explanation of the embodiment of the multiple supply chain scheduling method based on the global Critic multi-agent algorithm is also applicable to the multiple supply chain scheduling system based on the global Critic multi-agent algorithm of this embodiment, and will not be repeated here.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0100] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0101] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0102] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0103] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0104] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-supply chain scheduling method based on global critic multi-agent algorithm, It is characterized in that include: Building a multi-agent supply chain environment; A global critic network is introduced, and a multi-agent deep deterministic policy gradient algorithm is used to perform reinforcement learning on the multi-agent supply chain environment to build a multi-agent decision model. The supply chain demand data is input into the multi-agent decision model for optimized scheduling processing to generate a scheduling strategy that maximizes the overall profit of the supply chain.

2. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 1, It is characterized in that Build a multi-agent supply chain environment, including: According to the overall business logic of multiple supply chains, the state space, action space and reward function are designed respectively; A multi-agent supply chain environment is constructed using the state space, the action space and the reward function.

3. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 2, It is characterized in that The state space consists of t The state space of the agent responsible for factory production at all times , and in t The state space of an agent that is responsible for purchasing goods from retailers at all times , and their expressions are as follows: ， ； ， ， ； In this multi-agent supply chain environment, To distinguish between factory and retailer numbers, When , it indicates that the retailer is being operated on, and is the number of steps per training round, is the factory's inventory level, To meet the factory's order requirements, Indicates j The quantity of goods that a retailer purchases from a factory, K is the total number of retailers; represents the order demand of all retailers. represents the order demand of retailer 1 at time t, represents the order demand of retailer k at time t, Indicates the maximum order quantity, represents the disturbance coefficient, Indicates rounding down. .

4. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 3, It is characterized in that The action space consists of t The action space of the agent responsible for factory production at all times , and in t Action space of an agent that is responsible for stocking the retailer , and the total action space set of all agents .

5. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 4, It is characterized in that The reward function is the sorting profit in a multi-level supply chain environment, and its expression is as follows: in, is the price of the factory's products, For Retailers The quantity of goods purchased from the factory, The cost of producing products for the factory, Indicates the quantity of products produced by the factory. is the retailer's product price, For Retailers j exist t The order demand at the moment, is the warehousing cost of retailer j, is the inventory level of retailer j, is the penalty coefficient for lack of goods in the warehouse, is the penalty coefficient for exceeding storage capacity, is the maximum storage capacity of retailer j, is the transportation cost of retailer j, is the maximum cargo capacity of each truck of retailer j.

6. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 5, It is characterized in that The supply chain demand data is input into the multi-agent decision model for optimization and scheduling processing, and a scheduling strategy is generated to maximize the overall profit of the supply chain, including: The supply chain demand data is input into the multi-agent decision model, and the agent selects the optimal action to respond to the demand change based on the current state and environmental information; After the agent takes corresponding actions, the state of the agent is updated using the state transfer function to simulate the new state of the supply chain after each decision cycle; Use the global critic network to evaluate the efficiency and effectiveness of the current action selection strategy; Adjust the agent’s action selection strategy based on the evaluation results of the global critic network; Repeat the above steps until the overall profit of the supply chain is maximized and the final supply chain scheduling strategy is generated.

7. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 6, It is characterized in that The expression of the state transfer function is as follows: in, For t The state space of the agent responsible for factory production at all times, For t The state space of the agent that is responsible for purchasing goods from the retailer at any given moment, represents the order demand of all retailers. is the factory's inventory level, Indicates the quantity of products produced by the factory. For Retailers The quantity of goods purchased from the factory, is the storage capacity of the factory, is the inventory level of retailer 1, The quantity that retailer 1 purchases from the factory, For retailers 1 t The order demand at the moment, is the storage capacity of retailer 1, For Retailers K inventory levels, Retailers K The quantity of goods purchased from the factory, For Retailers K exist t The order demand at the moment, For Retailers K storage capacity, To meet the factory's order requirements, is the retailer’s order demand at time t, is the order demand of the retailer at time t-1.

8. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to claim 1, It is characterized in that A global critic network is introduced, and a multi-agent deep deterministic policy gradient algorithm is used to perform reinforcement learning on the multi-agent supply chain environment to build a multi-agent decision-making model, including: Randomly extracting a data sample batch of a predetermined size, and integrating the data sample batches to form a data set including the current state, next state, action, reward, and completion flag of the agent; Based on the current state, use the target Actor network to predict the action set for the next state; Using the global critic network, the TD target value is calculated based on the action set of the next state; According to the TD target value, the global critic network loss is calculated, and the parameters of the global critic network are updated through back propagation; Use the updated global critic network to score the action set of the next state, calculate the Actor network loss of each agent, and update the Actor network parameters of each agent through back propagation according to the Actor network loss; Repeat the above steps until the multi-agent supply chain environment converges and obtains the multi-agent decision-making model.

9. The multiple supply chain scheduling method based on the global critic multi-agent algorithm according to any one of claims 1 to 8, It is characterized in that The supply chain demand data includes but is not limited to the revenue of retailers from selling products, the demand of factories, the revenue of factories from selling products, the cost of factories producing products, storage costs, penalties for overdue goods, penalties for exceeding storage limits and transportation costs; The agent represents a link in the supply chain, including factories and retailers.

10. A multi-supply chain scheduling system based on global critic multi-agent algorithm, It is characterized in that include: The first building block is used to build a multi-agent supply chain environment; The second building module is used to introduce a global critic network, use a multi-agent deep deterministic policy gradient algorithm to perform reinforcement learning on the multi-agent supply chain environment, and build a multi-agent decision model; The generation module is used to input the supply chain demand data into the multi-agent decision model for optimization and scheduling processing, and generate a scheduling strategy that maximizes the overall profit of the supply chain.

Images