A supply chain automation contract agent method based on multi-agent reinforcement learning

Abstract

The present invention discloses a supply chain automation contract agent method based on multi-agent reinforcement learning, and the steps are as follows: S1: two typical scenarios of supply chain transactions are specifically modeled according to the Stackelberg model and the Rubinstein model; S2: expert experience is obtained through the Stackelberg model and the Rubinstein model, and the agent behavior network is pre-trained based on expert experience; S3: according to the game characteristics, the value prediction based on the opponent's strategy is introduced, the feature extraction of the historical transaction sequence is performed using the time series model, and the priority experience playback mechanism is used to gradually optimize the strategy of the agent. The present invention significantly improves the algorithm performance, handles complex and dynamic negotiation tasks, enhances the forward-looking decision-making ability of the agent, accelerates the learning process in the early stage of training, focuses more on important experience in the learning process, and improves the overall learning efficiency and strategy maturity.

Description

Technical Field

[0001] The present invention relates to the dual fields of game theory and artificial intelligence, and in particular to a supply chain automation contract agency method based on multi-agent reinforcement learning. Background Art

[0002] 1. Multi-agent reinforcement learning MARL focuses on optimizing the interaction of multiple decision-making entities in a dynamic environment. By integrating game theory and machine learning principles, MARL enables agents to learn and explore social interaction scenarios, thereby improving decision-making quality and the efficiency of strategic planning.

[0003] 2. In supply chain transactions, game theory and mutually beneficial cooperation have a significant impact on society. This cooperative relationship not only stabilizes market prices, but also provides impetus for economic growth.

[0004] 3. In the multi-agent system MAS, agents have independent negotiation capabilities and can reach consensus on goals or action plans in a rapidly changing environment, which is crucial for scenarios where quick decisions need to be made. Summary of the invention

[0005] In order to solve the above-mentioned technical problems, the present invention proposes a supply chain automation contract agency method based on multi-agent reinforcement learning.

[0006] A supply chain automation contract agent method based on multi-agent reinforcement learning includes the following steps:

[0007] S1: Two typical scenarios of supply chain transactions are specifically modeled based on the Stackelberg model and the Rubinstein model;

[0008] S2: Obtain expert experience through the Stackelberg model and the Rubinstein model, and pre-train the agent behavior network based on expert experience;

[0009] S3: According to the characteristics of the game, we introduce value prediction based on the opponent's strategy, apply the time series model to extract the features of the historical transaction sequence, and the priority experience playback mechanism. By continuously obtaining information from the experience pool, we optimize the network, integrate the architecture algorithm of the opponent's strategy model, and gradually optimize the strategy of the intelligent agent.

[0010] Furthermore, a supply chain automation contract agent method based on multi-agent reinforcement learning includes the following steps:

[0011] S1: Two typical scenarios of supply chain transactions are specifically modeled based on the Stackelberg model and the Rubinstein model;

[0012] S2: Obtain expert experience through the Stackelberg model and the Rubinstein model, and pre-train the agent behavior network based on expert experience;

[0013] S3: According to the characteristics of the game, we introduce value prediction based on the opponent's strategy, apply the time series model to extract the features of the historical transaction sequence, and the priority experience playback mechanism. By continuously obtaining information from the experience pool, we optimize the network, integrate the architecture algorithm of the opponent's strategy model, and gradually optimize the strategy of the intelligent agent.

[0014] Furthermore, in a supply chain automation contract agent method based on multi-agent reinforcement learning, step S1 includes the following sub-steps:

[0015] S11: In the constructed Stackelberg model game environment, two roles are set: supplier and retailer. In each transaction round, the supplier sets the wholesale price of the goods, and the retailer determines the order quantity based on the wholesale price of the goods, and jointly completes the formulation of the contract;

[0016] S12: After completing the contract, both parties interact with the market. The supplier calculates the profit based on the wholesale price, the retailer's order quantity, and the market's cost of goods. The retailer calculates the profit based on the market demand, its own order quantity, and the difference between wholesale and retail prices. The formula is:

[0017] ;

[0018] in, Represents the supplier's profit, Represents the retailer’s profit, wholesale prices set on behalf of suppliers, represents the retailer’s order quantity, represents the supplier's cost of goods, Represents the total demand in the market;

[0019] S13: Calculate the state transfer equation, state, and action of both parties through the profit obtained after the transaction between the two roles is completed. The formula is:

[0020] ;

[0021] in, Represents the time step The state of occurrence, Represents the supplier time step The action that occurs, Represents the supplier time step The action that occurs, Represents the transaction content, represents the time step of the transaction, represents the empty set, Represents immediate reward;

[0022] S14: In the constructed Rubinstein model game environment, the agents play the roles of suppliers and retailers respectively. The environment simulates the key characteristics of alternately providing quotations. The decay factor is introduced to simulate the situation where the utility of the quotation gradually decreases as the negotiation drags on. In each round, the supplier proposes a wholesale price and the retailer responds with an order quantity. Whether a deal is reached depends on the negotiation dynamics of that round.

[0023] S15: Calculate the rewards of both roles based on the sales results and costs, adjust the value reduced by the delay of the agreement, and conduct multiple rounds of incomplete information conditional negotiation games until the transaction is reached or the preset maximum number of rounds is reached. The value of future benefits is discounted by the decay factor;

[0024] S16: If the two parties fail to reach a deal, they will continue to negotiate. The expected utility of both parties will decay according to the decay factor. The two roles will alternately perform contract formulation and contract decision-making. If the supplier formulates the contract and decides the wholesale price and order quantity in the contract, the retailer will decide whether to accept the contract in this round. In the next round, the roles of the two parties will be reversed. The retailer will decide the wholesale price and order quantity in the contract and the supplier will decide whether to accept it, until the two parties reach an agreement.

[0025] S17: Determine the profits of both parties based on the number of rounds and the content of the contract itself:

[0026] If the number of decision-making rounds of both parties exceeds the upper limit of the game round number N, the profit formula of the two agents is:

[0027] ;

[0028] in, represents the profit of the agent supplier, represents the profit of the agent retailer, represents the decay factor, reflecting the loss of potential opportunities over time, and ranges from , Represents the demand, Represents the minimum value, Represents the retail price of the retailer, which is determined by the market. represents the time step;

[0029] If the number of decision-making rounds of both parties exceeds the upper limit of the game round number N, the profit of both parties is 0;

[0030] S18: Since the state transfer equations of the two intelligent agents differ only in different rounds, the state transfer equations of the intelligent agents are calculated by the contract proposer and the contract decision maker. The formula is:

[0031] ;

[0032] in, represents the state transfer equation, Represents the transaction situation, , Including wholesale prices and order quantity , Represents all historical quote information in this round Whether the contract content given by the contract proposer is accepted , Represents all historical transaction quotation information except for this round, Represents the profit earned by the supplier or retailer agent.

[0033] Furthermore, in a supply chain automation contract agent method based on multi-agent reinforcement learning, step S2 includes the following sub-steps:

[0034] S21: Calculate the actions of both suppliers and retailers based on the expert experience calculation method in the Stackelberg model and To generate expert data w(t) and q(w), the wholesale price given by the supplier By the time step of the current transaction The retailer's order quantity By the wholesale price in the current transaction Decide;

[0035] S22: In the Rubinstein model, both parties follow the stubborn-weak strategy set by humans, based on the time step Generate corresponding expert experience;

[0036] S23: After obtaining expert experience, use the expert experience to pre-train the agent:

[0037] Expert-based agent imitation learning: by introducing a large margin classification loss J_E(Q) to fit the actions of both actors’ networks with the corresponding actions in the expert experience;

[0038] Expert-based agent demonstration learning: By putting expert data into the long-term priority replay experience pool in advance, the actor network and the commentator network are optimized based on expert experience to integrate the opponent's strategy model.

[0039] Furthermore, a supply chain automation contract agent method based on multi-agent reinforcement learning, said S21 comprises the following sub-steps:

[0040] S211: Use pre-trained expert experience through the Stackelberg model, the formula is:

[0041] ;

[0042] in, Represents the wholesale price given to suppliers , the dominant order quantity in the retailer's decision, represents the supplier's cost of goods, Represents the total demand in the market;

[0043] Where v represents the temperature coefficient, which is used to control the severity of retailers' reaction to wholesale price increases. The higher the temperature coefficient v, the higher the wholesale price. Increased order quantity will decrease at a faster rate, Represents a Gaussian distribution, which is used to balance the decision between expert experience and exploration actions;

[0044] S212: Calculate the wholesale price given by the supplier based on time in the expert experience. The formula is:

[0045] ;

[0046] in, Represents the time step is the wholesale price given by the supplier of the variable, For is the action value of the time step, and N represents the upper limit of the number of game rounds;

[0047] S213: Imitation learning and agent strategies and expert strategies, IQL imitation learning process, large margin classification loss It is introduced to improve the efficiency of the initial strategy, so that the agent can quickly explore the effective strategy. The formula is:

[0048] ;

[0049] in, represents the large-margin classification loss, Representative Action A collection of represents the action taken by the expert experience in state s, Representing the network, Represents the value of the current state-action pair;

[0050] in, Represents when a = , and a positive margin function otherwise, forcing the value of other actions to be at least one margin lower than the value of the demonstration action.

[0051] Furthermore, in a supply chain automation contract agent method based on multi-agent reinforcement learning, the S22 comprises the following sub-steps:

[0052] S221: Assume that the upper limit of the number of game rounds is N rounds, the final agreement is contract∗, the stubborn negotiation agreement is contractb, and the weak negotiation agreement is contract;

[0053] Then the stubborn negotiation strategy will modify its own agreement contractb near the final round and quickly change it to contract∗, which means sticking to its original decision in the whole round of game tasks;

[0054] Under the weak negotiation strategy, the agent modifies its own agreement contract to quickly become contract∗ in the first few rounds, and is prone to compromise with the other party in the entire round of game tasks;

[0055] S222: The expert strategy used in the Rubinstein environment is a combination of stubborn negotiation strategies and weak negotiation strategies to reach a deal in different rounds, which are given to the intelligent agent for learning as a common experience of human traders in reaching agreements in supply chain transactions.

[0056] Furthermore, a supply chain automation contract agent method based on multi-agent reinforcement learning, said S3 comprises the following sub-steps:

[0057] S31: Environment sending status The agent gets input information from the environment and makes decisions. Input decision feedback to the next state. After the agent makes a decision, the environment provides feedback based on the decision and enters the next state , and provide an immediate reward to the agent ;

[0058] S32: Storing short-term experience The current decision and environmental feedback are stored in the short-term priority replay experience pool for subsequent optimization. When storing long-term experience, the current decision and feedback are also stored in the long-term priority replay experience pool for optimization over a longer time scale.

[0059] S33: Sampling data from the experience pool Extract samples from the short-term and long-term priority experience pools for training, optimize the strategy, and use the priority experience playback mechanism to update and decide the data that needs to be sampled;

[0060] S34: Adversary Strategy Network Optimization Optimize the opponent's strategy network according to the loss function of the expected strategy Train the opponent strategy network model to predict the opponent's behavior. The target behavior network predicts the opponent's strategy for the next round, and the agent uses the optimized strategy to predict the opponent's behavior.

[0061] S35: Target Behavior Network Optimization Optimize the target behavior network, modify actions based on feedback from the opponent's strategy, and optimize the commentator network based on the optimized experience samples;

[0062] S36: Use the feedback obtained from the critic network to finally optimize the behavior network, adjust and optimize the performance of the model according to the TD-Loss temporal difference loss, and use the priority experience replay mechanism to update the priority of the data.

[0063] Furthermore, a supply chain automation contract proxy method based on multi-agent reinforcement learning, the priority experience replay mechanism includes the following sub-steps:

[0064] I1: Calculate the error for each experience , the formula is:

[0065] ;

[0066] in, Representative rewards, represents the discount factor, represents the value of the current state-action pair, represents the maximum expected value of the next state, represents the error of each experience;

[0067] I2: By accumulation The absolute value of plus the constant Obtain the priority of important indicators for evaluating experience learning, the formula is:

[0068] ;

[0069] Among them, z represents the priority, Represents a constant, set to , ensuring that each sample has a chance to be selected;

[0070] I3: Sampling probability of each sample Proportional to the priority, the formula is:

[0071] ;

[0072] in, represents the sampling probability of the sample, represents the sampling priority of sample i, represents the sum of the priorities of all samples k. Represents a non-negative parameter, which is used to adjust the impact of priority on sampling probability. When , the sampling probability of all samples is equal, degenerating into uniform sampling.

[0073] Furthermore, a supply chain automation contract agent method based on multi-agent reinforcement learning, said S34 comprises the following sub-steps:

[0074] S341: Minimize the mean squared error between the next state predicted by the model and the actual next state observed in the environment, as:

[0075] ;

[0076] in, represents the loss function, Representation status Take Action The next state actually observed after Represents the use of parameters Model, input , The next state generated , represents the model parameters;

[0077] S342: Update model parameters using gradient descent , minimize the loss function, the formula is:

[0078] ;

[0079] Among them, β represents the learning rate, represents the updated model parameters, Represents the gradient of the loss function with respect to the model parameters;

[0080] S343: Calculate the value of the state by predicting the future state through the model, and complete the value expansion.

[0081] Furthermore, a supply chain automation contract agent method based on multi-agent reinforcement learning, said S343 comprises the following sub-steps:

[0082] S3431: Trajectory sampling: From the current state Start, according to the strategy Select Action , predict the opponent's strategy through the opponent's strategy model M ;

[0083] S3432: Multi-step expansion: predict the next state, continuously predict multiple steps, and obtain a state sequence , … ;

[0084] Where T represents the time range of the forecast;

[0085] S3433: Calculate cumulative returns: For each predicted state , use the trained value network V to evaluate the value and calculate the cumulative return from time t to t + T. The formula is:

[0086] ;

[0087] in, Represents accumulated returns, represents the discount factor, r represents the prediction reward obtained from the model, represents a variable from 0... to T, Represents the predicted The state of the moment;

[0088] S3434: Optimization: Using an extended value perspective, update the policy or value function by minimizing the difference between the true return and the predicted return or maximizing the predicted return.

[0089] Furthermore, a supply chain automation contract agent method based on multi-agent reinforcement learning, said S13 comprises the following sub-steps:

[0090] S131: Demonstrating the observable states of intelligent agents in the supply chain transaction environment of the Rubinstein bargaining game framework , the matrix is:

[0091] ;

[0092] Each row contains data for a transaction round, including but not limited to contract details, transaction status, and profits of both parties. Represents the supplier at time step Profits, On behalf of retailers Profit at a time step;

[0093] S132: LSTM and Transformer models are used for sequence decision modeling. The LSTM model can effectively handle long-term and short-term dependencies through gating units, and the Transformer model uses position embedding to process the temporal information in the sequence.

[0094] S133: The position embedding component of the Transformer model enables the model to capture the relative or absolute position relationship of elements in the input sequence by adding a unique position dependency signal to each sequence element. Combining these two models, the state matrix is input into the transaction sequence feature learning network.

[0095] The beneficial effects of the present invention are as follows: a negotiation game model is introduced into the application background of the supply chain through a supply chain automation contract agency method based on multi-agent reinforcement learning. A multi-agent reinforcement learning algorithm is designed based on two supply chain transaction environments, namely the Stackelberg model and the Rubinstein bargaining game framework. The algorithm performance is significantly improved by integrating the Monte Carlo method, imitation learning, priority replay experience pool and using Transformer for transaction sequence modeling, and complex and dynamic negotiation tasks are handled. The opponent strategy prediction and value development technology are applied to enhance the forward-looking decision-making ability of the agent. The pre-training method based on imitation learning accelerates the learning process in the early stage of training. The introduction of the priority replay experience pool makes the learning process more focused on important experience, thereby improving the overall learning efficiency and strategy maturity. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 It is the supply chain transaction flow chart of the Stackelberg model.

[0097] Figure 2 It is a flowchart of the actions of retailers in the Stackelberg model game.

[0098] Figure 3 It is a flowchart of the action of the supplier in the Stackelberg model game.

[0099] Figure 4 It is a flowchart of the architecture algorithm optimization of the integrated opponent strategy model based on strategy optimization. DETAILED DESCRIPTION

[0100] The present invention is further described below, but the protection scope of the present invention is not limited to the following description.

[0101] (1) Supply chain transaction scenario modeling based on the Stackelberg model and the Rubinstein model

[0102] As attached Figure 1As shown in the figure, two key roles are set in the constructed supply chain game environment: suppliers and retailers. In each transaction round, suppliers set a certain wholesale price for goods, and retailers determine their order quantity based on this price, thereby jointly completing the formulation of the contract. After completing the contract, both roles will interact with the market. Suppliers calculate their profits based on wholesale prices, retailers' order quantities, and market commodity costs; in contrast, retailers make profits based on market demand, their own order quantities, and the difference between wholesale and retail prices. The specific profit calculation formula is as follows:

[0103] ;

[0104] in, Represents the supplier's profit, Represents the retailer’s profit, wholesale prices set on behalf of suppliers, represents the retailer’s order quantity, represents the supplier's cost of goods, Represents the total demand in the market;

[0105] Supplier Status All are empty sets , the state transfer equations of both parties are as follows:

[0106] ;

[0107] in, Represents the time step The state of occurrence, Represents the supplier time step The action that occurs, Represents the supplier time step The action that occurs, Represents the transaction content, represents the time step of the transaction, represents the empty set, Represents immediate reward;

[0108] As attached Figure 2~3As shown in the figure, in this environment, agents play the roles of suppliers and retailers respectively, and conduct multiple rounds of incomplete information conditional negotiation games. The environment simulates the key characteristics of alternating quotations and introduces a decay factor to simulate the situation where the utility of quotations gradually decreases as the negotiations drag on. In each round, the supplier proposes a wholesale price and the retailer responds with an order volume. Whether a deal is reached depends on the negotiation dynamics of that round. The rewards of both parties are calculated based on the sales results and costs, and the value reduced due to the delay in the agreement is adjusted. This setting makes it possible to dynamically explore negotiation strategies in a controlled and realistic complex supply chain scenario. The value of future gains is discounted through the decay factor, reflecting the loss of potential opportunities over time. The profits of the two agents are calculated as follows:

[0109] ;

[0110] in, represents the profit of the agent supplier, represents the profit of the agent retailer, represents the decay factor, reflecting the loss of potential opportunities over time, and ranges from , Represents the demand, Represents the minimum value, Represents the retail price of the retailer, which is determined by the market. represents the time step;

[0111] If no deal is reached and negotiations continue, the expected utility of both parties will be Attenuation, regarding the mechanism of the environment, the two roles will alternately perform contract formulation and contract decision-making. For example, in the first round of a round of the game, the supplier formulates the contract and determines the wholesale price and order quantity in the contract. In this round, the retailer decides whether to accept the contract; in the next round, the roles of the two parties are swapped, and the retailer determines the wholesale price and order quantity in the contract while the supplier decides whether to accept it until the two parties reach an agreement. At this time, both parties reap profits according to the formula, that is, through the number of rounds in this round and the content of the contract itself. If the number of rounds decided by both parties exceeds the upper limit of the number of game rounds N, the benefits of both parties are 0. The state transfer equation of the agent under the Rubinstein bargaining game framework can be summarized by the following equation. It should be noted that the state transfer equations of the two agents are different only in different rounds, so the state transfer equations of the agents are described by the contract proposer and the contract decision-maker:

[0112] ;

[0113] in, represents the state transfer equation, Represents the transaction situation, , Including wholesale prices and order quantity , Represents all historical quote information in this round Whether the contract content given by the contract proposer is accepted , Represents all historical transaction quotation information except for this round, Represents the profit earned by the supplier or retailer agent.

[0114] (2) Overall architecture of the algorithm

[0115] In the policy-based algorithm architecture, this mainly involves building a network of critics and executors. These networks work together to evaluate the value of decisions made by the intelligent agent in different states and guide the execution of the optimal decision. Specifically, the critic network is responsible for learning the potential value behind different state and action combinations, while the executor network optimizes its strategy through the gradient ascent method based on the critic's value evaluation. In the actual interaction process with the environment, the critic is mainly used to guide decision-making.

[0116] In this process, the opponent strategy model is used to analyze historical transaction data, and data enhancement technology is used to conduct value assessment and predict the opponent behavior that may be caused by each decision step. In addition, based on the predicted opponent behavior, the target actor will generate a response action, and then the target critic will conduct value assessment on these newly generated data. These steps synthesize a batch of data sets for optimization. The current critic network is updated through these data sets and the target critic network, and the temporal difference loss Temporal Difference Loss (TD-Loss) is used to calculate the value error, and the gradient ascent method is used to refine the executor's decision. TD-Loss is also used to update the priority of each experience in the experience replay pool to optimize the learning process and improve the efficiency of the strategy.

[0117] As attached Figure 4 As shown, the algorithm execution and training steps in the game model are:

[0118] S1: Environment sending status to the agent, which takes input from the environment and makes decisions;

[0119] S2: After the decision is entered, the agent will Enter the corresponding decision;

[0120] S3: Feedback on the next state After the agent makes a decision, the environment provides feedback based on the decision and enters the next state , and provide an immediate reward to the agent;

[0121] S4: Store short-term experience The current decision and environmental feedback are stored in the short-term priority playback experience pool for subsequent optimization;

[0122] S5: While storing long-term experience, current decisions and feedback will also be stored in the long-term priority playback experience pool for optimization over a longer time scale;

[0123] S6: Sampling data from the experience pool: Samples are extracted from the short-term and long-term priority experience pools for training, strategy optimization is performed, and the priority experience replay mechanism of the subsequent part is used to update and determine the data that needs to be sampled;

[0124] S7: Adversary strategy network optimization optimizes the opponent's strategy network and trains the opponent's strategy network model based on the loss of the expected strategy;

[0125] S8: Predicting the opponent's behavior Based on the target behavior network, the opponent's strategy for the next round is predicted, and the agent uses the optimized strategy to predict the opponent's behavior;

[0126] S9: Target Behavior Network Optimization Optimize the target behavior network and modify actions based on feedback from the opponent's strategy;

[0127] S10: Optimize the commentator network Based on the optimized experience samples, optimize the commentator network to improve the effect of the overall strategy;

[0128] S11: Behavior Network Optimization Finally, the behavior network is further optimized using the feedback obtained from the critic network, and the performance of the model is further adjusted and optimized based on the TD-Loss temporal difference loss, while using the priority of some update data described in the TD-Loss temporal difference loss and priority experience replay mechanism section.

[0129] The overall process is a cyclical process. By continuously acquiring information from the experience pool and performing network optimization, the strategy of the intelligent agent is gradually optimized.

[0130] Before the algorithm officially starts to execute in the environment and collect data for updating, we need to pre-train the agent actor network based on expert experience. First, according to the expert experience calculation method in the Stackelberg model given in the section Pre-training of Agents Based on Expert Experience, calculate the actions of both suppliers and retailers to generate expert data w(t) and q(w). That is, the wholesale price given by the supplier is determined by the time step of the current transaction, and the purchase amount of the retailer is determined by the wholesale price in the current transaction. In the Rubinstein model, both parties follow the stubborn-weak strategy set by humans and generate corresponding expert experience based on the time step. After obtaining the expert experience, use the expert experience to pre-train the agent, which includes the following two parts, which are carried out simultaneously:

[0131] (1) Agent imitation learning based on expert experience: by introducing a large margin classification loss J_E(Q) to fit the network actions of both actors with the corresponding actions in the expert experience;

[0132] (2) Agent demonstration learning based on expert experience: By placing expert data into the long-term priority replay experience pool in advance, the actor network and the critic network can perform the optimization process shown in the figure above based on expert experience.

[0133] (3) Priority Experience Replay Mechanism

[0134] By evaluating the timeliness of the feedback information obtained, we first improve the implementation of the priority experience replay strategy, assign a priority to all elements in the experience pool, and use TD-Loss, the temporal difference loss value, which is the difference between the current estimate and the actual observed result, to assess the importance of each experience.

[0135] 1. Calculation priority: Priority is calculated by calculating the error TD-Loss for each experience, which is an important indicator for evaluating experience learning:

[0136] Calculate the error for each experience , the formula is:

[0137] ;

[0138] in, Representative rewards, represents the discount factor, represents the value of the current state-action pair, represents the maximum expected value of the next state, represents the error of each experience;

[0139] By accumulating The absolute value of plus the constant Obtain the priority of important indicators for evaluating experience learning, the formula is:

[0140] ;

[0141] Among them, z represents the priority, Represents a constant, set to , ensuring that each sample has a chance to be selected;

[0142] 2. Calculate sampling probability: the sampling probability of each sample Proportional to the priority, the formula is:

[0143] ;

[0144] in, represents the sampling probability of the sample, represents the sampling priority of sample i, represents the sum of the priorities of all samples k. Represents a non-negative parameter, which is used to adjust the impact of priority on sampling probability. When , the sampling probability of all samples is equal, degenerating into uniform sampling.

[0145] (4) Agent pre-training based on expert experience

[0146] Regarding the exploration problem of intelligent agents, the more traditional method is to conduct random exploration, by adding noise to the decision-making of the intelligent agent so that the intelligent agent has a certain exploration ability, or by exploring by directly making random decisions in certain rounds to enhance the exploration ability of the intelligent agent. Another common method is to use imitation learning to pre-train the off-policy algorithm, and use the imitation learning method to improve the efficiency of RL algorithm learning and exploration. A part of the expert experience is put into the playback experience pool in advance, and pre-trained for N rounds before the formal start. Different expert experiences are designed according to the characteristics of different game models in two experimental scenarios. The pre-trained expert experience used in the Stackelberg environment is summarized as follows:

[0147] ;

[0148] in, Represents the wholesale price given to suppliers , the dominant order quantity in the retailer's decision, represents the supplier's cost of goods, Represents the total demand in the market;

[0149] Where v represents the temperature coefficient, which is used to control the severity of retailers' reaction to wholesale price increases. The higher the temperature coefficient v, the higher the wholesale price. Increased order quantity will decrease at a faster rate, Represents a Gaussian distribution, which is used to balance the decision between expert experience and exploration actions;

[0150] Calculate the wholesale price given by suppliers based on time in expert experience. The formula is:

[0151] ;

[0152] in, Represents the time step is the wholesale price given by the supplier of the variable, For is the action value of the time step, and N represents the upper limit of the number of game rounds;

[0153] Imitation learning simulates the strategy of the agent and the expert strategy, the imitation learning process of IQL, and the large margin classification loss It is introduced to improve the efficiency of the initial strategy, so that the agent can quickly explore the effective strategy. The formula is:

[0154] ;

[0155] in, represents the large-margin classification loss, Representative Action A collection of represents the action taken by the expert experience in state s, Representing the network, Represents the value of the current state-action pair;

[0156] in, Represents when a = , otherwise it is a positive margin function, forcing the value of other actions to be at least one margin lower than the value of the demonstration action. This forces the value of other actions to be at least one margin lower than the value of the demonstration action. Doing so forces the value function to mimic the demonstrator's greedy strategy. If there is no supervision loss, The network will have unrestricted values ​​between successive states and will not be able to satisfy the Bellman equation required to improve the policy.

[0157] For the Rubinstein environment, the role of expert experience is to give the agent some learning trajectories for reference, rather than fitting the strategy model to the behavior of expert experience in the Stackelberg environment. The expert experience in the Rubinstein environment is a combination of stubborn and weak negotiation strategies to reach a deal in different rounds. The specific expert behavior trajectory pattern is: assuming that the maximum number of rounds is N rounds, and the final agreement is contract ∗, then the stubborn negotiation strategy will modify its own agreement contract b near the final round and quickly change to contract ∗, and in the whole round of the game task, it will show that it is more persistent in its initial decision; while under the weak negotiation strategy, the agent is likely to modify its own agreement contract c in the first few rounds and quickly change to contract ∗, and in the whole round of the game task, it will show that it is easier to compromise with the other party. The expert strategy used in the Rubinstein environment is to combine the stubborn negotiation strategy and the weak negotiation strategy to reach a deal in different rounds, which is used as a common experience for human traders to form agreements in supply chain transactions and give it to the agent for learning.

[0158] (5) Value prediction based on opponent strategy modeling

[0159] The fitting model targets the opponent's strategy. Since the main trading environment is bilateral, by fitting the opponent's strategy, it can better assist the value estimation components in the agent optimization process, such as the Q table of MADQN and the commentator network of MADDPG.

[0160] The optimization process of the opponent strategy fitting model is as follows:

[0161] 1. Loss function: A common approach is to minimize the mean squared error between the next state predicted by the model and the actual next state observed in the environment:

[0162] ;

[0163] in, represents the loss function, Representation status Take Action The next state actually observed after Represents the use of parameters Model, input , The next state generated , represents the model parameters;

[0164] 2. Update model parameters using gradient descent , minimize the loss function, the formula is:

[0165] ;

[0166] Among them, β represents the learning rate, represents the updated model parameters, Represents the gradient of the loss function with respect to the model parameters;

[0167] 3. Use the model to predict the future state and calculate the value of the state to complete the value expansion:

[0168] Trajectory sampling: From the current state Start by choosing an action according to strategy π , predict the opponent's strategy through the opponent's strategy model M ;

[0169] Multi-step expansion: predict the next state, continuously predict multiple steps, and get a state sequence , … ;

[0170] Where T represents the time range of the forecast;

[0171] Calculate the cumulative return: For each predicted state , use the trained value network V to evaluate the value and calculate the cumulative return from time t to t + T. The formula is:

[0172] ;

[0173] in, represents the accumulated return, γ represents the discount factor, r represents the predicted reward obtained from the model, represents a variable from 0... to T, Represents the predicted The state of the moment;

[0174] Optimization: Using this extended value perspective to update the strategy or value function, usually by minimizing the difference between the actual return and the predicted return or maximizing the predicted return, can make the strategy more focused on long-term benefits rather than just the immediate benefits, and help generate more robust and far-sighted decisions. In practical applications, a relatively accurate environmental model is required to ensure the quality of the prediction, and the amount of computation increases as the number of steps over time increases.

[0175] (6) Transaction sequence modeling

[0176] In the supply chain transaction environment based on the Rubinstein bargaining game framework, the observable state of the agent Including historical quote information of all rounds in the current round and the complete transaction history of previous rounds, this state can be represented by the following matrix form:

[0177] ;

[0178] Each row contains data for a transaction round, including but not limited to contract details, transaction status, and profits of both parties. Represents the supplier at time step Profits, On behalf of retailers Profit at a time step;

[0179] In order to deeply model the supply chain transaction environment based on the Rubinstein bargaining game, LSTM and Transformer models are used for sequential decision modeling. The LSTM model can effectively handle long-term and short-term dependencies through its gating units, while the Transformer model uses position embedding to process timing information in the sequence, which is crucial for correctly interpreting transaction dynamics. The position embedding component of the Transformer model adds a unique position dependency signal to each sequence element, enabling the model to capture the relative or absolute position relationship of elements in the input sequence. In the application of transaction sequence modeling, this allows the Transformer to understand and decode the time order in which transactions occur. The key is to be able to analyze the complex interactions between transaction data at different time points through the self-attention mechanism. Combining these two models, the state matrix It is input into a transaction sequence feature learning network, which not only learns patterns from historical data, but also guides the agent to make optimal strategy choices based on these patterns.

[0180] This solution uses a supply chain automation contract agency method based on multi-agent reinforcement learning to introduce a negotiation game model in the application context of the supply chain. Based on two supply chain transaction environments of the Stackelberg model and the Rubinstein bargaining game framework, a multi-agent reinforcement learning algorithm is designed. By integrating the Monte Carlo method, imitation learning, priority replay experience pool and using Transformer for transaction sequence modeling, the algorithm performance is significantly improved to handle complex and dynamic negotiation tasks. The application of opponent strategy prediction and value development technology enhances the forward-looking decision-making ability of the agent. The pre-training method based on imitation learning accelerates the learning process in the early stages of training. The introduction of the priority replay experience pool makes the learning process more focused on important experience, improving the overall learning efficiency and strategy maturity.

[0181] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A supply chain automation contract agent method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S1: Two typical scenarios of supply chain transactions are specifically modeled based on the Stackelberg model and the Rubinstein model; S2: Obtain expert experience through the Stackelberg model and the Rubinstein model, and pre-train the agent behavior network based on expert experience; S3: According to the characteristics of the game, it introduces value prediction based on the opponent's strategy, applies the time series model to extract the features of the historical transaction sequence, and the priority experience playback mechanism. By continuously obtaining information from the experience pool, it optimizes the network, integrates the architecture algorithm of the opponent's strategy model, and gradually optimizes the strategy of the intelligent agent; The step S1 comprises the following sub-steps: S11: In the constructed Stackelberg model game environment, two roles are set: supplier and retailer. In each transaction round, the supplier sets the wholesale price of the goods, and the retailer determines the order quantity based on the wholesale price of the goods, and jointly completes the formulation of the contract; S12: After completing the contract, both parties interact with the market. The supplier calculates the profit based on the wholesale price, the retailer's order quantity, and the market's cost of goods. The retailer calculates the profit based on the market demand, its own order quantity, and the difference between wholesale and retail prices. The formula is: ； in, Represents the supplier's profit, Represents the retailer’s profit, wholesale prices set on behalf of suppliers, represents the retailer’s order quantity, represents the supplier's cost of goods, Represents the total demand in the market; S13: Calculate the state transfer equation, state, and action of both parties through the profit obtained after the transaction between the two roles is completed. The formula is: ； in, Represents the time step The state of occurrence, Represents the supplier time step The action that occurs, Represents the supplier time step The action that occurs, Represents the transaction content, represents the time step of the transaction, represents the empty set, Represents immediate reward; S14: In the constructed Rubinstein model game environment, the agents play the roles of suppliers and retailers respectively. The environment simulates the key characteristics of alternately providing quotations. The decay factor is introduced to simulate the situation where the utility of the quotation gradually decreases as the negotiation drags on. In each round, the supplier proposes a wholesale price and the retailer responds with an order quantity. Whether a deal is reached depends on the negotiation dynamics of that round. S15: Calculate the rewards of both roles based on the sales results and costs, adjust the value reduced by the delay of the agreement, and conduct multiple rounds of incomplete information conditional negotiation games until the transaction is reached or the preset maximum number of rounds is reached. The value of future benefits is discounted by the decay factor; S16: If the two parties fail to reach a deal, they will continue to negotiate. The expected utility of both parties will decay according to the decay factor. The two roles will alternately perform contract formulation and contract decision-making. If the supplier formulates the contract and decides the wholesale price and order quantity in the contract, the retailer will decide whether to accept the contract in this round. In the next round, the roles of the two parties will be reversed. The retailer will decide the wholesale price and order quantity in the contract and the supplier will decide whether to accept it, until the two parties reach an agreement. S17: Determine the profits of both parties based on the number of rounds and the content of the contract itself: If the number of decision-making rounds of both parties does not exceed the upper limit of the number of game rounds N, the profit formula of the two agents is: ； in, represents the profit of the agent supplier, represents the profit of the agent retailer, represents the decay factor, reflecting the loss of potential opportunities over time, and ranges from , Represents the demand, Represents the minimum value, Represents the retail price of the retailer, which is determined by the market. represents the time step; If the number of decision-making rounds of both parties exceeds the upper limit of the game round number N, the profit of both parties is 0; S18: Since the state transfer equations of the two intelligent entities differ only in different rounds, the state transfer equations of the intelligent entities are calculated by the contract proposer and the contract decision maker. The formula is: ； in, represents the state transfer equation, Represents the transaction status, abbreviated as , Including wholesale prices and order quantity , Including historical transaction quotation information , Transaction Content And the transaction situation , Contains historical transaction quote information , Transaction Content , Represents all historical transaction quotation information except for this round, The profit earned by the agent representing the supplier or retailer; The S3 comprises the following sub-steps: S31: Environment sending status The agent gets input information from the environment and makes decisions. Input decision feedback to the next state. After the agent makes a decision, the environment provides feedback based on the decision and enters the next state , and provide an immediate reward to the agent ; S32: Storing short-term experience The current decision and environmental feedback are stored in the short-term priority replay experience pool for subsequent optimization. When storing long-term experience, the current decision and feedback are also stored in the long-term priority replay experience pool for optimization over a longer time scale. S33: Sampling data from the experience pool Extract samples from the short-term and long-term priority experience pools for training, optimize the strategy, and use the priority experience playback mechanism to update and decide the data that needs to be sampled; S34: Adversary Strategy Network Optimization Optimize the opponent's strategy network according to the loss function of the expected strategy Train the opponent strategy network model to predict the opponent's behavior. The target behavior network predicts the opponent's strategy for the next round, and the agent uses the optimized strategy to predict the opponent's behavior. S35: Target Behavior Network Optimization Optimize the target behavior network, modify actions based on feedback from the opponent's strategy, and optimize the commentator network based on the optimized experience samples; S36: Use the feedback obtained from the critic network to finally optimize the behavior network, adjust and optimize the performance of the model according to the TD-Loss temporal difference loss, and use the priority experience replay mechanism to update the priority of the data.

2. A supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21: Calculate the actions of both suppliers and retailers based on the expert experience calculation method in the Stackelberg model and To generate expert data w(t) and q(w), the wholesale price given by the supplier By the time step of the current transaction The retailer's order quantity By the wholesale price in the current transaction Decide; S22: In the Rubinstein model, both parties follow the stubborn-weak strategy set by humans, based on the time step Generate corresponding expert experience; S23: After obtaining expert experience, use the expert experience to pre-train the agent: Expert-based agent imitation learning: by introducing a large margin classification loss J_E(Q) to fit the actions of both actors’ networks with the corresponding actions in the expert experience; Expert-based agent demonstration learning: By putting expert data into the long-term priority replay experience pool in advance, the actor network and the commentator network are optimized based on expert experience to integrate the opponent's strategy model.

3. A supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 2, characterized in that: The S21 comprises the following sub-steps: S211: Use pre-trained expert experience through the Stackelberg model, the formula is: ； in, Represents the wholesale price given to suppliers , the order quantity of the retailer's advantage in decision making, represents the supplier's cost of goods, Represents the total demand in the market; Where v represents the temperature coefficient, which is used to control the severity of retailers' response to wholesale price increases. The higher the temperature coefficient v, the higher the wholesale price. Increased order quantity will decrease at a faster rate, Represents a Gaussian distribution, which is used to balance the decision between expert experience and exploration actions; S212: Calculate the wholesale price given by the supplier based on time in the expert experience. The formula is: ； in, Represents the time step is the wholesale price given by the supplier of the variable, For is the action value of the time step, and N represents the upper limit of the number of game rounds; S213: Imitation learning and agent strategies and expert strategies, IQL imitation learning process, large margin classification loss It is introduced to improve the efficiency of the initial strategy, so that the agent can quickly explore the effective strategy. The formula is: ； in, represents the large-margin classification loss, Representative Action A collection of represents the action taken by the expert experience in state s, Representing the network, Represents the value of the current state-action pair; in, Representative , and a positive margin function otherwise, forcing the value of other actions to be at least one margin lower than the value of the demonstration action.

4. A supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 2, characterized in that: The S22 comprises the following sub-steps: S221: Assume that the upper limit of the number of game rounds is N rounds, the final agreement is contract∗, the stubborn negotiation agreement is contractb, and the weak negotiation agreement is contract; Then the stubborn negotiation strategy will modify its own agreement contractb near the final round and quickly change it to contract∗, which means sticking to its original decision in the whole round of game tasks; Under the weak negotiation strategy, the agent modifies its own agreement contract to quickly become contract∗ in the first few rounds, and is likely to compromise with the other party in the entire round of the game task; S222: The expert strategy used in the Rubinstein environment is a combination of stubborn negotiation strategies and weak negotiation strategies to reach a deal in different rounds, which are given to the intelligent agent for learning as a common experience of human traders in reaching agreements in supply chain transactions.

5. The supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The priority experience playback mechanism includes the following sub-steps: I1： Calculate the error for each experience , the formula is: ； in, Representative rewards, represents the discount factor, represents the value of the current state-action pair, represents the maximum expected value of the next state, represents the error of each experience; I2: By accumulation The absolute value of plus the constant Obtain the priority of important indicators for evaluating experience learning, the formula is: ； Among them, z represents the priority, Represents a constant, set to , ensuring that each sample has a chance to be selected; I3: Sampling probability of each sample Proportional to the priority, the formula is: ； in, represents the sampling probability of the sample, represents the sampling priority of sample i, represents the sum of the priorities of all samples k. Represents a non-negative parameter, which is used to adjust the impact of priority on sampling probability. When , the sampling probability of all samples is equal, degenerating into uniform sampling.

6. A supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 1, characterized in that: The S34 comprises the following sub-steps: S341: Minimize the mean squared error between the next state predicted by the model and the actual next state observed in the environment, as: ； in, represents the loss function, Representation status Take Action The next state actually observed after Represents the use of parameters Model, input , The next state generated , represents the model parameters; S342: Update model parameters using gradient descent , minimize the loss function, the formula is: ； Among them, β represents the learning rate, represents the updated model parameters, Represents the gradient of the loss function with respect to the model parameters; S343: Calculate the value of the state by predicting the future state through the model, and complete the value expansion.

7. A supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 6, characterized in that: The S343 includes the following sub-steps: S3431: Trajectory sampling: From the current state Start, according to the strategy Select Action , predict the opponent's strategy through the opponent's strategy model M ; S3432: Multi-step expansion: predict the next state, continuously predict multiple steps, and obtain a state sequence , … ; Where T represents the time range of the forecast; S3433: Calculate cumulative returns: For each predicted state , use the trained value network V to evaluate the value and calculate the cumulative return from time t to t + T. The formula is: ； in, Represents accumulated returns, represents the discount factor, r represents the prediction reward obtained from the model, represents a variable from 0... to T, Represents the predicted The state of the moment; S3434: Optimization: Using an extended value perspective, update the policy or value function by minimizing the difference between the true return and the predicted return or maximizing the predicted return.

8. The supply chain automation contract agent method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The S18 comprises the following sub-steps: S181: Demonstrating the observable state of intelligent agents in the supply chain transaction environment of the Rubinstein bargaining game framework , the matrix is: ； Each row contains data for a transaction round, including but not limited to contract details, transaction status, and profits of both parties. Represents the supplier at time step Profits, On behalf of retailers Profit at a time step; S182: LSTM and Transformer models are used for sequence decision modeling. The LSTM model can effectively handle long-term and short-term dependencies through gating units, and the Transformer model uses position embedding to process the temporal information in the sequence. S183: The position embedding component of the Transformer model enables the model to capture the relative or absolute position relationship of elements in the input sequence by adding a unique position dependency signal to each sequence element. Combining these two models, the state matrix is input into the transaction sequence feature learning network.

Images