A supply chain inventory management method based on a transformer

Abstract

The application discloses a supply chain inventory management method based on a Transformer. In view of the problems of rapid scale growth and high decision complexity of an aircraft manufacturing supply chain inventory management model, the application converts a multi-agent reinforcement learning problem into a sequence modeling problem of an encoder-decoder architecture based on the essence of multi-agent reinforcement learning sequence decision, and logically reduces the complexity of the algorithm. The application designs a lightweight encoder module and an autoregressive decoder module. The lightweight encoder module replaces a self-attention mechanism with high complexity by utilizing the duality of Fourier transform, and reduces the algorithm complexity in implementation. The autoregressive decoder module ensures the monotone improvement of the joint strategy of the whole supply chain by the autoregressive sequential decision mode based on the idea of the information transmission mechanism. The method can effectively simplify the training time, model parameter quantity and the like relative to other existing methods while maintaining the performance.

Description

Technical fields:

[0001] This invention relates to the field of supply chain inventory management, and in particular to a supply chain inventory management method based on Transformer. Background technology:

[0002] Supply chain inventory management refers to the methods used in each link of the supply chain to rationally plan and control the reserve quantities of materials, components, finished products, etc., in order to optimize supply chain operations, reduce costs, and improve efficiency.

[0003] In recent years, with the rapid development of artificial intelligence, supply chain inventory management has made continuous progress. In particular, with the success of reinforcement learning, it has been widely applied in the field of supply chain inventory management.

[0004] Closest to existing technology and its evaluation:

[0005] Oroojlooy et al. (Oroojlooy A. Applications of machine learning in supply chains[D]. Lehigh University, 2019.) used the Deep Q-Network (DQN) algorithm to solve the supply chain inventory management problem.

[0006] Kemmer et al (Kemmer L, von Kleist H, de D, et al. Reinforcement learning for supply chain optimization[C] / / European Workshop on Reinforcement Learning.2018,14(10).) The State-Action-Reward-State-Action (SARSA) algorithm is used to solve the inventory management problem of the secondary supply chain.

[0007] Alves and Mateus et al. (Alves JC, Mateus G R. Deep reinforcement learning and optimization approach for multi-echelon supply chain with uncertain demands[C] / / Computational Logistics: 11th International Conference, ICCL 2020, Enschede, The Netherlands, September 28–30, 2020, Proceedings 11. Springer International Publishing, 2020: 584-599.) used a single-agent reinforcement learning method to solve the inventory management problem in a four-level supply chain.

[0008] Alves and Silva et al. (Alves JC, Silva DM, Mateus G R. Applying and comparing policy gradient methods to multi-echelon supply chains with uncertain demands and lead times[C] / / Artificial Intelligence and SoftComputing:20th International Conference,ICAISC 2021,Virtual Event,June 21–23,2021,Proceedings,Part II.Cham:Springer International Publishing,2021:229-239.) compared the efficiency of several classical reinforcement learning methods in a cooperative supply chain inventory management environment.

[0009] Wang et al. (Wang H, Tao J, Peng T, et al. Dynamic inventory replenishments strategy for aerospace manufacturing supply chain: combining reinforcement learning and multi-agent simulation[J]. International Journal of Production Research, 2022, 60(13): 4117-4136.) used reinforcement learning to solve the inventory management problem in the aircraft manufacturing industry.

[0010] While the methods described above can provide some management of supply chain inventory issues, several problems remain to be addressed: the complexity of the problem increases exponentially with the number of supply chain participants; although the training of each supply chain participant's agent is logically parallel, a serial update strategy is still required in practice to ensure monotonic updates during performance; and different supply chain participants' agents can observe different states and take different actions, requiring separate design for these heterogeneous agents. Summary of the Invention

[0011] The purpose of this invention is to provide a supply chain inventory management method based on Transformer.

[0012] This invention addresses three problems faced by existing methods in solving inventory management in the aircraft manufacturing supply chain: high method complexity, reliance on serial updates by each agent during training, and cumbersome handling of heterogeneous agents. It proposes a Transformer-based supply chain inventory management method. Based on the sequential decision-making nature of multi-agent reinforcement learning, this method utilizes a lightweight encoder module and an autoregressive decoder module designed in this invention to transform the multi-agent reinforcement learning problem into a sequence modeling problem. It solves the problems of rapid model scaling and high decision complexity faced by existing methods from three angles: simplifying the model using Fourier transform, using a centrally maintained encoder-decoder structure for parallel training, and using a masked attention mechanism to avoid heterogeneous agents. Compared with existing multi-agent reinforcement learning-based supply chain inventory management methods, the algorithm proposed in this invention can improve model performance while simplifying the multi-agent reinforcement learning model structure.

[0013] The technical solution of this invention is as follows:

[0014] A Transformer-based supply chain inventory management method is characterized by including a main manufacturer agent and a supplier agent, and the agents are able to interact according to specific logic to simulate inventory changes during the supply chain operation.

[0015] The number of main manufacturer agents is one, and the main manufacturer agent has a corresponding policy function π0 and value function V0 in logic;

[0016] The number of supplier agents is N (where N is a positive integer greater than or equal to 1), and each supplier agent logically has a corresponding policy function π. i And value function V i (where i = 1, 2, 3, ..., N, is the sequence number of the supplier's intelligent agent);

[0017] The value function V of each supply chain participant's intelligent agent i (where i = 0, 1, 2, 3, ..., N) is fitted by a lightweight encoder;

[0018] The policy function π of each supply chain participant's intelligent agent i (where i = 0, 1, 2, 3, ..., N) is fitted by an autoregressive decoder set;

[0019] The specific logic used to interactively simulate the operation of a master manufacturer-supplier model supply chain executes a training algorithm process at each time t, where time t is in the form of a natural week.

[0020] By adopting the above solution, the beneficial effects of the present invention are:

[0021] This invention proposes a Transformer-based supply chain inventory management method to address three problems faced by traditional reinforcement learning approaches in solving inventory management within the aircraft manufacturing supply chain. Based on the sequential decision-making nature of multi-agent reinforcement learning, this method utilizes a lightweight encoder module and an autoregressive decoder module designed in this invention to transform the multi-agent reinforcement learning problem into a sequence modeling problem. It addresses the issues of rapid model scaling and high decision complexity in existing methods by simplifying the model using Fourier transform, performing parallel training using a centrally maintained encoder-decoder structure, and avoiding heterogeneous agents through a masked attention mechanism.

[0022] Experimental results show that, compared with existing supply chain inventory management methods based on multi-agent reinforcement learning, the algorithm proposed in this invention can improve model performance while simplifying the structure of multi-agent reinforcement learning models. Attached Figure Description

[0023] Figure 1 This is a process flow diagram in a reinforcement learning environment according to an embodiment of the present invention.

[0024] Figure 2 This is the model network structure diagram of the present invention.

[0025] Figure 3 This invention relates to the training process of a master manufacturer-supplier model supply chain inventory management method based on multi-agent reinforcement learning.

[0026] Figure 4 This is the main flowchart of the master manufacturer-supplier model supply chain inventory management method based on multi-agent reinforcement learning of the present invention.

[0027] Figure 5 This is a comparison chart of the experimental results of the present invention and the prior art. Detailed Implementation

[0028] A Transformer-based supply chain inventory management method includes a master manufacturer agent and a supplier agent, and the agents can interact according to specific logic to simulate inventory changes during the supply chain operation.

[0029] The number of main manufacturer agents is one, and the main manufacturer agent has a corresponding policy function π0 and value function V0 in logic;

[0030] The number of supplier agents is N (where N is a positive integer greater than or equal to 1), and each supplier agent logically has a corresponding policy function π. i And value function V i (where i = 1, 2, 3, ..., N, is the sequence number of the supplier's intelligent agent);

[0031] The value function V of each supply chain participant's intelligent agent i (where i = 0, 1, 2, 3, ..., N) is fitted by a lightweight encoder;

[0032] The policy function π of each supply chain participant's intelligent agent i (where i = 0, 1, 2, 3, ..., N) is fitted by an autoregressive decoder set;

[0033] The specific logic used to interactively simulate the operation of a master manufacturer-supplier model supply chain executes a training algorithm process at each time t, where time t is in the form of a natural week.

[0034] The technical solution of the present invention will be further described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments.

[0035] This embodiment involves applications such as Figure 1 shown: 10 process flows abstracted from the aircraft final assembly process. Assume that each process flow involves three types of materials: corresponding structural parts, system parts, and standard parts. Among them, the structural parts and system parts of each process flow are different, and two standard part materials are shared by all process flows. Therefore, a total of 22 types of materials are involved as shown in Table 1. In this embodiment, assume that each material is provided by a single supplier, and each supplier only provides one material. Then, a total of 23 agents are involved in the environment, including 1 main manufacturer agent and 22 supplier agents.

[0036] First, construct a network. The network structure of the model is as Figure 2 shown.

[0037] The entire network consists of two parts, namely a lightweight encoder that encodes the observation information of each supply chain participant agent and an autoregressive decoder that generates the optimal actions of each supply chain participant agent.

[0038] During the encoding process, the joint observation information o t After passing through the Multilayer Perceptron (MLP) module, it is encoded by multiple lightweight encoders into the encoded observation information The encoded observation information After passing through the MLP module, the value evaluation of each agent can be obtained Specifically, the lightweight encoder internally uses the Fast Fourier Transform (FFT) module and the MLP module for feature extraction and encoding.

[0039] During the decoding process, the shifted joint action After being encoded by the MLP module, it is gradually autoregressively decoded through the autoregressive decoder and the MLP module to obtain the optimal action of agent i Specifically, the first masked attention mechanism module inside the autoregressive decoder uses the shifted joint action a t encoded by the MLP module as the input of K, Q, and V of the attention mechanism. By using a mask, it is ensured that for supply chain participant j, the attention calculation is only performed between the r-th and j-th action heads where all r < j, in order to maintain the sequential update between agents; the second masked attention mechanism module, based on the first masked attention mechanism module, uses the encoded observation information as the input Q of the attention mechanism, and uses the features processed by the first masked attention mechanism as the input K and V.

[0040] Second, the training process is as Figure 3As shown, the steps are as follows:

[0041] Step 1: Initialize time t and related information of the main manufacturer agent and all supplier agents;

[0042] Step 2: The main manufacturer's intelligent agent updates its own demand information. Transportation information trn t The supplier's intelligent agent updates its own demand information. Production Information (where i = 1, 2, 3, ..., 22, representing the sequence number of the supplier's intelligent agent), in subsequent steps, the main manufacturer uses its current demand information... Inventory information Transportation information trn t As observation information

[0043] Step 3: The main manufacturer agent updates the shipping information based on the trn data. t Update its own inventory information The supplier agent, based on the updated production information Update its own inventory information (where i = 1, 2, 3, ..., 22, representing the sequence number of the supplier's intelligent agent), in subsequent steps, each supplier uses its current demand information... Inventory information Production Information As observation information

[0044] Step 4: Supply chain participants' intelligent agents use current joint observation information As input, the information is encoded using the lightweight encoder to obtain the encoded observation information.

[0045] Step 4.1: First, observe the information of each supply chain participant i = 0, 1, 2, ..., 22. Combined observation information Embedded representation:

[0046] x obv =embedding(o t )

[0047] Step 4.2: Next, analyze the joint observation information. t The embedding representation x is subjected to Fourier transform to replace the self-attention mechanism in the classic Transformer model. Fourier transforms are performed on the feature dimension and the sequence dimension in sequence, and the real part of the complex result is taken:

[0048]

[0049] in and These represent the Fourier transform operations in the feature dimension and the sequence dimension, respectively; This indicates the operation of taking the real part of a complex number.

[0050] The real part obtained after Fourier transform Multiplication is performed in the forward layer. Due to the convolutional properties of Fourier transform, multiplication in the frequency domain is equivalent to convolution in the time domain. Therefore, the essence of this series of operations is equivalent to performing a convolution operation with a large convolution kernel on the observation information of the input supply chain participants.

[0051] On the other hand, due to the duality of the Fourier transform, after superimposing multiple coding blocks, the input information is repeatedly subjected to Fourier transform and inverse transform. This is equivalent to repeatedly fusing the observation information among the intelligent agents of various supply chain participants through convolution and multiplication. This achieves the same function as the self-attention module using a simpler method, encoding the observation information into...

[0052] Step 5: Encode Observational Information The optimal actions of each agent are obtained sequentially through the autoregressive decoder. (where i = 0, 1, 2, 3, ..., 22);

[0053] Step 6: The main manufacturer's intelligent agent and the supplier's intelligent agent receive their actions based on the reward and punishment mechanism. Post-state rewards and punishments (where i = 0, 1, 2, 3, ..., 22);

[0054] Step 7: Repeat steps 2 to 6 until the preset trajectory length L is reached, and store the acquired data samples in the data cache D;

[0055] Step 8: The main manufacturer agent and the supplier agent update the network parameters of the lightweight encoder and autoregressive decoder based on the data in the data cache D described in Step 7.

[0056] Step 8.1: Lightweight Encoder Parameter Update. The training objective of the lightweight encoder is to minimize the following Bellman error:

[0057]

[0058] Where T is the sampling trajectory length; n is the number of intelligent agents involved in the supply chain; r tγ represents the team return in the supply chain at time t; φ represents the discount factor; V represents the encoder parameters; φ The value function fitted to the encoder module.

[0059] Step 8.2: Autoregressive decoder parameter update. The training objective of the autoregressive decoder is to minimize the following proximal policy optimization objective function:

[0060]

[0061] Where T is the sampling trajectory length; n is the number of intelligent agents participating in the supply chain; clip(·) represents the truncation function; ∈ represents the hyperparameter truncation coefficient that limits the update step size; Let θ be the advantage function of the supply chain at time t; θ is the strategy function parameter. The update step size is calculated according to the following formula:

[0062]

[0063] Step 9: Repeat steps 1 to 8 until the preset number of tracks T is reached.

[0064] After training is completed, the overall flowchart of the implementation of the method proposed in this invention is as follows: Figure 4 As shown.

[0065] As shown in Table 1, there are a total of 22 types of materials. In this embodiment, it is assumed that each material is provided by a single supplier, and each supplier provides only one type of material. Therefore, the environment involves a total of 23 agents, including one main manufacturer agent and 22 supplier agents. Figure 5 The results shown are from a training example demonstrating the supply chain team's feedback. These are macro-level indicators of the overall supply chain participants in the environment, reflecting how the overall supply chain cost changes during training. In terms of algorithm convergence level and speed, the algorithm proposed in this invention... Figure 5 The Ours-Transformer algorithm converges to a team reward of around -250 after approximately 300 training trajectories. The Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, which has become increasingly popular in recent years, converges after approximately 700 training trajectories, with a team reward of around -400. Single-Agent Reinforcement Learning (SRL), which treats multi-agent reinforcement learning as a single-agent problem, converges after approximately 700 training trajectories, with a team reward of around -600.

Claims

1. A supply chain inventory management method based on Transformer, characterized in that, It includes main manufacturer agents and supplier agents, and the agents can interact according to specific logic to simulate inventory changes during the operation of the supply chain; The number of main manufacturer agents is one, and the main manufacturer agent has a corresponding policy function in logic. and value function ; The number of supplier agents is N, where N is a positive integer greater than or equal to 1, and each supplier agent has a corresponding policy function. and value function ,in =1, 2, 3, ..., N, where N is the sequence number of the supplier's intelligent agent; The value function of each supply chain participant's intelligent agent ,in =0, 1, 2, 3, ..., N, fitted by a lightweight encoder; The policy functions of each supply chain participant's intelligent agent ,in =0, 1, 2, 3, ..., N, fitted by an autoregressive decoder set; The specific logic used for interactive simulation of the master manufacturer-supplier model supply chain operation, at each time... The time during the execution of the training algorithm process Using the natural week as the unit; First, build the model network, then execute the training algorithm. The model network structure is as follows: The entire network consists of two parts: a lightweight encoder that encodes the observation information of each supply chain participant's intelligent agent and an autoregressive decoder that generates the optimal actions of each supply chain participant's intelligent agent. During the encoding process, jointly observe information After passing through the multilayer perceptron module, the information is encoded into coded observation information by multiple lightweight encoders. Encoding observation information The value assessment results of each agent after passing through the MLP module The multilayer perceptron (MLP) is a type of perceptron. The lightweight encoder internally uses a fast Fourier transform (FFT) module and an MLP module for feature extraction and encoding. During decoding, shift joint action After encoding by the MLP module, the agent is obtained through autoregressive decoding and MLP module, step by step. Optimal action The first mask attention mechanism module inside the autoregressive decoder utilizes the shift joint action encoded by the MLP module. As inputs to the K, Q, and V of the attention mechanism, a mask is used to ensure that the attention of supply chain participants is considered. Only in all The The and the first Attention calculations are performed between individual action heads to maintain the order updates between agents; the second masked attention mechanism module, based on the first masked attention mechanism module, encodes the observation information. The features processed by the first mask attention mechanism are used as inputs to the attention mechanism, namely Q, K, and V.

2. The method as described in claim 1, characterized in that, The process of executing the training algorithm includes the following steps: Step 1: Initialize time And related information about the main manufacturer's intelligent agent and all supplier intelligent agents; Step 2: The main manufacturer's intelligent agent updates its own demand information. Transportation information The supplier's intelligent agent updates its own demand information. Production information ,in =1, 2, 3, ..., N, where N is the sequence number of the supplier's intelligent agent. In subsequent steps, the main manufacturer will use its current demand information. Inventory information Transportation information As observation information ; Step 3: The main manufacturer's intelligent agent updates the shipping information. Update its own inventory information The supplier's intelligent agent, based on the updated production information Update its own inventory information ,in =1, 2, 3, ..., N, where N is the sequence number of the supplier agent. In subsequent steps, each supplier uses its current demand information. Inventory information Production information As observation information ; Step 4: Supply chain participant intelligent agents use the current joint observation information As input, the information is encoded using the lightweight encoder to obtain the encoded observation information. ; Step 5: Encode Observational Information The optimal actions of each agent are obtained sequentially through the autoregressive decoder. ,in =0,1,2,3,...,N; Step 6: The main manufacturer's intelligent agent and the supplier's intelligent agent receive their actions based on the reward and punishment mechanism. Post-state rewards and punishments ,in =0,1,2,3,......,N; Step 7: Repeat steps 2 through 6 until the preset trajectory length is reached. The acquired data samples are stored in the data cache. middle; Step 8: The main manufacturer's intelligent agent and the supplier's intelligent agent cache the data as described in Step 7. The data in the dataset is used to update the network parameters of the lightweight encoder and autoregressive decoder. Step 9: Repeat steps 1 through 8 until the preset number of tracks is reached. .

3. The method as described in claim 2, characterized in that, The lightweight encoder in step 4 performs the encoding: Step 4.1: First, address the needs of each supply chain participant. Observational information Combined observation information Embedded representation: Step 4.2: Next, analyze the joint observation information. Embedded representation To replace the self-attention mechanism in the classic Transformer model, Fourier transforms are performed sequentially on both the feature dimension and the sequence dimension, and the real part of the resulting complex number is taken: in and These represent the Fourier transform operations in the feature dimension and the sequence dimension, respectively; This indicates the operation of taking the real part of a complex number; The real part obtained after Fourier transform When multiplication is performed during forward propagation, due to the convolutional properties of Fourier transform, multiplication in the frequency domain is equivalent to convolution in the time domain. Essentially, it is equivalent to performing a convolution operation with a large convolution kernel on the observation information of the input supply chain participants. Due to the duality of the Fourier transform, after superimposing multiple coding blocks, the input information is repeatedly subjected to Fourier transforms and inverse transforms. This is equivalent to repeatedly fusing the observation information among the intelligent agents of various supply chain participants through convolution and multiplication. This achieves the same function as the self-attention module using a simpler method, encoding the observation information into... .

4. The method as described in claim 3, characterized in that, The autoregressive decoder in step 5 decodes to obtain the optimal action: Encoded observation information When performing decoding to generate the optimal action, the first step is to consider the various supply chain participants. action Combination of shifting actions Perform embedded representation; The designed autoregressive encoder module has two mask attention mechanisms: the first mask self-attention mechanism ensures that for supply chain participants... Only in all The The and the first Attention is calculated between individual action heads to maintain the order updates between agents; a second masked attention mechanism is used to calculate the encoding of action heads and observation information. To reduce the additional maintenance costs for heterogeneous agents by focusing attention on each other.

5. The method as described in claim 3, characterized in that, Network parameter update in step 8: Step 8.1: Lightweight encoder parameter update; The training objective of the lightweight encoder is to minimize the following Bellman error: in The length of the sampling trajectory; The number of intelligent agents among supply chain participants; for Team rewards for continuous supply chain operations; This is the discount factor; For encoder parameters; The value function fitted to the encoder module; Step 8.2: Autoregressive decoder parameter update; the training objective of the autoregressive decoder is to minimize the following proximal policy optimization objective function: in The length of the sampling trajectory; The number of intelligent agents among supply chain participants; This represents the truncation function; The truncation coefficient represents the hyperparameter that limits the update step size; For the supply chain The advantage function at time; These are the parameters for the policy function; The update step size is calculated according to the following formula: 。

Images