A Collaborative Optimization Method for Refinery Production and Procurement Based on Multi-Agent Reinforcement Learning

Abstract

This invention relates to a collaborative optimization method for refinery production and procurement based on multi-agent reinforcement learning. The method includes the following steps: S1, constructing a production agent, a procurement agent, and a production plan model; setting the action space, state space, reward function, and environment updated based on the production plan model after the agent outputs an action for the production agent and the procurement agent; S2, setting the action interaction, state interaction, and reward interaction between the production agent and the procurement agent; S3, based on the interactions set in S2, training the production agent and the procurement agent using the asynchronous MAPPO method, acquiring actual refinery production data, and inputting the actual refinery production data into the trained production agent and the procurement agent to obtain the actual production and procurement plan. Compared with the prior art, this invention has advantages such as improving production efficiency, reducing operating costs, and maintaining balanced crude oil inventory.

Description

Technical Field

[0001] This invention relates to the field of refinery production scheduling, and in particular to a collaborative optimization method for refinery production and procurement based on multi-agent reinforcement learning. Background Technology

[0002] In the petroleum industry, refineries play a crucial role, primarily responsible for extracting and producing various petroleum products from crude oil, including gasoline, diesel, aviation fuel, and chemical feedstocks. The refining industry is a complex system involving dynamic interactions between various processes, resources, and decisions. Among these, production planning and procurement management are critical to ensuring operational efficiency and profitability. Specifically, production scheduling affects crude oil consumption ratios, constraining crude oil procurement decisions; crude oil procurement determines production rhythm, impacting production scheduling performance. However, these two levels of decision-making are often conducted independently, lacking a mechanism for establishing and coordinating their coupling, leading to decision conflicts and infeasible solutions. Furthermore, traditional optimization methods cannot adequately address dynamic market demand and supply chain complexity, resulting in production scheduling strategies that are inefficient, have high operating costs, and fail to maintain balanced crude oil inventory. Summary of the Invention

[0003] The purpose of this invention is to propose a collaborative optimization method for refinery production and procurement based on multi-agent reinforcement learning, which aims to improve production efficiency, reduce operating costs, and maintain balanced crude oil inventory.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A collaborative optimization method for oil refinery production and procurement based on multi-agent reinforcement learning, comprising the following steps:

[0006] S1. Construct production agents and procurement agents, as well as a production planning model. Set up the action space, state space, reward function, and the environment updated based on the production planning model after the agents output actions.

[0007] S2. Set up action interactions, status interactions, and reward interactions between the production agent and the procurement agent;

[0008] S3. Based on the interaction set in S2, the asynchronous MAPPO method is used to train the production agent and the procurement agent to obtain actual refinery production data. The actual refinery production data is then input into the trained production agent and the procurement agent to obtain the actual production and procurement plan.

[0009] Furthermore, the action space for the production agent and the procurement agent is as follows:

[0010]

[0011] in, To purchase the action space of the intelligent agent, q c,t This represents the quantity of crude oil c purchased at time t, where Crude represents the set of all types of crude oil available for purchase, and q represents the quantity of crude oil c purchased at time t. p,t Let P represent the production quantity of product p at time t, where P represents the set of products. The action space for producing intelligent agents.

[0012] Furthermore, the state spaces of the production agent and the procurement agent are as follows:

[0013]

[0014] in, To purchase the state space of the intelligent agent, Price crude,t D represents the price of crude oil. product,t Indicates product requirements, Inv crude,t Indicates the inventory level of raw materials. To generate the state space of intelligent agents, O prod,t Inv indicates the current status information of the product order. prod,t This indicates the product's inventory level.

[0015] Furthermore, the local reward functions for the producing agent and the purchasing agent are as follows:

[0016]

[0017] Among them, Price crude,t It is the price of crude oil at time t, Cost c,t It is the unit holding cost of inventory, Inv c,t This represents the crude oil inventory level at time t, where T represents the time interval between two purchasing actions, and profit. PA,t′ It is the profit generated by the production agent at each time step t′ within this time interval, where β represents the weighting coefficient;

[0018]

[0019] Among them, Price p,t Cost m,t Cost v,t and Cost u,t Let Sell be the selling price of product p at time t, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each unit u. p,t Buy m,t Inv v,tThese are the volumetric flow rate of product p at time t, the purchase quantity of raw material m, and the inventory quantity in storage tank v, respectively. u,t To describe the processing load of CDU.

[0020] Furthermore, the specific steps for training the production agent and procurement agent using the asynchronous MAPPO method in the S2 interaction settings are as follows:

[0021] The production agent and the procurement agent acquire the current environmental state within their respective update frequencies, output the current action, calculate the weighted embedding reward function with the other party's reward, store their respective trajectories in their respective buffers, and update their respective neural network parameters through the MAPPO method. The production agent and the procurement agent are trained independently.

[0022] Furthermore, the production agent and the procurement agent, within their respective update frequencies, obtain the current environmental state and output the specific expression for the current action as follows:

[0023]

[0024] The meaning of this expression is: at time t, the productive agent is based on state... Select actions and influence the actions of the purchasing agent through interaction with the environment. Similarly, the procurement agent is state-based. Select actions and interact with the environment to influence the actions of the production agent. k reflects different update frequencies.

[0025] Furthermore, the optimization objectives for the production agent and the procurement agent are:

[0026]

[0027] in, and These represent the optimization decisions of the purchasing agent and the production agent, respectively, where E represents the expectation and γ represents the discount rate.

[0028] Furthermore, the specific steps for updating the respective neural networks using the MAPPO method through trajectories are as follows:

[0029] The sampling trajectory is used to adjust the neural network parameters according to the reward function until the round reward is maximized.

[0030] The policy loss function of the MAPPO method is:

[0031]

[0032] In the formula, L {CLIP}(θ) For loss function, Let be the expected value at time step t, and 'clip' be the clipping strategy. The clipped value will be in the interval [1-ε, 1+ε]. To estimate the advantage function of an action relative to a state, r t (θ) represents the ratio between the old and new strategies, and ε is a hyperparameter.

[0033] Furthermore, the advantage function is:

[0034]

[0035] Among them, V w(st) Represents state s t The value of the value function, r t Let represent the reward for the action at step t, γ be the discount rate used to control the impact on long-term returns, λ be the decay factor used to control the trade-off between bias and variance, and l be the current accumulated time step. This represents the timing difference error at point t+1.

[0036] Furthermore, the training objective of the Critic network in a neural network is to minimize the squared error.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1) This invention is a multi-cycle production and procurement collaborative optimization of oil refineries based on multi-agent reinforcement learning. It designs production agents and procurement agents that make decisions at different time scales, and constructs an asynchronous collaborative framework and information interaction mechanism to enhance the system's adaptability to dynamic market conditions, such as fluctuating crude oil prices and product demand, thereby improving production efficiency, reducing operating costs, and maintaining balanced crude oil inventory.

[0039] 2) This invention models the production process and equipment operating parameters respectively. This invention designs crude oil procurement and production planning as two intelligent agents that work asynchronously and independently, which can effectively solve the problem of supply and demand imbalance and improve production efficiency.

[0040] 3) This invention introduces an advantage function to evaluate the relative advantage of various actions compared to the current strategy, helping the agent to select actions more effectively and accelerating the convergence speed of reinforcement learning algorithms. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the process of the present invention.

[0042] Figure 2 This is a simplified process diagram of the refinery in the example;

[0043] Figure 3A comparison of the learning curves of production agents and procurement agents under different methods. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0045] This invention proposes a collaborative optimization method for refinery production and procurement based on multi-agent reinforcement learning, comprising the following steps: First, an asynchronous collaborative multi-agent optimization framework is established, in which the production agent and the procurement agent make independent decisions and update their strategies at different time intervals. Second, an asynchronous interactive collaborative decision-making mechanism is designed, considering the information interaction between the state space, action space, and reward function to ensure that the two agents can coordinate and cooperate at different times. Third, the multi-agent reinforcement learning method MAPPO is combined with asynchronous settings, allowing the agents to learn the optimal strategy based on continuous interaction, while taking into account fluctuations in uncertain factors such as crude oil prices, processing capacity, and inventory levels. Compared with existing technologies, this invention has advantages such as high solution efficiency and strong robustness. The flowchart of this invention is as follows: Figure 1 As shown. The steps of the present invention include:

[0046] S1. Construct production agents, procurement agents, and production planning models; set up the action space, state space, local reward function, and environment updated based on the production planning model after the agents output actions.

[0047] S2. Set up action interactions, status interactions, and reward interactions between the production agent and the procurement agent;

[0048] S3. Based on the interaction set in S2, the asynchronous MAPPO method is used to train the production agent and the procurement agent to obtain actual refinery production data. The actual refinery production data is then input into the trained production agent and the procurement agent to obtain the actual production and procurement plan.

[0049] In S1, an asynchronous and collaborative multi-agent optimization framework is built, in which the production agent and the procurement agent make independent decisions and update their strategies at different time intervals, thereby improving the system's flexibility and responsiveness to real-time changes.

[0050] In the asynchronous collaborative multi-agent optimization framework, the production environment includes uncertainties such as crude oil prices and product demand fluctuations, and a production planning model updates the states of the production and procurement agents. Two-way communication is established between the production and procurement agents, jointly influencing the production state, as represented by:

[0051]

[0052] In the formula, f is the state transition function. It is the action of purchasing intelligent agents. These are the actions of the production agent. The production agent and the procurement agent share information through states and actions, thereby achieving dynamic adjustments.

[0053] The production planning model includes constraints on product demand, product properties, equipment load, equipment operation, and storage tank processing load.

[0054] In the production environment, uncertainties such as crude oil prices and product demand are taken into account. A deterministic production planning model is constructed to respond to the action decisions of production agents and procurement agents and update their status information. The production planning model includes constraints on raw material supply, product demand, equipment operation, equipment processing capacity, product properties, and storage tank processing load.

[0055] like Figure 2 As shown, the refinery production operation involves three parts: crude oil blending and processing, secondary processing, and product blending. Crude oils of different properties and prices are blended and sent to the crude oil distillation unit (CDU), where they are separated into different fractions such as light hydrocarbons, gasoline, kerosene, diesel, and residues at different temperatures. These fractions flow to the secondary processing units for further processing. The main secondary processing units include catalytic cracking (FCC), continuous reforming (CRU), delayed coking unit (DCU), solvent deasphalting, hydrocracking (HCU), residue hydrotreating unit (RHU), and hydrotreating units.

[0056] This embodiment is for Figure 2 The process involves building a production planning model.

[0057] The formula describing product demand is:

[0058]

[0059] In the formula, and Let Sell be the minimum and maximum market demand for product p during the planning period t. p,t Let p be the sales volume of product p during the planning period t.

[0060] For atmospheric and vacuum distillation units (CDUs), due to the numerous nonlinearities and complexities involved in CDU modeling, a fixed yield model is used to describe the CDU modeling process. The processing flow rate of each type of crude oil at each time point is also considered. m,t The formula is:

[0061]

[0062] In the formula, CO is the set of crude oil m, and MCU m,u,t This represents the amount of crude oil m flowing into unit u (type CDU) at time t. U (U = CDU, PU, ​​SPL, MIX) represents the various units of the refinery, including atmospheric and vacuum distillation units, secondary processing units, separators, and mixers. Secondary processing units include catalytic cracking, continuous reforming, delayed coking, solvent deasphalting, hydrocracking, and hydrotreating units.

[0063] Describe the processing load flow of the CDU u,t The formula is:

[0064]

[0065] The formula describing the machining load constraint range of the CDU is:

[0066]

[0067] In the formula, and These are the minimum and maximum values ​​of the CDU processing load.

[0068] The content of different distillate oils in the CDU was calculated by multiplying the processing volume of different crude oils by a fixed yield using Yield. s,u,t The formula is:

[0069]

[0070] In the formula, S (s∈{1,2,…S}) is the set of all flows, and UO=(u,s) represents all outlet flows of the device.

[0071] for Figure 2 The devices mentioned can be classified into two categories: single-mode processing devices and multi-mode processing devices. The following settings are made for different types of devices:

[0072] For a single-mode processing unit, the formulas describing the material balance, processing load, and product output of the single-mode processing unit u are as follows:

[0073]

[0074]

[0075] For a multi-mode processing unit, each unit can operate in production mode M (m∈{1,2,…M}) according to production requirements. The product yield of the unit varies under different operating modes. Based on material balance, the formulas describing the total flow rate of the multi-mode processing unit u under mode op (different schemes with different product and raw material ratios) and the total load and load constraints of unit u are:

[0076]

[0077] In the formula, UI = (u, s) represents all inlet flows of the device.

[0078] The formula describing the process of splitting and merging traffic to ensure that quality balance is maintained is as follows:

[0079]

[0080] The quality of blended products must meet specific standards and specifications, including requirements for the content of various components, physical properties (such as density and viscosity), and chemical properties (such as sulfur content and aromatic hydrocarbon content).

[0081] The formula describing the yield of product p, calculated from each harmonic component c∈BC, is as follows:

[0082]

[0083] The formula describing the volumetric properties of component c in the blended product p is:

[0084] QV c,p,q,t =Flow c,p,t ·Q c,q,t c∈BC, p∈P, q∈Q, t∈T

[0085] In the formula, Q c,q,t Let q represent the properties of the harmonic component c in period t. Q (q∈1,2,…Q) is a set of product attributes where the elements represent sulfur content, density, octane number, cetane number, olefin content, and aromatics.

[0086] The formula describing product property constraints is:

[0087]

[0088] In the formula, and Let p be the minimum and maximum values ​​of the product property.

[0089] Multi-cycle production planning connects raw material inventory and intermediate product inventory. The formula describing the coupling relationship between inventory levels in each cycle is as follows:

[0090]

[0091] In the formula, and These represent the flow rates flowing into and out of storage tank v during period t, respectively.

[0092] The formula describing the processing load constraint of storage tank v is:

[0093]

[0094] In the formula, and Let v be the minimum and maximum values ​​of the processing load of the storage tank.

[0095] The state space of the purchasing agent can be represented as:

[0096]

[0097] In the formula, Price crude,t D represents the price of crude oil. product,t Indicates product requirements, Inv crude,t This indicates the inventory level of raw materials.

[0098] The action space of the purchasing agent can be represented as:

[0099]

[0100] In the formula, q c,t This represents the amount of crude oil c purchased at time t, where Crude represents the set of all types of crude oil available for purchase.

[0101] The reward function for the purchasing agent aims to balance cost-effectiveness and product demand satisfaction, and consists of two main components: 1) crude oil procurement and inventory costs; and 2) the production profit generated by the production agent between two procurement actions. The reward function can be expressed as follows:

[0102]

[0103] In the formula, Price crude,t It is the price of crude oil at time t, Cost c,t It is the unit holding cost of inventory, Inv c,t This represents the inventory level of crude oil at time t. T represents the time interval between two purchasing actions, and profit... PA,t′ It is the profit generated by the production agent at each time step t′ within this time interval.

[0104] The state space of a production agent can be represented as:

[0105]

[0106] In the formula, O prod,t Inv indicates the current status information of the product order. prod,t This indicates the product's inventory level.

[0107] The action space of a production agent can be represented as:

[0108]

[0109] In the formula, q p,t This represents the production volume of product p at time t.

[0110] The reward function for the production agent is the difference between product sales revenue and the cumulative costs related to raw material procurement, inventory holding, and operating costs. The calculation expression is:

[0111]

[0112]

[0113] In the formula, Price p,t Cost m,t Cost v,t and Cost u,t Let Sell be the selling price of product p at time t, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each unit u. p,t Buy m,t Inv v,t These are the volumetric flow rate of product p at time t, the purchase quantity of raw material m, and the inventory quantity of storage tank v.

[0114] In S2, to reflect the collaborative decision-making relationship between the two agents, an asynchronous interactive collaborative decision-making mechanism is designed, which considers the information interaction between the state space, action space and reward function to ensure that the two agents can coordinate and cooperate at different times to maximize the overall benefits of the refinery.

[0115] The production agent and the procurement agent adopt a distributed design, with independent observation space and action space, and use an asynchronous interactive collaborative decision-making mechanism for information exchange.

[0116] The production and procurement agents employ asynchronous decision-making, information sharing mechanisms, and asynchronous buffering insertion strategies to achieve independent reinforcement learning training and updates. Furthermore, the asynchronous interaction protocol defines the frequency of state exchange and updates between the production and procurement agents, expressed as:

[0117]

[0118] In the formula, k is an interval that reflects different update frequencies.

[0119] The asynchronous setup is modeled as a distributed partially observable Markov decision process (Dec-POMDP), where each agent optimizes its policy based on local observations. The optimization objective for each agent is expressed as follows:

[0120]

[0121] In the formula, the policy of each agent and They are all trained based on local observations, with the aim of maximizing expected returns while asynchronously adapting to the ever-changing environment.

[0122] S2 specifically includes:

[0123] Within the asynchronous framework, production and procurement agents employ asynchronous decision-making, information sharing mechanisms, and asynchronous buffering insertion strategies to achieve independent reinforcement learning training and updates. The asynchronous coordination mechanism improves responsiveness and adaptability to dynamic production environments such as fluctuations in crude oil prices and changes in product demand. Furthermore, the asynchronous interaction protocol defines the frequency of state exchange and updates between the production and procurement agents. The production agent provides the procurement agent with its predicted production demand and current inventory levels. In return, the procurement agent shares its expected procurement quantity and expected supply cost with the production agent. Each agent independently updates its policy and state at its own frequency. Under the premise of independent policy updates, a collaborative reward signal is designed, with each agent softly embedding the reward of the other, promoting global cooperation without significantly interfering with the independent updates of individual policies. Both the production and procurement agents receive their own local reward signals. and And the weighted embedding of the other party's reward (for the purchasing agent as) For production intelligent agents Through a soft-embedding strategy, each agent can take into account the goals of other agents without being overly constrained by them, thus striking a balance between global cooperation and the autonomy required for efficient asynchronous policy updates. These reward signals are then used to train the two agents using the Async-MAPPO algorithm, further enhancing their ability to adapt to dynamic conditions and maximizing the efficiency of the overall system. The asynchronous setup is modeled as a distributed partially observable Markov decision process (Dec-POMDP), where each agent optimizes its policy based on local observations. The optimization objective for each agent is expressed as follows:

[0124]

[0125] In the formula, the policy of each agent and They are all trained based on local observations, with the aim of maximizing expected returns while asynchronously adapting to the ever-changing environment.

[0126] In S3, the multi-agent reinforcement learning method MAPPO is combined with asynchronous settings, allowing agents to learn optimal policies on the basis of continuous interaction, while taking into account the fluctuations of uncertain factors such as crude oil prices, processing capacity and inventory levels.

[0127] The MAPPO algorithm's training process for learning production and maintenance decision-making strategies includes: sampling production decision trajectories, adjusting network parameters according to the reward function until the round reward is maximized to generate a strategy that maximizes expected returns, and solving for the optimal production plan and crude oil procurement scheme. For a complete event τ, the total reward obtained throughout the process is... The weighted sum of rewards obtained from each sampling event τ for a given policy with parameter θ is expressed as:

[0128]

[0129] The goal is to obtain the total reward by adjusting the policy parameter θ. The larger the better. The most commonly used gradient estimator. The expression is:

[0130]

[0131] Because the Monte Carlo sampling method has a large variance, the training process is difficult to converge. Therefore, an advantage function is introduced. To help agents choose actions more effectively and accelerate the convergence speed of reinforcement learning algorithms, generalized advantage estimation (GAE) is commonly used to calculate the advantage function. This involves introducing the advantage function. The formula used to evaluate the relative advantages of various actions compared to the current strategy is as follows:

[0132]

[0133] In the formula, V w(st) Represents state s t The value of the value function, r t Let represent the reward for the action at step t, γ be the discount rate used to control the impact on long-term returns, λ be the decay factor used to control the trade-off between bias and variance, and l be the current accumulated time step. This represents the timing difference error at point t+1.

[0134] To prevent policy collapse or instability in the learning process, PPO enhances the algorithm's stability by limiting the magnitude of policy updates. The core policy loss function is defined as follows:

[0135]

[0136] In the formula, L {CLIP}(θ) For loss function, Let be the expected value at time step t, and 'clip' be the clipping strategy. The clipped value will be in the interval [1-ε, 1+ε]. To estimate action a t Relative to state s t Advantages, r t (θ) represents the ratio between the old and new strategies, and L is a hyperparameter used to ensure that the difference between the old and new strategies is not too large when multiple strategy updates are performed using the same batch of data.

[0137] Furthermore, the training objective of the Critic network is to minimize the squared error (MSE), calculated using the following formula:

[0138] L(θ)=E[(G t -V(s t ;θ)) 2 ]

[0139] In the formula, θ represents the parameters of the Critic network, V(s) t ;θ) represents the Critic network's response to state s t The estimated value of G t Indicates from state s t The actual returns that begin to be observed.

[0140] To verify the performance of the present invention, the following experiment was designed.

[0141] The experiment compared the Asynchronous Cooperative Multi-Agent Framework (ACO) with the Independent Distributed Multi-Agent Framework (DNCO) in a real-world refinery case. The refinery case included six candidate crude oils, 14 final products, and 28 processing units. Additionally, there were five blending units: a diesel blending unit, a G92 blending unit, a G95 blending unit, a jet fuel blending unit, and a naphtha blending unit. The production agent's time step was one month, while the crude oil procurement agent's time step was three months. This setup reflects the refinery's quarterly optimization of its crude oil procurement plan. During each procurement cycle, crude oil prices fluctuated based on market conditions, while product demand was adjusted monthly. The results showed that in DNCO, the procurement agent reduced crude oil purchases to lower costs, leading to production interruptions. In ACO, the procurement and production agents effectively addressed the supply-demand imbalance through an asynchronous cooperative mechanism. Furthermore, compared to DNCO, ACO resulted in higher production profits, lower crude oil costs, and the storage of more crude oil during periods of low prices. Comparison of learning curves for production agents and procurement agents under different methods, for example Figure 3 As shown.

[0142] Compared with the prior art, the present invention has the following beneficial effects:

[0143] 1) This invention is a multi-cycle production and procurement collaborative optimization of oil refineries based on multi-agent reinforcement learning. It designs production agents and procurement agents that make decisions at different time scales, and constructs an asynchronous collaborative framework and information interaction mechanism to enhance the system's adaptability to dynamic market conditions, such as fluctuating crude oil prices and product demand, thereby improving production efficiency, reducing operating costs, and maintaining balanced crude oil inventory.

[0144] 2) This invention models the production process and equipment operating parameters respectively. This invention designs crude oil procurement and production planning as two intelligent agents that work asynchronously and independently, which can effectively solve the problem of supply and demand imbalance and improve production efficiency.

[0145] 3) This invention introduces an advantage function to evaluate the relative advantage of various actions compared to the current strategy, helping the agent to select actions more effectively and accelerating the convergence speed of reinforcement learning algorithms.

[0146] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A collaborative optimization method for refinery production and procurement based on multi-agent reinforcement learning, characterized in that, The method includes the following steps: S1. Construct production agents and procurement agents, as well as a production planning model. Set up the action space, state space, reward function, and the environment updated based on the production planning model after the agents output actions. S2. Set up action interactions, status interactions, and reward interactions between the production agent and the procurement agent; S3. Based on the interaction set in S2, the asynchronous MAPPO method is used to train the production agent and the procurement agent to obtain the actual refinery production data. The actual refinery production data is then input into the trained production agent and the procurement agent to obtain the actual production and procurement plan. The action space for the production agent and the procurement agent is: in, To provide the action space for the procurement of intelligent agents, Indicates time Crude oil Purchase volume, This represents the set of all types of crude oil available for purchase. Indicates time product Production volume. Describe a collection of products. The action space for producing intelligent agents; The state spaces of the production agent and the procurement agent are as follows: in, The state space for purchasing intelligent agents. Indicates crude oil price, Indicate product demand, This indicates the inventory level of raw materials. The state space for producing intelligent agents. This indicates the current status information of the product order. Indicates the product's inventory level; The local reward functions for the production agent and the procurement agent are: in, It is time crude oil prices, It is the unit holding cost of inventory. Indicates crude oil in time Inventory levels, Indicates the time interval between two procurement actions. It is the time step of the production agent within this time interval. The profits generated Indicates the weighting coefficient; in, , , and They are time Product of the Times Sales price, raw materials Purchase price, storage tank Inventory costs and per unit Operating costs, , , They are time Product of the Times Volumetric flow rate, raw materials Purchase volume and storage tanks Inventory levels To describe the processing load of the atmospheric and vacuum distillation unit (CDU); The specific steps for training the production agent and procurement agent using the asynchronous MAPPO method based on the S2-configured interaction are as follows: The production agent and the procurement agent acquire the current environmental state within their respective update frequencies, output the current action, calculate the weighted embedding reward function with the other party's reward, store their respective trajectories in their respective buffers, and update their respective neural network parameters through the MAPPO method. The production agent and the procurement agent are trained independently.

2. The refinery production and procurement collaborative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The production agent and the procurement agent, within their respective update frequencies, obtain the current environmental state and output the specific expression for their current action: The meaning of this expression is: In time Production agents are state-based Select actions and influence the actions of the purchasing agent through interaction with the environment. Similarly, the procurement agent is state-based. Select actions and interact with the environment to influence the actions of the production agent. , This reflects different update frequencies.

3. The refinery production and procurement collaborative optimization method based on multi-agent reinforcement learning according to claim 2, characterized in that, The optimization objectives for the production agent and the procurement agent are: in, and These are the optimization decisions for the procurement agent and the production agent, respectively. Expressing expectations, This indicates the discount rate.

4. The refinery production and procurement collaborative optimization method based on multi-agent reinforcement learning according to claim 3, characterized in that, The specific steps for updating the respective neural networks using the MAPPO method through trajectories are as follows: The sampling trajectory is used to adjust the neural network parameters according to the reward function until the round reward is maximized. The policy loss function of the MAPPO method is: In the formula, For loss function, Let the expected value be at time step t. For the clipping strategy, the clipped value is in [ ] interval, To estimate the advantage function of an action relative to a state, This represents the ratio between the old and new strategies. It's a hyperparameter.

5. The refinery production and procurement collaborative optimization method based on multi-agent reinforcement learning according to claim 4, characterized in that, The advantage function is: in, Representing state The value of the value function, Indicates the first Rewards for step actions, The discount rate is used to control the impact on long-term returns. This is a decay factor used to control the trade-off between bias and variance. This is the current accumulated number of time steps. for Timing difference error at the location.

6. The refinery production and procurement collaborative optimization method based on multi-agent reinforcement learning according to claim 5, characterized in that, The training objective of the Critic network in a neural network is to minimize the squared error.

Images