Oil refinery production and purchase collaborative optimization method based on multi-agent reinforcement learning

Abstract

The invention relates to an oil refinery production and purchase collaborative optimization method based on multi-agent reinforcement learning, and the method comprises the following steps: S1, constructing a production agent, a purchase agent and a production plan model, setting an action space, a state space and a reward function of a production agent and a purchasing agent and an environment updated based on a production plan model after the agents output actions; s2, setting action interaction, state interaction and reward interaction between the production agent and the purchasing agent; and S3, training a production agent and a purchasing agent by adopting an asynchronous MAPPO method based on the interaction set in the S2, obtaining actual oil refinery production data, and inputting the actual oil refinery production data into the trained production agent and purchasing agent to obtain an actual production and purchasing plan. Compared with the prior art, the method has the advantages of improving the production efficiency, reducing the operation cost, keeping the stock balance of crude oil and the like.

Description

Technical Field

[0001] The present invention relates to the field of refinery production scheduling, and in particular to a refinery production procurement collaborative optimization method based on multi-agent reinforcement learning. Background Art

[0002] In the oil industry, refineries play a pivotal role, mainly responsible for extracting and producing various petroleum products from crude oil, including gasoline, diesel, aviation fuel, and chemical raw materials. The oil refining industry is a complex system involving dynamic interactions between various processes, resources, and decisions. Among them, production planning and procurement management play a vital role in ensuring operational efficiency and profitability. Specifically, the production scheduling plan affects the crude oil consumption ratio and restricts the crude oil procurement decision; crude oil procurement determines the production rhythm and affects the production scheduling performance. However, these two levels of decisions are often made independently, lacking the construction and coordination mechanism of the coupling relationship, resulting in decision conflicts and infeasible solutions. In addition, traditional optimization methods cannot fully cope with the dynamic market demand and the complexity of the supply chain, resulting in the analyzed production scheduling strategy not meeting the production efficiency, high operating costs, and failure to maintain a balanced crude oil inventory. Summary of the invention

[0003] The purpose of the present invention is to propose a collaborative optimization method for refinery production and procurement based on multi-agent reinforcement learning in order to improve production efficiency, reduce operating costs and maintain crude oil inventory balance.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A method for collaborative optimization of production and procurement of an oil refinery based on multi-agent reinforcement learning, the method comprising the following steps:

[0006] S1. Construct production agents, procurement agents and production planning models, set the action space, state space, reward function of the production agents and procurement agents, and the environment based on the production planning model update after the agents output actions;

[0007] S2, setting the action interaction, state interaction and reward interaction between the production agent and the procurement agent;

[0008] S3. Based on the interaction set in S2, the asynchronous MAPPO method is used to train the production agent and the procurement agent to obtain the actual refinery production data. The actual refinery production data is input into the trained production agent and the procurement agent to obtain the actual production procurement plan.

[0009] Furthermore, the action space of the production agent and the procurement agent is:

[0010]

[0011] in, is the action space of the purchasing agent, q c,t represents the purchase volume of crude oil c at time t, Crude represents the set of all crude oil types available for purchase, and q p,t represents the production of product p at time t, P represents the product set, The action space of the production agent.

[0012] Furthermore, the state space of the production agent and the procurement agent is:

[0013]

[0014] in, is the state space of the purchasing agent, Price crude,t represents the crude oil price, D product,t Indicates product demand, Inv crude,t Indicates the inventory level of raw materials, is the state space of the production agent, O prod,t Indicates the status information of the current product order, Inv prod,t Indicates the stock level of a product.

[0015] Furthermore, the local reward functions of the production agent and the procurement agent are:

[0016]

[0017] Among them, Price crude,t is the crude oil price at time t, Cost c,t is the inventory unit holding cost, Inv c,t represents the inventory level of crude oil at time t, T represents the time interval between two purchase actions, and profit PA,t′ is the profit generated by the production agent at each time step t′ in this time interval, and β represents the weight coefficient;

[0018]

[0019] Among them, Price p,t Cost m,t Cost v,t and Cost u,t are the selling price of product p, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each device u at time t. p,t 、Buy m,t 、Inv v,tare the volume flow of product p, the purchase volume of raw material m, and the inventory volume of storage tank v at time t. u,t To describe the processing load of CDU.

[0020] Furthermore, the interaction set by S2 uses the asynchronous MAPPO method to train the production agent and the procurement agent in the following specific steps:

[0021] The production agent and the procurement agent obtain the current state of the environment within their respective update frequencies, output the current action, and respectively calculate the weighted embedded reward function with the other party's reward. The respective trajectories are stored in their corresponding buffers, and the MAPPO method is used to update their respective neural network parameters through the trajectory. The production agent and the procurement agent are trained independently.

[0022] Furthermore, the production agent and the procurement agent obtain the current state of the environment within their respective update frequencies and output the specific expression of the current action as follows:

[0023]

[0024] The meaning of this expression is: at time t, the production agent is based on the state Select actions and influence the actions of the purchasing agent by interacting with the environment Similarly, the purchasing agent is based on the state Select actions and interact with the environment to influence the actions of the production agent k reflects different update frequencies.

[0025] Furthermore, the optimization objectives of the production agent and the procurement agent are:

[0026]

[0027] in, and They are the optimization decisions of the purchasing agent and the production agent respectively, E represents the expectation, and γ represents the discount rate.

[0028] Furthermore, the specific steps of using the MAPPO method to update the respective neural networks through trajectories are:

[0029] Sample trajectories and adjust the neural network parameters according to the reward function until the round reward is maximized;

[0030] The strategy loss function of the MAPPO method is:

[0031]

[0032] Where, L {CLIP}(θ) is the loss function, is the expected value at time step t, clip is the clipping strategy, and the clipped value is in the interval [1-ε,1+ε], To estimate the advantage function of the action relative to the state, r t (θ) represents the ratio between the new and old strategies, and ε is a hyperparameter.

[0033] Furthermore, the advantage function is:

[0034]

[0035] Among them, V w(st) Indicates state s t The value of the value function, r t represents the reward of the t-th action, γ is the discount rate, which is used to control the impact of long-term benefits, λ is the decay factor, which is used to control the trade-off between deviation and variance, l is the current accumulated time step, is the timing differential error at t+1.

[0036] Furthermore, the training objective of the Critic network in the neural network is to minimize the squared error.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1) The present invention is a multi-period production and procurement collaborative optimization of an oil refinery based on multi-agent reinforcement learning. Production agents and procurement agents that make decisions on different time scales are designed, an asynchronous collaborative framework and information interaction mechanism are constructed, and the system's adaptability to dynamic market conditions, such as fluctuating crude oil prices and product demand, is enhanced, thereby improving production efficiency, reducing operating costs, and maintaining a balanced crude oil inventory.

[0039] 2) The present invention models the production process, device operating parameters, etc. respectively. The present invention designs crude oil procurement and production planning as two intelligent entities, which collaborate asynchronously and make independent decisions, which can well solve the problem of supply and demand imbalance and improve production efficiency.

[0040] 3) The present invention introduces an advantage function to evaluate the relative advantages of various actions compared with the current strategy, helping the intelligent agent to choose actions more effectively and speeding up the convergence of the reinforcement learning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the process of the present invention.

[0042] Figure 2 This is a simplified process diagram of an oil refinery in the embodiment;

[0043] Figure 3Comparison of learning curves of production agents and procurement agents under different methods. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] The present invention proposes a method for collaborative optimization of refinery production and procurement based on multi-agent reinforcement learning, which includes the following steps: building an asynchronous collaborative multi-agent optimization framework, in which the production agent and the procurement agent make independent decisions and update strategies at different time intervals; designing an asynchronous interactive collaborative decision-making mechanism, considering the information interaction between the state space, action space and reward function to ensure that the two agents can coordinate and cooperate at different times; combining the multi-agent reinforcement learning method MAPPO with an asynchronous setting, allowing the agent to learn the optimal strategy based on continuous interaction, while taking into account the fluctuations of uncertain factors such as crude oil prices, processing capacity and inventory levels. Compared with the prior art, the present invention has the advantages of high solution efficiency and strong robustness. The flow chart of the present invention is as follows Figure 1 The steps of the present invention include:

[0046] S1. Construct production agents, procurement agents and production planning models, set the action space, state space, local reward function of the production agents and procurement agents, and the environment based on the production planning model update after the agents output actions;

[0047] S2, setting the action interaction, state interaction and reward interaction between the production agent and the procurement agent;

[0048] S3. Based on the interaction set in S2, the asynchronous MAPPO method is used to train the production agent and the procurement agent to obtain the actual refinery production data. The actual refinery production data is input into the trained production agent and the procurement agent to obtain the actual production procurement plan.

[0049] In S1, an asynchronous collaborative multi-agent optimization framework was built, in which the production agent and the procurement agent made independent decisions and updated their strategies at different time intervals, thereby improving the system's flexibility and responsiveness to real-time changes.

[0050] In the asynchronous collaborative multi-agent optimization framework, the production environment includes fluctuations in uncertain factors such as crude oil prices and product demand, as well as the production planning model to update the status of the production agent and the procurement agent, and establish a two-way communication between the production agent and the procurement agent to jointly influence the production status, which is expressed as:

[0051]

[0052] Where f is the state transition function, is the action of the purchasing agent, It is the action of the production agent. The production agent and the procurement agent share information through status and action, thus achieving dynamic adjustment.

[0053] The production planning model includes product demand constraints, product property constraints, device load constraints, device operation constraints and tank processing load constraints.

[0054] In the production environment, the fluctuations of uncertain factors such as crude oil prices and product demand are taken into account, and a deterministic production planning model is constructed to respond to the action decisions of the production agent and the procurement agent and update the status information. The production planning model includes raw material supply constraints, product demand constraints, device operation constraints, device processing capacity constraints, product property constraints and tank processing load constraints.

[0055] like Figure 2 As shown in the figure, the production operation process of a refinery involves three parts, namely crude oil blending and processing, secondary processing and product blending. Crude oils of different properties and prices are mixed and sent to the crude distillation unit (CDU) to be separated into different fractions such as light hydrocarbon gas, gasoline, kerosene, diesel and residue at different temperatures. These fractions flow to the secondary processing unit for further processing. The main secondary processing units include catalytic cracking (FCC), continuous reforming (CRU), delayed coking (DCU), solvent deasphalting, hydrocracking (HCU), residue hydrotreating unit (RHU) and hydroprocessing.

[0056] This embodiment is aimed at Figure 2 The production planning model is constructed through the process.

[0057] The formula describing the demand for a product is:

[0058]

[0059] In the formula, and is the minimum and maximum market demand of product p in planning period t, Sell p,t is the sales volume of product p in planning period t.

[0060] For the atmospheric and vacuum unit, due to the many nonlinearities and complexities in the atmospheric and vacuum unit (CDU) modeling process, a fixed yield model is used to describe the CDU modeling process. m,t The formula is:

[0061]

[0062] Where CO is the set of crude oil m, MCU m,u,t It represents the amount of crude oil m flowing into unit u of type CDU at time t. U (U = CDU, PU, ​​SPL, MIX) represents the various units of the refinery, including atmospheric and vacuum crude distillation units, secondary processing units, separators and mixers. Secondary processing units include catalytic cracking, continuous reforming, delayed coking, solvent deasphalting, hydrocracking and hydrotreating units.

[0063] Describe the CDU processing load flow u,t The formula is:

[0064]

[0065] The formula describing the processing load constraint range of CDU is:

[0066]

[0067] In the formula, and are the minimum and maximum values ​​of CDU processing load.

[0068] The content of different distillates in CDU is calculated by multiplying the processing volume of different crude oils by the fixed yield. s,u,t , the formula is:

[0069]

[0070] Where S(s∈{1,2,…S}) is the set of all flows, and UO=(u,s) represents all outlet flows of the device.

[0071] for Figure 2 The devices mentioned in the above can be classified into two categories: single-mode processing devices and multi-mode processing devices. The following settings are made for different types of devices:

[0072] For a single-mode process unit, the formula describing the material balance, process load and unit product output of a single-mode process unit u is:

[0073]

[0074]

[0075] For multi-mode processing devices, each device can be operated in M ​​(m∈{1,2,…M}) production modes according to production requirements, and the product yields of the devices under different operation modes are different. According to material balance, the total flow of multi-mode processing device u under mode op (different schemes, different ratios of products and raw materials), the total load of device u and the load constraint formula are:

[0076]

[0077] Where UI = (u, s) represents all inlet flows of the device.

[0078] The formula that describes the process of flow division and confluence to ensure that the mass balance is maintained is:

[0079]

[0080] The quality of blended products must meet specific standards and specifications, including requirements on the content of various ingredients, physical properties (such as density, viscosity), chemical properties (such as sulfur content, aromatics content), etc.

[0081] The formula describing the output of product p calculated by each blending component c∈BC is:

[0082]

[0083] The formula describing the volumetric properties of component c of the blend product p is:

[0084] QV c,p,q,t =Flow c,p,t Q c,q,t c∈BC,p∈P,q∈Q,t∈T

[0085] In the formula, Q c,q,t represents the properties of the blending component c in period t. Q(q∈1,2,…Q) is a set of product attributes, whose elements represent sulfur content, density, research octane number, cetane number, olefin content, and aromatics.

[0086] The formula describing the product property constraints is:

[0087]

[0088] In the formula, and are the minimum and maximum values ​​of the product p property.

[0089] The multi-period production plan is connected through the raw material inventory and the intermediate product inventory. The formula describing the coupling relationship of each period inventory is:

[0090]

[0091] In the formula, and They represent the flow rate flowing into tank v and the flow rate flowing out of tank v in period t respectively.

[0092] The formula describing the process load constraint for tank v is:

[0093]

[0094] In the formula, and are the minimum and maximum values ​​of the processing load of tank v.

[0095] The state space of the purchasing agent can be expressed as:

[0096]

[0097] In the formula, Price crude,t represents the crude oil price, D product,t Indicates product demand, Inv crude,t Indicates the stock level of raw materials.

[0098] The action space of the purchasing agent can be expressed as:

[0099]

[0100] In the formula, q c,t represents the purchase quantity of crude oil c at time t, and Crude represents the set of all crude oil types available for purchase.

[0101] The reward function of the purchasing agent aims to balance cost-effectiveness and product demand satisfaction, and consists of two main parts: 1) crude oil procurement and inventory costs; 2) production profits generated by the production agent between two procurement actions. The reward function can be expressed as

[0102]

[0103] In the formula, Price crude,t is the crude oil price at time t, Cost c,t is the inventory unit holding cost, Inv c,t represents the inventory level of crude oil at time t. T represents the time interval between two purchase actions, profit PA,t′ is the profit generated by the production agent at each time step t′ during this time interval.

[0104] The state space of the production agent can be expressed as:

[0105]

[0106] In the formula, O prod,t Indicates the status information of the current product order, Inv prod,t Indicates the stock level of a product.

[0107] The action space of the production agent can be expressed as:

[0108]

[0109] In the formula, q p,t Represents the production volume of product p at time t.

[0110] The reward function of the production agent is the difference between the product sales revenue and the accumulated costs related to raw material procurement, inventory holding and operating costs, calculated as:

[0111]

[0112]

[0113] In the formula, Price p,t Cost m,t Cost v,t and Cost u,t are the selling price of product p, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each device u at time t. p,t 、Buy m,t 、Inv v,t They are the volume flow of product p, the purchase quantity of raw material m and the inventory in tank v at time t.

[0114] In S2, in order to reflect the collaborative decision-making relationship between the two, an asynchronous interactive collaborative decision-making mechanism is designed, considering the information interaction between the state space, action space and reward function, to ensure that the two intelligent agents can coordinate and cooperate at different times to maximize the overall benefits of the refinery.

[0115] The production agent and procurement agent adopt a distributed design, have independent observation space and action space, and use asynchronous interactive collaborative decision-making mechanism to exchange information.

[0116] The production agent and the procurement agent adopt asynchronous decision-making, information sharing mechanism and asynchronous buffer insertion strategy to achieve independent reinforcement learning training and updating. In addition, the asynchronous interaction protocol defines the state exchange and update frequency between the production agent and the procurement agent, which is expressed as:

[0117]

[0118] In the formula, k is an interval, reflecting different update frequencies.

[0119] The asynchronous setting is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), where each agent optimizes its policy based on local observations. The optimization objective of each agent is expressed as follows:

[0120]

[0121] In the formula, the strategy of each agent is and They are all trained based on local observations to maximize expected rewards while asynchronously adapting to a changing environment.

[0122] S2 specifically includes:

[0123] Under the asynchronous framework, the production agent and the procurement agent adopt asynchronous decision-making, information sharing mechanism and asynchronous buffer insertion strategy to achieve independent reinforcement learning training and updating. The asynchronous coordination mechanism can improve the response speed and adaptability to dynamic production environments such as crude oil price fluctuations and changes in product demand. In addition, the asynchronous interaction protocol defines the state exchange and update frequency between the production agent and the procurement agent. The production agent provides the procurement agent with its predicted production demand and current inventory level. In return, the procurement agent shares its expected purchase quantity and expected supply cost with the production agent. Each agent independently updates its strategy and state at its own frequency. Under the premise of independent update strategy, a collaborative reward signal is designed, and each agent introduces a soft embedding of the other party's reward, so that it can promote global collaboration without significantly interfering with the independent update of individual strategies. Both the production agent and the procurement agent will receive their own local reward signal and and the weighted embedding of the other party’s reward (for the purchasing agent For production agents ). By soft-embedded policies, each agent is able to take into account the goals of other agents without being overly constrained by these goals, thereby striking a balance between global cooperation and the autonomy required for effective asynchronous policy updates. These reward signals are then used to train both agents via the Async-MAPPO algorithm, further improving their ability to adapt to dynamic conditions and maximizing the efficiency of the entire system. The asynchronous setting is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), where each agent optimizes its policy based on local observations. The optimization objective of each agent is expressed as follows:

[0124]

[0125] In the formula, the strategy of each agent is and They are all trained based on local observations to maximize expected rewards while asynchronously adapting to a changing environment.

[0126] In S3, the multi-agent reinforcement learning method MAPPO is combined with an asynchronous setting, allowing agents to learn optimal strategies based on continuous interactions while taking into account fluctuations in uncertain factors such as crude oil prices, processing capacity, and inventory levels.

[0127] The process of MAPPO algorithm training and learning production and maintenance decision strategies includes: sampling production decision trajectories, adjusting network parameters according to the reward function, until the round reward is maximized, to generate a strategy that maximizes the expected return, and solving the optimal production plan and crude oil procurement plan. For a complete event τ, the total reward obtained in the whole process is For a given policy with parameters θ, the weighted sum of rewards obtained from each sampled event τ is expressed as:

[0128]

[0129] The goal is the total reward obtained by adjusting the policy parameters θ The larger the better. The most commonly used gradient estimator The expression is:

[0130]

[0131] Due to the large variance of the Monte Carlo sampling method, the training process is difficult to converge. Introducing the advantage function To help agents choose actions more effectively and speed up the convergence of reinforcement learning algorithms, generalized advantage estimation (GAE) is often used to calculate the advantage function. Used to evaluate the relative advantages of various actions compared to the current strategy, the calculation expression is:

[0132]

[0133] Where V w(st) Indicates state s t The value of the value function, r t represents the reward of the t-th action, γ is the discount rate, which is used to control the impact of long-term benefits, λ is the decay factor, which is used to control the trade-off between deviation and variance, l is the current accumulated time step, is the timing differential error at t+1.

[0134] In order to prevent policy collapse or unstable learning process, PPO enhances the stability of the algorithm by limiting the amplitude of policy updates. The core policy loss function is defined as follows:

[0135]

[0136] Where, L {CLIP}(θ) is the loss function, is the expected value at time step t, clip is the clipping strategy, and the clipped value is in the interval [1-ε,1+ε], To estimate action a t Relative to state s t The advantage of r t (θ) represents the ratio between the new and old strategies, and L is a hyperparameter used to ensure that the gap between the new and old strategies will not be too large when multiple strategy updates are performed using the same batch of data.

[0137] In addition, the training goal of the Critic network is to minimize the square error (MSE), which is calculated as follows:

[0138] L(θ)=E[(G t -V(s t ;θ)) 2 ]

[0139] In the formula, θ represents the parameters of the Critic network, V(s t ; θ) represents the Critic network's response to state s t The estimated value of G t Indicates that from state s t Start observing actual returns.

[0140] In order to verify the performance of the present invention, the following experiments were designed.

[0141] The experiment compares the asynchronous collaborative multi-agent framework (ACO) with the independent distributed multi-agent framework (DNCO) on a real refinery case. In the refinery case, there are 6 candidate crude oils, 14 final products and 28 processing units. In addition, there are 5 blending units, namely diesel blending unit, G92 blending unit, G95 blending unit, jet fuel oil blending unit and naphtha blending unit. The time step of the production agent is one month, while the time step of the crude oil procurement agent is three months. This setting reflects the quarterly optimization of the crude oil procurement plan of the refinery. In each procurement cycle, the crude oil price fluctuates according to market conditions, while the product demand is adjusted monthly. The results show that in DNCO, the procurement agent reduces the crude oil procurement volume in order to reduce the procurement cost, resulting in the interruption of the production agent's production; in ACO, the procurement agent and the production agent solve the supply and demand imbalance problem well through the asynchronous coordination mechanism. In addition, compared with DNCO, ACO brings higher production profits, lower crude oil costs, and stores more crude oil when the crude oil price is low. Comparison of learning curves of production agents and procurement agents under different methods Figure 3 shown.

[0142] Compared with the prior art, the present invention has the following beneficial effects:

[0143] 1) The present invention is a multi-period production and procurement collaborative optimization of an oil refinery based on multi-agent reinforcement learning. Production agents and procurement agents that make decisions on different time scales are designed, an asynchronous collaborative framework and information interaction mechanism are constructed, and the system's adaptability to dynamic market conditions, such as fluctuating crude oil prices and product demand, is enhanced, thereby improving production efficiency, reducing operating costs, and maintaining a balanced crude oil inventory.

[0144] 2) The present invention models the production process, device operating parameters, etc. respectively. The present invention designs crude oil procurement and production planning as two intelligent entities, which collaborate asynchronously and make independent decisions, which can well solve the problem of supply and demand imbalance and improve production efficiency.

[0145] 3) The present invention introduces an advantage function to evaluate the relative advantages of various actions compared with the current strategy, helping the intelligent agent to choose actions more effectively and speeding up the convergence of the reinforcement learning algorithm.

[0146] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for collaborative optimization of production and procurement in a refinery based on multi-agent reinforcement learning, characterized in that: The method comprises the following steps: S1. Construct production agents, procurement agents and production planning models, set the action space, state space, reward function of the production agents and procurement agents, and the environment based on the production planning model update after the agents output actions; S2, setting the action interaction, state interaction and reward interaction between the production agent and the procurement agent; S3. Based on the interaction set up in S2, the asynchronous MAPPO method is used to train the production agent and the procurement agent to obtain the actual refinery production data. The actual refinery production data is input into the trained production agent and the procurement agent to obtain the actual production procurement plan.

2. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 1, characterized in that: The action space of the production agent and the procurement agent is: in, is the action space of the purchasing agent, q c,t represents the purchase volume of crude oil c at time t, Crude represents the set of all crude oil types available for purchase, and q p,t represents the production of product p at time t, P represents the product set, The action space of the production agent.

3. The method for collaborative optimization of production and procurement of an oil refinery based on multi-agent reinforcement learning according to claim 2 is characterized in that: The state space of the production agent and the procurement agent is: in, is the state space of the purchasing agent, Price crude,t represents the crude oil price, D product,t Indicates product demand, Inv crudd,t Indicates the inventory level of raw materials, is the state space of the production agent, O prod,t Indicates the status information of the current product order, Inv prod,t Indicates the stock level of a product.

4. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 3 is characterized in that: The local reward functions of the production agent and the procurement agent are: Among them, Price crude,t is the crude oil price at time t, Cost c,t is the inventory unit holding cost, Inv c,t represents the inventory level of crude oil at time t, T represents the time interval between two purchase actions, and profit PA,t′ is the profit generated by the production agent at each time step t′ in this time interval, and β represents the weight coefficient; Among them, Price p,t Cost m,t Cost v,t and Cost u,t are the selling price of product p, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each device u at time t. p,t 、Buy m,t 、Inv v,t are the volume flow of product p, the purchase volume of raw material m, and the inventory volume of storage tank v at time t. u,t To describe the processing load of CDU.

5. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 1, characterized in that: The specific steps of training the production agent and the procurement agent using the asynchronous MAPPO method based on the interaction set by S2 are as follows: The production agent and the procurement agent obtain the current state of the environment within their respective update frequencies, output the current action, and respectively calculate the weighted embedded reward function with the other party's reward. The respective trajectories are stored in their corresponding buffers, and the MAPPO method is used to update their respective neural network parameters through the trajectory. The production agent and the procurement agent are trained independently.

6. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 5, characterized in that: The production agent and the procurement agent obtain the current state of the environment within their respective update frequencies and output the specific expression of the current action: The meaning of this expression is: at time t, the production agent is based on the state Select actions and influence the actions of the purchasing agent by interacting with the environment Similarly, the purchasing agent is based on the state Select actions and interact with the environment to influence the actions of the production agent k reflects different update frequencies.

7. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 6 is characterized in that: The optimization objectives of the production agent and the procurement agent are: in, and They are the optimization decisions of the purchasing agent and the production agent respectively, E represents the expectation, and γ represents the discount rate.

8. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 7, characterized in that: The specific steps of using the MAPPO method to update the respective neural networks through trajectories are: Sample trajectories and adjust the neural network parameters according to the reward function until the round reward is maximized; The strategy loss function of the MAPPO method is: Where, L {CLIP}(θ) is the loss function, is the expected value at time step t, clip is the clipping strategy, and the clipped value is in the interval [1-ε,1+ε], To estimate the advantage function of the action relative to the state, r t (θ) represents the ratio between the new and old strategies, and ε is a hyperparameter.

9. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 8, characterized in that: The advantage function is: in, Indicates state s t The value of the value function, r t represents the reward of the t-th action, γ is the discount rate, which is used to control the impact of long-term benefits, λ is the decay factor, which is used to control the trade-off between deviation and variance, l is the current accumulated time step, is the timing differential error at t+1.

10. The method for collaborative optimization of production and procurement of a refinery based on multi-agent reinforcement learning according to claim 9, characterized in that: The training objective of the Critic network in the neural network is to minimize the squared error.

Images