Oil refinery production and maintenance collaborative optimization method based on reinforcement learning
By applying reinforcement learning in oil refineries to optimize production planning and maintenance cycles, the frequent maintenance of residual oil hydrogen refueling equipment due to metal deposition and impact of crude oil price fluctuations is solved, and more efficient production scheduling and stable operation of the equipment are achieved.
Patent Information
- Application Number
- CN202411023826.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
The residual oil hydrogenation device in the refinery is frequently repaired due to metal deposition problems, has a short operating cycle, and the existing mathematical planning methods cannot effectively deal with uncertain factors such as fluctuations in crude oil prices, resulting in low production scheduling efficiency and affecting profits.
Adopting a collaborative optimization method for refinery production and maintenance based on reinforcement learning, by constructing Markov decision-making model and production planning model, combining PPO algorithm, optimize production planning and residual oil hydrogen refueling equipment maintenance cycle, and designing a weighted reward function to balance production efficiency and maintenance Cost, use supervised learning to extract decision-making knowledge from production cases, initialize the Actor network, and optimize crude oil procurement strategies.
It improves the calculation efficiency and stability of the refinery, extends the maintenance cycle of the residual oil hydrogenation device, improves the production efficiency and stable operation of the device, and can more effectively deal with crude oil price fluctuations and metal deposition problems.
Smart Images

Figure CN118966431A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of refinery production scheduling, and in particular to a refinery production maintenance collaborative optimization method based on reinforcement learning. Background Art
[0002] During the operation of the refinery, impurities such as asphaltene and metals in the residual oil form coke and metal deposits covering the catalyst surface, causing the catalyst to deactivate. In recent years, with the serious deterioration of crude oil, the residual oil hydrotreating unit (RHU) has been frequently overhauled due to metal deposition problems. Its operation cycle is only 1 to 2 years, while the operation cycle of the downstream catalytic cracking unit is generally about 4 to 5 years. The mismatch of operation cycles is an important problem faced by the fixed-bed residual oil hydrotreating-catalytic cracking combined process, which seriously affects the profits of the refinery. In addition, the refinery will face disturbances from various uncertain factors during the production process, such as crude oil price fluctuations. There is a combinatorial explosion problem in the production scheduling problem of large-scale long-term refineries. The existing mathematical programming methods are inefficient in solving and cannot cope with the disturbances of uncertain factors. Summary of the invention
[0003] The purpose of the present invention is to provide a method for collaborative optimization of refinery production maintenance based on reinforcement learning in order to overcome the defects of low solution efficiency and poor stability in the above-mentioned prior art.
[0004] The purpose of the present invention can be achieved by the following technical solutions:
[0005] A method for collaborative optimization of refinery production maintenance based on reinforcement learning, comprising the following steps:
[0006] S1: Collect production cases, extract production decision knowledge from production cases through supervised learning, and initialize the Actor network of reinforcement learning;
[0007] S2: Construct Markov decision model and production planning model for multi-period production and maintenance collaborative optimization;
[0008] S3: A production and maintenance collaborative optimization framework based on hierarchical reinforcement learning combines the Markov decision model and the production planning model in a complementary way through two-way information exchange, and then uses the PPO algorithm to optimize and solve the optimized production plan and residue hydrotreating unit maintenance cycle.
[0009] Furthermore, the production planning model considers the metal deposition calculation of the residue oil hydrotreating unit and establishes the coupling relationship between the metal deposition amount and the crude oil selection of the production plan. The calculation expression of the coupling relationship is:
[0010]
[0011] In the formula, Inv v,t 、Invv,t-1 are the processing loads of tank v at time t and t-1 respectively, and They represent the flow rate into and out of tank v in period t respectively.
[0012] Furthermore, the process of using the PPO algorithm to train and learn the production and maintenance decision-making strategy includes: sampling the production decision trajectory, adjusting the network parameters according to the reward function until the round reward is maximized, and introducing the advantage function Used to evaluate the relative advantages of various actions compared to the current strategy, the calculation expression is:
[0013]
[0014]
[0015] In the formula, Indicates state s t The value of the value function, r t represents the reward of the t-th action, γ is the discount rate, λ is the decay factor, l is the current accumulated time step, is the timing differential error at t+1, For advantage.
[0016] Furthermore, the strategy loss function of the PPO algorithm is:
[0017]
[0018] Where, L {CLIP}(θ) is the loss function, is the expected value at time step t, clip is the clipping strategy, and the clipped value is in the interval [1-ε,1+ε], To estimate the advantage function of the action relative to the state, r t (θ) represents the ratio between the new and old strategies, and ε is a hyperparameter.
[0019] Furthermore, the production planning model includes crude oil selection, unit processing plan, unit processing volume, inventory and product blending plan. The maintenance cycle of the residue oil hydrotreating unit arranges the unit shutdown and maintenance time according to the changes in metal deposition in the residue oil hydrotreating unit.
[0020] Furthermore, the PPO algorithm includes a weighted reward function, including a production efficiency reward function and a maintenance cost penalty function. The production efficiency reward function r 1t The expression is:
[0021]
[0022] In the formula, Price p,t 、Pricem,t Cost v,t and D u,t are the selling price of product p at time t, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each unit u. p,t 、Buy m,t 、Inv v,t They are the volume flow of product p, the purchase volume of raw material m and the inventory volume of tank v at time t, and the maintenance cost penalty function r 2t The expression is:
[0023]
[0024] In the formula, eta q is the upper limit of RHU metal deposition, Flow c,t and Q c,q They represent the processing volume of crude oil c and the metal content of crude oil c at time t respectively.
[0025] Furthermore, step S1 specifically includes sampling the uncertainty of crude oil prices, constructing a state-action pair data set, extracting production scheduling decision knowledge using a supervised learning method, and migrating the production decision knowledge to the Actor network of reinforcement learning.
[0026] Furthermore, the network used in supervised learning is consistent with the Actor network used in reinforcement learning.
[0027] Furthermore, the construction of the Markov decision model includes defining the state space as the inventory level, crude oil price, and the status information of the residue hydrotreating unit, defining the action space as the crude oil purchase volume, setting a weighted reward function to balance the conflict between production decisions and unit maintenance decisions, and a state transition function.
[0028] Furthermore, the production planning model includes raw material supply constraints, product demand constraints, product property constraints, device load constraints, device operation constraints, and tank processing load constraints;
[0029] The formula for raw material supply constraint is:
[0030]
[0031] In the formula, and is the minimum and maximum purchase amount of raw material (i.e. crude oil) m in the planning period t, b m is a coefficient with a value of 0 or 1;
[0032] The formula describing the demand for a product is:
[0033]
[0034] In the formula, and is the minimum and maximum market demand of product p in planning period t, Sell p,t is the sales volume of product p in planning period t.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] 1) The present invention is a multi-cycle production and maintenance collaborative optimization of a refinery based on reinforcement learning. A hierarchical optimization framework is constructed. The upper Markov decision model determines the crude oil procurement plan under different production conditions, and the lower production planning model optimizes the production decision under the crude oil procurement plan. A weighted reward function is designed to comprehensively consider the refinery production efficiency and RHU unit maintenance cost, and the optimized production plan and residue oil hydrogenation unit maintenance cycle are obtained, which improves the calculation efficiency and enhances the stability, which is beneficial to the improvement of refinery production efficiency and the stable operation of the unit.
[0037] 2) The present invention models the production process, device operating parameters, etc. respectively. The present invention designs crude oil procurement as a reinforcement learning action. By calculating the crude oil procurement volume, it can match the needs of the production plan and improve production efficiency.
[0038] 3) The present invention introduces an advantage function to evaluate the relative advantages of various actions compared with the current strategy, helping the intelligent agent to choose actions more effectively and speeding up the convergence of the reinforcement learning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a schematic diagram of the process of the present invention.
[0040] Figure 2 This is a simplified process diagram of an oil refinery in the embodiment.
[0041] Figure 3 It is a simplified process flow diagram of the residue oil hydrogenation unit in the embodiment. DETAILED DESCRIPTION
[0042] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0043] Example 1
[0044] like Figure 1 As shown, the present invention is a method for collaborative optimization of refinery production maintenance based on reinforcement learning, comprising the following steps:
[0045] S1: Collect production cases, extract production decision knowledge from production cases through supervised learning, and initialize the Actor network of reinforcement learning;
[0046] By sampling the uncertainty of crude oil prices, a deterministic production planning model is constructed to optimize and solve production cases under different states, including raw material supply constraints, product demand constraints, device operation constraints, device processing capacity constraints, product property constraints and tank processing load constraints. A state-action pair dataset is constructed to extract production scheduling decision knowledge using supervised learning methods, and the production decision knowledge is transferred to the Actor network of reinforcement learning.
[0047] Among them, the network used in supervised learning is consistent with the Actor network used in reinforcement learning.
[0048] like Figure 2 As shown in the figure, the production operation process of a refinery involves three parts, namely crude oil blending and processing, secondary processing and product blending. Crude oils of different properties and prices are mixed and sent to the crude distillation unit (CDU) to be separated into different fractions such as light hydrocarbon gas, gasoline, kerosene, diesel and residue at different temperatures. These fractions flow to the secondary processing unit for further processing. The main secondary processing units include catalytic cracking (FCC), continuous reforming (CRU), delayed coking (DCU), solvent deasphalting, hydrocracking (HCU), residue hydrotreating unit (RHU) and hydroprocessing.
[0049] like Figure 3 As shown, the residual oil from the atmospheric and vacuum distillation unit is pressurized by a high-pressure pump, and after heat exchange with the reactor generated oil, it is mixed with hydrogen and then enters the heating furnace for heating; the heated oil-gas mixture enters the reactor, and under the high temperature and catalyst environment, harmful impurities such as sulfur, nitrogen and metal in the residual oil molecules react with hydrogen and hydrogen sulfide respectively to generate hydrogen sulfide, ammonia and metal sulfides, and at the same time, some larger molecules in the residual oil are cracked and hydrogenated to become ideal components with smaller molecules, and the metal sulfides generated by the reaction are deposited on the catalyst. The reaction generated oil enters the high-pressure separation system after a series of heat exchange and cooling; oil, gas and water are separated into three phases in the hot high-pressure separation tank, the cold high-pressure separation tank, the hot low-pressure separation tank and the cold low-pressure separation tank, and the oil tank is sent to the fractionation system to separate naphtha, diesel and hydrogenated tail oil, so as to provide raw materials for subsequent devices such as catalytic cracking and hydrocracking.
[0050] This embodiment is aimed at Figure 2 and Figure 3 The integrated model of production planning and equipment maintenance is constructed through the process.
[0051] The formula describing the raw material supply constraint is:
[0052]
[0053] In the formula, and is the minimum and maximum value of the purchase amount of raw material (i.e. crude oil) m in the planning period t, b m is a coefficient that takes a value of 0 or 1.
[0054] The formula describing the demand for a product is:
[0055]
[0056] In the formula, and is the minimum and maximum market demand of product p in planning period t, Sell p,t is the sales volume of product p in planning period t.
[0057] For the atmospheric and vacuum unit, due to the many nonlinearities and complexities in the atmospheric and vacuum unit (CDU) modeling process, a fixed yield model is used to describe the CDU modeling process. m,t The formula is:
[0058]
[0059] Where CO is the set of crude oil m, MCU m,u,t It represents the amount of crude oil m flowing into unit u of type CDU at time t. U (U = CDU, PU, SPL, MIX) represents the various units of the refinery, including atmospheric and vacuum crude distillation units, secondary processing units, separators and mixers. Secondary processing units include catalytic cracking, continuous reforming, delayed coking, solvent deasphalting, hydrocracking and hydrotreating units.
[0060] Describe the processing load flow of CDU u,t The formula is:
[0061]
[0062] The formula describing the processing load constraint range of CDU is:
[0063]
[0064] In the formula, and are the minimum and maximum values of CDU processing load.
[0065] The content of different distillates in CDU is calculated by multiplying the processing volume of different crude oils by the fixed yield Yield. s,u,t , the formula is:
[0066]
[0067] Where S(s∈{1,2,…S}) is the set of all flows, and UO=(u,s) represents all outlet flows of the device.
[0068] for Figure 2 , Figure 3 The devices mentioned in the above can be classified into two categories: single-mode processing devices and multi-mode processing devices. The following settings are made for different types of devices:
[0069] For a single-mode process unit, the formula describing the material balance, process load and unit product output for a single-mode process unit u is:
[0070]
[0071]
[0072]
[0073] For multi-mode processing devices, each device can be operated in M (m∈{1,2,…M}) production modes according to production requirements, and the product yields of the devices under different operation modes are different. According to material balance, the total flow of multi-mode processing device u under mode op (different schemes, different ratios of products and raw materials), the total load of device u and the load constraint formula are:
[0074]
[0075]
[0076]
[0077] Where UI = (u, s) represents all inlet flows of the device.
[0078] The formula that describes the process of flow division and confluence to ensure that the mass balance is maintained is:
[0079]
[0080] The quality of blended products must meet specific standards and specifications, including requirements on the content of various ingredients, physical properties (such as density, viscosity), chemical properties (such as sulfur content, aromatics content), etc.
[0081] The formula describing the output of product p calculated by each blending component c∈BC is:
[0082]
[0083] The formula describing the volumetric properties of component c of the blend product p is:
[0084] QV c,p,q,t Flow c,p,t Q c,q,t c∈BC,p∈P,q∈Q,t∈T
[0085] In the formula, Q c , q , t represents the properties of the blending component c in period t. Q(q∈1,2,...Q) is a set of product attributes, where the elements represent sulfur content, density, research octane number, cetane number, olefin content, and aromatics.
[0086] The formula describing the product property constraints is:
[0087]
[0088] In the formula, and are the minimum and maximum values of the product p property.
[0089] The multi-period production plan is connected through the raw material inventory and the intermediate product inventory. The formula describing the coupling relationship of each period inventory is:
[0090]
[0091] In the formula, and They represent the flow rate into and out of tank v in period t respectively.
[0092] In order to maintain inventory balance throughout the entire production cycle and prevent the inventory from being cleared out in the last cycle, the formula describing that the inventory at the end of the last cycle should be equal to the inventory at the beginning of the first cycle is:
[0093] Inv v,T =Inv v ,0 v∈V,t∈T
[0094] The formula describing the process load constraint for tank v is:
[0095]
[0096] In the formula, and are the minimum and maximum values of the processing load of tank v.
[0097] Describe the energy consumption ED of each processing unit u,t The formula that is linearly related to the processing load is:
[0098] ED u,t =Cost u,t Flow u,t
[0099] Description: When the metal deposition amount of the residue oil hydroprocessing unit is less than the threshold value δ, the catalyst activity is good and the unit can operate normally. Otherwise, the residue oil hydroprocessing unit needs to be shut down to replace the catalyst:
[0100]
[0101]
[0102] In the formula, x t =1 indicates that the residue oil hydrotreating unit is in normal operation.
[0103] The equation describing the relationship between metal deposition in a residue hydroprocessing unit and crude oil selection for production planning is:
[0104]
[0105] In the formula, Flow m,t Indicates the processing volume of each crude oil, Q m , q Indicates the metal content of each crude oil.
[0106] The formula describing the metal deposition of the residue hydroprocessing unit under different conditions is:
[0107] Metal t =0 if y t =1
[0108]
[0109] S2: Construct Markov decision model and production planning model for multi-period production and maintenance collaborative optimization;
[0110] This step specifically includes:
[0111] 1) State space design: Generally speaking, the system state of the refinery production process mainly involves two types of information: physical system state (inventory, unit operation status) and market information (price, demand, supply). Specifically, the model state in this study includes inventory information, RHU state information, and crude oil price information. The inventory information is represented by the inventory level of all storage tanks, and the inventory is limited by the tank capacity. The RHU state information is obtained by dividing the remaining metal deposition in the RHU by the metal content of various crude oils, reflecting the impact of different RHU states on crude oil selection.
[0112] 2) Action Space Design: The impact of crude oil procurement on the multi-period production and maintenance collaborative optimization problem is multifaceted. First, the quality, price, and supply of crude oil directly affect the production efficiency, cost, and profitability of the refinery. A reasonable crude oil procurement strategy can effectively control production costs, thereby bringing economic benefits to production operations. Second, crude oil quality is closely related to the metal deposition rate of the RHU, which affects the operating status and service life of the unit. When processing raw materials containing a large amount of metals, resins, and asphaltene, these impurities can lead to the formation of coke and metal deposition on the catalyst surface, which can lead to catalyst deactivation and bring challenges to the operation of the RHU. Reasonable crude oil procurement decisions can optimize the maintenance requirements of the RHU and ensure its stable operation and service life. In addition, crude oil price fluctuations and supply interruptions also bring risks. By formulating appropriate procurement strategies, these risks can be effectively controlled to ensure the stable execution of production plans. Reasonable crude oil procurement decisions play a vital role in the efficiency, benefits, and risk management of the entire production system. Therefore, crude oil procurement is designed as a reinforcement learning action. Each element of the action vector represents the purchase volume of each crude oil in each period. The action vector is defined as [c, c∈Crude], whose length is the length of the crude oil category set (Crude).
[0113] 3) Reward function design: The reward function is a key factor that directly affects the performance of reinforcement learning. In the multi-task optimization problem of the refinery, the optimization goal should consider both the production plan and the RHU maintenance scheduling, and guide the agent to learn appropriate strategies through the reward function. In HTDRL, the operation cycle of the RHU is used as a round of reinforcement learning. The reward function encourages the agent to choose a strategy to extend the operation cycle of the RHU to improve the overall economic benefits.
[0114] To this end, this paper designs a weighted reward function, which mainly consists of two parts: rewards for production efficiency and penalties for RHU maintenance costs.
[0115] The reward for production efficiency is determined by the optimal solution of the single-cycle production model, including product revenue, raw material cost, inventory cost, and operating cost:
[0116]
[0117] In the formula, r 1t As a reward for productivity, Price p,t 、Price m,t Cost v,t and D u,t are the selling price of product p at time t, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each unit u. p,t 、Buy m,t 、Invv,t They are the volume flow of product p, the purchase quantity of raw material m and the inventory in tank v at time t.
[0118] The penalty for RHU maintenance cost is calculated based on the stable operation time and the RHU metal deposition status, which can limit the proportion of heavy oil processed by the RHU, thereby extending its operation cycle. The lower the metal deposition level, the longer the RHU operation cycle and the higher the reward, and vice versa. Specifically, the reward function is designed based on the impact of the metal deposition amount on the RHU operation cycle in each cycle:
[0119]
[0120] In the formula, eta q is the upper limit of RHU metal deposition, Flow c,t and Q c,q They represent the processing volume of crude oil c and the metal content of crude oil c at time t respectively.
[0121] In order to coordinate the two goals of production and RHU maintenance, we designed a weighted reward function that takes both production efficiency and RHU reliability into account: t =r 1t +βr 2t It allows weights to be adjusted in different environments and conditions, thereby solving multi-task conflicts and collaborative optimization problems.
[0122] In the hierarchical reinforcement learning framework, the Markov decision model and the production planning model are effectively combined in a complementary way through bidirectional information exchange. Among them, the production planning model accepts crude oil purchase decisions from the Markov decision model actor network, and the state transitions and rewards of the Markov decision model within a planning cycle are determined by the optimization simulation of the production planning model. The maintenance of the RHU involves decisions across time scales, which will produce metal deposition over time. By combining the Markov decision model and the production planning model, the relevant fast time scale dynamics and uncertainties can be better captured, so as to make robust decisions and respond quickly throughout the operation cycle. The production planning model takes into account constraints such as production capacity, equipment availability, raw material supply, product demand, quality and safety standards, ensuring the feasibility and effectiveness of the production plan. Since the production planning model is a small-scale mixed integer problem without random factors, it can be solved efficiently using Gurobi.
[0123] S3: A production and maintenance collaborative optimization framework based on hierarchical reinforcement learning combines the Markov decision model and the production planning model in a complementary way through two-way information exchange, and then uses the PPO algorithm to optimize and solve the optimized production plan and the maintenance cycle of the residue oil hydrotreating unit.
[0124] The PPO reinforcement learning algorithm is used for trajectory sampling and production decision-making strategies are learned to generate strategies that maximize expected returns. The optimal production schedule and device maintenance cycle are solved for scheduling production.
[0125] For a complete event τ, the total reward obtained during the whole process is For a given policy with parameters θ, the weighted sum of rewards obtained from each sampled event τ is expressed as:
[0126]
[0127] The goal is the total reward obtained by adjusting the policy parameters θ The larger the better. The most commonly used gradient estimator The expression is:
[0128]
[0129] Due to the large variance of the Monte Carlo sampling method, the training process is difficult to converge. Introducing the advantage function It is used to evaluate the relative advantages of various actions compared with the current strategy, help the agent select actions more effectively, and speed up the convergence of the reinforcement learning algorithm. The generalized advantage estimation (GAE) is usually used to calculate the advantage function.
[0130]
[0131]
[0132] In the formula, Indicates state s t The value of the value function, r t represents the reward of the t-th action, γ is the discount rate, which is used to control the impact of long-term benefits, λ is the decay factor, which is used to control the trade-off between deviation and variance, l is the current accumulated time step, is the timing differential error at t+1.
[0133] In order to prevent policy collapse or unstable learning process, PPO enhances the stability of the algorithm by limiting the amplitude of policy updates. The core policy loss function is defined as follows:
[0134]
[0135] Where, L {CLIP}(θ) is the loss function, is the expected value at time step t, clip is the clipping strategy, and the clipped value is in the interval [1-ε,1+ε], To estimate action a t Relative to state lt The advantage of r t (θ) represents the ratio between the new and old strategies, and ε is a hyperparameter used to ensure that the gap between the new and old strategies will not be too large when multiple strategy updates are performed using the same batch of data.
[0136] In addition, the training goal of the Critic network is to minimize the square error (MSE), which is calculated as follows:
[0137] L(θ)=E(G t -V(s t ;θ)) 2 ]
[0138] In the formula, θ represents the parameters of the Critic network, V(s t ; θ) represents the Criti network for state s t The estimated value of G t Indicates that from state s t Start observing actual returns.
[0139] In order to verify the performance of the present invention, the following experiments were designed.
[0140] The experiment compared the knowledge transfer-based deep reinforcement learning method (HTDRL) with existing uncertainty methods, including stochastic programming and distributed robust optimization, on different refinery cases. In the refinery case, there are 6 candidate crude oils, 14 final products, and 28 processing units. In addition, there are 5 blending units, namely diesel blending unit, G92 blending unit, G95 blending unit, jet fuel oil blending unit, and naphtha blending unit. The crude oil price fluctuates 10% above and below the benchmark price. It is assumed that the upper limit of metal deposition in the RHU is 36,000 ppm, that is, when the metal accumulation reaches this value, the RHU must be shut down to replace the catalyst. In all three cases, this method has better production efficiency.
[0141] Table 1 Comparative experimental results of HTDRL model and other models on different refinery cases
[0142]
[0143] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A method for collaborative optimization of refinery production maintenance based on reinforcement learning, characterized in that: The following steps are involved: S1: Collect production cases, extract production decision knowledge from production cases through supervised learning, and initialize the Actor network of reinforcement learning; S2: Construct Markov decision model and production planning model for multi-period production and maintenance collaborative optimization; S3: A production and maintenance collaborative optimization framework based on hierarchical reinforcement learning combines the Markov decision model and the production planning model in a complementary way through two-way information exchange, and then uses the PPO algorithm to optimize and solve the optimized production plan and the maintenance cycle of the residue oil hydrotreating unit.
2. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: The production planning model considers the metal deposition calculation of the residue oil hydrotreating unit and establishes the coupling relationship between the metal deposition amount and the crude oil selection of the production plan. The calculation expression of the coupling relationship is: In the formula, Inv v,t 、Inv v,t-1 are the processing loads of the tank v at time t and t-1 respectively, and They represent the flow rate flowing into tank v and the flow rate flowing out of tank v in period t respectively.
3. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: The process of using the PPO algorithm to train and learn the production and maintenance decision strategy includes: sampling the production decision trajectory, adjusting the network parameters according to the reward function until the round reward is maximized, and introducing the advantage function Used to evaluate the relative advantages of various actions compared to the current strategy, the calculation expression is: In the formula, Indicates state s t The value of the value function, r t represents the reward of the t-th action, γ is the discount rate, λ is the decay factor, l is the current accumulated time step, is the timing differential error at t+1, For advantage.
4. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 3 is characterized in that: The strategy loss function of the PPO algorithm is: Where, L {CLIP}(θ) is the loss function, is the expected value at time step t, clip is the clipping strategy, and the clipped value is in the interval [1-ε,1+ε], To estimate the advantage function of the action relative to the state, r t (θ) represents the ratio between the new and old strategies, and ε is a hyperparameter.
5. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: The production planning model includes crude oil selection, device processing plan, device processing volume, inventory and product blending plan. The maintenance cycle of the residual oil hydrogenation device arranges the device shutdown and maintenance time according to the changes in metal deposition in the residual oil hydrogenation device.
6. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: The PPO algorithm includes a weighted reward function, including a production efficiency reward function and a maintenance cost penalty function. The production efficiency reward function r 1t The expression is: In the formula, Price p,t 、Price m,t Cost v,t and D u,t are the selling price of product p at time t, the purchase price of raw material m, the inventory cost of storage tank v, and the operating cost of each unit u. p,t 、Buy m,t 、Inv v,t They are the volume flow of product p, the purchase volume of raw material m and the inventory volume of tank v at time t, and the maintenance cost penalty function r 2t The expression is: In the formula, eta q is the upper limit of RHU metal deposition, Flow c,t and Q c,q They represent the processing volume of crude oil c and the metal content of crude oil c at time t respectively.
7. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: Specifically, step S1 samples the uncertainty of crude oil prices, constructs a state-action pair data set, uses supervised learning methods to extract production scheduling decision knowledge, and transfers the production decision knowledge to the Actor network of reinforcement learning.
8. The method for collaborative optimization of oil refinery production maintenance based on reinforcement learning according to claim 7, characterized in that: The network used in the supervised learning is consistent with the Actor network used in the reinforcement learning.
9. The method for collaborative optimization of refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: The construction of the Markov decision model includes defining the state space as inventory level, crude oil price, and state information of the residue hydrotreating unit, defining the action space as the crude oil purchase volume, setting a weighted reward function to balance the conflict between production decisions and unit maintenance decisions, and a state transfer function.
10. The method for collaborative optimization of oil refinery production maintenance based on reinforcement learning according to claim 1, characterized in that: The production planning model includes raw material supply constraints, product demand constraints, product property constraints, device load constraints, device operation constraints and tank processing load constraints; The formula for the raw material supply constraint is: In the formula, and is the minimum and maximum value of the purchase amount of raw material (i.e. crude oil) m in the planning period t, b m is a coefficient with a value of 0 or 1; The formula describing the demand for a product is: In the formula, and is the minimum and maximum market demand of product p in planning period t, Sell p,t is the sales volume of product p in planning period t.
Citation Information
Cited By
Oil refinery production and purchase collaborative optimization method based on multi-agent reinforcement learning
CN119809516A
Wind turbine generator predictive maintenance decision-making method and system applying reinforcement learning
CN120563110A
Event-triggered safety reinforcement learning based optimization method for crude oil scheduling in oil refineries
CN122840544A
