A dynamic scheduling method for gas system based on improved proximal strategy optimization

Abstract

The present invention discloses a method for dynamic scheduling of a gas system based on improved proximal strategy optimization, comprising: (1) determining the production plan interval and gas production and consumption devices according to the scheduling optimization process of the gas system, and establishing a gas pipe network model; (2) determining the initial state of the gas pipe network model according to the gas production and the initial value of the gas pipe network pressure, and iteratively updating it on this basis; (3) constructing a reward function for evaluating the quality of actions in the current state according to the optimization objective function; (4) using a reinforcement learning agent model to obtain an action a based on the state, updating the model until a scheduling is completed, recording the state, action, and reward of the scheduling process, and updating the network parameters of the model; (5) after the iterative training is completed, applying the model to a test set for testing; and (6) using the trained model to optimize the scheduling of the gas system. Utilizing the present invention, the load capacity of the gas pipe network can be better predicted, and the pressure balance level of the gas pipe network can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the fields of gas system balance and artificial intelligence applications, and in particular to a gas system dynamic scheduling method based on improved proximal strategy optimization. Background Art

[0002] The gas system is a crucial component of an oil refinery's energy system and one of its primary fuel sources. The gas pipeline network is the primary equipment for gas transportation, but the network's inherent pressure is constrained by upper and lower limits. If gas production significantly exceeds consumption, the gas will exceed the upper pressure limit, potentially posing a safety hazard. If gas production significantly exceeds consumption, the gas will fall below the lower pressure limit, potentially causing mechanical failures.

[0003] Currently, there are relatively few scheduling optimization algorithms practically applied to gas systems, and most still rely on manual experience or traditional methods. Whether using manual methods or traditional methods based on heuristic algorithms, the performance of the solutions they generate is generally mediocre. The actual effectiveness of scheduling rules determined by manual experience varies significantly depending on the decision maker. Furthermore, the scheduling strategies that heuristic algorithms can generate within a limited timeframe are only slightly better than those generated manually, and often exhibit a degree of randomness, making them incapable of achieving stable results.

[0004] Chinese patent publication CN101794119A discloses a method for balancing and optimizing gas system scheduling based on forecast data, comprising: obtaining data required to trigger the gas system from a scheduling system; predicting the gas generation and energy demand of each production unit within a preset future time period based on the required data to obtain forecast data; determining whether the gas system's production and demand are balanced within the preset future time period based on the forecast data; and optimizing the scheduling strategy and scheduling plan for each production unit within the preset future time period based on the forecast data if the production and demand of the gas system are unbalanced within the preset future time period; and displaying the optimized scheduling strategy and scheduling plan for each production unit within the preset future time period via a client to facilitate scheduling. However, this method relies on relatively accurate historical data for predicting gas system production and demand. If the historical data is insufficient or the current production and demand data has significantly changed compared to the historical situation, the accuracy of the forecast cannot be guaranteed. Furthermore, the forecast model proposed by this method requires prior production plan scheduling data, which makes it difficult to achieve accurate forecasts in scenarios where production plans are dynamically changing. In addition, the mixed integer linear programming algorithm for the scheduling system proposed in this method requires repeated iterations and has a long running time, making it difficult to achieve real-time scheduling.

[0005] Compared to traditional methods, applying deep reinforcement learning to scheduling optimization problems is a new data-driven approach. It has the following advantages:

[0006] (1) Generalization ability: Traditional methods usually need to start from scratch for a new problem and obtain a relatively better solution through iteration. Deep reinforcement learning methods enable algorithms to have learning capabilities. By analyzing and solving some problems, they can effectively obtain solutions when given a new problem.

[0007] (2) Scalability: Deep reinforcement learning methods have the potential to reduce time complexity to linear, and combined with mature parallel acceleration capabilities, they can be applied to large-scale problems.

[0008] (3) Universality: The trained model can be applied to problems of different scales and parameters, without the need to design new parameter training and learning for each problem.

[0009] However, there is currently little research and application in academia and engineering circles on the optimization of gas system scheduling in refineries using deep reinforcement learning algorithms. Summary of the Invention

[0010] The present invention provides a gas system dynamic scheduling method based on improved proximal strategy optimization. Dynamic scheduling for the gas system can better predict the load capacity of the gas pipeline network and effectively improve the pressure balance level of the gas pipeline network.

[0011] A gas system dynamic scheduling method based on improved proximal strategy optimization includes:

[0012] (1) According to the scheduling optimization process of the gas system, the production plan interval, gas production equipment and gas consumption equipment are determined, and the gas pipeline network model is established;

[0013] (2) Determine the initial state of the gas network model based on the initial values ​​of gas production and gas network pressure, and iteratively update it on this basis;

[0014] (3) Based on the optimization objective function, a reward function is constructed to evaluate the quality of the action in the current state. The reward function is represented by the income generated by the gas consumption device and the pressure balance of the gas network. The formula is as follows:

[0015]

[0016] Where x ik represents the state of the i-th device in the k-th time period, p ik represents the maximum profit that the i-th device can obtain in the k-th time period, c ik W represents the maximum consumption that the i-th device can achieve in the k-th time period. kIndicates the pressure in the gas network during the kth time period, W normal Indicates the pressure of the gas pipe network in a completely balanced state; α k represents the penalty factor for the pressure imbalance in the pipeline network in the kth time period; n represents the number of devices;

[0017] (4) Build a reinforcement learning agent model, which obtains action a through the state, updates the reinforcement learning agent model until a scheduling is completed, records the state, action, and reward of the scheduling process, updates the network parameters of the reinforcement learning agent model, and improves the reward through iterative training;

[0018] (5) After the iterative training is completed, the reinforcement learning agent model is applied to the test set, and the change process of the pipeline network pressure is visualized to ensure that the reinforcement learning agent model is safe and reliable;

[0019] (6) Save the reinforcement learning agent model and use the trained reinforcement learning agent model to directly optimize the scheduling of the gas system.

[0020] Furthermore, in step (1), the gas consuming devices in the gas system are divided into two categories. One category is a device that performs on-off selection of gas consumption, i.e., consumes all the gas provided, or consumes no gas; the other category is a device with a valve to adjust the gas consumption, which consumes gas between 0 and c. ik Continuous changes between

[0021] Assume that there are m devices of the first type, and their range of action is x ik ∈{0,1}, i=1,2,...,m; k=1,2,...,N; there are nm devices of the second type, and their range of action is x jk ∈[0, 1], j=m+1, m+2,...,n; k=1, 2,...,N.

[0022] In step (2), the state of the gas network model is represented by the state of each device at the current moment and the actual network pressure. By providing the current network pressure to the reinforcement learning agent model, it is enabled to have the ability to predict and control the network pressure to maintain balance and increase revenue.

[0023] In step (4), the reinforcement learning neural network completes a scheduling process, specifically including the following steps:

[0024] (4-1) First initialize the network parameters θ of the strategy 0 ,θ k is the parameter obtained in the previous training. Each time an iteration is performed, θ k Update and interact with the environment to obtain a set of state-action pairs, β is dynamically adjusted according to the KL divergence, and the advantage function is estimated using the proximal policy optimization formula

[0025] (4-2) The critic network learns to estimate the value of the current strategy And according to the current policy parameterization The expected sum of future discounted rewards is calculated

[0026] (4-3) Actor network learning is based on θ π The parameterized random policy π aims to take actions with the highest probability to maximize the sum of future rewards; therefore, the policy is given by θ π Parameterized and generates a probability distribution of the set of available actions at time t, which is expressed as:

[0027]

[0028] Where R represents the reward function evaluated by taking action a at state s and time t, and E represents the mathematical expectation;

[0029] (4-4) The parameters are updated by calculating the timing differential error TD-ERROR, which is expressed as follows:

[0030]

[0031] (4-5) The Tanh function is used for activation. The expression of the Tanh function is as follows:

[0032]

[0033] (4-6) The Adam optimization algorithm is used to optimize the cumulative loss and iteratively update the weights of the neural network based on the training data, so as to design independent adaptive learning rates for different parameters.

[0034] In step (4-1), the proximal strategy optimization formula is expressed as:

[0035]

[0036] Where, represents the objective function of optimization, β represents the penalty coefficient, KL(θ,θ k ) is used to measure θ and θ k degree of similarity.

[0037] In step (4-2), calculate the future discount reward The formula is expressed as:

[0038]

[0039] Among them, s tis the state at time t, R t It is from s t Convert to s t+1 The reward received at time t, T represents the total number of scheduled moments, Y is the discount factor, where 0<Y≤1, and E represents the mathematical expectation of future discounted rewards.

[0040] In step (4-3), during training, according to the probability output of the policy network, a set of available actions A is selected. t Middle pair action a t Sampling is performed to introduce some randomness into the chosen actions to encourage exploration; during testing, the action with the highest probability is chosen instead.

[0041] Preferably, in step (5), the trained reinforcement learning agent model is verified in a pre-generated test set, the total benefit is calculated, and a curve of the change of the pipeline network pressure during the test is drawn to verify the control effect of the model on the pipeline network pressure balance.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The dynamic scheduling method for a gas system based on improved proximal strategy optimization proposed in the present invention makes improvements based on the differences in actual gas consumption devices, so that the algorithm can solve the mixed problem of 0 / 1 consumption devices and non-0 / 1 consumption devices.

[0044] 2. The present invention addresses the problem of pressure imbalance in actual gas pipeline networks, with the goal of maximizing consumption benefits and minimizing pipeline network fluctuations. By improving the proximal strategy optimization, the model is trained. Using the trained model, a scheduling optimization strategy can be efficiently obtained, providing certain guidance for actual gas scheduling.

[0045] 3. The dynamic scheduling method for the gas system based on improved proximal strategy optimization proposed in the present invention shows, according to the pressure change curve, that the variation range of the pipeline network pressure does not exceed the upper and lower limits. It can be seen that the balance of the pipeline network pressure can be effectively achieved, thereby improving the safety of the scheduling process.

[0046] 4. The dynamic scheduling method for the gas system based on improved proximal strategy optimization proposed in the present invention has the advantages of short solution time and good solution effect. For production scenarios with high real-time scheduling strategies, a trained network model can be used to obtain a scheduling solution.

[0047] 5. The dynamic scheduling method for gas systems based on improved proximal strategy optimization proposed in the present invention is based on deep reinforcement learning. It can train the network on small-scale problems and directly migrate the trained network to large-scale scheduling problems, effectively solving large-scale optimization scheduling problems and obtaining good solution performance, thereby improving the adaptability of the scheduling optimization strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a topological diagram of a gas system pipe network in an embodiment of the present invention;

[0049] Figure 2 This is a structural diagram of the reinforcement learning agent model built in an embodiment of the present invention;

[0050] Figure 3 This is a training curve diagram of the improved proximal strategy optimization algorithm in an embodiment of the present invention;

[0051] Figure 4 This is a pressure change curve diagram of the improved proximal strategy optimization algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0053] The following takes the gas system in a certain oil refinery as an example to provide a detailed description of the establishment of a dynamic scheduling model for the gas system and the simulation research.

[0054] like Figure 1 As shown in FIG, a dynamic scheduling method for a gas system based on improved proximal strategy optimization mainly includes:

[0055] Step 1: Determine the production plan interval and the devices that produce and consume gas, and establish a gas pipeline network model.

[0056] In this embodiment, the initial pressure of the gas network is 100KPa, with upper and lower limits of 110KPa and 90KPa respectively; the unit planning interval is [0, T], T=30, and the gas production in each time period is y k , k=1,2,...,N,the gas consumption is c k , k = 1, 2, ..., N, there are n = 5 gas-consuming devices in total, of which m = 2 are first-type devices and nm = 3 are second-type devices.

[0057] Step 2: According to the initial values ​​of gas production and gas network pressure, the initial state of the gas network model is determined and iteratively updated on this basis.

[0058] In this embodiment, the initial state is formed by combining the state of each device at the current moment and the actual pipe network pressure. In the initial state, each production device is about to produce gas according to the production plan of the first time period, and the consumption device is inactive. s1 = [tank1, tank2, ..., tank n, W1], where tank1=tank2=…=tank n =0, W1=W normal +input1, input1 represents the methane gas input into the production device in the first time period.

[0059] Step 3: Based on the optimization objective function, a reward function is constructed to evaluate the quality of the action in the current state. The reward function is represented by the income generated by the gas consumption device and the pressure balance of the gas network. The formula is as follows:

[0060]

[0061] (4) The reinforcement learning agent model obtains action a through the state and updates the model until a scheduling is completed. The state, action, and reward of the scheduling process are recorded, and the network parameters are updated. After a certain number of iterations, the reward is increased.

[0062] In this embodiment, the model structure of the reinforcement learning agent model (actor-critic network) is as follows: Figure 2 As shown, the relevant parameter information is: hidden layers: 3, number of hidden layer neurons: 128, actor network learning rate: 5e-5, critic network learning rate: 1e-3, number of iterations: 2000 times, through a three-layer fully connected neural network and activated by the Tanh function, to obtain actions, update the state, and increase the total reward.

[0063] (5) After a certain number of iterations, the model is applied to the test set and the change process of the pipeline network pressure is visualized.

[0064] like Figure 3 As shown, in this embodiment, a total of 2000 iterative training and learning are performed. According to the training curve, it can be seen that the algorithm quickly rises and converges in a relatively short period of time. It can be seen that the scheduling method proposed in the present invention can effectively realize the dynamic scheduling of the gas pipe network system, and its training efficiency is high. After the training, the performance of the reinforcement learning intelligent agent model is stable, and higher benefits can be obtained while ensuring the basic balance of the gas pipe network pressure. It has good reliability and practicality.

[0065] like Figure 4 As shown, in this embodiment, according to the pressure change curve, it can be seen that when the algorithm is running in the test set, the variation range of the pipeline network pressure does not exceed the upper and lower limits. It can be seen that the improved proximal strategy optimization algorithm can effectively achieve the balance of pipeline network pressure and improve the safety of the scheduling process.

[0066] In addition, the average profit of the algorithm running on 30 test sets reached 751. While ensuring the pressure balance of the pipeline network, it was also able to fully utilize the gas generated by the production equipment, achieving positive returns and fully verifying the effectiveness of the scheduling.

[0067] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A gas system dynamic scheduling method based on improved proximal strategy optimization, characterized in that: include: (1) According to the scheduling optimization process of the gas system, the production plan interval, gas production equipment and gas consumption equipment are determined, and the gas pipeline network model is established; (2) Determine the initial state of the gas network model based on the initial values ​​of gas production and gas network pressure, and iteratively update it on this basis; (3) Based on the optimization objective function, a reward function is constructed to evaluate the quality of the action in the current state. The reward function is represented by the income generated by the gas consumption device and the pressure balance of the gas network. The formula is as follows: Where x ik represents the state of the i-th device in the k-th time period, p ik represents the maximum profit that the i-th device can obtain in the k-th time period, c ik W represents the maximum consumption that the i-th device can achieve in the k-th time period. k Indicates the pressure in the gas network during the kth time period, W normal Indicates the pressure of the gas pipe network in a completely balanced state; α k represents the penalty factor for the pressure imbalance in the pipeline network in the kth time period; n represents the number of devices; (4) Build a reinforcement learning agent model, which obtains action a through the state, updates the reinforcement learning agent model until a scheduling is completed, records the state, action, and reward of the scheduling process, updates the network parameters of the reinforcement learning agent model, and improves the reward through iterative training; (5) After the iterative training is completed, the reinforcement learning agent model is applied to the test set, and the change process of the pipeline network pressure is visualized to ensure that the reinforcement learning agent model is safe and reliable; (6) Save the reinforcement learning agent model and use the trained reinforcement learning agent model to directly optimize the scheduling of the gas system.

2. The gas system dynamic scheduling method based on improved proximal strategy optimization according to claim 1 is characterized in that: In step (1), the gas consuming devices in the gas system are divided into two categories. One category is a device that selects the on / off value of gas consumption, that is, it consumes all the gas provided, or consumes no gas; the other category is a device with a valve to adjust the gas consumption, and the gas consumed is between 0 and c ik Continuous changes between There are m devices of the first type, and their range of action is x ik ∈{0,1},i=1,2,...,m;k=1,2,...,N;There are nm devices of the second type, and their range of action is x jk ∈[0,1], j=m+1,m+2,…,n; k=1,2,…,N.

3. The method for dynamic scheduling of a gas system based on improved proximal strategy optimization according to claim 1, characterized in that: In step (2), the state of the gas network model is represented by the state of each device at the current moment and the actual network pressure. By providing the current network pressure to the reinforcement learning agent model, it is enabled to have the ability to predict and control the network pressure to maintain balance and increase revenue.

4. The method for dynamic scheduling of a gas system based on improved proximal strategy optimization according to claim 1, characterized in that: In step (4), the reinforcement learning neural network completes a scheduling process, specifically including the following steps: (4-1) First initialize the network parameters θ of the strategy 0 ,θ k is the parameter obtained in the previous training. Each time an iteration is performed, θ k Update and interact with the environment to obtain a set of state-action pairs, β is dynamically adjusted according to the KL divergence, and the advantage function is estimated using the proximal policy optimization formula (4-2) The critic network learns to estimate the value of the current strategy And according to the current policy parameterization The expected sum of future discounted rewards is calculated (4-3) Actor network learning is based on θ π The parameterized random policy π aims to take actions with the highest probability to maximize the sum of future rewards; therefore, the policy is given by θ π Parameterized and generates a probability distribution of the set of available actions at time t, which is expressed as: Where R represents the reward function evaluated by taking action a at state s and time t, and E represents the mathematical expectation; (4-4) The parameters are updated by calculating the timing differential error TD-ERROR, which is expressed as follows: (4-5) The Tanh function is used for activation. The expression of the Tanh function is as follows: (4-6) The Adam optimization algorithm is used to optimize the cumulative loss and iteratively update the weights of the neural network based on the training data, so as to design independent adaptive learning rates for different parameters.

5. The method for dynamic scheduling of a gas system based on improved proximal strategy optimization according to claim 4 is characterized in that: In step (4-1), the proximal strategy optimization formula is expressed as: Where, represents the objective function of optimization, β represents the penalty coefficient, KL(θ,θ k ) is used to measure θ and θ k degree of similarity.

6. The method for dynamic scheduling of a gas system based on improved proximal strategy optimization according to claim 4, characterized in that: In step (4-2), calculate the future discount reward The formula is expressed as: Among them, s t is the state at time t, R t It is from s t Convert to s t+1 The reward received at time t, T represents the total number of scheduled moments, γ is the discount coefficient, where 0<γ≤1, and E represents the mathematical expectation of future discounted rewards.

7. The method for dynamic scheduling of a gas system based on improved proximal strategy optimization according to claim 4, characterized in that: In step (4-3), during training, according to the probability output of the policy network, a set of available actions A is selected. t Middle pair action a t Sampling is performed to introduce some randomness into the chosen actions to encourage exploration; during testing, the action with the highest probability is chosen instead.

8. The method for dynamic scheduling of a gas system based on improved proximal strategy optimization according to claim 1, characterized in that: In step (5), the trained reinforcement learning agent model is verified in a pre-generated test set, the total benefit is calculated, and a curve of the change in pipeline network pressure during the test is drawn to verify the control effect of the model on the pipeline network pressure balance.

Images