Oxygen system scheduling optimization method based on deep reinforcement learning mixed action space

Abstract

The present application provides an oxygen system scheduling optimization method based on deep reinforcement learning hybrid action space, comprising the following steps: oxygen system scheduling optimization; drl-based method; results and analysis. The present application is based on deep reinforcement learning, and proposes an HAC algorithm for solving the oxygen system scheduling problem. The proposed algorithm extends the actor network of the AC algorithm, and designs an additional segmented reward function according to the market time-of-use electricity price. The HAC algorithm solves the mixed problem of discrete variables and continuous variables, and realizes energy saving and emission reduction while improving economic benefits. Experimental results show that the HAC algorithm places the scheduling problem in a hybrid action space, avoiding the problem of suboptimal action selection. Compared with the traditional algorithm, the HAC algorithm significantly improves the convergence and accuracy of the algorithm, making the oxygen system have higher oxygen utilization rate and economic benefits.

Description

Technical Field

[0001] This invention relates to oxygen system scheduling optimization, specifically to an oxygen system scheduling optimization method based on a deep reinforcement learning hybrid action space. Background Technology

[0002] Oxygen is an indispensable energy-consuming raw material in the steel manufacturing process. Frequent fluctuations in oxygen consumption and imbalances in oxygen supply and demand can easily lead to low oxygen utilization efficiency and energy waste, increasing production costs. Therefore, optimizing the scheduling of the oxygen system is crucial for steel companies to improve their economic efficiency.

[0003] The oxygen system scheduling problem in steel enterprises involves optimizing the operation of the entire system and the balance between oxygen supply and demand. It includes both continuous variables such as oxygen quantity and time, and discrete variables such as unit operation. This type of discrete-continuous mixed-variable scheduling optimization problem is highly complex and challenging. Air separation units play a crucial role in the oxygen system, and in recent years, many scholars both domestically and internationally have conducted extensive research on their production scheduling. To address the requirement of frequent load changes in multiple air separation units, Zhou et al. established a mixed-integer linear program (MILP) model for process scheduling, proposing an optimal scheduling strategy under separate modes to determine unit operation modes and optimizing the overall profit margin over a certain time period. Mitra et al. divided the operating states of air separation units into different modes and used the convex hull method to establish a deterministic mixed-integer linear program (MILP) model in discrete time, achieving high solution speed even with high model complexity. However, air separation units exhibit strong coupling and nonlinearity, which traditional linear programming methods cannot address. Therefore, Zou et al. proposed a robust control strategy combining a loosely coupled model with control variable priority settings, utilizing incremental load balancing calculations to handle nonlinearity and reduce energy consumption during production scheduling. Pattison and Baldea studied the optimal design of air separation units operating at variable capacity under variable electricity prices, proposing a new process modeling framework oriented towards pseudo-transient equations and combining the developed model with a time-relaxation-based optimization algorithm. Morgan T. Kelley et al. proposed a data-driven approach, using historical closed-loop operating data and an autoregressive model with extra inputs (ARX) for scheduling optimization in industrial air separation units. Each piece of equipment in an air separation unit also influences the overall system scheduling; therefore, studying the operating modes of individual equipment is crucial for overall system scheduling. For the nonlinear behavior of the thermally integrated air separation tower (HIASC) in an air separation unit, Fu and Liu designed a series of nonlinear fluctuation model predictive control schemes to achieve optimal scheduling objectives. Cao et al. used particle swarm optimization to optimize the scheduling of the liquefier, reducing power consumption and thus production costs.

[0004] The aforementioned studies all revolve around air separation equipment, optimizing the production scheduling process through improvements in air separation equipment production methods, primarily focusing on scheduling optimization for discrete variables. However, scheduling optimization for continuous variables such as oxygen demand, production, and dissipation is also crucial for energy conservation and emission reduction, and related scholars have conducted relevant research on scheduling optimization problems for these continuous variables. For the oxygen demand optimization problem in the multi-converter steelmaking stage of steel enterprises, Xu et al. established an optimal production scheduling model with the objectives of minimizing oxygen demand fluctuations and minimizing the penalty for casting start-up time deviations. To solve this model, a hybrid genetic algorithm combining variable neighborhood search (HGAVNS) was employed. Combining the fitted regression relationship between oxygen system production and electricity consumption, Han et al. established a nonlinear programming model for oxygen system scheduling and used the particle swarm optimization (PSO) algorithm to solve the proposed scheduling model considering electricity costs. To address the fluctuations in supply and demand caused by uncertainties in the manufacturing process, Jiang et al. proposed an optimal oxygen allocation strategy considering uncertain demand. They designed a two-stage robust optimization (TSRO) model based on a budget-constrained uncertainty set, mainly consisting of a Gaussian process-based time series model to predict the demand range of the continuous process and a capacity-constrained scheduling model to generate multi-scenario demand for the discrete process. Zhang et al. established a mixed-integer linear programming model for oxygen system scheduling to reduce pipeline pressure fluctuations caused by oxygen supply and demand imbalances, improving system safety while reducing oxygen emission rates. The above studies conducted scheduling research based on actual production and demand. Scheduling during production has a lag. To solve scheduling problems in real time, Han et al. applied a granular computation-based model to predict oxygen / nitrogen demand and developed a MILP-based optimization model for oxygen allocation. Zhang et al. developed a model-based decision support system that can minimize oxygen emission based on short-term oxygen demand forecasts, providing a timely and responsive solution to adjust all variables involved in the supply side. However, when using traditional methods to deal with problems involving discrete or continuous variables, there is still no scheduling method that explicitly addresses whether the variables are continuous or discrete to solve practical scheduling problems. Most methods are selected based on the scheduling optimization objective. Therefore, for complex problems involving mixed variables in the oxygen system of steel enterprises, traditional solution methods still need further improvement.

[0005] Deep reinforcement learning (RL) combines the perceptual capabilities of deep learning with the decision-making abilities of reinforcement learning, enabling it to handle complex scheduling problems involving high-dimensional variables. From the perspective of temporal difference update methods, deep reinforcement learning is divided into value function algorithms and policy gradient algorithms. Value function algorithms require sampling of actions and are used to handle discrete actions. Deep Q-Learning (DQN) is a classic deep reinforcement learning algorithm based on value functions. Bernd Waschneck et al. applied the DQN algorithm to industrial production scheduling to achieve Industry 4.0, and Wu et al. proposed an energy management strategy for hybrid buses based on deep Q-learning. However, real-world scheduling problems are complex and diverse, and the traditional DQN algorithm cannot solve all scheduling problems, necessitating improvements to the DQN algorithm. Luo et al. designed a novel training framework based on double DQN (DDQN) to solve the dynamic multi-objective flexible job shop scheduling problem with new task insertion. Ren et al. used a prediction-based Dueling-DDQN algorithm to optimize a home energy management system. For a factory reconfigurable manufacturing system, the Dueling-DDQN algorithm can be used to derive scheduling strategies. Policy gradient algorithms directly use policy networks to search for actions and can be used to handle continuous action scenarios. Commonly used algorithms include the deep deterministic policy gradient (DDPG) algorithm, and improved versions of DDPG such as the TD3 (Twin Delayed Deep Deterministic) algorithm and the PPO (Proximal Policy Optimization) algorithm. For continuous action spaces, many scholars have used the above algorithms to optimize the scheduling of different systems. Wang et al. used the DDPG algorithm to make decisions on the charging and discharging of electric vehicles in an electric vehicle cluster (CD) system, effectively reducing user charging costs. For virtual power plants, Guo et al. used the TD3 algorithm to schedule the charging and discharging of electric vehicles, while Zhou et al. used the TD3 algorithm to solve the energy management strategy for hybrid electric vehicles. Furthermore, the PPO algorithm was used to find the optimal strategy for job shop scheduling, and the DPPO algorithm was used to solve the economic scheduling problem of combined heat and power systems. The Actor-Critic algorithm, a combination of value function algorithm and policy gradient algorithm, allows for single-step updates and has significant advantages. Mao et al. proposed an Actor-Critic algorithm with a feedforward neural network to solve the taxi scheduling problem, while Ying et al. used the Actor-Critic algorithm to solve the subway train scheduling problem with limited vehicle flow.Therefore, deep reinforcement learning algorithms can be used to solve practical scheduling problems involving discrete or continuous variables, but there is no specific algorithm to solve the oxygen system scheduling problem involving mixed variables.

[0006] In summary, existing research mostly focuses on discrete variables related to unit operating characteristics to optimize the scheduling of the entire system's equipment operation, or only considers the oxygen production or consumption within the system to optimize the oxygen supply and demand balance. No scholar has combined equipment operation with oxygen supply and demand balance for holistic optimization scheduling, formulating a reasonable scheduling scheme to simultaneously control oxygen quantity and equipment operation. Furthermore, existing deep reinforcement learning algorithms cannot solve the scheduling problem of steel oxygen systems that simultaneously include discrete and continuous variables. Therefore, this invention designs a scheduling scheme for steel enterprise oxygen systems based on an improved deep reinforcement learning algorithm. Based on the actual scheduling needs of the oxygen system, and considering both equipment operating states such as equipment start-up / shutdown and oxygen production, a Hybrid Action Evaluation (HAC) algorithm is proposed. Addressing the characteristics of the hybrid action space, the actor network is extended within the Actor-Critic algorithm framework, subdividing the action space into discrete and continuous action spaces. Considering the coupling relationship between discrete and continuous actions, an association matrix is ​​constructed to schedule the overall oxygen system. Summary of the Invention

[0007] This invention provides an oxygen system scheduling optimization method based on a deep reinforcement learning hybrid action space to solve the problems in the prior art.

[0008] The technical problem solved by this invention is achieved by the following technical solution:

[0009] This invention provides a method for optimizing oxygen system scheduling based on a deep reinforcement learning hybrid action space, comprising the following steps:

[0010] Step 1: Optimization of Oxygen System Scheduling

[0011] 1.1 Identifying the Oxygen System Scheduling Optimization Problem: The oxygen system of an iron and steel enterprise is divided into three subsystems: oxygen generation system, storage system, and utilization system; the operation mode and production level of each piece of equipment in the system should be arranged to meet the oxygen demand of users, improve oxygen utilization, and maximize profits.

[0012] 1.2 Optimization Objectives

[0013] 1.3 Constraints

[0014] Step 2: DRL-based method

[0015] Step 3: Results and Analysis

[0016] To verify the effectiveness and superiority of the HAC algorithm for oxygen system scheduling optimization, we first present the comparative experimental results of scheduling using the HAC algorithm and actual production, and then analyze the proposed algorithm structure.

[0017] 3.1 Algorithm Verification and Analysis

[0018] The hybrid actor-critic algorithm uses a neural network to fit the reinforcement learning policy function and the state-action value function. It has three neural networks: a discrete actor network, a continuous actor network, and a critic network. The number of hidden layers in the neural network is set to 2, with 200 neurons in each layer. The activation function of the hidden layers is the ReLU function with a negative slope of 0.01. The output layer is a fully connected layer. The hyperparameters are selected using common deep learning techniques and then adjusted through trial and error based on actual training data.

[0019] 3.2 Algorithm Structure Analysis and Comparison

[0020] An additional reward function was set to consider the impact of peak and off-peak electricity consumption on the overall scheduling result. To verify its rationality, the original algorithm and the algorithm without additional reward were used to optimize the scheduling of the oxygen system. The optimized strategies were analyzed and calculated, and their respective economic indicators were compared according to the proposed objective function. Without the additional reward function, the difference in electricity consumption between peak and off-peak hours was small, and the overall economic benefits were consistently lower than those of the original algorithm. However, after scheduling with the original algorithm, the electricity consumption of machines during peak hours was lower than that during off-peak hours, and the overall economic benefits were higher, averaging 24% higher than those without the additional reward function.

[0021] In addition, the hybrid actor-critic algorithm establishes an correlation matrix D, which indicates the coupling relationship between the two actor networks; to verify the necessity of setting the correlation matrix, the HAC algorithm is used to solve the oxygen system scheduling strategy after removing the correlation matrix.

[0022] 3.3 Comparison of Results from Different Algorithms

[0023] The hybrid actor-critic algorithm was compared with the MP-DQN, P-DQN, and PADDPG algorithms. The hybrid actor-critic algorithm's reward curve converged around 10,000 rounds and had the highest reward value. The MP-DQN and P-DQN algorithms had longer convergence times and lower reward values ​​than the HAC algorithm. Although the PADDPG algorithm had a faster convergence time, its average reward value was too low to meet the scheduling optimization requirements.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. This invention studies the scheduling problem of an oxygen system in a steel enterprise, constructing a comprehensive economic index combining liquid oxygen sales revenue, equipment electricity costs, and oxygen release losses. Considering that traditional scheduling methods cannot simultaneously address the start-up and shutdown of system equipment and frequent changes in oxygen demand, resulting in low economic efficiency, this invention proposes the HAC algorithm based on deep reinforcement learning to solve the oxygen system scheduling problem. The proposed algorithm extends the actor network of the AC algorithm, designing an additional piecewise reward function based on market time-of-use electricity prices. The HAC algorithm solves the mixed problem of discrete and continuous variables, achieving energy conservation and emission reduction while improving economic efficiency. Experimental results show that the HAC algorithm places the scheduling problem in a mixed action space, avoiding suboptimal action selection. Compared with traditional algorithms, the HAC algorithm significantly improves convergence and accuracy, resulting in higher oxygen utilization and economic benefits for the oxygen system.

[0026] 2. The oxygen system scheduling optimization problem in steel enterprises focuses on the scheduling process of oxygen production, storage, and use. Existing scheduling strategies only consider single continuous or discrete variables. However, in actual production, it is necessary to consider the mixed problem involving both discrete and continuous variables such as equipment start-up and shutdown, oxygen production, and dissipation. Therefore, this paper proposes a hybrid actor-critic (HAC) algorithm based on deep reinforcement learning. This algorithm subdivides the traditional action space into a discrete action space and a continuous action space, and improves the Q-function to achieve accurate evaluation of the scheduling strategy. To address the non-uniqueness in the number of continuous actions corresponding to discrete actions, and the dimensional inconsistency of continuous actions corresponding to different discrete actions, an association matrix is ​​constructed. This allows the continuous actor network to have independent action parameter outputs, avoiding the Q-value obtained by adopting different mixed actions from depending on unrelated continuous actions, generating false gradients, and leading to suboptimal action selection. Experiments are conducted using actual data from steel enterprises to verify the algorithm. The results show that the proposed scheduling algorithm improves the reward value while increasing the convergence speed. Compared with traditional algorithms, it has better overall performance. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is an oxygen system according to an embodiment of the present invention;

[0029] Figure 2This is a diagram illustrating the reinforcement learning structure of an embodiment of the present invention.

[0030] Figure 3 This is a hybrid actor-critic structure according to an embodiment of the present invention;

[0031] Figure 4 This invention employs a hybrid actor-critic algorithm in its embodiments.

[0032] Figure 5 This refers to the reward curve during the training process in an embodiment of the present invention.

[0033] Figure 6 This is the result of optimizing the objective function in an embodiment of the present invention;

[0034] Figure 7 This is a comparison of the external sales of liquid oxygen in embodiments of the present invention;

[0035] Figure 8 A comparison of power consumption in embodiments of the present invention;

[0036] Figure 9 For comparison of emission rates in embodiments of the present invention;

[0037] Figure 10 For comparison of additional reward functions in embodiments of the present invention;

[0038] Figure 11 This is a Gantt chart showing the operation of the equipment in an embodiment of the present invention;

[0039] Figure 12 The output of different equipment in the embodiments of the present invention;

[0040] Figure 13 This invention provides a comparison of reward curves during the training process of different algorithms in embodiments of the invention.

[0041] Figure 14 This is a comparison of the economic indicators of different algorithms in the embodiments of the present invention. Detailed Implementation

[0042] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0043] Embodiments of the present invention provide a technical solution: an oxygen system scheduling optimization method based on a deep reinforcement learning hybrid action space, comprising the following steps:

[0044] Step 1: Problem Description

[0045] 1.1 Oxygen System Scheduling Optimization Problem

[0046] like Figure 1As shown, the oxygen system of a steel enterprise can be divided into three subsystems: oxygen generation system, storage system, and usage system.

[0047] The oxygen generation system consists of an oxygen generator and an oxygen compressor. Air is compressed to a certain pressure by an air compressor and then oxygenated by the oxygen generator. The generated gaseous oxygen and liquid oxygen are separated by cryogenic distillation. The gaseous oxygen is compressed by the oxygen compressor and transported to the user system through the oxygen pipeline network, while the liquid oxygen is stored in liquid tanks. When there is an excess of high-pressure oxygen, the amount of gas stored in the oxygen pipeline network and tanks will gradually increase, leading to a gradual increase in pipeline pressure. To avoid safety hazards caused by pipeline pressure exceeding the safe range, gaseous oxygen can be liquefied and stored in liquid tanks through a liquefaction device, or excess gas can be discharged. When high-pressure oxygen is insufficient, liquid oxygen can be converted into gaseous oxygen using a vaporization device to meet the needs of the oxygen user system. The oxygen user system includes steel plants, ironmaking plants, other users, and external users. Gaseous oxygen enters the steel plant for converter blowing and the ironmaking plant for blast furnace oxygen-enriched smelting. Liquid oxygen is mainly sold to external users to generate external profits.

[0048] In the steel manufacturing process, the oxygen demand fluctuates frequently due to the constant changes in steel products and their corresponding production processes, easily leading to an imbalance between production and consumption. Therefore, optimizing the oxygen system's scheduling to address this supply-demand imbalance is essential. Under normal circumstances, oxygen dissipation is directly proportional to oxygen production and inversely proportional to the adjustment rate; thus, oxygen dissipation also affects the overall system's supply-demand balance. Furthermore, the storage capacity of the oxygen system's storage devices determines the amount of oxygen used and stored during supply-demand imbalances. The conversion of liquid and gaseous oxygen using liquefiers and vaporizers also plays a regulatory role in maintaining the overall supply-demand balance. Therefore, the challenge of oxygen system scheduling is how to arrange the operating modes and production levels of various devices within the system to meet user oxygen needs while improving oxygen utilization and maximizing profits.

[0049] 1.2 Optimization Objectives

[0050] Typically, the optimization objectives for oxygen system scheduling include minimizing oxygen emission rates and maximizing economic benefits. This invention considers energy consumption, oxygen emission rates, and economic benefits to make optimal scheduling decisions regarding the operation modes and production levels of the main equipment in the oxygen system.

[0051] Taking the economic indicator F of the oxygen system as the scheduling optimization objective, and considering the revenue from the sale of liquid oxygen, the power consumption of each piece of equipment, and the oxygen release loss, its mathematical expression is:

[0052] F = J p -E-λ·J r (1)

[0053] Among them, J PE is the revenue from the sale of liquid oxygen from the oxygen system, and J is the cost of electricity used by the equipment. r It is the loss from oxygen release from the system.

[0054] Revenue from the sale of liquid oxygen from the oxygen system J P for:

[0055]

[0056] Where f is the unit price of liquid oxygen This refers to the amount of liquid oxygen sold externally. Let be the liquid oxygen output of the i-th oxygen generator at time t. Let be the liquefaction amount at time t of the j-th liquefier. Let Δt be the vaporization rate of the g-th vaporizer at time t. i Δt j Δt g α represents the operating time of each oxygen generator, liquefier, and vaporizer. i β j and ρ g These represent whether the oxygen generator, liquefier, and vaporizer are turned on or off, respectively, and can be discretized into 0-1 variables:

[0057]

[0058] β j and ρ g With α i similar.

[0059] The equipment electricity cost E is:

[0060]

[0061] Let be the gaseous oxygen output of the i-th oxygen generator at time t. and The electricity costs for the oxygen generator, liquefier, and vaporizer are respectively, H i (t), C j (t) and B g E(t) is the function relating the output and power consumption of each oxygen generator, liquefier, and vaporizer, and E(t) is the actual electricity price.

[0062] Oxygen system release loss J r for:

[0063]

[0064] Oxygen unit price, Oxygen dissipation It represents the oxygen storage at time t. Oxygen consumption at time t.

[0065] 1.3 Constraints

[0066] The constraints of the oxygen system scheduling problem include four parts: material balance constraints, the ratio of gaseous oxygen to liquid oxygen production constraints, the operational constraints of each piece of equipment, and the equipment activation constraints.

[0067] (3) Material balance constraints

[0068] At time t, the material balance of the oxygen pipeline network and the liquid oxygen storage tank can be expressed as:

[0069]

[0070]

[0071] in, Let represent the total amount of liquid oxygen, gaseous oxygen, vaporization, and liquefaction at time t, respectively. It represents the storage capacity of the liquid tank at time t.

[0072] (4) Oxygen and liquid oxygen confinement

[0073] The gaseous oxygen and liquid oxygen produced have the following relationship:

[0074]

[0075] (3) Operating constraints of each device

[0076] The oxygen system involves various types of equipment, such as oxygen generators, liquefiers, vaporizers, and liquid oxygen storage devices, all of which have an operating range.

[0077] 1) Oxygen generator operating constraints

[0078] Each oxygen concentrator has an upper and lower limit for oxygen production, which are approximately 80% and 105% of its production capacity, respectively. The oxygen output of the i-th oxygen concentrator is... The relevant constraints are:

[0079]

[0080] and Let be the lower and upper limits of the oxygen production capacity of the i-th oxygen generator.

[0081] 2) Liquefier operating constraints

[0082] Similar to oxygen generators, the load range of liquefaction units is generally between 60% and 100%.

[0083] The operating constraints of the liquefier are:

[0084]

[0085] These are the lower and upper limits of the liquefaction capacity of the j-th liquefier, respectively.

[0086] 3) Vaporizer operating constraints

[0087] The vaporization capacity of the vaporizer must not exceed the maximum load of the equipment.

[0088]

[0089] denoted as the upper and lower limits of the gasification volume of the g-th gasification unit.

[0090] 4) Liquid oxygen storage tank volume constraints

[0091] For equipment safety reasons, the actual capacity of the storage tank is limited to 10% of the rated capacity.

[0092] 95%. The volume of the liquid tank is... The constraint is:

[0093]

[0094] S LO2 (max), S LO2 (min) represents the upper and lower limits of the liquid oxygen storage tank volume, respectively.

[0095] (4) Equipment opening constraints

[0096] The activation action can only be performed when the equipment is idle, and the number of activated devices cannot exceed the total number of devices.

[0097] ∑α i ≤n (13)

[0098] ∑β j ≤m (14)

[0099] ∑ρ g ≤l (15)

[0100] The oxygen system scheduling optimization problem in steel enterprises involves a mixture of discrete and continuous decision variables, making the scheduling problem complex and involving numerous variable constraints, which traditional methods cannot solve. However, deep reinforcement learning algorithms have advantages in handling mixed variable problems, tailored to specific variable types. Therefore, this invention proposes a mixed variable optimization scheduling method based on improved reinforcement learning in Section 3 to solve the oxygen system scheduling optimization problem in steel enterprises.

[0101] Step 2: DRL-based method

[0102] This step first introduces the relevant knowledge of reinforcement learning and establishes a Markov model involving the specific characteristics of the oxygen system. It also elaborates on the specific content of the proposed HAC algorithm for solving the scheduling optimization problem.

[0103] 2.1 Basic Principles of Reinforcement Learning

[0104] Reinforcement learning is a machine learning method in which an intelligent agent takes certain actions to maximize cumulative rewards during continuous interaction with an unknown environment. It can be represented by a Markov Decision Process (MDP). Specifically, it can be represented by a quintuple.<s,a,p,r,γ> Let represent , where s represents all states, a represents all actions the agent can take, p represents the state transition probability, r represents the immediate reward for a specific state and action, and γ represents the return discount.

[0105] like Figure 2 As shown, the state at time t is s. t The agent, according to policy π t To perform a certain action a in the environment t Based on the state transition probability p, the state transitions to the new state s at time t+1. t+1 The environment provides feedback rewards to the agent in the new state. t+1 The agent is based on the new state s. t+1 and feedback rewards t+1 At time t+1, a new action is executed, and the environment is iteratively interacted with through feedback signals.

[0106] 2.2 Oxygen System Scheduling Optimization Markov Model

[0107] Based on the aforementioned reinforcement learning theory, a Markov model for oxygen system scheduling optimization is established, with the objective of maximizing economic indicators:

[0108] 6) State s

[0109] The observed oxygen system states include oxygen demand, oxygen emissions, liquid oxygen sales, liquid oxygen storage, and the current scheduling cycle. Therefore, state s can be represented as:

[0110]

[0111] 7) Action a

[0112] At time t, the operation of the oxygen system consists of two parts: discrete actions and continuous actions. The discrete actions include the activation of each oxygen generator, vaporizer, and liquefier. The continuous actions include the activation duration of each device, the amount of oxygen and liquid oxygen produced by the oxygen generation system, the amount of oxygen liquefied by the liquefier in the oxygen storage system, and the amount of oxygen vaporized by each vaporizer. Therefore, the mixed action can be represented as:

[0113]

[0114] 8) Reward r

[0115] The objective of oxygen system scheduling decisions is to maximize the system's economic indicators. This objective aligns with the classic reinforcement learning objective of maximizing rewards; therefore, the immediate reward obtained by the oxygen system can be represented as:

[0116] r1(s,a)=J p (s,a)-E(s,a)-λ·J r (s,a) (18)

[0117] Because electricity prices vary at different times, and dispatching objectives are affected by electricity prices, an additional reward function is added.

[0118]

[0119] Z D Z P Z represents the average electricity consumption during off-peak hours and the average electricity consumption during peak hours, respectively. D Z P It can be expressed as follows:

[0120]

[0121]

[0122] Therefore, the reward function r can be expressed as a weighted sum of the objective function and the additional reward value, i.e.

[0123] r=(1-τ)r1+τr2 (22)

[0124] in As training progresses, the weight of economic rewards gradually decreases, while the rewards gradually increase, with a focus on maximizing the optimization objective.

[0125] 9) Strategy

[0126] As mentioned above, the oxygen system determines its behavior based on its own state. In Markov decision processes, the policy is defined as the transition probability of the oxygen system from the observed state to the action state, and its policy is defined as...

[0127] π(a|s)=P[a t =a∣s t =s]s∈S,a∈A (23)

[0128] 10) Returns

[0129] State Action Value Function Q π(s,a) represents the expected cumulative reward after performing action a in state s. The state-action value function Bellman equation is expressed as:

[0130] Qπ(s t ,a t )=Eπ[r t +γQπ(s t+1 ,a t+1 )∣s t ,a t ] (twenty four)

[0131] The Bellman equation shows that the action value of the current state depends only on the current reward / penalty value and the action value of the next state, and can be solved iteratively.

[0132] Solving for the optimal strategy of the oxygen system is equivalent to solving for the optimal state-action value function, i.e.:

[0133] π * =argmaxQ π (s,a) (25)

[0134] The corresponding Bellman equation is:

[0135]

[0136] To enable oxygen systems to perceive complex environmental conditions and construct optimal action strategies, a deep reinforcement learning (DRL) approach is proposed to address the oxygen system scheduling optimization problem. Most existing DRL algorithms only consider discrete actions or only continuous actions. However, the scheduling system of this invention includes both discrete and continuous actions. In scenarios with a mixed discrete-continuous action space, traditional reinforcement learning methods often fail to effectively solve the practical problems of oxygen systems by discretizing the approximate mixed space or relaxing it into a continuous set.

[0137] 2.3 Hybrid Actor-Critic System Architecture

[0138] Based on existing actor-critic algorithms, this invention proposes a hybrid actor-critic algorithm for the hybrid action space scheduling optimization problem. For example... Figure 3 As shown, the hybrid actor-critic algorithm framework has two actor networks: a discrete actor network and a continuous actor network. The Q-network is used as the critic network to evaluate the quality of the strategies. Based on the current state s of the oxygen system, the discrete actor network provides discrete actions k, and then the correlation matrix D is derived from a series of discrete actions. k The state s and the correlation matrix D kA continuous actor network provides a series of actions. After performing the mixed actions, the critic network calculates the Q-value based on the input state and the actions performed, and then makes a decision on the actions.

[0139] The proposed HAC algorithm parameterizes the hybrid action space, which has a set of discrete actions A. d =[K]={k1,k2,…,k K}, where each k has a corresponding continuous action parameter x. k ∈X k Then the mixed action space can be represented as:

[0140] A={((k,x k )∣x k ∈X k ,k∈[K])} (27)

[0141] Traditional algorithms parameterize the continuous actions corresponding to discrete actions, assuming that the number of such continuous actions is unique. However, in practical oxygen scheduling problems, multiple different discrete actions correspond to different and unequal continuous actions, resulting in each parameterized discrete variable having a corresponding multidimensional continuous variable with unequal dimensions. To address this issue, the algorithm's critic network input is improved to enhance its accuracy.

[0142] Based on the actual problem, the continuous action parameter x belongs to the discrete action k. k Quantity m K Not equal to 0 and m1≠m2≠…≠m K Construct the correlation matrix D

[0143]

[0144] Matrix D is ∑m K ×∑m K A 3D square matrix, where each element a i'j' The value is

[0145]

[0146] Using state s, discrete action k, and action parameter matrix For the input of the critic network, where k = (k1, k2, ... k) K For each discrete action k, a forward pass is performed once. When the discrete action is k K hour, Combining deep learning with Bellman equations to update the critic network, the formula for the Bellman equation used in the hybrid action space is as follows:

[0147]

[0148] The parameter is θ Q The deep neural network will Q(s,k,x) k ) is represented as Q(s,k,x) k ;θ Q Mini-batch samples are sampled from the empirical replay pool B1, and the Q-network is updated using the method of minimizing the mean square Bellman error.

[0149]

[0150] where y=r+γmax Q(s′,k(s′),x k (s′); θ Q ).

[0151] In discrete actor networks, parameters θ are used. d Deep Neural Network Representation Strategy Although the strategy is not fully differentiable, we believe it can still be obtained by following... We use the policy gradient to train our policy. Define... for

[0152]

[0153] in For a parameter that maps from the state space to the discrete action space, The function. Therefore, the loss function of a single sample in state s with respect to discrete actions is:

[0154]

[0155] The policy gradient is

[0156]

[0157] Update network parameters for discrete actions using the policy gradient method.

[0158] For continuous actor networks, the parameter is θ. c Deep Neural Network Representation Strategy The network is updated using the same loss function as the discrete actor network, and its policy gradient is:

[0159]

[0160] Because the action parameter matrix was input The gradient of the unfolded motion parameters is given by

[0161]

[0162] Transform into

[0163]

[0164] Therefore, the HAC algorithm completely eliminates the influence of the neural network weights on the irrelevant action parameters of the input, and solves the problem of multidimensional mismatch between the parameterized continuous action and its corresponding discrete action.

[0165] 2.4 HAC Algorithm Steps

[0166] The hybrid actor-critic algorithm proposed in this invention solves practical problems involving hybrid action spaces, improving the algorithm's solution speed and accuracy. The algorithm flowchart is as follows: Figure 4 As shown.

[0167] The specific algorithm steps are as follows:

[0168] 11. Input step size Parameter ò, mini-batch size M, probability distribution ξ

[0169] 12. Initialize the neural network weights and Initialize experience pool B1

[0170] 13. For t=1 to T do

[0171] 14. Calculate the parameter vector of continuous motion. and discrete actions

[0172] 15. Based on selecting motion from discrete action space and continuous action space respectively

[0173]

[0174] 16. After performing the action, store (s, a, r, s') in experience pool B1.

[0175] 17. Sample from experience pool B1 in mini-batch size M.

[0176] 18. Define the target y

[0177]

[0178] 19. Calculate the stochastic gradient. and

[0179] 20. Update weights

[0180]

[0181]

[0182]

[0183] End for

[0184] Step 3: Results and Analysis

[0185] To verify the effectiveness and superiority of the HAC algorithm proposed in this invention for oxygen system scheduling optimization, the following was conducted: Figure 1 The data from the oxygen system of a steel plant in Jiangsu Province are used for experiments. This section first presents the comparative experimental results of scheduling using the HAC algorithm and actual production, and analyzes the structure of the proposed algorithm. Finally, the algorithm proposed in this invention is compared with other related algorithms. The experimental details are shown below.

[0186] 3.1 Algorithm Verification and Analysis

[0187] The hybrid actor-critic algorithm uses a neural network to fit the reinforcement learning policy function and state-action value function. It employs three neural networks: a discrete actor network, a continuous actor network, and a critic network. The hidden layers are set to two layers, each with 200 neurons. The activation function for the hidden layers is a ReLU function with a negative slope of 0.01, and the output layer is a fully connected layer. Hyperparameters are selected using common deep learning techniques, and then adjusted through trial and error based on actual training data. The specific training parameters are given in Table 1.

[0188] Table 1 Training parameters

[0189]

[0190] The operating parameters of the oxygen system equipment are shown in Table 2. The electricity price used in this invention is the peak-valley price stipulated by large power plants. Peak hours are 8:00-11:00 and 16:00-21:00, with a price of 0.7564 yuan / kWh; normal hours are 6:00-8:00 and 21:00-22:00, with a price of 0.5257 yuan / kWh; and valley hours are 22:00-6:00, with a price of 0.3110 yuan / kWh. The unit cost prices of gaseous oxygen and liquid oxygen are fixed at 0.35 yuan / ton and 470 yuan / ton, respectively.

[0191] Table 2 Equipment Operating Parameters

[0192]

[0193] First, the convergence of the algorithm was verified using the parameters shown in Table 1 for training. Figure 5 The figure shows the average reward value and maximum reward value curves every 10,000 cycles during the oxygen system training process. Continuous testing of the proposed algorithm shows that it converges after approximately 10,000 cycles, obtaining the optimal scheduling strategy. Because the oxygen system is initially unfamiliar with the environment, the reward value obtained in the early stages is relatively small. As the training process continues, the oxygen system continues to interact with the environment to gain experience and eventually achieves convergence.

[0194] After offline training of the algorithm, the obtained network parameters are saved for real-time scheduling of the oxygen system. The algorithm's scheduling results are compared with actual production scheduling results, focusing on the proposed objective function for the oxygen system and its three sub-components. Figure 6 The figure shows a comparison between the actual production process and the objective function (economic indicators) after algorithm scheduling. The comparison includes the amount of liquid oxygen sold, equipment power consumption, and gaseous oxygen released in the objective function. Figure 7 , 8 As shown in Figure 9.

[0195] This invention calculates the proposed objective function for the oxygen system based on actual production data and compares the result with the objective function after algorithm scheduling. For example... Figure 6 The optimization results of the objective function show that after scheduling optimization using the HAC algorithm, the average economic benefit of the HAC algorithm is 9274 yuan / h, while the average economic benefit of actual production is 7580 yuan / h. Therefore, the economic benefit of the HAC algorithm after scheduling is about 2000 yuan / h higher than that of actual production.

[0196] Combination Figure 7-9 Further analysis of the scheduling results reveals that, in actual production, the oxygen generator remains on and will not be shut down unless there are unforeseen circumstances, while the vaporizer and liquefaction unit operate less frequently, resulting in a relatively stable and low oxygen production volume. The HAC algorithm, however, schedules equipment start-up and shutdown more often than in actual production. Figure 7-8 As shown, the average liquid oxygen sales volume after scheduling using the algorithm of this invention is significantly more than twice that of the actual production process. Although the power consumption after scheduling using the HAC algorithm will fluctuate greatly and exceed the power consumption of the actual production process at certain times, it is generally lower than the power consumption of the actual production process. Figure 9 As shown, the algorithm's maximum average release rate is only 0.34%, which is lower than the actual release rate in the production process. Oxygen loss is near zero in multiple time periods, and energy consumption is significantly reduced while meeting scheduling requirements. Therefore, the comparative analysis of the above experimental results shows that using the HAC algorithm for oxygen system scheduling optimization improves the economic efficiency of the oxygen system, significantly increases the amount of liquid oxygen sold, further reduces electricity costs and oxygen release rate, and avoids energy waste.

[0197] 3.2 Algorithm Structure Analysis and Comparison

[0198] Because this invention incorporates an additional reward function in its algorithm reward function design, it considers the impact of peak and off-peak electricity consumption on the overall scheduling result. To verify its rationality, such as... Figure 10 As shown, the original algorithm and an algorithm without additional rewards are used to optimize the oxygen system. The optimized strategies are analyzed and calculated, and their respective economic indicators are compared according to the proposed objective function. The figure shows that without additional reward functions, the difference in power consumption between peak and off-peak hours is small, and the overall economic efficiency is consistently lower than that of the original algorithm. However, after using the original algorithm, the power consumption of machines during peak hours is lower than during off-peak hours, resulting in higher overall economic efficiency, averaging 24% higher than the efficiency without additional reward functions.

[0199] Furthermore, the hybrid actor-critic algorithm establishes an incidence matrix D, indicating the coupling relationship between the two actor networks. To verify the necessity of setting the incidence matrix, the HAC algorithm is used to solve for the oxygen system scheduling strategy after removing the incidence matrix. The algorithm solution results are as follows: Figure 11 and Figure 12 As shown, combined with Figure 11 Equipment operation Gantt chart and Figure 12 The output of different equipment shows that oxygen generator 1 maintained a constant oxygen production rate during its three start-stop cycles, and its output was not zero when it was not running. Other equipment also exhibited a mismatch between its operating rate and output. Therefore, setting up an association matrix effectively improved the algorithm's accuracy.

[0200] 3.3 Comparison of Results from Different Algorithms

[0201] like Figure 13 As shown, to further verify the feasibility and superiority of the algorithm proposed in this invention, the hybrid actor-critic algorithm is compared with the MP-DQN, P-DQN, and PADDPG algorithms. The reward curve of the hybrid actor-critic algorithm converges at approximately 10,000 rounds and has the highest reward value. The MP-DQN and P-DQN algorithms have longer convergence times and lower reward values ​​than the HAC algorithm. While the PADDPG algorithm has a faster convergence time, its average reward value is too low to meet the scheduling optimization requirements. Therefore, the algorithm proposed in this invention is more stable, has higher convergence and accuracy, and better meets the production scheduling optimization requirements. Figure 14 The paper presents a bar chart of the economic indicators for each 10-hour period for the scheduling optimization strategies solved by four different algorithms. It can be seen from the chart that the algorithm proposed in this invention has the highest economic benefits.

[0202] To visually illustrate the differences in performance among the four scheduling methods—MP-DQN, P-DQN, PADDPG, and HAC algorithms—oxygen emission and economic indicators were statistically analyzed. As shown in Table 3, the economic indicator after scheduling based on the HAC algorithm was 274,125 yuan, representing a 21.5% improvement over the MP-DQN algorithm, a 45% improvement over the P-DQN scheduling method, and an 83% improvement over the PADDPG algorithm. Therefore, the scheduling method based on the HAC algorithm proposed in this invention has significant advantages in improving the economic efficiency of oxygen systems in steel enterprises. Furthermore, in terms of gas emission, the HAC algorithm resulted in the lowest gas emission, reducing gas release losses.

[0203] Table 3. Oxygen Release and Economic Benefits under Different Dispatch Schemes

[0204]

[0205] The algorithm proposed in this invention operates in a hybrid action space, considering not only the output of each device but also its start-up and shutdown actions, as different devices consume different amounts of energy during start-up and shutdown. Table 2 shows the operating parameters of each device and their start-up costs. Frequent start-ups and shutdowns increase energy consumption. Using a reasonable scheduling strategy to arrange device start-ups and shutdowns can reduce costs and increase economic benefits. Table 4 compares the total start-up cost and total production cost of devices using four different algorithms. According to the data in the table, the scheduling cost of the HAC algorithm is approximately 63% of that of the PADDPG algorithm, and approximately 67% and 72% of that of the MP-DQN and P-DQN algorithms, respectively. Therefore, the HAC algorithm results in lower electricity costs, achieving the goal of cost savings.

[0206] Table 4. Cost Comparison of Different Algorithms

[0207]

[0208] Based on the comparison results of the different algorithms, the HAC algorithm yields the highest scheduling reward value and the fastest solution speed within the same time frame. After scheduling, the equipment production cost and start-up cost are lower than other solution algorithms, and the economic benefits are significantly increased within a certain range. Therefore, the oxygen system scheduling optimization method using the algorithm proposed in this invention improves economic efficiency while reducing energy consumption.

[0209] in conclusion

[0210] This invention studies the scheduling problem of an oxygen system in a steel enterprise, constructing a comprehensive economic index combining liquid oxygen sales revenue, equipment electricity costs, and oxygen emission losses. Considering that traditional scheduling methods cannot simultaneously address the start-up and shutdown of system equipment and frequent changes in oxygen demand, resulting in low economic efficiency, this invention proposes the HAC algorithm based on deep reinforcement learning to solve the oxygen system scheduling problem. The proposed algorithm extends the actor network of the AC algorithm, designing an additional piecewise reward function based on market time-of-use electricity prices. The HAC algorithm solves the mixed problem of discrete and continuous variables, achieving energy conservation and emission reduction while improving economic efficiency. Experimental results show that the HAC algorithm places the scheduling problem in a mixed action space, avoiding suboptimal action selection. Compared with traditional algorithms, the HAC algorithm significantly improves convergence and accuracy, resulting in higher oxygen utilization and economic benefits for the oxygen system.

[0211] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing oxygen system scheduling based on a deep reinforcement learning hybrid action space, characterized in that: Includes the following steps: Step 1: Economic indicators of oxygen systems The mathematical expression for the scheduling optimization objective is: in, E represents the revenue from the sale of liquid oxygen from the oxygen system, and E represents the cost of electricity used by the equipment. This is a loss due to oxygen release from the system; Oxygen system liquid oxygen sales revenue for: in Liquid oxygen unit price, The amount of liquid oxygen sold externally; Let be the liquid oxygen output of the i-th oxygen generator at time t. Let be the liquefaction amount at time t of the j-th liquefier. Let g be the vaporization rate at time t of the g-th vaporizer; , , The operating time for each oxygen generator, liquefier, and vaporizer. , and These represent whether the oxygen generator, liquefier, and vaporizer are turned on or off, respectively, and can be discretized into 0-1 variables: and and similar; Equipment electricity costs for: Let be the gaseous oxygen output of the i-th oxygen generator at time t. , and The electricity costs for the oxygen generator, liquefier, and vaporizer are respectively. , and This is a function relating the output and power consumption of each oxygen generator, liquefier, and vaporizer. This refers to the actual electricity price; Oxygen system release loss for: Oxygen unit price, Oxygen dissipation It represents the oxygen storage at time t. Oxygen consumption at time t; Step 2: Establish a Markov model for oxygen system scheduling optimization; 1) Status The oxygen system status includes oxygen demand, oxygen emissions, liquid oxygen sales, liquid oxygen storage, and the current scheduling cycle; therefore, the status... It can be represented as: It represents the liquid storage capacity of the tank at time t; 2) Actions At time t, the operation of the oxygen system consists of two parts: discrete actions and continuous actions. Discrete actions include the activation of each oxygen generator, vaporizer, and liquefier. Continuous actions include the activation duration of each device, the amount of oxygen and liquid oxygen produced by the oxygen generation system, the amount of oxygen liquefied by the liquefier in the oxygen storage system, and the amount of vaporized oxygen by each vaporizer. The mixed actions can be represented as: 3) Rewards The immediate reward obtained by the oxygen system can be represented as: Electricity prices vary at different times of day, therefore an additional reward function is added. in , These represent the average electricity consumption during off-peak hours and the average electricity consumption during peak hours, respectively. , It can be expressed as follows: reward function This can be expressed as a weighted sum of the objective function and the additional reward value, i.e. in , 4) Strategy The strategy is defined as the transition probability of the oxygen system from the observation state to the action state, and it is expressed as: 5) Returns State action value function Indicates the state Execute action The expected cumulative return after the state-action value function, expressed by the Bellman equation, is as follows: The Bellman equation shows that the action value of the current state depends only on the current reward / penalty value and the action value of the next state, and can be solved iteratively. Solving for the optimal strategy of the oxygen system is equivalent to solving for the optimal state-action value function, that is: The corresponding Bellman equation is: Step 3: Constructing the Hybrid Action Space The hybrid actor-critic algorithm framework has two actor networks: a discrete actor network and a continuous actor network. The Q-network is used as the critic network to evaluate the merits of the strategies. This is based on the current state of the oxygen system. Discrete actor networks provide discrete actions. Then, the correlation matrix is ​​derived based on a series of discrete actions. , by state and the correlation matrix A continuous actor network provides a series of actions. After performing the mixed actions, the critic network calculates the Q-value based on the input state and the actions performed, and makes decisions on the actions. The proposed HAC algorithm parameterizes the hybrid action space, which has a set of discrete actions. Each k has corresponding continuous action parameters. Then the mixed action space can be represented as: Continuous action parameters belonging to discrete action k quantity Not equal to 0 and , Step 4: Construct the association matrix matrix for A 3D square matrix, where each element... The value is by state Discrete Actions and action parameter matrix For the input of the critic network, where , For each discrete action Perform a forward pass when the discrete action is hour, ; Step 5: Combining deep learning with Bellman equations to update the critic network, the formula for the Bellman equation used in the hybrid action space is as follows: The parameters are as follows: Deep neural networks will Represented as From the experience replay pool We sample mini-batch samples and update the Q-network using the method of minimizing the mean square Bellman error: in ; In discrete actor networks, the parameters used are Deep Neural Network Representation Strategy The strategy is not fully differentiable, but it can be determined by following... We use the policy gradient to train our policy; define for in For a parameter that maps from the state space to the discrete action space, The function; therefore the state is The loss function for a single sample with respect to discrete actions is: The policy gradient is Update network parameters for discrete actions using the policy gradient method; For continuous actor networks, the parameters used are: Deep Neural Network Representation Strategy The network is updated using the same loss function as the discrete actor network, and its policy gradient is: Because the action parameter matrix was input The gradient of the unfolded action parameters is given by Transform into 。 2. The oxygen system scheduling optimization method based on deep reinforcement learning hybrid action space according to claim 1, characterized in that: The constraints of the oxygen system scheduling problem include four parts: material balance constraints, gaseous oxygen to liquid oxygen production ratio constraints, equipment operation constraints, and equipment start-up constraints. (1) Material balance constraints At time t, the material balance of the oxygen pipeline network and the liquid oxygen storage tank can be expressed as: in, , , , Let represent the total amount of liquid oxygen, gaseous oxygen, vaporization, and liquefaction at time t, respectively. This represents the storage capacity of the liquid tank at time t. (2) Oxygen and liquid oxygen confinement The gaseous oxygen and liquid oxygen produced have the following relationship: (3) Operating constraints of each piece of equipment The oxygen system involves various types of equipment, such as oxygen generators, liquefiers, vaporizers, and liquid oxygen storage devices, and all have an operating scope. 1) Oxygen generator operating constraints Each oxygen concentrator has an upper and lower limit for oxygen production, which are approximately 80% and 105% of its production capacity, respectively; the oxygen output of the i-th oxygen concentrator is... The relevant constraints are: and Let $i$ be the lower and upper limits of the oxygen production capacity of the $i$-th oxygen generator. 2) Liquefier operating constraints Similar to oxygen generators, the load range of liquefaction units is generally between 60% and 100%; therefore, the operating constraints of the liquefaction unit are: , These are the lower and upper limits of the liquefaction capacity of the j-th liquefier, respectively; 3) Vaporizer operating constraints The vaporization capacity of the vaporizer must not exceed the maximum load of the equipment. , The upper and lower limits of the gasification volume of the g-th gasification unit; 4) Liquid oxygen storage tank volume constraints For equipment safety reasons, the actual capacity of the storage tank is limited to 10% to 95% of its rated capacity; the tank volume is... Then the constraint is: , These are the upper and lower limits of the liquid oxygen storage tank volume, respectively. (4) Equipment opening constraints The activation action can only be performed when the equipment is idle, and the number of activated equipment cannot exceed the total number of equipment. 。

Images