An optimization scheduling method and system based on a deep deterministic policy gradient algorithm

Abstract

This invention discloses an optimization scheduling method and system based on a deep deterministic policy gradient algorithm, relating to the field of artificial intelligence technology. The method includes: collecting and preprocessing equipment data; constructing an environmental dynamic model; constructing an optimization scheduling model; and constructing and training a deep deterministic policy gradient algorithm model. This invention considers the storage safety of gas holders, the gas consumption safety of heating furnaces, and the safe operation of generator sets, ensuring the safe and stable operation of the system. It employs a deep deterministic policy gradient algorithm combined with deep neural networks and Actor-Critic networks to make the scheduling process more intelligent. Combined with time-of-use pricing, it ensures that the power generation of generator sets at different times better meets the electricity demand of the steel industry, reducing the amount of electricity purchased from the grid, maximizing the value of generated electricity, and utilizing blast furnace gas and converter gas more efficiently, reducing gas emissions, improving gas utilization efficiency, and reducing environmental pollution.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an optimization scheduling method and system based on a deep deterministic policy gradient algorithm. Background Technology

[0002] In the steel industry, coal gas power generation plays a crucial role in the energy management and economic efficiency of the entire steel plant. Coal gas produced by equipment such as blast furnaces and converters can serve as an important raw material for sintering machines, heating furnaces, and their own operation, reducing raw material costs. A portion of this coal gas is used to generate a large amount of electricity, which can reduce the amount of electricity purchased from the grid in the steel industry, improve the economic efficiency of the steel industry, and reduce the amount of coal gas emitted, thus reducing environmental pollution. The optimized scheduling of the coal gas system involves the flow and pressure control of coal gas generating equipment, storage equipment, consumption equipment, coal gas power generation equipment, and the coal gas system pipeline network. In addition, the stability and reliability of coal gas power generation are greatly affected by the quality and supply stability of coal gas, which may affect power generation efficiency and grid stability. Therefore, coal gas generating equipment and other equipment need to meet a certain dynamic balance. The complexity of this system lies in the fact that it not only needs to consider the coordinated management between material flow and energy flow, but also needs to cope with the uncertainties and variability in the production process. Traditional optimization methods, such as mathematical analytical methods (including linear programming, nonlinear programming, and dynamic programming), are effective in handling small-scale problems, but they often fail to meet the requirements when facing large-scale, dynamically changing real-world industrial problems due to limitations in computation speed and real-time performance.

[0003] In view of this, reinforcement learning methods have shown their superiority in dealing with complex, dynamic and highly uncertain systems. Among them, the deep deterministic policy gradient algorithm can handle both continuous and discrete action spaces. It can introduce an experience replay mechanism to store the experience gained by the agent in the interaction with the environment in an experience pool, and randomly draw small batches of data for learning during training to reduce the correlation between data. In terms of actions, noise can be added to encourage exploration, prevent overfitting, and find the optimal solution. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is: how to ensure computation speed and optimal computation results when processing large-scale, dynamically changing data.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: an optimization scheduling method based on a deep deterministic policy gradient algorithm, comprising: collecting and preprocessing device data; constructing an environmental dynamic model; constructing an optimization scheduling model; and constructing and training a deep deterministic policy gradient algorithm model.

[0007] As a preferred embodiment of the optimization scheduling method based on the deep deterministic strategy gradient algorithm described in this invention, the equipment data includes the gas generation and consumption of the blast furnace and converter, the gas consumption of each sintering machine, heating furnace, ladle, and gas generator set in the sintering process, the gas storage of the gas holder, the energy conversion efficiency of each generator set, and the normal operating output range and gas flow rate per unit time of equipment such as the blast furnace, converter, sintering furnace, heating furnace, ladle, and each generator set.

[0008] As a preferred embodiment of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in this invention, the preprocessing includes data denoising, data imputation, and data standardization. The data denoising is achieved by using a convolutional neural network to identify and eliminate random fluctuations and outliers in the data. The data imputation is achieved by filling in missing data through interpolation and mean replacement. The data standardization is achieved by scaling the data to give it a uniform dimension and distribution.

[0009] As a preferred embodiment of the optimization scheduling method based on deep deterministic policy gradient algorithm described in this invention, the construction of the environmental dynamic model includes establishing the relationships between the blast furnace and converter and their respective storage devices, consumption devices, and gas power generation devices, as expressed as:

[0010] Gas b (t)=Gas bc,b (t)+Gas s,b (t)+Gas h,b (t)+Gas l,b (t)+Gas p,b (t)+Gas sgb (t)

[0011] Gas c (t)=Gas cc,c (t)+Gas gc,c (t)+Gas l,c (t)+Gas p,c (t)+Gas sgc (t)

[0012] Among them, Gas b (t) represents the amount of blast furnace gas produced by the blast furnace during time period t. Gas bc,b(t) represents the amount of blast furnace gas required for the blast furnace to operate during time period t. Gas s,b (t) represents the amount of blast furnace gas consumed by the sintering machine during time period t. Gas h,b (t) represents the amount of blast furnace gas consumed by the heating furnace during time period t. Gas l,b (t) represents the amount of blast furnace gas consumed in the ladle during time period t. p,b (t) represents the amount of blast furnace gas supplied to the generator set during time period t. Gas sgb (t) represents the blast furnace gas emission rate during time period t. c (t) represents the amount of converter gas produced by the converter during time period t. Gas cc,c (t) represents the amount of converter gas required for the converter to operate during time period t. gc,c (t) represents the amount of converter gas stored in the gas holder during time period t. l,c (t) represents the amount of converter gas required for the ladle during time period t. p,c (t) represents the amount of converter gas supplied to the generator set during time period t. Gas sgc (t) represents the amount of converter gas released during time period t.

[0013] As a preferred embodiment of the optimization scheduling method based on the deep deterministic strategy gradient algorithm described in this invention, when the sintering machine needs maintenance or the production of sinter needs to be reduced, the sintering machine needs to be stopped or put into operation. Therefore, the operating conditions of the sintering machine are divided into two cases, as follows:

[0014]

[0015] Where μ represents the two operating conditions of the sintering machine, when μ = 0, it means that the sintering machine stops running at time t, and when μ = 1, it means that the sintering machine starts running at time t; a is the amount of gas required for the sintering machine to start running.

[0016] The physical relationship between the converter gas volume in the gas holder and its input and output is expressed as follows:

[0017]

[0018] Gas gc (t+1)=Gas gc (t)+Gas gc,c (t)

[0019] in, This represents the amount of converter gas input at the gas holder inlet during time period t. Gas represents the converter gas output at the gas holder outlet during time period t. gc,c(t) represents the amount of converter gas remaining in the gas holder after input and output during time period t. gc (t) represents the amount of converter gas in the gas holder at time t. gc (t+1) represents the total amount of converter gas in the gas holder at time t+1.

[0020] As a preferred embodiment of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in this invention, the optimization scheduling model is based on the dynamic environmental model, with further added constraints. Upper and lower limits are constrained for the amount of converter gas entering and leaving the gas holder and the amount of gas present in the gas holder at the current moment.

[0021]

[0022]

[0023]

[0024] in, These represent the upper and lower limits of the converter gas input at the gas holder inlet.

[0025] These represent the upper and lower limits of converter gas output from the gas holder outlet. Gas gc This indicates the upper and lower limits of the converter gas that the gas holder can store;

[0026] Limits are imposed on the upper and lower limits of gas consumption in the heating furnace:

[0027]

[0028] in, This indicates the upper limit of the amount of gas consumed by the heating furnace. Gas h,b This indicates the lower limit of the amount of gas consumed by the heating furnace;

[0029] Constrain the upper and lower limits of the generator set's power:

[0030] P b (t)+P c (t)=P(t)

[0031]

[0032] Among them, P b (t) represents the amount of electricity generated by the generator set using blast furnace gas during time period t, P c P(t) represents the amount of electricity generated by the generator set using converter gas during time period t, P(t) is the total amount of electricity generated by the generator set during time period t, P is the upper limit of the generator set power, and P is the lower limit of the generator set power.

[0033] Limits are imposed on the number of times generator sets can be started and stopped, and on the gradient rate.

[0034]

[0035] -R down ≤P(t+1)-P(t)≤R up

[0036] Where B is the unit start-up and shutdown budget constraint, and C is... e R represents the cost of each start-up and shutdown of the unit, N is the number of start-ups and shutdowns allowed for the unit, and R is the cost of each start-up and shutdown. up Indicates the maximum ramp rate of the generator set, -R down This indicates the maximum downhill slope of the generator unit.

[0037] As a preferred embodiment of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in this invention, the deep deterministic policy gradient algorithm model includes defining a state space S, determining key variables affecting the gas system scheduling decision, and the state variables including the amount of blast furnace gas produced by the blast furnace, the amount of converter gas produced by the converter, the amount of blast furnace gas consumed by the blast furnace itself, the amount of converter gas consumed by the converter itself, the amount of blast furnace gas consumed by the sintering machine, heating furnace, ladle, and generator set, the amount of converter gas consumed by the ladle and generator set, the amount of converter gas in the converter gas holder at the current moment, the amount of blast furnace gas, the amount of converter gas released, and the time-of-use electricity price for each time period. The state is determined based on these quantities, specifically:

[0038] S = [Gas] b (t),Gas c (t),Gas bc,b (t),Gas cc,c (t),Gas s,b (t),Gas h,b (t),Gas l,b (t),

[0039] Gas l,c (t),Gas p,b (t),Gas p,c (t),Gas sg,b (t),Gas sg,c (t),Gas gc (t),Price(t)]

[0040] Where Price(t) represents the time-of-use electricity price for each period;

[0041] Define action space A to determine the key control actions that can be taken in the gas system scheduling. Action variables include the operating conditions of the sintering machine, the operating power of the generator set, and the input and output of converter gas in the gas holder. Based on these quantities, the actions are determined as follows:

[0042] Secondly, another objective of this invention is to provide an optimization scheduling system based on a deep deterministic policy gradient algorithm, comprising: a data acquisition and preprocessing module, an environment dynamic model construction module, an optimization scheduling model construction module, and a deep deterministic policy gradient algorithm model construction module;

[0043] The data acquisition and preprocessing module is used to acquire the required equipment data and perform preprocessing.

[0044] The environmental dynamics model building module is used to build a basic environmental dynamics model;

[0045] The optimized scheduling model construction module is used to impose constraints based on the dynamic environmental model;

[0046] The Deep Deterministic Policy Gradient Algorithm Model Building Module is used to build and train a deep deterministic policy gradient algorithm model.

[0047] Thirdly, a computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm as described above.

[0048] Fourthly, a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm as described above.

[0049] The beneficial effects of this invention are as follows: This invention provides an optimized scheduling method and system based on a deep deterministic policy gradient algorithm, which incorporates various beneficial controls, considering the storage safety of gas holders, the gas consumption safety of heating furnaces, and the safe operation of generator sets, ensuring the safe and stable operation of the system. By employing a deep deterministic policy gradient algorithm combined with deep neural networks and Actor-Critic networks, the scheduling process becomes more intelligent. Combined with time-of-use pricing, the power generation of generator sets at different times better meets the electricity demand of the steel industry, reducing the amount of electricity purchased from the grid, maximizing the value of generated electricity, and more efficiently utilizing blast furnace gas and converter gas. This reduces gas emissions, improves gas utilization efficiency, reduces environmental pollution, enhances the overall economic efficiency of the steel industry, and meets the requirements of low-carbon and environmental protection. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 The following is an overall flowchart of an optimization scheduling method based on a deep deterministic policy gradient algorithm, provided as an embodiment of the present invention.

[0052] Figure 2 A gas system structure diagram for an optimization scheduling method based on a deep deterministic policy gradient algorithm, provided as an embodiment of the present invention;

[0053] Figure 3 A structural diagram illustrating the reinforcement learning principle of an optimization scheduling method based on a deep deterministic policy gradient algorithm, provided in one embodiment of the present invention.

[0054] Figure 4 An Actor-Critic network structure diagram for an optimization scheduling method based on a deep deterministic policy gradient algorithm, provided as an embodiment of the present invention;

[0055] Figure 5 A time-of-use electricity price distribution diagram of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided in one embodiment of the present invention;

[0056] Figure 6 A gas holder location distribution diagram provided as an embodiment of the present invention for an optimization scheduling method based on a deep deterministic strategy gradient algorithm;

[0057] Figure 7 The image shows the training results of a deep deterministic policy gradient algorithm agent for an optimization scheduling method based on a deep deterministic policy gradient algorithm, as provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0059] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0060] Example 1

[0061] Reference Figures 1-4 As an embodiment of the present invention, an optimization scheduling method based on a deep deterministic policy gradient algorithm is provided, comprising:

[0062] S1: Collect device data and perform preprocessing;

[0063] Furthermore, the gas system is an independent and complete subsystem of a steel enterprise, including gas generation equipment, storage equipment, consumption equipment, and gas power generation equipment. On-site surveys were conducted to obtain data on the gas generation, storage, consumption, and power generation equipment from the steel plant. The data to be obtained includes the gas generation and consumption of blast furnaces and converters, the gas consumption of each sintering machine, heating furnace, ladle, and gas generator set in the sintering process, the gas storage capacity of gas holders, the energy conversion efficiency of each generator set, and the normal operating output range and gas flow rate per unit time of equipment such as blast furnaces, converters, sintering furnaces, heating furnaces, ladles, and generator sets.

[0064] It should be noted that the data acquired from the steel plant may contain noisy data due to data acquisition errors, sensor malfunctions, human error, etc. Therefore, before modeling, it is necessary to denoise and fill in missing values ​​in the acquired data, and to standardize the data. Specifically:

[0065] ① Data Denoising: For noisy data in the dataset, convolutional neural networks are used to identify and eliminate random fluctuations and outliers in the data, thereby improving data accuracy and reducing their impact on model training.

[0066] ② Data imputation: By using techniques such as interpolation and mean replacement, missing data is filled in to ensure the integrity of the dataset and the accuracy of the analysis.

[0067] ③ Data standardization: The data is scaled to give it a uniform dimension and distribution, so as to facilitate comparison and processing.

[0068] S2: Construct a dynamic environmental model;

[0069] Furthermore, building dynamic environmental models can provide real-time information about environmental changes, enabling the decision-making process to adapt to these changes and helping algorithms better understand and predict environmental changes, which is the foundation for effective decision-making.

[0070] It should be noted that data analysis is performed on the dataset obtained after preprocessing in step S1 to determine the initial conditions of the gas system model, etc. The physical relationships between equipment form the basis of system optimization scheduling, which determines the storage and utilization efficiency of the equipment, ensures the supply and demand balance of gas, and improves energy utilization efficiency. Based on this, the physical relationships between the gas generating equipment, storage equipment, consumption equipment, and gas power generation equipment of the gas system are established, and then an environmental dynamic model is constructed. The network structure relationships between the gas generating equipment, storage equipment, consumption equipment, and gas power generation equipment of the gas system are as follows: Figure 2 As shown.

[0071] Furthermore, relationships are established between the blast furnace and converter and their respective storage, consumption, and gas-fired power generation equipment. The blast furnace primarily supplies its own gas to its own consumption, sintering machine, heating furnace, ladle, and generator set, with some gas released as it cannot be fully utilized. The converter primarily supplies its own gas to its own consumption, gas holder, and a portion directly supplied or indirectly supplied to the ladle and generator set via the gas holder. Like the blast furnace gas, some converter gas is released as it cannot be fully utilized. The gas required by the ladle is mainly supplied by the blast furnace gas; if the blast furnace gas is insufficient, it is supplied by the converter gas. This is illustrated below:

[0072] Gas b (t)=Gas bc,b (t)+Gas s,b (t)+Gas h,b (t)+Gas l,b (t)+Gas p,b (t)+Gas sgb (t)(1)

[0073] Gas c (t)=Gas cc,c (t)+Gas gc,c (t)+Gas l,c (t)+Gas p,c (t)+Gas sgc (t) (1)

[0074] Among them, Gas b (t) represents the amount of blast furnace gas produced by the blast furnace during time period t. Gas bc,b(t) represents the amount of blast furnace gas required for the blast furnace to operate during time period t. Gas s,b (t) represents the amount of blast furnace gas consumed by the sintering machine during time period t. Gas h,b (t) represents the amount of blast furnace gas consumed by the heating furnace during time period t. Gas l,b (t) represents the amount of blast furnace gas consumed in the ladle during time period t. p,b (t) represents the amount of blast furnace gas supplied to the generator set during time period t. Gas sgb (t) represents the blast furnace gas emission rate during time period t. c (t) represents the amount of converter gas produced by the converter during time period t. Gas cc,c (t) represents the amount of converter gas required for the converter to operate during time period t. gc,c (t) represents the amount of converter gas stored in the gas holder during time period t (the value is negative if released), Gas l,c (t) represents the amount of converter gas required for the ladle during time period t. p,c (t) represents the amount of converter gas supplied to the generator set during time period t. Gas sgc (t) represents the amount of converter gas released during time period t.

[0075] It should be noted that the amount of blast furnace gas supplied to the sintering machine is allocated according to the number of sintering machines in operation. The sintering machine requires regular maintenance and repair to prevent malfunctions during operation. When the sinter storage volume is too high or too low, the sintering machine needs to be stopped or put into operation. Therefore, the operating conditions of the sintering machine are divided into two cases, which are illustrated below using piecewise functions:

[0076]

[0077] Where μ represents the two operating conditions of the sintering machine, when μ = 0, it means that the sintering machine stops operating at time t, and when μ = 1, it means that the sintering machine starts operating at time t; a is the amount of gas required for the sintering machine to start operating.

[0078] Furthermore, in this invention, the gas holder is one of the core devices for achieving the invention's objective. The gas storage characteristics of the gas holder directly affect the power generation value of the generator set and the amount of gas released. As a storage device for converter gas, the gas holder involves the input and output of converter gas. Due to the difference between the input and output amounts of converter gas, the amount of converter gas in the gas holder will continuously change. The physical relationship between these changes is established as follows:

[0079]

[0080] Gas gc (t+1)=Gasgc (t)+Gas gc,c (t) (3)

[0081] in, This represents the amount of converter gas input at the gas holder inlet during time period t. Gas represents the converter gas output at the gas holder outlet during time period t. gc,c (t) represents the amount of converter gas remaining in the gas holder after input and output during time period t. gc (t) represents the amount of converter gas in the gas holder at time t. gc (t+1) represents the total amount of converter gas in the gas holder at time t+1.

[0082] S3: Construct an optimized scheduling model;

[0083] Furthermore, after constructing the environmental dynamic model, an optimized scheduling model is built on it. The optimized scheduling model considers the dynamic characteristics and constraints of the gas system, improves the efficiency of gas use, reduces energy waste, and helps maintain the stability of the gas system. On the basis of the basic environmental dynamic model, further constraints are added to make the environmental model more complete and accurate.

[0084] It should be noted that the input and output volumes of gas at the inlet and outlet of the gas holder, as well as the upper and lower limits of the gas volume within the holder, are crucial. These constraints ensure that the gas holder operates under safe and stable working pressure, preventing safety hazards caused by excessively high or low pressure. They also reduce environmental pollution and improve energy efficiency. Therefore, these upper and lower limits are subject to constraints.

[0085]

[0086] in, These represent the upper and lower limits of the converter gas input at the gas holder inlet. These represent the upper and lower limits of converter gas output from the gas holder outlet. Gas gc This indicates the upper limit of converter gas that the gas holder can store, and this indicates the lower limit of converter gas that the gas holder can store.

[0087] Furthermore, excessive gas supply to the heating furnace can lead to uncontrolled furnace temperature, equipment damage, or even explosions, while insufficient gas supply can result in inadequate furnace temperature, affecting heating efficiency and causing a decline in product quality. Both excessive and insufficient gas consumption by the heating furnace cause energy waste and environmental pollution, increasing industrial production costs and safety risks. Therefore, the gas consumption of the heating furnace needs to be reasonably controlled, with upper and lower limits set:

[0088]

[0089] in, This indicates the upper limit of the amount of gas consumed by the heating furnace. Gas h,b This indicates the lower limit of the amount of gas consumed by the heating furnace.

[0090] It should be noted that maximizing the value of the electrical energy generated by the generator set is an important objective in this invention. Therefore, ensuring the safe operation of the generator set is extremely important. Reasonable upper and lower power limits can prevent equipment damage or failure due to excessive load, while also avoiding the impact on power generation efficiency and stability due to excessively low load. Therefore, determining a specific power operating range for the generator set and constraining its upper and lower limits ensures the safe operation of the equipment.

[0091] P b (t)+P c (t)=P(t)

[0092]

[0093] Among them, P b (t) represents the amount of electricity generated by the generator set using blast furnace gas during time period t, P c P(t) represents the amount of electricity generated by the generator set using converter gas during time period t, and P(t) is the total amount of electricity generated by the generator set during time period t. P represents the upper limit of the generator set's power, and P represents the lower limit of the generator set's power.

[0094] Furthermore, frequent start-ups and shutdowns accelerate unit wear and tear, reducing the unit's actual service life. Considering equipment lifespan loss and operational economics, certain limits need to be placed on the number of start-ups and shutdowns and the ramp rate of generator units.

[0095]

[0096] -R down ≤P(t+1)-P(t)≤R up (8)

[0097] Where B is the unit start-up and shutdown budget constraint, and C is... e R represents the cost of each start-up and shutdown of the unit, N is the number of start-ups and shutdowns allowed for the unit, and R is the cost of each start-up and shutdown. up -R represents the maximum ramp rate of the generator set within the time period t. down This represents the maximum downhill rate of the generator unit within the time period t.

[0098] S4: Construct and train a deep deterministic policy gradient algorithm model.

[0099] It should be noted that the model to be built is a deep reinforcement learning model, which requires interaction between the agent and the environment. Appropriate reward functions and penalty terms are set to guide the agent, enabling it to continuously learn in such goal-oriented interactions, ultimately learning the optimal policy. Figure 3 This is a diagram illustrating the mechanism of interaction between an agent and its environment, where the environment provides the agent with the state s at time t. t and reward r t The agent generates action a at time t through a policy. t The environment provides guidance through actions generated by the agent, and the environment then feeds back a new state s to the agent. t+1 and reward r t The training continues until the maximum number of iterations is reached. After the environment model is built, the next step is to build the intelligent agent and set the reward function and penalty term.

[0100] Furthermore, a state space S is defined, and the key variables affecting the gas system scheduling decision are identified. These key state variables are integrated into a state vector s. The state variables include the amount of blast furnace gas produced by the blast furnace, the amount of converter gas produced by the converter, the amount of blast furnace gas consumed by the blast furnace itself, the amount of converter gas consumed by the converter itself, the amount of blast furnace gas consumed by the sintering machine, the heating furnace, the ladle, and the generator set, the amount of converter gas consumed by the ladle and the generator set, the amount of converter gas in the converter gas holder at the current moment, the amount of blast furnace gas, the amount of converter gas released, and the time-of-use electricity price for each time period. The state vector s is determined based on these quantities. Specifically:

[0101] S = [Gas] b (t),Gas c (t),Gas bc,b (t),Gas cc,c (t),Gas s,b (t),Gas h,b (t),Gas l,b (t),

[0102] Gas l,c (t),Gas p,b (t),Gas p,c (t),Gas sg,b (t),Gas sg,c (t),Gas gc (t),Price(t)](9)

[0103] Where Price(t) represents the time-of-use electricity price for each period.

[0104] A state space defines the set of all possible states that an agent can observe and exist in, helping the agent understand the complexity and diversity of its environment.

[0105] Define the action space A, determine the key control actions that can be taken in the gas system scheduling, and integrate the key control actions into an action vector a. The action variables include the operating conditions of the sintering machine, the operating power of the generator set, and the input and output of converter gas in the gas holder. Based on these quantities, determine the action vector a. Specifically:

[0106]

[0107] The action space defines all possible actions an agent can take in each state, providing the agent with behavioral diversity. The action space and state space are interdependent and together form the basis for the agent's decision-making.

[0108] Construct a deep neural network that captures and learns the constraints in the physical model, ensuring sufficient constraints during model execution and guaranteeing accurate and reasonable training results. First, define an input layer, clearly defining the number of input data samples; name this layer InputLayer. Next, define a fully connected layer containing 100 neurons, the number of which is determined by the complexity of the problem; name this fully connected layer FC1. Then, define an activation function layer, choosing the ReLU function. Next, define a second fully connected layer containing 5 neurons; this layer will receive the output of the previous activation layer and perform further feature transformation; name this layer OutputLayer. Finally, define an output layer to output the prediction results; name this layer RegressionOutput.

[0109] Define various parameters and behaviors of the neural network during training to control the training process and optimize model performance.

[0110] The data is divided into training, validation, and test sets, with the following specific proportions:

[0111]

[0112] Where S train S represents the proportion of the training set to be partitioned. val S represents the proportion of the validation set partition. test This indicates the proportion of the test set partition.

[0113] Unlike deep neural networks, Actor-Critic networks generate a set of actions conforming to a probability distribution based on the state, evaluate these actions, and select an appropriate policy. It is a basic architecture of a deep deterministic policy gradient algorithm agent. The environment provides the Actor network with state information `s`, the Actor feeds back a set of actions `a` to the Critic network and the environment, the environment generates a corresponding reward `r` based on action `a`, and the Critic network provides feedback to the Actor network based on the reward, state information, and actions generated by the Actor network. Figure 4 As shown.

[0114] First, a Critic network is constructed, consisting of two input layers to receive state and action information, respectively. An array of multiple neural network layers is then built to process the common representation of actions and states. These layers are constructed in the following order:

[0115] Define a connected layer that combines the state and action inputs, and name this layer 'concat'; define a fully connected layer with 1000 neurons, and name this layer 'FC1'; define an activation function layer, and name this layer 'Relu1'; define another fully connected layer with 1000 neurons, and name this layer 'FC2'; define another activation function layer, and name this layer 'Relu2'; define a final fully connected layer to represent the value of the state-action pair, and name this layer 'CriticOutput'.

[0116] The learning rate is set to 0.001, and the gradient threshold is set to 1.

[0117] To build an Actor network, first construct an array containing multiple neural network layers, including:

[0118] Define an input layer for receiving state information, named 'observation'; define a fully connected layer with 200 neurons, named 'ActorFC1'; define an activation function layer, named 'ActorRelu1'; define a fully connected layer with 200 neurons, named 'ActorFC2'; define an activation function layer, named 'ActorRelu2'; define another fully connected layer with the number of neurons specified by numAct, named 'ActorFC3'; define a tanh activation layer that limits the output to a certain range, named 'ActorTanh'; finally, define a scaling layer that scales the output of the tanh layer to the actual range of the action, named 'ActorTanh'.

[0119] The learning rate is set to 0.01, and the gradient threshold is set to 1.

[0120] To create a reinforcement learning agent based on the deep deterministic policy gradient algorithm, first, a representation of a deterministic policy is created, which receives observations as input and outputs actions. Options for the agent based on the deep deterministic policy gradient algorithm are configured using rlDDPGAgentOptions, including sampling time, experience pool length, number of samples drawn from the experience pool, and discount factor. Noise parameters are set for the agent when selecting actions. Finally, a deep deterministic policy gradient algorithm agent is created.

[0121] A deep deterministic policy gradient algorithm agent is created, and a reward function and a penalty term are set to ensure that the agent is properly guided during training. The reward function and penalty term are shown in equations (13) to (18):

[0122] Reward1=P×Δt×Price(t) (12)

[0123]

[0124] Penalty2 = Gas sgb (14)

[0125] Penalty3 = Gas sgc (15)

[0126]

[0127] Reward=Reward1-λ1·Penalty1-λ2·Penalty2-λ3·Penalty3-λ4·Penalty4(17)

[0128] λ1, λ2, λ3, and λ4 are coefficients set for Penalty1, Penalty2, Penalty3, and Penalty4 respectively, based on the degree of punishment.

[0129] Example 2

[0130] One embodiment of the present invention provides an optimization scheduling system based on a deep deterministic policy gradient algorithm, comprising: a data acquisition and preprocessing module, an environment dynamic model construction module, an optimization scheduling model construction module, and a deep deterministic policy gradient algorithm model construction module; the data acquisition and preprocessing module is used to acquire and preprocess the required equipment data; the environment dynamic model construction module is used to construct a basic environment dynamic model; the optimization scheduling model construction module is used to impose constraints based on the environment dynamic model; and the deep deterministic policy gradient algorithm model construction module is used to construct and train a deep deterministic policy gradient algorithm model.

[0131] Example 3

[0132] One embodiment of the present invention differs from the previous two embodiments in that:

[0133] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0135] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0136] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0137] Example 4

[0138] See Figures 5-7 This invention provides an optimized scheduling method based on a deep deterministic policy gradient algorithm, as one embodiment of the present invention. To verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculations and simulation experiments.

[0139] The data collection interval is on the order of minutes, and the specific data to be collected is shown in Table 1:

[0140]

[0141]

[0142] Time-of-use electricity pricing, such as Figure 5 As shown, the time-of-use electricity price is lowest from 23:00 to 7:00, which is the off-peak period for electricity demand; the price is between peak and off-peak periods from 7:00 to 9:00 and from 12:00 to 17:00, which is the period of stable electricity demand; the price is highest from 9:00 to 12:00 and from 17:00 to 23:00, which is the period of peak electricity demand. Incentives based on reasonable electricity prices can help alleviate the peak-shaving problem of the power grid and balance the grid load.

[0143] Figure 6 The diagram shows the gas level in the gas holder during different time periods. It can be seen that the gas holder level gradually increases between 23:00 and 6:00 and between 13:00 and 15:00, while it gradually decreases between 6:00 and 13:00 and between 15:00 and 22:00. During periods of lower electricity prices, gas is stored in the gas holder, while during peak electricity prices, the stored gas is used. This reduces the need to purchase electricity from the grid during peak hours, helps with peak shaving for the power grid, and improves economic efficiency.

[0144] Figure 7 The graph shows the reward curve during the training process. As can be seen from the graph, the reward value of the reinforcement learning agent is very low at the beginning of training, but after about several dozen training sessions, the reward value gradually reaches its peak and tends to stabilize. This shows that deep reinforcement learning based on the deep deterministic policy gradient algorithm converges very quickly to obtain the optimal solution and has a good effect on the scheduling of the gas system. Although there are some slight fluctuations in the reward value, the theoretically optimal solution can be obtained.

Claims

1. An optimization scheduling method based on a deep deterministic policy gradient algorithm, characterized in that, The method comprises the following steps: Collecting and preprocessing device data; Building an environment dynamic model; Building an optimization scheduling model; Building a deep deterministic policy gradient algorithm model and training it; The construction of the environment dynamic model includes the establishment of the relationship between the blast furnace and the converter and the respective corresponding storage devices, consumption devices and coal gas power generation devices, which is represented as: Gas b (t) = Gas bc,b (t) + Gas s,b (t) + Gas h,b (t) + Gas l,b (t) + Gas p,b (t) + Gas sg,b (t) Gas c (t) = Gas cc,c (t) + Gas gc,c (t) + Gas l,c (t) + Gas p,c (t) + Gas sg,c (t) wherein, Gas b (t) represents the amount of blast furnace gas produced by the blast furnace at the t period, Gas bc,b (t) represents the amount of blast furnace gas required for the blast furnace itself to operate at the t period, Gas s,b (t) represents the amount of blast furnace gas consumed by the sintering machine at the t period, Gas h,b (t) represents the amount of blast furnace gas consumed by the heating furnace at the t period, Gas l,b (t) represents the amount of blast furnace gas consumed by the ladle at the t period, Gas p,b (t) represents the amount of blast furnace gas supplied to the generator set at the t period, Gas sg,b (t) represents the amount of blast furnace gas diffused at the t period, Gas c (t) represents the amount of converter gas produced by the converter at the t period, Gas cc,c (t) represents the amount of converter gas required for the converter itself to operate at the t period, Gas gc,c (t) represents the amount of converter gas stored in the gas tank at the t period, Gas l,c (t) represents the amount of converter gas required for the ladle at the t period, Gas p,c (t) represents the amount of converter gas supplied to the generator set at the t period, Gas sg,c (t) represents the amount of converter gas diffused at the t period, Gas The optimization scheduling model is based on the environment dynamic model and further adds constraint conditions to the converter gas entering and exiting the gas cabinet and the existing gas amount in the gas cabinet at the current time, and constraints the upper and lower limits of the converter gas amount: wherein, respectively represent upper and lower limits of the converter gas input amount at the inlet of the gas holder, respectively represent upper and lower limits of the converter gas output amount at the outlet of the gas holder, Gas gc represent upper and lower limits of the converter gas storable in the gas holder; The upper and lower limits of the gas consumption of the heating furnace are constrained: wherein, represents an upper limit of the amount of coal gas consumed by the heating furnace, Gas h,b represents a lower limit of the amount of coal gas consumed by the heating furnace; The upper and lower limits of the power of the generator set are constrained: P b (t)+P c (t)=P(t) wherein P b (t) represents the power generated by the generator set using blast furnace gas at time period t, c (t) represents the power generated by the generator set using converter gas at time period t, P(t) is the total power generated by the generator set at time period t, is the upper limit of the power of the generator set, P is the lower limit of the power of the generator set. The start-stop times and the climbing rate of the generator set are limited: -R down ≤ P(t + 1) - P(t) ≤ R up where B is the budget limit of unit start-stop, C e is the cost of each start-stop of the unit, N is the number of start-stop allowed for the unit, R up represents the maximum ramp-up rate of the unit, -R down represents the maximum ramp-down rate of the unit; The deep deterministic policy gradient algorithm model includes defining a state space S, determining key variables that affect the scheduling decision of the gas system, and state variables including the amount of blast furnace gas generated by the blast furnace, the amount of converter gas generated by the converter, the amount of blast furnace gas consumed by the blast furnace itself, the amount of converter gas consumed by the converter itself, the amount of blast furnace gas consumed by the sintering machine, the heating furnace, the ladle and the generator set, the amount of converter gas consumed by the ladle and the generator set, the amount of converter gas in the converter gas cabinet at the current time, the amount of blast furnace gas, the amount of converter gas, the amount of converter gas, and the time-of-use electricity price of each time period, based on these quantities to determine the state, specifically: S = [Gas b (t), Gas c (t), Gas bc,b (t), Gas cc,c (t), Gas s,b (t), Gas h,b (t), Gas l,b (t), Gas l,c (t), Gas p,b (t), Gas p,c (t), Gas sg,b (t), Gas sg,c (t), Gas gc (t), Price(t)] Where Price(t) represents the time-of-use electricity price of each period; Define the action space A, determine the key control actions that can be taken in the scheduling of the coal gas system. The action variables include the operating conditions of the sintering machine, the operating power of the generator set, and the converter gas input and output quantities of the coal gas tank. Based on these quantities, the actions are determined, specifically: Where μ(t) represents the two working conditions of the sintering machine. 2.The optimization scheduling method based on deep deterministic policy gradient algorithm according to claim 1, wherein: The device data includes the amount of gas generated by the blast furnace and the converter and the amount of gas consumed by the blast furnace itself, the amount of gas consumed by each sintering machine in the sintering process, the heating furnace, the ladle, each gas generator set, the amount of gas stored in the gas cabinet, the energy conversion efficiency of each generator set, and the normal working output range and gas flow rate per unit time of the blast furnace, the converter, the sintering machine, the heating furnace, the ladle and each generator set. 3.The optimization scheduling method based on deep deterministic policy gradient algorithm according to claim 1, wherein: The preprocessing includes data denoising, data filling and data standardization, the data denoising uses a convolutional neural network to identify and eliminate random fluctuations and outliers in the data, the data filling fills in missing data by interpolation and mean replacement, and the data standardization performs scale conversion on the data to have a uniform dimension and distribution. 4.The optimization scheduling method based on deep deterministic policy gradient algorithm according to claim 3, wherein: When the sintering machine needs to be repaired or needs to reduce the production of sintered ore, the sintering machine needs to be stopped or put into operation, so the working condition of the sintering machine is divided into two cases, which is represented as: Where μ(t) represents the two working conditions of the sintering machine, μ(t)=0 represents that the sintering machine stops running at t time, μ(t)=1 represents that the sintering machine is put into operation at t time; a is the numerical value of the amount of gas required when the sintering machine is put into operation; The physical relationship between the amount of converter gas in the gas cabinet and the input and output is represented as: Gas gc (t+1) = Gas gc (t) + Gas gc,c (t) wherein, Gas(t) represents the amount of converter gas at the inlet of the gas holder at time t, Gas(t) represents the amount of converter gas at the outlet of the gas holder at time t, gc,c Gas(t) represents the amount of converter gas at the inlet of the gas holder at time t, gc Gas(t) represents the amount of converter gas at the inlet of the gas holder at time t, gc Gas(t+1) represents the total amount of converter gas at the inlet of the gas holder at time t+1.

5. A system employing the optimization scheduling method based on the deep deterministic policy gradient algorithm according to any one of claims 1 to 4, characterized in that, The method comprises the following steps: Data acquisition and preprocessing module, environment dynamic model construction module, optimization scheduling model construction module and deep deterministic policy gradient algorithm model construction module; The data acquisition and preprocessing module is used to collect the required device data and preprocess it; The environment dynamic model construction module is used to build a basic environment dynamic model; The optimization scheduling model construction module is configured to make constraints based on the environment dynamic model; The deep deterministic policy gradient algorithm model construction module is configured to construct a deep deterministic policy gradient algorithm model and train the deep deterministic policy gradient algorithm model. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5. The processor, when executing the computer program, implements the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm in any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm in any one of claims 1 to 4.

Images