Optimized scheduling method and system based on depth deterministic policy gradient algorithm

Abstract

The invention discloses an optimal scheduling method and system based on a depth deterministic strategy gradient algorithm, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting equipment data, and carrying out the preprocessing; constructing an environment dynamic model; constructing an optimal scheduling model; and a depth deterministic strategy gradient algorithm model is constructed and trained. Storage safety of a gas cabinet, gas consumption safety of a heating furnace, safe operation of a generator set and the like are considered, safe and stable operation of the system is ensured, the deep deterministic strategy gradient algorithm is adopted and combined with the deep neural network and the Actor-Critic network, the scheduling process is more intelligent, time-of-use electricity price is combined, and the scheduling efficiency is improved. The power generation power of the generator set in each time period better meets the power utilization requirement of the iron and steel industry, the electricity purchased from a power grid is reduced, the value of generated electric energy is maximized, blast furnace gas and converter gas are utilized more efficiently, the diffusion amount of the gas is reduced, the utilization efficiency of the gas is improved, and pollution to the environment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optimization scheduling method and system based on a deep deterministic policy gradient algorithm. Background Art

[0002] In the steel industry, coal gas power generation plays an extremely important role in the energy management and economic benefits of the entire steel plant. The coal gas generated by blast furnaces, converters and other equipment can be used as an important raw material for the operation of sintering machines, heating furnaces and the like, reducing the cost of raw materials. Part of the coal gas produces a large amount of electricity through power generation equipment. The produced electricity can reduce the amount of electricity purchased from the power grid in the steel industry, improve the economic benefits of the steel industry, and reduce the amount of coal gas emissions and environmental pollution. The optimal scheduling of the gas system involves the flow and pressure control of gas generation equipment, storage equipment, consumption equipment, and gas power generation equipment and gas system pipelines. In addition, the stability and reliability of gas power generation are greatly affected by the quality and supply stability of gas, which may affect the power generation efficiency and the stability of the power grid. Therefore, the gas generation equipment and other equipment need to meet a certain dynamic balance. The complexity of this system lies in that it not only needs to consider the coordinated management between material flow and energy flow, but also needs to deal with the uncertainty and variability in the production process. Traditional optimization methods, such as mathematical analytical methods (including linear programming, nonlinear programming and dynamic programming), are effective in dealing with small-scale problems. However, when faced with large-scale, dynamically changing actual industrial problems, they are often unable to meet the needs due to limitations in computing speed and real-time performance.

[0003] In view of this, reinforcement learning methods have shown their superiority in dealing with complex, dynamic and highly uncertain systems, and the deep deterministic policy gradient algorithm can process both continuous action space and discrete action space. An experience replay mechanism can be introduced to store the experience gained from the interaction between the agent and the environment in an experience pool, and randomly extract small batches of data for learning during the training process to reduce the correlation between data. Noise can be added to encourage exploration in action to prevent overfitting and find the optimal solution. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: how to ensure the calculation speed and the optimal calculation result when processing large-scale, dynamically changing data.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: an optimization scheduling method based on a deep deterministic policy gradient algorithm, comprising: collecting device data and preprocessing it; building an environmental dynamic model; building an optimization scheduling model; building a deep deterministic policy gradient algorithm model and training it.

[0007] As a preferred scheme of the optimization scheduling method based on deep deterministic policy gradient algorithm described in the present invention, the equipment data includes the gas generation and gas consumption of blast furnace and converter, the gas consumption of each sintering machine, heating furnace, ladle and each gas generator set in the sintering process, the gas storage capacity of the gas tank, the energy conversion efficiency of each generator set, the normal working output range of equipment such as blast furnace, converter, sintering, heating furnace, ladle and each generator set, and the gas flow rate per unit time.

[0008] As a preferred solution of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in the present invention, the preprocessing includes data denoising, data filling and data standardization. The data denoising is to use a convolutional neural network to identify and eliminate random fluctuations and outliers in the data. The data filling is to fill the missing data by interpolation and mean replacement. The data standardization is to scale the data to have a unified dimension and distribution.

[0009] As a preferred solution of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in the present invention, the construction of the environmental dynamic model includes establishing the relationship between the blast furnace and the converter and their corresponding storage equipment, consumption equipment and gas power generation equipment, which is expressed as:

[0010] Gas b (t) = Gas bc,b (t)+Gas s,b (t)+Gas h,b (t)+Gas l,b (t)+Gas p,b (t)+Gas sgb (t)

[0011] Gas c (t) = Gas cc,c (t)+Gas gc,c (t)+Gas l,c (t)+Gas p,c (t)+Gas sgc (t)

[0012] Among them, Gas b (t) represents the amount of blast furnace gas produced by the blast furnace during period t, Gas bc,b(t) represents the amount of blast furnace gas required for the blast furnace to operate during the period t. s,b (t) represents the amount of blast furnace gas consumed by the sintering machine during period t, Gas h,b (t) represents the amount of blast furnace gas consumed by the heating furnace during period t, Gas l,b (t) represents the amount of blast furnace gas consumed by the ladle during period t, Gas p,b (t) represents the amount of blast furnace gas supplied to the generator set during period t, Gas sgb (t) represents the amount of blast furnace gas released during period t, Gas c (t) represents the amount of converter gas produced by the converter during period t, Gas cc,c (t) represents the amount of converter gas required for the converter to work during the period t. gc,c (t) represents the amount of converter gas stored in the gas tank during period t, Gas l,c (t) represents the amount of converter gas required by the ladle during the period t. p,c (t) represents the amount of converter gas supplied to the generator set during period t, Gas sgc (t) represents the amount of converter gas released during period t.

[0013] As a preferred solution of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in the present invention, when the sintering machine needs to be repaired or the production of sintered ore needs to be reduced, the sintering machine needs to be stopped or put into operation. Therefore, the operating conditions of the sintering machine are divided into two cases, which are expressed as:

[0014]

[0015] Wherein μ represents two working conditions of the sintering machine. When μ=0, it means that the sintering machine stops running at time t; when μ=1, it means that the sintering machine is put into operation at time t; a is the value of the amount of gas required when the sintering machine is put into operation;

[0016] The physical relationship between the converter gas volume in the gas tank and the input and output is expressed as:

[0017]

[0018] Gas gc (t+1)=Gas gc (t)+Gas gc,c (t)

[0019] in, represents the converter gas input at the gas tank inlet during period t, Gas represents the output of converter gas at the gas tank outlet during period t. gc,c(t) represents the amount of converter gas remaining in the gas tank after input and output during period t. gc (t) represents the amount of converter gas in the gas tank at time t, Gas gc (t+1) represents the total amount of converter gas in the gas tank at time t+1.

[0020] As a preferred solution of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in the present invention, the optimization scheduling model is to further add constraints on the basis of the environmental dynamic model, and to constrain the upper and lower limits of the converter gas volume entering and flowing out of the gas holder and the gas volume in the gas holder at the current moment:

[0021]

[0022]

[0023]

[0024] in, They represent the upper and lower limits of converter gas input at the gas tank inlet,

[0025] They represent the upper and lower limits of converter gas output at the gas tank outlet, Gas gc Indicates the upper and lower limits of converter gas that can be stored in the gas tank;

[0026] Constrain the upper and lower limits of the gas consumption of the heating furnace:

[0027]

[0028] in, Indicates the upper limit of gas consumption of the heating furnace. Gas h,b Indicates the lower limit of gas consumption of the heating furnace;

[0029] Constrain the upper and lower limits of the power of the generator set:

[0030] P b (t)+P c (t) = P(t)

[0031]

[0032] Among them, P b (t) represents the power generation of the generator set using blast furnace gas in period t, P c (t) represents the power generation of the generator set using converter gas in period t, P(t) is the total power generation of the generator set in period t, P is the upper limit of the power of the generator set, and P is the lower limit of the power of the generator set;

[0033] Limit the number of starts and stops and the ramp rate of the generator set:

[0034]

[0035] -R down ≤P(t+1)-P(t)≤R up

[0036] Among them, B is the unit start-stop budget limit, C e is the cost of each start and stop of the unit, N is the number of starts and stops allowed for the unit, R up Indicates the maximum climbing rate of the unit, -R down Indicates the maximum downslope rate of the unit.

[0037] As a preferred solution of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in the present invention, the deep deterministic policy gradient algorithm model includes defining a state space S, determining key variables that affect the scheduling decision of the gas system, and the state variables include the amount of blast furnace gas generated by the blast furnace, the amount of converter gas generated by the converter, the amount of blast furnace gas consumed by the blast furnace itself, the amount of converter gas consumed by the converter itself, the amount of blast furnace gas consumed by the sintering machine, the heating furnace, the ladle and the generator set, the amount of converter gas consumed by the ladle and the generator set, the amount of converter gas in the converter gas cabinet at the current moment, the blast furnace gas, the release amount of converter gas and the time-of-use electricity price in each time period, and the state is determined based on these quantities, specifically:

[0038] S=[Gas b (t),Gas c (t),Gas bc,b (t),Gas cc,c (t),Gas s,b (t),Gas h,b (t),Gas l,b (t),

[0039] Gas l,c (t),Gas p,b (t),Gas p,c (t),Gas sg,b (t),Gas sg,c (t),Gas gc (t),Price(t)]

[0040] Among them, Price(t) represents the time-of-use electricity price in each period;

[0041] Define the action space A and determine the key control actions that can be taken in the gas system scheduling. The action variables include the operating conditions of the sintering machine, the operating power of the generator set, and the converter gas input and output of the gas tank. The actions are determined based on these quantities, specifically:

[0042] In a second aspect, another object of the present invention is to provide an optimization scheduling system based on a deep deterministic policy gradient algorithm, comprising: a data acquisition and preprocessing module, an environment dynamic model building module, an optimization scheduling model building module and a deep deterministic policy gradient algorithm model building module;

[0043] The data acquisition and preprocessing module is used to acquire the required equipment data and perform preprocessing;

[0044] The environment dynamic model building module is used to build a basic environment dynamic model;

[0045] The optimization scheduling model building module is used to constrain based on the environmental dynamic model;

[0046] The deep deterministic policy gradient algorithm model building module is used to build a deep deterministic policy gradient algorithm model and train it.

[0047] In a third aspect, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm as described above are implemented.

[0048] In a fourth aspect, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm as described above.

[0049] Beneficial effects of the present invention: The present invention provides an optimization scheduling method and system based on a deep deterministic policy gradient algorithm, which performs a variety of beneficial controls, takes into account the storage safety of gas cabinets, the gas consumption safety of heating furnaces, and the safe operation of generator sets, etc., to ensure the safe and stable operation of the system. The deep deterministic policy gradient algorithm is adopted in combination with a deep neural network and an Actor-Critic network to make the scheduling process more intelligent. Combined with time-of-use electricity prices, the power generation of the generator set in each period is more in line with the electricity demand of the steel industry, reducing the amount of electricity purchased from the power grid, maximizing the value of the generated electricity, and more efficiently utilizing blast furnace gas and converter gas, reducing the amount of gas released, improving the utilization efficiency of gas, reducing pollution to the environment, improving the economy of the overall operation of the steel industry, and meeting the requirements of low-carbon and environmental protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0051] Figure 1 An overall flow chart of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

[0052] Figure 2 A gas system structure diagram of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

[0053] Figure 3 A reinforcement learning principle structure diagram of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

[0054] Figure 4 An Actor-Critic network structure diagram of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

[0055] Figure 5 A time-of-use electricity price distribution diagram of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

[0056] Figure 6 A gas cabinet position distribution diagram of an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

[0057] Figure 7 A diagram of training results of a deep deterministic policy gradient algorithm agent for an optimization scheduling method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0060] Example 1

[0061] Reference Figure 1-Figure 4 , is an embodiment of the present invention, and provides an optimization scheduling method based on a deep deterministic policy gradient algorithm, comprising:

[0062] S1: Collect equipment data and pre-process it;

[0063] Furthermore, the gas system is an independent and complete subsystem of the steel enterprise, which includes gas generation equipment, storage equipment, consumption equipment and gas power generation equipment. Conduct on-site surveys to obtain data on equipment such as gas generation equipment, storage equipment, consumption equipment and gas power generation equipment from the steel plant. The data that needs to be obtained include the gas generation and gas consumption of blast furnaces and converters, the gas consumption of sintering machines, heating furnaces, ladles, and gas generators in the sintering process, the gas storage capacity of gas tanks, the energy conversion efficiency of each generator, the normal working output range of blast furnaces, converters, sintering, heating furnaces, ladles, and generators, and the gas flow rate per unit time.

[0064] It should be noted that the data obtained in the steel plant may contain noise data due to data collection errors, sensor failures, human errors, etc. Therefore, before modeling, it is necessary to denoise the acquired data, eliminate the noise, fill in the missing values, and standardize the data. Specifically:

[0065] ① Data denoising: For the noisy data in the data set, convolutional neural networks are used to identify and eliminate random fluctuations and outliers in the data, improve data accuracy, and reduce its impact on model training.

[0066] ②Data filling: fill in missing data through interpolation, mean replacement and other techniques to ensure the integrity of the data set and the accuracy of the analysis.

[0067] ③Data standardization: Scale the data to make it have a unified dimension and distribution for easy comparison and processing.

[0068] S2: Building a dynamic model of the environment;

[0069] Furthermore, building a dynamic model of the environment can provide real-time information about environmental changes, allowing the decision-making process to adapt to these changes and helping the algorithm to better understand and predict environmental changes, which is the basis for making effective decisions.

[0070] It should be noted that the data set obtained through preprocessing in step S1 is analyzed to determine the initial conditions of the gas system model, etc. The physical relationship between the equipment constitutes the basis for system optimization and scheduling, which determines the storage and use efficiency of the equipment, ensures the balance of gas supply and demand, and improves energy efficiency. Based on this, the physical relationship between the gas system's production equipment, storage equipment, consumption equipment, and gas power generation equipment is established, and then the environmental dynamic model is constructed. Among them, the network structure relationship between the gas system's gas production equipment, storage equipment, consumption equipment, and gas power generation equipment is as follows: Figure 2 shown.

[0071] Furthermore, the relationship between the blast furnace and the converter and their corresponding storage equipment, consumption equipment and gas power generation equipment is established respectively, wherein the blast furnace mainly supplies blast furnace gas to the blast furnace itself, sintering machine, heating furnace, ladle, generator set, and there is also a part of gas that cannot be fully utilized and is released; the converter mainly supplies converter gas to the converter itself, gas holders, and directly supplied and indirectly supplied to the ladle and generator set through the gas holder, and like the blast furnace gas, there will be a part of the converter gas that cannot be fully utilized and is released; the gas required by the ladle is mainly supplied by the blast furnace gas generated by the blast furnace. If the blast furnace gas is insufficient, it will be supplied by the converter gas. As shown below:

[0072] Gas b (t) = Gas bc,b (t)+Gas s,b (t)+Gas h,b (t)+Gas l,b (t)+Gas p,b (t)+Gas sgb (t)(1)

[0073] Gas c (t) = Gas cc,c (t)+Gas gc,c (t)+Gas l,c (t)+Gas p,c (t)+Gas sgc (t) (1)

[0074] Among them, Gas b (t) represents the amount of blast furnace gas produced by the blast furnace during period t, Gas bc,b(t) represents the amount of blast furnace gas required for the blast furnace to operate during the period t. s,b (t) represents the amount of blast furnace gas consumed by the sintering machine during period t, Gas h,b (t) represents the amount of blast furnace gas consumed by the heating furnace during period t, Gas l,b (t) represents the amount of blast furnace gas consumed by the ladle during period t, Gas p,b (t) represents the amount of blast furnace gas supplied to the generator set during period t, Gas sgb (t) represents the amount of blast furnace gas released during period t, Gas c (t) represents the amount of converter gas produced by the converter during period t, Gas cc,c (t) represents the amount of converter gas required for the converter to work during the period t. gc,c (t) represents the amount of converter gas stored in the gas tank during period t (the value is negative if released), Gas l,c (t) represents the amount of converter gas required by the ladle during the period t. p,c (t) represents the amount of converter gas supplied to the generator set during period t, Gas sgc (t) represents the amount of converter gas released during period t.

[0075] It should be noted that the amount of blast furnace gas supplied to the sintering machine is allocated according to the number of sintering machines put into operation. The sintering machine needs regular maintenance and overhaul to prevent sintering machine failures during operation. When the storage amount of sintered ore is too much or too little, the sintering machine needs to be stopped or put into operation. Therefore, the operating conditions of the sintering machine are divided into two cases. The following uses a piecewise function to list these two cases:

[0076]

[0077] Wherein μ represents two working conditions of the sintering machine. When μ=0, it means that the sintering machine stops running at time t; when μ=1, it means that the sintering machine is put into operation at time t; a is the value of the amount of gas required when the sintering machine is put into operation.

[0078] Furthermore, in the present invention, the gas holder is one of the core devices for achieving the invention objective. The gas storage characteristics of the gas holder directly affect the power generation value of the generator set and the amount of gas released. As a storage device for converter gas, the gas holder has input and output of converter gas. Since there is a certain difference between the input and output converter gas amounts, the converter gas amount in the gas holder will change continuously. The physical relationship is established as follows:

[0079]

[0080] Gas gc (t+1)=Gasgc (t)+Gas gc,c (t) (3)

[0081] in, represents the converter gas input at the gas tank inlet during period t, Gas represents the output of converter gas at the gas tank outlet during period t. gc,c (t) represents the amount of converter gas remaining in the gas tank after input and output during period t. gc (t) represents the amount of converter gas in the gas tank at time t, Gas gc (t+1) represents the total amount of converter gas in the gas tank at time t+1.

[0082] S3: Build an optimization scheduling model;

[0083] Furthermore, after constructing the environmental dynamic model, an optimization scheduling model is built on its basis. The optimization scheduling model takes into account the dynamic characteristics and constraints of the gas system, improves the efficiency of gas use, reduces energy waste, and helps maintain the stability of the gas system. On the basis of the basic environmental dynamic model, further constraints are added to make the environment more complete and accurate.

[0084] It should be noted that the input and output of the gas tank's inlet and outlet as well as the upper and lower limits of the gas volume in the tank are crucial. It ensures that the gas tank operates under a safe and stable working pressure, prevents safety hazards caused by excessively high or low pressure, and also reduces environmental pollution and improves energy efficiency. The upper and lower limits are constrained:

[0085]

[0086] in, They represent the upper and lower limits of converter gas input at the gas tank inlet, They represent the upper and lower limits of converter gas output at the gas tank outlet, Gas gc Indicates the upper limit of converter gas that can be stored in the gas tank, and indicates the lower limit of converter gas that can be stored in the gas tank.

[0087] Furthermore, if the amount of gas supplied to the heating furnace is too high, it may cause the furnace temperature to be out of control, equipment damage or even explosion accidents, while if the amount of gas is too low, the furnace temperature may be insufficient, affecting the heating efficiency and causing the product quality to decline. If the amount of gas consumed by the heating furnace is too high or too low, it will cause energy waste and environmental pollution, increase industrial production costs and safety risks. Therefore, the gas consumption of the heating furnace needs to be reasonably controlled and its upper and lower limits must be constrained:

[0088]

[0089] in, Indicates the upper limit of gas consumption of the heating furnace. Gas h,b Indicates the lower limit of gas consumption of the heating furnace.

[0090] It should be noted that in the present invention, maximizing the value of the electric energy generated by the generator set is an important goal, so it is extremely important to ensure the safe operation of the generator set. Reasonable upper and lower power limits can prevent the generator set from being damaged or malfunctioning due to excessive load, and can also avoid affecting the power generation efficiency and stability due to too low load. Therefore, the specific power operating range of the generator set is determined to ensure the safe operation of the equipment, and its upper and lower limits are constrained:

[0091] P b (t)+P c (t) = P(t)

[0092]

[0093] Among them, P b (t) represents the power generation of the generator set using blast furnace gas in period t, P c (t) represents the power generation of the generator set using converter gas in period t, P(t) is the total power generation of the generator set in period t, is the upper limit of the generator set power, and P is the lower limit of the generator set power.

[0094] Furthermore, frequent start-stop and other operations will accelerate the wear of the unit and reduce the actual operating life of the unit. Considering the equipment life loss and operating economy, the start-stop frequency and ramp rate of the generator set need to be restricted to a certain extent:

[0095]

[0096] -R down ≤P(t+1)-P(t)≤R up (8)

[0097] Among them, B is the unit start-stop budget limit, C e is the cost of each start and stop of the unit, N is the number of starts and stops allowed for the unit, R up Indicates the maximum ramp rate of the unit during time period t, -R down It indicates the maximum downslope rate of the unit in the time period t.

[0098] S4: Build a deep deterministic policy gradient algorithm model and train it.

[0099] It should be noted that the model to be established is a deep reinforcement learning model, which requires the agent to interact with the environment and set appropriate reward functions and penalty items to guide the agent, so that the agent can continuously learn in such interactions with clear goals and eventually learn the optimal strategy. Figure 3 This is the mechanism diagram of the interaction between the agent and the environment, where the environment provides the state s of the agent at time t t and reward r t , the agent generates action a at time t through the strategy t The environment is guided by the actions generated by the agent, and the environment then feeds back the new state s to the agent. t+1 and reward r t , and continue training until the maximum number of iterations is reached. After the environment model is built, it is time to build the agent and set the reward function and penalty term.

[0100] Furthermore, the state space S is defined, and the key variables that affect the scheduling decision of the gas system are determined. The key state variables are integrated into a state vector s. The state variables include the amount of blast furnace gas generated by the blast furnace, the amount of converter gas generated by the converter, the amount of blast furnace gas consumed by the blast furnace itself, the amount of converter gas consumed by the converter itself, the amount of blast furnace gas consumed by the sintering machine, heating furnace, ladle and generator set, the amount of converter gas consumed by ladle and generator set, the amount of converter gas in the converter gas holder at the current moment, the amount of blast furnace gas, the amount of converter gas released and the time-of-use electricity price in each time period. The state vector s is determined based on these quantities. Specifically:

[0101] S=[Gas b (t),Gas c (t),Gas bc,b (t),Gas cc,c (t),Gas s,b (t),Gas h,b (t),Gas l,b (t),

[0102] Gas l,c (t),Gas p,b (t),Gas p,c (t),Gas sg,b (t),Gas sg,c (t),Gas gc (t),Price(t)](9)

[0103] Among them, Price(t) represents the time-of-use electricity price in each period.

[0104] The state space defines the set of all possible states that the agent can observe and exist in, helping the agent understand the complexity and diversity of its environment.

[0105] Define the action space A, determine the key control actions that can be taken in the gas system scheduling, and integrate the key control actions into an action vector a. The action variables include the operating conditions of the sintering machine, the operating power of the generator set, and the converter gas input and output of the gas tank. The action vector a is determined based on these quantities. Specifically:

[0106]

[0107] The action space specifies all possible actions that the agent can take in each state, which provides the agent with behavioral diversity. The action space and the state space are interdependent and together form the basis for the agent's decision-making.

[0108] Construct a deep neural network, and through its capture and learning of the constraints in the physical model, make the model have strong enough constraints when running, and ensure the accuracy and rationality of the training results. First, define an input layer, clearly define the number of samples of the input data, and name this layer InputLayer; define a fully connected layer, which contains 100 neurons. The number of neurons is determined according to the complexity of the problem. Name this fully connected layer FC1; define an activation function layer, and select the activation function as the Relu function; define a second fully connected layer, which contains 5 neurons. This layer will receive the output of the previous activation layer and perform further feature conversion. Name this layer OutputLayer; finally, define an output layer to output the prediction results, and name this layer RegressionOutput.

[0109] Define various parameters and behaviors of the neural network during training, control the training process and optimize model performance.

[0110] The data is divided into training set, validation set and test set. The specific division ratio is as follows:

[0111]

[0112] Where S train represents the training set division ratio, S val represents the validation set division ratio, S test Indicates the test set division ratio.

[0113] Constructing an Actor-Critic network. Unlike a deep neural network, the Actor-Critic network generates a set of actions that conform to the probability distribution according to the state, and evaluates these actions to select the appropriate strategy. It is a basic architecture of the deep deterministic policy gradient algorithm agent. The state information s given to the Actor network by the environment, the Actor feeds back a set of actions a to the Critic network and the environment. The environment generates a corresponding reward r through the action a. The Critic network gives the Actor network a feedback through the reward and state information given by the environment and the action generated by the Actor network, such as Figure 4 shown.

[0114] First, we build a Critic network and two input layers, one for receiving state information and the other for action information. We build an array of multiple neural network layers to process the joint representation of action and state. The layers we build include:

[0115] Define a connection layer to merge the state and action inputs together, and name this layer 'concat'; define a fully connected layer with 1000 neurons, and name this layer 'FC1'; define an activation function activation layer, and name this layer 'Relu1'; define another fully connected layer with 1000 neurons, and name this layer 'FC2'; define another activation function activation layer, and name this layer 'Relu2'; define the last fully connected layer used to represent the value of the state-action pair, and name this layer 'CriticOutput'.

[0116] The learning rate is set to 0.001 and the gradient threshold is set to 1.

[0117] To build an Actor network, first build an array containing multiple neural network layers. These layers include:

[0118] Define an input layer for receiving state information, and name this layer 'observation'; define a fully connected layer with 200 neurons, and name this layer 'ActorFC1'; define an activation function activation layer, and name this layer 'ActorRelu1'; define a fully connected layer with 200 neurons, and name this layer 'ActorFC2'; define an activation function activation layer, and name this layer 'ActorRelu2'; define another fully connected layer with the number of neurons specified by numAct, and name this layer 'ActorFC3'; define a tanh activation layer that limits the output to a certain range, and name this layer 'ActorTanh'; finally, define a scaling layer that scales the output of the tanh layer to the actual range of the action, and name this layer 'ActorTanh'.

[0119] Set the learning rate to 0.01 and the gradient threshold to 1.

[0120] Create a reinforcement learning agent based on the deep deterministic policy gradient algorithm. First, create a representation of a deterministic policy, receive observations as input, and output actions; configure the options of the deep deterministic policy gradient algorithm agent through rlDDPGAgentOptions, including sampling time, experience pool length, number of samples drawn from the experience pool, and discount factor; set the noise parameters added by the agent when selecting actions; finally, create a deep deterministic policy gradient algorithm agent.

[0121] Create a deep deterministic policy gradient algorithm agent, set the reward function and penalty term, and ensure that the agent is correctly guided during the training process. The reward function and penalty term are shown in equations (13) to (18):

[0122] Reward1=P×Δt×Price(t) (12)

[0123]

[0124] Penalty2=Gas sgb (14)

[0125] Penalty3=Gas sgc (15)

[0126]

[0127] Reward=Reward1-λ1·Penalty1-λ2·Penalty2-λ3·Penalty3-λ4·Penalty4(17)

[0128] λ1, λ2, λ3, and λ4 are coefficients set for Penalty1, Penalty2, Penalty3, and Penalty4 respectively according to the degree of penalty.

[0129] Example 2

[0130] One embodiment of the present invention provides an optimization scheduling system based on a deep deterministic policy gradient algorithm, including: a data acquisition and preprocessing module, an environment dynamic model construction module, an optimization scheduling model construction module and a deep deterministic policy gradient algorithm model construction module; the data acquisition and preprocessing module is used to acquire required device data and perform preprocessing; the environment dynamic model construction module is used to build a basic environment dynamic model; the optimization scheduling model construction module is used to impose constraints based on the environment dynamic model; the deep deterministic policy gradient algorithm model construction module is used to build a deep deterministic policy gradient algorithm model and perform training.

[0131] Example 3

[0132] An embodiment of the present invention is different from the first two embodiments in that:

[0133] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0135] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0136] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] Example 4

[0138] See also Figure 5-Figure 7 , which is an embodiment of the present invention, provides an optimization scheduling method based on a deep deterministic policy gradient algorithm. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0139] The interval time of data collection is in the order of minutes, and the specific data to be collected are shown in Table 1:

[0140]

[0141]

[0142] Time-of-use electricity prices Figure 5 As shown, it can be seen that the time-of-use electricity price is the lowest from 23:00 to 7:00, which is the low electricity load period; the electricity price is between the peak and the trough from 7:00 to 9:00 and from 12:00 to 17:00, which is the stable electricity load period; the electricity price is the highest from 9:00 to 12:00 and from 17:00 to 23:00, which is the peak electricity load period. Formulating rewards based on reasonable electricity prices will help alleviate the peak load problem of the power grid and balance the load of the power grid.

[0143] Figure 6 This is a diagram of the gas level in the gas tank in each time period. It can be seen from the figure that the gas tank level is gradually increasing from 23:00 to 6:00 and 13:00 to 15:00, and is gradually decreasing from 6:00 to 13:00 and 15:00 to 22:00. During periods of lower electricity prices, gas is stored in the gas tank, and during periods of peak electricity prices, the gas stored in the gas tank is used. This reduces the purchase of electricity from the power grid during peak electricity price periods, helps with peak load regulation of the power grid, and improves economic benefits.

[0144] Figure 7 This is a reward curve diagram during the training process. It can be seen from the figure that the reward value is very low when the reinforcement learning agent starts training, but after about dozens of training times, the reward value gradually reaches a peak and tends to stabilize. This shows that the deep reinforcement learning based on the deep deterministic policy gradient algorithm converges quickly to obtain the optimal solution, and has a good effect on the scheduling of the gas system. Although there are certain slight fluctuations in the reward value, the theoretically optimal solution can be obtained.

Claims

1. An optimization scheduling method based on a deep deterministic policy gradient algorithm, characterized in that: include: Collect equipment data and pre-process it; Build dynamic models of the environment; Build an optimized scheduling model; Build a deep deterministic policy gradient algorithm model and train it.

2. The optimization scheduling method based on the deep deterministic policy gradient algorithm according to claim 1, characterized in that: The equipment data include the gas generation and gas consumption of blast furnaces and converters, the gas consumption of each sintering machine, heating furnace, ladle and gas generator sets in the sintering process, the gas storage capacity of the gas tank, the energy conversion efficiency of each generator set, the normal working output range of blast furnaces, converters, sintering, heating furnaces, ladles and generator sets, and the gas flow rate per unit time.

3. The optimization scheduling method based on the deep deterministic policy gradient algorithm according to claim 1, characterized in that: The preprocessing includes data denoising, data filling and data standardization. The data denoising is to use convolutional neural networks to identify and eliminate random fluctuations and outliers in the data. The data filling is to fill missing data through interpolation and mean replacement. The data standardization is to scale the data to make it have a unified dimension and distribution.

4. The optimization scheduling method based on the deep deterministic policy gradient algorithm according to claim 3, characterized in that: The construction of the environmental dynamic model includes establishing the relationship between the blast furnace and the converter and their corresponding storage equipment, consumption equipment and gas power generation equipment, which is expressed as: Gas b (t)=Gas bc,b (t)+Gas s,b (t)+Gas h,b (t)+Gas l,b (t)+Gas p,b (t)+Gas sgb (t) Gas c (t)=Gas cc,c (t)+Gas gc,c (t)+Gas l,c (t)+Gas p,c (t)+Gas sgc (t) Among them, Gas b (t) represents the amount of blast furnace gas produced by the blast furnace during period t, Gas bc,b (t) represents the amount of blast furnace gas required for the blast furnace to operate during the period t. s,b (t) represents the amount of blast furnace gas consumed by the sintering machine during period t, Gas h,b (t) represents the amount of blast furnace gas consumed by the heating furnace during period t, Gas l,b (t) represents the amount of blast furnace gas consumed by the ladle during period t, Gas p,b (t) represents the amount of blast furnace gas supplied to the generator set during period t, Gas sgb (t) represents the amount of blast furnace gas released during period t, Gas c (t) represents the amount of converter gas produced by the converter during period t, Gas cc,c (t) represents the amount of converter gas required for the converter to work during the period t. gc,c (t) represents the amount of converter gas stored in the gas tank during period t, Gas l,c (t) represents the amount of converter gas required by the ladle during the period t. p,c (t) represents the amount of converter gas supplied to the generator set during period t, Gas sgc (t) represents the amount of converter gas released during period t.

5. The optimization scheduling method based on the deep deterministic policy gradient algorithm according to claim 4, characterized in that: When the sintering machine needs to be repaired or the production of sintered ore needs to be reduced, the sintering machine needs to be stopped or put into operation. Therefore, the operating conditions of the sintering machine are divided into two conditions, which are expressed as: Wherein μ represents two working conditions of the sintering machine. When μ=0, it means that the sintering machine stops running at time t; when μ=1, it means that the sintering machine is put into operation at time t; a is the value of the amount of gas required when the sintering machine is put into operation; The physical relationship between the converter gas volume in the gas tank and the input and output is expressed as: Gas gc (t+1)=Gas gc (t)+Gas gc,c (t) in, represents the converter gas input at the gas tank inlet during period t, Gas represents the output of converter gas at the gas tank outlet during period t. gc,c (t) represents the amount of converter gas remaining in the gas tank after input and output during period t. gc (t) represents the amount of converter gas in the gas tank at time t, Gas gc (t+1) represents the total amount of converter gas in the gas tank at time t+1.

6. The optimization scheduling method based on the deep deterministic policy gradient algorithm according to claim 5, characterized in that: The optimization scheduling model is based on the environmental dynamic model, and further adds constraints to constrain the upper and lower limits of the converter gas volume entering and flowing out of the gas tank and the gas volume in the gas tank at the current moment: in, They represent the upper and lower limits of converter gas input at the gas tank inlet, They represent the upper and lower limits of converter gas output at the gas tank outlet, Gas gc Indicates the upper and lower limits of converter gas that can be stored in the gas tank; Constrain the upper and lower limits of the gas consumption of the heating furnace: in, Indicates the upper limit of gas consumption of the heating furnace. Gas h,b Indicates the lower limit of gas consumption of the heating furnace; Constrain the upper and lower limits of the power of the generator set: P b (t)+P c (t)=P(t) Among them, P b (t) represents the power generation of the generator set using blast furnace gas in period t, P c (t) represents the power generation of the generator set using converter gas in period t, P(t) is the total power generation of the generator set in period t, is the upper limit of the power of the generator set, and P is the lower limit of the power of the generator set; Limit the number of starts and stops and the ramp rate of the generator set: -R down ≤P(t+1)-P(t)≤R up Among them, B is the unit start-stop budget limit, C e is the cost of each start and stop of the unit, N is the number of starts and stops allowed for the unit, R up Indicates the maximum climbing rate of the unit, -R down Indicates the maximum downslope rate of the unit.

7. The optimization scheduling method based on the deep deterministic policy gradient algorithm according to claim 6, characterized in that: The deep deterministic policy gradient algorithm model includes defining a state space S, determining key variables that affect the scheduling decision of the gas system, and the state variables include the amount of blast furnace gas generated by the blast furnace, the amount of converter gas generated by the converter, the amount of blast furnace gas consumed by the blast furnace itself, the amount of converter gas consumed by the converter itself, the amount of blast furnace gas consumed by the sintering machine, the heating furnace, the ladle and the generator set, the amount of converter gas consumed by the ladle and the generator set, the amount of converter gas in the converter gas cabinet at the current moment, the amount of blast furnace gas, the amount of converter gas released and the time-of-use electricity price in each time period. The state is determined based on these quantities, specifically: S=[Gas b (t),Gas c (t),Gas bc,b (t),Gas cc,c (t),Gas s,b (t),Gas h,b (t),Gas l,b (t), Gas l,c (t),Gas p,b (t),Gas p,c (t),Gas sg,b (t),Gas sg,c (t),Gas gc (t),Price(t)] Among them, Price(t) represents the time-of-use electricity price in each period; Define the action space A and determine the key control actions that can be taken in the gas system scheduling. The action variables include the operating conditions of the sintering machine, the operating power of the generator set, and the converter gas input and output of the gas tank. The actions are determined based on these quantities, specifically:

8. A system using the optimization scheduling method based on the deep deterministic policy gradient algorithm as described in any one of claims 1 to 7, characterized in that: include: Data acquisition and preprocessing module, environment dynamic model building module, optimization scheduling model building module and deep deterministic policy gradient algorithm model building module; The data acquisition and preprocessing module is used to acquire the required equipment data and perform preprocessing; The environment dynamic model building module is used to build a basic environment dynamic model; The optimization scheduling model building module is used to constrain based on the environmental dynamic model; The deep deterministic policy gradient algorithm model construction module is used to construct a deep deterministic policy gradient algorithm model and perform training.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the optimization scheduling method based on the deep deterministic policy gradient algorithm described in any one of claims 1 to 7 are implemented.

Images