MPC heating furnace oxygen-enriched combustion control system parameter optimization method based on reinforcement learning

Abstract

The invention is suitable for the technical field of automatic control, and provides an MPC heating furnace oxygen-enriched combustion control system parameter optimization method based on reinforcement learning, which comprises the following steps: step 1, production line data is acquired, and the acquisition of the production line data is mainly to acquire real data of an actual cement production control production line, including MV, OV and DV data; and step 2, obtaining a state space model, the obtaining of the state space model comprising establishing a model of converting a first-order transfer function into a state space, the MPC parameter adjustment is performed by using the prominent learning ability and adaptive ability of a reinforcement learning (RL) method, the dynamic adjustment according to the change of the system can be realized, and the control accuracy of the system is improved. And training the current oxygen-enriched combustion control algorithm according to the state information under the existing decision-making condition so as to output the expected furnace temperature, and evaluating the effectiveness of the control algorithm through the prepared evaluation standard.

Description

Technical Field

[0001] This invention belongs to the field of automatic control technology, and in particular relates to a parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning. Background Technology

[0002] Model predictive control (MPC) algorithms are used in oxygen-enriched combustion control of hot rolling furnaces in steel industry because they have the ability to consider input-output constraints, perform rolling time-domain optimization, and provide optimal solutions. Traditional MPC control parameter tuning is generally done manually or semi-automatically. Under different operating conditions and unknown disturbances, the parameters obtained by this method have a certain degree of uncertainty. Therefore, a parameter tuning method that does not rely on human experience, has intelligent thinking, and can adaptively select the optimal parameters according to environmental changes is needed. Summary of the Invention

[0003] This invention provides a parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning, aiming to solve the above-mentioned problems.

[0004] This invention is implemented as follows: a parameter optimization method for an MPC (Mechanical Control System) oxygen-enriched combustion control system for a heating furnace based on reinforcement learning, comprising the following steps:

[0005] Step 1: Obtain production line data. This mainly involves obtaining actual data from the cement production control line, including MV, OV, and DV data.

[0006] Step 2: Obtain the state space model, which includes establishing a model that transforms the first-order transfer function into the state space.

[0007] Step 3: MPC parameter optimization. The DDPG algorithm under the Actor-Critic (AC) framework with continuous input and output space is used. That is, the DDPG deep reinforcement learning algorithm is selected to adjust the important parameters of MPC in real time, and then input into the MPC controller to achieve precise control of the target. In the oxygen-enriched combustion control of the heating furnace, the deep learning agent receives furnace pressure, air pressure, oxygen quantity, gas flow rate and NOx status information generated by the interaction between the surrounding environment and the agent. By setting the desired furnace temperature, the upper-level DDPG MPC control algorithm finally outputs the valve opening to achieve constant furnace temperature control.

[0008] Step 4: Obtain the optimal parameter values. After obtaining the optimal control parameters, perform MPC closed-loop testing to finally determine the optimal control parameters.

[0009] Preferably, the first-order time-delay transfer system function in step two is as follows:

[0010]

[0011] Among them, K p T is the gain coefficient. p T is the time constant. d The time delay coefficient is obtained through system identification.

[0012] The above system function is transformed into state-space equation form as follows:

[0013]

[0014] Δcv=C*x p

[0015] Where x is the initial system state vector x p A is the predicted system state vector; A is the system state transition matrix; mv is the input matrix, corresponding to the given values ​​of the opening and closing degrees of the air valve and gas valve in the preheating section of the heating furnace; B is the input transition matrix; C is the output transition matrix; ov is the output matrix, corresponding to the temperature in the preheating section of the heating furnace; Δov is the output difference.

[0016] Preferably, in step three, the agent in the DDPG algorithm consists of an actor network and a critic network. The actor network, as the action decision unit, outputs MPC parameters to adjust the working state of the system, while the critic network evaluates the action based on the operating results of the upper-level control system.

[0017] Preferably, in step four, the Actor network and Critic network are updated based on the evaluation results. The control performance of the upper-level control system is continuously optimized by combining the MPC control parameters, ultimately achieving the optimal parameter combination for system operation.

[0018] Preferably, the system includes a state space and an action space. The state space is defined as a vector formed by environmental data such as furnace pressure, air pressure, oxygen quantity, gas flow rate, air flow rate, and NOx quantity. The action space is defined as MPC control parameters.

[0019] Preferably, the Actor network outputs the optimal control parameters for MPC based on the system's state information. It obtains three components, V, K, and Q, by multiplying the input vector by three weight matrices (obtained through training). Then, it performs a dot product on the corresponding Q and K to obtain the score for each position. It performs a softmax operation on the calculated scores to normalize the scores so that they are all positive and their sum is multiplied by the vector of V position by position. Finally, it adds up all the weighted vectors to produce the output result for that position.

[0020] Preferably, the Critic network's basic process involves the agent interacting with environmental information, perceiving the corresponding environmental state information, and then outputting MPC parameters after training with the Actor network. At the next time step, the environmental state information changes, resulting in a reward. The Critic network is updated based on the accumulated reward value. Subsequently, the Actor network updates its weights in the same direction as the Critic network, and this process is repeated until the Actor network performs well.

[0021] Preferably, the reward feedback includes a reward function, which includes a tiered reward design for positive incentives for actions, and the initial reward function is defined as 0.

[0022] Preferably, the reward function further includes:

[0023] (1) Nitrogen oxide content;

[0024] (2) Air-fuel ratio;

[0025] (3) Fluctuations in gas flow rate;

[0026] (4) Oxygen content;

[0027] (5) Scrolling prediction of furnace temperature;

[0028] (6) Other situations.

[0029] Preferably, the nitrogen oxide content is such that when the predicted NOx concentration is ≤240mg / Nm3, the environment should reward it, and a reward of +10 is set for this condition; the air-fuel ratio is such that when the air-fuel ratio in the preheating section is in the range of 1.98 to 2.43, a reward of +10 is set for this condition; the gas flow fluctuation is such that when the fluctuation range before and after the change in gas flow does not exceed 0.5%, a reward of +20 is set for this condition; when the fluctuation range before and after the change in gas flow is in the range of 0.5% to 1%, a reward of +10 is set for this condition; the oxygen content is such that when the residual oxygen content is in the range of 3% to 5%, the environment should reward it, and a reward of +10 is set for this condition; the furnace temperature rolling prediction is based on the following cost function, where both the cost function and the evaluation index adopt the rolling time domain RMSE, and the cost function and evaluation index adopt the RMSE, defined as follows:

[0030]

[0031] Among them, y i y represents the furnace temperature prediction vector obtained by solving mv in the i-th reciprocal step of MPC. o The measured target vector represents the furnace temperature. n represents the length of the reverse-ordered MPC solution iterations.

[0032] When RMSE≤0.1, set the reward to +20.

[0033] In other cases, reward-20.

[0034] Compared with the prior art, the embodiments of this application have the following main advantages:

[0035] This invention utilizes the superior learning and adaptive capabilities of reinforcement learning to adjust MPC parameters, enabling dynamic adjustments based on system changes and improving control accuracy. The current oxygen-enriched combustion control algorithm is trained based on state information under existing decision conditions to output the desired furnace temperature, and the effectiveness of the control algorithm is evaluated using predetermined criteria. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the process structure of a parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning, according to the present invention.

[0037] Figure 2 This is the MPC online optimization framework diagram. Detailed Implementation

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0039] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0040] This invention provides a parameter optimization method for an MPC (Mechanical Control and Processing) furnace oxygen-enriched combustion control system based on reinforcement learning, as shown in the figure, including the following steps:

[0041] Step 1: Obtain production line data. Obtaining production line data mainly involves acquiring actual data from the cement production control line, including MV, OV, and DV data.

[0042] Step 2: Obtain the state-space model. Obtaining the state-space model includes establishing a model that transforms the first-order transfer function into the state space.

[0043] Step 3: MPC parameter optimization. The DDPG algorithm under the Actor-Critic (AC) framework with continuous input and output space is used. That is, the DDPG deep reinforcement learning algorithm is selected to adjust the important parameters of MPC in real time, and then input into the MPC controller to achieve precise control of the target. In the oxygen-enriched combustion control of the heating furnace, the deep learning agent receives furnace pressure, air pressure, oxygen quantity, gas flow rate and NOx status information generated by the interaction between the surrounding environment and the agent. By setting the desired furnace temperature, the upper-level DDPG MPC control algorithm finally outputs the valve opening to achieve constant furnace temperature control.

[0044] Step 4: Obtain the optimal parameter values. After obtaining the optimal control parameters, perform MPC closed-loop testing to finally determine the optimal control parameters.

[0045] The first-order time-delay transfer system function in step two is as follows:

[0046]

[0047] Among them, K p T is the gain coefficient. p T is the time constant. d The time delay coefficient is obtained through system identification.

[0048] The above system function is transformed into state-space equation form as follows:

[0049]

[0050] Δcv=C*x p

[0051] Where x is the initial system state vector x p A is the predicted system state vector; A is the system state transition matrix; mv is the input matrix, corresponding to the given values ​​of the opening and closing degrees of the air valve and gas valve in the preheating section of the heating furnace; B is the input transition matrix; C is the output transition matrix; ov is the output matrix, corresponding to the temperature in the preheating section of the heating furnace; Δov is the output difference.

[0052] In step three, the agent in the DDPG algorithm consists of an Actor network and a Critic network. The Actor network, acting as the action decision unit, outputs MPC parameters to adjust the system's operating state. The Critic network evaluates the action based on the results of the upper-level control system. The optimization framework is as follows: Figure 2 As shown.

[0053] In step four, the Actor network and Critic network are updated based on the evaluation results. The control performance of the upper-level control system is continuously optimized by combining the MPC control parameters, so as to achieve the optimal parameter combination for system operation.

[0054] The system includes a state space and an action space. The state space is defined as a vector formed by environmental data such as furnace pressure, air pressure, oxygen quantity, gas flow rate, air flow rate, and NOx quantity. The action space is defined as the MPC control parameters. Details are as follows:

[0055]

[0056]

[0057]

[0058] The Actor network outputs the optimal control parameters for MPC based on the system's state information. It is trained by multiplying the input vector by three weight matrices to obtain three components: V, K, and Q. Then, the corresponding Q and K are sequentially multiplied to obtain the score for each position. The calculated scores are then subjected to a softmax operation to normalize the scores so that they are all positive and their sum is multiplied by the vector of V position by position. Finally, all weighted vectors are summed to produce the output result for that position.

[0059] The basic process of the Critic network is as follows: After the agent interacts with environmental information, it perceives the corresponding environmental state information. After training with the Actor network, it outputs MPC parameters. At the next time step, the environmental state information changes, and the agent receives a reward. The Critic network is updated based on the accumulated reward value. Then, the Actor network updates its weights in the same direction as the Critic network. This process is repeated until the Actor network performs well.

[0060] The reward feedback includes a reward function, which includes positive incentives for actions and adopts a tiered reward design. The initial reward function is defined as 0.

[0061] The reward function also includes the following cases:

[0062] (1) Nitrogen oxide content;

[0063] (2) Air-fuel ratio;

[0064] (3) Fluctuations in gas flow rate;

[0065] (4) Oxygen content;

[0066] (5) Scrolling prediction of furnace temperature;

[0067] (6) Other situations.

[0068] For nitrogen oxide (NOx) content, a reward of +10 is set when the predicted NOx concentration is ≤240 mg / Nm3. For air-fuel ratio, a reward of +10 is set when the preheating section air-fuel ratio is between 1.98 and 2.43. For gas flow fluctuation, a reward of +20 is set when the fluctuation range before and after the change does not exceed 0.5%, and a reward of +10 is set when the fluctuation range is between 0.5% and 1%. For oxygen content, a reward of +10 is set when the residual oxygen content is between 3% and 5%. The rolling furnace temperature prediction cost function is as follows: Both the cost function and the evaluation index use rolling time-domain RMSE, and are defined as follows:

[0069]

[0070] Among them, y i y represents the furnace temperature prediction vector obtained by solving mv in the i-th reciprocal step of MPC. o The measured target vector represents the furnace temperature. n represents the length of the reverse-ordered MPC solution iterations.

[0071] When RMSE≤0.1, set the reward to +20.

[0072] In other cases, the reward is -20. Detailed implementation method:

[0074] The steps are as follows:

[0075] (1) Obtain actual production line operation data from the actual control system

[0076] Taking the oxygen-enriched combustion data of a steel plant from April 1, 2024 to April 11, 2024 as an example, the following fields were retrieved.

[0077]

[0078]

[0079] (2) Obtain the furnace temperature control system model

[0080] The model is a state-space equation with a first-order transfer function. After identification, the model matrix is ​​as follows:

[0081]

[0082] The model is a 2-input, 1-output state-space matrix model. The inputs include 2 mv values ​​and 1 cv value. The mv values ​​include the preheating section gas valve opening and the preheating section air valve opening given values. The cv value is the calculated preheating section furnace temperature.

[0083] (3) MPC Online Optimization Framework

[0084] Optimization is achieved through an optimization framework.

[0085] (4) Obtain control parameters

[0086] The obtained parameters are used in the actual control system for control.

[0087] All electrical components mentioned in this article are electrically connected to the controller and power supply. The control method of this invention is controlled by the controller. The control circuit of the controller can be implemented by simple programming by those skilled in the art. The power supply is also common knowledge in the art. Furthermore, this invention is mainly used to protect mechanical devices, so the control method and circuit connection will not be explained in detail.

[0088] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0089] It should be understood that the disclosed apparatus can be implemented in other ways, given the several embodiments provided in this application. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units described above may be implemented in other ways in practice. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or communication connections shown or discussed may be through some interfaces; indirect coupling or communication connections between devices or units may be telecommunications or other forms.

[0090] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can still combine, add, delete, or otherwise adjust the features of the various embodiments of the present invention according to the circumstances without conflict or creative effort, thereby obtaining different technical solutions that do not fundamentally depart from the concept of the present invention. These technical solutions also fall within the scope of protection of the present invention.

Claims

1. A parameter optimization method for an MPC (Mechanical Control and Propulsion) heating furnace oxygen-enriched combustion control system based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain production line data. This mainly involves obtaining actual data from the cement production control line, including MV, OV, and DV data. Step 2: Obtain the state space model, which includes establishing a model that transforms the first-order transfer function into the state space. Step 3: MPC parameter optimization. The DDPG algorithm under the Actor-Critic (AC) framework with continuous input and output space is used. That is, the DDPG deep reinforcement learning algorithm is selected to adjust the important parameters of MPC in real time, and then input into the MPC controller to achieve precise control of the target. In the oxygen-enriched combustion control of the heating furnace, the deep learning agent receives furnace pressure, air pressure, oxygen quantity, gas flow rate and NOx status information generated by the interaction between the surrounding environment and the agent. By setting the desired furnace temperature, the upper-level DDPG MPC control algorithm finally outputs the valve opening to achieve constant furnace temperature control. Step 4: Obtain the optimal parameter values. After obtaining the optimal control parameters, perform MPC closed-loop testing to finally determine the optimal control parameters.

2. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 1, characterized in that, The first-order time-delay transfer system function in step two is as follows: Among them, K p T is the gain coefficient. p T is the time constant. d The time delay coefficient is obtained through system identification. The above system function can be transformed into state-space equation form as follows: Δcv=C*x p Where x is the initial system state vector x p A is the predicted system state vector; A is the system state transition matrix; mv is the input matrix, corresponding to the given values ​​of the opening and closing degrees of the air valve and gas valve in the preheating section of the heating furnace; B is the input transition matrix; C is the output transition matrix; ov is the output matrix, corresponding to the temperature in the preheating section of the heating furnace; Δov is the output difference.

3. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 2, characterized in that, In step three, the agent in the DDPG algorithm consists of an actor network and a critic network. The actor network, as the action decision unit, outputs MPC parameters to adjust the working state of the system, while the critic network evaluates the action based on the operating results of the upper-level control system.

4. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 3, characterized in that, In step four, the Actor network and Critic network are updated based on the evaluation results. The control performance of the upper-level control system is continuously optimized by combining the MPC control parameters, ultimately achieving the optimal parameter combination for system operation.

5. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 4, characterized in that, The system includes a state space and an action space. The state space is defined as a vector formed by environmental data such as furnace pressure, air pressure, oxygen quantity, gas flow rate, air flow rate, and NOx quantity. The action space is defined as MPC control parameters.

6. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 4, characterized in that, The Actor network outputs the optimal control parameters for MPC based on the system's state information. It obtains three components, V, K, and Q, by multiplying the input vector by three weight matrices (obtained through training). Then, it performs a dot product on the corresponding Q and K components to obtain the score for each position. The calculated scores are then subjected to a softmax operation to normalize the scores so that they are all positive and their sum is multiplied by the vector of V position by position. Finally, all weighted vectors are summed to produce the output result for that position.

7. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 6, characterized in that, The Critic network operates on the following basic principle: After the agent interacts with environmental information, it perceives the corresponding environmental state information. Following training with the Actor network, it outputs MPC parameters. At the next time step, the environmental state information changes, resulting in a reward. The Critic network updates based on the accumulated reward value. The Actor network then updates its weights in the same direction as the Critic network. This process is repeated until the Actor network performs well.

8. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 7, characterized in that, The reward feedback includes a reward function, which includes positive incentives for actions using a tiered reward design, with the initial reward function `reward` defined as 0.

9. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 8, characterized in that, The reward function specifically also includes: 1) Nitrogen oxide content; 2) Air-fuel ratio; 3) Fluctuations in gas flow rate; 4) Oxygen levels; 5) Continuous furnace temperature prediction; 6) Other situations.

10. The parameter optimization method for an MPC heating furnace oxygen-enriched combustion control system based on reinforcement learning as described in claim 9, characterized in that, The nitrogen oxide content is set at a reward of +10 when the predicted NOx concentration is ≤240mg / Nm3. The air-fuel ratio is set at a reward of +10 when the air-fuel ratio in the preheating section is in the range of 1.98 to 2.

43. The gas flow fluctuation is set at a reward of +20 when the fluctuation range before and after the change does not exceed 0.5%, and at a reward of +10 when the fluctuation range before and after the change is between 0.5% and 1%. The oxygen content is set at a reward of +10 when the residual oxygen content is in the range of 3% to 5%. The furnace temperature rolling prediction is based on the following cost function, where both the cost function and the evaluation index adopt the rolling time domain RMSE, and are defined as follows: Among them, y i y represents the furnace temperature prediction vector obtained by solving mv in the i-th reciprocal step of MPC. o The measured target vector represents the furnace temperature. n represents the length of the reverse-ordered MPC solution iterations. When RMSE≤0.1, set the reward to +20. In other cases, reward-20.

Images