An optimization control method and system for boiler sootblower based on reinforcement learning

Through a reinforcement learning-based method, using policy networks and value networks to train the optimized control model of boiler soot blowers, the problem of insufficient control capabilities under multiple operating conditions in the existing technology is solved, and the efficient operation of the boiler and resource saving is achieved.

CN115097729BActive Publication Date: 2024-11-26GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210703948.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-21
Publication Date
2024-11-26
Estimated Expiration
2042-06-21

AI Technical Summary

Technical Problem

The existing boiler soot blowing control method has insufficient real-time dynamic control capabilities in multi-working environments, and it is impossible to obtain a global optimal control solution.

Method used

Using a reinforcement learning-based method, by obtaining the state value of the boiler soot blower, using the policy network, value network and target network to establish an experience pool, train an optimized control model, and dynamically adjust the control parameters of the soot blower to achieve the optimal control strategy.

Benefits of technology

Real-time dynamic control in a multi-working environment is realized, and a global optimal control solution is obtained, which improves boiler operation efficiency and resource utilization and reduces costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115097729B_ABST
    Figure CN115097729B_ABST
Patent Text Reader

Abstract

The present invention discloses an optimization control method and system for a boiler sootblower based on reinforcement learning, and relates to the technical field of boiler sootblowing. The method comprises: obtaining a state value of the boiler sootblower, determining an action value, an action state value, and a reward value of the boiler sootblower according to the state value and a strategy network, and establishing an experience pool; training a value network, a target value network, a target strategy network, and the strategy network based on the experience data in the experience pool to obtain an optimization control model for the sootblower; determining control parameters of the sootblower according to the action value, the reaction value, and the optimization control model of the sootblower, so as to solve the technical problem that the existing sootblowing control method is insufficient in real-time dynamic control capability of the sootblower under multiple working conditions and cannot obtain a global optimal control solution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of boiler sootblowing, and in particular to an optimization control method and system for a boiler sootblower based on reinforcement learning. Background Art

[0002] The power station boiler of a coal-fired power plant is an important equipment for generating steam and generating electricity. After the fuel is burned in the boiler, fly ash will be deposited on the boiler surface. The deposition of fly ash on the heating surface is considered to be slagging or fouling. These foulings increase the heat transfer resistance of the heating surface and the ventilation resistance of the flue, affecting the normal operation of the boiler, reducing the thermal efficiency of the boiler and increasing the coal consumption of the unit, which has an adverse effect on the performance of the boiler. At present, about 50% of the fuel used by large units in my country is slagging fuel. Therefore, almost all heating surfaces of the units have slagging problems to varying degrees, causing the units to operate at reduced load or shut down. In severe cases, it will cause major accidents, affecting the safe and economical operation of the boiler.

[0003] During boiler operation, there are many technologies to reduce boiler scaling, such as the use of mineral additives, ash blowers, wet pretreatment of coal, chemical treatment, coal blending, internal cameras, pulse detonation engines, etc. Among the various technical measures to avoid and reduce serious ash accumulation and slagging, steam or air blowing of the heating surface is an effective and commonly used method. Conventional soot blowing methods include steam soot blowing, sonic soot blowing, gas pulse soot blowing, etc., among which steam soot blowing has the widest application range. However, whether it is sonic soot blowing, gas pulse soot blowing or steam soot blowing, a lot of energy is consumed. Taking steam soot blowing as an example, the amount of steam consumed by steam soot blowing generally accounts for 1% of the total steam production, and the heat loss of steam, the heat loss of throttling and the exhaust loss consume about 0.7% of the boiler efficiency. At present, the sootblowing scheme adopted by major power plants in my country is still mainly based on sootblowing at a fixed time every day and every shift. Although the scheduled sootblowing every day brings higher efficiency to the boiler, the steam used for sootblowing is part of the steam production. Excessively frequent sootblowing means that more fuel is needed to generate steam, which also means more cost investment and air pollution emissions generated by coal combustion. In addition, sootblowing in the clean part will cause corrosion of the tube, accelerate the degradation of the surface tube and reduce the service life. Therefore, it is necessary to optimize the control of the sootblower, that is, to blow soot in time to ensure the normal operation efficiency of the boiler, and to control the duration and frequency of sootblowing to reduce resource consumption.

[0004] With the rapid development of computer technology, neural networks have been widely used. At present, the optimization strategy of boiler sootblower based on remaining time prediction is widely used. This sootblowing strategy first uses a prediction model based on remaining time to predict the health status of the boiler heating surface and the sootblowing threshold, and then uses the mechanism update theorem to calculate the boiler sootblowing cost based on the prediction results. The prediction interval and sootblowing threshold are used as optimization variables, and the optimization model is established with the minimum average sootblowing cost as the goal, so as to obtain the relationship model of sootblowing cost rate, sootblowing threshold and monitoring period. Finally, the genetic algorithm is used to optimize and solve the relationship model, and the optimal monitoring period and sootblowing threshold of the boiler are obtained, and the average cost rate of boiler operation is minimized. However, this optimization also has disadvantages. The model optimized by genetic algorithm is good at predicting and controlling the sootblower under single working condition and fixed state, but in the actual scenario of multiple working conditions, it cannot achieve the ideal dynamic optimization control effect, and the algorithm is easy to fall into the local optimal solution, so that the global optimal solution cannot be obtained. Summary of the invention

[0005] The present invention provides a reinforcement learning-based optimization control method and system for a boiler sootblower, which is used to solve the technical problem that the existing sootblowing control method has insufficient real-time dynamic control capability for the sootblower under multiple working conditions and cannot obtain a global optimal control solution.

[0006] The present invention provides an optimization control method for a boiler sootblower based on reinforcement learning, comprising:

[0007] Acquire the state value of the boiler sootblower, determine the action value, action state value, and reward value of the boiler sootblower according to the state value and the strategy network, and establish an experience pool;

[0008] Based on the experience data in the experience pool, the value network, the target value network, the target strategy network and the strategy network are trained to obtain a sootblower optimization control model;

[0009] The control parameters of the sootblower are determined according to the action value, the reaction value and the sootblower optimization control model of the boiler sootblower.

[0010] Preferably, the state value of the boiler soot blower is obtained, and the action value, action state value, and reward value of the boiler soot blower are determined according to the state value and the strategy network, and the experience pool is established as follows:

[0011] The state value is used as the input of the strategy network to obtain an action, the data corresponding to the action is recorded as the action value, and the action state value and the reward value corresponding to the action are determined according to the action value;

[0012] Establishing experience data based on the state value, action value, action state value and reward value, and storing the experience data in a preset experience pool;

[0013] Determine whether the amount of experience data in the experience pool reaches a threshold, if so, end the creation of experience data, if not, return to the step of using the state value as an input to the policy network.

[0014] Preferably, the value network, the target value network, the target strategy network and the strategy network are trained based on the experience data in the experience pool to obtain the sootblower optimization control model as follows:

[0015] Randomly obtain experience data in the experience pool; input the action state value and the preset noise into the target strategy network to obtain the action value corresponding to the action state value, and record the action value corresponding to the action state value as the secondary action value;

[0016] Inputting the state value and the action value into the value network to obtain a first weight and a second weight; wherein the value network includes a first value network and a second value network;

[0017] Inputting the action state value, the secondary action value and the reward value into a target value network to obtain a first target weight and a second target weight; wherein the target value network includes a first target value network and a second target value network;

[0018] Determining a target value according to the first target weight and the second target weight;

[0019] Calculate the TD error according to the target value, the first weight, and the second weight; wherein the TD error includes a first TD error and a second TD error;

[0020] updating the value network parameters of the value network according to the TD error; wherein the value network parameters include: a first value parameter and a second value parameter, the first value parameter being a network parameter of the first value network, and the second value parameter being a network parameter of the second value network;

[0021] Obtaining policy network parameters, calculating a policy gradient according to the value network parameters and the policy network parameters, and updating the policy network parameters using the policy gradient until a preset number of updates is reached; wherein the policy network parameters are network parameters of the policy network;

[0022] Determine whether the update times of the target strategy network and the target value network reach the iteration threshold; if so, smooth the target strategy network and the target value network and output the sootblower optimization control model; if not, return to the step of randomly acquiring experience data in the experience pool.

[0023] Preferably, the control parameters of the sootblower determined according to the action value, the reaction value and the sootblower optimization control model of the boiler sootblower are specifically:

[0024] Using the state value as the input of the policy network to obtain an action, recording the data corresponding to the action as the action value, determining an action opposite to the action according to the action, recording the data corresponding to the opposite action as the counter-action value, and determining the counter-action state value and the counter-action reward value corresponding to the counter-action according to the counter-action value;

[0025] Inputting the state value, the action value, the action state value and the reward value into the sootblower optimization control model to obtain a first control parameter and a first evaluation value;

[0026] Inputting the state value, the reaction value, the reaction state value and the reaction reward value into the sootblower optimization control model to obtain a second control parameter and a second evaluation value;

[0027] A control parameter output by the sootblower optimization control model is determined according to the first evaluation value and the second evaluation value.

[0028] Preferably, the control parameter output by the sootblower optimization control model is determined according to the first evaluation value and the second evaluation value as follows:

[0029] Determine the difference between the first evaluation value and the second evaluation value:

[0030] X c =ω(XX op )

[0031] When X c When X is positive, the control parameter output by the sootblower optimization control model is the first control parameter; when X c When it is a negative value, the control parameter output by the sootblower optimization control model is the second control parameter;

[0032] Wherein, X is the first evaluation value, X op is the second evaluation value, ω is the regression parameter, X c For reward and punishment values.

[0033] Preferably, the determining of the target value according to the first target weight and the second target weight is specifically:

[0034] y=r+γmin(Q' 1 , Q' 2 )

[0035] Among them, γ is the discount rate, r is the reward value, and y is the target value.

[0036] Preferably, the updating of the value network parameters of the value network according to the TD error is specifically as follows:

[0037] The error back propagation algorithm is used to update the value network parameters:

[0038]

[0039] Among them, y is the target value, L is the mean square error loss value of TD error, N is the number of updates; Q is the weight value of the value network.

[0040] Preferably, the acquiring of the policy network parameters, calculating the policy gradient according to the value network parameters and the policy network parameters, and updating the policy network parameters using the policy gradient until a preset number of updates is reached are specifically:

[0041] Obtaining policy network parameters, and calculating parameter gradients according to the policy network parameters;

[0042] Calculating action gradients based on the value network parameters;

[0043] The policy gradient of the policy network is calculated based on the parameter gradient and the action gradient:

[0044]

[0045] J(φ) is the policy objective function, Q θ is the value network; φ is the strategy network; φ is the strategy network parameter, N is the number of updates, a is the action value, s is the state value, θ is the value network parameter, and the value network parameter θ includes the first value parameter θ 1 and the second value parameter θ 2 ;in,

[0046] Preferably, the smoothing of the target strategy network and the target value network includes:

[0047] θ'=τθ+(1-τ)θ'

[0048] φ'=τφ+(1-τ)φ'

[0049] Among them, θ is the value network parameter, θ' is the target value network parameter, φ is the policy network parameter, φ' is the target policy network parameter, and τ is the update coefficient.

[0050] The present invention also provides an optimization control system for a boiler sootblower based on reinforcement learning, comprising:

[0051] The first module is used to obtain the state value of the boiler sootblower, determine the action value, action state value, reward value of the boiler sootblower according to the state value and the strategy network, and establish an experience pool;

[0052] The second module is used to train the value network, the target value network, the target strategy network and the strategy network based on the experience data in the experience pool to obtain a sootblower optimization control model;

[0053] The third module is used to determine the control parameters of the sootblower according to the action value, the reaction value and the sootblower optimization control model of the boiler sootblower.

[0054] It can be seen from the above technical scheme that the present invention has the following advantages: the present invention provides an optimization control method for a boiler sootblower based on reinforcement learning, applies the deep reinforcement algorithm to the optimization field of the controllable operating parameters of the boiler sootblower, obtains the state value of the boiler sootblower, determines the action value, action state value, reward value of the boiler sootblower according to the state value and the strategy network, and establishes an experience pool; based on the experience data in the experience pool, the value network, the target value network, the target strategy network and the strategy network are trained to obtain a sootblower optimization control model; then, the control parameters of the sootblower are determined according to the action value, reaction value and the sootblower optimization control model.

[0055] According to the actual environment, a model reward mechanism is designed. At the same time, the action and reaction of the boiler sootblower are compared. Based on the sootblower optimization control model, the balance point between sootblowing benefit and sootblowing cost is found to achieve dynamic control of the control variables of the sootblower, thereby obtaining the optimal control strategy, and solving the technical problems that the existing sootblowing control method is insufficient in real-time dynamic control of the sootblower under multiple operating conditions and cannot obtain the global optimal control solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A method flow chart of an optimization control method for a boiler sootblower based on reinforcement learning provided by an embodiment of the present invention;

[0058] Figure 2 A flow chart of sootblower optimization control model training provided by an embodiment of the present invention;

[0059] Figure 3A demonstration diagram of case 1 provided for an embodiment of the present invention;

[0060] Figure 4 This is a demonstration diagram of Case 2 provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The embodiment of the present invention provides an optimization control method and system for a boiler sootblower based on reinforcement learning, which relates to the field of boiler sootblowing technology. The deep reinforcement algorithm is applied to the optimization field of controllable operating parameters of the boiler sootblower, and a model reward mechanism is designed according to the actual environment to dynamically control the selection of various control variables of the sootblower, so as to obtain the optimal control strategy and realize dynamic and automatic control of the boiler sootblower of a coal-fired power plant, so as to achieve the multi-objective optimization task of ensuring the operating efficiency of the boiler and saving sootblowing resources. The technical problem that the existing sootblowing control method is insufficient in real-time dynamic control of the sootblower under multiple operating conditions and cannot obtain the global optimal control solution is solved.

[0062] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The present application embodiment provides an optimization control method for a boiler sootblower based on reinforcement learning. Figure 1 , control methods include:

[0064] 100. Obtain a state value of a boiler soot blower, determine an action value, an action state value, and a reward value of the boiler soot blower according to the state value and a strategy network, and establish an experience pool.

[0065] Step 100 is specifically as follows:

[0066] Collect the state change coefficient of the boiler heating surface, and record the change coefficient as the state value s; use the state value s as the input of the strategy network to obtain the action, record the data corresponding to the action as the action value a, and determine the action state value s' and the reward value r corresponding to the action according to the action value a; establish experience data (s, a, s', r) based on the state value, action value, action state value and reward value, and store the experience data in a preset experience pool; determine whether the number of experience data in the experience pool reaches a threshold, if so, end the establishment of experience data, if not, return to the step of collecting the state change coefficient of the boiler heating surface.

[0067] Combined with the application scenarios of boiler sootblowers, the meaning of the empirical data is explained as follows: the environmental feedback variable (state value) s is the heat transfer coefficient of the boiler heating surface state change; the action value a is the controllable variable of the sootblowing system for sootblowing: sootblowing cycle (sootblowing duration, sootblowing frequency), sootblower outlet diameter, sootblower offset angle, sootblower outlet steam flow rate, total steam flow and other parameters; the reward value r is the boiler steam production efficiency calculated by the heat transfer coefficient in the environmental system, and the next feedback state of the environment can be determined by the reward value r.

[0068] Among them, the policy network is a fully connected neural network structure, whose input value is the environmental observation state, which is the state value s in this application, and the output is the action, which is the action value a in this application. The policy network is also called a decision network, and its main function is to interact with the environment. When the boiler sootblower performs an action according to the action value a, the state value s will change and a new state will be formed. We record the heat transfer coefficient of the boiler heating surface state change corresponding to the new state formed after the action is performed according to the action value a as the action state value s'. After the action is performed according to the action value a, a reward r will be generated, and the reward can be positive or negative. For example: a is to perform sootblowing. The sootblowing action consumes a certain amount of steam, but at the same time, the steam generation rate of the boiler also increases. When the benefit brought by the increase in the steam generation rate of the boiler is greater than the amount of steam consumed, we believe that the sootblowing action has obtained a positive reward value. Correspondingly, when the benefit brought by the increase in the steam generation rate of the boiler is less than the amount of steam consumed, it means that sootblowing is not the best choice at this time, and it is believed that the sootblowing action has obtained a negative reward value.

[0069] By setting up an experience pool to store the experience data (s, a, s', r) generated by the interaction between the strategy network and the environment, when the experience pool storage reaches a certain level, a specified amount of experience data is randomly extracted to train the soot blower optimization control model. In this way, the data generated in the past can also be used to train the soot blower optimization control model, which effectively improves the problem of insufficient data in reinforcement learning. Because the data of a specified batch is randomly extracted, the correlation between the training data is greatly reduced, which improves the training efficiency.

[0070] In the boiler sootblowing scenario, in order to make the sootblower optimization control model have better adaptability and prevent the trial-and-error interaction with the environment to accumulate experience due to insufficient experience data, the experience data in the experience pool is preferably set to 10 5 To meet their learning needs.

[0071] 200. Based on the experience data in the experience pool, the value network, the target value network, the target strategy network and the strategy network are trained to obtain a sootblower optimization control model.

[0072] 300. Determine control parameters of the sootblower according to the action value, the reaction value and the sootblower optimization control model of the boiler sootblower.

[0073] The present invention provides an optimization control method for a boiler sootblower based on reinforcement learning, which applies a deep reinforcement algorithm to the optimization field of controllable operating parameters of a boiler sootblower, obtains a state value of the boiler sootblower, determines an action value, an action state value, and a reward value of the boiler sootblower according to the state value and a strategy network, and establishes an experience pool; based on the experience data in the experience pool, a value network, a target value network, a target strategy network, and the strategy network are trained to obtain a sootblower optimization control model; and then determines the control parameters of the sootblower according to the action value, reaction value, and the sootblower optimization control model.

[0074] According to the actual environment, a model reward mechanism is designed. At the same time, the action and reaction of the boiler sootblower are compared. Based on the sootblower optimization control model, the balance point between sootblowing benefit and sootblowing cost is found to achieve dynamic control of the control variables of the sootblower, thereby obtaining the optimal control strategy, and solving the technical problems that the existing sootblowing control method is insufficient in real-time dynamic control of the sootblower under multiple operating conditions and cannot obtain the global optimal control solution.

[0075] As a preferred embodiment, see Figure 2 , Figure 2 The sootblower optimization control model training flow chart provided in the embodiment of the present invention, step 200 specifically includes:

[0076] 201. Randomly obtain experience data (s, a, s', r) from the experience pool; input the action state value s' and the preset noise into the target strategy network to obtain the action value corresponding to the action state value s', and record the action value corresponding to the action state value s' as the secondary action value a';

[0077] 202. Input the state value s and the action value a into the value network to obtain the first weight Q 1 and the second weight Q 2 ; Wherein, the value network includes a first value network and a second value network;

[0078] 203. Input the action state value s', the secondary action value a' and the reward value r into the target value network to obtain the first target weight Q' 1 and the second objective weight Q' 2 ; Wherein, the target value network includes a first target value network and a second target value network;

[0079] 204. According to the first target weight Q' 1 and the second target weight Q' 2Determine the target value y;

[0080] 205. According to the target value y and the first weight Q 1 The second weight Q 2 Calculating a TD error; wherein the TD error includes a first TD error and a second TD error;

[0081] 206. Update the value network parameters of the value network according to the TD error; wherein the value network parameters include: a first value parameter θ 1 and the second value parameter θ 2 , the first value parameter θ 1 is the network parameter of the first value network, and the second value parameter θ 2 is the network parameter of the second value network;

[0082] 207. Obtain a policy network parameter φ, calculate a policy gradient according to the value network parameter and the policy network parameter, and update the policy network parameter using the policy gradient until a preset number of updates is reached; wherein the policy network parameter is a network parameter of the policy network;

[0083] 208. Determine whether the update times of the target strategy network and the target value network reach the iteration threshold. If so, smooth the target strategy network and the target value network and output the sootblower optimization control model. If not, return to the step of randomly acquiring the experience data (s, a, s', r) in the experience pool.

[0084] It should be noted that step 206 and step 207 are performed in no particular order.

[0085] In a preferred embodiment, step 204 is specifically as follows:

[0086] Compare the first target weight Q' 1 and the second target weight Q' 2 Determine the target value y:

[0087] y=r+γmin(Q' 1 , Q' 2 )

[0088] Among them, γ is the discount rate, r is the reward value, and y is the target value.

[0089] In the prior art, a target value network is usually used to guide the update of the value network. The output value of the target value network is the target value y. Because there is noise in the sample, the value of the action value estimate with error is usually larger than the true value in reality. This reduces the accuracy of the valuation function. Since the calculation of the valuation method is based on the Bellman equation, even if the valuation is updated using subsequent states, the accuracy will be seriously reduced. Every time the strategy is updated, using an inaccurate estimate will cause errors to be accumulated. These accumulated errors will cause a bad state to be overestimated, which ultimately leads to the inability to optimize the strategy to the optimal state and the inability of the model to converge. The present application uses two target value networks to evaluate the action value, and then selects the smaller target value network to output the value, so that the phenomenon of overestimation of the action value can be alleviated.

[0090] In a preferred embodiment, step 205 is specifically as follows:

[0091] The error back propagation algorithm is used to update the value network parameters:

[0092]

[0093] Among them, y is the target value, L is the mean square error loss value of TD error, N is the number of updates; Q is the weight value of the value network.

[0094] It should be noted that since the updating method of the strategy network parameters is consistent with the updating method of the value network parameters mentioned above, it will not be elaborated on here.

[0095] It can be understood that updating the value network / strategy network means updating the parameters of the value network / strategy network.

[0096] In the prior art, a method of delayed updating of the policy network is usually adopted, that is, the policy network is updated after the value network is updated multiple times. On the one hand, it can reduce unnecessary repeated updates, and on the other hand, it can also reduce the accumulated errors in multiple updates. In the present application, the policy network / value network is updated first, and when the preset number of updates is reached, the target policy network / target value network is updated.

[0097] In a preferred embodiment, step 207 is specifically as follows:

[0098] Obtaining a policy network parameter φ, and calculating a parameter gradient according to the policy network parameter;

[0099] Calculating action gradients based on the value network parameters;

[0100] The policy gradient of the policy network is calculated based on the parameter gradient and the action gradient:

[0101]

[0102] J(φ) is the policy objective function, Q θ is the value network; φ is the strategy network; φ is the strategy network parameter, N is the number of updates, a is the action value, s is the state value, θ is the value network parameter, and the value network parameter θ includes the first value parameter θ 1 and the second value parameter θ 2 ;in,

[0103] The policy network can obtain a certain action based on the input state value and the policy network parameters (the policy network parameters mentioned here are actually the probability distribution of the policy network selecting a certain action in a certain state. The policy network will select the action with a large probability distribution as the output). When the appropriate policy network parameters are obtained, the action will obtain a positive reward. This application uses the gradient ascent method to find the appropriate policy network parameters.

[0104] In a preferred embodiment, step 208 is specifically as follows:

[0105] θ'=τθ+(1-τ)θ'

[0106] φ'=τφ+(1-τ)φ'

[0107] Among them, θ is the value network parameter, θ' is the target value network parameter, φ is the policy network parameter, φ' is the target policy network parameter, and τ is the update coefficient.

[0108] In a preferred embodiment, step 300 specifically includes:

[0109] 301. Obtain the state value s of the boiler sootblower, use the state value s as the input of the strategy network, obtain an action, record the data corresponding to the action as the action value a, determine the action opposite to the action according to the action, record the data corresponding to the opposite action as the counter-action value a op , according to the reaction value a op Determine the reaction state value s' op The counter-action reward value r corresponding to the counter-action op .

[0110] 302. Input the state value s, the action value a, the action state value s′ and the reward value r into the sootblower optimization control model to obtain a first control parameter and a first evaluation value;

[0111] 303. The state value s and the reaction value a op , the reaction state value s' opand the counter-action reward value r op Inputting the sootblower optimization control model to obtain a second control parameter and a second evaluation value;

[0112] Determine the difference between the first evaluation value and the second evaluation value:

[0113] X c =ω(XX op )

[0114] When X c When X is positive, the control parameter output by the sootblower optimization control model is the first control parameter; when X c When it is a negative value, the control parameter output by the sootblower optimization control model is the second control parameter;

[0115] Wherein, X is the first evaluation value, is the second evaluation value, ω is the regression parameter (taking a positive value), X c For reward and punishment values.

[0116] Since sootblowers also need to consume steam, excessive sootblowing will consume the amount of steam originally used for boiler power generation, resulting in a waste of resources. In order to ensure the power generation efficiency of the boiler, and to minimize the number of unnecessary sootblowing times and save sootblowing costs, we need to control the duration and frequency of sootblowing. The key to control is the evaluation of the sootblowing benefits and sootblowing costs generated by boiler sootblowing. In this application, the output action of the sootblower optimization control model is the control parameter of the sootblower, and the reward and punishment mechanism of the model serves as a dynamic evaluation parameter between the boiler sootblowing benefits and sootblowing costs. The model goal is to minimize resource consumption while ensuring the power generation efficiency of the boiler.

[0117] When the action a output by the policy network is blowing dust, under the reward and punishment mechanism, the counter-action a will be simulated at the same time. op Without soot blowing, the amount of steam for power generation is calculated synchronously. The difference between soot blowing and not soot blowing is multiplied by a regression parameter ω, and its value is controlled within a certain range, which is used as the environmental feedback reward value X of the input action a. c When the reward is positive, the model will be more inclined to output actions with large reward values ​​next time; when the reward is negative, the model will try to minimize the selection of this action.

[0118] In the technology of the aforementioned step 200, this application provides two specific cases for illustration:

[0119] Case 1: (Situations where dust blowing is required)

[0120] like Figure 3As shown in the figure, assuming that the amount of steam used for power generation in the boiler at the end of a certain period is 1 t, the steam generation rate is 0.8 t / h, and at this time, the ash is seriously accumulated. If soot blowing is not performed at this moment, the steam generation rate will be reduced to 0.6 t / h at the end of the next period, and the amount of steam generated during this period is steam for power generation X = 0.75 t; if soot blowing is selected at this moment, the steam generation rate will increase to 0.9 t / h at the end of the next period, and the total amount of steam generated during this period is 0.85 t, which is obtained by deducting the steam used for soot blowing 0.008 t to obtain steam for power generation X op =0.842t. If the soot blowing action is not performed at the end of this period, then according to X c =ω(XX op ) Calculate the reward value, obviously X c <0, the reward value for not blowing the dust is negative, which is regarded as a penalty. If you choose to blow the dust, then X c >0, the reward value obtained by the soot blowing action is a positive number and is regarded as a reward.

[0121] Case 2: (Situations where dust blowing should not be done)

[0122] like Figure 4 As shown in the figure, assuming that the amount of steam used for power generation in the boiler at the end of a certain period is 1 t, and the steam generation rate is 0.998 t / h. At this time, there is slight ash accumulation. If soot blowing is selected at this moment, the steam generation rate will increase to 1 t / h at the end of the next period. The amount of steam generated during this period is 0.999 t of steam for power generation. After deducting the amount of steam used for soot blowing 0.008 t, the amount of steam used for power generation is X = 0.991 t. If soot blowing is not selected at this moment, the steam generation rate will decrease to 0.997 t / h at the end of the next period. The total amount of steam generated during this period is X. op =0.998t. If soot blowing is performed at the end of this period, then according to X c =ω(XX op ) Calculate the reward value, obviously X c <0, the reward value of blowing dust is negative, which is regarded as a penalty. If you choose not to blow dust at this time, then X c >0, the reward value obtained by not blowing dust is a positive number and is regarded as a reward.

[0123] The reward and punishment mechanism determines the probability of the action of the input environment being selected next time, so the reward setting should be set according to the production environment. In the application scenario of boiler soot blowing, a steam soot blower is used, which consumes steam, and boiler power generation also requires steam to drive the turbine to operate. Therefore, when the soot blower is not working, most of the steam is used for boiler power generation. When ash is accumulated, the heat transfer coefficient of the heating surface decreases, and the boiler evaporation rate (the amount of steam generated by the boiler per hour) also decreases, and the total amount of steam used for boiler power generation decreases. In order to ensure the normal power generation of the boiler, the soot blower needs to be enabled. The soot blower also consumes steam for soot blowing, but as the ash is removed, the heat transfer coefficient increases, and the total amount of steam generated also increases, thereby increasing the amount of steam used for boiler power generation and ensuring the normal power generation efficiency of the boiler. Obviously, in actual operation, it is more hoped that the optimization control model can output actions that obtain positive reward values.

[0124] The present application applies the deep reinforcement algorithm to the field of optimization of the controllable operating parameters of the boiler sootblower. By obtaining the state value of the boiler sootblower, the action value, action state value, and reward value of the boiler sootblower are determined according to the state value and the strategy network, and an experience pool is established; based on the experience data in the experience pool, the value network, the target value network, the target strategy network, and the strategy network are trained to obtain a sootblower optimization control model; and then the control parameters of the sootblower are determined according to the action value, reaction value, and the sootblower optimization control model.

[0125] According to the actual environment, a model reward mechanism is designed. At the same time, the action and reaction of the boiler sootblower are compared. Based on the sootblower optimization control model, the balance point between sootblowing benefit and sootblowing cost is found to achieve dynamic control of the control variables of the sootblower, thereby obtaining the optimal control strategy, and solving the technical problems that the existing sootblowing control method is insufficient in real-time dynamic control of the sootblower under multiple operating conditions and cannot obtain the global optimal control solution.

[0126] Another embodiment of the present invention further provides an optimization control system for a boiler sootblower based on reinforcement learning, comprising:

[0127] The first module is used to obtain the state value of the boiler sootblower, determine the action value, action state value, reward value of the boiler sootblower according to the state value and the strategy network, and establish an experience pool;

[0128] The second module is used to train the value network, the target value network, the target strategy network and the strategy network based on the experience data in the experience pool to obtain a sootblower optimization control model;

[0129] The third module is used to determine the control parameters of the sootblower according to the action value, the reaction value and the sootblower optimization control model of the boiler sootblower.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimization control method for boiler sootblower based on reinforcement learning, It is characterized in that include: Acquire the state value of the boiler sootblower, determine the action value, action state value, and reward value of the boiler sootblower according to the state value and the strategy network, and establish an experience pool; Based on the experience data in the experience pool, the value network, the target value network, the target strategy network and the strategy network are trained to obtain a sootblower optimization control model; Determining control parameters of the sootblower according to the action value, the reaction value and the sootblower optimization control model of the boiler sootblower; The value network, the target value network, the target strategy network and the strategy network are trained based on the experience data in the experience pool to obtain the sootblower optimization control model as follows: Randomly obtain experience data in the experience pool; input the action state value and the preset noise into the target strategy network to obtain the action value corresponding to the action state value, and record the action value corresponding to the action state value as the secondary action value; Inputting the state value and the action value into the value network to obtain a first weight and a second weight; wherein the value network includes a first value network and a second value network; Inputting the action state value, the secondary action value and the reward value into a target value network to obtain a first target weight and a second target weight; wherein the target value network includes a first target value network and a second target value network; Determining a target value according to the first target weight and the second target weight; Calculate the TD error according to the target value, the first weight, and the second weight; wherein the TD error includes a first TD error and a second TD error; updating the value network parameters of the value network according to the TD error; wherein the value network parameters include: a first value parameter and a second value parameter, the first value parameter being a network parameter of the first value network, and the second value parameter being a network parameter of the second value network; Obtaining policy network parameters, calculating a policy gradient according to the value network parameters and the policy network parameters, and updating the policy network parameters using the policy gradient until a preset number of updates is reached; wherein the policy network parameters are network parameters of the policy network; Determine whether the update times of the target strategy network and the target value network reach the iteration threshold; if so, perform smoothing on the target strategy network and the target value network and output the sootblower optimization control model; if not, return to the step of randomly acquiring experience data in the experience pool; The control parameters of the soot blower are determined according to the action value, the reaction value and the soot blower optimization control model of the boiler soot blower as follows: Using the state value as the input of the policy network to obtain an action, recording the data corresponding to the action as the action value, determining an action opposite to the action according to the action, recording the data corresponding to the opposite action as the reaction value, and determining the reaction state value and the reaction reward value corresponding to the reaction according to the reaction value; Inputting the state value, the action value, the action state value and the reward value into the sootblower optimization control model to obtain a first control parameter and a first evaluation value; Inputting the state value, the reaction value, the reaction state value and the reaction reward value into the sootblower optimization control model to obtain a second control parameter and a second evaluation value; A control parameter output by the sootblower optimization control model is determined according to the first evaluation value and the second evaluation value.

2. The optimization control method according to claim 1, It is characterized in that The obtaining of the state value of the boiler soot blower, determining the action value, action state value, and reward value of the boiler soot blower according to the state value and the strategy network, and establishing the experience pool are specifically as follows: The state value is used as the input of the strategy network to obtain an action, the data corresponding to the action is recorded as the action value, and the action state value and the reward value corresponding to the action are determined according to the action value; Establishing experience data based on the state value, action value, action state value and reward value, and storing the experience data in a preset experience pool; Determine whether the amount of experience data in the experience pool reaches a threshold, if so, end the creation of experience data, if not, return to the step of using the state value as an input to the policy network.

3. According to the optimization control method of claim 1, It is characterized in that The control parameter output by the sootblower optimization control model is determined according to the first evaluation value and the second evaluation value as follows: Determine the difference between the first evaluation value and the second evaluation value: ; when When is a positive value, the control parameter output by the sootblower optimization control model is the first control parameter; when When it is a negative value, the control parameter output by the sootblower optimization control model is the second control parameter; Among them, the is the first evaluation value, is the second evaluation value, is the regression parameter, Reward and punishment value.

4. The optimization control method according to claim 1, It is characterized in that The determination of the target value according to the first target weight and the second target weight is specifically as follows: ; in, is the discount rate, is the reward value, is the target value, is the first target weight, is the second target weight.

5. The optimization control method according to claim 1, It is characterized in that The updating of the value network parameters of the value network according to the TD error is specifically as follows: The error back propagation algorithm is used to update the value network parameters: ; in, is the target value, is the mean square error loss value of the TD error, is the number of updates; is the weight value of the value network.

6. The optimization control method according to claim 1, It is characterized in that The obtaining of the policy network parameters, calculating the policy gradient according to the value network parameters and the policy network parameters, and updating the policy network parameters using the policy gradient until a preset number of updates is reached is specifically: Obtaining policy network parameters, and calculating parameter gradients according to the policy network parameters; Calculating action gradients based on the value network parameters; The policy gradient of the policy network is calculated based on the parameter gradient and the action gradient: ; is the policy objective function, For the value network; For strategic networks; are policy network parameters, is the number of updates, is the action value, is the status value, is the value network parameter, the value network parameter Include the first value parameter and the second value parameter ;in, .

7. The optimization control method according to claim 1, It is characterized in that The smoothing of the target strategy network and the target value network comprises: ; ; in, is the value network parameter, is the target value network parameter, are policy network parameters, is the target strategy network parameter, is the update coefficient.