T-S fuzzy and TD3-based slow dynamic unknown coal-fired power generation system coordination control method

Abstract

The invention discloses a T-S fuzzy and TD3-based slow dynamic unknown coal-fired power generation system coordination control method, which comprises the following steps: firstly, decomposing a coal-fired power generation system into a fast subsystem and a slow subsystem by using a singular perturbation theory, and decomposing an original control task into a stabilization task of the fast subsystem and a tracking task of the slow subsystem; for the fast subsystem, a fuzzy set is selected on a definition domain of the fast subsystem by using the slow variation characteristic of steam pressure, and a T-S fuzzy model is constructed; a fast subsystem controller is obtained by solving algebraic Riccati equations corresponding to a plurality of linear systems. For a slow subsystem, a double-Q network is adopted to reduce overestimation of a Q value, a dynamic learning rate and a batch size adjustment mechanism are introduced to accelerate training convergence, and learning is carried out under a TD3 algorithm framework to obtain control input of the slow subsystem. The fast subsystem and the slow subsystem control inputs to be combined and then act on the original system to obtain state information at the next moment, and the intelligent agent interacts with the original system to complete collaborative optimization.

Description

Technical Field

[0001] The present invention relates to the field of data-driven control of coal-fired power generation systems, and mainly to a coordinated control method for a slow-dynamic unknown dual-time-scale coal-fired power generation system based on TS fuzzy and TD3 algorithms. Background Art

[0002] As global energy demand continues to grow, coal-fired power generation is an important part of the modern power system, and its safe and efficient operation is crucial to the stability of the power grid. Coal-fired power generation systems have highly nonlinear and strong coupling characteristics between fast and slow dynamics, and optimizing their operation strategies has become a research focus. The coordinated control system receives load instructions from the power grid, controls the boiler and steam turbine as an integrated unit, and sends instructions to coordinate the slow dynamics of the boiler and the fast dynamics of the steam turbine. The boiler has a slow response due to its large thermal inertia, while the steam turbine responds quickly. The main goal of the coordinated control system is to meet the load demand of the power grid by adjusting the output power, while maintaining the turbine pressure and boiler water level at a stable level to ensure stable operation of the unit. In recent years, scholars have used singular perturbation theory to decompose coal-fired power generation systems into fast and slow subsystems, in which the fast subsystem responds quickly and adjusts stably through the TS fuzzy controller, and the slow subsystem ensures asymptotic stability through the Lyapunov method. However, this method has a high dependence on accurate system modeling and lacks the adaptive ability to cope with environmental and parameter changes, which limits its application under complex working conditions. In addition, although existing data-driven methods can adaptively adjust the output of boilers and turbines to optimize power generation efficiency and respond to load changes, these methods usually fail to fully consider the dual-time scale characteristics and may cause numerical ill-conditioned problems, thereby affecting the stability and reliability of the control system.

[0003] The integration of singular perturbation theory and reinforcement learning is a feasible approach to solving the coordinated control of coal-fired power generation systems with fast and slow dynamic coupling, where accurate models are difficult to construct. Singular perturbation theory decomposes the fast and slow dynamic coupling system into a fast subsystem and a slow subsystem, reducing the original control problem to the stabilization task of the fast subsystem and the tracking task of the slow subsystem. As an advanced reinforcement learning algorithm, TD3 significantly improves the stability and convergence of training by introducing dual critic networks, delayed policy updates, and soft updates of the target network, thereby enhancing the ability to handle complex system dynamics. TS fuzzy modeling approximates nonlinear systems into multiple linear subsystems through fuzzy rules, which can simplify the control strategy design of the fast subsystem.

[0004] Therefore, there is an urgent need to develop a coordinated optimization control method for coal-fired power generation systems with self-learning ability by combining the stability of TD3 and the modeling ability of TS fuzzy. Summary of the invention

[0005] The present invention provides a slow dynamic unknown dual time scale coal-fired power generation system coordinated control method based on TS fuzzy and TD3 algorithm, which solves the problems of obvious difference between fast and slow dynamics, incomplete dynamic information and insufficient adaptability in the coordinated control of coal-fired power generation system. See the following description for details:

[0006] A coordinated control method for a slow dynamic unknown dual-time scale coal-fired power generation system based on TS fuzzy and TD3 algorithm comprises the following steps:

[0007] Step 1, decompose the original coal-fired power generation system into a fast subsystem and a slow subsystem, and decompose the original control task into a stabilization task of the fast subsystem and a tracking task of the slow subsystem;

[0008] Step 2, modeling the tracking task of the slow subsystem as a Markov decision process, defining a four-tuple (S, A, R, γ), where A is the action space, representing the set of operations performed by the reinforcement learning agent;

[0009] S is the state space, and the slow component of the original coal-fired power generation system state is selected to construct the slow subsystem state data;

[0010] R is the reward function; γ is the discount factor; the TD3 algorithm framework is adopted, and loss clipping, dynamic learning rate and batch size adjustment mechanism are introduced to optimize the training process, and the control input a of the slow subsystem is obtained. k ;

[0011] Step 3: For the fast subsystem stabilization task, a TS fuzzy model is established, the controller gain is obtained by solving the algebraic Riccati equation, and the fast subsystem control input u is obtained by the parallel compensation method. 2f ;

[0012] Step 4, fast system control input a k and the slow subsystem control input u 2f After the combination, the original coal-fired power generation system is acted on to obtain the state at the next moment, and the system returns to step 2 to calculate the reward information according to the reward function R. The intelligent agent optimizes the control strategy according to the state at the next moment and the reward information until the set maximum number of steps is reached.

[0013] Furthermore, step 1 specifically includes the following contents:

[0014] Step 101, consider the following dual time scale coal-fired power generation system

[0015]

[0016] Among them, x1 represents the slow state variables of the coal-fired power generation system, namely, steam pressure and steam water density, x2 represents the fast state variables of the coal-fired power generation system, namely, output power, u is the control input, namely, the valve opening of fuel flow, steam flow, and feed water flow, 0<ε<1 is the time scale parameter, which characterizes the degree of difference between the fast and slow dynamic time scales, and f, g are vector or matrix equations of appropriate dimensions.

[0017] Step 102, let ε = 0, and get x 2s =φ s (x 1s ,u s ,t), and substituting into (35a) we can obtain the state equation of the slow subsystem

[0018]

[0019] Here, the subscript "s" in the variable indicates its slow part.

[0020] Step 103, state x2 contains the slow component x 2s and fast component x 2f , that is, x2=x 2s +x 2f , based on (35b), we can get the state equation of the fast subsystem:

[0021]

[0022] The subscript "f" in the variable indicates its fast part.

[0023] Furthermore, the control goal of the original system is to achieve the tracking task, that is, to ensure that the drum pressure, coal flow and steam flow can accurately track their preset target values ​​while reducing energy consumption as much as possible. Its performance indicators are defined as

[0024]

[0025] Among them, Q and R are the weight matrices of error state and control input, respectively, and T s is the target state, u is the control input of the system, and γ is the discount factor.

[0026] Then the performance indicators of the decomposed fast and slow subsystems are as follows:

[0027]

[0028] Among them, J f is the performance index of the fast subsystem, Q f and R f are the weight matrices corresponding to the state and control input of the fast subsystem, J s is the performance index of the slow subsystem, Q s and Rs are the weight matrices corresponding to the error state and control input of the slow subsystem, x e is the deviation between the slow component of the current state and the slow component of the target state.

[0029] Furthermore, step 2 specifically includes the following contents:

[0030] Step 201, formulate the tracking problem of the coal-fired power generation slow subsystem under the reinforcement learning framework, and define a four-tuple (S, A, R, γ), specifically:

[0031] (1) A is the action space, which represents the set of operations that the reinforcement learning agent can perform. For the coordination control problem, the control input can be used as the action of the agent, that is, a = [u 1s ,u 2s ,u 3s ] T , where u 1s ,u 2s ,u 3s They represent the slow components of the fuel flow, steam flow and drum feedwater flow control valve openings. Each valve can rotate in the range of (0,1), and each control variable is a continuous action space, 0 means fully closed, and 1 means fully open.

[0032] (2) S is the state space, which represents the set of information observed by the agent. Select the slow component of the system state, the boiler drum steam pressure x 1s (kg / cm 2 ), turbine power x 2s (MW) and the fluid density x in the boiler drum 3s (kg / cm 3 ), the slow component of the current state and the target state T s The deviation between the slow components is e = [e1, e2, e3] T The complete observation information can be described as

[0033] s=[x 1s ,x 2s ,x 3s ,e1,e2,e3] T (39)

[0034] (3) r is the reward function, which is used to evaluate the goals and effects of the agent when performing actions. s The reward becomes smaller when the system state is close to the target state T s When the slow component is large, the reward becomes larger, so the reward r is set to

[0035] r=-ω1||e||-ω2a (40)

[0036] Among them, ω1 and ω2 are the tracking error penalty weight and the control consumption penalty weight, ||e|| represents the difference between the current system state and the target state T s The norm of the tracking error between .

[0037] (4)γ is a discount factor, which is used to measure the importance of future rewards. Specifically, the discount factor determines the degree to which the agent reduces future rewards;

[0038] Step 202, initialize two critic networks and and an actor network π φ , and initialize with random parameters θ1, θ2, φ, and assign the initial values ​​to the target network:

[0039] θ′1←θ1,θ′2←θ2,φ′←φ (41)

[0040] Where θ′1 and θ′2 are the parameters of the target critic network, and φ′ is the parameter of the target actor network.

[0041] Step 203: When the agent is training, the current state at the current time k is represented as s k ∈S. When taking action a t After π(s)+∈, ε~N(0,σ), it transfers to the next state s k+1 and obtain a reward r from the environment k , and store the data sequence (s, a, r, s′) of each time step into the experience pool;

[0042] Step 204, define Used to describe the coal-fired power generation system in state s at a specific time k. k When , execute action a k , and then the long-term expected control cost function of the coordinated control strategy π(s) is implemented, that is,

[0043]

[0044] Step 205: extract N (s, a, r, s′) data sequences from the experience pool for neural network training. When the state is s, select an action according to the target actor network. Using two target critic networks and Evaluate state s′ and target action The value of Q The minimum value among the values ​​to reduce the overestimation bias

[0045]

[0046] Step 206, the loss of the critic network is a measure of the difference between the predicted Q value and the target Q value

[0047]

[0048] Where B is the batch size to be extracted.

[0049] Step 207, randomly initialize the running mean μ of the critic network loss c1 and μ c2 , the decay rate β c1 and β c2 , standard deviation multiple n c Calculate the standard deviation based on the initial loss value Then, we can get the upper and lower bounds of loss based on the operation:

[0050]

[0051] Step 208: Update the new loss L according to the relationship between the running loss and the upper and lower limits. c :

[0052]

[0053] Step 209: Dynamically update the network loss mean μ c1 and μ c2 , used to solve the next loss upper and lower limits:

[0054]

[0055] Step 210, if L critic <μ c1 , dynamically adjust the Critic network learning rate:

[0056] α c ←(1-γ c )α c (48)

[0057] Among them, γ c Tune parameters for the Critic network learning rate.

[0058] If L critic >μ c1 , dynamically adjust the Critic network learning rate:

[0059] α c ←(1+γ c )α c (49)

[0060] Step 211, update critic network parameters:

[0061]

[0062] Step 212, continuously looping steps 206-211 until the training is completed and the optimal control strategy is obtained.

[0063] Step 213, the role of the actor network is to generate the optimal action under a given state, that is, to maximize the Q value output by the critic network. The loss of the actor network is defined as:

[0064]

[0065] Step 214, randomly initialize the running mean μ of the actor network loss a1 and μ a2 , the decay rate β a1 and β a2 , standard deviation multiple n c , calculate the standard deviation based on the initial loss value Then we can obtain the upper and lower bounds of the loss:

[0066]

[0067] Step 215: Update the new loss L according to the relationship between the running loss and the upper and lower limits. a :

[0068]

[0069] Step 216, dynamically update the network loss mean μ a1 and μ a2 , used to solve the next loss upper and lower limits:

[0070]

[0071] Step 217, if L actor <μ a1 , dynamically adjust the Actor network learning rate:

[0072] α a ←(1-γ a )α a (55)

[0073] Among them, γ a Tuning parameters for the Actor network learning rate.

[0074] If L actor >μ a1 , dynamically adjust the Actor network learning rate:

[0075] α a ←(1+γ a )α a (56)

[0076] Step 218, update actor network parameters:

[0077]

[0078] Step 219, continuously looping steps 213-218 until the training is completed and the optimal control strategy is obtained.

[0079] Step 220, dynamically adjust the batch size according to the current weighted loss. The weighted loss is L ac =tL actor +(1-t)L critic , where t is the weight factor, μ ac =tμ a1 +(1-t)μ c1 is the weighted average of the actor and critic losses. If L ac <μ ac , then increase the batch size

[0080] B←(1+ρ)B (58)

[0081] Here, ρ is the batch size adjustment parameter.

[0082] If L ac >μ ac , then reduce the batch size

[0083] B←(1-ρ)B (59)

[0084] Furthermore, step 3 specifically includes the following contents:

[0085] Step 301, divide the domain of the slow state variable x1 into multiple fuzzy sets, and establish corresponding fuzzy rules. Each fuzzy rule is expressed as follows:

[0086] Fast subsystem rule i:

[0087] If x1 belongs to the fuzzy set

[0088] but

[0089]

[0090] Where i = 1, 2, ..., s, s is the number of fuzzy rules, A fi and B fi is the corresponding constant matrix.

[0091] Step 302: Select an appropriate fuzzy set based on the slow variable x1 and calculate its membership degree μ i (x1)

[0092]

[0093] Among them, a i is the left endpoint of the fuzzy set, b i is the midpoint of the fuzzy set, c i is the right endpoint of the fuzzy set.

[0094] Solving for membership

[0095]

[0096] Step 303, the overall TS fuzzy fast subsystem model is as follows:

[0097]

[0098] Step 304, with T f The fast subsystem is discretized for the sampling period, and the discretized fast subsystem is in the following form:

[0099]

[0100] in, h i (k) = h i (x1(k)).

[0101] Solve the following algebraic Riccati equation

[0102]

[0103] Among them, P i is the solution of the Riccati equation, and then the feedback gain matrix K i :

[0104]

[0105] Further, the fuzzy fast controller is obtained:

[0106]

[0107] Furthermore, step 4 specifically includes the following contents:

[0108] Step 401, add the slow subsystem control input selected in step 204 and the fast subsystem control input obtained in step 305 to obtain a combined control input:

[0109] u=a k +u 2f (k) (67)

[0110] Step 402, the combined control input interacts with the original environment to obtain the next state, reward and other information to provide data support for the subsequent training of the intelligent agent. Through this interaction process, the intelligent agent continuously updates the network parameters and optimizes the control strategy.

[0111] The beneficial effects of the technical solution provided by the present invention are:

[0112] 1) By combining the TD3 algorithm with TS fuzzy modeling, the coordinated control problem of slow dynamic unknown coal-fired power generation system is effectively addressed, and the adaptability and robustness of the control strategy to the dynamic changes of complex systems are significantly improved.

[0113] 2) By introducing a dynamic learning rate and batch size adjustment mechanism into the TD3 algorithm, the training process of the neural network is optimized, enabling the algorithm to automatically adjust the training parameters at different stages, avoiding the problem of gradient explosion or vanishing, improving the training stability and efficiency, and accelerating the convergence process. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 It is a coordinated control flow chart of slow dynamic unknown dual time scale coal-fired power generation system based on TS fuzzy and TD3 algorithm;

[0115] Figure 2 It is a diagram of slow subsystem training;

[0116] Figure 3 It is the trajectory diagram of steam pressure x1 and error e1;

[0117] Figure 4 It is the trajectory diagram of steam turbine power x2 and error e2;

[0118] Figure 5 is the trajectory diagram of fluid density x3 and error e3 in the boiler drum;

[0119] Figure 6 It is a trajectory diagram of valve opening controlled by controlling input fuel flow u1, steam flow u2 and drum feed water flow u3. DETAILED DESCRIPTION

[0120] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0121] The present invention provides a coordinated control method for a slow dynamic unknown dual-time scale coal-fired power generation system based on TS fuzzy and TD3 algorithm, such as Figure 1 As shown, the method comprises the following steps:

[0122] Step 1: Use singular perturbation theory to decompose the coal-fired power generation system into a fast subsystem and a slow subsystem, and decompose the original control task into a stabilization task of the fast subsystem and a tracking task of the slow subsystem. The details are as follows:

[0123] Step 101, consider the following dual time scale coal-fired power generation system

[0124]

[0125] Among them, x1 represents the slow state variables of the coal-fired power generation system, namely, steam pressure and steam water density, x2 represents the fast state variables of the coal-fired power generation system, namely, output power, u is the control input, namely, the valve opening of fuel flow, steam flow, and feed water flow, 0<ε<1 is the time scale parameter, which characterizes the degree of difference between the fast and slow dynamic time scales, and f, g are vector or matrix equations of appropriate dimensions.

[0126] The control goal of the original system is to achieve the tracking task, that is, to ensure that the drum pressure, coal flow and steam flow can accurately track their preset target values ​​while reducing energy consumption as much as possible. Its performance indicators are defined as

[0127]

[0128] Among them, Q and R are the weight matrices of error state and control input, respectively, and T s is the target state, u is the control input of the system, and γ is the discount factor.

[0129] Step 102, let ε = 0, and get x 2s =φ s (x 1s ,u s ,t), and substituting into (69a) we can obtain the state equation of the slow subsystem

[0130]

[0131] Here, the subscript "s" in the variable indicates its slow part.

[0132] Step 103, state x2 contains the slow component x 2s and fast component x 2f , that is, x2=x 2s +x 2f , based on (69b), we can get the state equation of the fast subsystem

[0133]

[0134] The subscript "f" in the variable indicates its fast part.

[0135] Step 104, the performance indicators of the decomposed fast and slow subsystems are as follows

[0136]

[0137] Among them, J f is the performance index of the fast subsystem, Q f and R f are the weight matrices corresponding to the state and control input of the fast subsystem, J s is the performance index of the slow subsystem, Q s and R s are the weight matrices corresponding to the error state and control input of the slow subsystem, x e is the deviation between the slow component of the current state and the slow component of the target state.

[0138] Step 2: For the tracking control problem of the slow subsystem, it is modeled as a Markov decision process. The TD3 algorithm framework is used to collect the slow state data from the original system to reconstruct the slow subsystem state data. Loss clipping, dynamic learning rate and batch size adjustment mechanisms are introduced to optimize the training process to obtain the control input of the slow subsystem. The details are as follows:

[0139] Step 201, formulate the tracking problem of the coal-fired power generation slow subsystem under the reinforcement learning framework, and define a four-tuple (S, A, R, γ), specifically:

[0140] (1) A is the action space, which represents the set of operations that the reinforcement learning agent can perform. For the coordination control problem, the control input can be used as the action of the agent, that is, a = [u 1s ,u 2s ,u 3s ] T , where u 1s ,u 2s ,u 3s They represent the slow components of the fuel flow, steam flow and drum feedwater flow control valve openings. Each valve can rotate in the range of (0,1), and each control variable is a continuous action space, 0 means fully closed, and 1 means fully open.

[0141] (2) S is the state space, which represents the set of information observed by the agent. Select the slow component of the system state, the boiler drum steam pressure x 1s (kg / cm 2 ), turbine power x 2s (MW) and the fluid density x in the boiler drum 3s (kg / cm 3 ), the slow component of the current state and the target state T s The deviation between the slow components is e = [e1, e2, e3] T The complete observation information can be described as

[0142] s=[x 1s ,x 2s ,x 3s ,e1,e2,e3] T (73)

[0143] (3) r is the reward function, which is used to evaluate the goals and effects of the agent when performing actions. s The reward becomes smaller when the system state is close to the target state T s When the slow component is large, the reward becomes larger, so the reward r is set to

[0144] r=-ω1||e||-ω2a (74)

[0145] Among them, ω1 and ω2 are the tracking error penalty weight and the control consumption penalty weight, ||e|| represents the difference between the current system state and the target state T s The norm of the tracking error between .

[0146] (4)γ is a discount factor, which is used to measure the importance of future rewards. Specifically, the discount factor determines the degree to which the agent reduces future rewards;

[0147] Step 202, initialize two critic networks and and an actor network π φ , and initialize with random parameters θ1, θ2, φ, and assign the initial values ​​to the target network:

[0148] θ1′←θ1,θ2′←θ2,φ′←φ(75)

[0149] Where θ1′ and θ2′ are the parameters of the target critic network, and φ′ is the parameter of the target actor network.

[0150] Step 203: When the agent is training, the current state at the current time k is represented as s k ∈S. When taking action a t After π(s)+∈, ε~N(0,σ), it transfers to the next state s k+1 and obtain a reward r from the environment k , and store the data sequence (s, a, r, s′) of each time step into the experience pool;

[0151] Step 204, define Used to describe the coal-fired power generation system in state s at a specific time k. k When , execute action a k , and then the long-term expected control cost function of the coordinated control strategy π(s) is implemented, that is,

[0152]

[0153] Step 205: extract N (s, a, r, s′) data sequences from the experience pool for neural network training. When the state is s, select an action according to the target actor network. Using two target critic networks and Evaluate state s′ and target action The value of Q The minimum value among the values ​​to reduce the overestimation bias

[0154]

[0155] Step 206, the loss of the critic network is a measure of the difference between the predicted Q value and the target Q value

[0156]

[0157] Where B is the batch size to be extracted.

[0158] Step 207, randomly initialize the running mean μ of the critic network loss c1 and μ c2 , the decay rate β c1 and β c2 , standard deviation multiple n c The decay rate β c1 and β c2 The adjustment factor used to dynamically adjust the variable update, through the smooth weighted method, the decay rate controls the degree of integration of the new value and the historical value;

[0159] Calculate the standard deviation based on the initial loss value Then, we can get the upper and lower bounds of loss based on the operation:

[0160]

[0161] Step 208: Update the new loss L according to the relationship between the running loss and the upper and lower limits. c :

[0162]

[0163] Step 209: Dynamically update the network loss mean μ c1 and μ c2 , used to solve the next loss upper and lower limits:

[0164]

[0165] Step 210, if Lcritic <μ c1 , dynamically adjust the Critic network learning rate:

[0166] α c ←(1-γ c )α c (82)

[0167] Among them, γ c Tune parameters for the Critic network learning rate.

[0168] If L critic >μ c1 , dynamically adjust the Critic network learning rate:

[0169] α c ←(1+γ c )α c (83)

[0170] Step 211, update critic network parameters:

[0171]

[0172] Step 212, continuously looping steps 206-211 until the training is completed and the optimal control strategy is obtained.

[0173] Step 213, the role of the actor network is to generate the optimal action under a given state, that is, to maximize the Q value output by the critic network. The loss of the actor network is defined as:

[0174]

[0175] Step 214, randomly initialize the running mean μ of the actor network loss a1 and μ a2 , the decay rate β a1 and β a2 , standard deviation multiple n c ; where the decay rate β a1 and β a2 , used to dynamically adjust the adjustment factor of variable updates. By smoothing weighted, the decay rate controls the degree of integration of new values ​​and historical values;

[0176] Calculate the standard deviation based on the initial loss value Then we can obtain the upper and lower bounds of the loss:

[0177]

[0178] Step 215: Update the new loss L according to the relationship between the running loss and the upper and lower limits. a :

[0179]

[0180] Step 216, dynamically update the network loss mean μ a1 and μ a2 , used to solve the next loss upper and lower limits:

[0181]

[0182] Step 217, if L actor <μ a1 , dynamically adjust the Actor network learning rate:

[0183] α a ←(1-γ a )α a (89)

[0184] Among them, γ a Tuning parameters for the Actor network learning rate.

[0185] If L actor >μ a1 , dynamically adjust the Actor network learning rate:

[0186] α a ←(1+γ a )α a (90)

[0187] Step 218, update actor network parameters:

[0188]

[0189] Step 219, continuously looping steps 213-218 until the training is completed and the optimal control strategy is obtained.

[0190] Step 220, dynamically adjust the batch size according to the current weighted loss. The weighted loss is L ac =tL actor +(1-t)L critic , where t is the weight factor, μ ac =tμ a1 +(1-t)μ c1 is the weighted average of the actor and critic losses. If L ac <μ ac , then increase the batch size

[0191] B←(1+ρ)B (92)where ρ is the batch size adjustment parameter.

[0192] If L ac >μ ac , then reduce the batch size

[0193] B←(1-ρ)B (93)

[0194] Step 3: For the stabilization problem of the fast subsystem, combined with the slow variation characteristics of the slow steam pressure, select appropriate fuzzy sets in its definition domain and calculate the membership degree, establish the TS fuzzy model, solve the algebraic Riccati equation to generate the controller gain and obtain the control input of the fast subsystem through the parallel compensation method; specifically, it includes the following:

[0195] Step 301, divide the domain of the slow state variable x1 into multiple fuzzy sets, and establish corresponding fuzzy rules. Each fuzzy rule is expressed as follows:

[0196] Fast subsystem rule i:

[0197] If x1 belongs to the fuzzy set

[0198] but

[0199]

[0200] Where i = 1, 2, ..., s, s is the number of fuzzy rules, A fi and B fi is the corresponding constant matrix.

[0201] Step 302: Select an appropriate fuzzy set based on the slow variable x1 and calculate its membership degree μ i (x1)

[0202]

[0203] Among them, a i is the left endpoint of the fuzzy set, b i is the midpoint of the fuzzy set, c i is the right endpoint of the fuzzy set.

[0204] Solving for membership

[0205]

[0206] Step 303, the overall TS fuzzy fast subsystem model is as follows:

[0207]

[0208] Step 304, with T f The fast subsystem is discretized for the sampling period, and the discretized fast subsystem is in the following form:

[0209]

[0210] in, hi (k) = h i (x1(k)).

[0211] Solve the following algebraic Riccati equation

[0212]

[0213] Among them, P i is the solution of the Riccati equation, and then the feedback gain matrix K i :

[0214]

[0215] Further, the fuzzy fast controller is obtained:

[0216]

[0217] Step 4: The fast and slow subsystem control inputs are combined and acted on the original system to obtain the next state and reward information. The agent updates the network parameters and optimizes the control strategy based on the feedback from the environment. The details are as follows:

[0218] Step 401, add the slow subsystem control input selected in step 204 and the fast subsystem control input obtained in step 305 to obtain a combined control input:

[0219] u=a k +u 2f (k) (101)

[0220] Step 402, the combined control input interacts with the original environment (69) to obtain the next state, reward and other information to provide data support for the subsequent training of the intelligent agent. Through this interaction process, the intelligent agent continuously updates the network parameters and optimizes the control strategy.

[0221] In order to enable those skilled in the art to better understand the present invention, the coordinated control method of a slow dynamic unknown dual-time scale coal-fired power generation system based on TS fuzzy and TD3 algorithm is described in detail below in combination with specific embodiments.

[0222] When designing a coordinated controller for a coal-fired power generation system based on the combination of TD3 algorithm and TS fuzzy algorithm, a 160MW coal-fired power generation system is taken as an example. For the fast subsystem, the TS fuzzy model is used to deal with the problem of slow variables in the system model. Specifically, by setting three fuzzy sets of steam pressure - low pressure (100 to 110), medium pressure (105 to 115), high pressure (110 to 120), the triangular membership medium a i , b i and c iare the left endpoint, midpoint and right endpoint of each fuzzy set respectively. For the slow subsystem, the TD3 algorithm framework is adopted, the actor and critic networks use ReLU as the activation function, and the output layer of the actor network uses the Tanh activation function. The optimizer uses Adam, the learning rate is 1e-4, the minimum batch size is 128, and the soft update rate ξ is set to 5e-3.

[0223] Set the initial state and target state to x(0) = [102, 60, 438.93] T and T s =[h1,h2,h3]=[121,90,389.92], the initial control input is a0=[0.3102,0.6711,0.3967]. The control cost function is

[0224] r=-ω1||e||-ω2a (102)

[0225] Through interaction with the actual coal-fired power generation environment, system operation data is collected and used for network training. In order to verify the control effect, this study conducted multiple operation tests in this environment. The average reward curve of the slow subsystem training process is shown in Figure 2. Figure 2 As shown in , the horizontal axis represents the number of training rounds and the vertical axis represents the corresponding reward value. The solid line is the average reward of multiple runs, and the shaded area represents the variance of the return. Figure 2 It can be seen that the reward curve gradually converges after about 600 training rounds. After the training, the TS fuzzy and TD3 algorithms are applied to the coal-fired power generation system to test its tracking target state T s The effect of the original system state variables x1, x2, x3 and the corresponding tracking targets h1, h2, h3 and the error trajectory diagrams are as follows: Figure 3 , Figure 4 and Figure 5 As shown, the trajectory of the system state variables under combined control is as follows: Figure 6 As shown, it can be seen that the system can eventually track the given target value.

[0226] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for coordinated control of a coal-fired power generation system, characterized in that: The steps include: Step 1, decompose the original coal-fired power generation system into a fast subsystem and a slow subsystem, and decompose the original control task into a stabilization task of the fast subsystem and a tracking task of the slow subsystem; Step 2, modeling the tracking task of the slow subsystem as a Markov decision process, defining a four-tuple (S, A, R, γ), where A is the action space, representing the set of operations performed by the reinforcement learning agent; S is the state space, and the slow component of the original coal-fired power generation system state is selected to construct the slow subsystem state data; R is the reward function; γ is the discount factor; the TD3 algorithm framework is adopted, and loss clipping, dynamic learning rate and batch size adjustment mechanism are introduced to optimize the training process, and the control input a of the slow subsystem is obtained. k ; Step 3: For the fast subsystem stabilization task, a TS fuzzy model is established, the controller gain is obtained by solving the algebraic Riccati equation, and the fast subsystem control input u is obtained by the parallel compensation method. 2f ; Step 4, fast system control input a k and the slow subsystem control input u 2f After the combination, the original coal-fired power generation system is acted on to obtain the state at the next moment, and the system returns to step 2 to calculate the reward information according to the reward function R. The intelligent agent optimizes the control strategy according to the state at the next moment and the reward information until the set maximum number of steps is reached.

2. A method for coordinated control of a coal-fired power generation system according to claim 1, characterized in that: The steps include: Step 101, the coal-fired power generation system is written in the following form: Among them, x1 represents the slow state variables of the coal-fired power generation system, namely, steam pressure and steam water density, x2 represents the fast state variables of the coal-fired power generation system, namely, output power, and u is the control input, namely, the valve opening of fuel flow, steam flow, and feed water flow. represents the derivative of x1, represents the derivative of x2, 0<ε<1 is the time scale parameter, which characterizes the degree of difference between the fast and slow dynamic time scales, and f, g are vector or matrix equations; Step 102, let ε = 0, and get x 2s =φ s (x 1s ,u s ,t), and substituting it into formula (1a) yields the state equation of the slow subsystem: The subscript "s" in the variable indicates the slow component in the variable, x 1s represents the slow component in variable x1, Representation variables The slow component in s represents the slow component in the variable φ, u s represents the slow component in the variable u; Step 103, the fast state variable x2 contains the slow component x 2s and fast component x 2f , that is, x2=x 2s +x 2f Based on formula (1b), the state equation of the fast subsystem is obtained The subscript "f" in the variable represents the fast component of the variable.

3. A method for coordinated control of a coal-fired power generation system according to claim 1, characterized in that: The performance index of the coal-fired power generation system is defined as Among them, Q and R are the weight matrices of error state and control input respectively, T s is the target state, u is the control input of the coal-fired power generation system, γ is the discount factor, x represents the current state, i represents the time step index, and k represents the current moment.

4. A method for coordinated control of a coal-fired power generation system according to claim 3, characterized in that: Performance index of fast subsystem J f and the performance index of the slow subsystem J s They are respectively expressed as follows: Among them, J f is the performance index of the fast subsystem, that is, the performance index of the system is optimized while ensuring the stability of the system; Q f and R f are the weight matrices corresponding to the state and control input of the fast subsystem respectively; J s It is the performance index of the slow subsystem, that is, the performance index is optimized while completing the tracking task; Q s and R s are the weight matrices corresponding to the error state and control input of the slow subsystem, x e is the deviation between the slow component of the current state and the slow component of the target state; Expresses the fast component x 2f The transpose of .

5. A method for coordinated control of a coal-fired power generation system according to claim 3, characterized in that: Step 2 specifically includes the following: Step 201, the tracking task of the slow subsystem is modeled as a Markov decision process. First, a four-tuple (S, A, R, γ) is defined, specifically: (1) A is the action space. For the coordinated control problem, the control input is the action of the agent, that is, a = [u 1s ,u 2s ,u 3s ] T , where u 1s Indicates the slow component of the fuel flow control valve opening, u 2s Indicates the slow component of the steam flow control valve opening, u 3s Indicates the slow component of the drum feedwater flow control valve opening; each valve can rotate in the range of (0,1), and each control variable is a continuous action space, 0 means fully closed, and 1 means fully open; (2) S is the state space, which represents the set of information observed by the agent; the slow component of the original system state is selected to construct the slow subsystem state data, the boiler drum steam pressure x 1s , turbine power x 2s and the fluid density x in the boiler drum 3s , the slow component x of the current state 1s 、x 2s 、x 3s With the target state T s The deviation between the slow components is e = [e1, e2, e3] T ; The complete observation information is described as s=[x 1s ,x 2s ,x 3s ,e1,e2,e3] T (6) (3) r is an immediate function, set to: r=-ω1||e||-ω2a (7) Among them, ω1 and ω2 are the tracking error penalty weight and the control consumption penalty weight, ||e|| represents the difference between the current system state and the target state T s The tracking error norm between ; (4)γ is a discount factor used to measure the importance of future rewards; Step 202: Initialize two critic networks using the TD3 algorithm framework and and an actor network π φ , and initialize with random parameters θ1, θ2, φ, and assign the initial values ​​to the target network: θ′1←θ1,θ′2←θ2,φ′←φ (8) Among them, θ1 and θ2 are critic networks respectively. and is the network parameter of the network π φ , θ1′ and θ2′ are the parameters of the target critic network, and φ′ is the parameter of the target actor network; Step 203: When the agent is training, the current state at time k is represented as s k ∈S; when taking action a t After π(s)+∈, ε~N(0,σ), the agent moves to the next state s k+1 And get the reward r at the current k moment from the environment k , and store the data sequence (s, a, r, s′) of each time step into the experience pool; Step 204, define Q π (s k ,a k ): Used to describe the coal-fired power generation system in state s at a specific time k. k When , execute action a k , and then the long-term expected control cost function of the coordinated control strategy π(s) is implemented, that is, E stands for Expectation Step 205: Extract B (s, a, r, s′) data sequences from the experience pool for training the actor and critic networks. When the state is s, select an action according to the target actor network. Using two target critic networks and Evaluate state s′ and target action The value of Minimum value among values Step 206, the loss of the critic network is a measure of the difference between the predicted Q value and the target Q value Where B is the batch size of the extraction; Step 207 introduces loss clipping and randomly initializes the running mean μ of the critic network loss c1 and μ c2 , the decay rate β c1 and β c2 , standard deviation multiple n c ; Calculate the standard deviation based on the initial loss value Then, we can get the upper and lower bounds of the critic network loss by running: Step 208: Based on the relationship between the running loss and the upper and lower limits, the new loss L of the critic network is updated. c : Step 209, dynamically update the running mean μ of the network loss c1 and μ c2 , used to solve the next loss upper and lower limits: Step 210, if L critic <μ c1 , dynamically adjust the critic network learning rate: a c ←(1-c c )a c (15) Among them, γ c Adjust parameters for critic network learning rate; If L critic >μ c1 , dynamically adjust the critic network learning rate: a c ←(1+c c )a c (16) Step 211, update critic network parameters: Step 212, continuously looping steps 206-211 until the training is completed and the optimal control strategy is obtained; In step 213, the actor network is used to generate the optimal action for a given state, that is, to maximize the Q value output by the critic network. The loss of the actor network is defined as: Step 214, randomly initialize the running mean μ of the actor network loss a1 and μ a2 , the decay rate β a1 and β a2 , standard deviation multiple n c , calculate the standard deviation based on the initial loss value Then we can obtain the upper and lower bounds of the actor network loss: Step 215: Update the new loss L of the actor network based on the relationship between the operating loss of the actor network and the upper and lower limits. a : Step 216, dynamically update the running mean μ of the actor network loss a1 and μ a2 , used to solve the next loss upper and lower limits: Step 217, if L actor <μ a1 , dynamically adjust the Actor network learning rate: a a ←(1-c a )a a (22) Among them, γ a Adjust parameters for the Actor network learning rate; If L actor >μ a1 , dynamically adjust the Actor network learning rate: a a ←(1+c a )a a (23) Step 218, update actor network parameters: Step 219, continuously looping steps 213-218 until the training is completed and the optimal control strategy is obtained; Step 220, dynamically adjust the batch size according to the current weighted loss, update the batch size B, return to step 205 to re-extract the data sequence, and train the actor and critic networks to obtain the optimal execution action a k ; The weighted loss is: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> ac <h2 style=";text-align:left;direction:ltr"> =tL<h2 style=";text-align:left;direction:ltr"> actor <h2 style=";text-align:left;direction:ltr"> +(1-t)L<h2 style=";text-align:left;direction:ltr"> critic <h2 style=";text-align:left;direction:ltr"> ; Where t is the weight factor; μ ac is the weighted average of the actor and critic losses μ ac =tμ a1 +(1-t)μ c1 ; If L ac <μ ac , then increase the batch size B←(1+ρ)B (25) Where ρ is the batch size adjustment parameter; If L ac >μ ac , then reduce the batch size B←(1-ρ)B (26).

6. A method for coordinated control of a coal-fired power generation system according to claim 1, characterized in that: The steps include: Step 3 is as follows: Step 301, dividing the domain of the slow state variable x1 into multiple fuzzy sets and establishing corresponding fuzzy rules; Each fuzzy rule is expressed as follows: Fast subsystem rule i: If the slow state variable x1 belongs to the fuzzy set but Where i = 1, 2, ..., n, n is the number of fuzzy rules, A fi and B fi is a constant matrix; Step 302: Select an appropriate fuzzy set based on the slow state variable x1 and calculate the membership degree μ of the slow state variable x1. i (x1) Among them, a i Fuzzy Set The left endpoint, b i Fuzzy Set The midpoint of i Fuzzy Set The right endpoint of ; Solving for membership j represents the index of the fuzzy set, and n represents the total number of fuzzy sets; Step 303, the overall TS fuzzy fast subsystem model is as follows: Step 304, with T f The fast subsystem is discretized for the sampling period, and the discretized fast subsystem is in the following form: in, h i (k) = h i (x1(k)); T f Represents the sampling period, s t The variable representing the time integral; Solve the following algebraic Riccati equation Among them, P i is the solution of the Riccati equation, and the feedback gain matrix K is obtained i , that is, the controller gain: Furthermore, the fuzzy fast controller is obtained through the parallel compensation method:

7. A method for coordinated control of a coal-fired power generation system according to claim 1, characterized in that: The method comprises the following steps: The combination of the fast system control input and the slow subsystem control input in step 4 is specifically: u=a k +u 2f (k)。

Images