Multi-condition adaptive control method for combined heat and power units considering deep reinforcement learning

Through deep reinforcement learning technology and MA-DDPG algorithm, a multi-condition adaptive control model was established, which solved the problem that traditional control methods could not cope with large-scale random output fluctuations and state parameter changes, and achieved rapid response and self-response of cogeneration units. Adapt to control, improving the overall control effect.

CN115421390BActive Publication Date: 2024-11-01HEILONGJIANG QINGXIANG THERMAL POWER CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211166934.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-23
Publication Date
2024-11-01
Estimated Expiration
2042-09-23

AI Technical Summary

Technical Problem

The traditional cogeneration unit control method cannot effectively deal with large-scale random output fluctuations and system state parameter changes, resulting in slow response speed and poor dynamic performance, and the fixed controller cannot adapt to the switching of multi-model control operating conditions.

Method used

Adopting deep reinforcement learning technology, establish a multi-condition adaptive control model, design the MA-DDPG algorithm to optimize control parameters, and realize fast load tracking of cogeneration units under multiple operating conditions through the synchronous operation of multiple agents and reward function assistance mechanism. and system adaptive control.

Benefits of technology

It significantly improves the adaptability of cogeneration units in complex environments, ensures system control reliability and fast load tracking, and optimizes the power generation load, steam extraction flow rate and pre-machine pressure control effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115421390B_ABST
    Figure CN115421390B_ABST
Patent Text Reader

Abstract

A multi-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning, taking into account that some state parameters of the CHP unit will undergo nonlinear changes when operating with random output in a wide range of conditions, a CHP unit state operation model characterizing different output conditions is established; based on the established CHP unit state operation model, a multi-condition adaptive control model is established taking into account the uncertain changes in system state parameters; for the parameter optimization problem of the control module in the multi-condition adaptive control model, a MA‑DDPG algorithm multi-condition adaptive control parameter optimization strategy is designed; through the above steps, the load rapid tracking of the cogeneration unit under multi-condition adaptive control is achieved. The control method of the present invention aims to achieve rapid load tracking of the cogeneration unit under multi-condition adaptive control while ensuring the reliability of system control, and significantly improve the adaptive ability of the control system in the face of uncertain and complex environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cogeneration technology control, and in particular to a multi-operating condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning. Background Art

[0002] With the continuous progress of the flexibility transformation of cogeneration, the large-scale installed combined heat and power (CHP) units in the "Three Norths" can participate in the rapid adjustment of grid flexibility within a wide range of output. When CHP units operate with a wide range of random output, the delay and nonlinear characteristics of the controlled objects will show significant differences, making the unit response process more complicated or even uncontrollable. Therefore, it is of great significance to carry out control research that considers the complex characteristics of the entire output condition to improve the rapid response of the unit in a wide range and stable heating, thereby meeting the requirements of grid flexibility operation.

[0003] At present, the traditional control method adopts the steam extraction heating regulation, which effectively improves the load response rate of the CHP unit under rated operating conditions while ensuring the heating quality. However, this control method only focuses on the rated output condition of the unit, and rarely pays attention to the influence of the unit's specific operating condition model on the large-scale random output fluctuation and the easy neglect of important information on the change of system state parameters, resulting in poor overall control effect; and the traditional control method of CHP units often uses a fixed controller, which cannot adapt to the multi-model control condition switching process. Summary of the invention

[0004] In order to further solve the problems that CHP units are affected by large-scale random output fluctuations in traditional control methods and are prone to ignoring important information such as changes in system state parameters, resulting in slow unit response speed and poor dynamic performance. The present invention provides a multi-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning, which aims to ensure the reliability of system control while achieving rapid load tracking of the cogeneration unit under multi-condition adaptive control, and significantly improve the adaptive ability of the control system in the face of uncertain and complex environments.

[0005] The technical solution adopted by the present invention is:

[0006] The multi-condition adaptive control method of a combined heat and power unit taking into account deep reinforcement learning includes the following steps:

[0007] Step 1: Considering that some state parameters of the CHP unit will undergo nonlinear changes when it is operating at random output in a wide range of conditions, a state operation model of the CHP unit under different output conditions is established;

[0008] Step 2: Based on the CHP unit state operation model established in step 1, a multi-condition adaptive control model is established taking into account the uncertain changes of system state parameters;

[0009] Step 3: Aiming at the parameter optimization problem of the control module in the multi-condition adaptive control model, a multi-condition adaptive control parameter optimization strategy of the MA-DDPG algorithm is designed;

[0010] Through the above steps, rapid load tracking of the cogeneration unit under multi-condition adaptive control is achieved.

[0011] In step 1, when the CHP unit is operating randomly under a wide range of conditions, some state parameters will undergo nonlinear changes. Its dynamic response process is a complex process with multiple conditions including mechanical energy conversion, heat exchange and two-phase flow. The nonlinear dynamic process of a typical extraction steam CHP unit is:

[0012] Inertia description of positive pressure direct blowing powder making system:

[0013]

[0014] In formula (1), T b is the milling inertia; t is the time, unit: s; r B (t) is the boiler combustion conversion rate, unit: t / h; is the derivative of boiler combustion conversion rate with respect to time t; u B is the coal feeding amount, unit: t / h; τ is the pulverizing delay time, unit: s;

[0015] Boiler energy balance equation:

[0016]

[0017] In formula (2), C b is the heat storage coefficient; p d is the drum pressure, unit: MPa; is the derivative of the drum pressure with respect to time t; K3 is the turbine gain; p T is the pressure before the machine, unit: MPa; u T is the high-speed valve opening, unit: %; K1 is the fuel gain; r B It is the boiler combustion rate, unit: t / h.

[0018] Description of the pressure difference characteristics before the superheater:

[0019] p T =p d -K2(K1r B ) 1.5 (3);

[0020] In formula (3), p T is the pressure before the machine, unit: MPa; p dis the drum pressure, unit: MPa; K2 is the pressure difference fitting coefficient; K1 is the fuel gain; r B is the boiler combustion rate.

[0021] Steam turbine energy balance equation:

[0022]

[0023] In formula (4), T t is the inertia time of the steam turbine, unit: s; P e is the generating load of the unit, unit: MW; is the derivative of the unit's power generation load with respect to time t; K5 is the steam work gain of the low-pressure cylinder; p z is the heating extraction steam pressure, unit: MPa; u H is the opening of the extraction butterfly valve, unit: %; K4 is the proportion of high and medium pressure cylinders in the turbine work; K3 is the turbine gain; p T is the pressure before the machine, unit: MPa; u T It is the high-profile door opening, unit: %.

[0024] Dynamic relationship of intermediate extraction steam pressure:

[0025]

[0026] In formula (5), C h is the heat storage coefficient of the heating network heater; p z is the heating extraction steam pressure, unit: MPa; is the derivative of the heating extraction steam pressure with respect to time t; K6 is the effective specific heat capacity of the heating network circulating water; q b is the circulating water flow rate of the heating network, unit: t / h; t c is the return water temperature of the heat network circulating water, unit: °C; K4 is the proportion of high and medium pressure cylinders in the turbine work; K3 is the turbine gain; p T is the pressure before the machine, unit: MPa; u T is the high-pressure valve opening, unit: %; K5 is the steam work gain of the low-pressure cylinder; u H is the opening of the steam extraction butterfly valve, unit: %.

[0027] Extraction steam flow calculation:

[0028] q m =K7K6q b (96p z -t c +103)(6);

[0029] In formula (6), q m is the extraction steam flow rate, unit: t / h; K7 is the effective heat of the extraction steam for heating converted into steam flow coefficient; K6 is the effective specific heat capacity of the circulating water in the heating network; qb is the circulating water flow rate of the heating network, unit: t / h; p z Heating steam extraction pressure, unit: MPa; t c It is the return water temperature of the heating network circulating water, unit: ℃.

[0030] When the unit is operating randomly under a wide range of working conditions, the state parameters of the controlled object represented by the above equations (1) to (6) have obvious nonlinear changes. The present invention considers them as variables and establishes a more accurate CHP unit state operation model characterizing different output conditions, as shown below:

[0031]

[0032] In formula (7): F i is the i-th operating condition model divided; T b,i is the milling inertia under the i-th working condition model; r B (t) is the boiler combustion conversion rate, unit: t / h; is the derivative of boiler combustion conversion rate with respect to time t; u B,i is the coal feeding amount under the i-th operating model, unit: t / h; τ i is the milling delay time under the i-th working condition model, unit: s;

[0033] C b,i is the heat storage coefficient under the i-th operating condition model; p d is the drum pressure, unit: MPa; K3 is the turbine gain; p T is the pressure before the machine, unit: MPa; u T,i is the high-pressure valve opening under the i-th working condition model, unit: %; K 1,i is the fuel gain under the i-th operating condition model;

[0034] K2 is the pressure difference fitting coefficient; T t is the inertia time of the steam turbine, unit: s; P e is the generating load of the unit, unit: MW; K5 is the steam work gain of the low-pressure cylinder; p z is the heating extraction steam pressure, unit: MPa;

[0035] u H,i is the opening of the extraction butterfly valve under the i-th operating model, unit: %; K4 is the proportion of high and medium pressure cylinders in the turbine work;

[0036] C h is the heat storage coefficient of the heating network heater; K6 is the effective specific heat capacity of the circulating water in the heating network; q b,i is the circulating water flow rate of the heating network under the i-th operating model, unit: t / h; t c,iis the return water temperature of the heat network circulating water under the i-th operating model, unit: °C; q m is the extraction steam flow rate, unit: t / h; K7 is the effective heat of the heating extraction steam converted into steam flow coefficient; is the derivative of boiler combustion conversion rate with respect to time t; is the derivative of the drum pressure with respect to time t; is the derivative of the unit's power generation load with respect to time t; Derivative of the heating extraction steam pressure with respect to time t.

[0037] In step 2, the process of establishing a multi-condition adaptive control model includes two processes: multi-condition model switching and multivariable control system PID adaptive control design.

[0038] Among them, the multivariable control system has three control loops. The CHP unit sets the pressure deviation △p T Sent to the first high-pressure valve opening control loop, the power generation load deviation △P e The second butterfly valve opening control loop is fed with the third fuel quantity control loop input. e and extraction steam flow disturbance △q m Equivalent power generation load deviation composition.

[0039] Therefore, the CHP unit control system is:

[0040]

[0041] In formula (8), Y is the output matrix of the three control loops, where △y i ,y i,0 , △x i , k p,i , k I,i , k d,i They are the output increment, output initial value, input deviation and three control parameters of the i-th control loop controller, which determine the regulation performance of the unit.

[0042] The multi-condition model switching is completed by the integral function switching mechanism, which specifically includes the following steps:

[0043] S3.1: According to formula (7), a model library containing n working conditions is constructed to cover the operating characteristics of the entire range of the controlled object:

[0044] Ω={F i |i=1,2,…,n}(9);

[0045] In formula (9): F i is the system model corresponding to the i-th working condition; Ω is the model element F i A collection of

[0046] S3.2: Establish the operating condition model control system according to formula (8):

[0047] C={Y i |i=1,2,…n}(10);

[0048] In formula (10): Y i is the sub-control system corresponding to the i-th operating condition sub-model, and C is its set.

[0049] S3.3: Use the integral function switching index to evaluate the matching degree between each working condition sub-model and the current controlled object, and switch to the optimal working condition model F o :

[0050]

[0051]

[0052] In formula (11): J is the switching mechanism; △x i,j (t) is the jth control loop input x between the ith operating condition model and the actual controlled object i.j (t) and x ref,j (t); α and β are the weights of the current transient error and the error within the memory length; γ is the forgetting factor to ensure that J i Convergence of (t), α>0, β>0, γ>0.

[0053] S3.4: Achieve disturbance-free switching by delayed switching method. Suppose sampling time t, the system sampling pair (F i ,Y i ), the next sampling time J k =min{J i (t)}, the switch is as follows:

[0054]

[0055] In formula (12): F i is the system model corresponding to the i-th working condition; Y i is the sub-control system corresponding to the i-th operating condition sub-model; J i is the switching mechanism corresponding to the i-th operating condition sub-model; δ is the lag time;

[0056] The best matching working condition model can be obtained by the integral function switching mechanism, and its output With the unit load command x ref (t) is subtracted to form a closed-loop optimal operating condition control model.

[0057] In step 3, the MA-DDPG algorithm is a multi-agent synchronous operation mechanism proposed on the basis of the traditional DDPG algorithm. The MA-DDPG algorithm takes into account the multi-loop control structure and control indicators of the rapid response of the power generation load control loop of the cogeneration unit, the heating stability of the extraction steam flow control loop, and the operating safety of the front pressure control loop, ensuring that each agent trains the corresponding control loop relatively independently, overcoming the shortcomings of the long training cycle and difficult convergence of a single agent for multiple control loops; at the same time, with the help of the reward function auxiliary mechanism, the learning actions between the agents are coordinated to achieve the optimal decision-making of each agent loop and the optimal global control, effectively improving the control effect of the entire output range of the unit.

[0058] For the MA-DDPG algorithm, at any optimization time, the three agents jointly explore the working condition sub-model environment, use the experience replay pool R to quickly update their respective network parameters and strategy formulation with N random samples within the training cycle T, and timely use the minimization loss function L(Q) and deterministic policy gradient (DPG) training to update Q and μ network parameters. Therefore, the gradient update formula of each agent is:

[0059]

[0060]

[0061] In formula (13) and formula (14), N is the total number of samples; i is the random sample number of the playback pool,

[0062]

[0063] is the response performance index penalty of the controller controlled variable; s is the current state; a is the decision action; ω is the Q value iteration discount factor; μ(s|θ μ ) is the Actor network used to make decision actions; Q(s,a|θ Q ) is the Critic network used for action evaluation; The target network copied by the current network μ, Q; For the status Next action Q value; θ Q , θ μ , The current network Q, μ network parameters and target network of each Agent's Critic and Actor network parameters.

[0064] Next, update the parameters θ of the Agent's Critic network and Actor network Q , and θ μ ,

[0065]

[0066]

[0067]

[0068]

[0069] Where: and are Q, μ and Network parameters; φ and are the current network learning weight values ​​of Critic and Actor respectively; σ is the target network update weight; j is the number of the agent; Y j is the target value function of the jth agent; c is the number of steps delayed by the target network relative to the current network;

[0070] In step 3, the design of the reward function auxiliary mechanism should ensure that the decisions of the three agents are relatively independent, coordinate the learning actions of each agent, and achieve the optimal decision of each agent and the global optimal decision:

[0071]

[0072]

[0073]

[0074] In formula (19) to formula (21): u T0 is the initial value of high-pressure door opening; u B0 is the initial value of coal feeding; t is the number of iterations; r 1,t 、r 2,t 、r 3,t are the response performance index penalties of the controlled variables of the PID controllers of the three agents respectively; △x 1,t , △x 2,t , △x 3,t are the load deviation values ​​of the 1st, 2nd and 3rd control loops respectively; △x 20 is the initial deviation of the generating load of the second control loop; ξ0, ξ1, ξ2, and ξ3 are obtained through experiments, ξ0=2, ξ1=5, ξ2=3, and ξ3=15 respectively.

[0075] Since the multi-agent DDPG synchronously trains multiple agents to obtain the maximum cumulative reward respectively, the multi-objective control problem of the invention is equivalent to maximizing the expected reward value under the optimal response strategy of each control loop controller:

[0076] MaxE[R i |π i ],i=1,2,3(22);

[0077] In formula (22): π i They respectively represent the total reward and penalty value and the optimal response strategy of the i-th controller acting on the operating condition sub-model within period T.

[0078] In step 3, the multi-condition adaptive control parameter optimization strategy of the MA-DDPG algorithm is taken into account, and the specific process is as follows:

[0079] Step 1: Multi-agent Actor Network μ(s|θ μ ), according to the current state s k Decision action a.

[0080] Step 2: Get the new state s by executing decision action a k+1 , and calculate the reward value r through equations (19), (20), and (21) 1,t 、r 2,t 、r 3,t .

[0081] Step 3: Critic network Q(s,a|θ Q ) estimates the current action strategy, and in the process of interacting with the environment, through the offline training-online testing operation mode, the intelligent agent continuously evaluates the Q value and corrects the strategy to achieve the optimal decision for the expected return and obtain the optimal sub-control system of the current working condition model.

[0082] During the offline training process, the basic thermal and electrical demand of the operating condition sub-model should be kept unchanged, and then as many random disturbances as possible should be considered. In the process of repeated iterations of "action-evaluation", the optimal μ and Q networks are obtained, and the corresponding sub-control systems are obtained, thereby constructing an offline database of operating condition sub-model-sub-control system pairs.

[0083] During the online learning decision-making process, the system continuously outputs offline experience data to guide the online adaptive learning process, that is, it can use the online reserve knowledge to ultra-fast update the μ and Q network parameters of each agent. When there is a difference between the actual environment and the offline environment, DDPG quickly adapts to the new operating environment, thereby online guiding the CHP unit control system to respond to demand faster and more effectively.

[0084] The present invention provides a multi-operating-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning, and the technical effects are as follows:

[0085] 1) The present invention proposes multi-model control to solve the complex problems of nonlinearity, variable operating conditions and parameter uncertainty, and provides a new idea for solving the problem of state parameter changes during a large range of CHP unit output.

[0086] 2) The present invention designs a multi-condition adaptive control parameter optimization strategy for the MA-DDPG algorithm, which integrates the reward value optimization model of deep reinforcement learning and the ability to quickly optimize and make decisions on high-dimensional data. After introducing it into the adaptive design link of the control system, it can significantly improve the adaptive ability in complex environments while ensuring the reliability of system control.

[0087] 3) The present invention takes into account the changes in the dynamic response characteristics of the CHP unit under different output conditions, and integrates deep reinforcement learning into the multi-model control module. It utilizes its reward value optimization model and the ability to quickly optimize and make decisions on high-dimensional data. On the basis of ensuring the reliability of system control, it significantly improves the adaptability of the CHP unit in the face of complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is the block diagram of the adaptive control model for the multi-condition model.

[0089] Figure 2 Design a flow chart for the DDPG multi-condition model adaptive control.

[0090] Figure 3 Schematic diagram of DDPG multi-agent synchronous operation.

[0091] Figure 4 This is a diagram of the multi-agent DDPG offline and online interaction model.

[0092] Figure 5(a) is a graph showing the error response curve of the power generation load of a typical CHP unit under different model control methods;

[0093] Figure 5(b) is a graph showing the error response curve of the extraction steam flow rate under typical operating conditions of the CHP unit under different model control methods;

[0094] Figure 5(c) is the error response curve of the pressure before the CHP unit under typical operating conditions under different model control methods.

[0095] Figure 6 Cumulative reward curve for different algorithm training processes.

[0096] Figure 7(a) is a cluster diagram of the error response curves of the CHP unit under different operating conditions for step disturbance power generation load;

[0097] Fig. 7(b) is a cluster diagram of the error response curves of extraction steam flow rate under step disturbance in different operating conditions of CHP unit;

[0098] Figure 7(c) is a cluster diagram of the pressure error response curves before the CHP unit to step disturbances under different operating conditions.

[0099] Figure 8(a) is a cluster diagram of the error response curves of the continuous disturbance power generation load under low load conditions of the CHP unit;

[0100] Fig. 8(b) is a cluster diagram of the error response curves of the continuous disturbance extraction steam flow rate under low load conditions of the CHP unit;

[0101] Figure 8(c) is a cluster diagram of the pressure error response curves before continuous disturbance of the CHP unit under low load conditions. DETAILED DESCRIPTION

[0102] The multi-condition adaptive control method of a combined heat and power unit taking into account deep reinforcement learning includes the following steps:

[0103] Step 1: Considering that some state parameters of the CHP unit will undergo nonlinear changes when it is operating at random output in a wide range of conditions, a state operation model of the CHP unit under different output conditions is established;

[0104] Step 2: Based on the CHP unit state operation model established in step 1, taking into account the uncertain changes in system state parameters, a multi-condition adaptive control model is established;

[0105] Step 3: Aiming at the parameter optimization problem of the control module in the multi-condition adaptive control model, a multi-condition adaptive control parameter optimization strategy of the MA-DDPG algorithm is designed;

[0106] Through the above steps, rapid load tracking of the cogeneration unit under multi-condition adaptive control is achieved.

[0107] Based on the multivariable feedback control architecture of CHP units, a multi-condition adaptive control model is proposed which takes into account the uncertain changes of system state parameters. That is, multiple condition sub-models are constructed to accurately approximate the characteristics of the entire output range of the unit to eliminate the influence of the uncertain changing parameters of the system on the unit control. The corresponding optimal sub-control system is selected for multiple condition sub-models by using the integral function switching mechanism, so as to construct an offline database of condition sub-model-sub-control system pairs. Figure 1 shown.

[0108] On this basis, in view of the poor adaptability of fixed controllers in traditional control methods and the limited model library models, an online adaptive control method of operating condition sub-model-sub-control system based on multi-agent deep deterministic policy gradient is proposed. That is, the DDPG offline database is used to quickly guide the online operation decision-making process, and the online rapid adaptive control of the complex control requirements of the unit in a large range of output range is realized. Specifically, Figure 2 As shown. Considering the multi-loop control structure and its control indexes of the rapid response of the power generation load control loop of the cogeneration unit, the heating stability of the extraction steam flow control loop, and the operation safety of the front pressure control loop, a MA-DDPG synchronous operation strategy is proposed to ensure that each agent trains the corresponding control loop relatively independently, overcoming the shortcomings of long training cycle and difficult convergence of single agent for multiple control loops; at the same time, with the help of the reward function auxiliary mechanism, the learning actions between the agents are coordinated to achieve the optimal decision-making of each agent loop and the optimal global control, effectively improving the overall control effect of the entire output range of the unit.

[0109] Example:

[0110] Taking the 330MW extraction steam CHP unit as an example, the corresponding simulation model is built in Matlab2020b / Simulink. The changes in key state parameters of the unit model under different operating conditions are shown in Table 1.

[0111] Table 1 Changes in key parameters of typical CHP unit output conditions

[0112]

[0113] The invention designs the following example to verify the effectiveness of the DDPG adaptive control method for multi-condition models:

[0114] Example 1: Comparison of the effect of the ideal full-condition model control method and the actual multi-condition model control method;

[0115] This example analyzes the effectiveness of the multi-condition model control method taking into account the changes in system state parameters. After the CHP unit response reaches a steady state, a +20MW load disturbance is applied at 1800s, and the error response curve clusters of the unit under different model control methods are observed in Figure 5.

[0116] Table 2 Generation load error response index under different model control methods

[0117]

[0118] Table 2 shows the performance index of its power generation load error response curve. The unit control system adopts a fixed controller with the following control parameters: [(12, 0.1, 0), (25, 0.0004, 0), (0.6, 0.0028, 0)].

[0119] contrast Figure 5(a) to Figure 5(c) It can be seen from the dashed and solid lines and Table 2 that when the CHP unit is operated under a wide range of load conditions, the load rise time is extended from 244.11s to 360.30s on average, the extraction flow rate responds slowly, the steady-state error reaches 35t / h, the unit response process fluctuation trend is complex, the transition time is different, and the tracking effect is significantly different. It can be seen that the actual state parameter changes during a wide range of conditions have a great impact on the unit characteristics. The ideal full-condition model control method cannot accurately describe the entire interval characteristics of the unit, and the control effect is also poor.

[0120] Example 2: For the multi-condition model control method, the adaptive convergence performance analysis of the MA-DDPG proposed by the invention is compared with that of other algorithms;

[0121] This example compares the convergence performance of the proposed MA-DDPG with other algorithms. Taking Agent2, which trains the power generation load control loop in the unit sub-control system, as an example, the training process of different algorithms such as MA-DDPG, DDPG and DQN is as follows: Figure 6 shown.

[0122] from Figure 6 It can be seen that at the beginning of training, the cumulative rewards of different algorithm agents are all negative, and the learning effect is poor. However, in the process of MA-DDPG where multiple agents learn from each other and make independent decisions, the reward value continues to increase until it basically converges after about 75 iterations.

[0123] Table 3 Algorithm performance comparison

[0124]

[0125] As shown in Table 3, compared with the benchmark methods DDPG, DQN and GA algorithms, the cumulative reward value of the MA-DDPG proposed in the invention has effectively increased by 18.76%, 38.28% and 12.55%. It can be seen that the collaboration of multiple agents has a faster convergence speed and better training effect than the continuous and discrete action algorithms of a single agent. In addition, Table 3 also shows the average running time of MA-DDPG, DDPG, DQN and GA algorithms, confirming the convergence speed advantage of deep reinforcement learning. This is due to the rapid guidance of offline knowledge in the online testing link, which avoids a lot of iterative work.

[0126] Example 3: Comparison of the control effect of step disturbance under a wide range of output conditions of the invented multi-condition model under the traditional control scheme and the DDPG adaptive control scheme;

[0127] This example verifies the control effect of the multi-operating condition model on a wide range of operating condition step disturbances under the DDPG adaptive control scheme.

[0128] Table 4 Typical operating condition model-sub-control system pair adaptation results

[0129]

[0130]

[0131] Table 4 shows the adaptive results of the operating sub-model and sub-control system of MA-DDPG. After the CHP unit reaches a steady state, a +20MW load disturbance is applied at 1800s. The results are compared under the conventional control scheme and the adaptive control scheme of the invention. Figure 5(a) to Figure 5(c) A cluster of error response curves.

[0132] Depend on Figure 7(a) to Figure 7(c) As shown in Table 4, compared with the traditional control scheme, after the CHP unit adopts the adaptive control scheme, the power generation load rise time is greatly shortened from 252 to 345 seconds to 57 to 82 seconds, and the response time is reduced by an average of 37.34%; the extraction steam flow error quickly stabilizes to near zero after a short-term fluctuation; the peak pressure in front of the machine is reduced by an average of 25.07%, effectively avoiding the safety instability phenomenon. It can be seen that under the adaptive action of the condition sub-model-sub-control system of MA-DDPG, the CHP unit can respond to the load demand quickly and accurately in a wide range without affecting the heating and operation safety.

[0133] Example 4: Comparison of the continuous disturbance control effect of the invented multi-condition model under the traditional control scheme and the DDPG adaptive control scheme in low-load output conditions.

[0134] This example verifies the control effect of the multi-condition model on continuous disturbances under low load conditions under the DDPG adaptive control scheme. The experiment was carried out under 30% to 50% THA low load conditions. The CHP unit power generation load fluctuates between 83 and 160MW, operates at a constant pressure of 9.96MPa, and the extraction steam flow rate is maintained at 220t / h. The corresponding simulation results are shown in Figure 8(a) to Figure 8(c) The error response curve clusters based on the traditional control scheme and the adaptive control scheme are shown in Table 5. The response indicators are shown in Table 5.

[0135] Table 5 Step disturbance response indicators of CHP units under different operating conditions

[0136]

[0137] Depend on Figure 8(a) to Figure 8(c) As shown in Table 5, compared with the slow response of power generation load and the long-term, large fluctuation and instability trend of extraction steam flow and front-end pressure under the traditional control scheme, the invented adaptive control scheme reduces the unit power generation load rise time by an average of 79.54%, and the response time is greatly shortened from 220-445s to 71-162s, and the transition of the extraction steam flow and front-end pressure response process is extremely rapid and smooth.

[0138] Table 6 Continuous disturbance response indicators of CHP units under low load conditions

[0139]

[0140] In summary, according to Table 5, Table 6 and Figure 7(a) to Figure 7(c) , Figure 8(a) to Figure 8(c) Analysis shows that, in the face of a wide range of random output requirements in the entire operating range, the multi-operating condition model DDPG adaptive control of power generation load rapidity, heating stability and operation safety are superior to traditional control.

Claims

1. A multi-condition adaptive control method for a combined heat and power unit taking into account deep reinforcement learning, characterized in that The following steps are involved: Step 1: Establish a CHP unit state operation model that characterizes different output conditions; Step 2: Based on the CHP unit state operation model established in step 1, a multi-condition adaptive control model is established taking into account the uncertain changes of system state parameters; Step 3: Aiming at the parameter optimization problem of the control module in the multi-condition adaptive control model, a multi-condition adaptive control parameter optimization strategy of the MA-DDPG algorithm is designed; In step 2, the multi-condition adaptive control model process is established, including two processes: multi-condition model switching and multivariable control system PID adaptive control design; Among them, the multivariable control system has three control loops. The CHP unit sets the pressure deviation △p T Sent to the first high-pressure valve opening control loop, the power generation load deviation △P e The second butterfly valve opening control loop is fed with the third fuel quantity control loop input. e and extraction steam flow disturbance △q m Equivalent generation load deviation composition; Therefore, the CHP unit control system is: In formula (8), Y is the output matrix of the three control loops, where △y i ,y i,0 , △x i , k p,i , k I,i , k d,i are the output increment, output initial value, input deviation and three control parameters of the controller of the i-th control loop, which determine the regulating performance of the unit; The multi-condition model switching is completed by the integral function switching mechanism, which specifically includes the following steps: S3.1: According to formula (7), a model library containing n working conditions is constructed to cover the operating characteristics of the entire range of the controlled object: Ω={F i |i=1,2,...,n}(9); In formula (9): F i is the system model corresponding to the i-th working condition; Ω is the model element F i A collection of S3.2: Establish the operating condition model control system according to formula (8): C={Y i |i=1,2,…n}(10); In formula (10): Y i is the sub-control system corresponding to the i-th working condition sub-model, and C is its set; S3.3: Use the integral function switching index to evaluate the matching degree between each working condition sub-model and the current controlled object, and switch to the optimal working condition model F o : In formula (11): J is the switching mechanism; △x i,j (t) is the jth control loop input x between the ith operating condition model and the actual controlled object i.j (t) and x ref,j (t); α and β are the weights of the current transient error and the error within the memory length; γ is the forgetting factor to ensure that J i (t) convergence, α>0, β>0, γ>0; S3.4: Assume sampling time t, system sampling pair (F i ,Y i ), the next sampling time J k =min{J i (t)}, the switch is as follows: In formula (12): F i is the system model corresponding to the i-th working condition; Y i is the sub-control system corresponding to the i-th operating condition sub-model; J i is the switching mechanism corresponding to the i-th operating condition sub-model; δ is the lag time; The best matching working condition model can be obtained by the integral function switching mechanism, and its output With the unit load command x ref (t) is subtracted to form a closed-loop optimal operating condition control model; In step 3, for the MA-DDPG algorithm, at any optimization time, the three agents jointly explore the working condition sub-model environment, use the experience replay pool R to quickly update their respective network parameters and strategy formulation with N random samples within the training cycle T, and timely update the Q and μ network parameters by minimizing the loss function L(Q) and deterministic policy gradient DPG training; therefore, the gradient update formula of each agent is: In formula (13) and formula (14), N is the total number of samples; i is the random sample number of the playback pool; is the response performance index penalty of the controller controlled variable; s is the current state; a is the decision action; ω is the Q value iteration discount factor; μ(s|θ μ ) is the Actor network used to make decision actions; Q(s,a|θ Q ) is the Critic network used for action evaluation; The target network copied by the current network μ, Q; For the status Next action Q value; θ Q , θ μ , The current network Q, μ network parameters and target network of each Agent's Critic and Actor Network parameters; Next, update the parameters θ of the Agent's Critic network and Actor network Q , and θ μ , Where: and are Q, μ and Network parameters; φ and are the current network learning weight values ​​of Critic and Actor respectively; σ is the target network update weight; j is the number of the agent; Y j is the target value function of the jth agent; c is the number of steps delayed by the target network relative to the current network; Through the above steps, rapid load tracking of the cogeneration unit under multi-condition adaptive control is achieved.

2. The multi-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning according to claim 1, characterized in that: In step 1, when the CHP unit is operating randomly under a wide range of conditions, some state parameters will undergo nonlinear changes. Its dynamic response process is a complex process with multiple conditions including mechanical energy conversion, heat exchange and two-phase flow. The nonlinear dynamic process of a typical extraction steam CHP unit is: Inertia description of positive pressure direct blowing powder making system: In formula (1), T b is the milling inertia; t is the time; r B (t) is the boiler combustion conversion rate; is the derivative of boiler combustion conversion rate with respect to time t; u B is the coal feeding amount; τ is the pulverizing delay time s; Boiler energy balance equation: In formula (2), C b is the heat storage coefficient; p d is the drum pressure; is the derivative of the drum pressure with respect to time t; K3 is the turbine gain; p T is the pressure before the machine; u T is the high-speed valve opening; K1 is the fuel gain; r B is the boiler combustion rate; Description of the pressure difference characteristics before the superheater: p T =p d -K2(K1r B ) 1.5 (3); In formula (3), p T is the pressure before the machine; p d is the drum pressure; K2 is the pressure difference fitting coefficient; K1 is the fuel gain; r B is the boiler combustion rate; Steam turbine energy balance equation: In formula (4), T t is the inertia time of the steam turbine; P e The power generation load of the unit; is the derivative of the unit's power generation load with respect to time t; K5 is the steam work gain of the low-pressure cylinder; p z is the heating extraction steam pressure; u H is the opening of the extraction butterfly valve; K4 is the proportion of high and medium pressure cylinders in the turbine work; K3 is the turbine gain; p T is the pressure before the machine; u T For high-profile door opening; Dynamic relationship of intermediate extraction steam pressure: In formula (5), C h is the heat storage coefficient of the heating network heater; p z is the extraction steam pressure for heating; is the derivative of the heating extraction steam pressure with respect to time t; K6 is the effective specific heat capacity of the heating network circulating water; q b is the circulating water flow rate of the heating network; t c is the return water temperature of the heat network circulating water; K4 is the proportion of high and medium pressure cylinders in the turbine work; K3 is the turbine gain; p T is the pressure before the machine; u T is the high-pressure valve opening; K5 is the steam work gain of the low-pressure cylinder; u H is the opening of the extraction butterfly valve; Extraction steam flow calculation: q m =K7K6q b (96p z -t c +103) (6); In formula (6), q m is the extraction steam flow rate; K7 is the effective heat of the extraction steam for heating converted into steam flow coefficient; K6 is the effective specific heat capacity of the circulating water in the heating network; q b is the circulating water flow of the heating network; p z Heating extraction steam pressure; t c The return water temperature of the heat network circulating water; When the unit operates randomly under a wide range of operating conditions, the state parameters of the controlled object represented by the above equations (1) to (6) change significantly in a nonlinear manner.

3. The multi-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning according to claim 2 is characterized in that: A CHP unit state operation model is established to characterize different output conditions, as shown below: In formula (7): F i is the i-th operating condition model divided; T b,i is the milling inertia under the i-th working condition model; r B (t) is the boiler combustion conversion rate; is the derivative of boiler combustion conversion rate with respect to time t; u B,i is the coal feeding amount under the i-th operating model; τ i is the milling delay time under the i-th operating model; C b,i is the heat storage coefficient under the i-th operating condition model; p d is the drum pressure; K3 is the turbine gain; p T is the pressure before the machine; u T,i is the high-pressure door opening under the i-th working condition model; K 1,i is the fuel gain under the i-th operating condition model; K2 is the pressure difference fitting coefficient; T t is the inertia time of the steam turbine; P e is the generating load of the unit; K5 is the steam work gain of the low-pressure cylinder; p z is the extraction steam pressure for heating; u H,i is the opening of the extraction butterfly valve under the i-th operating model; K4 is the proportion of high and medium pressure cylinders in the turbine work; C h is the heat storage coefficient of the heating network heater; K6 is the effective specific heat capacity of the circulating water in the heating network; q b,i is the circulating water flow rate of the heating network under the i-th operating model; t c,i is the return water temperature of the heat network circulating water under the i-th operating model; q m is the extraction steam flow rate; K7 is the steam flow coefficient converted from the effective heat of the heating extraction steam; is the derivative of boiler combustion conversion rate with respect to time t; is the derivative of the drum pressure with respect to time t; is the derivative of the unit's power generation load with respect to time t; Derivative of the heating extraction steam pressure with respect to time t.

4. The multi-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning according to claim 1, characterized in that: The design of the reward function auxiliary mechanism ensures that the decisions of the three agents are relatively independent, coordinates the learning actions among the agents, and achieves the optimal decision of each agent and the global optimal decision: In formula (19) to formula (21): u T0 is the initial value of high-pressure door opening; u B0 is the initial value of coal feeding amount; t is the number of iterations; r 1,t 、r 2,t 、r 3,t are the response performance index penalties of the controlled variables of the PID controllers of the three agents respectively; △x 1,t , △x 2,t , △x 3,t are the load deviation values ​​of the 1st, 2nd and 3rd control loops respectively; △x 20 is the initial deviation of the generating load of the second control loop; ξ0, ξ1, ξ2, ξ3 are obtained through experiments, ξ0=2, ξ1=5, ξ2=3, ξ3=15 respectively; Since multi-agent DDPG trains multiple agents synchronously to obtain the maximum cumulative reward, the multi-objective control problem is equivalent to maximizing the expected reward value under the optimal response strategy of each control loop controller: MaxE[R i |π i ],i=1,2,3 (22); In formula (22): π i They respectively represent the total reward and penalty value and the optimal response strategy of the i-th controller acting on the operating condition sub-model within period T.

5. The multi-operating-condition adaptive control method for a cogeneration unit taking into account deep reinforcement learning according to claim 4, characterized in that: In step 3, the multi-condition adaptive control parameter optimization strategy of the MA-DDPG algorithm is taken into account, and the specific process is as follows: Step 1: Multi-agent Actor Network μ(s|θ μ ), according to the current state s k Decision action a; Step 2: Get the new state s by executing decision action a k+1 , and calculate the reward value r through equations (19), (20), and (21) 1,t 、r 2,t 、r 3,t ; Step 3: Critic network Q(s,a|θ Q ) Estimate the current action strategy, and in the process of interacting with the environment, through the offline training-online testing operation mode, the intelligent agent continuously evaluates the Q value and corrects the strategy to achieve the optimal decision of expected return and obtain the optimal sub-control system of the current working condition model; During offline training, the basic thermal and electrical requirements of the operating condition sub-model should be kept unchanged, and then as many random disturbances as possible should be considered. In the process of repeated iterations of "action-evaluation", the optimal μ and Q networks are obtained, and the corresponding sub-control systems are obtained, thereby constructing an offline database of operating condition sub-model-sub-control system pairs. During the online learning decision-making process, the system continuously outputs offline experience data to guide the online adaptive learning process, and uses online reserve knowledge to ultra-fast update the μ and Q network parameters of each agent; When there is a difference between the actual environment and the offline environment, DDPG quickly adapts to the new operating environment, thereby guiding the CHP unit control system online to respond to demand faster and more effectively.