A method and system for frequency control of a thermal power plant based on reinforcement learning algorithm

Abstract

The application relates to a frequency control method for a thermal power plant based on a reinforcement learning algorithm, wherein the method comprises the following steps: obtaining primary frequency regulation evaluation data and current combined cycle unit working condition data to form state data; determining a state-action pair in a preset algorithm model, wherein the preset algorithm model is obtained by training a Q learning algorithm through reinforcement learning based on the state data of the thermal power plant; obtaining evaluation values of each state-action pair based on a preset estimation rule through the preset algorithm model, and obtaining a target state-action pair according to the evaluation values; generating a control instruction according to an environmental interaction action in the target state-action pair, and adjusting the output power and action time of an electric energy storage device through the control instruction. Through the application, the problem that the service life of an electric energy storage device of a combined cycle unit is low due to long-term high-load working state in the related art is solved, accurate frequency regulation control is realized, power waste of the electric energy storage device is reduced, and the service life of the electric energy storage device is improved.

Description

Technical Field

[0001] This application relates to the field of frequency regulation control in power systems, and in particular to a frequency control method and system for a thermal power plant. Background Technology

[0002] With the gradual grid connection of new energy units, the proportion of conventional generating units has continued to decline. Since new energy units lack frequency response capabilities, the overall frequency response capability of the power system has also gradually weakened. Therefore, the focus must be on improving the frequency response capability of conventional combined heat and power (CHP) plants. Consequently, various local power grids have proposed specific primary frequency regulation assessment methods for CHP plants.

[0003] Traditional gas-fired power plants generally use combined cycle units. Due to the complex coupling characteristics between the gas turbine and steam turbine in a combined cycle unit, it is impossible to accurately control the frequency output of the gas turbine and steam turbine.

[0004] To address the aforementioned issues, related technologies employ an electrochemical energy storage device to assist frequency regulation in the turbine of a combined cycle unit. This device offers advantages such as precise instantaneous response, strong ramp-up capability, flexible power output, and high adaptability. Through this device, the regulation of the combined cycle unit can be decoupled (i.e., individual control can be achieved), thereby improving the performance of primary frequency regulation.

[0005] However, the electric energy storage devices used in gas-steam combined cycle units generally have a short service life due to the high-frequency load conditions, requiring frequent replacements. This undoubtedly has an adverse impact on the normal operation of the power plant and increases operation and maintenance costs. Summary of the Invention

[0006] This application provides a frequency control method, system, computer equipment, and computer-readable storage medium for combined cycle power plants based on reinforcement learning algorithms, to at least address the problem of low lifespan of electrical energy storage devices in related technologies.

[0007] In a first aspect, embodiments of this application provide a frequency control method for thermal power plants based on reinforcement learning algorithms, applied in the primary frequency regulation control scenario of thermal power plants, the method comprising:

[0008] The status data is composed of primary frequency regulation assessment data and current operating condition data of the combined cycle unit. The operating condition data includes: frequency deviation data, turbine operating condition data, and energy storage device operating condition data.

[0009] State-action pairs are determined in a preset algorithm model, wherein the state-action pairs consist of the state data and currently selectable environmental interaction actions, the environmental interaction actions are the control actions of the energy storage device, and the preset algorithm model is obtained by reinforcement learning training through Q-learning algorithm based on the state data of the thermal power plant.

[0010] The preset algorithm model is used to obtain the evaluation value of each state-action pair based on the preset estimation rules, and the target state-action pair is obtained based on the evaluation value.

[0011] Based on the environmental interaction actions in the target state action pair, control commands are generated, and the output power and action time of the energy storage device are adjusted through the control commands.

[0012] In some embodiments, the reinforcement learning training process of the preset algorithm model includes the following steps:

[0013] Step S1: Obtain the status data by combining the primary frequency regulation test data and the operating condition data of the combined cycle unit;

[0014] Step S2: Define a first value function, a learning factor, and a state-action pair in the Q-learning algorithm model, wherein the state-action pair uses the state data as the state variable and the environmental interaction action as the action variable;

[0015] Step S3: Determine the first state data. Under the first state data, estimate the first environmental interaction action based on the first value function. After instructing the energy storage device to respond to the first environmental interaction action, save the feedback first evaluation value and fill it into the evaluation value list.

[0016] Step S4: Under the first state data, iteratively learn the Q-learning algorithm model;

[0017] Step S5: Determine whether the results of the evaluation value list have converged. If not, update the first state data to the second state data, and repeat steps S3 and S4 until all state data have been traversed.

[0018] In some embodiments, step S4 includes:

[0019] Step S41: Update the first value function to the second value function according to the preset iterative formula;

[0020] Step S42: Under the first state data, estimate the second environmental interaction action based on the second value function, instruct the energy storage device to respond to the second environmental interaction action, save the feedback second evaluation value, and fill it into the evaluation value list;

[0021] Step S43: Repeat steps S41 and S42.

[0022] In some embodiments, the first value function is updated to the second value function using the following preset iterative formula:

[0023]

[0024] Among them, s t It is the state variable of the Q-learning algorithm model, a t It is the action quantity of the Q-learning algorithm model, (s) t a t Let α be the state-behavior pair of the decision-making process at time t. t The learning factor is used to determine the degree to which new information covers old information, and γ is the discount factor, which reflects the importance of the evaluation value of the next action to the evaluation value of the current action.

[0025] In some embodiments, the frequency offset data includes: a frequency offset value and the duration of the frequency offset value;

[0026] The turbine operating data includes: turbine power and turbine speed inequality.

[0027] The operating data of the energy storage device includes: state of charge and power of the energy storage device.

[0028] In some embodiments, the preset estimation rule is defined as follows: when the real-time frequency deviation is greater than the preset frequency deviation range, the frequency regulation capability of the thermal power plant can meet the requirements of the primary frequency regulation assessment data and can improve the service life of the energy storage device.

[0029] The generation process of the preset rules includes:

[0030] Using any of the aforementioned state data as the state variables of the agent in the Q-learning algorithm, after executing the first environmental interaction action, the response result obtained from executing the first environmental interaction action is acquired.

[0031] If the response result indicates that the assessment cost is 0 and the expected impact on the lifespan of the energy storage device is less than the preset impact value, then for the first environmental interaction action, an evaluation signal r=1 is fed back to the intelligent agent.

[0032] If the response result indicates that the assessment cost is greater than 0, or the expected impact on the lifespan of the energy storage device is greater than the preset impact value, an evaluation signal r = 0 is fed back to the intelligent agent in response to the first environmental interaction action.

[0033] The agent selects a second environmental interaction action based on the evaluation signal, wherein the selection criterion for the second environmental interaction action is to increase the probability of obtaining the evaluation signal r=1.

[0034] The preset rules are generated based on the selection strategies accumulated during the learning process.

[0035] In some embodiments, when the preset algorithm model identifies that the combined cycle unit's own frequency regulation capability is able to adjust the frequency offset to meet the requirements of the primary frequency regulation assessment,...

[0036] The control command generated by the preset algorithm model instructs the output power of the energy storage device to be adjusted to zero.

[0037] Secondly, embodiments of this application provide a frequency control system for a thermal power plant, applied in a primary frequency regulation control scenario of a thermal power plant. The system includes: a data acquisition module, a data processing module, and an instruction generation module, wherein;

[0038] The data acquisition module is used to acquire status data composed of primary frequency regulation assessment data and current combined cycle unit operating condition data, wherein the operating condition data includes: frequency deviation data, turbine operating condition data and energy storage device operating condition data;

[0039] The data processing module is used to determine state-action pairs in a preset algorithm model, wherein the state-action pair consists of the state data and currently selectable environmental interaction actions, the environmental interaction actions being the control actions of the energy storage device, and the preset algorithm model is obtained based on the state data of the thermal power plant through reinforcement learning training using the Q-learning algorithm.

[0040] The evaluation value of each state-action pair is obtained by using a preset algorithm model and based on a preset estimation rule, and the target state-action pair is obtained based on the evaluation value.

[0041] The instruction generation module is used to generate control instructions based on the environmental interaction actions in the target state action pair, and to adjust the output power and action time of the energy storage device through the control instructions.

[0042] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0044] Compared to related technologies, the frequency control method for thermal power plants provided in this application acquires state data composed of primary frequency regulation assessment data and current operating condition data of the combined cycle unit; determines state-action pairs in a preset algorithm model; obtains evaluation values ​​for each state-action pair based on preset estimation rules using the preset algorithm model, and obtains target state-action pairs based on the evaluation values; generates control commands based on environmental interaction actions in the target state-action pairs, and adjusts the output power and action time of the energy storage device through the control commands. This application solves the problem in related technologies where the energy storage device of a combined cycle unit has a short service life due to long-term high-load operation. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a schematic diagram of a gas-fired steam combined cycle unit according to an embodiment of this application;

[0047] Figure 2 This is a flowchart of a frequency control method for a thermal power plant according to an embodiment of this application;

[0048] Figure 3 This is a schematic diagram illustrating the principle of the reinforcement learning algorithm according to an embodiment of this application;

[0049] Figure 4 This is a flowchart of a training preset algorithm model according to an embodiment of this application;

[0050] Figure 5 This is a flowchart of another training preset algorithm model according to an embodiment of this application;

[0051] Figure 6 This is a structural block diagram of a frequency control system for a thermal power plant according to an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of a frequency control system for a thermal power plant according to an embodiment of this application;

[0053] Figure 8 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0055] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0058] In this document, it should be understood that the terms used may be technical means used to implement part of this invention or other summary technical terms. For example, terms may include:

[0059] Primary frequency regulation: refers to the automatic control process in which the control system of the generating units in the power grid automatically controls the increase or decrease of the active power of the generating units when the frequency of the power grid deviates from the rated value, thereby limiting the change of the power grid frequency and maintaining the stability of the power grid frequency.

[0060] Primary frequency regulation assessment: The evaluation and compensation mechanism for the response capability of power plants to primary frequency regulation can be judged by the following indicators: 15s output response index, 30s output response index, and power contribution index.

[0061] 15s output response index: Within 15 seconds from the start of the frequency deviation exceeding the dead zone, the percentage of the actual maximum output adjustment of the unit relative to the theoretical maximum output adjustment. Taking the "Two Detailed Rules for Grid-Connected Power Plants in North China" as an example, a 15s output response index of less than 75% is considered unqualified.

[0062] 30s Output Response Index: Within 30 seconds of the frequency deviation exceeding the dead zone, the percentage of the actual maximum output adjustment of the unit relative to the theoretical maximum output adjustment. Taking the "Two Detailed Rules for Grid-Connected Power Plants in North China" as an example: For coal-fired power units, a 30-second output response index of less than 90% is considered unqualified; for gas-fired and hydropower units, a 30-second output response index of less than 100% is considered unqualified.

[0063] Electricity contribution index: The percentage of actual electricity contributed by the unit during the frequency regulation period relative to the theoretical electricity contribution; taking the "Two Detailed Rules for Grid-Connected Power Plants in North China" as an example, an electricity contribution index of less than 75% is considered unqualified.

[0064] Speed ​​unequality rate: When a steam turbine is running alone, the percentage of the difference between the no-load speed and the full-load speed to the rated speed is called the speed unequality rate (or non-uniformity, speed variation rate, etc.) of the regulating system, denoted by the symbol δ.

[0065] Coupling characteristics: The coupling characteristics between gas and machinery in a combined cycle unit; Figure 1 This is a schematic diagram of a gas-fired steam combined cycle unit according to an embodiment of this application, as shown below. Figure 1 As shown, the gas turbine actively responds to the unit's load command, while the steam turbine passively follows. When the gas turbine load changes, it triggers a change in the steam heat load of the waste heat boiler. First, this process has a significant delay; second, the steam turbine's control valve remains unchanged in most cases, so the steam turbine's load also changes with the steam heat load of the waste heat boiler.

[0066] Reinforcement learning is a process of repeatedly interacting with and learning from the environment to enhance certain decisions. The optimization of this sequential decision-making relies on evaluative feedback signals. Its basic principle is that if the reward or punishment an agent receives for executing a certain behavioral strategy is positive, then the agent's tendency to adopt this strategy in future actions will be strengthened.

[0067] To address the issue of the short lifespan of energy storage devices in the aforementioned background technology, the inventors of this case learned that combined cycle power plants perform thousands of frequency adjustments daily during normal operation to meet primary frequency regulation requirements. Furthermore, through analysis of historical data, the inventors discovered that the combined cycle unit's own frequency regulation function can meet the vast majority of primary frequency regulation requirements; only in rare cases can the primary frequency regulation needs be met, necessitating the involvement of energy storage devices. Therefore, if the existing method is used to control the energy storage device to participate in frequency regulation when frequency fluctuations exceed the range, this is clearly unreasonable, significantly impacting the lifespan of the energy storage device and incurring high economic costs.

[0068] Furthermore, according to the primary frequency regulation assessment rules, the severity of the primary frequency regulation assessment is related not only to the instantaneous regulation power of the generating unit but also to the regulation duration. Therefore, even when energy storage devices are required to participate in frequency regulation, the output power required for frequency regulation by the energy storage devices should not only be related to the current frequency offset value but also to the current operating status of the energy storage devices and the state of the entire power grid at that time.

[0069] Based on the above, the inventors of this case learned that if the output power of the energy storage device is determined solely based on the frequency offset value according to the existing frequency modulation method, this is obviously passive and not accurate enough. In addition, when the frequency fluctuation is large, the existing method may also cause a large amount of power waste.

[0070] Therefore, the inventors of this case proposed a frequency control method for thermal power plants based on reinforcement learning. This method fully utilizes the decision-making characteristics of reinforcement learning algorithms, makes decisions based on all relevant data that may affect the primary frequency regulation assessment, and dynamically controls the output frequency of the energy storage device in real time. This not only enables the energy storage device to participate in the primary frequency regulation control of the unit in a timely manner to avoid exceeding the requirements of the primary frequency regulation assessment, but also improves the service life of the energy storage device.

[0071] Figure 2 This is a flowchart of a frequency control method for a thermal power plant based on a reinforcement learning algorithm, according to an embodiment of this application. Figure 2 As shown, the process includes the following steps:

[0072] S201, acquire the primary frequency regulation assessment data and the current operating condition data of the combined cycle unit to form status data, wherein the operating condition data includes: frequency deviation data, turbine operating condition data and energy storage device operating condition data;

[0073] The primary frequency regulation assessment data includes the aforementioned output response index, power contribution index, and other indicators.

[0074] It should be noted that the assessment indicators set by different local power grids are not exactly the same. However, it should be understood that the specific differences in the parameters do not affect the core inventive points of this application.

[0075] Furthermore, the frequency offset data includes the frequency offset value and the duration of the frequency offset value; the turbine operating condition data includes: turbine power and turbine speed inequality; the energy storage device operating condition data includes: state of charge data and energy storage device power.

[0076] S202, determine the state-action pair in the preset algorithm model, wherein the state-action pair consists of state data and currently selectable environmental interaction actions, the environmental interaction actions are the control actions of the energy storage device, and the preset algorithm model is based on the state data of the thermal power plant and is obtained by reinforcement learning training through the Q learning algorithm;

[0077] It should be noted that in the reinforcement learning algorithm model, in the environment state S t Next, choose any action a. t After execution, the environment state will be in action a. t Under the influence of the state, it changes to the next state S. t+1 At the same time, it will return the currently selected action 'a' to the model agent. t The obtained evaluation value, wherein the environmental state S is given above. t and action a t The status action pairs in this application.

[0078] Furthermore, since the embodiment of this application is a reinforcement learning model for the participation of an energy storage device in primary frequency regulation control, the corresponding environmental interaction actions available in a certain state are the control actions of the energy storage device, which include, but are not limited to, controlling the energy storage device to regulate power and the duration of the regulating power.

[0079] S203, by using a preset algorithm model and based on preset estimation rules, the evaluation value of each state-action pair is obtained, and the target state-action pair is obtained based on the evaluation value;

[0080] In step S202 above, the execution of action a has been described. t Then, the model will return the current state S to the model agent. t Select action a below t The obtained evaluation value corresponds to the evaluation value of the state-action pair obtained in this step.

[0081] It should be noted that this preset algorithm model is trained using the Q-learning algorithm through reinforcement learning. The basic logic of the Q-learning algorithm training process is as follows:

[0082] In a certain state, after selecting an action to execute, an evaluation value for executing the current action will be returned, and the current action will be updated at the same time.

[0083] The intelligent system continues to select the next action based on the evaluation signal and the current environmental state. The selection criterion for each action is to increase the probability of receiving a good reward.

[0084] Knowledge is gained through proactive exploration and evaluation, action rules are continuously improved and refined, and the optimal strategy is ultimately determined. The goal is to maximize the reward value. This optimal strategy can be expressed by the following formula:

[0085] π * (s t ) = argmax at Q(s t at )

[0086] Where, π * (s t ) is state s t The optimal strategy under the following conditions, (s t a t ) is a state-action pair, Q(s) t a t ) is a state-action pair (s) t a t The evaluation value of ).

[0087] Based on the above, it can be understood that in step S203, the process of obtaining the evaluation value through the preset algorithm model and based on the preset estimation rules is equivalent to the process of matching using the learning experience accumulated during the training process.

[0088] In addition, this application uses the Q-learning algorithm to obtain a preset algorithm model, which does not require knowledge of the environment model and state transition function. In actual decision-making, it only needs to select the action with the largest evaluation value from all state-action pairs based on learning experience. Compared with frequency control through other algorithm models (such as deep algorithm models), its decision-making process is simpler and has lower requirements for prior knowledge of the environment.

[0089] S204 generates control commands based on the environmental interaction actions in the target state action pair, and adjusts the output power and action time of the energy storage device through the control commands.

[0090] The environmental interaction actions in the aforementioned target state action pair are the control actions most suitable for the current environmental state obtained through reinforcement learning decision-making. Furthermore, by generating control commands based on these environmental interaction actions and then controlling the output frequency of the energy storage device, precise control of the energy storage device can be achieved in a single frequency regulation.

[0091] It should be understood that, given the pre-defined algorithm model's identification and the combined cycle unit's own frequency regulation capability, which allows it to adjust the frequency offset to meet the requirements of the primary frequency regulation assessment, the control commands generated by the pre-defined algorithm model will adjust the output power of the energy storage device to zero.

[0092] Through steps S201 to S204 above, compared to existing power plant frequency regulation methods, the method provided in this application, when performing primary frequency regulation control, utilizes data related to primary frequency regulation assessment and makes optimal decisions through a reinforcement learning-based algorithm model. This avoids the problem of short service life caused by high-frequency, high-load operation of existing point-of-use energy storage devices. By fully integrating environmental variables for precise frequency regulation control, power waste in energy storage devices is reduced, and their service life is extended.

[0093] It should be noted that although some schemes utilizing reinforcement learning for microgrid frequency control have emerged, these methods primarily involve applying reinforcement learning algorithms to estimate the most suitable droop adjustment parameters when the microgrid is operating in islanded mode. These parameters are then used to adjust the microgrid frequency for global frequency regulation. It should be understood that while both the scheme in this application and these technologies utilize reinforcement learning mechanisms for frequency regulation, reinforcement learning algorithms are widely used in the field of intelligent control. Therefore, the technical problems addressed and the technical methods employed in this application are fundamentally different from those technologies.

[0094] In some of these embodiments, Figure 3 This is a schematic diagram illustrating the principle of the reinforcement learning algorithm according to an embodiment of this application, such as... Figure 3 As shown:

[0095] Reinforcement learning views learning as a process of continuous trial and error. The agent continuously receives input states s from the environment and then selects an action a to continue executing based on some internal pre-set reasoning mechanisms.

[0096] The environment changes to a new state s under the influence of action 'a', and provides an evaluation signal (reward or penalty) to the agent's chosen action. Based on the evaluation signal and the current environment state, the agent continues to choose its next action, with each action chosen increasing the probability of receiving a positive reward. Each action chosen by the agent not only affects the current reward value but also influences the state at the next moment and even the final reward value.

[0097] Furthermore, reinforcement learning has the following characteristics:

[0098] (1) Intelligent agents need to actively test the environment rather than remain static or passive.

[0099] (2) The environment’s feedback to these probing actions is evaluative.

[0100] (3) The agent acquires knowledge in the process of actively exploring and obtaining environmental assessment, continuously improves and perfects its action plan, and finally adapts to the environment to complete the learning task.

[0101] Figure 4 This is a flowchart of a training preset algorithm model according to an embodiment of this application, such as... Figure 4 As shown, specifically, it includes the following steps:

[0102] Step S401: Obtain the status data composed of primary frequency regulation assessment data and combined cycle unit operating condition data; it should be noted that in this step, the status data used in the training process is not a single data point, but a dataset composed of accumulated historical data;

[0103] Step S402: Define a first value function, a learning factor, and state-action pairs in the Q-learning algorithm model. The state-action pairs use state data as state variables and environmental interaction actions as action variables. In addition, the value function is used to estimate the action with the highest probability that will obtain the highest evaluation value.

[0104] Step S403: Determine the first state data. Under the first state data, estimate the first environmental interaction action based on the first value function. After instructing the energy storage device to respond to the first environmental interaction action, save the feedback first evaluation value and fill it into the evaluation value list.

[0105] Step S404: Under the first state data, the Q-learning algorithm model is iteratively learned; the process of algorithm model iteration is equivalent to continuously selecting the corresponding action that can obtain the maximum evaluation value under different value functions by adjusting the value function. In this process, learning experience will be continuously accumulated to obtain different state-action pairs and their evaluation values.

[0106] Step S405: Determine if the evaluation value list has converged. If not, update the first state data to the second state data, and repeat steps S303 and S304 until all state data has been traversed. It should be noted that convergence of the evaluation value list can be reflected in the fact that the values ​​in the list no longer change significantly. Furthermore, it can be understood that in the first state S... t After the result converges, the state variable is changed to S. t+1 Continue training according to S303 and S304 above until all state data has been traversed.

[0107] In some of these embodiments, Figure 5 This is a flowchart of another training preset algorithm model according to an embodiment of this application, such as... Figure 5 As shown, step S404 specifically includes:

[0108] Step S4041: Update the first value function to the second value function according to the preset iterative formula;

[0109] Step S4042: Under the first state data, estimate the second environmental interaction action based on the second value function, instruct the energy storage device to respond to the second environmental interaction action, save the feedback second evaluation value, and fill it into the evaluation value list;

[0110] Step S4043: Repeat steps S4041 and S4042 until the results in the table converge.

[0111] In some embodiments, the first value function is updated to the second value function using the following preset iterative formula:

[0112]

[0113] Among them, s t It is the state variable of the Q-learning algorithm model, a t It is the action quantity of the Q-learning algorithm model, (s t a t Let α be the state-behavior pair of the decision-making process at time t. t γ is the learning factor, which determines the degree to which new information covers old information, and γ is the discount factor, which reflects the importance of the reward value of the next action to the Q value of the current action.

[0114] In some embodiments, the preset estimation rule is defined as follows: when the real-time frequency is greater than a preset frequency deviation range, the frequency regulation capability of the thermal power plant can meet the requirements of the primary frequency regulation assessment data and can improve the service life of the energy storage device; wherein, the generation process of the preset rule includes:

[0115] Using any state data as the state variable of the agent in the Q-learning algorithm, after performing the first environmental interaction action, the response result obtained from performing the first environmental interaction action is acquired.

[0116] When the response result indicates that the assessment cost is 0 and the expected impact on the lifespan of the energy storage device is less than the preset impact value (i.e., the impact on the lifespan of the energy storage device is small), an evaluation signal r=1 is fed back to the agent for the first environmental interaction action. When the response result indicates that the assessment cost is greater than 0, or the expected impact on the lifespan of the energy storage device is greater than the preset impact value (i.e., the impact on the lifespan of the energy storage device is large), an evaluation signal r=0 is fed back to the agent for the first environmental interaction action. The agent selects a second environmental interaction action based on the evaluation signal. The selection criterion for the second environmental interaction action is to increase the probability of obtaining the evaluation signal r=1.

[0117] Preset rules are generated based on the selection strategies accumulated during the learning process.

[0118] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0119] This embodiment also provides a frequency control system for a thermal power plant, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0120] Figure 6 This is a structural block diagram of a frequency control system for a thermal power plant according to an embodiment of this application, such as... Figure 6 As shown, the system includes: a data acquisition module 60, a data processing module 61, and an instruction generation module 62, wherein;

[0121] The data acquisition module 60 is used to acquire the status data composed of primary frequency regulation assessment data and current combined cycle unit operating condition data. The operating condition data includes: frequency deviation data, turbine operating condition data and energy storage device operating condition data.

[0122] Data processing module 61 is used to determine state-action pairs in a preset algorithm model. Each state-action pair consists of state data and currently selectable environmental interaction actions. These environmental interaction actions are control actions of the energy storage device. The preset algorithm model is obtained based on the state data of a thermal power plant, trained using a Q-learning algorithm and reinforcement learning.

[0123] The evaluation value of each state-action pair is obtained by using a preset algorithm model and based on preset estimation rules, and the target state-action pair is obtained based on the evaluation value.

[0124] The instruction generation module 62 is used to generate control instructions based on the environmental interaction actions in the target state action pair, and to adjust the output power and action time of the energy storage device through the control instructions.

[0125] In some of these embodiments, Figure 7 This is a schematic diagram of a frequency control system for a thermal power plant according to an embodiment of this application, as shown below. Figure 7 As shown,

[0126] This system uses the Q-learning algorithm to dynamically adjust the output of the energy storage devices participating in primary frequency regulation in real time, thereby effectively reducing the primary frequency regulation assessment costs of the power grid and extending the operating life of the energy storage devices, creating good economic benefits for gas-fired power plants.

[0127] In this system, the characteristic parameters, frequency deviation data, and primary frequency regulation assessment data of the combined cycle unit and energy storage device are the state variables belonging to the Q-learning algorithm. Adjusting the output power of the energy storage device based on the agent's decisions is the action of the Q-learning algorithm. The learning result is the evaluation value of each state-action pair, that is, the evaluation value of the state action that meets the requirements of reducing primary frequency regulation assessment and extending the life of the energy storage device when the grid frequency exceeds the normal operating range. After a period of iterative learning, the values ​​in the evaluation value table will stabilize, indicating that the learning results have converged. At this point, the system can be used for frequency regulation control to improve the economic efficiency of the gas-fired power plant.

[0128] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a frequency control system for a thermal power plant. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input devices may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0129] In one embodiment, Figure 8 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 8 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 8 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a frequency control system for a thermal power plant, and the database stores data.

[0130] Those skilled in the art will understand that Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A frequency control method for thermal power plants based on reinforcement learning algorithms, characterized in that, The method, applied in the primary frequency regulation control scenario of a thermal power plant, includes: The status data is composed of primary frequency regulation assessment data and current operating condition data of the combined cycle unit. The operating condition data includes: frequency deviation data, turbine operating condition data, and energy storage device operating condition data. The turbine operating condition data includes: turbine power and turbine speed inequality. The energy storage device operating condition data includes: state of charge data and energy storage device power. The frequency deviation data includes: frequency offset value and the duration of the frequency offset value. A state-action pair is determined in a preset algorithm model, wherein the state-action pair consists of the state data and the currently selectable environmental interaction action, and the environmental interaction action is the control action of the energy storage device. The preset algorithm model is obtained by reinforcement learning training through Q-learning algorithm based on the state data of the thermal power plant. The preset algorithm model is a reinforcement learning model for the energy storage device to participate in primary frequency regulation control. In a certain state, the selectable environmental interaction actions include: controlling the energy storage device to adjust the power and the duration of the adjustment power. Using the preset algorithm model and based on preset estimation rules, the evaluation value of each state-action pair is obtained, and the target state-action pair is determined based on the evaluation value. Based on the environmental interaction actions in the target state action pair, control commands are generated, and the output power and action time of the energy storage device are adjusted through the control commands. The preset estimation rule is defined as follows: when the real-time frequency is greater than the preset frequency deviation range, the frequency regulation capability of the thermal power plant can meet the requirements of the primary frequency regulation assessment data and can improve the service life of the energy storage device. The generation process of the preset estimation rule includes: Using any of the aforementioned state data as the state variables of the agent in the Q-learning algorithm, after executing the first environmental interaction action, the response result obtained from executing the first environmental interaction action is acquired. If the response result indicates that the assessment cost is 0 and the expected impact on the lifespan of the energy storage device is less than the preset impact value, then for the first environmental interaction action, an evaluation signal r=1 is fed back to the intelligent agent. If the response result indicates that the assessment cost is greater than 0, or the expected impact on the lifespan of the energy storage device is greater than the preset impact value, an evaluation signal r = 0 is fed back to the intelligent agent in response to the first environmental interaction action. The agent selects a second environmental interaction action based on the evaluation signal, wherein the selection criterion for the second environmental interaction action is to increase the probability of obtaining the evaluation signal r=1. The preset estimation rule is generated based on the selection strategy accumulated during the learning process.

2. The method according to claim 1, characterized in that, The reinforcement learning training process of the preset algorithm model includes the following steps: Step S1: Obtain the status data by combining the primary frequency regulation test data and the operating condition data of the combined cycle unit; Step S2: Define a first value function, a learning factor, and a state-action pair in the Q-learning algorithm model, wherein the state-action pair uses the state data as the state variable and the environmental interaction action as the action variable; Step S3: Determine the first state data. Under the first state data, estimate the first environmental interaction action based on the first value function. After instructing the energy storage device to respond to the first environmental interaction action, save the feedback first evaluation value and fill it into the evaluation value list. Step S4: Under the first state data, iteratively learn the Q-learning algorithm model; Step S5: Determine whether the results of the evaluation value list have converged. If not, update the first state data to the second state data, and repeat steps S3 and S4 until all state data have been traversed.

3. The method according to claim 2, characterized in that, Step S4 includes: Step S41: Update the first value function to the second value function according to the preset iterative formula; Step S42: Under the first state data, estimate the second environmental interaction action based on the second value function, and instruct the... After the energy storage device responds to the second environmental interaction action, it saves the feedback second evaluation value and fills it into the evaluation value list; Step S43: Repeat steps S41 and S42.

4. The method according to claim 3, characterized in that, The first value function is updated to the second value function using the following preset iterative formula: in, These are the state variables of the Q-learning algorithm model. It is the action quantity of the Q-learning algorithm model. Let be the state-behavior pair of the decision-making process at time t. The learning factor determines the degree to which new information covers older information. It is a discount factor used to reflect the importance of the evaluation value of the next action to the evaluation value of the current action. The state behavior pair at time t The evaluation value.

5. The method according to claim 1, characterized in that, Based on the preset algorithm model's identification, the combined cycle unit's own frequency regulation capability is sufficient to adjust the frequency offset to meet the requirements of the primary frequency regulation assessment. The control command generated by the preset algorithm model instructs the output power of the energy storage device to be adjusted to zero.

6. A frequency control system for a thermal power plant, characterized in that, The system, applied in the primary frequency regulation control scenario of a thermal power plant, includes: a data acquisition module, a data processing module, and an instruction generation module, wherein; The data acquisition module is used to acquire status data composed of primary frequency regulation assessment data and current combined cycle unit operating condition data. The operating condition data includes: frequency deviation data, turbine operating condition data, and energy storage device operating condition data. The turbine operating condition data includes: turbine power and turbine speed inequality. The energy storage device operating condition data includes: state of charge data and energy storage device power. The frequency deviation data includes: frequency offset value and the duration of the frequency offset value. The data processing module is used to determine state-action pairs in a preset algorithm model, wherein the state-action pair consists of the state data and currently selectable environmental interaction actions, the environmental interaction actions being the control actions of the energy storage device, and the preset algorithm model is obtained based on the state data of the thermal power plant through reinforcement learning training using the Q-learning algorithm. The evaluation value of each state-action pair is obtained by using a preset algorithm model and based on preset estimation rules, and the target state-action pair is determined according to the evaluation value. The instruction generation module is used to generate control instructions based on the environmental interaction actions in the target state action pair, and to adjust the output power and action time of the energy storage device through the control instructions; the preset algorithm model is a reinforcement learning model for the energy storage device to participate in primary frequency regulation control. In a certain state, the selectable environmental interaction actions include: controlling the energy storage device to adjust the power and the duration of the power adjustment. The preset estimation rule is defined as follows: when the real-time frequency is greater than the preset frequency deviation range, the frequency regulation capability of the thermal power plant can meet the requirements of the primary frequency regulation assessment data and can improve the service life of the energy storage device. The generation process of the preset estimation rule includes: Using any of the aforementioned state data as the state variables of the agent in the Q-learning algorithm, after executing the first environmental interaction action, the response result obtained from executing the first environmental interaction action is acquired. If the response result indicates that the assessment cost is 0 and the expected impact on the lifespan of the energy storage device is less than the preset impact value, then for the first environmental interaction action, an evaluation signal r=1 is fed back to the intelligent agent. If the response result indicates that the assessment cost is greater than 0, or the expected impact on the lifespan of the energy storage device is greater than the preset impact value, an evaluation signal r = 0 is fed back to the intelligent agent in response to the first environmental interaction action. The agent selects a second environmental interaction action based on the evaluation signal, wherein the selection criterion for the second environmental interaction action is to increase the probability of obtaining the evaluation signal r=1. The preset estimation rule is generated based on the selection strategy accumulated during the learning process.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images