An adaptive energy storage capacity configuration method, device, medium and product based on deep reinforcement learning for assisting secondary frequency modulation of a thermal power unit

Abstract

The application discloses a kind of based on deep reinforcement learning auxiliary thermal power unit secondary frequency modulation adaptive energy storage capacity configuration method, equipment, medium and product, it is related to secondary frequency modulation technical field, the method comprises: obtaining the power grid operating state data of current period;Input the strategy network trained by maximum entropy Markov decision process modeling, soft-actor-critic algorithm, obtain the action vector containing power regulation, capacity boundary component;In combination with energy storage rated parameter, operating boundary and converter limit, active instruction and online adjustable capacity limit are generated and issued by mapping;When fire storage collaborative frequency modulation, energy storage preferentially assumes high-frequency fast short-time regulation power, and remaining demand is distributed according to unit ramp rate;Capacity limit quantile value is calculated in the statistical period, and recommended configuration capacity is generated in combination with safety margin coefficient.The application can dynamically adjust energy storage frequency modulation parameter according to real-time state of power grid, with self-learning optimization ability.

Description

Technical Field

[0001] This application relates to the field of secondary frequency regulation technology, and in particular to an adaptive energy storage capacity configuration method, device, medium and product based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units. Background Technology

[0002] With the increasing penetration of new energy sources such as wind power and photovoltaics into the power system, the randomness and volatility of the power grid have significantly increased, posing a huge challenge to the stability of the grid frequency. Precise control of the grid frequency is the cornerstone of ensuring power quality and the safe and stable operation of the power grid. Secondary frequency regulation is a key step in restoring the system frequency to its rated value, and it is mainly undertaken by thermal power units with slower response speeds but larger regulation capacities.

[0003] However, thermal power units suffer from slow adjustment rates, limited ramp-up capabilities, and frequent adjustments that exacerbate unit wear and reduce operational economics. To address these challenges, introducing fast-response, precise-adjustment energy storage systems (such as battery storage) in conjunction with thermal power units for secondary frequency regulation has become a mainstream technical solution.

[0004] Existing energy storage configurations and frequency regulation control schemes still have the following shortcomings: 1. Fixed capacity configuration lacks dynamic adaptability. Traditional energy storage configuration methods are usually calculated based on typical, static load or grid disturbance scenarios to obtain a fixed energy storage capacity configuration. This approach cannot adapt to the real-time changing operating status and frequency regulation requirements of the power grid, which may lead to redundant energy storage capacity in some periods, resulting in wasted investment, or insufficient capacity in other periods, affecting frequency regulation performance.

[0005] 2. Configuration methods rely on accurate mathematical models. Some optimal configuration methods depend on accurate mathematical modeling of the power grid, generator units, and loads. However, modern power systems are becoming increasingly complex, making it very difficult to establish accurate mathematical models. The accuracy of the model directly affects the quality of the configuration results.

[0006] 3. Lack of online adaptive learning capabilities. Most existing solutions lack the ability to learn and adapt online, and cannot optimize energy storage configuration and frequency regulation strategies based on long-term operational data to achieve globally optimal operational efficiency. Summary of the Invention

[0007] The purpose of this application is to provide an adaptive energy storage capacity configuration method, device, medium and product based on deep reinforcement learning to assist secondary frequency regulation of thermal power units. It can dynamically adjust the power command and capacity limit of energy storage participating in frequency regulation according to the real-time status of the power grid, and has the ability to learn and optimize autonomously.

[0008] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units, including: Acquire grid operation status data for the current control cycle; the grid operation status data includes regional control deviation, grid frequency deviation, frequency deviation integral, energy storage system state of charge, current active power output of thermal power units, and short-term load forecast information; The power grid operation status data is input into a pre-trained policy network to obtain action vectors; the action vectors include energy storage power adjustment action components and energy storage capacity boundary action components; the policy network is modeled by a maximum entropy Markov decision process and trained based on a soft-actor-commentator deep reinforcement learning algorithm. Based on the rated parameters of the energy storage system, the state-of-charge operating boundary, and the power limit of the energy storage converter, the energy storage power regulation action component is mapped to the energy storage active power command, and the energy storage capacity boundary action component is mapped to the online callable capacity limit of the energy storage system participating in secondary frequency regulation during the current operating period. The energy storage active power command is sent to the energy storage converter to execute charge and discharge control, and the online callable capacity limit is sent to the energy storage coordination controller or battery management system as the online capacity constraint for energy storage to participate in secondary frequency regulation in the current period. During the coordinated frequency regulation of thermal power and energy storage, the control system prioritizes the energy storage system to undertake high-frequency, fast, and small-time-scale regulation power, and allocates the remaining frequency regulation demand that exceeds the capacity of the energy storage system to the thermal power units for execution according to the ramp rate limit of the thermal power units; Record and statistically analyze the continuously output online callable capacity limit within a set statistical period, calculate the quantile value within that statistical period, and generate a recommended configuration capacity by combining it with the safety margin coefficient.

[0009] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units as described above.

[0010] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units as described above.

[0011] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units as described above.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an adaptive energy storage capacity configuration method, device, medium, and product based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units. This application uses a policy network to output real-time energy storage power regulation action components and energy storage capacity boundary action components based on the regional control deviation, grid frequency deviation, frequency deviation integral, energy storage system state of charge, current active power output of the thermal power unit, and short-term load forecast information during the current control cycle. Within a set statistical period, it records and statistically calculates the quantile values ​​of the continuously output online callable capacity limits and combines them with a safety margin coefficient to generate a recommended configuration capacity. Compared with traditional schemes that calculate fixed energy storage capacity based on typical static scenarios, this application can dynamically adjust the online callable capacity boundary of energy storage according to the real-time operating status of the grid. When the grid disturbance intensity is high, it releases sufficient capacity to ensure frequency regulation needs, and actively limits the depth of energy storage participation to protect battery life when the grid is running smoothly. This effectively avoids the problems of redundancy waste and insufficient failure of fixed capacity configuration in some periods and in other periods. During the thermal power-storage coordinated frequency regulation process, this application controls the energy storage system to prioritize high-frequency, fast, and short-time-scale regulation power. Because the energy storage system has a second-level power regulation capability, it can compensate for the output delay and gap caused by the physical ramp rate limitation of thermal power units in real time, thus suppressing the system frequency deviation and regional control deviation more quickly. Experiments have shown that this mechanism significantly improves the response speed and regulation accuracy of secondary frequency regulation, ensuring that the frequency regulation evaluation index is superior to the traditional independent frequency regulation mode of thermal power. This application allocates the remaining frequency regulation demand exceeding the capacity of energy storage to thermal power units according to the ramp rate limitation of thermal power units, so that thermal power units only need to respond to the slow, continuous regulation power after energy storage smoothing, avoiding the thermal power units frequently performing reverse regulation actions to track high-frequency fluctuating loads, significantly reducing the fatigue wear of mechanical components such as unit valves and turbines, and extending the equipment maintenance cycle and service life. This application trains the policy network based on the soft-actor-commentator deep reinforcement learning algorithm, enabling the policy network to learn the optimal joint decision-making strategy of energy storage power and capacity from the six-dimensional state space composed of regional control deviation, frequency deviation, frequency deviation integral, energy storage state of charge, thermal power unit output, and short-term load forecast. Leveraging the exploratory nature of the maximum entropy framework, this strategy can flexibly adjust energy storage power commands and capacity limits based on real-time conditions when facing random and nonlinear disturbances brought about by the grid connection of new energy sources such as wind power and photovoltaics. This demonstrates stronger adaptability and control stability than traditional rule-based control. In the strategy network decision-making process, the generation of energy storage power commands and capacity limits in this application simultaneously considers physical constraints such as the rated parameters of the energy storage system itself, the state-of-charge operating boundary, and the power limits of the energy storage converter.By dynamically managing the online available capacity limits, the energy storage system can operate within a safe operating range, avoiding overcharging, over-discharging, and deep cycling, thus extending battery cycle life. At the same time, the dynamic matching of capacity limits also reduces the idle cost and configuration redundancy of energy storage capacity, improving the overall economic benefits of the energy storage system throughout its entire life cycle. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is an application environment diagram of an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units, according to one embodiment of this application. Figure 2 A flowchart illustrating an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units, provided as an embodiment of this application; Figure 3 A schematic diagram of the overall process of an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units, provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] This application provides a method for configuring energy storage capacity that dynamically adjusts according to the real-time state of the power grid and has the ability to learn and optimize autonomously, so as to improve the comprehensive benefits of combined thermal power and energy storage secondary frequency regulation. Based on the SoftActor-Critic (SAC) deep reinforcement learning algorithm, it can assist thermal power units in performing secondary frequency regulation.

[0017] This application provides a method for adaptive configuration and coordinated control of energy storage in an auxiliary frequency regulation system for thermal power units. The method uses a power grid energy management system (EMS) or a thermal power-storage coordinated controller as the execution subject. It performs real-time closed-loop control of the frequency regulation output of the thermal power unit and the charging and discharging power of the energy storage system, and dynamically adjusts the online configuration capacity of the energy storage system.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server.

[0020] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0021] In one exemplary embodiment, such as Figure 2 As shown, an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units is provided. This method is executed by computer equipment, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S1 to S6. Wherein: S1. Obtain the power grid operation status data for the current control cycle; the power grid operation status data includes regional control deviation, power grid frequency deviation, frequency deviation integral, energy storage system state of charge, current active power output of thermal power units, and short-term load forecast information.

[0022] S2. Input the power grid operation status data into the pre-trained policy network to obtain action vectors; the action vectors include energy storage power adjustment action components and energy storage capacity boundary action components; the policy network is modeled by a maximum entropy Markov decision process and trained based on a soft-actor-commentator deep reinforcement learning algorithm.

[0023] S3. Based on the rated parameters of the energy storage system, the state-of-charge operating boundary, and the power limit of the energy storage converter, the energy storage power regulation action component is mapped to the energy storage active power command, and the energy storage capacity boundary action component is mapped to the online callable capacity limit of the energy storage system participating in secondary frequency regulation during the current operating period.

[0024] S4. The energy storage active power command is sent to the energy storage converter to execute charge and discharge control, and the online callable capacity limit is sent to the energy storage coordination controller or battery management system as the online capacity constraint for energy storage to participate in secondary frequency regulation in the current period.

[0025] S5. During the coordinated frequency regulation of thermal power and energy storage, the control system prioritizes the energy storage system to undertake high-frequency, fast, and small-time-scale regulation power, and allocates the remaining frequency regulation demand that exceeds the capacity of the energy storage system to the thermal power units for execution according to the ramp rate limit of the thermal power units.

[0026] S6. Record and statistically analyze the continuously output online callable capacity limit within the set statistical period, calculate the quantile value within the statistical period, and generate a recommended configuration capacity in combination with the safety margin coefficient.

[0027] This embodiment uses a strategy network to output real-time energy storage power regulation components and energy storage capacity boundary components based on the regional control deviation, grid frequency deviation, frequency deviation integral, energy storage system state of charge, current active power output of thermal power units, and short-term load forecast information during the current control cycle. Within a set statistical period, it records and statistically calculates the quantile values ​​of the continuously output online callable capacity limits, and combines this with a safety margin coefficient to generate a recommended configuration capacity. Compared to traditional schemes that calculate fixed energy storage capacity based on typical static scenarios, this embodiment can dynamically adjust the online callable capacity boundary of energy storage according to the real-time operating status of the grid. It releases sufficient capacity to ensure frequency regulation needs when grid disturbances are strong, and actively limits the depth of energy storage participation to protect battery life when the grid is running smoothly. This effectively avoids the problems of redundancy waste and insufficient failure of fixed capacity configuration in some periods. In the thermal power-storage coordinated frequency regulation process, this embodiment controls the energy storage system to prioritize high-frequency, fast, and small-time-scale regulation power. Because the energy storage system has second-level power regulation capabilities, it can compensate for the output delay and gap caused by the physical ramp-up rate limitation of thermal power units in real time, enabling faster suppression of system frequency deviation and regional control deviation. Experiments have shown that this mechanism significantly improves the response speed and regulation accuracy of secondary frequency regulation, ensuring that the frequency regulation evaluation indicators are superior to those of traditional independent frequency regulation modes for thermal power plants. This embodiment allocates the remaining frequency regulation demand exceeding the capacity of energy storage to thermal power units based on their ramp rate limitations. This allows the thermal power units to respond only to the slow, continuous power regulation after energy storage has smoothed the load, avoiding frequent reverse regulation actions by the units to track high-frequency fluctuating loads. This significantly reduces fatigue wear on mechanical components such as valves and turbines, extending equipment maintenance cycles and service life. This embodiment trains the policy network based on a soft-actor-commentator deep reinforcement learning algorithm, enabling the network to learn the optimal joint decision-making strategy for energy storage power and capacity from a six-dimensional state space comprised of regional control deviation, frequency deviation, frequency deviation integral, energy storage state of charge, thermal power unit output, and short-term load forecasting. Leveraging the exploratory nature of the maximum entropy framework, this strategy can flexibly adjust energy storage power commands and capacity limits based on real-time conditions when facing random and nonlinear disturbances brought about by the grid connection of new energy sources such as wind power and photovoltaics. This demonstrates stronger adaptability and control stability than traditional rule-based control. In this embodiment, the generation of energy storage power commands and capacity limits during the strategy network decision-making process simultaneously considers physical constraints such as the rated parameters of the energy storage system itself, the state-of-charge operating boundary, and the power limits of the energy storage converter. Through dynamic management of online callable capacity limits, the energy storage system can operate within a safe operating range, avoiding overcharging, over-discharging, and deep cycling, thus extending battery cycle life. Simultaneously, the dynamic matching of capacity limits also reduces the idle cost and configuration redundancy of energy storage capacity, improving the overall economic benefits of the energy storage system throughout its entire lifecycle.

[0028] In another exemplary embodiment of this application, the training step of the policy network in step S2 above includes: Step 1: Construct a maximum entropy Markov decision process model for the secondary frequency regulation environment of the power grid, defining the state space, action space, and reward function. The state vector in the state space is the power grid operating state data, and the action vector in the action space is a two-dimensional continuous action vector composed of the energy storage power regulation action component and the energy storage capacity boundary action component. The reward function is used to evaluate the frequency regulation effect and operating cost generated after executing the action vector in the current state.

[0029] Step 2: Construct a deep reinforcement learning agent network structure based on the soft-actor-critic algorithm. The agent network structure includes a policy network, a first value network, a second value network, a first target value network, and a second target value network. The policy network is used to implement the mapping from the state space to the action space, that is, to output action vectors based on the input state vectors. The first value network and the second value network are used to evaluate the long-term benefits obtained under the state vectors and action vectors, respectively, and provide guiding signals for the update of the policy network. The first target value network and the second target value network have the same structure as the first value network and the second value network, respectively, and are used to provide target values ​​during the training process to stabilize the training of the value network.

[0030] Step 3: Through interaction with the power grid secondary frequency regulation simulation environment, the agent network structure is trained offline. In each iteration, the power grid operation status data of the current period is first input into the policy network as the current state to obtain the training action vector. The training action vector is then input into the power grid secondary frequency regulation simulation environment. The next moment's state and immediate reward are calculated based on the current state and action vector. The current state, the training action vector, the immediate reward, and the next moment's state are stored as a set of experience samples in the experience replay pool. Then, experience samples are sampled from the experience replay pool. The sum of the expected returns of the immediate reward and the next moment's state under the target value network is used as a supervision signal to update the first value network and the second value network. The Q values ​​output by the first value network and the second value network are used as the guiding signal for action evaluation to update the policy network and update the temperature coefficient parameters. The parameters of the first target value network and the second target value network are synchronized through a soft update method until the policy network converges. The converged policy network parameters are then saved.

[0031] The policy network takes a state vector as input and outputs a distribution of action vectors. The distribution of action vectors is characterized by the mean and standard deviation. The action vectors include the energy storage power regulation action component and the energy storage capacity boundary action component. During training, the policy network samples the action vectors based on their distribution. The first value network and the second value network take the state vector and action vector as input, respectively, and output corresponding state-action values. The state-action values ​​are used to evaluate the long-term comprehensive benefits that can be obtained after executing the action vectors under the current grid operating state. The first target value network and the second target value network have the same structure as the first value network and the second target value network, respectively. Their parameters are gradually synchronized by the corresponding value network parameters through a soft update method, and are used to calculate the target Q value during the value network update process.

[0032] Experience samples are sampled from the experience replay pool. The sum of the immediate reward and the expected return of the next state under the target value network is used as a supervision signal to update the first value network and the second value network. The Q-values ​​output by the first value network and the second value network are used as guiding signals for action evaluation to update the policy network, and the parameters of the temperature coefficient are updated. Specifically, this includes: Experience samples are sampled from the experience replay pool. The sum of the immediate reward and the expected return of the next time state under the target value network is used as a supervision signal. The parameters of the first value network and the second value network are updated by minimizing the Bellman residual loss function. The Q-values ​​output by the first value network and the second value network are used as a guiding signal for action evaluation to update the policy network. The parameters of the temperature coefficient are automatically adjusted by gradient descent.

[0033] Specifically, the steps in step one of this embodiment for constructing the maximum entropy Markov decision process (MDP) model for the secondary frequency regulation environment of the power grid include: First, we define the three elements of a reinforcement learning environment: state space, action space, and reward function, which serve as the basis for agent interaction.

[0034] State Space: The environmental state vector observed by the agent Defined as the following vector: ; in, S t express t The environmental state vector perceived by the agent at any given time. This represents the regional control deviation at time t, reflecting the current power imbalance in the regional power grid. This represents the power grid frequency deviation at time t. Its integral reflects frequency stability; express t The state of charge of the energy storage system at all times constrains the physical limitations of energy storage; P thermal,t express t Real-time output of thermal power units reflects their operating conditions; L forecast,t express t Short-term load forecast information at any given time.

[0035] Action Space: In this embodiment, the action space of the intelligent agent is used to describe the real-time control decisions of the energy storage system during the secondary frequency regulation process of thermal power and energy storage. Considering that the energy storage system needs to determine both the charging and discharging power within the current control cycle and the online callable capacity boundary under the current operating conditions when participating in secondary frequency regulation, the action space is defined as a two-dimensional continuous vector: ; in, A t for t The action vector output by the agent at any given time. This is the energy storage power regulation action component, used to generate the active power command of the energy storage system in the current control cycle; These are the energy storage capacity boundary action components, used to generate the online callable capacity limit for the energy storage system to participate in secondary frequency regulation during the current operating period. For ease of policy network output and training, both action components are normalized to [value missing]. Within the range.

[0036] In actual execution, the action vector is not directly used as equipment control command. Instead, the EMS or thermal power-storage co-controller maps the normalized action quantity into control commands with physical meaning according to the rated parameters of the energy storage system, the SOC operating boundary, the PCS power limit, and the adjustment constraints of the thermal power unit. The specific execution steps are as follows.

[0037] First, within each frequency regulation control cycle, the EMS or the thermal power storage co-control system collects the regional control deviation in real time. Grid frequency deviation Frequency deviation integral, energy storage system state of charge Current active power output of thermal power units and short-term load forecast information The aforementioned state variables are then subjected to dimensional unification and normalization to form the agent's input state vector. .

[0038] Secondly, the state vector Input the trained policy network, and the policy network outputs the action vector at the current time step. .in, Characterizes the proportion of frequency regulation power or the direction of regulation that the energy storage system should undertake. This characterizes the capacity participation boundary that the energy storage system should open under current operating conditions.

[0039] Then, the controller performs physical quantity mapping on the motion components. For the power motion component... Based on the maximum discharge power of the energy storage system Maximum charging power PCS power limits and SOC constraints are mapped to energy storage active power commands. .when When, it indicates that the energy storage system is discharging and providing positive frequency regulation power to the grid; when When, it indicates that the energy storage system is charging and absorbing excess power; when When this occurs, it indicates that the energy storage system does not participate in the power regulation of the current cycle. This applies to the capacity action component. According to the rated capacity of the energy storage system The upper and lower limits of SOC and the preset safety margin are mapped to the online callable capacity limit for energy storage to participate in secondary frequency regulation during the current period. .

[0040] The above mapping relationship can be expressed as: in, This is a power action mapping function used to convert normalized power actions into active power setpoints that can be executed by the energy storage PCS. This is a capacity boundary mapping function used to convert normalized capacity actions into a capacity limit that the energy storage system can participate in frequency regulation during the current time period. This capacity limit does not represent a change in the physical rated capacity of the energy storage device during operation, but rather represents the effective capacity range that the energy storage system is allowed to utilize under the current SOC state, safety constraints, and frequency regulation requirements.

[0041] Controller for mapping and Perform a safety check. If the energy storage power command meets the PCS power limit, SOC safety range, charge / discharge rate constraint, and capacity boundary constraint, then... The command is sent to the energy storage converter to perform charge and discharge control, and... The command is sent to the energy storage coordination controller or BMS as an online capacity constraint for energy storage participation in secondary frequency regulation during the current period. If the energy storage power command or capacity boundary exceeds the allowable range, the controller will limit and correct the command, and allocate the remaining frequency regulation demand exceeding the capacity of energy storage to the thermal power units for execution.

[0042] In the process of coordinated frequency regulation of thermal power and energy storage, the energy storage system prioritizes the handling of high-frequency, rapid, and short-time-scale regulation power, while the thermal power unit handles the remaining slow and continuous regulation power. Therefore, Used to determine the magnitude and direction of the charging and discharging power of the energy storage system within the current control cycle. This is used to determine the capacity support level that the energy storage system can participate in secondary frequency regulation during the current operating period. Through joint decision-making based on power commands and capacity boundaries, this embodiment can meet the performance requirements of secondary frequency regulation while avoiding overcharging, over-discharging, or long-term deep cycling operation of the energy storage system, thereby achieving adaptive matching between energy storage capacity and the real-time frequency regulation needs of the power grid.

[0043] Furthermore, to generate capacity configuration recommendations, the controller can perform continuous output analysis within a set statistical period. Perform recording and statistical analysis. For example, using 15 minutes, 1 hour, or a typical FM operating period as a statistical window, calculate the results within that window. The recommended capacity is generated by combining the high quantile value or maximum demand value with a safety margin factor. This recommended capacity is used to guide the capacity planning or operational capacity boundary setting of the energy storage system, thereby transforming the online frequency regulation control results into a basis for energy storage capacity configuration.

[0044] Reward Function: Construct a comprehensive reward function R t To guide agent optimization: ; in, These are the weighting coefficients; the first term is the penalty for ACE bias; the second term... The third term is a function of coal consumption and wear cost of thermal power units. Costs associated with energy storage operation and capacity occupancy; P ess,t The power command that the energy storage system needs to participate in frequency regulation. The online callable capacity limit for the energy storage system to participate in frequency regulation during the current time period, capacity The larger the capacity, the greater the penalty, thus forcing the agent to find the minimum capacity configuration.

[0045] By constructing the state space and action space as described above, the process of energy storage system participating in the secondary frequency regulation of thermal power units is modeled as a control problem that can be directly decided by an intelligent agent. The first component of the action vector... The corresponding energy storage system's charging and discharging power decision within the current frequency regulation cycle is used to generate an energy storage power command after scaling; the second component in the action vector The corresponding energy storage system's capacity decision results for participating in secondary frequency regulation under current operating conditions are used to form energy storage capacity participation instructions, which constrain the available adjustment range of energy storage in subsequent frequency regulation processes.

[0046] The energy storage power command and capacity participation command work together in the secondary frequency regulation process of the thermal power unit. The energy storage system takes priority in undertaking high-frequency and fast power regulation, while the thermal power unit performs slow power compensation according to the remaining frequency regulation demand, thereby reducing the regulation pressure of the thermal power unit.

[0047] In step two of this embodiment, the deep reinforcement learning agent network structure based on the SAC algorithm is constructed, which specifically includes the following steps: Based on the aforementioned Markov decision process for secondary frequency regulation of the power grid, a deep reinforcement learning agent network structure based on the SoftActor-Critic (SAC) algorithm is further constructed. This agent is used to learn from the power grid operating state. To energy storage power command and capacity participation command The mapping relationship provides a basis for decision-making regarding the coordinated secondary frequency regulation of thermal power units and energy storage systems.

[0048] Among them, the input state of the agent Maintaining consistency with the state space defined above, specifically including: ; In the formula, This is the regional control deviation, used to characterize the current degree of power imbalance in the power grid; This refers to the power grid frequency deviation, which reflects the degree to which the frequency deviates from the rated value. This is the integral of the frequency deviation, used to characterize the cumulative effect of the frequency deviation; State of charge (SBC) of an energy storage system reflects its current charge and discharge capacity. This represents the current active power output of the thermal power unit, used to reflect the operating conditions and regulation margin of the thermal power unit. This information serves as short-term load forecasting, reflecting subsequent trends in frequency regulation demand. These state variables collectively constitute the intelligent agent's perception of the operational status of the power grid, thermal power units, and energy storage systems.

[0049] The agent's output action vector Consistent with the action space defined above, specifically: ; in, The energy storage power regulation action component, after being mapped by physical boundaries, forms the energy storage active power command. ; The boundary action components of energy storage capacity are mapped to the physical boundary to form the online callable capacity limit of energy storage. Therefore, the neural network here does not redefine the control variables independently of the above, but rather uses the state space and action space constructed above as the input and output basis to learn the nonlinear mapping relationship between "grid state - energy storage power / capacity decision".

[0050] The SAC agent mainly includes a policy network, two value networks, and two target value networks, as detailed below.

[0051] Policy Network (Actor) ): Policy networks use state vectors As input, output action distribution mean and standard deviation The action distribution is a two-dimensional continuous distribution, corresponding to the energy storage power regulation action component. and energy storage capacity boundary action component During training, the policy network samples action vectors based on this distribution. It is used to interact with the power grid secondary frequency regulation simulation environment; in actual online applications, it can generate energy storage power commands and capacity participation commands based on the average or deterministic actions output by the strategy network.

[0052] Two value networks (Critic, , ): Value networks are based on state and actions As input, the corresponding state-action value is output. This is used to evaluate the long-term comprehensive benefits obtained after making a certain energy storage power and capacity boundary decision under the current grid operating conditions. This benefit comprehensively reflects the secondary frequency regulation effect, the regulation cost of thermal power units, the energy storage operating cost, and the capacity occupancy cost. Using two value networks can reduce the Q-value overestimation problem and improve the stability of the strategy update process.

[0053] Two target value networks (Target Critic, , ): The target value network has the same structure as the value network and is used to calculate the target Q-value during the value network update process. Its parameters are gradually synchronized with the value network parameters through a soft update method to avoid drastic changes in the target value and improve the convergence stability of the training process.

[0054] Through the aforementioned network structure, the policy network is responsible for generating the energy storage power and capacity boundary action distribution based on the current grid state, the value network is responsible for evaluating the long-term benefits of this action in the thermal power-storage coordinated secondary frequency regulation process, and the target value network is responsible for providing stable training objectives. The three types of networks work together to enable the agent to learn a control strategy that balances frequency regulation performance, equipment safety, and operational economy, while satisfying energy storage SOC constraints, PCS power constraints, and thermal power unit ramp-up constraints.

[0055] After training, the action vectors output by the policy network are mapped through physical boundaries to generate energy storage power commands. and online callable capacity limits .in, Used to control the charging and discharging power of the energy storage system during the current control cycle. This is used to limit the capacity range of the energy storage system participating in secondary frequency regulation during the current operating period, thereby providing a decision-making basis for subsequent online adaptive capacity configuration and thermal-storage coordinated frequency regulation control.

[0056] In step three of this embodiment, the offline training and parameter update process of the agent is as follows: In the preceding text, the process of thermal power units and energy storage systems participating in secondary frequency regulation has been modeled as a Markov decision process, and the observable state vector of the agent has been clearly defined. Outputtable action vectors And a reward function for evaluating frequency modulation performance and operating costs. Furthermore, a policy network, value network, and target value network based on the SAC algorithm were established to realize the transformation of state... To action Nonlinear mapping and action value evaluation.

[0057] Building upon this, historical frequency modulation data or simulation environment interaction data are used here to perform offline training and iterative updates on the parameters of the agent network constructed above. Specifically, the agent reads the current state within each training step. Actions are output by the policy network. And through physical boundary mapping, energy storage power commands are generated. and online callable capacity limits Subsequently, the power grid secondary frequency regulation simulation environment returns to the state of the next time step based on this action. and instant rewards This forms an empirical sample ( , , , The data is stored in the experience replay pool for subsequent parameter updates of the value network, policy network, and temperature coefficient.

[0058] Therefore, the logical relationship between these three specific steps is as follows: Step 1 provides the definitions of state, action and reward required for reinforcement learning training; Step 2 provides a trainable agent network structure; Step 3 enables the policy network to gradually learn the optimal energy storage power allocation and capacity boundary setting strategy under different grid frequency deviations, energy storage SOC states and thermal power unit operating conditions through environmental interaction, experience sampling and parameter updates.

[0059] During the training process described above, the SAC algorithm maximizes the maximum entropy objective function. This allows the agent to maintain a certain level of strategy exploration capability while obtaining high long-term rewards. Its objective function can be expressed as: ; in, α This represents the temperature coefficient. Wherein, T This indicates the length of time steps contained in a single training trajectory or a training epoch. ρ π This represents the joint state-action distribution formed by the interaction between the agent and the power grid secondary frequency regulation simulation environment under policy π; R ( s t , a t ) indicates that the intelligent agent is in t Always in a state of readiness s t And perform the action a t The immediate reward obtained afterward is used to comprehensively evaluate the regional control deviation suppression effect, the adjustment cost of thermal power units, and the operation and capacity occupancy costs of energy storage; α represents the temperature coefficient, used to adjust the weight of the entropy term in the objective function, thereby balancing strategy exploration capability and long-term benefit optimization; H(π(·|st)) represents the value obtained in state s t The entropy of the action distribution output by the next policy π is used to measure the randomness and exploratory ability of the policy. The larger the entropy value, the higher the degree of exploration the agent maintains for different actions in the current state. E ( s t , a t )~ ρ π [·] represents the expected value of a sample under the state-action distribution induced by policy π.

[0060] The intelligent agent interacts with the power grid simulation environment, transferring experience ( ) is stored in the experience replay pool. Among them, ( s t )express t The state vector of the power grid's secondary frequency regulation observed by the agent at any given time; a t ) indicates that the agent is in a state ( s t The action vector output by the policy network includes the energy storage power regulation action component and the energy storage capacity boundary action component; r t ) indicates that the agent performs an action ( a t The immediate reward value fed back from the power grid secondary frequency regulation simulation environment is used to characterize the comprehensive effect of this action in terms of frequency regulation deviation suppression, thermal power unit regulation cost, energy storage operation cost, and capacity occupancy cost; s t+1 ) indicates the execution of an action ( a t The state vector obtained after the transition.

[0061] Sample from the playback pool and update as follows at each step: Update value network parameters : Minimize the Bellman residual loss function : ; Where D represents the Replay Buffer, which is used to store experience samples generated during the interaction between the agent and the power grid secondary frequency regulation simulation environment; Indicates the first i The Bellman residual loss function of a value network is used to measure the deviation between the current output value of the value network and the target value. Let represent the network parameters of the i-th value network (i=1,2), corresponding to the first value network and the second value network, respectively; This means performing mathematical expectation calculations after sampling state-action samples from the experience replay pool D. The parameter is The i-th value network is in state s t Next action a t The estimated state-action value; y tThis represents the target value used to update the value network. Its calculation process includes the immediate reward, the target value estimate in the next time step, and the entropy regularization term.

[0062] Target value Calculations including entropy terms: ; In the formula, This represents the discount factor. r ( s t , a t ) indicates that the intelligent agent is in t Always in a state of readiness s t And perform the action a t The immediate reward value fed back by the power grid secondary frequency regulation simulation environment is used to evaluate the immediate impact of the current action on frequency regulation deviation suppression, thermal power unit regulation cost and energy storage operation cost. Indicates the state at the next moment. s t+1 Next, according to the current strategy Sampling to obtain the action at the next moment a t+1 And perform mathematical expectation calculation on the target value corresponding to the action; s t+1 This indicates that the agent is performing the current action. a t The state at the next moment obtained from the environmental transition; a t+1 Indicates the state at the next moment. s t+1 The action vector is obtained from the output or sampling of the policy network; This means taking the smaller of the two target value network estimates for the next state-action pair to reduce overestimation bias in the value estimation process; This represents the entropy regularization correction term, used to reflect the impact of the strategy exploration level in the calculation of the target value.

[0063] Update strategy network parameters : By updating the policy by minimizing the KL divergence, the actions produced by the policy can achieve higher Q values ​​while maintaining high entropy. ; in, This is a reparameterized sampling function. This represents the loss function of the policy network, used to update the policy network parameters so that the policy network can output actions with higher long-term returns while maintaining a certain degree of exploratoryness. Indicates the policy network parameters; This indicates that the sample was obtained from the experience replay pool D. t Time-state sample; This represents a random noise variable sampled from a standard normal distribution, used to implement reparameterized sampling of policy actions; This represents a reparameterized sampling function, i.e., under a given state. s t and random noise Under these conditions, action samples are generated by the policy network; Indicates the policy network in state s t Down Output Action The probability density; This represents the optimization term corresponding to the policy entropy term, used to maintain the policy's exploration capability; This represents the smaller of the two value network estimates for the current state-action pair, used to guide the policy network to select the action with higher long-term returns and reduce the impact of overestimation of value; j represents the value network number, which can be 1 or 2.

[0064] Adaptive adjustment of temperature coefficient : Automatic adjustment via gradient descent To dynamically balance "exploration" and "exploitation" at different stages of training: ; in, Let be the target entropy hyperparameter.

[0065] The strategy network parameters obtained through training enable the generated energy storage power commands to effectively reduce the adjustment range and number of adjustments for thermal power units under the same grid frequency deviation conditions.

[0066] Finally, online adaptive capacity configuration and frequency modulation control are performed, wherein the policy network parameters are saved after training convergence. In practical applications: 1. Real-time data acquisition: The EMS system acquires the real-time status of the power grid every control cycle (e.g., 1 second or 4 seconds). .

[0067] 2. Forward reasoning: Input policy network, output deterministic actions .

[0068] 3. Closed-loop control: (1) Issue power command Control the charging and discharging of energy storage.

[0069] (2) Record the output capacity instruction This value represents the optimal energy storage capacity required to meet frequency regulation needs under the current operating conditions.

[0070] (3) Statistical analysis of data over a period of time (e.g., 15 minutes or 1 hour). The capacity is distributed, and its quantile (such as the 95th percentile) is taken as the recommended capacity configuration for that period, so as to realize dynamic adaptive capacity management.

[0071] The energy storage power command is superimposed or decoupled from the original AGC command of the thermal power unit to jointly complete the regional secondary frequency regulation control.

[0072] This application aims to address the issues of response delay, low regulation accuracy, and unreasonable energy storage system configuration in thermal power units during secondary frequency regulation by using intelligent strategies generated through deep reinforcement learning technology. Its core technical solution is as follows: 1. A decision-making and control framework for secondary frequency regulation in a coordinated thermal power generation and energy storage system is established. This method uses reinforcement learning algorithms as the technical means to generate coordinated control strategies. By constructing a multi-dimensional sensing model that includes grid frequency status, thermal power unit output characteristics, and energy storage state of charge (SOC), the power allocation decision in the secondary frequency regulation process is mapped into an optimal control problem in a continuous action space, enabling the system to have dynamic response capabilities for complex operating conditions.

[0073] 2. Physically Constrained Adaptive Capacity and Power Scheduling: This approach incorporates physical constraints related to grid operation and equipment safety into the control process. The intelligent agent perceives grid area control error (ACE) and load fluctuations in real time. Under the premise of meeting the constraints of energy storage system charge / discharge rate, cycle life degradation characteristics, and thermal power unit ramp-up rate limits, it adaptively outputs the optimal configuration capacity and power commands for energy storage to participate in frequency regulation, achieving precise matching between energy storage resources and frequency regulation needs.

[0074] 3. A multi-objective collaborative dynamic optimization strategy: This method balances the robustness and economy of the control strategy through a maximum entropy framework. Under the core premise of ensuring compliance with the grid secondary frequency regulation performance indicators (such as CPS standards), it effectively reduces the wear and tear from frequent reverse regulation of thermal power units by dynamically adjusting the output ratio of thermal power units and energy storage systems, while also considering the long-life operation of energy storage systems, thereby minimizing the overall operating cost of the thermal-energy storage collaborative system.

[0075] Please see Figure 3 This application proposes an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units. By optimizing the physical output allocation logic and dynamic boundary management of thermal power and energy storage, it achieves the following significant technical effects in terms of grid regulation performance and power generation equipment protection: Firstly, it significantly improves the dynamic response speed and regulation accuracy of secondary frequency regulation. By introducing a fast response mechanism from an energy storage system, the collaborative control strategy developed in this application can utilize the second-level power regulation capability of energy storage to compensate for the output gap of thermal power units caused by the limitation of ramp rate (physical inertia) in real time. Experiments have shown that this collaborative control mechanism significantly reduces the regional control error (ACE) and ensures that the frequency regulation evaluation indicators (such as CPS1 and CPS2) are superior to those of the traditional independent frequency regulation mode of thermal power.

[0076] Secondly, it effectively reduces the mechanical wear of thermal power units and extends equipment life. This method, through a thermal-storage coordination mechanism, prioritizes the allocation of high-frequency, large-amplitude fluctuating loads in the power grid to the energy storage system. This control strategy avoids frequent reverse regulation actions by thermal power units, significantly reducing fatigue wear on mechanical components such as unit valves and turbines, and effectively alleviating the inefficient operation of thermal power units that are underutilized.

[0077] Third, it achieves a physical match between the scale of energy storage configuration and real-time frequency regulation requirements. Unlike traditional fixed-capacity configuration schemes, the adaptive configuration mechanism established in this application can dynamically adjust the online available capacity of energy storage according to the intensity of grid disturbances. This dynamic boundary management ensures sufficient energy to support grid security under extreme operating conditions, while protecting battery life by limiting the depth of charge and discharge under stable operating conditions, thereby maximizing the economic efficiency of the energy storage system throughout its entire life cycle.

[0078] Fourth, it enhances the robustness of the control system under complex and variable power grid conditions. Based on a decision framework with high exploratory capabilities, the cooperative control strategy of this application exhibits stronger adaptability to random disturbances brought about by the grid connection of wind and solar new energy sources. Compared to fixed PID control or rule-based control, the command pairs generated by this method can more flexibly respond to nonlinear load changes, ensuring that the frequency regulation system maintains stable control performance under various typical operating conditions.

[0079] Appendix: Variable Description

[0080] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units.

[0081] Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0082] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0083] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0085] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0086] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units, characterized in that, include: Obtain the power grid operating status data for the current control cycle; The power grid operation status data includes regional control deviation, power grid frequency deviation, frequency deviation integral, energy storage system state of charge, current active power output of thermal power units, and short-term load forecast information. The power grid operation status data is input into a pre-trained policy network to obtain action vectors; the action vectors include energy storage power adjustment action components and energy storage capacity boundary action components; the policy network is modeled by a maximum entropy Markov decision process and trained based on a soft-actor-commentator deep reinforcement learning algorithm. Based on the rated parameters of the energy storage system, the state-of-charge operating boundary, and the power limit of the energy storage converter, the energy storage power regulation action component is mapped to the energy storage active power command, and the energy storage capacity boundary action component is mapped to the online callable capacity limit of the energy storage system participating in secondary frequency regulation during the current operating period. The energy storage active power command is sent to the energy storage converter to execute charge and discharge control, and the online callable capacity limit is sent to the energy storage coordination controller or battery management system as the online capacity constraint for energy storage to participate in secondary frequency regulation in the current period. During the coordinated frequency regulation of thermal power and energy storage, the control system prioritizes the energy storage system to undertake high-frequency, fast, and small-time-scale regulation power, and allocates the remaining frequency regulation demand that exceeds the capacity of the energy storage system to the thermal power units for execution according to the ramp rate limit of the thermal power units; Record and statistically analyze the continuously output online callable capacity limit within a set statistical period, calculate the quantile value within that statistical period, and generate a recommended configuration capacity by combining it with the safety margin coefficient.

2. The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units according to claim 1, characterized in that, The training steps for the policy network include: A maximum entropy Markov decision process model for the secondary frequency regulation environment of the power grid is constructed, defining a state space, an action space, and a reward function. The state vector in the state space is the power grid operating state data, and the action vector in the action space is a two-dimensional continuous action vector composed of the energy storage power regulation action component and the energy storage capacity boundary action component. The reward function is used to evaluate the frequency regulation effect and operating cost generated after executing the action vector in the current state. A deep reinforcement learning agent network structure based on the soft-actor-critic algorithm is constructed. The agent network structure includes a policy network, a first value network, a second value network, a first target value network, and a second target value network. The policy network is used to map from the state space to the action space, i.e., outputting action vectors based on the input state vectors. The first and second value networks are used to evaluate the long-term gains obtained under the state vectors and action vectors, respectively, providing guiding signals for the policy network's updates. The first and second target value networks have the same structure as the first and second value networks, respectively, and are used to provide target values ​​during training to stabilize the training of the value networks. The agent network structure is trained offline by interacting with the power grid secondary frequency regulation simulation environment. In each iteration, the power grid operation status data of the current period is first input into the policy network as the current state to obtain the training action vector. The training action vector is then input into the power grid secondary frequency regulation simulation environment. The next moment's state and immediate reward are calculated based on the current state and action vector. The current state, the training action vector, the immediate reward, and the next moment's state are stored as a set of experience samples in the experience replay pool. Then, experience samples are sampled from the experience replay pool. The sum of the expected returns of the immediate reward and the next moment's state under the target value network is used as a supervision signal to update the first value network and the second value network. The Q-values ​​output by the first value network and the second value network are used as the guiding signal for action evaluation to update the policy network and update the temperature coefficient parameters. The parameters of the first target value network and the second target value network are synchronized through a soft update method until the policy network converges. The converged policy network parameters are then saved.

3. The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units according to claim 2, characterized in that, The state space is defined as follows: ； in, S t express t The environmental state vector perceived by the agent at any given time. Indicates the regional control deviation at time t; △ f t This represents the power grid frequency deviation at time t; express t The state of charge of the energy storage system at all times; P thermal,t express t Real-time output of thermal power units; L forecast,t express t Short-term load forecast information at any given time; The action space is defined as a two-dimensional continuous action vector: ； in, A t for t The action vector output by the agent at any given time. For the energy storage power regulation action component, This represents the boundary action component of the energy storage capacity; The reward function is defined as follows: ； in, These are the weighting coefficients; This is a function of coal consumption and wear cost of thermal power units; Costs associated with energy storage operation and capacity occupancy; P ess,t The power command that the energy storage system needs to participate in frequency regulation. This refers to the online available capacity limit for the energy storage system to participate in frequency regulation during the current time period.

4. The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units according to claim 2, characterized in that, The policy network takes a state vector as input and outputs a distribution of action vectors. The distribution of action vectors is characterized by the mean and standard deviation. The action vectors include the energy storage power adjustment action component and the energy storage capacity boundary action component. During training, the policy network samples the action vectors according to the distribution of the action vectors. The first value network and the second value network take the state vector and the action vector as inputs, respectively, and output the corresponding state-action value; the state-action value is used to evaluate the long-term comprehensive benefits that can be obtained by executing the action vector under the current power grid operating state. The first target value network and the second target value network have the same structure as the first value network and the second value network, respectively. Their parameters are gradually synchronized by the corresponding value network parameters through a soft update method, and are used to calculate the target Q value during the value network update process.

5. The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units according to claim 2, characterized in that, Experience samples are sampled from the experience replay pool. The sum of the immediate reward and the expected return of the next state under the target value network is used as a supervision signal to update the first value network and the second value network. The Q-values ​​output by the first value network and the second value network are used as guiding signals for action evaluation to update the policy network, and the parameters of the temperature coefficient are updated. Specifically, this includes: Experience samples are sampled from the experience replay pool, and the sum of the immediate reward and the expected return of the next time state under the target value network is used as a supervision signal. The parameters of the first value network and the second value network are updated by minimizing the Bellman residual loss function. The policy network is updated using the Q values ​​output by the first value network and the second value network as guiding signals for action evaluation, and the parameters of the temperature coefficient are automatically adjusted through gradient descent.

6. The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units according to claim 1, characterized in that, The mapping relationship between the energy storage power regulation action component and the energy storage active power command is as follows: in, For energy storage active power commands, For power action mapping function, For the energy storage power regulation action component, for t The state of charge of the energy storage system at all times. This represents the maximum discharge power of the energy storage system. This represents the maximum charging power of the energy storage system. The mapping relationship between the energy storage capacity boundary action component and the online callable capacity limit of the energy storage system participating in secondary frequency regulation during the current operating period is as follows: in, The limit on the online callable capacity for secondary frequency modulation during the current runtime period. For the boundary action component of energy storage capacity, For the rated capacity of the energy storage system, This is the limit of the state of charge. This represents the upper limit of the state of charge.

7. The adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units according to claim 1, characterized in that, Before sending the energy storage active power command to the energy storage converter to execute charge and discharge control, the process also includes: Determine whether the energy storage active power command meets the energy storage converter power limit, state of charge safety range and charge / discharge rate constraints; Determine whether the online callable capacity limit meets the capacity boundary constraints; If the energy storage active power command or the online callable capacity limit exceeds the allowable range, the controller will limit and correct the corresponding command, and allocate the remaining frequency regulation demand that exceeds the energy storage's capacity to the thermal power unit for execution.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the adaptive energy storage capacity configuration method for secondary frequency regulation of thermal power units based on deep reinforcement learning as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the adaptive energy storage capacity configuration method based on deep reinforcement learning-assisted secondary frequency regulation of thermal power units as described in any one of claims 1-7.

Images