A Photovoltaic-Storage-Day-ahead Optimization Scheduling Method and System Based on Security Reinforcement Learning

Abstract

This invention belongs to the field of new energy technology and provides a day-ahead optimal scheduling method and system for photovoltaic (PV) and energy storage based on secure reinforcement learning. It acquires predicted PV power generation data, grid demand data, and electricity price data. Considering the revenue from electricity sales at PV / energy storage sites, the cost of curtailment, and the cost of energy storage losses, a day-ahead optimal scheduling model is constructed with the goal of maximizing the net profit of PV / energy storage sites. A constrained Markov decision process is used to describe the optimal scheduling problem of the PV / energy storage system, determining the state space, action space, constraint violation quantities, and reward function. The state space reflects the internal operating state of the PV / energy storage system and external observation information; the action space is defined as the charging and discharging power of energy storage; the constraint violation quantities reflect the degree of violation of the system's power balance constraints; and the reward function is designed based on the optimization objective of the day-ahead optimal scheduling model. A secure reinforcement learning algorithm is used to solve the problem, resulting in an optimized day-ahead scheduling scheme for PV / energy storage. This invention further enhances safety while ensuring the economic efficiency of the sites.

Description

Technical Field

[0001] This invention belongs to the field of new energy technology, specifically relating to a solar-storage day-ahead optimization scheduling method and system based on security reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of photovoltaic (PV) installed capacity, PV grid connection has become a prominent feature of the power system. While providing clean energy to the grid, PV power generation's low inertia poses new challenges to the active power balance and frequency security of the power system. Low-inertia power systems are highly sensitive to frequency disturbances; even minor disturbances can lead to high frequency change rates and frequency deviations, threatening the safe and stable operation of the power system and even causing widespread blackouts. Therefore, grid regulations require PV power plants to have a certain amount of active power reserve to enhance the frequency stability of the power system.

[0004] Energy storage, due to its flexible active power control capabilities, can regulate frequency through rapid charging and discharging when system frequency fluctuations occur. Although configuring energy storage in photovoltaic power plants increases costs, it helps improve the frequency stability of the power system. The economical operation of photovoltaic-energy storage power plants relies on fast and efficient optimization scheduling methods. Currently, photovoltaic-energy storage optimization scheduling methods can be divided into two categories: The first category is methods based on programming theory; mixed-integer programming is one of its typical representatives. These methods can obtain accurate and theoretically optimal solutions for small- to medium-scale problems, but as the problem dimension increases, the solution space grows exponentially, and the computational complexity and solution time increase significantly, making it difficult to meet scheduling requirements. The second category is methods based on heuristic algorithms (such as particle swarm optimization and genetic algorithms). These methods are inspired by biological populations or evolutionary mechanisms and can solve the photovoltaic-energy storage power plant optimization scheduling problem without gradient information. However, these algorithms are unstable and prone to getting trapped in local optima. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a pre-planning optimization method and system for photovoltaic-storage-day planning based on security reinforcement learning. This invention describes the photovoltaic-storage optimization scheduling problem as a constrained Markov decision process. The agent learns by interacting with the environment, utilizing information such as reward feedback and constraint violation rates to learn the optimal scheduling strategy within the safety boundary. The addition of Lagrange multipliers effectively handles the safety constraints, balancing economic objectives and safety constraints while ensuring solution speed.

[0006] According to some embodiments, the present invention adopts the following technical solution:

[0007] A pre-holiday optimization scheduling method for optical storage based on security reinforcement learning includes the following steps:

[0008] Obtain forecasted photovoltaic power generation data, grid-side demand data, and electricity price data;

[0009] Taking into account the electricity sales revenue of photovoltaic and energy storage power plants, the cost of curtailment of photovoltaic power plants, and the cost of energy storage losses, a day-ahead optimization dispatch model is constructed with the goal of maximizing the net profit of photovoltaic and energy storage power plants.

[0010] The constrained Markov decision process is used to describe the optimal scheduling problem of the photovoltaic-storage system. The state space, action space, constraint violation quantity and reward function are determined. The state space reflects the internal operating state of the photovoltaic-storage system and external observation information. The action space is defined as the charging and discharging power of energy storage. The constraint violation quantity can reflect the degree of violation of the system power balance constraint. The reward function is designed based on the optimization objective of the day-ahead optimal scheduling model.

[0011] A secure reinforcement learning algorithm is used to solve the optimization scheduling problem of the photovoltaic-storage system, resulting in an optimized day-ahead scheduling scheme for photovoltaic-storage systems.

[0012] As an alternative implementation, the objective function of the day-ahead optimization scheduling model is to maximize the net profit of the photovoltaic-storage power station, which consists of electricity sales revenue, curtailment costs, and energy storage loss costs.

[0013]

[0014] in, For the net profit of photovoltaic and energy storage facilities, For time range, For the electricity sales revenue of photovoltaic and energy storage power plants, For energy storage loss costs, Costs associated with solar curtailment.

[0015] As an alternative implementation, the constraints of the day-ahead optimization scheduling model include energy storage. Constraints include energy storage charge / discharge state constraints, energy storage charge / discharge power constraints, and system power balance constraints.

[0016] As an alternative implementation, in describing the optimal scheduling problem of a photovoltaic-storage system using a constrained Markov decision process, the constrained Markov decision process uses a seven-tuple. describe, For state space, For the action space, This is the state transition function. For the reward function, To constrain the amount of violations, The initial state distribution, This is the discount factor.

[0017] As a further defined implementation method, the reward function is designed based on the objective function and is expressed as:

[0018]

[0019] in, It is a contractile factor. For the electricity sales revenue of photovoltaic and energy storage power plants, For energy storage loss costs, Costs associated with solar curtailment;

[0020] The constraint violation quantity, based on the system power balance constraint design, is expressed as:

[0021]

[0022] in, It is a contractile factor. for Real-time power grid planned demand for The planned grid-connected power of the photovoltaic and energy storage station.

[0023] As an alternative implementation method, the process of solving the optimal scheduling problem of a photovoltaic-storage system using a secure reinforcement learning algorithm includes:

[0024] Initialize the parameters;

[0025] Initialize the PV-BESS environment status The intelligent agent is based on the state of the environment. and random noise Select Action The environment performs actions and provides feedback on reward values ​​and constraint violation amounts. Based on the selected actions, the environment will change accordingly to generate new states.

[0026] To replay the experience, The samples are stored in the experience replay pool. When the number of samples in the experience replay pool reaches the preset number, random samples are taken from the experience replay pool for the update and training of the Actor and Critic networks.

[0027] The gradient descent method is used to update the Critic network;

[0028] The goal of the Actor network is to maximize the Lagrange function, and the gradient ascent method is used to update the Actor network.

[0029] Update the Lagrange multipliers using the dual gradient ascent method;

[0030] A soft update factor is introduced to update the target network.

[0031] As a further defined implementation, the parameter initialization process includes: randomly initializing the parameters of the Actor, reward, and cost Critic networks. , , Then, the corresponding target network is initialized with the same parameters. , , Initialize Lagrange multipliers and experience replay pool .

[0032] As a further defined implementation, the process of updating the Critic network using gradient descent includes: updating the Critic network mainly involves updating the Reward network and the Cost network, using gradient descent to update both Critic networks, with the loss functions being:

[0033]

[0034] in, and The target Q value can be further expressed as:

[0035]

[0036] in, As a discount factor, , , All are network parameters;

[0037] Q-value function and Approximated by two sets of neural networks, according to the Bellman equation, the value function of the current state-action pair is expressed as:

[0038] .

[0039] As a further defined implementation, with maximizing the Lagrangian function as the objective of the Actor network, the process of updating the Actor network using the gradient ascent method includes:

[0040]

[0041] in, To set the upper limit of the power safety threshold, the Actor network parameters are updated using the gradient ascent method. :

[0042] .

[0043] A day-ahead optimization scheduling system for photovoltaic energy storage based on security reinforcement learning includes:

[0044] The data acquisition module is configured to acquire predicted photovoltaic power generation data, grid-side demand data, and electricity price data;

[0045] The day-ahead optimization scheduling model construction module is configured to comprehensively consider the electricity sales revenue of photovoltaic and energy storage power plants, the cost of curtailment of photovoltaic power plants, and the cost of energy storage losses, and construct a day-ahead optimization scheduling model with the goal of maximizing the net profit of photovoltaic and energy storage power plants.

[0046] The constrained Markov decision process (CDM) construction module is configured to describe the optimal scheduling problem of a photovoltaic-storage system using a constrained Markov decision process. It determines the state space, action space, constraint violation quantity, and reward function. The state space reflects the internal operating state of the photovoltaic-storage system and external observation information. The action space is defined as the charging and discharging power of the energy storage. The constraint violation quantity can reflect the degree of violation of the system's power balance constraints. The reward function is designed based on the optimization objective of the day-ahead optimal scheduling model.

[0047] The secure reinforcement learning algorithm solution module is configured to use the secure reinforcement learning algorithm to solve the optimization scheduling problem of the photovoltaic-storage system, and obtain the optimized day-ahead scheduling scheme for photovoltaic-storage.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] (1) This invention models the optical-storage optimization scheduling problem as a constrained Markov decision process. Under this framework, the agent learns by interacting with the environment and uses information such as reward feedback and constraint violation to learn the optimal scheduling strategy within the safety boundary, thereby achieving synergistic optimization of economy and safety.

[0050] (2) This invention adopts an optimized scheduling method for optical-storage systems based on security reinforcement learning. This method combines the Deep Deterministic Policy Gradient (DDPG) algorithm with the Lagrange multiplier method. The addition of Lagrange multipliers can effectively handle security constraints and balance economic objectives and security constraints.

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0053] Figure 1 This is a flowchart of the optical storage day-ahead optimization scheduling method based on security reinforcement learning in an embodiment of the present invention;

[0054] Figure 2 The above are typical daily electricity price, photovoltaic and grid demand forecast output curves in the embodiments of the present invention;

[0055] Figure 3 This is a schematic diagram of the constrained Markov decision process in an embodiment of the present invention;

[0056] Figure 4 This is a flowchart illustrating the training process based on security reinforcement learning in an embodiment of the present invention.

[0057] Figure 5 This is a schematic diagram of the photovoltaic-storage grid-connected system structure in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the reward curves for different reinforcement learning algorithms in the embodiments of the present invention;

[0059] Figure 7 This is a schematic diagram of constraint violation curves for different reinforcement learning algorithms in embodiments of the present invention;

[0060] Figure 8 This is a schematic diagram of the power balance of the photovoltaic energy storage system in an embodiment of the present invention. Detailed Implementation

[0061] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0062] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0063] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0064] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0065] Example 1

[0066] In one or more embodiments, a pre-holiday optimization scheduling method for optical storage based on security reinforcement learning is disclosed, such as... Figure 1 As shown, the specific process includes the following:

[0067] (1) Obtain predicted photovoltaic power generation data, grid-side demand data, and electricity price data. The typical daily data used in this embodiment is as follows: Figure 2 As shown, when the photovoltaic power generation at the solar-energy storage power station is not generating power, the power grid does not set any grid connection requirements for the power station.

[0068] (2) The day-ahead optimization scheduling model of photovoltaic and energy storage includes an objective function and constraints. Its objective function is to maximize the net profit of the photovoltaic and energy storage station.

[0069] 1) In this embodiment, the net profit of the photovoltaic-storage power station comprehensively considers electricity sales revenue, curtailment costs, and energy storage loss costs:

[0070]

[0071] in, For the net profit of photovoltaic and energy storage facilities, For time range, For the electricity sales revenue of photovoltaic and energy storage power plants, For energy storage loss costs, Costs associated with solar curtailment;

[0072] Electricity sales revenue It can be represented as:

[0073]

[0074] in, for The on-grid electricity price for photovoltaic and energy storage at any time for The planned grid-connected power of the photovoltaic and energy storage power station at any given time;

[0075] Energy storage loss cost It can be represented as:

[0076]

[0077] in, The unit operating loss cost of energy storage for The discharge power that stores energy at all times. For the discharge efficiency of energy storage, for Charging power for continuous energy storage For energy storage charging efficiency;

[0078] Cost of solar curtailment It can be represented as:

[0079]

[0080] in, The unit cost of curtailment of photovoltaic power. for Real-time photovoltaic curtailment power;

[0081] 2) In this embodiment, the constraints include energy storage. Constraints include: energy storage charge / discharge state constraints, energy storage charge / discharge power constraints, and system power balance constraints. (Energy storage) The constraint can be represented as:

[0082]

[0083] in, express The state of charge that stores energy at all times. and These represent the upper and lower limits of the state of charge, respectively. This can be further expressed as:

[0084]

[0085] in, express State of charge (SOC) that stores energy at all times;

[0086] The energy storage charge / discharge state constraints can be expressed as:

[0087]

[0088] in, Let the Boolean variable represent the energy storage discharge state. Let it be a Boolean variable representing the energy storage charging state;

[0089] The energy storage charging and discharging power constraint can be expressed as:

[0090]

[0091]

[0092] in, To store the maximum discharge power, This represents the maximum charging power for energy storage.

[0093] The system power balance constraint can be expressed as:

[0094]

[0095] in, for The power demand of the power grid at any given time.

[0096] (3) The constrained Markov decision process is used to describe the optimal scheduling problem of the photovoltaic-storage grid-connected system, and the state space, action space, constraint violation quantity and reward function are determined as follows:

[0097] In security reinforcement learning, the agent continuously interacts with the environment, accumulates experience through trial and error, and learns new strategies. This process can be explained by constrained Markov decision processes, such as... Figure 3 As shown, each change in the environment can be represented by a state. The agent will take an action based on this state value, and the environment will provide feedback on the selected action, including reward values ​​and constraint violation amounts. Furthermore, based on the selected action, the environment will change accordingly, generating a new state.

[0098] Generally speaking, constrained Markov decision processes can be represented by 7-tuples. To describe. For the optimization problem constructed in this embodiment, wherein, For state space, For the action space, This is the state transition function. For the reward function, To constrain the amount of violations, The initial state distribution, Discount factor:

[0099] 1) The state space can reflect the internal operating state of the photovoltaic-storage system. and external observation information, including Constant state of energy storage Photovoltaic power generation forecast Power grid planned demand Photovoltaic-storage grid-connected electricity price , can be represented as:

[0100]

[0101] 2) The action space can be defined as the charging and discharging power of energy storage. , can be represented as:

[0102]

[0103] in, for The charging and discharging power of energy stored at all times.

[0104] 3) The reward function is designed based on the objective function and can be expressed as:

[0105]

[0106] in, It is a contraction factor.

[0107] 4) The constraint violation quantity, based on the system power balance constraint design, can be expressed as:

[0108]

[0109] in, It is a contraction factor.

[0110] (4) Solve the problem using a secure reinforcement learning algorithm, such as... Figure 4 As shown, the specific process includes the following:

[0111] Initialization: Randomly initialize the parameters of the Actor, reward, and cost networks. , , Then, the corresponding target network is initialized with the same parameters. , , Initialize Lagrange multipliers and experience replay pool .

[0112] 1) Interaction: Initialize the PV-BESS environment state The intelligent agent is based on the state of the environment. and random noise Select Action The environment performs actions to provide feedback on reward values ​​and constraint violations, and based on the selected actions, the environment will change accordingly to generate new states.

[0113] 2) Experience Review: First, […] The samples are stored in the experience replay pool. When the number of samples in the experience replay pool reaches the preset number, random samples are taken from the experience replay pool for the update and training of the Actor and Critic networks.

[0114] 3) Critic Network Update: The Critic network update mainly involves updating the Reward network and the Cost network. Gradient descent is used to update both Critic networks, with the following loss functions:

[0115]

[0116] in, and The target Q value can be further expressed as:

[0117]

[0118] in, This is the discount factor.

[0119] Q-value function and Approximated by two sets of neural networks, according to the Bellman equation, the value function of the current state-action pair can be expressed as:

[0120]

[0121] 4) Actor Network Update: The goal of the Actor network is to maximize the Lagrange function.

[0122]

[0123] in, To set the upper limit of the power safety threshold, the Actor network parameters are updated using the gradient ascent method. :

[0124]

[0125] 5) Lagrange multiplier update: To ensure that constraints are satisfied as much as possible during the optimization process, the Lagrange multipliers can be updated using the following formula:

[0126]

[0127] 6) Target network update: The target network can be updated using a soft update method.

[0128]

[0129] in, This is a soft update factor.

[0130] The photovoltaic-storage grid-connected system studied in this embodiment is as follows: Figure 5 As shown, a photovoltaic-storage grid-connected system consists of three core components: a photovoltaic system, a grid-connected energy storage system (BESS), and the power grid. The photovoltaic system converts solar energy into electricity, which can be stored in the BESS or directly supplied to the grid. The BESS exchanges energy with the photovoltaic system and the grid through charging and discharging. Specifically, when the predicted photovoltaic power generation exceeds the grid's planned demand, the BESS stores the excess energy. Conversely, when photovoltaic power generation falls short of the grid's planned demand, the BESS releases the stored energy to compensate for the power shortage.

[0131] To verify the effectiveness of the proposed algorithm, it was compared with the traditional DDPG algorithm. Both algorithms were trained for 2000 rounds to learn the optimal scheduling strategy for the optical storage system. To more intuitively illustrate the changes in reward values, this embodiment adds a smoothed cumulative reward during training, such as... Figure 6 As shown, the convergence rounds of the DDPG algorithm and the proposed algorithm are approximately 750 and 500, respectively. The proposed algorithm exhibits a faster convergence speed, and its reward value is significantly higher than that of the DDPG algorithm. Figure 7 The power constraint violation curves of different methods were compared. It can be seen from the small window that the power violation of the proposed algorithm is lower than that of the DDPG algorithm. Figure 8The optimal scheduling results of the photovoltaic-storage system obtained by the proposed method on a typical day are shown. The results demonstrate that the proposed algorithm satisfies the system's power balance requirements. The profits of the photovoltaic-storage system optimized by the DDPG algorithm and the proposed algorithm are RMB 22,508.83 and RMB 22,664.5, respectively, indicating that the proposed method achieves an improvement of approximately 0.69%. The constraint violation amount of the proposed algorithm is 28.14 kW, which is approximately 12.7% lower than the 32.23 kW of the DDPG algorithm. This demonstrates that the proposed algorithm improves the economics of photovoltaic-storage sites while ensuring safety.

[0132] Example 2

[0133] In one or more embodiments, a pre-holiday optimization scheduling system for optical storage based on security reinforcement learning is disclosed, comprising:

[0134] The data acquisition module is configured to acquire predicted photovoltaic power generation data, grid-side demand data, and electricity price data;

[0135] The day-ahead optimization scheduling model construction module is configured to comprehensively consider the electricity sales revenue of photovoltaic and energy storage power plants, the cost of curtailment of photovoltaic power plants, and the cost of energy storage losses, and construct a day-ahead optimization scheduling model with the goal of maximizing the net profit of photovoltaic and energy storage power plants.

[0136] The constrained Markov decision process (CDM) construction module is configured to describe the optimal scheduling problem of a photovoltaic-storage system using a constrained Markov decision process. It determines the state space, action space, constraint violation quantity, and reward function. The state space reflects the internal operating state of the photovoltaic-storage system and external observation information. The action space is defined as the charging and discharging power of the energy storage. The constraint violation quantity can reflect the degree of violation of the system's power balance constraints. The reward function is designed based on the optimization objective of the day-ahead optimal scheduling model.

[0137] The secure reinforcement learning algorithm solution module is configured to use the secure reinforcement learning algorithm to solve the optimization scheduling problem of the photovoltaic-storage system, and obtain the optimized day-ahead scheduling scheme for photovoltaic-storage.

[0138] The specific implementation process of each of the above modules is the same as that in Example 1, and will not be described in detail again.

[0139] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0140] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A pre-schedule optimization scheduling method for optical storage based on security reinforcement learning, characterized in that, Includes the following steps: Obtain forecasted photovoltaic power generation data, grid-side demand data, and electricity price data; Taking into account the electricity sales revenue of photovoltaic and energy storage power plants, the cost of curtailment of photovoltaic power plants, and the cost of energy storage losses, a day-ahead optimization dispatch model is constructed with the goal of maximizing the net profit of photovoltaic and energy storage power plants. The constrained Markov decision process (CMM) is used to describe the optimal scheduling problem of a photovoltaic (PV) energy storage system. The state space, action space, constraint violations, and reward function are determined. The state space reflects the internal operating state of the PV system and external observation information. The action space is defined as the charging and discharging power of the energy storage. Constraint violations reflect the degree of violation of the system's power balance constraints. The reward function is designed based on the optimization objective of the day-ahead optimal scheduling model. The constraint violations are expressed as follows: in, It is a contractile factor. for Real-time power grid planned demand for The planned grid-connected power of the photovoltaic and energy storage power station at any given time; A secure reinforcement learning algorithm is used to solve the optimal scheduling problem of a photovoltaic-storage system, resulting in an optimized day-ahead scheduling scheme. The process includes: updating the Critic network using gradient descent; updating the Actor network using gradient ascent with the goal of maximizing the Lagrange function; updating the Lagrange multipliers using dual gradient ascent; and introducing a soft update factor to update the target network. The Actor network update process is as follows: in, and The value function of the state-action pair. s i for i The state at any given moment, a i for i Actions at any moment For Lagrange multipliers, , , For network parameters, To set the upper limit of the power safety threshold, the Actor network parameters are updated using the gradient ascent method. : in, This represents the initial state distribution.

2. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 1, characterized in that, The objective function of the day-ahead optimization scheduling model is to maximize the net profit of the photovoltaic-storage power station, which consists of electricity sales revenue, curtailment costs, and energy storage loss costs. in, For the net profit of photovoltaic and energy storage facilities, For time range, For the electricity sales revenue of photovoltaic and energy storage power plants, For energy storage loss costs, Costs associated with solar curtailment.

3. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 1, characterized in that, The constraints of the day-ahead optimization scheduling model include energy storage. Constraints include energy storage charge / discharge state constraints, energy storage charge / discharge power constraints, and system power balance constraints.

4. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 1, characterized in that, In describing the optimal scheduling problem of a photovoltaic-storage system using a constrained Markov decision process, the constrained Markov decision process uses 7-tuples. describe, For state space, For the action space, This is the state transition function. For the reward function, To constrain the amount of violations, The initial state distribution, This is the discount factor.

5. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 1, characterized in that, The reward function is based on the objective function and is expressed as: in, It is a contractile factor. For the electricity sales revenue of photovoltaic and energy storage power plants, For energy storage loss costs, Costs associated with solar curtailment.

6. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 1, characterized in that, The process of solving the optimal scheduling problem of a photovoltaic-storage system using a secure reinforcement learning algorithm includes: Initialize the parameters; Initialize the PV-BESS environment status The intelligent agent is based on the state of the environment. and random noise Select Action The environment performs actions and provides feedback on reward values ​​and constraint violation amounts. Based on the selected actions, the environment will change accordingly to generate new states. To replay the experience, The samples are stored in the experience replay pool. When the number of samples in the experience replay pool reaches the preset number, random samples are taken from the experience replay pool for the update and training of the Actor and Critic networks.

7. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 6, characterized in that, The parameter initialization process includes: randomly initializing the parameters of the Actor, reward, and cost Critic networks. , , Then, the corresponding target network is initialized with the same parameters. , , Initialize Lagrange multipliers and experience replay pool .

8. The optical storage day-ahead optimization scheduling method based on security reinforcement learning as described in claim 6, characterized in that, The process of updating the Critic network using gradient descent includes: updating the Reward network and the Cost network, using gradient descent to update both Critic networks, with the following loss functions: in, and The target Q value is expressed as: in, As a discount factor, , , All are network parameters; Q-value function and Approximated by two sets of neural networks, according to the Bellman equation, the value function of the current state-action pair is expressed as: 。 9. A day-ahead optimization scheduling system for photovoltaic energy storage based on security reinforcement learning, characterized in that it includes: The data acquisition module is configured to acquire predicted photovoltaic power generation data, grid-side demand data, and electricity price data; The day-ahead optimization scheduling model construction module comprehensively considers the electricity sales revenue of photovoltaic and energy storage power plants, the cost of curtailment of photovoltaic power plants, and the cost of energy storage losses, and constructs a day-ahead optimization scheduling model with the goal of maximizing the net profit of photovoltaic and energy storage power plants. The Constrained Markov Decision Process (CPM) construction module is configured to describe the optimal scheduling problem of a photovoltaic (PV) energy storage system using CPM. It determines the state space, action space, constraint violations, and reward function. The state space reflects the internal operating state of the PV system and external observation information. The action space is defined as the charging and discharging power of the energy storage. Constraint violations reflect the degree of violation of the system's power balance constraints. The reward function is designed based on the optimization objective of the day-ahead optimal scheduling model. The constraint violations are expressed as: in, It is a contractile factor. for Real-time power grid planned demand for The planned grid-connected power of the photovoltaic and energy storage power station at any given time; The secure reinforcement learning algorithm solution module is configured to solve the photovoltaic-storage system optimization scheduling problem using a secure reinforcement learning algorithm, obtaining an optimized photovoltaic-storage day-to-day scheduling scheme. The process includes: updating the Critic network using gradient descent; updating the Actor network using gradient ascent with maximizing the Lagrange function as the objective; updating the Lagrange multipliers using dual gradient ascent; and introducing a soft update factor to update the target network. The Actor network update process is as follows: in, and The value function of the state-action pair. s i for i The state at any given moment, a i for i Actions at any moment For Lagrange multipliers, , , For network parameters, To set the upper limit of the power safety threshold, the Actor network parameters are updated using the gradient ascent method. : in, This represents the initial state distribution.

Images