Multi-scene energy storage adaptive configuration method, system and device for migration deep reinforcement learning oriented to light storage direct flexibility, and medium

Abstract

The invention relates to the technical field of energy storage configuration of a direct-current flexible storage system, and discloses a light-storage direct-current flexible-oriented migration deep reinforcement learning multi-scene energy storage adaptive configuration method, system and device, and a medium, and the method comprises the steps: obtaining historical operation and device parameters, and constructing a typical operation scene set based on a clustering algorithm; constructing a double-layer Markov decision model; the upper-layer model takes the optimal annual comprehensive economic benefit of the system as a reward function, investment out-of-limit penalty is introduced, and the lower-layer model takes the optimal daily operation economic benefit as a reward function, and system security constraints are combined; a TD3 algorithm is improved based on a multi-scene migration strategy and a smooth constraint technology, a double-layer Markov decision model is solved, and a multi-scene energy storage adaptive configuration scheme considering investment benefits and operation benefits is formed. According to the invention, adaptive planning configuration of the energy storage capacity of the optical storage direct-flexible system can be realized under the condition of multiple scenes, the system investment and operation and maintenance cost are reduced, and the power supply reliability and operation economy of the optical storage direct-flexible system are improved.

Description

Technical Field

[0001] This invention relates to the field of energy storage configuration technology for photovoltaic-storage-direct-flexible systems, and in particular to a method, system, device, and medium for adaptive configuration of energy storage in multiple scenarios using transfer deep reinforcement learning for photovoltaic-storage-direct-flexible systems. Background Technology

[0002] With the large-scale integration of renewable energy into the grid and the development of distributed energy systems, the importance of photovoltaic-storage-DC-flexible systems (PV-SSDS-Flexible Systems) in improving regional power supply reliability, optimizing energy dispatch, and achieving green and low-carbon goals is becoming increasingly prominent. Under conditions of uncertainty in photovoltaic power generation and load fluctuations, PV-SSDS-Flexible Systems can achieve local power self-sufficiency through flexible dispatch of energy storage devices, reducing dependence on the external grid and improving energy utilization efficiency. However, in complex operating environments with multiple scenarios, how to scientifically configure energy storage capacity and formulate adaptive operating strategies, balancing system economy, energy storage lifespan, and power supply reliability, remains a key issue restricting the efficient operation of PV-SSDS-Flexible Systems.

[0003] In recent years, deep reinforcement learning has offered a potential solution to this problem due to its strong adaptability and ability to optimize long-term benefits. Some research focuses on photovoltaic-storage-direct current-flexible energy storage systems, optimizing capacity and power models with economic goals, and employing particle swarm optimization (PSO) to optimize the configuration of these parameters. However, this method only optimizes energy storage in a single scenario and exhibits poor generalization ability across multiple scenarios. Other research has proposed a method for designing the capacity configuration of key equipment in photovoltaic-storage-direct current-flexible energy storage systems for civil buildings, based on an improved energy valley optimization algorithm. While this method can achieve unified configuration of multiple devices, including energy storage, it does not consider the subsequent intraday optimization of the system. Still other technologies transform the system's energy storage operation problem into an agent-based optimization problem, proposing a TD3 algorithm that considers system operational constraints to solve it, ensuring the safe operation of the energy storage system. However, the TD3 algorithm in this method shows poor training performance in multiple scenarios, and the output strategy exhibits significant oscillations, which will affect the system's operational stability and equipment lifespan. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a transfer deep reinforcement learning-based adaptive configuration method and system for photovoltaic-storage-direct-flexible systems in multiple scenarios, which solves the problems of poor generalization ability, difficulty in taking into account intraday optimization operation, and large oscillations in the output strategy of current configuration methods for photovoltaic-storage-direct-flexible systems in multiple scenarios.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a transfer deep reinforcement learning-based adaptive energy storage configuration method for multiple scenarios, oriented towards optical-storage integration and flexibility, comprising: Obtain historical photovoltaic, energy storage, load, electricity price and equipment parameters, and construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; Based on a set of typical operating scenarios, a two-layer Markov decision model is constructed. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces penalties for exceeding investment limits. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. The TD3 algorithm is improved based on multi-scenario migration strategy and smoothing constraint technique; An improved TD3 algorithm is used to solve a two-layer Markov decision model. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed.

[0007] As a preferred embodiment of the adaptive energy storage configuration method for multi-scenario energy storage using transfer deep reinforcement learning, which is oriented towards direct and flexible photovoltaic energy storage as described in this invention, the upper-layer model uses the optimal comprehensive economic benefit of the system throughout the year as the reward function and introduces an investment limit penalty, including: Construct an upper-level state space, which includes external environment information and system equipment parameters for typical scenarios, specifically: photovoltaic power output characteristics, load characteristics, electricity price levels, and equipment-related parameters for each scenario; Construct an upper-level action space, defining the action space as the rated installed capacity and rated power of energy storage. The upper-level reward function includes the system's annual operating revenue, the equivalent annual investment cost of energy storage equipment, the annual operation and maintenance cost, and penalties for exceeding investment limits.

[0008] As a preferred embodiment of the adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for a direct-to-flexible photovoltaic-energy storage system, as described in this invention, the lower-level model uses the optimal daily operating economic benefit as the reward function, combined with system safety constraints, and includes: Construct a lower-level state space and set the state space for daily operation, specifically including the current energy level of energy storage, photovoltaic power generation capacity, load demand and electricity price signals, while distinguishing the operating conditions under different typical scenarios; The lower-level action space is defined as the charging and discharging power of energy storage at each moment and the amount of electricity purchased and sold by the system to the grid at each moment; The lower-level reward function includes the system's daily operating revenue and the system's safety constraint penalty, where the safety constraint penalty includes penalties for exceeding the energy storage state of charge limit and penalties for exceeding the grid-connected power limit.

[0009] As a preferred embodiment of the adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for optical-storage integration described in this invention, the lower-layer model further includes: A mathematical model of the energy storage device is used to update the state of charge of the energy storage device according to the charging and discharging actions; The mathematical model defines the self-discharge rate, charging efficiency, and discharging efficiency of the energy storage device, and ensures that its state of charge always operates within the preset safety limits.

[0010] As a preferred embodiment of the adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for a direct-to-flexible optical-storage system, as described in this invention, the TD3 algorithm is improved based on a multi-scenario transfer strategy and smoothing constraint technology, including: In the multi-scenario migration strategy, scenario information is integrated into the action policy network and value evaluation network, enabling the algorithm to share knowledge and adaptively adjust between different scenarios. Introduce action smoothing loss term and parameter smoothing loss term into the loss function of action policy network to suppress action jumps between adjacent time steps and limit parameter drift.

[0011] As a preferred embodiment of the adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for a direct-to-flexible photovoltaic-energy storage system, as described in this invention, the method utilizes an improved TD3 algorithm to solve a two-layer Markov decision model. Through iterative interaction between the upper and lower layers, a multi-scenario adaptive configuration scheme for energy storage that balances investment and operational efficiency is formed, including: Initialize the upper and lower layer policy networks, dual Critic networks, target networks, and experience replay pools; Based on empirical rules and economic constraints, a preliminary energy storage configuration scheme is selected, while noise training is conducted and the scheme is then tailored to the feasible region. Based on the energy storage configuration capacity and power given by the upper layer, the lower layer Actor network is trained and outputs the energy storage charging and discharging and system power purchase and sale actions for each time period of the day, while returning real-time revenue and constraint feedback. The lower layer stores interaction samples and updates the upper-layer TD3 dual Critic network; the Actor network is updated under latency conditions, and action and parameter smoothing regularization is added; The daily operational efficiency and constraint results output from the lower layer are fed back to the upper layer, and combined with investment and maintenance costs and investment over-limit penalties, a reward is formed for the upper layer. The upper layer stores interaction samples and updates the lower layer TD3 dual Critic network Actor network, also introducing action and parameter smoothing regularization; during sampling, samples from multiple scenarios are mixed to complete transfer learning; The updated upper-layer energy storage configuration scheme is passed to the lower layer. The lower layer's intraday operation scheduling strategy is retrained under the new energy storage configuration capacity and the results are returned, forming a cyclical iterative optimization. When the training reaches the set maximum number of iterations, the optimal energy storage configuration and corresponding operating strategy are output.

[0012] As a preferred embodiment of the transfer deep reinforcement learning-based multi-scenario adaptive energy storage configuration method for optical-storage-linear-flexible systems described in this invention, the method includes: constructing a typical operating scenario set for the optical-storage-linear-flexible system based on a clustering algorithm, comprising: The preprocessed multidimensional data is concatenated into a sample vector; Based on sample vectors, a clustering algorithm is used to cluster historical multidimensional operating data to generate a set of scenarios that can characterize different typical operating conditions of the system.

[0013] Secondly, the present invention provides a transfer deep reinforcement learning-based multi-scenario adaptive energy storage configuration system for optical-storage integration, comprising: The acquisition module is used to acquire historical photovoltaic, energy storage, load, electricity price and equipment parameters, and to construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; The two-layer Markov decision model construction module is used to build a two-layer Markov decision model based on a set of typical operating scenarios. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces investment over-limit penalties. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. The optimization module is used to improve the TD3 algorithm based on multi-scenario migration strategies and smoothing constraint techniques; The output module is used to solve the two-layer Markov decision model using the improved TD3 algorithm. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed.

[0014] Thirdly, the present invention provides a computer device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of a transfer deep reinforcement learning method for adaptive energy storage configuration in multiple scenarios with optical-storage direct-flexible energy storage.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the aforementioned adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for optical-storage direct-flexible energy storage.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: First, it acquires relevant information such as photovoltaic, energy storage, load, electricity price, and system equipment parameters, and constructs a typical operating scenario set for a photovoltaic-storage-direct current-flexible system based on a clustering algorithm. Second, it constructs a two-layer Markov decision model: the upper layer uses the optimal comprehensive economic benefit of the system throughout the year as the reward function and introduces an investment over-limit penalty to determine the energy storage capacity configuration; the lower layer uses the optimal daily operating economic benefit as the reward function, while considering system safety constraints to optimize the energy storage charging and discharging strategy, achieving multi-scenario adaptive operation. Then, it improves the TD3 algorithm based on multi-scenario migration strategies and smoothing constraint techniques to enhance cross-scenario adaptability and suppress strategy oscillations, thereby improving the algorithm's convergence and stability. Finally, it uses the improved TD3 algorithm to solve the two-layer Markov decision model, and through interactive iteration between the upper and lower layers, forms a multi-scenario adaptive energy storage configuration scheme that balances investment and operational benefits. The method proposed in this invention can achieve adaptive planning and configuration of energy storage capacity for a photovoltaic-storage-direct current-flexible system under multiple scenario conditions, reducing system investment and operation and maintenance costs, and improving the power supply reliability and operational economy of the photovoltaic-storage-direct current-flexible system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall process of a transfer deep reinforcement learning-based adaptive configuration method for energy storage in multiple scenarios, which is oriented towards optical-storage direct-flexible energy storage, according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the two-layer model structure of energy storage configuration in a transfer deep reinforcement learning method for adaptive energy storage configuration in multiple scenarios, which is oriented towards optical-storage direct-flexible energy storage according to an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the improved TD3 algorithm structure in a multi-scenario adaptive energy storage configuration method based on transfer deep reinforcement learning for optical-storage direct-flexible energy storage, as described in an embodiment of the present invention.

[0021] Figure 4 This is a comparison chart of the average training reward values ​​of the upper-layer model under different algorithms in a transfer deep reinforcement learning method for adaptive energy storage configuration in multiple scenarios, which is oriented towards optical storage and flexible energy storage according to an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0023] Example 1, referring to Figure 1 As one embodiment of the present invention, a transfer deep reinforcement learning-based adaptive configuration method for multi-scenario energy storage, oriented towards optical-storage integration and flexibility, is provided, comprising: S100: Obtain historical photovoltaic, energy storage, load, electricity price and equipment parameters, and construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; S200: Based on a set of typical operating scenarios, a two-layer Markov decision model is constructed. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces penalties for exceeding investment limits. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. S300: An improvement to the TD3 algorithm based on multi-scenario migration strategy and smoothing constraint technology; S400: The improved TD3 algorithm is used to solve the two-layer Markov decision model. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed.

[0024] Specifically, most current energy storage configuration methods are designed for a single scenario and lack the ability to adapt to complex operating environments in multiple scenarios. Furthermore, some methods focus on equipment capacity configuration and fail to effectively integrate with the subsequent intraday optimization operation of the system, resulting in a disconnect between configuration and operation strategies. The methods have poor training effects in multiple scenarios, and the output strategies also exhibit significant oscillations, affecting the stability of system operation and equipment lifespan.

[0025] Therefore, this invention sets up S100-S400 to construct a multi-scenario set through clustering, and constructs a Markov decision model with division of labor between upper and lower layers, while optimizing long-term investment and short-term operation of energy storage; then improves the reinforcement learning algorithm to enhance the adaptability of multiple scenarios and the stability of the strategy, and finally solves the model through the algorithm to output a solution for energy storage configuration and operation that can automatically adapt to different scenarios and take into account both economy and safety.

[0026] Example 2, refer to Figures 1-3 As an embodiment of the present invention, based on the above embodiment, a multi-scenario adaptive energy storage configuration method using transfer deep reinforcement learning for optical-storage integration is provided. The method includes: S100: Obtain historical photovoltaic, energy storage, load, electricity price and equipment parameters, and construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; Specifically, in step S100, the multi-dimensional data required for the planning and configuration of the photovoltaic-storage-direct-drive-flexible system are obtained, mainly including historical photovoltaic, energy storage, load, electricity price and equipment parameters; The historical photovoltaic data includes: solar irradiance, module temperature, and actual output power of photovoltaic units under historical meteorological conditions, forming a real power output sample sequence; historical energy storage data includes: user load power curves for different time periods, covering typical days and seasons, to show the fluctuation pattern of load energy consumption; historical load data includes: information such as the state of charge, charge and discharge efficiency, lifespan degradation characteristics, and rated capacity of energy storage equipment, to reflect the dynamic characteristics of energy storage equipment under different scenarios; historical electricity prices include: time-of-use electricity prices and peak electricity prices, combined with historical fluctuation patterns, to provide input for economic constraints; and equipment parameters include: equipment parameters such as the capacity limit and power loss coefficient of DC transmission lines.

[0027] In one optional implementation, after acquiring the data in S100, data preprocessing is also included, which involves resampling time series data such as photovoltaic, load, and electricity price according to a uniform sampling interval; scaling data of different dimensions such as power, electricity price, and state of charge to a uniform scale using a standard deviation normalization method; and filling in missing data using linear interpolation.

[0028] In this embodiment of the application, step S100, which constructs a typical operating scenario set for the optical-storage-direct-drive-flexible system based on a clustering algorithm, includes the following steps A1-A2: A1: Concatenate the preprocessed multidimensional data into a sample vector; Specifically, concatenating the preprocessed multidimensional data into a sample vector can be represented as: In the formula, For sample vectors; Photovoltaic output power; Power required by the system load; This represents the upper limit of the state of charge for energy storage devices. This is the limit of the state of charge of energy storage devices; The charging efficiency of energy storage devices; The discharge efficiency of energy storage devices; The electricity price that the system purchases from the grid; The electricity price sold by the system to the grid; This refers to the rated power of the inverter. This represents the maximum power exchanged between the system and the power grid.

[0029] A2: Based on sample vectors, a clustering algorithm is used to cluster historical multidimensional operating data to generate a set of scenarios that can characterize different typical operating conditions of the system; Specifically, clustering algorithms are used to perform cluster analysis on historical multi-day operational data, and the resulting set of typical operational scenarios is denoted as follows: In the formula, A collection of scenes; For scenario 1; For the scene ; For the scene The total number of scenes is .

[0030] It should be noted that the constructed scenario set should be able to comprehensively characterize the operating characteristics of the optical-storage-direct-flexible system under different typical operating conditions, so as to provide environmental state input for the subsequent training of deep reinforcement learning strategies.

[0031] S200: Based on a set of typical operating scenarios, a two-layer Markov decision model is constructed. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces penalties for exceeding investment limits. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. It should be noted that the adaptive configuration problem of energy storage in a photovoltaic-storage-direct-drive-flexible system has the dual characteristics of long-term investment planning and short-term operation optimization. Therefore, it can be modeled as a two-layer Markov decision model to describe the annual capacity configuration and the daily operation strategy, respectively. The upper layer is used to determine the energy storage capacity configuration, and the lower layer is used to optimize the energy storage charging and discharging strategy to achieve adaptive operation in multiple scenarios. Overall optimization is achieved through interactive iteration between the upper and lower layers.

[0032] The constructed two-level Markov decision model is as follows: In this embodiment of the application, the upper-level model in step S200 uses the optimal comprehensive economic benefit of the system throughout the year as the reward function and introduces an investment over-limit penalty, including the following steps B1-B3: B1: Construct the upper-level state space, which includes external environment information and system equipment parameters for typical scenarios, specifically: photovoltaic power output characteristics, load characteristics, electricity price level and equipment-related parameters under each scenario; Specifically, the state space mainly includes external environmental information and system equipment parameters for typical scenarios: photovoltaic output characteristics, load characteristics, electricity price levels, and equipment-related parameters under each scenario, which can be represented as: In the formula, These are the state variables of the upper-level model; For scene sequence number; For the scene Characteristics of photovoltaic power generation under the following conditions; For the scene Under the following load characteristics; For the scene The current electricity price level; This refers to the rated power of the inverter. This represents the maximum interaction power between the system and the power grid. This represents the total number of scenes.

[0033] B2: Construct the upper-level action space, defining the action space as the rated installed capacity and rated power of energy storage; Specifically, the action space is used to determine the configuration scheme at the system investment level, and is defined as the rated installed capacity and rated power of energy storage: In the formula, The amount of motion in the upper-level model; The rated installed capacity of the energy storage device; The rated charge and discharge power of the energy storage device.

[0034] B3: The upper-level reward function includes the system's annual operating revenue, the equivalent annual investment cost of energy storage equipment, the annual operation and maintenance cost, and the penalty for exceeding the investment limit.

[0035] Specifically, the energy storage configuration optimization layer of a photovoltaic-storage-direct-drive-flexible system focuses on investment planning costs while also considering the investment budget. Therefore, the reward function is defined as follows: In the formula, This is the reward function for the upper-level model; For the daily operating revenue of the photovoltaic-storage-DC-flexible system; This represents the equivalent annual investment cost of energy storage equipment. The annual operation and maintenance cost of energy storage equipment; Penalties will be imposed for exceeding investment limits.

[0036] Annualized investment cost item: In the formula, This represents the equivalent annual investment cost of energy storage equipment. The unit power installation cost of energy storage equipment; The unit capacity installation cost of energy storage equipment; The discount rate; This refers to the lifespan of energy storage equipment.

[0037] Annual maintenance costs: In the formula, The annual operation and maintenance cost of energy storage equipment; This represents the total number of scenes; For the scene The weight of the overall scenario; For the scene Annual operation and maintenance costs of energy storage equipment.

[0038] Penalties for exceeding investment limits: In the formula, Penalties for exceeding investment limits; The penalty coefficient for exceeding investment limits; This represents the annual investment budget for the system.

[0039] In this embodiment of the application, the lower-level model in step S200 uses the optimal daily operating economic efficiency as the reward function, combined with system security constraints, and includes the following steps C1-C3: C1: Construct the lower-level state space and set the state space for intraday operation, specifically including the current energy level of energy storage, photovoltaic power generation capacity, load demand and electricity price signals, while distinguishing the operating conditions under different typical scenarios; Specifically, the intraday operating state space includes the current energy level of energy storage, photovoltaic power generation capacity, load demand and electricity price signals, while also distinguishing the operating conditions under different typical scenarios.

[0040] In the formula, These are the state variables of the lower-level model; For photovoltaics Actual output at any given moment; For system load in Power required at any given moment; For energy storage devices The state of charge at any given moment; for The electricity price that the time-of-use system purchases from the power grid; for The electricity price sold to the grid by the time-of-use system; For the operating scenario.

[0041] C2: Construct the lower-level action space, defined as the charging and discharging power of energy storage at each moment and the amount of electricity purchased and sold by the system to the grid at each moment; Specifically, the action space is defined as the charging and discharging power of the energy storage at each moment and the amount of electricity purchased and sold by the system to the grid at each moment: In the formula, For the motion quantities of the lower-level model; for The charging power of the energy storage device at any given time; for The discharge power of the energy storage device at any given moment; This refers to the amount of electricity the system purchases from the power grid. This refers to the amount of electricity sold by the system to the power grid.

[0042] C3: The lower-level reward function includes the system's daily operating revenue and the system's safety constraint penalty, where the safety constraint penalty includes the energy storage state of charge exceeding the limit penalty and the grid-connected power exceeding the limit penalty; Specifically, the daily operation optimization of the photovoltaic-storage-DC-flexible system aims to minimize daily maintenance costs while also considering system operational security. Therefore, the reward function is defined as follows: In the formula, This is the reward function for the lower-level model; For the daily operating revenue of the photovoltaic-storage-DC-flexible system; System security constraints and penalties.

[0043] Operation and maintenance costs: In the formula, For the daily operating revenue of the photovoltaic-storage-DC-flexible system; The operation and maintenance cost per unit output power of photovoltaic power; The operation and maintenance cost per unit charging power of energy storage equipment; The operation and maintenance cost per unit discharge power of energy storage equipment; The electricity price that the system purchases from the grid; The electricity price sold by the system to the grid; For marginal electricity price, take and The larger value in; This refers to the power loss in DC transmission lines.

[0044] Safety constraint penalties: In the formula, This is the SOC over-limit penalty coefficient; This is the minimum value for the charged state of energy storage devices; This represents the upper limit of the state of charge of the energy storage device. Penalty factor for exceeding grid-connected power limits; This refers to the power exchanged between the system and the power grid. This represents the maximum allowable switching power of the power grid.

[0045] In this embodiment of the application, the lower-level model in step S200 further includes C21: a mathematical model of the energy storage device, used to update the state of charge of the energy storage device according to the charging and discharging action; The mathematical model defines the self-discharge rate, charging efficiency, and discharging efficiency of the energy storage device, and ensures that its state of charge always operates within the preset safety limits.

[0046] Specifically, the mathematical model of an energy storage device can be shown as follows: In the formula, The self-discharge rate of the energy storage device; For energy storage devices The state of charge at any given moment; The charging efficiency of energy storage devices; The discharge efficiency of energy storage devices; Unit of time.

[0047] The system must meet the power balance condition: .

[0048] It should be noted that the mathematical model of an energy storage device is used to describe the physical behavior of the device during charging and discharging. It serves as a crucial bridge connecting actions and states. Specifically, it connects charging and discharging decisions with changes in the state of charge of the stored energy, defines the dynamic response law of the energy storage device, and is also part of the reinforcement learning environment model. It is mainly used to simulate the actual behavioral changes of the energy storage device after receiving control commands.

[0049] like Figure 2As shown, the left side represents the energy storage configuration model (upper layer), whose reward function consists of annual operating revenue, annualized investment cost, annual operation and maintenance cost, and investment over-limit penalty. Decision actions include the rated capacity and rated charging / discharging power of the energy storage. The right side represents the intraday scheduling model (lower layer), whose reward function consists of operation and maintenance cost, SOC over-limit penalty, and power over-limit penalty. Decision actions include the charging / discharging power of the energy storage and the electricity purchased from the grid. Both models are solved using the improved TD3 algorithm. The upper-layer model optimizes to obtain the optimal energy storage scale scheme, while the lower-layer model obtains the optimal operating strategy under a given configuration. The energy storage capacity and rated power decision results output by the upper-layer model serve as boundary conditions for the lower-layer model. The daily operating revenue obtained by the lower-layer model is then fed back to the energy storage configuration model to correct and optimize long-term investment decisions. Through iterative interaction, the optimal energy storage configuration scheme is finally obtained.

[0050] In this embodiment of the application, step S300 improves the TD3 algorithm based on a multi-scene migration strategy and smoothing constraint technology, including the following steps D1-D2: D1: In the multi-scene migration strategy, scene information is integrated into the action policy network and value evaluation network, enabling the algorithm to share knowledge and adaptively adjust between different scenes; It should be noted that traditional TD3 has high learning efficiency in a single scenario. Its relevant action strategy and Q-function calculation formula are as follows: In the formula, for The action to be performed at any given moment; For the actor policy network in the environment state The output action below; for Environmental state information at any given time; TD target value; for Instant rewards for each moment; Discount factor; There are two target Q-networks. To obtain the smaller state estimate from the two target Q-networks; This is a state estimate; for Environmental state information at any given time; For the actor policy network in the environment state The output action below.

[0051] However, when this strategy is trained in optical storage direct-flexible systems under multiple scenarios, its generalization ability is poor, resulting in low training efficiency and weak cross-scenario learning ability.

[0052] Therefore, the present invention will [describe the scenario]. The action policy learning of the TD3 algorithm is integrated into the Q function to represent different operating scenarios; in this way, the same network can share some knowledge in different scenarios and make adjustments according to scenario differences.

[0053] Specifically, the improved policy and value evaluation network (Q-function) in D1 is as follows: In the formula, For the actor policy network in the environment state With scene The output action below; For the scene The estimated state value under the given conditions.

[0054] It should also be noted that by binding the action policy, value estimation network and scene information, D1 enables the algorithm to reuse existing policy parameters in different scenes, improves cross-scene transfer capability, and avoids the high cost of retraining when changing scenes.

[0055] D2: Introduce action smoothing loss term and parameter smoothing loss term into the loss function of the action policy network to suppress action jumps between adjacent time steps and limit parameter drift; It should be noted that in a photovoltaic-storage-DC-flexible system, the smoothness of the strategy output is crucial for the scheduling of energy storage and photovoltaics. Excessive strategy oscillations will lead to frequent and drastic fluctuations in charging and discharging power, reducing efficiency and shortening energy storage lifespan. However, the traditional TD3 algorithm lacks this limitation, making it prone to this situation. The actor network loss function of the traditional TD3 algorithm is as follows: In the formula, The loss function for the actor network; To get from the experience pool Sampling status Calculate the expected value.

[0056] To address the aforementioned issues, step D2 of this invention adds an "action smoothing constraint" to the loss function of the actor network in the TD3 algorithm. This encourages that the actions output at adjacent time steps should not differ too much, while also limiting parameter drift.

[0057] The specific improvements are as follows: To penalize action jumps between adjacent time steps, an action smoothing loss function is added; to limit parameter drift during a single update, a parameter smoothing loss function is added. In the formula, Loss due to motion smoothing; For smoothing coefficients; This is the operation of squaring the Euclidean distance; For parameter smoothing loss; The regularization coefficient is used. These are the current network parameters of the Actor network; The Actor parameters from the previous iteration; This represents the magnitude of the parameter change.

[0058] The improved TD3 algorithm's actor network loss function is as follows: In the formula, This is the loss function for the improved actor network.

[0059] like Figure 3 As shown, to improve the generalization ability of the TD3 algorithm under multiple scenario conditions while reducing computational costs, this invention introduces scenario information into the action policy learning process and the Q function, enabling the algorithm to represent the characteristics of different operating scenarios. This allows for knowledge sharing across multiple scenarios and adaptive adjustment based on scenario differences. Furthermore, to avoid significant oscillations in the energy storage charging and discharging control quantities due to rapid policy updates during the TD3 algorithm's training process, this invention introduces an action smoothing constraint term into the loss function during Actor network parameter updates. This ensures the continuity of actions in adjacent time steps, avoids excessive fluctuations, effectively suppresses parameter drift, and improves policy stability.

[0060] It should be noted that the above improvements to D2 can solve the problem of the TD3 algorithm updating the strategy too quickly during the training process, thereby making the output results smoother and avoiding large fluctuations in the energy storage charging and discharging control of the optical-storage-direct-flexible system.

[0061] In this embodiment of the application, step S400 uses the improved TD3 algorithm to solve the two-layer Markov decision model. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed, including the following steps E1-E8: E1: Initialize the upper and lower layer policy networks, dual Critic networks, target networks, and experience replay pools; E2: Based on empirical rules and economic constraints, a preliminary energy storage configuration scheme is selected, while noise training is conducted and the scheme is then tailored to the feasible region. E3: Based on the energy storage configuration capacity and power given by the upper layer, the lower layer Actor network is trained and outputs the energy storage charging and discharging and system power purchase and sale actions for each time period of the day, while returning real-time revenue and constraint feedback. E4: The lower layer stores interaction samples and updates the upper layer TD3 dual Critic network; the Actor network is updated under delay conditions, and action and parameter smoothing regularization is added; E5: The daily operational efficiency and constraint results output by the lower layer are fed back to the upper layer, and combined with investment and maintenance costs and investment over-limit penalties, a reward is formed for the upper layer; E6: The upper layer stores interaction samples and updates the lower layer TD3 dual Critic network Actor network. It also introduces action and parameter smoothing regularization. During sampling, it mixes samples from multiple scenarios to complete transfer learning. E7: The updated upper-layer energy storage configuration scheme is passed to the lower layer. The lower layer's intraday operation scheduling strategy is retrained under the new energy storage configuration capacity and the results are returned, forming a cyclical iterative optimization. E8: When the training reaches the set maximum number of iterations, output the optimal energy storage configuration and corresponding operating strategy.

[0062] In summary, this invention, by constructing a set of typical scenarios and a two-layer optimization model, enables adaptive adjustment of energy storage configuration and operation strategies to different lighting and load conditions, avoiding the limitations of a single scenario and thus improving the overall economic benefits of the system over longer periods and in more complex environments. The upper-layer capacity investment decision and the lower-layer intraday operation strategy interact and iterate through reinforcement learning, which can solve the problem of the disconnect between planning and operation. Through transfer learning and smoothing constraint techniques, the oscillation of strategy output is reduced, and the training efficiency and convergence stability of the algorithm in multiple scenarios are also improved.

[0063] Example 3, referring to Figure 4 According to Table 1, this embodiment provides an application scheme for a multi-scenario adaptive energy storage configuration method based on transfer deep reinforcement learning for optical-storage integration, in order to verify the feasibility and effectiveness of the present invention.

[0064] Figure 4 This compares the average training reward values ​​of the upper-layer model under different algorithms.

[0065] Depend on Figure 4It can be seen that the average reward values ​​of the upper-layer model under the traditional TD3 algorithm and the improved TD3 algorithm proposed in this invention converge to approximately 9.94 and 11.20 at 528 and 274 training epochs, respectively. It can be observed that under the improved TD3 algorithm, the photovoltaic-storage-direct-drive-flexible system can obtain a better energy storage configuration scheme faster. This is because the method proposed in this paper introduces a multi-scenario transfer strategy on the basis of the traditional TD3 algorithm, enabling the algorithm to learn energy storage configuration strategies under different scenarios, thereby obtaining a higher final average reward value faster and thus achieving higher system returns. Furthermore, as shown in the figure, the fluctuations after strategy convergence are smaller under the improved TD3 algorithm. This is because the smoothing constraint technique introduced in this invention constrains strategy changes, making the system more stable. This indicates that the system can obtain the optimal configuration scheme more stably under the improved algorithm.

[0066] Table 1: Optimal Energy Storage Configurations for Photovoltaic-Storage-Direct-Way Flexible Systems in Different Scenarios

[0067] As shown in Table 1, the optimal energy storage configuration schemes for the photovoltaic-storage-direct current-flexible system vary significantly across different scenarios: In Scenario 1, with abundant solar resources and a large peak-valley electricity price difference, a configuration of 3.86MWh / 1.36MW can achieve the highest daily and annual benefits; in Scenario 2, although the configuration capacity is larger, the marginal benefits decrease significantly due to the less ideal electricity price difference and solar conditions compared to Scenario 1; in Scenario 3, due to insufficient solar resources and a small electricity price difference, the optimal configuration capacity and power are both low, resulting in the worst benefit level. This demonstrates that energy storage configuration and operational benefits are significantly affected by changes in the scenario. The proposed two-layer optimization model and its transfer deep reinforcement learning solution method can adaptively obtain the optimal configuration and scheduling scheme under different typical scenarios, exhibiting superior performance.

[0068] Example 4 illustrates a schematic scheme for a multi-scenario adaptive energy storage configuration method based on transfer deep reinforcement learning for optical-storage-linear-flexible energy storage. It should be noted that the technical solution of this system for multi-scenario adaptive energy storage configuration based on transfer deep reinforcement learning for optical-storage-linear-flexible energy storage is based on the same concept as the aforementioned method for multi-scenario adaptive energy storage configuration based on transfer deep reinforcement learning for optical-storage-linear-flexible energy storage. Details not described in detail in this embodiment can be found in the description of the aforementioned method for multi-scenario adaptive energy storage configuration based on transfer deep reinforcement learning for optical-storage-linear-flexible energy storage.

[0069] This embodiment also provides a transfer deep reinforcement learning-based multi-scenario adaptive energy storage configuration system for optical-storage integration, including: The acquisition module is used to acquire historical photovoltaic, energy storage, load, electricity price and equipment parameters, and to construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; The two-layer Markov decision model construction module is used to build a two-layer Markov decision model based on a set of typical operating scenarios. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces investment over-limit penalties. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. The optimization module is used to improve the TD3 algorithm based on multi-scenario migration strategies and smoothing constraint techniques; The output module is used to solve the two-layer Markov decision model using the improved TD3 algorithm. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed.

[0070] This embodiment also provides a computer device applicable to a multi-scenario adaptive energy storage configuration method for optical-storage direct-flexible transfer deep reinforcement learning, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the multi-scenario adaptive energy storage configuration method for optical-storage direct-flexible transfer deep reinforcement learning proposed in the above embodiment.

[0071] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a multi-scenario adaptive energy storage configuration method based on transfer deep reinforcement learning for optical-storage direct-flexible energy storage, as proposed in the above embodiment.

[0072] The storage medium proposed in this embodiment and the method for adaptive configuration of energy storage in multiple scenarios based on transfer deep reinforcement learning for optical storage with direct and flexible optical storage proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0073] From the above description of the implementation methods, those skilled in the art will clearly understand that the present invention can be implemented using software and necessary general-purpose hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0074] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A transfer deep reinforcement learning-based adaptive energy storage configuration method for multiple scenarios, characterized in that: include: Obtain historical photovoltaic, energy storage, load, electricity price and equipment parameters, and construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; Based on a set of typical operating scenarios, a two-layer Markov decision model is constructed. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces penalties for exceeding investment limits. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. The TD3 algorithm is improved based on multi-scenario migration strategy and smoothing constraint technique; An improved TD3 algorithm is used to solve a two-layer Markov decision model. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed.

2. The adaptive configuration method for multi-scenario energy storage based on transfer deep reinforcement learning for optical-storage integration, as described in claim 1, is characterized in that... The upper-level model uses the optimal overall economic benefit of the system throughout the year as the reward function, and introduces penalties for exceeding investment limits, including: Construct an upper-level state space, which includes external environment information and system equipment parameters for typical scenarios, specifically: photovoltaic power output characteristics, load characteristics, electricity price levels, and equipment-related parameters for each scenario; Construct an upper-level action space, defining the action space as the rated installed capacity and rated power of energy storage. The upper-level reward function includes the system's annual operating revenue, the equivalent annual investment cost of energy storage equipment, the annual operation and maintenance cost, and penalties for exceeding investment limits.

3. The adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for optical-storage integration, as described in claim 2, is characterized in that... The lower-level model uses the optimal daily operating economic efficiency as the reward function, combined with system safety constraints, including: Construct a lower-level state space and set the state space for daily operation, specifically including the current energy level of energy storage, photovoltaic power generation capacity, load demand and electricity price signals, while distinguishing the operating conditions under different typical scenarios; The lower-level action space is defined as the charging and discharging power of energy storage at each moment and the amount of electricity purchased and sold by the system to the grid at each moment; The lower-level reward function includes the system's daily operating revenue and the system's safety constraint penalty, where the safety constraint penalty includes penalties for exceeding the energy storage state of charge limit and penalties for exceeding the grid-connected power limit.

4. The adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for optical-storage integration, as described in claim 3, is characterized in that... The lower-level model also includes: A mathematical model of the energy storage device is used to update the state of charge of the energy storage device according to the charging and discharging actions; The mathematical model defines the self-discharge rate, charging efficiency, and discharging efficiency of the energy storage device, and ensures that its state of charge always operates within the preset safety limits.

5. The adaptive configuration method for multi-scenario energy storage based on transfer deep reinforcement learning for optical-storage integration, as described in claim 4, is characterized in that... The TD3 algorithm is improved based on multi-scenario migration strategies and smoothing constraint techniques, including: In the multi-scenario migration strategy, scenario information is integrated into the action policy network and value evaluation network, enabling the algorithm to share knowledge and adaptively adjust between different scenarios. Introduce action smoothing loss term and parameter smoothing loss term into the loss function of action policy network to suppress action jumps between adjacent time steps and limit parameter drift.

6. The adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for optical-storage integration, as described in claim 5, is characterized in that... An improved TD3 algorithm is used to solve a two-layer Markov decision model. Through iterative interaction between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that balances investment and operational efficiency is formed, including: Initialize the upper and lower layer policy networks, dual Critic networks, target networks, and experience replay pools; Based on empirical rules and economic constraints, a preliminary energy storage configuration scheme is selected, while noise training is conducted and the scheme is then tailored to the feasible region. Based on the energy storage configuration capacity and power given by the upper layer, the lower layer Actor network is trained and outputs the energy storage charging and discharging and system power purchase and sale actions for each time period of the day, while returning real-time revenue and constraint feedback. The lower layer stores interaction samples and updates the upper-layer TD3 dual Critic network; the Actor network is updated under latency conditions, and action and parameter smoothing regularization is added; The daily operational efficiency and constraint results output from the lower layer are fed back to the upper layer, and combined with investment and maintenance costs and investment over-limit penalties, a reward is formed for the upper layer. The upper layer stores interaction samples and updates the lower layer TD3 dual Critic network Actor network, also introducing action and parameter smoothing regularization; during sampling, samples from multiple scenarios are mixed to complete transfer learning; The updated upper-layer energy storage configuration scheme is passed to the lower layer. The lower layer's intraday operation scheduling strategy is retrained under the new energy storage configuration capacity and the results are returned, forming a cyclical iterative optimization. When the training reaches the set maximum number of iterations, the optimal energy storage configuration and corresponding operating strategy are output.

7. The adaptive configuration method for multi-scenario energy storage using transfer deep reinforcement learning for optical-storage integration as described in claim 6, characterized in that, A set of typical operating scenarios for a photovoltaic-storage-direct-drive-flexible system was constructed based on a clustering algorithm, including: The preprocessed multidimensional data is concatenated into a sample vector; Based on sample vectors, a clustering algorithm is used to cluster historical multidimensional operating data to generate a set of scenarios that can characterize different typical operating conditions of the system.

8. A transfer deep reinforcement learning-based multi-scenario adaptive energy storage configuration system for optical-storage systems, employing the method described in any one of claims 1-7, characterized in that... include: The acquisition module is used to acquire historical photovoltaic, energy storage, load, electricity price and equipment parameters, and to construct a typical operating scenario set for photovoltaic-storage-DC-flexible systems based on clustering algorithms; The two-layer Markov decision model construction module is used to build a two-layer Markov decision model based on a set of typical operating scenarios. The upper-layer model uses the optimal comprehensive economic benefits of the system throughout the year as the reward function and introduces investment over-limit penalties. The lower-layer model uses the optimal economic benefits of daily operation as the reward function and combines system safety constraints to achieve adaptive operation in multiple scenarios. The optimization module is used to improve the TD3 algorithm based on multi-scenario migration strategies and smoothing constraint techniques; The output module is used to solve the two-layer Markov decision model using the improved TD3 algorithm. Through interactive iteration between the upper and lower layers, a multi-scenario adaptive energy storage configuration scheme that takes into account both investment and operational benefits is formed.

9. A computer device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the transfer deep reinforcement learning multi-scenario adaptive energy storage configuration method for optical-storage direct-flexible energy storage as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a processor, implement the steps of the adaptive configuration method for multi-scenario energy storage based on transfer deep reinforcement learning for optical-storage direct-flexible energy storage as described in any one of claims 1 to 7.

Images