Multi-scale electric energy flow control method and system based on deep reinforcement learning

Abstract

The invention discloses a multi-scale electric energy flow control method and system based on deep reinforcement learning, and the method comprises the steps: S1, constructing prediction models of different time scales, and integrating the trained prediction models into a calculation platform; s2, the state space of the energy storage system optimization control algorithm comprises the charge state of the current energy storage system, real-time load data, predicted load and photovoltaic power generation power, real-time electricity price information and time characteristics; s3, the action of the energy storage system optimization control algorithm is the charging and discharging power of the energy storage system, and the action space is a continuous action space; s4, a reward function of the energy storage system optimization control algorithm considers electric power purchase cost, demand electric charge, energy storage loss, green energy consumption and real-time load fluctuation; and S5, training the energy storage system optimization control algorithm in the simulation environment to output an optimal energy storage control strategy. According to the invention, changes of photovoltaic power generation and real-time load are accurately predicted, and scheduling and optimization requirements of a power system are met.

Description

Technical Field

[0001] The present invention relates to a multi-scale electric energy flow control method and system based on deep reinforcement learning. Background Art

[0002] With the growth of global energy demand and the enhancement of environmental protection awareness, renewable energy, especially photovoltaic power generation, is increasingly widely used in power systems. However, photovoltaic power generation is intermittent and volatile, and is affected by weather, seasons and day and night changes, resulting in the instability of its output power. At the same time, the real-time load (electricity demand) in the power system is also random and volatile, and is affected by multiple factors such as user behavior, production activities, seasons and emergencies. This dual uncertainty poses a huge challenge to the dispatch and stable operation of the power system.

[0003] In the prior art, there are mainly the following methods and problems for the prediction of photovoltaic power generation and real-time load and the optimization control of energy storage systems:

[0004] 1. Usually, a single time scale forecasting model is used, such as short-term load forecasting or ultra-short-term photovoltaic output forecasting, using traditional statistical methods or simple neural network models. These models are difficult to meet the forecasting needs at different time scales at the same time, and they do not take into account the volatility of real-time loads, have limited forecasting accuracy, and cannot fully capture the nonlinearity and complexity of data, resulting in insufficient guidance for grid dispatching and control.

[0005] 2. In the optimization control of energy storage systems, the dynamic changes of real-time loads are often ignored or only the average load is used as a reference. This method cannot respond to instantaneous load fluctuations in a timely manner, which may lead to an imbalance in power supply and demand and affect the stability and economy of the power grid.

[0006] 3. The control of energy storage systems is usually based on preset rules or simple optimization algorithms, which fail to fully consider the impact of real-time loads and the dynamic characteristics of energy storage systems. There is a lack of comprehensive optimization of real-time loads and renewable energy output. The control strategy is rigid and difficult to adjust in time, making it impossible to achieve intelligent and efficient management of the power system.

[0007] Therefore, the existing technology has obvious shortcomings in high-precision prediction at multiple time scales, optimal control of energy storage systems, and comprehensive consideration of the impact of real-time loads. An innovative method that can solve these problems at the same time is urgently needed. Summary of the invention

[0008] The first object of the present invention is to provide a multi-scale power flow control method based on deep reinforcement learning, which improves prediction accuracy, improves system flexibility and adaptability, and enhances economic benefits.

[0009] The first object of the present invention is achieved by the following technical measures: a multi-scale power flow control method based on deep reinforcement learning, characterized in that it includes the following steps:

[0010] S1. Preliminarily build multiple prediction models of different time scales, optimize the parameters of each prediction model through big data training, and then integrate the trained prediction models into the computing platform;

[0011] S2. The state space of the energy storage system optimization control algorithm includes the current state of charge of the energy storage system, real-time load data, predicted load and photovoltaic power generation, real-time electricity price information, and time characteristics, where the predicted load and photovoltaic power generation are calculated by the prediction model;

[0012] S3. The action of the energy storage system optimization control algorithm is the charge and discharge power of the energy storage system, and the action space adopts a continuous action space;

[0013] S4. The reward function of the energy storage system optimization control algorithm takes into account the electricity purchase cost, demand electricity charge, energy storage loss, green energy consumption and real-time load fluctuation;

[0014] S5. Train the energy storage system optimization control algorithm in a simulation environment so that it outputs the optimal energy storage control strategy.

[0015] The present invention constructs a high-precision multi-time-scale prediction model for prediction needs at different time scales, which can accurately predict changes in photovoltaic power generation and real-time loads, meet the requirements of power system scheduling and optimization, and improve the prediction accuracy of multiple time scales; the present invention innovatively defines state space, action space and reward function, which can be effectively learned and applied in a complex power system environment, and optimizes the control strategy of the energy storage system. In addition, the present invention realizes the multi-objective dynamic balance of power system operation by considering factors such as electricity cost, energy storage loss, green energy utilization rate and real-time load fluctuation through the design of a comprehensive optimization reward function, improves the overall benefits and system flexibility, realizes the dynamic balance of multiple objectives, and enhances the system's adaptability to real-time load changes.

[0016] In the step S1 of the present invention, the prediction models of multiple different time scales are an ultra-short-term prediction model corresponding to a time scale of 0 to 4 hours, a short-term prediction model corresponding to a time scale of 0 to 72 hours, and a medium- and long-term prediction model corresponding to a time scale of 1 month to 1 year, wherein:

[0017] The ultra-short-term prediction model adopts a long short-term memory network model, and its input features are current and historical load data, photovoltaic power generation data, real-time weather information and time characteristics, and predicts the load and photovoltaic power generation within the next 0-4 hours;

[0018] The short-term prediction model adopts a model combining a convolutional neural network and a long short-term memory network. The input features are load and photovoltaic data with a longer historical span (usually more than a month) and weather forecast information. The local features are first extracted through the convolutional neural network, and then the time dependency is captured through the long short-term memory network to predict the load and photovoltaic power generation power in the next 0-72 hours.

[0019] The medium- and long-term prediction model adopts a recurrent neural network with an attention mechanism. The input features are historical data with a long span, seasonal factors and holiday information, and predicts the load and photovoltaic power generation power in the next 1 month to 1 year.

[0020] In the step S1 of the present invention, a sliding window, data enhancement and cross-validation method are used to perform big data training to optimize model parameters and improve the prediction accuracy of real-time load and photovoltaic power generation.

[0021] In the step S3, the present invention adopts a proximal strategy optimization algorithm to process the continuous action space.

[0022] In the step S3 of the present invention, the energy storage system optimization control algorithm adopts an Actor-Critic network architecture, inputs the state space into the trained energy storage system optimization control algorithm, and outputs the optimal energy storage control strategy.

[0023] The reward function of the present invention is:

[0024] R t =-(C t ·P grid,t +β1·Loss t +β2·ΔL t -γ·P pv,used,t )

[0025] in:

[0026] P grid,t : Power purchased from the grid;

[0027] Loss t : Loss of energy storage system;

[0028] ΔL t : The amplitude of real-time load fluctuation;

[0029] P pv,used,t : The photovoltaic power consumed;

[0030] β1β2γ: weight coefficient.

[0031] The present invention balances the importance of different optimization objectives by adjusting the value of β1β2γ to meet different operational strategy requirements. According to the real-time operation situation, the weight coefficient is dynamically adjusted to achieve flexible strategy control.

[0032] The second object of the present invention is to provide a system using the above-mentioned multi-scale power flow control method based on deep reinforcement learning.

[0033] The second purpose of the present invention is achieved through the following technical measures: A system using the above-mentioned multi-scale electric energy flow control method based on deep reinforcement learning, characterized in that it includes a photovoltaic power generation system, an energy storage system, a load device, a data acquisition device, and a computing platform integrated with a prediction model and an energy storage system optimization control algorithm, the photovoltaic power generation system is connected to the energy storage system, the energy storage system and the photovoltaic power generation system are respectively connected to the load device, the data acquisition device is connected to the computing platform, and the data acquisition device is used to collect historical load data, photovoltaic power generation data, weather data and time characteristic data and pre-process these data, and then transmit the pre-processed data to the computing platform for analysis and processing.

[0034] The data acquisition device of the present invention preprocesses data including the following steps:

[0035] ⑴Data cleaning: use interpolation, forward filling or deletion methods to process missing data points, identify and process outliers in the data;

[0036] ⑵Data normalization, scaling numerical data to a set range;

[0037] ⑶ Feature engineering, converting time information into numerical features that can be used by the model;

[0038] ⑷ Dataset division: divide the data set into training set, validation set and test set.

[0039] Compared with the prior art, the present invention has the following significant effects:

[0040] ⑴ The present invention can improve the prediction accuracy and comprehensively consider the impact of real-time load: by adopting advanced deep learning models, a high-precision prediction model is constructed for prediction requirements of different time scales; the dynamic changes of real-time loads are comprehensively considered, which improves the accuracy of load prediction and provides reliable data support for the optimal scheduling of power systems.

[0041] ⑵The present invention can optimize energy storage control and improve system flexibility and adaptability: the energy storage optimization control algorithm based on deep reinforcement learning can adjust the charging and discharging strategy of the energy storage system in real time; improve the utilization efficiency of the energy storage system, reduce energy storage loss, enhance the system's adaptability to load fluctuations, and ensure the balance of electricity supply and demand.

[0042] ⑶ The present invention can reduce electricity costs and improve economic benefits: through the comprehensive optimization of the reward function design, the electricity purchase cost, demand electricity charges, energy storage losses and load fluctuations are balanced, and the electricity operating costs are significantly reduced; by smoothing the load curve, the maximum load demand is reduced, and the expenditure on demand electricity charges is reduced.

[0043] (4) The present invention realizes multi-objective dynamic balance and enhances system flexibility: through innovative reward function design, multi-objective dynamic balance is achieved to meet the diversified needs of power system operation; the adaptive ability of reinforcement learning algorithm enables the system to optimize the control strategy in real time according to real-time load and environmental changes, thereby enhancing the flexibility and adaptability of the system. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with embodiments to help those skilled in the art better understand the inventive concept of the present invention. However, the protection scope of the claims of the present invention is not limited to the following embodiments. For those skilled in the art, all other embodiments obtained without creative work without departing from the inventive concept of the present invention belong to the protection scope of the present invention.

[0045] The present invention provides a multi-scale power flow control method based on deep reinforcement learning, which specifically includes the following steps:

[0046] S1. Preliminarily build multiple prediction models of different time scales, optimize the parameters of each prediction model through big data training, and then integrate the trained prediction models into the computing platform to provide accurate reference for the optimization control of the energy storage system and the dispatch of the power system, especially the dynamic response to the real-time load;

[0047] The prediction models with different time scales are respectively the ultra-short-term prediction model with a time scale of 0 to 4 hours, the short-term prediction model with a time scale of 0 to 72 hours, and the medium- and long-term prediction model with a time scale of 1 month to 1 year.

[0048] Ultra-short-term prediction model: The long short-term memory network (LSTM) model is used to process the short-term dependencies of time series data. The appropriate number of LSTM layers and neurons is set to prevent overfitting or underfitting. The input features are the current and historical load data L t , Photovoltaic power generation data pv,t , Real-time weather information t and time characteristics, predicting load and PV power generation within the next 0–4 hours;

[0049] h t =LSTM(X t ,h t-1 ) Formula ⑴

[0050] Among them, X t =[L t ,P pv,t ,W t ] is the input feature, h t Hidden state (Hidden state is often used in recurrent neural networks. Hidden state is used to store historical information of sequence data to help the model consider previous context when processing current input). Predict load and photovoltaic output at future moments

[0051] Short-term prediction model: A model combining convolutional neural network and LSTM (CNN-LSTM) is used to input historical load and photovoltaic data {L t-n ,...,L t},{P pv,t-n ,...,P pv,t}、Weather forecast t+1 ,...,W t+m}, etc., extract spatial and temporal features and adapt to the complexity of short-term prediction.

[0052] First, extract the local feature F through the convolutional neural network t =CNN(X t ), and then capture the time dependency h through the long short-term memory network t =LSTM(F t ,h t-1 ), forecasting load and PV power generation for the next 0–72 hours

[0053] Medium- and long-term forecasting model: A recurrent neural network with an attention mechanism is used to input historical load and photovoltaic data {L t-N ,...,L t},{P pv,t-N ,...,P pv,t}, Seasonal factor S t 、Holiday Information Loss t etc., focus on long-term trends and key influencing factors, and improve the accuracy of medium- and long-term forecasts.

[0054] RNN extracts sequence features h t =RNN(X t ,h t-1 ), the attention mechanism calculates the context vector:

[0055] α t =softmax(W a h t +b a )

[0056]

[0057] Combine the context vector for prediction

[0058] Predict the load and photovoltaic power generation power in the next month to one year.

[0059] Through big data training, sliding windows, data enhancement, cross-validation and other methods are used to optimize model parameters and improve the prediction accuracy of real-time load and photovoltaic power generation.

[0060] Loss function: The mean square error (MSE) is used as the loss function:

[0061]

[0062] Optimization method: Use Adam optimizer and learning rate adjustment strategy to prevent overfitting.

[0063] Integrating prediction models of different time scales into a unified computing platform provides an accurate reference for the optimal control of energy storage systems and the dispatch of power systems, especially for the dynamic response to real-time loads.

[0064] S2. The state space of the energy storage system optimization control algorithm includes the current state of charge of the energy storage system, real-time load data, predicted load and photovoltaic power generation, real-time electricity price information, and time characteristics, where the predicted load and photovoltaic power generation are calculated by the prediction model;

[0065] S3. The action of the energy storage system optimization control algorithm is the charge and discharge power of the energy storage system, and the action space adopts a continuous action space;

[0066] S4. The reward function of the energy storage system optimization control algorithm takes into account the electricity purchase cost, demand electricity charge, energy storage loss, green energy consumption and real-time load fluctuation;

[0067] State space, action space and reward function are concepts of reinforcement learning in the prior art.

[0068] S5. Train the energy storage system optimization control algorithm in a simulation environment so that it outputs the optimal energy storage control strategy.

[0069] Design of energy storage optimization control algorithm based on deep reinforcement learning:

[0070] 1. Build a reinforcement learning environment

[0071] (1) Innovative definition of state space:

[0072] Comprehensive information fusion: The state space includes the current state of charge (SoC) of the energy storage system, real-time load data, predicted load and photovoltaic power generation, real-time electricity price information, time characteristics, etc., and various state quantities are combined into a vector as the input of the algorithm.

[0073] State Space

[0074] in:

[0075] SoC t : The current state of charge of the energy storage system.

[0076] L t : Real-time load data.

[0077] The predicted load at the next moment.

[0078] Real-time photovoltaic power generation.

[0079] Predicted photovoltaic power generation at the next moment.

[0080] C t : Real-time electricity price information.

[0081] P min : Time characteristics (such as hours, weeks).

[0082] Real-time load impact: In particular, the dynamic changes of real-time load are incorporated into the state space to reflect the instantaneous supply and demand conditions of the power system and enhance the algorithm's sensitivity and responsiveness to load fluctuations.

[0083] (2) Innovative definition of action space:

[0084] The action is the charge and discharge power of the energy storage system, which uses a continuous action space within the physical limitations of the energy storage device and can flexibly adjust the output of the energy storage system to adapt to real-time load changes. t is the charging and discharging power of the energy storage system, ranging from:

[0085] A t ∈[P min ,P max ]

[0086] Among them, P min and P max is the minimum and maximum charging and discharging power of the energy storage device.

[0087] (3) Design state transfer function: Calculate the state at the next moment (change in SoC) based on the current state and action.

[0088] 2. Innovative design of reward function:

[0089] a) Comprehensive optimization goal: reward function R t At the same time, factors such as electricity purchase cost, demand electricity charge, energy storage loss, green energy consumption and real-time load fluctuations are taken into consideration.

[0090] b) Mathematical expression:

[0091] R t =-(C t ·P grid,t +β1·Loss t +β2·ΔL t -γ·P pv,used,t )

[0092] in:

[0093] P grid,t : Power purchased from the grid.

[0094] Loss t : The loss of energy storage system is usually related to the charging and discharging power.

[0095] ΔL t : The amplitude of real-time load fluctuation.

[0096] P pv,used,t : The photovoltaic power consumed.

[0097] β1β2γ: Weight coefficient, which adjusts the importance of each goal.

[0098] Minimizing the cost of purchasing electricity means reducing the cost of purchasing electricity from the power grid. t ·P grid,t ; Minimize energy storage loss, that is, reduce the loss of energy storage system Loss t ; Load smoothing means reducing load fluctuation ΔL t , smooth load curve; maximize the utilization of green energy, that is, improve the self-consumption of photovoltaic power generation pv,used,t .

[0099] 3. Algorithm implementation (training reinforcement learning agents):

[0100] a) The Proximal Policy Optimization (PPO) algorithm is used to process continuous action space, which has good stability and convergence.

[0101] b) Construct an Actor-Critic network architecture, including a policy network (Actor) and a value network (Critic), input the state space, including real-time load information, and output the optimal control strategy for the energy storage system.

[0102] Actor Network: Input state St , output action A t The probability distribution of .

[0103] Critic network: input state S t , estimate the state value function V(S t ).

[0104] Actor loss function:

[0105]

[0106] in, P pv,t is the advantage function.

[0107] Critic loss function:

[0108]

[0109] Algorithm flow:

[0110] 1. Initialize the policy network and value network parameters.

[0111] 2. Sampling data: interact with the environment and collect state, action, and reward data.

[0112] 3. Calculate the advantage function: Based on the collected data, calculate the advantage function

[0113] 4. Update the network: use the loss function L Actor and L Critic Update network parameters.

[0114] 5. Cycle training: Repeat steps 2-4 until convergence.

[0115] c) Train the intelligent agent in a simulation environment so that it can learn and output the optimal energy storage control strategy and respond to real-time load changes in a timely manner.

[0116] The present invention adopts a comprehensive and optimized reward function design to achieve a dynamic balance of multiple objectives and enhance the system's adaptability to real-time load changes:

[0117] (1) Multi-objective optimization strategy: By adjusting the value of β1β2γ, the importance of different optimization objectives is balanced to meet different operational strategy requirements. According to the real-time operation situation, the weight coefficient is dynamically adjusted to achieve flexible strategy control.

[0118] (2) Real-time response mechanism: The algorithm can adjust the strategy in real time according to environmental changes to adapt to the fluctuations of load and photovoltaic power generation. When a sudden increase or decrease in load is detected, the intelligent agent can quickly adjust the charging and discharging power of the energy storage system to maintain the balance of the power system. Through strategy optimization, the fluctuation amplitude of the load curve is reduced, the peak-to-valley difference is reduced, and the stability of the power system is improved.

[0119] The present invention provides a system using the multi-scale power flow control method based on deep reinforcement learning, comprising:

[0120] Photovoltaic power generation system, calculate the required photovoltaic power generation capacity according to the actual load demand, and select suitable solar panels and inverters.

[0121] The energy storage system selects energy storage batteries (such as lithium-ion batteries) and bidirectional converters (PCS) of appropriate capacity and power according to load characteristics and photovoltaic power generation capacity, and calculates the required energy storage capacity by considering peak and valley load differences and backup requirements.

[0122] Load Equipment, lists the equipment and systems that need to be powered, including their power requirements and operating hours.

[0123] The data acquisition device is equipped with current, voltage, power and other sensors to monitor the status of photovoltaic, energy storage and load in real time.

[0124] A computing platform that integrates prediction models and energy storage system optimization control algorithms, a computer or server with performance that meets the requirements, an operating system and necessary software libraries installed, including frameworks for deep learning and reinforcement learning.

[0125] The photovoltaic power generation system is connected to the energy storage system, the energy storage system and the photovoltaic power generation system are connected to the load equipment respectively, and the data acquisition device is connected to the computing platform. The data acquisition device is used to collect historical load data, photovoltaic power generation data, weather data and time characteristic data and pre-process these data, and then transmit the pre-processed data to the computing platform for analysis and processing.

[0126] The data acquisition device collects the following data:

[0127] Historical load data: Obtain the power consumption record of the load in the past year through the data provided by the load monitoring system.

[0128] Photovoltaic power generation data: Get real-time and historical power generation and load data from monitoring equipment in the photovoltaic system.

[0129] Weather data: Obtain meteorological parameters that affect photovoltaic power generation, including solar radiation intensity, temperature, humidity, wind speed, etc., from the meteorological bureau data and third-party meteorological service API.

[0130] Time characteristic data: time information such as date, time, day of the week, holidays, etc., used to capture periodic changes in load and power generation.

[0131] The data acquisition device preprocesses the data including the following steps:

[0132] ⑴Data cleaning: use interpolation, forward filling or deletion methods to process missing data points, identify and process outliers in the data, and ensure data quality;

[0133] ⑵Data normalization: scaling numerical data to a set range (such as 0 to 1) to eliminate the impact of dimensional differences on the model;

[0134] ⑶ Feature engineering, converting time information into numerical features that can be used by the model, such as using sine and cosine functions to represent periodicity;

[0135] (4) Dataset division: divide the dataset into a training set (accounting for 70% of the total data), a validation set (accounting for 15% of the total data) and a test set (accounting for 15% of the total data).

[0136] The system integration and deployment of the present invention are as follows:

[0137] 1. Deploy the prediction model:

[0138] ⑴Model integration: Load the trained prediction model into the system, use real-time data to input the model, and obtain the load and photovoltaic power generation prediction results at future times.

[0139] ⑵Interface design: Design the interface for data collection and transmission to achieve connection with sensors and databases. Design the input and output interfaces of the model to ensure that the model can be called by other modules.

[0140] 2. Deployment of optimized control strategies

[0141] ⑴Integrated reinforcement learning agent: Deploy the trained agent into the control system and generate control instructions for the energy storage system based on the real-time status.

[0142] ⑵ Safety mechanism: Before deployment, verify the strategy of the agent to ensure its safety and reliability. Design anomaly detection and processing mechanisms to prevent system risks caused by strategy failure.

[0143] 3. System monitoring and maintenance

[0144] ⑴ Real-time monitoring: Real-time monitoring of key system parameters, such as SoC, power purchase power from the power grid, load demand, etc. Set thresholds and trigger alarms when parameters exceed the normal range.

[0145] ⑵ Regular maintenance: Regularly update the data set of the prediction model and retrain to maintain the accuracy of the model. Regularly check the operating status of the hardware equipment and perform necessary maintenance and care.

Claims

1. A multi-scale power flow control method based on deep reinforcement learning, characterized in that The following steps are involved: S1. Preliminarily build multiple prediction models of different time scales, optimize the parameters of each prediction model through big data training, and then integrate the trained prediction models into the computing platform; S2. The state space of the energy storage system optimization control algorithm includes the current state of charge of the energy storage system, real-time load data, predicted load and photovoltaic power generation, real-time electricity price information, and time characteristics, where the predicted load and photovoltaic power generation are calculated by the prediction model; S3. The action of the energy storage system optimization control algorithm is the charge and discharge power of the energy storage system, and the action space adopts a continuous action space; S4. The reward function of the energy storage system optimization control algorithm takes into account the electricity purchase cost, demand electricity charge, energy storage loss, green energy consumption and real-time load fluctuation; S5. Train the energy storage system optimization control algorithm in a simulation environment so that it can output the optimal energy storage control strategy.

2. The multi-scale power flow control method based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the prediction models of multiple different time scales are respectively an ultra-short-term prediction model corresponding to a time scale of 0 to 4 hours, a short-term prediction model corresponding to a time scale of 0 to 72 hours, and a medium- and long-term prediction model corresponding to a time scale of 1 month to 1 year, wherein: The ultra-short-term prediction model adopts a long short-term memory network model, and its input features are current and historical load data, photovoltaic power generation data, real-time weather information and time characteristics, and predicts the load and photovoltaic power generation within the next 0-4 hours; The short-term prediction model adopts a model combining a convolutional neural network and a long short-term memory network. The input features are load and photovoltaic data with a longer historical span and weather forecast information. The local features are first extracted through the convolutional neural network, and then the time dependency is captured through the long short-term memory network to predict the load and photovoltaic power generation power in the next 0-72 hours. The medium- and long-term prediction model adopts a recurrent neural network with an attention mechanism. The input features are historical data with a long span, seasonal factors and holiday information, and predicts the load and photovoltaic power generation power in the next 1 month to 1 year.

3. The multi-scale power flow control method based on deep reinforcement learning according to claim 2 is characterized in that: In step S1, a sliding window, data enhancement and cross-validation method are used to perform big data training to optimize model parameters.

4. The multi-scale power flow control method based on deep reinforcement learning according to claim 3 is characterized in that: In step S3, a proximal strategy optimization algorithm is used to process the continuous action space.

5. The multi-scale power flow control method based on deep reinforcement learning according to claim 4 is characterized in that: The energy storage system optimization control algorithm adopts an Actor-Critic network architecture, inputs the state space into the trained energy storage system optimization control algorithm, and outputs the optimal energy storage control strategy.

6. The multi-scale power flow control method based on deep reinforcement learning according to claim 5 is characterized in that: The reward function is: R t =-(C t ·P grid,t +β1·Loss t +β2·ΔL t -γ·P pv,used,t ) in: P grid,t : Power purchased from the grid; Loss t : Loss of energy storage system; ΔL t : The amplitude of real-time load fluctuation; P pv,used,t : The photovoltaic power consumed; β1β2γ: weight coefficient.

7. A system using the multi-scale power flow control method based on deep reinforcement learning according to any one of claims 1 to 6, characterized in that: It includes a photovoltaic power generation system, an energy storage system, a load device, a data acquisition device, and a computing platform integrated with a prediction model and an energy storage system optimization control algorithm. The photovoltaic power generation system is connected to the energy storage system, and the energy storage system and the photovoltaic power generation system are respectively connected to the load device. The data acquisition device is connected to the computing platform. The data acquisition device is used to collect historical load data, photovoltaic power generation data, weather data and time characteristic data and pre-process these data, and then transmit the pre-processed data to the computing platform for analysis and processing.

8. The system according to claim 7, characterized in that: The data acquisition device preprocesses the data including the following steps: ⑴Data cleaning: use interpolation, forward filling or deletion methods to process missing data points, identify and process outliers in the data; ⑵Data normalization, scaling numerical data to a set range; ⑶ Feature engineering, converting time information into numerical features that can be used by the model; ⑷ Dataset division: divide the data set into training set, validation set and test set.

Images