Photovoltaic energy storage collaborative optimization scheduling prediction method and system based on reinforcement learning

Abstract

The invention provides a photovoltaic energy storage collaborative optimization scheduling prediction method and system based on reinforcement learning, and relates to the technical field of photovoltaic energy storage, and the method comprises the steps: constructing a multi-target evaluation system containing peak-valley difference minimization and demand response income, and combining a hierarchical strategy selection mechanism of adaptive weight to determine an optimal scheduling strategy, and parameter optimization and rolling prediction are carried out by using the dual Q network action value evaluation model, so that optimal scheduling of the energy storage system is realized. The economic benefit of the photovoltaic energy storage system can be effectively improved, the peak-valley difference of the power grid is reduced, and the stability and reliability of system operation are improved.

Description

Technical Field

[0001] The present invention relates to photovoltaic energy storage technology, and in particular to a photovoltaic energy storage collaborative optimization scheduling prediction method and system based on reinforcement learning. Background Art

[0002] As the proportion of renewable energy in the power system continues to increase, the intermittent and fluctuating nature of photovoltaic power generation poses significant challenges to the safe and stable operation of the power grid. Energy storage systems, as a key regulatory tool, can effectively smooth out fluctuations in photovoltaic output, improving system reliability and economic efficiency. Currently, the coordinated optimization and scheduling of photovoltaic and energy storage systems primarily utilizes traditional optimization algorithms and forecasting methods, building mathematical models to achieve optimal system operation. While these methods have achieved some success in practical application, several technical bottlenecks remain that require overcoming.

[0003] Conventional PV-energy storage coordinated scheduling methods often employ fixed optimization objective weights, failing to adapt to dynamic changes in the system's operating status and external environment. This results in optimization results that fail to meet actual operational requirements. Furthermore, existing forecasting methods primarily rely on statistical analysis of historical data, insufficiently considering the system's dynamic characteristics and uncertainties, resulting in poor forecasting accuracy and adaptability. Furthermore, traditional methods often employ simple linear weighting approaches when addressing multi-objective optimization problems, making it difficult to effectively balance multiple objectives, such as system economics and reliability, and thus failing to achieve true global optimization.

[0004] These technical deficiencies severely restrict the collaborative optimization of PV-energy storage systems. To overcome these issues, it is necessary to develop new intelligent optimization and scheduling methods that fully utilize the advantages of artificial intelligence technologies such as reinforcement learning to achieve adaptive optimization and accurate prediction of the system, thereby improving the operational efficiency and economic benefits of PV-energy storage systems. Summary of the Invention

[0005] The embodiments of the present invention provide a photovoltaic energy storage collaborative optimization scheduling prediction method and system based on reinforcement learning, which can solve the problems in the existing technology.

[0006] According to a first aspect of the embodiments of the present invention, Provides a reinforcement learning-based PV and energy storage collaborative optimization scheduling prediction method, including: A multi-objective evaluation system, encompassing peak-to-valley difference minimization and demand response benefits, is constructed based on grid exchange power data, load regulation, and real-time electricity price data corresponding to photovoltaic power stations. This system determines the optimal dispatch strategy through a hierarchical strategy selection mechanism with adaptive weights. This optimal dispatch strategy is then smoothed to generate charge and discharge instructions for the energy storage system. The adaptive weights are then dynamically adjusted based on real-time operational data feedback from the energy storage system executing these instructions. The system state vector adjusted based on the adaptive weight is input into the dual Q network action value evaluation model to calculate the action value. The optimal action is selected based on the action value through the ε-greedy strategy. The optimal action is executed to obtain the system reward and the next state. The system state vector, optimal action, system reward and next state are sequentially composed into a state transition sample and stored in the experience replay pool. Training data is sampled from the experience replay pool and used to optimize the parameters of the dual-Q network action value evaluation model. The dual-Q network action value evaluation model with optimized parameters is combined with the hierarchical reinforcement learning framework. Based on the combined dual-Q network action value evaluation model, a rolling forecast for future time periods is made and an optimized scheduling plan for the energy storage system is generated.

[0007] A multi-objective evaluation system, encompassing peak-valley difference minimization and demand response benefits, is constructed based on grid exchange power data, load regulation, and real-time electricity price data corresponding to photovoltaic power plants. This system determines the optimal dispatch strategy through a hierarchical strategy selection mechanism with adaptive weights. This optimal dispatch strategy is then smoothed to generate charge and discharge instructions for the energy storage system. The adaptive weights are dynamically adjusted based on real-time operational feedback from the energy storage system executing the charge and discharge instructions. These weights include: Based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station, a multi-objective evaluation system is established, including the peak-to-valley difference minimization objective and the demand response benefit objective. The peak-to-valley difference minimization objective is calculated by the difference between the maximum and minimum values of the grid exchange power data, and the demand response benefit objective is calculated by multiplying the real-time electricity price data by the load regulation amount. The multi-objective evaluation system is input into the adaptive weighted hierarchical strategy selection mechanism. The adaptive weighted hierarchical strategy selection mechanism calculates the weight coefficient based on the performance index of each objective through a normalized exponential function. The performance index is calculated as the deviation between the actual value of the objective and the expected value. The comprehensive scores of different scheduling strategies are calculated based on the weight coefficients and the multi-objective evaluation system. The comprehensive score is obtained by weighting the normalized value of each objective and the corresponding weight coefficient. The strategy with the highest comprehensive score is selected as the optimal scheduling strategy. The optimal scheduling strategy is smoothed using an exponentially weighted sliding average method. The energy storage charging and discharging power in adjacent time periods is weighted averaged using an adaptively adjusted smoothing coefficient to obtain the optimized charging and discharging instructions. Real-time feedback data of the energy storage system operation is obtained, and the control deviation is calculated based on the real-time feedback data. The distribution coefficient of the adaptive weight is updated through the gradient of the control deviation. The strategy selection and smoothing processing are re-executed according to the updated adaptive weight to achieve dynamic optimization of the scheduling strategy.

[0008] Based on the grid exchange power data, load regulation and real-time electricity price data corresponding to the photovoltaic power station, a multi-objective evaluation system is constructed, including the peak-valley difference minimization target and the demand response benefit target. The peak-valley difference minimization target is calculated by the difference between the maximum and minimum values of the grid exchange power data, and the demand response benefit target is calculated by multiplying the real-time electricity price data and the load regulation amount. The maximum and minimum values of the power exchange data of the power grid are obtained by traversing the power exchange data of the power grid in the scheduling period, and the peak-to-valley difference minimization target is obtained by subtracting the minimum value of the power exchange data from the maximum value of the power exchange data of the power grid; The benchmark load is calculated based on the historical load operation data for the same period, and the load adjustment amount is calculated. The load adjustment amount is the difference between the actual load and the benchmark load. The load adjustment amount is multiplied by the real-time electricity price data and accumulated within the scheduling period to obtain the demand response profit target.

[0009] The system state vector adjusted based on the adaptive weight is input into the dual Q network action value evaluation model to calculate the action value. The optimal action is selected based on the action value through the ε-greedy strategy. The optimal action is executed to obtain the system reward and the next state. The system state vector, optimal action, system reward and next state are sequentially composed into a state transition sample and stored in the experience replay pool, including: The system state vector adjusted based on the adaptive weight coefficient is input into the first Q network and the second Q network to obtain evaluation results of the first action value and the second action value, respectively; a random number in the range of zero to one is generated, and when the random number is less than a preset exploration probability, an execution action is randomly sampled from the action space based on the ratio of the preset exploration probability to the size of the action space, where the size of the action space represents the capacity of the action space; when the random number is greater than or equal to the preset exploration probability, the action corresponding to the minimum action value is selected as the optimal action by comparing the first action value and the second action value; Execute the optimal action, calculate the grid interaction benefit, energy storage state maintenance reward, and power deviation penalty, sum the product of the grid interaction benefit and the first weight coefficient, the product of the energy storage state maintenance reward and the second weight coefficient, and the product of the power deviation penalty and the third weight coefficient to obtain the system reward value, and at the same time obtain the next state to which the system transfers after executing the optimal action; The system state vector, optimal action, system reward value and next state vector are constructed as state transition samples. The temporal difference error, Bellman error and novelty measure are calculated based on the state transition samples. The sample importance score is calculated according to the temporal difference error, Bellman error and novelty measure. The state transition samples are stored in different priority storage pools according to the sample importance score, and the state transition samples are migrated between different priority storage pools based on the priority update value. The life cycle of the state transition sample is determined according to the comprehensive weight calculated according to the temporal weight and the sample importance score, and the sample is stored in the experience replay pool.

[0010] The system state vector, optimal action, system reward value and next state vector are constructed as state transition samples. The temporal difference error, Bellman error and novelty metric are calculated based on the state transition samples. The sample importance score is calculated based on the temporal difference error, Bellman error and novelty metric, including: The system state vector, optimal action, system reward value and next state vector are sequentially concatenated to construct a state transition sample; the time series difference error of the state transition sample is calculated, and the time series difference error is the absolute value of the difference between the sum of the system reward value and the discounted value of the next state value function minus the current state value function; the Bellman error of the state transition sample is calculated, and the Bellman error is the absolute value of the difference between the current state action value function and the sum of the discounted value of the system reward value and the next state maximum action value function; Calculate the novelty metric of the state transition sample, which is the minimum Euclidean distance between the state transition sample and existing samples in the experience pool; multiply the time series difference error by the first weight coefficient to obtain a first weight value, which is used to represent the importance of the time series characteristics; multiply the Bellman error by the second weight coefficient to obtain a second weight value, which is used to represent the accuracy requirement of the value estimate; Multiply the novelty measure by the third weight coefficient to obtain a third weighted value, where the third weight coefficient is used to characterize the attention paid to the novelty of the sample; the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is one; and add the first weighted value, the second weighted value and the third weighted value to obtain the sample importance score.

[0011] Training data is sampled from the experience replay pool and used to optimize the parameters of the dual-Q network action value evaluation model. The optimized dual-Q network action value evaluation model is then combined with the hierarchical reinforcement learning framework. Based on this combined dual-Q network action value evaluation model, a rolling forecast for future time periods is performed to generate an optimized scheduling plan for the energy storage system, including: Constructing an action value evaluation model, which includes a first Q network and a second Q network, constructing a state space and an action space, inputting the state in the state space into the first Q network and the second Q network, and obtaining the first Q value and the second Q value of each action in the current state; Based on the first Q value and the second Q value, an ε greedy strategy is used to select actions. An action is randomly selected with an ε probability. The action corresponding to the maximum value of the first Q value and the second Q value is selected as the optimal action with a probability of 1-ε. The optimal action is executed to obtain an immediate reward and the next state. Randomly sample batches of state transition samples from the experience replay pool, and calculate the target Q value based on the batch state transition samples. The target Q value is calculated by summing the product of the immediate reward, the discount factor, and the minimum Q value of the next state. The first Q network and the second Q network are trained using the target Q value, and the temporal difference error of the state transition samples is used as the loss function to update the parameters of the first Q network and the second Q network; The first Q network and the second Q network after parameter update are constructed into a hierarchical reinforcement learning framework, which includes a first-layer policy network and a second-layer execution network. The current state in the state space is input into the first-layer policy network to generate sub-goals, and constraints are constructed based on the sub-goals and the current state. The constraints are input into the second-layer execution network. The second-layer execution network and constraints are used to make rolling predictions for future time periods. Based on the rolling prediction results, a charging and discharging power timing optimization plan for the energy storage system is generated, and the charging and discharging power timing optimization plan for the energy storage system is used as a scheduling instruction for the energy storage system.

[0012] Based on the first Q value and the second Q value, an ε greedy strategy is used to select an action. An action is randomly selected with an ε probability, and the action corresponding to the maximum value of the first Q value and the second Q value is selected as the optimal action with a probability of 1-ε. Executing the optimal action to obtain an immediate reward and the next state includes: Based on the first Q value and the second Q value, the ε greedy strategy is used to select actions, generating a uniform random number between 0 and 1. When the uniform random number is less than the preset exploration probability ε, the execution action is randomly sampled from the action space. When the execution action is greater than or equal to the preset exploration probability ε, the largest Q value between the first Q value and the second Q value is selected through comparison operation, and the action index corresponding to the largest Q value is mapped to the action space to obtain the optimal action; Execute the optimal action to interact with the environment. Input the optimal action into the environment through the environment interaction interface and perform simulation. Calculate the immediate reward value according to the preset reward function. The reward function comprehensively considers multiple evaluation indicators of action execution, including goal achievement, resource consumption, and constraint satisfaction. At the same time, obtain the observation value of the environment after executing the optimal action as the next state.

[0013] A second aspect of an embodiment of the present invention provides a photovoltaic energy storage collaborative optimization scheduling prediction system based on reinforcement learning, including: The first unit is used to construct a multi-objective evaluation system that includes peak-to-valley difference minimization and demand response benefits based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station. The multi-objective evaluation system determines the optimal scheduling strategy through a hierarchical strategy selection mechanism with adaptive weights. The optimal scheduling strategy is smoothed to obtain charging and discharging instructions for the energy storage system. The adaptive weights are dynamically adjusted based on the real-time operating data feedback of the energy storage system executing the charging and discharging instructions. The second unit is used to input the system state vector adjusted based on the adaptive weight into the dual Q network action value evaluation model to calculate the action value, select the optimal action based on the action value through the ε-greedy strategy, execute the optimal action to obtain the system reward and the next state, and sequentially form the system state vector, optimal action, system reward and next state into a state transition sample and store it in the experience replay pool; The third unit is used to sample training data from the experience replay pool, use the training data to optimize the parameters of the dual-Q network action value evaluation model, combine the dual-Q network action value evaluation model with the hierarchical reinforcement learning framework, and perform rolling predictions for future time periods based on the combined dual-Q network action value evaluation model to generate an optimized scheduling plan for the energy storage system.

[0014] The third aspect of the embodiment of the present invention An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0016] The beneficial effects of this application are as follows: The present invention realizes the dynamic optimization adjustment of the charging and discharging instructions of the energy storage system by constructing a hierarchical strategy selection mechanism based on adaptive weights, effectively improves the coordinated operation efficiency of the photovoltaic energy storage system, significantly reduces the peak-to-valley difference of the system, and enhances the demand response capability.

[0017] The present invention adopts a dual Q network action value evaluation model for action value calculation and optimal action selection, and combines it with an experience replay pool for model training and parameter optimization, thereby improving the convergence and stability of the reinforcement learning algorithm and ensuring the reliability and robustness of the energy storage system scheduling strategy.

[0018] The present invention combines the optimized dual Q network action value assessment model with a hierarchical reinforcement learning framework to achieve rolling predictions for future time periods and the generation of optimized scheduling plans for energy storage systems. It has strong practicality and promotion value and can provide effective technical support for the intelligent operation and management of photovoltaic energy storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the process of the photovoltaic energy storage collaborative optimization scheduling prediction method based on reinforcement learning in an embodiment of the present invention; Figure 2 Schematic diagram comparing peak-to-valley difference reduction and demand response benefits of different scheduling strategies according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the photovoltaic energy storage collaborative optimization system - dual-Q network intelligent scheduling according to an embodiment of the present invention; Figure 4 Schematic diagram showing performance comparison of different technical solutions according to the embodiment of the present invention; Figure 5 Schematic diagram comparing the convergence performance of different algorithms in embodiments of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 FIG. 1 is a flow chart of a photovoltaic energy storage collaborative optimization scheduling prediction method based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes: A multi-objective evaluation system, encompassing peak-to-valley difference minimization and demand response benefits, is constructed based on grid exchange power data, load regulation, and real-time electricity price data corresponding to photovoltaic power stations. This system determines the optimal dispatch strategy through a hierarchical strategy selection mechanism with adaptive weights. This optimal dispatch strategy is then smoothed to generate charge and discharge instructions for the energy storage system. The adaptive weights are then dynamically adjusted based on real-time operational data feedback from the energy storage system executing these instructions. The system state vector adjusted based on the adaptive weight is input into the dual Q network action value evaluation model to calculate the action value. The optimal action is selected based on the action value through the ε-greedy strategy. The optimal action is executed to obtain the system reward and the next state. The system state vector, optimal action, system reward and next state are sequentially composed into a state transition sample and stored in the experience replay pool. Training data is sampled from the experience replay pool and used to optimize the parameters of the dual-Q network action value evaluation model. The dual-Q network action value evaluation model with optimized parameters is combined with the hierarchical reinforcement learning framework. Based on the combined dual-Q network action value evaluation model, a rolling forecast for future time periods is made and an optimized scheduling plan for the energy storage system is generated.

[0023] In an optional embodiment, a multi-objective evaluation system including peak-valley difference minimization and demand response benefits is constructed based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station. The multi-objective evaluation system determines the optimal scheduling strategy through a hierarchical strategy selection mechanism with adaptive weights. The optimal scheduling strategy is smoothed to obtain charging and discharging instructions for the energy storage system. The adaptive weights are dynamically adjusted based on real-time operational feedback from the energy storage system executing the charging and discharging instructions, including: Based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station, a multi-objective evaluation system is established, including the peak-to-valley difference minimization objective and the demand response benefit objective. The peak-to-valley difference minimization objective is calculated by the difference between the maximum and minimum values of the grid exchange power data, and the demand response benefit objective is calculated by multiplying the real-time electricity price data by the load regulation amount. The multi-objective evaluation system is input into the adaptive weighted hierarchical strategy selection mechanism. The adaptive weighted hierarchical strategy selection mechanism calculates the weight coefficient based on the performance index of each objective through a normalized exponential function. The performance index is calculated as the deviation between the actual value of the objective and the expected value. The comprehensive scores of different scheduling strategies are calculated based on the weight coefficients and the multi-objective evaluation system. The comprehensive score is obtained by weighting the normalized value of each objective and the corresponding weight coefficient. The strategy with the highest comprehensive score is selected as the optimal scheduling strategy. The optimal scheduling strategy is smoothed using an exponentially weighted sliding average method. The energy storage charging and discharging power in adjacent time periods is weighted averaged using an adaptively adjusted smoothing coefficient to obtain the optimized charging and discharging instructions. Real-time feedback data of the energy storage system operation is obtained, and the control deviation is calculated based on the real-time feedback data. The distribution coefficient of the adaptive weight is updated through the gradient of the control deviation. The strategy selection and smoothing processing are re-executed according to the updated adaptive weight to achieve dynamic optimization of the scheduling strategy.

[0024] Grid exchange power data is obtained from the PV power station's management system, representing the power exchanged between the PV station and the public grid. For example, the grid exchange power data (in kW) for a particular PV station over a given day is: [-120, -150, -180, -160, -130, -50, 100, 250, 380, 450, 480, 500, 490, 460, 400, 320, 200, 100, -50, -100, -140, -160, -170, -150], reflecting the characteristic of power consumption at night and power transmission during the day.

[0025] Obtain the load regulation data for the energy storage system, representing the load adjustment achieved through charging and discharging. For example, if the energy storage system's regulation capacity is capped at ±200 kW, the actual regulation data would be: [150, 180, 200, 180, 150, 50, -50, -100, -150, -200, -200, -180, -150, -150, -100, -50, 0, 50, 100, 150, 180,200, 180, 150], reflecting a late-night charging and daily-discharging pattern.

[0026] Get real-time electricity price data, such as the daily data (yuan / kWh) for a certain region: [0.3, 0.28, 0.25, 0.28, 0.32, 0.45, 0.6, 0.8, 1.0, 1.2, 1.3, 1.25, 1.2, 1.1, 1.0, 0.9, 0.8, 0.75, 0.7, 0.6, 0.5, 0.45, 0.4, 0.35], showing the changes from the early morning low to the midday peak and then the evening decline.

[0027] A multi-objective evaluation system was constructed, encompassing both peak-to-valley differential minimization and demand response revenue objectives. This differential minimization was achieved by processing grid exchange power data. The energy storage regulation capability was superimposed on the original grid exchange power. For example, if the exchange power in the first hour was -120kW and the energy storage charged 150kW, the adjusted power would be -270kW. The difference between the adjusted maximum and minimum values across all time periods was calculated as the peak-to-valley differential. For example, a peak-to-valley differential of 680kW before adjustment could be reduced to 420kW through appropriate regulation, a reduction of approximately 38%.

[0028] Demand response benefits are calculated by multiplying the electricity price by the amount of regulation. For example, if the electricity price for the first hour is 0.3 yuan / kWh and the energy storage charges at 150kW, the cost is 45 yuan. The costs or benefits are calculated for each period over a 24-hour period and aggregated to obtain the net benefit. For example, under a certain strategy, charging at a low electricity price costs 500 yuan, while discharging at a high electricity price generates 900 yuan, resulting in a net benefit of 400 yuan.

[0029] The multi-objective evaluation results are fed into an adaptive weighted hierarchical strategy selection mechanism. This mechanism calculates weight coefficients based on each objective performance indicator. Performance indicators are determined by the deviation between the actual value and the expected value. For example, the expected peak-to-valley difference target value is 300kW, the actual value is 420kW, and the deviation is 120kW; the expected profit target value is 500 yuan, the actual value is 400 yuan, and the deviation is 100 yuan.

[0030] Normalize the deviations to make indicators of different dimensions comparable. Normalization is done by dividing the deviation by the maximum possible deviation. For example, the normalized deviation for peak-to-valley difference is 0.176, and the normalized deviation for return is 0.2. Weight coefficients are calculated using a normalized exponential function, with targets with larger deviations receiving greater weights, such as a weight of 0.55 for peak-to-valley difference and 0.45 for return.

[0031] Calculate the overall scores of different strategies by adding the normalized values of each objective and the weight. For example, Strategy A has a normalized peak-to-valley difference of 0.82, a normalized return of 0.75, and an overall score of 0.79; Strategy B scores 0.76, and Strategy C scores 0.81. Select Strategy C, which has the highest score, as the optimal strategy.

[0032] The optimal strategy is smoothed using an exponentially weighted moving average method to avoid frequent energy storage state switching. This method weights adjacent power periods using an adaptively adjusted smoothing coefficient. For example, the original power of three consecutive periods [180kW, -50kW, -120kW] becomes [180kW, 111kW, -49.7kW] after smoothing, creating a smoother curve and reducing energy storage stress.

[0033] The system obtains energy storage operation feedback data and calculates the control deviation (the difference between actual power and command power). The weight coefficient is updated based on the deviation gradient, with larger deviations leading to larger adjustments. If the deviations over multiple time periods are concentrated in the peak-to-valley difference, the weight of the peak-to-valley difference is reduced, while the weight of the return is increased. Strategy selection and smoothing are re-executed based on the new weights, forming a dynamic optimization closed loop.

[0034] For example, if the initial peak-valley difference weight is 0.6 and the revenue weight is 0.4, if the energy storage system finds it difficult to execute the peak-valley difference adjustment instructions, the system automatically adjusts them to 0.5 for each weight. The instructions generated under these new weights prioritize economic efficiency and improve their feasibility. Through operation, the system has found a balance between technical and economic indicators, achieving optimal coordinated scheduling.

[0035] Technical Benefits: A multi-objective evaluation system and adaptive weighting mechanism are established to effectively achieve coordinated optimization of photovoltaics and energy storage. Minimizing peak-valley variations smooths the grid's exchange power curve, alleviating peak-shaving pressure and improving system stability. Field measurements show a 35%-40% reduction in peak-valley variations. Smoothing reduces frequent energy storage switching and extends battery life by approximately 20%. Economic Benefits: Economic returns are improved by optimizing charging and discharging behavior during different electricity price periods. Field measurements show a 25%-30% increase in the annualized rate of return for participating in demand response. The adaptive weighting mechanism dynamically adjusts strategies based on market and system conditions to maximize economic benefits. Reduced peak-valley variations also reduce grid investment and operating costs. Management Benefits: Automated and intelligent dispatching is achieved, reducing manual intervention and improving operational efficiency. The system adjusts strategies in real time based on feedback, rapidly responding to grid demand and market changes. The adaptive mechanism adapts to different PV plant types and weather conditions, reducing manual parameter adjustments. Field measurements show a reduction in management complexity of approximately 40% and an improvement in operational reliability of over 15%.

[0036] In an optional embodiment, a multi-objective evaluation system including a peak-valley difference minimization objective and a demand response benefit objective is constructed based on the grid exchange power data, load regulation amount, and real-time electricity price data corresponding to the photovoltaic power station. The peak-valley difference minimization objective is calculated by the difference between the maximum and minimum values of the grid exchange power data, and the demand response benefit objective is calculated by multiplying the real-time electricity price data and the load regulation amount. The system includes: The maximum and minimum values of the power exchange data of the power grid are obtained by traversing the power exchange data of the power grid in the scheduling period, and the peak-to-valley difference minimization target is obtained by subtracting the minimum value of the power exchange data from the maximum value of the power exchange data of the power grid; The benchmark load is calculated based on the historical load operation data for the same period, and the load adjustment amount is calculated. The load adjustment amount is the difference between the actual load and the benchmark load. The load adjustment amount is multiplied by the real-time electricity price data and accumulated within the scheduling period to obtain the demand response profit target.

[0037] Collect and process grid exchange power data from photovoltaic power plants. This data is collected in real time through smart meters or energy management systems. The sampling interval is set to 15 minutes, and a total of 96 data points are collected over a 24-hour scheduling cycle. For example, the grid exchange power data for a photovoltaic power plant on a typical summer day has a minimum of -50 kilowatts at 3:00 a.m. and a maximum of 200 kilowatts at 2:00 p.m.

[0038] Calculate the objective of minimizing the peak-to-valley difference in grid exchange power. Using a data traversal and comparison method, determine the maximum and minimum values among the 96 sampling points. Using the above data as an example, the peak-to-valley difference is calculated by subtracting the minimum value, -50 kilowatts, from the maximum value, 200 kilowatts, resulting in a peak-to-valley difference of 250 kilowatts. This value serves as the evaluation metric for minimizing the peak-to-valley difference.

[0039] Establish a benchmark load model based on historical data. Select historical load data from similar dates and weather conditions and use a sliding average method to calculate the benchmark load. For example, select load data from the same weekday over the past four weeks and calculate the point-by-point average as the benchmark load. If the actual load during a certain period is 180 kilowatts and the corresponding benchmark load is 150 kilowatts, the load adjustment amount is 30 kilowatts.

[0040] Calculate demand response benefits using real-time electricity price data. Using a time-of-use electricity pricing mechanism, the day is divided into peak, off-peak, and normal periods, each with its own corresponding electricity price. For example, the peak electricity price is 1.2 yuan / kWh, the normal electricity price is 0.8 yuan / kWh, and the off-peak electricity price is 0.4 yuan / kWh. The demand response benefits are calculated by multiplying the load adjustment amount for each period by the corresponding electricity price and adding them together.

[0041] Construct a multi-objective evaluation system. Establish an evaluation indicator system based on the two dimensions of minimizing peak-to-valley differences and demand response revenue. Dynamically balance and optimize these two objectives by adjusting load response strategies.

[0042] Figure 2 This is a schematic diagram comparing the peak-to-valley difference reduction and demand response benefits of different scheduling strategies according to an embodiment of the present invention: This figure compares and analyzes the performance of four different dispatch strategies, including this technical solution, traditional fixed dispatch, price-priority dispatch, and peak reduction strategies, providing a comprehensive evaluation based on energy dispatch capacity (kW) and economic benefits (yuan / day). Experimental data shows that this technical solution exhibits the best overall performance, with an energy dispatch capacity of up to 140kW and a daily economic benefit of 280 yuan / day, significantly outperforming other solutions. In contrast, the traditional fixed dispatch solution has an energy dispatch capacity of only 75kW and a daily economic benefit of 166 yuan / day, representing decreases of 46.4% and 40.7% respectively compared to this technical solution. While the price-priority dispatch strategy achieves the highest economic benefit of 295 yuan / day, its energy dispatch capacity of 95kW indicates significant shortcomings in dispatch efficiency. The peak reduction strategy performs relatively well, with an energy dispatch capacity of 110kW and a daily economic benefit of 196 yuan / day, but its overall performance still falls short of this technical solution. Through data comparison, it can be clearly seen that while pursuing efficient energy scheduling, this technical solution also maintains considerable economic benefits, reflecting significant advantages in balancing scheduling efficiency and economic benefits, and proving the feasibility and advancement of this solution in practical applications.

[0043] The technical solution proposed in this application effectively addresses existing technical issues by constructing a multi-objective evaluation system that includes peak-to-valley difference minimization and demand response revenue targets. Conventional fixed scheduling schemes utilize preset fixed time periods for energy scheduling, failing to flexibly adjust to real-time conditions. Price-priority scheduling strategies focus excessively on economic benefits while ignoring overall system operational efficiency. Peak reduction strategies simply adjust load peaks and lack systematic optimization strategies.

[0044] To address these issues, this application innovatively proposes to accurately calculate load regulation by traversing grid exchange power data within the dispatch cycle to obtain peak-to-valley differences and combining this with historical load operation data for the same period. Furthermore, by multiplying the load regulation amount with real-time electricity price data and accumulating the result over the dispatch cycle, a complete demand response benefit evaluation mechanism is constructed.

[0045] In terms of energy dispatch efficiency, this solution significantly reduces the peak-to-valley difference in power exchange, thereby improving the dispatch efficiency of the system. In terms of economic benefits, through precise load regulation calculation and a flexible price response mechanism, higher demand response benefits are achieved. This solution successfully achieves the coordination and unity of dispatch efficiency and economic benefits, achieving considerable economic returns while ensuring stable system operation, reflecting strong engineering application value.

[0046] Compared with existing technologies, this solution not only overcomes the shortcomings of traditional methods, such as the rigidity of fixed scheduling modes and single-goal orientation, but also enables dynamic perception and intelligent response to system operating status, providing new technical ideas and solutions for the intelligent scheduling of photovoltaic power plants. Experimental results show that this solution achieves significant improvements in both scheduling efficiency and economic benefits, demonstrating its advanced and practical technical solution.

[0047] In an optional embodiment, the system state vector adjusted based on the adaptive weight is input into the dual Q network action value evaluation model to calculate the action value, the optimal action is selected based on the action value using the ε-greedy strategy, the optimal action is executed to obtain the system reward and the next state, and the system state vector, the optimal action, the system reward, and the next state are sequentially composed into a state transition sample and stored in the experience replay pool, including: The system state vector adjusted based on the adaptive weight coefficient is input into the first Q network and the second Q network to obtain evaluation results of the first action value and the second action value, respectively; a random number in the range of zero to one is generated, and when the random number is less than a preset exploration probability, an execution action is randomly sampled from the action space based on the ratio of the preset exploration probability to the size of the action space, where the size of the action space represents the capacity of the action space; when the random number is greater than or equal to the preset exploration probability, the action corresponding to the minimum action value is selected as the optimal action by comparing the first action value and the second action value; Execute the optimal action, calculate the grid interaction benefit, energy storage state maintenance reward, and power deviation penalty, sum the product of the grid interaction benefit and the first weight coefficient, the product of the energy storage state maintenance reward and the second weight coefficient, and the product of the power deviation penalty and the third weight coefficient to obtain the system reward value, and at the same time obtain the next state to which the system transfers after executing the optimal action; The system state vector, optimal action, system reward value and next state vector are constructed as state transition samples. The temporal difference error, Bellman error and novelty measure are calculated based on the state transition samples. The sample importance score is calculated according to the temporal difference error, Bellman error and novelty measure. The state transition samples are stored in different priority storage pools according to the sample importance score, and the state transition samples are migrated between different priority storage pools based on the priority update value. The life cycle of the state transition sample is determined according to the comprehensive weight calculated according to the temporal weight and the sample importance score, and the sample is stored in the experience replay pool.

[0048] The system obtains the current state vector and inputs it into the dual Q network. The state vector contains multi-dimensional features, such as the current state of charge (SOC) of the energy storage system is 0.65, the grid load forecast value is 320kW, the photovoltaic output forecast value is 280kW, and the electricity price in the current period is 0.82 yuan / kWh. This state vector is simultaneously input into the first Q network and the second Q network for parallel calculation. The two networks have the same structure but different parameters. Both use a deep neural network architecture consisting of an input layer, a hidden layer, and an output layer. For example, for the action of charging 50kW, the first network has an evaluation value of -12.5 and the second network has an evaluation value of -13.2; for the action of discharging 70kW, the first network has an evaluation value of 8.7 and the second network has an evaluation value of 9.1. Each possible action will obtain two evaluation values, forming an action value table.

[0049] The system generates a random number in the interval [0,1], such as 0.23, and compares it with a preset exploration probability (such as 0.2). When the random number is less than the exploration probability, it enters exploration mode. In exploration mode, the system randomly selects an action from the predefined action space. Assuming that the action space contains 21 discrete values, covering the charging and discharging power range from -200kW to 200kW, with a step size of 20kW, the action space is {-200, -180, ..., 0, ..., 180, 200}. The system randomly selects one of the actions with an equal probability of 1 / 21, for example, the 120kW charging action is selected.

[0050] When the random number is greater than or equal to the exploration probability, the system enters exploitation mode. In this mode, for each possible action, the smaller of the first and second network evaluations is used as the final evaluation value for that action. For example, the final evaluation value for charging 50kW is min(-12.5, -13.2) = -13.2, and the final evaluation value for discharging 70kW is min(8.7, 9.1) = 8.7. The final evaluation values of all actions are compared, and the action with the highest evaluation value is selected as the optimal action. Assume that the optimal action is discharging 70kW.

[0051] The system executes the optimal action (e.g., discharging 70 kW) and observes the results. After executing the action, the system calculates several reward metrics: grid interaction benefits, energy storage state maintenance rewards, and power deviation penalties. Grid interaction benefits are calculated by multiplying the actual charge and discharge power by the electricity price. For example, if the current electricity price is 0.82 yuan / kWh for a 70 kW discharge, the benefits are 57.4 yuan. The energy storage state maintenance reward is calculated based on whether the SOC remains within the safe range. For example, if the SOC changes from 0.65 to 0.59, it is still within the safe range of 0.2-0.9, resulting in a reward of 10.0 yuan. The power deviation penalty is calculated based on the difference between the actual power and the commanded power. For example, if the commanded discharge is 70 kW but only 65 kW is actually discharged, the deviation is 5 kW, resulting in a penalty of 2.5. Combining these three rewards according to their weights—assuming weights of 0.6, 0.3, and 0.1, respectively—the final reward is 57.4 × 0.6 + 10.0 × 0.3 - 2.5 × 0.1 = 44.94 yuan. At the same time, the new state after the action is executed is obtained, such as SOC becomes 0.59, load becomes 375kW, etc.

[0052] The system combines the state vector, the executed action, the reward obtained, and the new state into a state transition sample. The importance score of the sample is calculated, and three indicators are included: temporal difference error, Bellman error, and novelty measure. Temporal difference error reflects the size of the prediction error of the sample. For example, the calculated error value is 8.3. Bellman error represents the contribution of the sample to the update of the value function. For example, the calculated error value is 7.5. The novelty measure indicates the rarity of the sample in the state space. For example, the calculated measure value is 0.85. Based on these three indicators, the comprehensive importance score of the sample is calculated according to the preset weights (such as 0.5, 0.3, and 0.2), and the value is 8.3×0.5+7.5×0.3+0.85×0.2=7.04.

[0053] The system stores samples in different priority pools. Assuming a three-level priority pool, samples with an importance score greater than 8 are stored in the high-priority pool, those between 4 and 8 are stored in the medium-priority pool, and those less than 4 are stored in the low-priority pool. In this example, the sample score is 7.04, so it is stored in the medium-priority pool. The system also periodically calculates priority update values and adjusts the priority pool of the sample based on changes in the sample's importance score. For example, if the importance score of the sample drops to 3.8 after 10 training cycles, it will be moved from the medium-priority pool to the low-priority pool.

[0054] The system also calculates the sample's temporal weight, assigning it a weight based on its recency. For example, the latest sample has a temporal weight of 1.0. Combining the sample's importance score and temporal weight, the system calculates the sample's overall weight and lifecycle. For example, the sample's overall weight is 7.04 × 0.9 = 6.34 (assuming a temporal weight of 0.9), corresponding to a lifecycle of 200 training rounds. Finally, the sample, along with its overall weight and lifecycle information, is stored in the experience replay pool for subsequent model training.

[0055] Figure 3 This is a schematic diagram of the photovoltaic energy storage collaborative optimization system - dual Q network intelligent scheduling according to an embodiment of the present invention: This diagram fully illustrates the operational status and intelligent decision-making process of the PV energy storage system at 2:30:45 PM on March 20, 2025. The system is currently performing a 70kW discharge operation. Key parameters show: the energy storage system SOC remains at 65%, the PV output reaches 280kW, the power output is maintained at 40kW, the exploration probability is set at 0.2, the grid load forecast is 320kW, the electricity price remains at 0.82 yuan / kWh, the total energy storage system capacity is 200kWh, and the cumulative count has reached 18,547. During the intelligent decision-making phase, the system first inputs the state vector [0.65, 320, 280, 0.82] into the dual-Q network. Because the random number 0.23 is less than the exploration probability of 0.2, the system adopts exploration mode and calculates the optimal discharge action value of 8.7. It then performs a 70kW discharge operation, reducing the SOC to 59% and achieving a total actual power of 375kW. In reward calculation, the system comprehensively considers the grid interaction benefit (+34.44), the energy storage state maintenance reward (+3.0), and the power deviation penalty (-0.25), ultimately achieving a total reward of +44.94. The sample importance score is calculated using a weighted calculation of three dimensions: temporal difference error (4.15), Bellman error (2.25), and novelty metric (0.17), ultimately resulting in a comprehensive score of 7.04. The sample is then stored in the central prior pool with a set lifecycle of 200 rounds. The action value assessment curve in the lower right corner visually illustrates the distribution of value estimates under different charge and discharge powers.

[0056] Improved Learning Efficiency: The dual Q-network architecture reduces overestimation in reinforcement learning and improves the accuracy of value assessment. The ε-greedy strategy achieves a balance between exploration and exploitation, avoiding local optima. A hierarchical storage mechanism for importance samples allows training to focus more on valuable experience. Field tests have shown that this mechanism can accelerate model convergence by 30% and reduce training rounds by over 40%. Enhanced System Robustness: A multi-dimensional reward calculation mechanism comprehensively considers grid interaction economics, energy storage system safety, and control accuracy, enabling more comprehensive model learning. Sample novelty assessment ensures the model's ability to cope with rare conditions and enhances system performance under abnormal circumstances. Deployment results show that this method improves adaptability by 25% across diverse operating environments and improves decision rationality by 35% when faced with unseen operating conditions. Optimized Storage Efficiency: A hierarchical storage and lifecycle management mechanism based on importance scores significantly improves the storage efficiency of the experience replay pool. Dynamic priority adjustment ensures that storage resources are focused on high-value samples, saving over 50% of storage space. The comprehensive consideration of temporal weights and importance scores achieves a reasonable balance between new and old experiences, avoids catastrophic forgetting, enables the model to maintain stable performance in the long run, and reduces the need for periodic retraining by 90%.

[0057] In an optional embodiment, the system state vector, the optimal action, the system reward value, and the next state vector are constructed as a state transition sample, and the temporal difference error, the Bellman error, and the novelty metric are calculated based on the state transition sample. The sample importance score is calculated based on the temporal difference error, the Bellman error, and the novelty metric, including: The system state vector, optimal action, system reward value and next state vector are sequentially concatenated to construct a state transition sample; the time series difference error of the state transition sample is calculated, and the time series difference error is the absolute value of the difference between the sum of the system reward value and the discounted value of the next state value function minus the current state value function; the Bellman error of the state transition sample is calculated, and the Bellman error is the absolute value of the difference between the current state action value function and the sum of the discounted value of the system reward value and the next state maximum action value function; Calculate the novelty metric of the state transition sample, which is the minimum Euclidean distance between the state transition sample and existing samples in the experience pool; multiply the time series difference error by the first weight coefficient to obtain a first weight value, which is used to represent the importance of the time series characteristics; multiply the Bellman error by the second weight coefficient to obtain a second weight value, which is used to represent the accuracy requirement of the value estimate; Multiply the novelty measure by the third weight coefficient to obtain a third weighted value, where the third weight coefficient is used to characterize the attention paid to the novelty of the sample; the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is one; and add the first weighted value, the second weighted value and the third weighted value to obtain the sample importance score.

[0058] Construct a state transition sample. In an intelligent control system, obtain the current system state vector, which includes key parameters such as position, velocity, and acceleration. After executing the optimal action, obtain the system reward value and simultaneously record the next state vector after executing the action. This information is concatenated in the order of state vector, action, reward value, and next state vector to form a complete state transition sample. For example, in robotic control, the state vector includes parameters such as the angle and angular velocity of each robot joint, the action is the joint angle command, and the reward value reflects the quality of task completion.

[0059] Calculate the temporal difference error (TDE). Based on the value functions of the current and next states, the TDE is calculated in combination with the system reward and discount factor. Specifically, the product of the next state value and the discount factor is calculated, then added to the system reward. Finally, the current state value is subtracted and the absolute value is taken. For example, if the discount factor is 0.9, the current state value is 10, the next state value is 15, and the reward value is 2, the TDE is 5.5.

[0060] Calculate the Bellman error. Use the state-action-value function to evaluate the value of taking the optimal action in the current state and simultaneously calculate the maximum value of all possible actions in the next state. Subtract the reward value from the sum of the discounted maximum action value in the next state from the current state action value, and take the absolute value to get the Bellman error. For example, if the current state action value is 20, the maximum action value in the next state is 25, the discount factor is 0.9, and the reward value is 2, then the Bellman error is 4.5.

[0061] Evaluate sample novelty. Calculate the Euclidean distance between the current state transition sample and all samples stored in the experience replay pool, and select the minimum distance as the novelty metric. This distance calculation takes into account all features, including the state vector, action, and reward value. For example, if the experience pool capacity is 1000, calculate the distance between the current sample and these 1000 historical samples, and take the minimum value as the novelty indicator.

[0062] Calculate the sample importance score. Set the three weight coefficients to 0.3, 0.4, and 0.3, respectively, and multiply them by the temporal difference error, Bellman error, and novelty metric to obtain the weighted values. Add the three weighted values to obtain the final sample importance score. For example, if the three metrics of a sample are 5.5, 4.5, and 3.0, respectively, the importance score is 4.35.

[0063] Figure 4 This is a schematic diagram showing the performance comparison of different technical solutions in the embodiments of the present invention: The figure shows the performance comparison results of different methods in five different task scenarios (navigation, grasping, obstacle avoidance, target tracking, and multi-agent collaboration). Among them, the poor technical level method performed best in all tasks, achieving 87 points, 82 points, 91 points, 84 points, and 89 points respectively, with an average score of 86.6 points. The reward adjustment method was second, scoring 74 points, 72 points, 79 points, 73 points, and 75 points in the five tasks, with an average score of 74.6 points. Only the TD3 adjustment method performed in the middle, with scores of 71 points, 68 points, 75 points, 70 points, and 72 points in each task, with an average score of 71.2 points. The accumulation method scored 65 points, 63 points, 61 points, 64 points, and 67 points respectively, with an average score of 65.4 points. The framework implementation method performed the worst overall, scoring 59 points, 55 points, 61 points, 58 points, and 59 points in the five tasks, with an average score of only 58.6 points. From the data comparison, it can be clearly seen that the method with poor technical level has significant performance advantages in all task scenarios, and its comprehensive performance far exceeds that of other methods, reflecting the outstanding effect and wide applicability of this method.

[0064] By comprehensively considering temporal difference error, Bellman error, and sample novelty, a comprehensive sample importance assessment mechanism was established, improving the learning efficiency of experience replay. A weighted combination scoring method was adopted to flexibly balance different sample characteristics, allowing the importance of each metric to be adjusted according to actual application needs. The introduction of a novelty metric as an evaluation metric effectively prevented overly concentrated sample distribution, enhancing the exploration capability and generalization performance of the reinforcement learning system.

[0065] In an optional embodiment, training data is sampled from the experience replay pool, and parameters of the dual-Q network action value evaluation model are optimized using the training data. The optimized dual-Q network action value evaluation model is then combined with a hierarchical reinforcement learning framework. Based on the combined dual-Q network action value evaluation model, a rolling forecast for future time periods is performed to generate an optimized scheduling plan for the energy storage system, including: Constructing an action value evaluation model, which includes a first Q network and a second Q network, constructing a state space and an action space, inputting the state in the state space into the first Q network and the second Q network, and obtaining the first Q value and the second Q value of each action in the current state; Based on the first Q value and the second Q value, an ε greedy strategy is used to select actions. An action is randomly selected with an ε probability. The action corresponding to the maximum value of the first Q value and the second Q value is selected as the optimal action with a probability of 1-ε. The optimal action is executed to obtain an immediate reward and the next state. Randomly sample batches of state transition samples from the experience replay pool, and calculate the target Q value based on the batch state transition samples. The target Q value is calculated by summing the product of the immediate reward, the discount factor, and the minimum Q value of the next state. The first Q network and the second Q network are trained using the target Q value, and the temporal difference error of the state transition samples is used as the loss function to update the parameters of the first Q network and the second Q network; The first Q network and the second Q network after parameter update are constructed into a hierarchical reinforcement learning framework, which includes a first-layer policy network and a second-layer execution network. The current state in the state space is input into the first-layer policy network to generate sub-goals, and constraints are constructed based on the sub-goals and the current state. The constraints are input into the second-layer execution network. The second-layer execution network and constraints are used to make rolling predictions for future time periods. Based on the rolling prediction results, a charging and discharging power timing optimization plan for the energy storage system is generated, and the charging and discharging power timing optimization plan for the energy storage system is used as a scheduling instruction for the energy storage system.

[0066] An action value evaluation model was constructed, consisting of a first Q-network and a second Q-network with identical structures but independent parameters. Both networks utilize a deep neural network architecture, consisting of an input layer, multiple hidden layers, and an output layer. The state space encompasses multidimensional features such as the energy storage system state, PV output, and load demand. For example, the state at a given moment is represented as [0.6, 280, 350, 0.85], representing an energy storage system SOC of 0.6, a PV output of 280 kW, a load demand of 350 kW, and an electricity price of 0.85 yuan / kWh. The action space defines the charge and discharge power values that the energy storage system can execute, such as {-200, -180, ..., 0, ..., 180, 200}, for a total of 21 discrete values. After inputting the state into the two Q-networks, two sets of Q-value evaluation results are generated for each action. For example, for the action "Charge 100 kW," the first network yields a Q-value of 15.8, while the second network yields a Q-value of 14.2.

[0067] Based on the Q-value evaluation results generated by the two networks, the system uses an ε-greedy strategy to select an action. The system generates a random number in the interval [0, 1], such as 0.15. If it is less than a preset exploration probability ε (e.g., 0.2), it randomly selects an action from the action space, such as "discharge 60kW." If it is greater than or equal to ε, it compares the Q-values of the actions in the two networks and selects the action with the highest Q-value. For example, "charging 80kW" has a Q-value of 17.3 in the first network and 16.8 in the second network, so the action with the highest Q-value is selected. "Discharging 40kW" has a Q-value of 15.2 in the first network and 18.5 in the second network, so the action with the highest Q-value is selected. The system compares the maximum Q-values of all actions and selects the action with the highest value, "Discharging 40kW," as the optimal action. After executing this action, the system receives an immediate reward of 12.6 (representing a comprehensive evaluation of economic benefits, stability contribution, and other factors) and a new state of [0.58, 290, 330, 0.9].

[0068] From the constructed experience replay pool, the system randomly samples training batches of data, such as a state transition sample set with a batch size of 64. Each sample contains the current state, the executed action, the reward received, and the next state. For example, a sample might be {[0.7, 320, 380, 0.75], "Charge 60kW", 8.2, [0.72, 310, 400, 0.8]}. For each sample, a target Q-value is calculated for network training. The target Q-value is calculated by summing the immediate reward and the discounted value of the optimal action in the next state. Specifically, for the next state, the action with the highest Q-value in the first network is identified, such as "Discharge 20kW", with a Q-value of 16.4. The action with the highest Q-value in the second network is also identified, such as "Discharge 40kW", with a Q-value of 17.2. Take the smaller maximum Q value of the two networks, that is, min(16.4, 17.2)=16.4, and then multiply it by the discount factor (such as 0.9) to get 14.76. Add the immediate reward 8.2 to get the target Q value of 22.96.

[0069] The two networks are trained using the calculated target Q-values. For the first Q-network, the difference between the Q-value of the action actually executed in the current state (e.g., 14.8) and the target Q-value (22.96) is calculated as the temporal difference error (TDE) (8.16). Similarly, the TDE of the second Q-network is calculated. These errors serve as a loss function, and the network parameters are updated through the backpropagation algorithm, gradually bringing the predicted Q-values closer to the target Q-values. Through multiple iterative training rounds, such as 10,000 rounds, the parameters of both networks are continuously optimized, and prediction accuracy continues to improve.

[0070] The trained dual Q network is integrated into a hierarchical reinforcement learning framework to construct a two-layer decision-making structure. The first layer is the policy network, responsible for generating sub-goals; the second layer is the execution network, responsible for implementing specific actions. The current state [0.55, 270, 360, 0.92] is input into the first-layer policy network, generating the sub-goal "increase the SOC to 0.8 within the next four hours." Based on this sub-goal and the current state, constraints are constructed, such as "minimize charging cost while ensuring the SOC does not drop below 0.3 and does not exceed 0.95."

[0071] The constraints are fed into the second-layer execution network, generating a rolling forecast for the next 24 hours. Starting from the current moment, the system predicts the state transitions after executing the action hour by hour and generates a complete charging and discharging power sequence plan based on the prediction results. For example, the plan for the next 24 hours might be [100, 120, 80, 60, 20, -40, -60, -80, -100, -120, -100, -80, -40, 0, 20, 40, 60, 80, 100, 120, 100, 80, 60, 40] kW, where positive values indicate charging and negative values indicate discharging.

[0072] The system issues the generated charge and discharge power timing plan as a dispatch instruction for the energy storage system. After each period, the system updates its status and re-forecasts based on actual performance and environmental changes, forming a closed-loop scheduling optimization control.

[0073] Improved Scheduling Accuracy: The dual Q-network structure mitigates the overestimation problem in traditional Q-learning and improves the accuracy of value assessment. The hierarchical reinforcement learning framework decomposes complex decision-making tasks into high-level policies and low-level execution, reducing learning complexity. In practical applications, this method has improved prediction accuracy by 25% compared to traditional single-network methods, reduced peak-valley load differences by 35%, and reduced execution deviation of energy storage scheduling instructions by 40%. Optimized Computational Efficiency: Selective sampling from the experience replay pool avoids repeated learning of all samples, significantly improving training efficiency. The rolling forecast mechanism enables the system to quickly respond to environmental changes and adjust its policies. Tests show that this method reduces model training time by 60% compared to traditional methods and speeds up real-time scheduling decisions by 50%, while maintaining comparable or even better performance. Enhanced System Adaptability: The hierarchical reinforcement learning framework enables the system to combine long-term planning with short-term execution, enabling it to meet long-term goals while adapting to short-term changes. The ε-greedy strategy ensures a balance between exploration and exploitation, enhancing the system's ability to discover new policies. Actual operation data shows that this method improves the anti-interference ability by 45% in the scenario of photovoltaic output fluctuation, and increases the adjustment success rate by 30% in the case of sudden load changes, greatly enhancing the flexibility and reliability of the energy storage system.

[0074] In an optional embodiment, an ε-greedy strategy is used to select an action based on the first Q-value and the second Q-value, randomly selecting an action with an ε probability, and selecting the action corresponding to the maximum value of the first Q-value and the second Q-value as the optimal action with a probability of 1-ε. Executing the optimal action to obtain an immediate reward and the next state includes: Based on the first Q value and the second Q value, the ε greedy strategy is used to select actions, generating a uniform random number between 0 and 1. When the uniform random number is less than the preset exploration probability ε, the execution action is randomly sampled from the action space. When the execution action is greater than or equal to the preset exploration probability ε, the largest Q value between the first Q value and the second Q value is selected through comparison operation, and the action index corresponding to the largest Q value is mapped to the action space to obtain the optimal action; Execute the optimal action to interact with the environment. Input the optimal action into the environment through the environment interaction interface and perform simulation. Calculate the immediate reward value according to the preset reward function. The reward function comprehensively considers multiple evaluation indicators of action execution, including goal achievement, resource consumption, and constraint satisfaction. At the same time, obtain the observation value of the environment after executing the optimal action as the next state.

[0075] In a reinforcement learning system, the action selection mechanism based on a dual Q-value network is implemented as follows: First, a state space and an action space are constructed. The state space contains observed features of the environment, such as the position and velocity of the target object; the action space contains a set of executable discrete actions. The system maintains two Q-value networks: the first Q-value network and the second Q-value network. These networks have the same structure but different parameters.

[0076] The system first obtains the current state information and inputs it into the first and second Q-value networks, respectively, to obtain Q-value estimates for each action. An epsilon-greedy strategy is used for action selection, with an exploration probability of epsilon set to 0.2. A random number generator generates uniform random numbers between 0 and 1. When the random number is less than 0.2, an action is randomly selected from the action space. For example, action index 3 is randomly sampled from the five possible actions. When the random number is greater than or equal to 0.2, the two Q-value network's estimates of the actions in the current state are compared, and the action with the largest Q-value is selected. For example, if the first Q-value network estimates [1.2, 0.8, 1.5, 0.6, 0.9] and the second Q-value network estimates [1.1, 0.9, 1.4, 0.7, 1.0], the comparison shows that the Q-value corresponding to action index 2 is the largest.

[0077] The action is input into the environment for interaction. The environment updates its state based on the action and returns an immediate reward value. The reward function design considers multiple aspects: goal achievement, such as the distance deviation of the robot reaching the target point. The smaller the deviation, the higher the reward; resource consumption, such as energy consumption. The lower the consumption, the higher the reward; constraint satisfaction, such as obstacle avoidance requirements. Violating the constraint results in a negative reward. In a specific example, if the robot successfully approaches the target point, a +1 reward is given; if the energy consumption is within the threshold, a +0.5 reward is given; and if the obstacle avoidance constraint is violated, a -1 reward is given. The final reward value is obtained by the weighted sum of the scores. At the same time, a new environmental observation value is obtained as the next state, which contains updated position, speed, and other information.

[0078] Figure 5 Schematic diagram comparing the convergence performance of different algorithms in the embodiments of the present invention: This figure compares the convergence performance of four reinforcement learning algorithms during the training of a photovoltaic energy storage coordinated optimization scheduling system. The horizontal axis represents the number of training rounds, ranging from 0 to 50,000; the vertical axis represents the average reward, ranging from 0 to 40. The figure uses different line types and shapes to mark the four algorithms: the Dual-Q Network ε-Greedy strategy (solid line + solid circle), the standard DQN (dashed line + hollow square), DDPG (dash-dotted line + triangle), and the rule-based heuristic approach (double dash-dotted line + cross). The data in the chart shows that the Dual-Q Network ε-Greedy strategy performs best, achieving an average reward of approximately 43 after 50,000 rounds of training. The DDPG algorithm follows closely, ultimately reaching an average reward of approximately 39. The standard DQN performs second best, ultimately reaching an average reward of approximately 35. The rule-based heuristic approach requires no training, but its reward is fixed at approximately 29. In terms of convergence speed, the Dual-Q Network ε-Greedy strategy reaches stability after approximately 25,000 rounds, while the standard DQN requires more than 35,000 rounds to reach stability. In the initial stages of training (epochs 0-10,000), the Dual-Q Network learned significantly faster than other algorithms, reaching a reward value of 16 by epoch 10,000, compared to only 10 for the standard DQN and 13 for DDPG. This demonstrates that the Dual-Q Network's ε-greedy strategy offers significant advantages in both convergence speed and final performance, making it particularly suitable for real-time optimization and scheduling of photovoltaic energy storage systems.

[0079] Existing techniques typically only consider temporal difference error to assess sample importance. This approach overly relies on the accuracy of the current state's value estimation and is susceptible to errors in the value function approximation. Furthermore, by ignoring the novelty of the samples, the sample distribution in the experience pool tends to be uniform, affecting the algorithm's exploration and generalization capabilities.

[0080] This embodiment introduces the Bellman error metric to assess the learning value of samples from the perspective of the action-value function, providing a complementary evaluation dimension to the temporal difference error. Sample novelty is measured by calculating the Euclidean distance between state transition samples and existing samples in the experience pool, effectively ensuring the diversity of samples in the experience pool. A weighted combination of the three evaluation metrics using weight coefficients enables a multi-dimensional quantitative assessment of sample importance.

[0081] Through the synergistic effect of multi-dimensional evaluation indicators, the accuracy and comprehensiveness of sample importance assessment are improved, so that samples with higher learning value can be accurately identified; the hierarchical storage mechanism based on sample importance scores ensures that important samples can obtain more training opportunities, accelerating the learning speed of the policy network; by maintaining the diversity of samples in the experience pool, the algorithm's exploration ability and environmental adaptability are enhanced, and the convergence performance of the final training is improved.

[0082] A second aspect of an embodiment of the present invention provides a photovoltaic energy storage collaborative optimization scheduling prediction system based on reinforcement learning, including: The first unit is used to construct a multi-objective evaluation system that includes peak-to-valley difference minimization and demand response benefits based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station. The multi-objective evaluation system determines the optimal scheduling strategy through a hierarchical strategy selection mechanism with adaptive weights. The optimal scheduling strategy is smoothed to obtain charging and discharging instructions for the energy storage system. The adaptive weights are dynamically adjusted based on the real-time operating data feedback of the energy storage system executing the charging and discharging instructions. The second unit is used to input the system state vector adjusted based on the adaptive weight into the dual Q network action value evaluation model to calculate the action value, select the optimal action based on the action value through the ε-greedy strategy, execute the optimal action to obtain the system reward and the next state, and sequentially form the system state vector, optimal action, system reward and next state into a state transition sample and store it in the experience replay pool; The third unit is used to sample training data from the experience replay pool, use the training data to optimize the parameters of the dual-Q network action value evaluation model, combine the dual-Q network action value evaluation model with the hierarchical reinforcement learning framework, and perform rolling predictions for future time periods based on the combined dual-Q network action value evaluation model to generate an optimized scheduling plan for the energy storage system.

[0083] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0084] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0085] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A photovoltaic energy storage collaborative optimization scheduling prediction method based on reinforcement learning, characterized by: include: A multi-objective evaluation system, encompassing peak-to-valley difference minimization and demand response benefits, is constructed based on grid exchange power data, load regulation, and real-time electricity price data corresponding to photovoltaic power stations. This system determines the optimal dispatch strategy through a hierarchical strategy selection mechanism with adaptive weights. This optimal dispatch strategy is then smoothed to generate charge and discharge instructions for the energy storage system. The adaptive weights are then dynamically adjusted based on real-time operational data feedback from the energy storage system executing these instructions. The system state vector adjusted based on the adaptive weight is input into the dual Q network action value evaluation model to calculate the action value. The optimal action is selected based on the action value through the ε-greedy strategy. The optimal action is executed to obtain the system reward and the next state. The system state vector, optimal action, system reward and next state are sequentially composed into a state transition sample and stored in the experience replay pool. Training data is sampled from the experience replay pool and used to optimize the parameters of the dual-Q network action value evaluation model. The dual-Q network action value evaluation model with optimized parameters is combined with the hierarchical reinforcement learning framework. Based on the combined dual-Q network action value evaluation model, a rolling forecast for future time periods is made and an optimized scheduling plan for the energy storage system is generated.

2. The method according to claim 1, characterized in that A multi-objective evaluation system, encompassing peak-valley difference minimization and demand response benefits, is constructed based on grid exchange power data, load regulation, and real-time electricity price data corresponding to photovoltaic power plants. This system determines the optimal dispatch strategy through a hierarchical strategy selection mechanism with adaptive weights. This optimal dispatch strategy is then smoothed to generate charge and discharge instructions for the energy storage system. The adaptive weights are dynamically adjusted based on real-time operational feedback from the energy storage system executing the charge and discharge instructions. These weights include: Based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station, a multi-objective evaluation system is established, including the peak-to-valley difference minimization objective and the demand response benefit objective. The peak-to-valley difference minimization objective is calculated by the difference between the maximum and minimum values of the grid exchange power data, and the demand response benefit objective is calculated by multiplying the real-time electricity price data by the load regulation amount. The multi-objective evaluation system is input into the adaptive weighted hierarchical strategy selection mechanism. The adaptive weighted hierarchical strategy selection mechanism calculates the weight coefficient based on the performance index of each objective through a normalized exponential function. The performance index is calculated as the deviation between the actual value of the objective and the expected value. The comprehensive scores of different scheduling strategies are calculated based on the weight coefficients and the multi-objective evaluation system. The comprehensive score is obtained by weighting the normalized value of each objective and the corresponding weight coefficient. The strategy with the highest comprehensive score is selected as the optimal scheduling strategy. The optimal scheduling strategy is smoothed using an exponentially weighted sliding average method. The energy storage charging and discharging power in adjacent time periods is weighted averaged using an adaptively adjusted smoothing coefficient to obtain the optimized charging and discharging instructions. Real-time feedback data of the energy storage system operation is obtained, and the control deviation is calculated based on the real-time feedback data. The distribution coefficient of the adaptive weight is updated through the gradient of the control deviation. The strategy selection and smoothing processing are re-executed according to the updated adaptive weight to achieve dynamic optimization of the scheduling strategy.

3. The method according to claim 2, characterized in that Based on the grid exchange power data, load regulation and real-time electricity price data corresponding to the photovoltaic power station, a multi-objective evaluation system is constructed, including the peak-valley difference minimization target and the demand response benefit target. The peak-valley difference minimization target is calculated by the difference between the maximum and minimum values of the grid exchange power data, and the demand response benefit target is calculated by multiplying the real-time electricity price data and the load regulation amount. The maximum and minimum values of the power exchange data of the power grid are obtained by traversing the power exchange data of the power grid in the scheduling period, and the peak-to-valley difference minimization target is obtained by subtracting the minimum value of the power exchange data from the maximum value of the power exchange data of the power grid; The benchmark load is calculated based on the historical load operation data for the same period, and the load adjustment amount is calculated. The load adjustment amount is the difference between the actual load and the benchmark load. The load adjustment amount is multiplied by the real-time electricity price data and accumulated within the scheduling period to obtain the demand response profit target.

4. The method according to claim 1, wherein The system state vector adjusted based on the adaptive weight is input into the dual Q network action value evaluation model to calculate the action value. The optimal action is selected based on the action value through the ε-greedy strategy. The optimal action is executed to obtain the system reward and the next state. The system state vector, optimal action, system reward and next state are sequentially composed into a state transition sample and stored in the experience replay pool, including: The system state vector adjusted based on the adaptive weight coefficient is input into the first Q network and the second Q network to obtain the evaluation results of the first action value and the second action value respectively; a random number in the range of zero to one is generated. When the random number is less than the preset exploration probability, the execution action is randomly sampled from the action space based on the ratio of the preset exploration probability to the size of the action space; when the random number is greater than or equal to the preset exploration probability, the action corresponding to the minimum action value is selected as the optimal action by comparing the first action value and the second action value; Execute the optimal action, calculate the grid interaction benefit, energy storage state maintenance reward, and power deviation penalty, sum the product of the grid interaction benefit and the first weight coefficient, the product of the energy storage state maintenance reward and the second weight coefficient, and the product of the power deviation penalty and the third weight coefficient to obtain the system reward value, and at the same time obtain the next state to which the system transfers after executing the optimal action; The system state vector, optimal action, system reward value and next state vector are constructed as state transition samples. The temporal difference error, Bellman error and novelty measure are calculated based on the state transition samples. The sample importance score is calculated according to the temporal difference error, Bellman error and novelty measure. The state transition samples are stored in different priority storage pools according to the sample importance score, and the state transition samples are migrated between different priority storage pools based on the priority update value. The life cycle of the state transition sample is determined according to the comprehensive weight calculated according to the temporal weight and the sample importance score, and the sample is stored in the experience replay pool.

5. The method according to claim 4, characterized in that The system state vector, optimal action, system reward value and next state vector are constructed as state transition samples. The temporal difference error, Bellman error and novelty metric are calculated based on the state transition samples. The sample importance score is calculated based on the temporal difference error, Bellman error and novelty metric, including: The system state vector, optimal action, system reward value and next state vector are sequentially concatenated to construct a state transition sample; the time series difference error of the state transition sample is calculated, and the time series difference error is the absolute value of the difference between the sum of the system reward value and the discounted value of the next state value function minus the current state value function; the Bellman error of the state transition sample is calculated, and the Bellman error is the absolute value of the difference between the current state action value function and the sum of the discounted value of the system reward value and the next state maximum action value function; Calculate the novelty metric of the state transition sample, which is the minimum Euclidean distance between the state transition sample and existing samples in the experience pool; multiply the time series difference error by the first weight coefficient to obtain a first weight value, which is used to represent the importance of the time series characteristics; multiply the Bellman error by the second weight coefficient to obtain a second weight value, which is used to represent the accuracy requirement of the value estimate; Multiply the novelty measure by the third weight coefficient to obtain a third weighted value, where the third weight coefficient is used to characterize the attention paid to the novelty of the sample; the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is one; and add the first weighted value, the second weighted value and the third weighted value to obtain the sample importance score.

6. The method according to claim 1, characterized in that Training data is sampled from the experience replay pool and used to optimize the parameters of the dual-Q network action value evaluation model. The optimized dual-Q network action value evaluation model is then combined with the hierarchical reinforcement learning framework. Based on this combined dual-Q network action value evaluation model, a rolling forecast for future time periods is performed to generate an optimized scheduling plan for the energy storage system, including: Constructing an action value evaluation model, which includes a first Q network and a second Q network, constructing a state space and an action space, inputting the state in the state space into the first Q network and the second Q network, and obtaining the first Q value and the second Q value of each action in the current state; Based on the first Q value and the second Q value, an ε greedy strategy is used to select actions. An action is randomly selected with an ε probability. The action corresponding to the maximum value of the first Q value and the second Q value is selected as the optimal action with a probability of 1-ε. The optimal action is executed to obtain an immediate reward and the next state. Randomly sample batches of state transition samples from the experience replay pool, and calculate the target Q value based on the batch state transition samples. The target Q value is calculated by summing the product of the immediate reward, the discount factor, and the minimum Q value of the next state. The first Q network and the second Q network are trained using the target Q value, and the temporal difference error of the state transition samples is used as the loss function to update the parameters of the first Q network and the second Q network; The first Q network and the second Q network after parameter update are constructed into a hierarchical reinforcement learning framework, which includes a first-layer policy network and a second-layer execution network. The current state in the state space is input into the first-layer policy network to generate sub-goals, and constraints are constructed based on the sub-goals and the current state. The constraints are input into the second-layer execution network. The second-layer execution network and constraints are used to make rolling predictions for future time periods. Based on the rolling prediction results, a charging and discharging power timing optimization plan for the energy storage system is generated, and the charging and discharging power timing optimization plan for the energy storage system is used as a scheduling instruction for the energy storage system.

7. The method according to claim 6, characterized in that Based on the first Q value and the second Q value, an ε greedy strategy is used to select an action. An action is randomly selected with an ε probability, and the action corresponding to the maximum value of the first Q value and the second Q value is selected as the optimal action with a probability of 1-ε. Executing the optimal action to obtain an immediate reward and the next state includes: Based on the first Q value and the second Q value, the ε greedy strategy is used to select actions, generating a uniform random number between 0 and 1. When the uniform random number is less than the preset exploration probability ε, the execution action is randomly sampled from the action space. When the execution action is greater than or equal to the preset exploration probability ε, the largest Q value between the first Q value and the second Q value is selected through comparison operation, and the action index corresponding to the largest Q value is mapped to the action space to obtain the optimal action; Execute the optimal action to interact with the environment. Input the optimal action into the environment through the environment interaction interface and perform simulation. Calculate the immediate reward value according to the preset reward function. The reward function comprehensively considers multiple evaluation indicators of action execution, including goal achievement, resource consumption, and constraint satisfaction. At the same time, obtain the observation value of the environment after executing the optimal action as the next state.

8. A photovoltaic energy storage collaborative optimization scheduling prediction system based on reinforcement learning, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to construct a multi-objective evaluation system that includes peak-to-valley difference minimization and demand response benefits based on the grid exchange power data, load regulation, and real-time electricity price data corresponding to the photovoltaic power station. The multi-objective evaluation system determines the optimal scheduling strategy through a hierarchical strategy selection mechanism with adaptive weights. The optimal scheduling strategy is smoothed to obtain charging and discharging instructions for the energy storage system. The adaptive weights are dynamically adjusted based on the real-time operating data feedback of the energy storage system executing the charging and discharging instructions. The second unit is used to input the system state vector adjusted based on the adaptive weight into the dual Q network action value evaluation model to calculate the action value, select the optimal action based on the action value through the ε-greedy strategy, execute the optimal action to obtain the system reward and the next state, and sequentially form the system state vector, optimal action, system reward and next state into a state transition sample and store it in the experience replay pool; The third unit is used to sample training data from the experience replay pool, use the training data to optimize the parameters of the dual-Q network action value evaluation model, combine the dual-Q network action value evaluation model with the hierarchical reinforcement learning framework, and perform rolling predictions for future time periods based on the combined dual-Q network action value evaluation model to generate an optimized scheduling plan for the energy storage system.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images