Source load storage dynamic strategy verification method based on double-layer reinforcement learning

Abstract

The invention discloses a source load storage dynamic strategy verification method based on double-layer reinforcement learning, and the method comprises the following steps: S1, collecting the operation data of a source load storage system, and constructing a standardized operation data set; s2, constructing a double-layer reinforcement learning model, generating a global scheduling strategy by an upper-layer strategy network, and outputting an action decision strategy by a lower-layer strategy network; s3, performing joint training on the double-layer reinforcement learning model by adopting a strategy gradient optimization method, and outputting a scheduling strategy; s4, introducing an integral gradient method to analyze a scheduling strategy, and constructing a key scheduling state node set; s5, optimizing the generation logic of the global scheduling strategy to obtain an optimized double-layer reinforcement learning model; s6, constructing a plurality of source-load-storage system operation scenes to form a typical operation scene set; and S7, deploying the optimized double-layer reinforcement learning model in the typical operation scene set, and outputting a strategy verification result. According to the invention, through combination of double-layer reinforcement learning and an integral gradient method, source-load-storage dynamic strategy verification is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent power system dispatching and reinforcement learning, and in particular to a source-load-storage dynamic strategy verification method based on double-layer reinforcement learning. Background Art

[0002] With the continuous optimization of the energy structure, the coordinated dispatching system of sources, grids, loads, and storage plays an increasingly critical role in the new power system. By integrating the generation side, the grid side, the load side, and the energy storage system, this system achieves a coordinated matching of energy and information flows to enhance the flexibility, economy, and security of the overall energy system. In this context, how to formulate scientific and efficient source-load-storage dispatching strategies in a dynamic and changing environment has become a core issue in the research and engineering practice of intelligent dispatching systems. Especially in the complex scenarios of increasing renewable energy penetration, gradual marketization of electricity pricing mechanisms, and strong uncertainty in power loads, dispatching strategies must not only respond to changes in system status in real time but also possess strong generalization and stability to achieve optimal performance in long-term operation.

[0003] Currently, the formulation of source-load-storage scheduling strategies still primarily relies on model predictive control (MPC), rolling optimization (RO), and rule-based approaches. Most of these approaches are based on system models and deterministic forecast data. They first predict the future system state within a certain range and then perform finite-step optimization in conjunction with economic and safety constraints. However, in actual operation, the source-load-storage system state is highly nonlinear, subject to frequent disturbances, and subject to significant external uncertainties, such as renewable power output fluctuations, sudden load changes, and drastic price fluctuations. These traditional approaches often exhibit poor adaptability, low update frequency, and easily ineffective optimization when faced with real-time and global requirements. Furthermore, traditional approaches struggle to fully exploit the deep patterns in system state data, resulting in a lack of forward-looking insights into future trends in their policy outputs, making it difficult to meet the current requirements for intelligent and proactive scheduling in complex systems.

[0004] In recent years, with the development of deep reinforcement learning technology, research on scheduling strategies based on reinforcement learning has gradually attracted attention. Reinforcement learning continuously optimizes strategies through trial and error through interactive learning between an agent and its environment. It has the advantages of being model-independent, highly adaptable, and exploratory, making it particularly suitable for handling scheduling problems in complex and uncertain environments. Previous studies have attempted to use algorithms such as DQN, DDPG, and PPO for end-to-end control of source-load-storage systems, with some success. However, most of these methods remain stuck in a single-layer decision-making structure, making it difficult to distinguish between global scheduling intent and local action execution logic. This results in a lack of hierarchy and structure in the strategies during long-term operation, making it difficult to support the multi-scale control requirements of complex systems.

[0005] To address the above problems, some studies have introduced a two-layer reinforcement learning architecture (Hierarchical Reinforcement Learning, HRL), which splits the scheduling task into the upper-layer global sub-goal planning and the lower-layer specific action execution, thereby improving the structure and interpretability of the system strategy. Although this method has improved the hierarchical division of labor of the strategy to a certain extent, there are still many challenges. First, the feedback channel between the upper and lower-layer intelligent agents is unclear, which easily leads to inconsistencies between the upper-layer sub-goals and the lower-layer actions, affecting the scheduling efficiency and system stability. Second, the sub-goal boundary division method generally relies on manual setting, lacks the ability to dynamically adjust the boundaries according to the system response characteristics, resulting in limited strategy generalization ability. Third, as a black box structure, the reinforcement learning model is difficult to understand the impact of input features on the strategy output. The model has poor interpretability, which limits its engineering deployment in actual power systems.

[0006] Furthermore, in existing reinforcement learning frameworks, the policy optimization process typically relies on a single objective function, such as maximizing cumulative rewards or minimizing execution costs, and fails to fully consider the trade-offs between multi-dimensional performance indicators during the scheduling process. For example, frequent charging and discharging of energy storage systems may yield higher immediate benefits, but this can also lead to hidden dangers such as drastic fluctuations in the state of charge and frequent policy switching, which can affect the operational stability of the system. Traditional reinforcement learning algorithms often struggle to achieve global optimality when faced with such multi-objective conflicts, and can only achieve single-point improvements in local policies. Furthermore, the performance of policies in different scenarios is highly volatile, and there is a lack of a unified policy evaluation system to support robustness verification and model migration evaluation in multiple scenarios.

[0007] More importantly, most current reinforcement learning scheduling methods lack support for model interpretability and the ability to identify key states. This makes it difficult to identify the most critical state variables and time nodes in the policy output, making it difficult to achieve structural adaptive optimization of the scheduling model. Although some studies have introduced attention mechanisms or attribution methods based on reward differences to attempt to explain model behavior, their accuracy, stability, and generalization capabilities still cannot meet the transparency and controllability requirements of complex power systems. This is especially true when scheduling strategies require manual review or rule correction by power dispatchers or dispatch master station systems. The lack of interpretability and the ability to identify key nodes become major bottlenecks limiting the application of reinforcement learning.

[0008] Therefore, how to provide a source-load-storage dynamic strategy verification method based on double-layer reinforcement learning is a problem that technicians in this field urgently need to solve. Summary of the Invention

[0009] One purpose of the present invention is to propose a source-load-storage dynamic strategy verification method based on double-layer reinforcement learning. The present invention realizes hierarchical scheduling decision-making of the source-load-storage system in complex operation scenarios by constructing a double-layer reinforcement learning model including an upper-level sub-goal generation network and a lower-level action execution network; adopts a joint optimization method based on policy gradient to train parameters of the double-layer structure to improve the global optimal ability of the strategy in the long-term operation process; further introduces the integral gradient attribution method to perform interpretable analysis on the policy model after training, and identify key state features and key scheduling time nodes that have a significant impact on the policy output; on this basis, constructs a dynamic sub-goal interval, automatically optimizes the structure generation logic of the upper-level strategy, and obtains a structurally adaptive optimized scheduling model; finally, the optimization strategy is verified under a variety of typical operation scenarios constructed to evaluate its comprehensive performance in terms of system economy, energy storage efficiency, scheduling stability and strategy switching frequency.

[0010] The source-load-storage dynamic strategy verification method based on dual-layer reinforcement learning according to an embodiment of the present invention includes the following steps:

[0011] S1. Collect the operating data of the source-load-storage system, construct an operating data set, standardize the operating data set, and construct a standardized operating data set;

[0012] S2. Based on a standardized operational data set, a two-layer reinforcement learning model is constructed. The upper-layer policy network uses a gated recurrent unit to extract the dynamic evolution characteristics of the source, load, and storage to generate a global scheduling strategy. The lower-layer policy network uses a multi-layer perceptron to receive the global scheduling strategy and output an action decision strategy.

[0013] S3. Use the policy gradient optimization method to jointly train the two-layer reinforcement learning model and output the scheduling strategy;

[0014] S4. Introduce the integral gradient method to perform attribution analysis on the scheduling strategy, generate feature contribution vectors, and construct a set of key scheduling state nodes;

[0015] S5. Optimize the generation logic of the global scheduling strategy of the upper-layer strategy network based on the key scheduling state nodes to obtain an optimized two-layer reinforcement learning model;

[0016] S6. Construct multiple source-load-storage system operation scenarios to form a set of typical operation scenarios;

[0017] S7. Deploy the optimized two-layer reinforcement learning model in a set of typical operating scenarios, execute scheduling actions, and output policy verification results.

[0018] Optionally, the operating data set includes electricity price data, load data, energy storage charge state data, renewable energy output data and frequency data.

[0019] Optionally, the S2 specifically includes:

[0020] S21. Construct a two-layer reinforcement learning model based on the standardized operation data set. The input data of the upper policy network is the system state vector s of the source-load-storage system. t =[p t ,l t ,soc t ,r t ,f t ], where p t is the electricity price data at time t, l t is the load data at time t, soc t is the energy storage charge state at time t, r t is the renewable energy output data at time t, f t is the frequency data at time t;

[0021] S22, the first layer strategy network uses gated recurrent units to extract s1, s2, ..., s t The dynamic evolution characteristics of source, load and storage are used to generate sub-target states and build a global scheduling strategy:

[0022]

[0023] Among them, π H (g t ) is the global scheduling strategy at time t, t is the time step index, g t is the sub-goal state at time t, To select the state that maximizes the objective function among all possible states, R k is the strategic benefit obtained under the guidance of the sub-goal at time t, K is the length of time considered in the future, k is the index of the future time step rolling forward from the current time t, and γ is the discount factor;

[0024] S23, merging the global scheduling strategy with the system state vector in the standardized operation data set to form a joint vector, and inputting the joint vector into the lower-layer strategy network;

[0025] S24, the lower layer strategy network adopts a multi-layer perceptron structure, outputs the action decision strategy according to the input joint vector, and generates the local action decision strategy at time t:

[0026]

[0027] Among them, π L (a t ) is the action decision strategy at time t, a t is the scheduling action selected at time t, To select the action that maximizes the objective function among all possible actions, M is the number of future time steps considered by the lower-level strategy when making decisions, j is the index of the future time step rolling forward from the current time t, and r j is the instant reward obtained at the jth moment, λ1 is the weight coefficient of the energy storage charge state error penalty term, Δ s oc is the deviation between the target state of charge and the actual state of charge of the energy storage system, Δ l oad is the deviation between the system expected load and the actual load after scheduling.

[0028] Optionally, the optimization method of the policy gradient includes constructing an upper-layer policy network optimization target, constructing a lower-layer policy network optimization target, and joint training;

[0029] The optimization goal of the upper-layer strategy network is to maximize the expected weighted sum of the strategy benefits at all moments from the current moment to the Kth moment in the future under the action of the discount factor under the upper-layer strategy network parameters;

[0030] The optimization goal of the lower-level strategy network is to maximize the value of the weighted sum of the instantaneous rewards at all moments from the current moment to the Mth moment in the future under the parameters of the lower-level strategy network, minus the weighted square term of the energy storage state of charge deviation and the weighted square term of the load change deviation.

[0031] The joint training is to jointly train the parameters of the upper-layer policy network and the lower-layer policy network through a policy gradient update algorithm to obtain a scheduling strategy.

[0032] Optionally, the S4 specifically includes:

[0033] S41, the system state vector s in the standardized operation data set t =[p t ,l t ,soc t ,r t ,f t ] is the target input, with all zero vector s′ t =[0,0,0,0,0] is the reference input, and the state interpolation path is constructed:

[0034]

[0035] in, is the interpolation path from the reference state to the current state, t is the time step index, s′ t is an all-zero vector, s t is the system state vector, α is the interpolation scale coefficient;

[0036] S42. Perform gradient integration on the interpolation path for the output function in the upper or lower policy network to obtain the integrated gradient of the i-th input feature:

[0037]

[0038] Among them, IG i (s t ) is the i-th input feature in state s t The integral gradient value under , i is the index of the i-th input feature in the system state vector, s t,i is the value of the i-th input feature in the system state vector at the current time t, t is the time step index, s′ t,i is the i-th input feature value in the all-zero vector, is the output result of the selected strategy network in the interpolation state;

[0039] S43. The integrated gradient results of all dimensions are used to form a feature contribution vector, and the contribution of each dimensional input feature in multiple consecutive time steps is analyzed to obtain a feature contribution sequence;

[0040] S44, calculate the change rate or fluctuation degree of each feature contribution sequence, and determine the time step t by detecting the local maximum value, mutation point or the position where the change rate exceeds the preset threshold. * is a candidate key node;

[0041] S45. Combine the contribution change trends of multiple feature dimensions, set key node determination rules, screen the candidate time step set, and finally output the key scheduling state node set.

[0042] Optionally, the key node determination rules include identifying time steps with local maximum values, identifying moments when the change in feature contribution per unit time exceeds a set threshold, and identifying moments when the weighted sum of the absolute values ​​of all-dimensional contributions exceeds an overall scheduling sensitivity threshold.

[0043] Optionally, the S5 specifically includes:

[0044] S51, based on key scheduling state node set In chronological order, the time period between any two adjacent key nodes is divided into a sub-goal boundary interval to form a sub-goal boundary interval set;

[0045] S52. Extracting a subsequence of system state vectors within each sub-goal boundary interval, and generating a new global scheduling strategy using an upper-layer strategy network based on the subsequence of system state vectors;

[0046] S53, arranging all new global scheduling strategies in chronological order to form an optimized global scheduling strategy set;

[0047] S54. Under the guidance of the optimized global scheduling strategy set, the input structure of the lower-level strategy is updated to obtain an optimized two-layer reinforcement learning model.

[0048] Optionally, the input structure of updating the lower-level strategy is that within each sub-goal boundary interval, if the current moment belongs to the interval, the input of the lower-level strategy network is composed of the current system state vector and the corresponding global scheduling strategy, thereby generating the action decision strategy corresponding to the moment.

[0049] Optionally, the S6 specifically includes:

[0050] S61. Construct a source-load-storage system simulation model and set basic boundary parameters, including load curve type, energy storage system rated capacity, initial state of charge, electricity price curve model, renewable energy output model, and time resolution;

[0051] S62. Set a load disturbance function, construct a load mutation scenario by setting the mutation time, amplitude variation range, and disturbance probability, and output a set of load mutation scenarios including sudden jump load disturbance, multi-step disturbance, and high-frequency disturbance load;

[0052] S63. Set a renewable energy disturbance function, combine historical wind speed and irradiation data, superimpose artificial shading probability and instantaneous fluctuation factor, and output a set of renewable energy fluctuation scenarios including photovoltaic shading fluctuation, wind power instantaneous fluctuation, and multi-energy coordinated disturbance;

[0053] S64. Define a family of electricity price disturbance curves, construct three types of drastic change patterns: linear jump type, high-low interval reversal type, and random step disturbance type, and output a set of electricity price disturbance scenarios;

[0054] S65: Construct an energy storage operation restricted scenario, set the initial state of charge to full load or near zero, set the maximum charging power limit, the minimum discharge power lower limit, and the state switching delay constraint, and output the energy storage constraint scenario set;

[0055] S66: Combine and cross the load mutation scenario set, renewable energy fluctuation scenario set, electricity price disturbance scenario set, and energy storage limited scenario set to generate a typical operation scenario set.

[0056] Optionally, the S7 specifically includes:

[0057] S71. Deploy the optimized two-layer reinforcement learning model in the typical operation scenario set, perform a simulation of a complete scheduling cycle, and record the state trajectory, action instructions, energy storage power, electricity price data, and strategy output records in each scenario;

[0058] S72. Calculate the total system scheduling cost:

[0059]

[0060] Among them, C total is the total cost of system scheduling, T is the total duration of the scheduling cycle, t is the time step index, P grid,t is the actual power purchased from the grid at time t, p t is the market electricity price at time t, C switch,t is the scheduling switching cost at time t;

[0061] S73. Calculate the energy storage utilization efficiency based on the ratio between the charge and discharge time and the maximum power capacity of the energy storage system within the scheduling period;

[0062] S74, calculating the ratio of the number of times the upper-layer policy network outputs the global scheduling policy and the lower-layer policy network outputs the scheduling action switching to the time during the scheduling process to obtain the policy switching frequency;

[0063] S75. Record the system frequency at each moment, calculate the difference between the frequency at all moments and the average frequency, and obtain the root mean square value based on the sum of the squares of the differences to obtain the fluctuation degree of the system frequency;

[0064] S76. Use the total system scheduling cost, energy storage utilization efficiency, strategy switching frequency, and system frequency fluctuation as scheduling strategy verification indicators, and output strategy verification results.

[0065] The beneficial effects of the present invention are:

[0066] The present invention is based on a two-layer reinforcement learning structure and for the first time introduces a decision-making framework with decoupling of upper and lower layers in the construction of dynamic strategies for source-load-storage systems. Unlike traditional scheduling methods that rely on a single model for centralized strategy output, the present invention constructs a two-layer policy network with structural division of labor. The upper-layer policy network generates a global scheduling strategy with global guidance capabilities to guide the real-time action decisions of the lower-layer policy network, thereby achieving structural "global-local" coordination of the scheduling strategy and improving the stability and adaptability of the strategy in a complex system environment. At the same time, in order to break through the bottleneck of weak upper and lower layer feedback links and reliance on manual experience in the setting of sub-goal boundaries in traditional reinforcement learning methods, the present invention innovatively introduces an interpretable attribution analysis method based on integral gradients. By evaluating the feature contribution of the strategy output results, identifying key state variables and key time nodes, and reversely guiding the adjustment of the sub-goal structure from the perspective of system response, the dynamic reconstruction and structural adaptive optimization of the sub-goal generation mechanism are achieved.

[0067] At the policy optimization level, the present invention constructs a joint optimization objective function for a two-layer structure, in which the upper-layer objective function is centered on the global scheduling strategy and focuses on maximizing the long-term scheduling return, while the lower-layer objective function introduces the constraint expression of the energy storage charge state deviation and the load change deviation on the basis of immediate reward optimization, thereby taking into account the system stability and economy in the local response. By jointly updating the upper and lower layer network parameters through the policy gradient method, the present invention achieves the optimal approximation of the global performance of the entire policy cycle, and effectively solves the problems of inconsistent upper and lower layer optimization objectives and global training difficulties in existing scheduling strategies. In addition, the present invention uses the attribution analysis results to construct a set of key scheduling state nodes, reconstructs the sub-target action interval based on the time distance between nodes, and adjusts the policy input structure and generation logic accordingly, and finally forms an optimized two-layer policy model with structural self-adaptation and feedback enhancement.

[0068] In terms of strategy validation, the present invention constructed a set of operating scenarios encompassing typical operating conditions, including sudden load changes, dramatic fluctuations in renewable power output, electricity price fluctuations, and limited energy storage. The optimized scheduling model was deployed in various scenarios for simulation validation, forming a strategy evaluation system based on indicators such as system scheduling cost, energy storage utilization efficiency, strategy switching frequency, and system frequency fluctuation. Verification results demonstrate that the proposed scheduling model not only excels in economic indicators but also exhibits significant strategy stability and generalization capabilities, maintaining consistent performance across multiple scenarios.

[0069] Overall, this invention bridges the entire technical chain for intelligent source-load-storage scheduling, from strategy generation, structural optimization, attribution explanation, to simulation verification. This, for the first time, realizes a closed-loop mechanism for scheduling strategies: "structure-behavior-feedback-verification." Compared to existing approaches, this invention offers multiple advantages, including a clear structural hierarchy, traceable feedback loops, explainable scheduling strategies, controllable optimization processes, and verifiable deployment and application. It provides new ideas and an engineering foundation for building high-performance, highly adaptable, and explainable scheduling agents for future power systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0071] Figure 1 This is a flow chart of the source-load-storage dynamic strategy verification method based on double-layer reinforcement learning proposed by the present invention;

[0072] Figure 2 This is a schematic diagram of the structure of the double-layer reinforcement learning model in the source-load-storage dynamic strategy verification method based on double-layer reinforcement learning proposed in the present invention;

[0073] Figure 3This is a diagram of the characteristic contribution calculation process of the integral gradient in the source-load-storage dynamic strategy verification method based on double-layer reinforcement learning proposed in this invention. DETAILED DESCRIPTION

[0074] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0075] refer to Figure 1-3 The source-load-storage dynamic strategy verification method based on two-layer reinforcement learning includes the following steps:

[0076] S1. Collect the operating data of the source-load-storage system, construct an operating data set, standardize the operating data set, and construct a standardized operating data set;

[0077] S2. Based on a standardized operational data set, a two-layer reinforcement learning model is constructed. The upper-layer policy network uses a gated recurrent unit to extract the dynamic evolution characteristics of the source, load, and storage to generate a global scheduling strategy. The lower-layer policy network uses a multi-layer perceptron to receive the global scheduling strategy and output an action decision strategy.

[0078] S3. Use the policy gradient optimization method to jointly train the two-layer reinforcement learning model and output the scheduling strategy;

[0079] S4. Introduce the integral gradient method to perform attribution analysis on the scheduling strategy, generate feature contribution vectors, and construct a set of key scheduling state nodes;

[0080] S5. Optimize the generation logic of the global scheduling strategy of the upper-layer strategy network based on the key scheduling state nodes to obtain an optimized two-layer reinforcement learning model;

[0081] S6. Construct multiple source-load-storage system operation scenarios to form a set of typical operation scenarios;

[0082] S7. Deploy the optimized two-layer reinforcement learning model in a set of typical operating scenarios, execute scheduling actions, and output policy verification results.

[0083] The present invention proposes a source-load-storage dynamic strategy verification method based on two-layer reinforcement learning, and constructs a complete closed-loop process from data collection, model construction, strategy training, feature attribution, structure optimization to multi-scenario verification. In this method, a two-layer strategy structure is used to implement hierarchical modeling of the global scheduling strategy, an integral gradient attribution mechanism is introduced to identify key scheduling nodes, and the sub-goal generation logic is optimized accordingly to obtain a reinforcement learning model with structural self-adaptation capabilities. Compared with existing methods, the present invention significantly improves the interpretability, stability, and generalization ability of the scheduling strategy, is suitable for intelligent scheduling systems in complex scenarios, and has good engineering deployment prospects and practical application value.

[0084] In this embodiment, the operating data set includes electricity price data, load data, energy storage charge state data, renewable energy output data and frequency data.

[0085] By carefully defining the source-load-storage system operational dataset, this invention ensures that the input data covers key scheduling elements such as electricity price, load, energy storage state of charge, renewable energy output, and system frequency. This data provides excellent temporal integrity and state representation capabilities, providing a high-quality data foundation for strategic model training and attribution. After standardization, this dataset is input into a two-layer reinforcement learning model. The upper layer extracts global evolutionary features, while the lower layer performs fine-grained action generation. This effectively improves the model's ability to perceive the state of multivariable coupled systems and its adaptability to strategic decision-making, enhancing the model's generalization and stability in complex operating environments.

[0086] In this embodiment, S2 specifically includes:

[0087] S21. Construct a two-layer reinforcement learning model based on the standardized operation data set. The input data of the upper policy network is the system state vector s of the source-load-storage system. t =[p t ,l t ,soc t ,r t ,f t ], where p t is the electricity price data at time t, l t is the load data at time t, soc t is the energy storage charge state at time t, r t is the renewable energy output data at time t, f t is the frequency data at time t;

[0088] S22, the first layer strategy network uses gated recurrent units to extract s1, s2, ..., s t The dynamic evolution characteristics of source, load and storage are used to generate sub-target states and build a global scheduling strategy:

[0089]

[0090] Among them, π H (g t ) is the global scheduling strategy at time t, t is the time step index, g t is the sub-goal state at time t, To select the state that maximizes the objective function among all possible states, R kis the strategic benefit obtained under the guidance of the sub-goal at time t, K is the length of time considered in the future, k is the index of the future time step rolling forward from the current time t, and γ is the discount factor;

[0091] S23, merging the global scheduling strategy with the system state vector in the standardized operation data set to form a joint vector, and inputting the joint vector into the lower-layer strategy network;

[0092] S24, the lower layer strategy network adopts a multi-layer perceptron structure, outputs the action decision strategy according to the input joint vector, and generates the local action decision strategy at time t:

[0093]

[0094] Among them, π L (a t ) is the action decision strategy at time t, a t is the scheduling action selected at time t, To select the action that maximizes the objective function among all possible actions, M is the number of future time steps considered by the lower-level strategy when making decisions, j is the index of the future time step rolling forward from the current time t, and r j is the instant reward obtained at the jth moment, λ1 is the weight coefficient of the energy storage charge state error penalty term, Δ s oc is the deviation between the target state of charge and the actual state of charge of the energy storage system, Δ l oad is the deviation between the system expected load and the actual load after scheduling.

[0095] By constructing a two-layer reinforcement learning model based on the system state vector, the present invention realizes the structural decoupling and collaborative optimization between the upper-layer global scheduling strategy and the lower-layer local action execution. The upper-layer strategy network inputs key variables such as electricity price, load, state of charge, output and frequency, and uses gated cyclic units to extract the dynamic evolution characteristics of source, load and storage, generate sub-target states with time-guided significance, and then construct a global scheduling strategy. At the lower layer, this strategy is input into the multi-layer perceptron network in conjunction with the current system state, outputs local scheduling actions, and realizes the unification of local response and global goals. This structure significantly enhances the model's target stratification capability, strategy transmission efficiency and decision-making accuracy, and provides support for improving the full-cycle stability and global optimality of the scheduling agent.

[0096] In this embodiment, the optimization method of the policy gradient includes constructing an upper-layer policy network optimization target, constructing a lower-layer policy network optimization target, and joint training;

[0097] The optimization goal of the upper-layer strategy network is to maximize the expected weighted sum of the strategy benefits at all moments from the current moment to the Kth moment in the future under the action of the discount factor under the upper-layer strategy network parameters;

[0098] The optimization goal of the lower-level strategy network is to maximize the value of the weighted sum of the instantaneous rewards at all moments from the current moment to the Mth moment in the future under the parameters of the lower-level strategy network, minus the weighted square term of the energy storage state of charge deviation and the weighted square term of the load change deviation.

[0099] The joint training is to jointly train the parameters of the upper-layer policy network and the lower-layer policy network through a policy gradient update algorithm to obtain a scheduling strategy.

[0100] By constructing a two-layer policy network optimization target and introducing a policy gradient joint training mechanism, the present invention achieves the coordinated unification of global scheduling strategy and local action execution at the optimization target level. The upper-layer policy network aims to maximize long-term policy benefits and guides the generation direction of sub-target states through a discount weighted method; the lower-layer policy network introduces penalty constraints on energy storage charge state and load changes on the basis of optimizing immediate rewards, effectively balancing scheduling economy and operational stability. The policy gradient algorithm is used in the joint training stage to synchronously update the upper and lower layer parameters, so that the strategy maintains a consistent optimization direction on different time scales. This method effectively solves the problems of upper and lower layer target separation and feedback incoordination in traditional policy optimization, and significantly improves the practicality and scheduling effect of reinforcement learning models in complex power systems.

[0101] In this embodiment, the S4 specifically includes:

[0102] S41, the system state vector s in the standardized operation data set t =[p t ,l t ,soc t ,r t ,f t ] is the target input, with all zero vector s′ t =[0,0,0,0,0] is the reference input, and the state interpolation path is constructed:

[0103]

[0104] in, is the interpolation path from the reference state to the current state, t is the time step index, s′ t is an all-zero vector, s t is the system state vector, α is the interpolation scale coefficient;

[0105] S42. Perform gradient integration on the interpolation path for the output function in the upper or lower policy network to obtain the integrated gradient of the i-th input feature:

[0106]

[0107] Among them, IG i (s t ) is the i-th input feature in state s t The integral gradient value under , i is the index of the i-th input feature in the system state vector, s t,i is the value of the i-th input feature in the system state vector at the current time t, t is the time step index, s′ t,i is the i-th input feature value in the all-zero vector, is the output result of the selected strategy network in the interpolation state;

[0108] S43. The integrated gradient results of all dimensions are used to form a feature contribution vector, and the contribution of each dimensional input feature in multiple consecutive time steps is analyzed to obtain a feature contribution sequence;

[0109] S44, calculate the change rate or fluctuation degree of each feature contribution sequence, and determine the time step t by detecting the local maximum value, mutation point or the position where the change rate exceeds the preset threshold. * is a candidate key node;

[0110] S45. Combine the contribution change trends of multiple feature dimensions, set key node determination rules, screen the candidate time step set, and finally output the key scheduling state node set.

[0111] By introducing the integral gradient attribution mechanism, the present invention realizes the importance identification of the input features of the scheduling strategy and the automatic extraction of key time nodes. The state interpolation path construction method is adopted to map the input changes between the standardized system state and the all-zero reference state into a continuous path, and the output function of the policy network is gradient integrated on the path to obtain the integral gradient of each input feature at the current moment to form a feature contribution vector. Further combined with the changing trend of the feature contribution within continuous time steps, local extreme values, mutation points or highly sensitive time points are identified, and finally the key scheduling state nodes are screened out. This mechanism significantly enhances the interpretability and structural adaptability of the model, provides a basis for subsequent policy structure optimization, and improves the model's responsiveness and stability to system changes.

[0112] In this embodiment, the key node determination rules include identifying time steps with local maximum values, identifying moments when the change in feature contribution per unit time exceeds a set threshold, and identifying moments when the weighted sum of the absolute values ​​of all-dimensional contributions exceeds the overall scheduling sensitivity threshold.

[0113] By constructing key node identification rules that include local maximum detection, rate-of-change threshold determination, and weighted sensitivity assessment, this paper achieves multi-dimensional, precise identification of critical moments in the scheduling process. This mechanism, combining temporal fluctuations in feature contributions with full-dimensional sensitivity assessment, improves the accuracy and robustness of key node identification, providing a data-driven basis for subsequent sub-goal interval partitioning and strategy structure optimization, effectively enhancing the responsiveness and adaptability of scheduling strategies.

[0114] In this embodiment, the S5 specifically includes:

[0115] S51, based on key scheduling state node set In chronological order, the time period between any two adjacent key nodes is divided into a sub-goal boundary interval to form a sub-goal boundary interval set;

[0116] S52. Extracting a subsequence of system state vectors within each sub-goal boundary interval, and generating a new global scheduling strategy using an upper-layer strategy network based on the subsequence of system state vectors;

[0117] S53, arranging all new global scheduling strategies in chronological order to form an optimized global scheduling strategy set;

[0118] S54. Under the guidance of the optimized global scheduling strategy set, the input structure of the lower-level strategy is updated to obtain an optimized two-layer reinforcement learning model.

[0119] By dynamically dividing the scheduling timeline based on key scheduling state nodes, the present invention constructs a set of sub-goal boundary intervals, then extracts state subsequences within each interval, and regenerates the global scheduling strategy by the upper-level strategy network, thus realizing the reconstruction and refinement of the strategy structure in the time dimension. All updated global scheduling strategies are arranged in chronological order to form a strategy set with phased guidance capabilities, which is used to update the input structure of the lower-level strategy and construct an optimized two-layer reinforcement learning model. This mechanism breaks through the limitations of the fixed sub-goal division method, enabling the scheduling structure to adaptively adjust as the system state changes, significantly improving the strategy's responsiveness, structural adaptability, and overall scheduling stability in a dynamic environment.

[0120] In this embodiment, the input structure of the updated lower-level strategy is that within each sub-goal boundary interval, if the current moment belongs to this interval, the input of the lower-level strategy network is composed of the current system state vector and the corresponding global scheduling strategy, thereby generating the action decision strategy corresponding to this moment.

[0121] By dynamically updating the input structure of the lower-level policy network within each sub-goal boundary interval, the present invention achieves a precise match between the scheduling policy input and the global goal. Specifically, the lower-level policy receives the combined input of the current system state vector and the corresponding global scheduling policy at each time step, making the action generation process phase-by-phase guidance and goal adaptability. This mechanism enhances the responsiveness of the lower-level policy to the intentions of the upper-level policy, effectively improving the coherence, pertinence, and optimization efficiency of scheduling behavior at different operating stages, and significantly improving the consistency of policy execution and system stability in complex scenarios.

[0122] In this embodiment, S6 specifically includes:

[0123] S61. Construct a source-load-storage system simulation model and set basic boundary parameters, including load curve type, energy storage system rated capacity, initial state of charge, electricity price curve model, renewable energy output model, and time resolution;

[0124] S62. Set a load disturbance function, construct a load mutation scenario by setting the mutation time, amplitude variation range, and disturbance probability, and output a set of load mutation scenarios including sudden jump load disturbance, multi-step disturbance, and high-frequency disturbance load;

[0125] S63. Set a renewable energy disturbance function, combine historical wind speed and irradiation data, superimpose artificial shading probability and instantaneous fluctuation factor, and output a set of renewable energy fluctuation scenarios including photovoltaic shading fluctuation, wind power instantaneous fluctuation, and multi-energy coordinated disturbance;

[0126] S64. Define a family of electricity price disturbance curves, construct three types of drastic change patterns: linear jump type, high-low interval reversal type, and random step disturbance type, and output a set of electricity price disturbance scenarios;

[0127] S65: Construct an energy storage operation restricted scenario, set the initial state of charge to full load or near zero, set the maximum charging power limit, the minimum discharge power lower limit, and the state switching delay constraint, and output the energy storage constraint scenario set;

[0128] S66: Combine and cross the load mutation scenario set, renewable energy fluctuation scenario set, electricity price disturbance scenario set, and energy storage limited scenario set to generate a typical operation scenario set.

[0129] By constructing a multi-dimensional simulation model of the source-load-storage system and systematically setting the disturbance function, the present invention generates a set of typical operating scenarios covering sudden load changes, renewable energy fluctuations, drastic changes in electricity prices, and limited energy storage operation. Various scenarios are constructed by fusing parameter perturbations with historical data, which can simulate high-frequency disturbances and extreme operating conditions in actual operation, ensuring that the verification environment is representative and challenging. By combining and crossing different types of disturbance sets, the constructed multi-scenario system has a high coverage and structural diversity, providing a high-quality testing basis for stability verification and performance comparison of scheduling strategies in complex environments, and effectively improving the comprehensiveness, scientificity and application reliability of model evaluation.

[0130] In this embodiment, the S7 specifically includes:

[0131] S71. Deploy the optimized two-layer reinforcement learning model in the typical operation scenario set, perform a simulation of a complete scheduling cycle, and record the state trajectory, action instructions, energy storage power, electricity price data, and strategy output records in each scenario;

[0132] S72. Calculate the total system scheduling cost:

[0133]

[0134] Among them, C total is the total cost of system scheduling, T is the total duration of the scheduling cycle, t is the time step index, P grid,t is the actual power purchased from the grid at time t, p t is the market electricity price at time t, C switch,t is the scheduling switching cost at time t;

[0135] S73. Calculate the energy storage utilization efficiency based on the ratio between the charge and discharge time and the maximum power capacity of the energy storage system within the scheduling period;

[0136] S74, calculating the ratio of the number of times the upper-layer policy network outputs the global scheduling policy and the lower-layer policy network outputs the scheduling action switching to the time during the scheduling process to obtain the policy switching frequency;

[0137] S75. Record the system frequency at each moment, calculate the difference between the frequency at all moments and the average frequency, and obtain the root mean square value based on the sum of the squares of the differences to obtain the fluctuation degree of the system frequency;

[0138] S76. Use the total system scheduling cost, energy storage utilization efficiency, strategy switching frequency, and system frequency fluctuation as scheduling strategy verification indicators, and output strategy verification results.

[0139] By deploying the optimized two-layer reinforcement learning model in a set of typical operating scenarios, the present invention realizes the systematic verification of scheduling strategies under complex and variable working conditions. During the simulation process, the state trajectory, action execution, energy storage operation and strategy output are recorded in real time to ensure that the evaluation data fully covers the entire scheduling process. Based on multi-dimensional indicators such as system scheduling cost, energy storage utilization efficiency, strategy switching frequency and frequency fluctuation, the present invention constructs a scientific and complete performance evaluation system that can quantify the comprehensive performance of the model in terms of economy, stability and execution efficiency, significantly improving the accuracy, pertinence and credibility of scheduling strategy evaluation, and providing data support for strategy iteration and actual deployment.

[0140] Example 1:

[0141] To verify the feasibility of this invention, we applied it to a deployed integrated energy dispatch platform with integrated sources, loads, and storage. The proposed "two-layer reinforcement learning source-load-storage dynamic strategy verification method based on integrated gradient perception" was deployed and validated over a typical operating period from July 1, 2024, to verify its effectiveness. This system encompasses multiple types of electricity loads, renewable energy sources such as photovoltaic and wind power, and a centralized energy storage system with a total capacity of 6MW / 12MWh. It exhibits rich operational data and significant fluctuations.

[0142] In this scenario, the system's daily load peak-to-valley difference exceeds 30%, with significant load fluctuations between midday and evening. PV output is concentrated during the day and fluctuates frequently, while wind power output fluctuates frequently due to wind speed. The hourly adjustment of the floating electricity price mechanism, combined with demand response triggering rules, leads to significant price fluctuations. Traditional dispatch methods, using fixed structure strategies or rolling optimization algorithms, suffer from issues such as an inability to respond promptly to multi-source disturbances, frequent energy storage charging and discharging, and unstable system operation. In particular, in high-frequency disturbance scenarios, strategy failure rates are high, energy utilization efficiency is low, and dispatch costs increase significantly.

[0143] During this operation cycle, the present invention was deployed on the platform system and compared with the traditional rolling optimization scheduling method. The specific implementation process includes: first, collecting five types of operating data in the system, such as electricity price, load, energy storage charge state, renewable output and frequency, to construct a standardized operation data set. Subsequently, a two-layer reinforcement learning model is constructed based on the data set. The upper layer uses a gated recurrent unit to extract the dynamic evolution characteristics of the source, load and storage, and generate a time-series global scheduling strategy; the lower layer policy network receives the joint input of the current state and the global scheduling strategy, and outputs a specific action strategy for controlling the charging and discharging of energy storage and the purchased power. The model completes joint training through the policy gradient algorithm, and after the policy training is completed, the integral gradient method is applied to perform attribution analysis, identify the key state nodes in the scheduling process, and construct a two-layer model with optimized structure. Finally, the strategy is verified in the constructed typical disturbance scenarios, including 10 representative operating situations such as load mutation, electricity price jump, and wind and solar output fluctuation.

[0144] Verification results show that the present invention demonstrates significant improvements over traditional methods across multiple key performance indicators. In terms of total system dispatch costs, the cumulative electricity procurement cost of the present invention's strategy during the test period was 71,345 yuan, an 11.2% reduction compared to traditional methods. In terms of energy storage efficiency, the ratio of effective energy storage utilization time to the total dispatch period increased to 85.4%, higher than the 68.1% of traditional methods. In terms of strategy switching frequency, due to the high stability of the sub-goal-guided structured scheduling strategy, the strategy adjustment frequency dropped to 0.025 times / minute (compared to 0.048 times / minute with the original strategy). The root mean square value of system frequency fluctuations decreased by 18.6%, demonstrating the scheduling strategy's excellent ability to suppress high-frequency disturbances.

[0145] The experimental results verify the significant advantages of the method of the present invention in improving system stability, reducing scheduling costs, and increasing the utilization efficiency of energy storage systems. In particular, it has stronger robustness and engineering adaptability in dealing with multi-source disturbances, high-frequency changes, and typical extreme operating conditions, providing an effective solution for strategy deployment and scheduling optimization in actual energy management platforms.

[0146] Table 1 Performance comparison data of the present invention and traditional scheduling method in a typical cycle

[0147]

[0148] The comparative data in Table 1 above demonstrates that, in a typical source-load-storage scheduling scenario, the present invention significantly outperforms traditional rolling optimization methods across multiple key performance indicators, validating its practicality and optimization value under complex operating conditions. Specifically, regarding the total system scheduling cost, the present invention controls the power procurement and strategy execution costs for the entire scheduling cycle to 71,345 yuan, while the traditional method costs as much as 80,318 yuan, a cost reduction of 11.2%. This effectively demonstrates the advantages of reinforcement learning strategies in optimizing long-term returns and market responsiveness.

[0149] In terms of energy storage system efficiency, the dual-layer structure strategy proposed in this invention effectively reduces the idle rate of energy storage resources, increasing the proportion of energy storage charge and discharge time to 85.4%, far exceeding the 68.1% of traditional methods. This not only optimizes energy storage system utilization, but also extends the energy storage lifespan and reduces asset operating pressure, demonstrating the invention's enhanced dynamic energy regulation capabilities.

[0150] In terms of policy execution stability, this invention significantly reduces the policy switching frequency to 0.025 times / minute by introducing a sub-goal state structure, a 47.9% reduction compared to the 0.048 times / minute of the traditional policy. This reduction in policy switching frequency enhances the continuity of control instructions and the stability of system behavior, reducing execution jitter and system response lag caused by frequent actions.

[0151] The system also demonstrates significant advantages in operational safety and stability. By calculating the root mean square fluctuation of the system frequency during the dispatch process, the proposed solution achieved a value of 0.036 Hz, an 18.6% decrease from the 0.044 Hz of the traditional solution. This demonstrates its ability to more effectively smooth disturbances caused by variations in renewable output and load, maintaining stable frequency operation.

[0152] In addition, the present invention demonstrates higher robustness and response speed in highly dynamic scenarios. For example, during periods of significant electricity price fluctuations, the present invention can maintain a 92.1% SOC stability rate, which is higher than the 76.5% of the traditional solution. In the event of a sudden load change, the average response delay of the present invention's strategy is 3.2 seconds, which is nearly half the 6.1 seconds of the traditional method, significantly improving the dispatch response speed. During periods of drastic fluctuations in wind and solar power output, the accuracy of the present invention's action output reached 89.7%, significantly better than the 74.2% of the traditional strategy, fully demonstrating the strategy's adaptability and generalization effect in highly uncertain scenarios.

[0153] In summary, this invention achieves comprehensive improvements in the dispatch system's economy, efficiency, stability, and robustness through hierarchical modeling of policy structures, attribution-driven structural optimization, and an integrated gradient feedback mechanism. This solution not only effectively reduces overall operating costs but also enhances the energy dispatch system's intelligent response capabilities to multi-source disturbances. This provides a practical technical path and engineering value support for policy deployment and dynamic optimization in smart power systems.

[0154] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A source-load-storage dynamic strategy verification method based on two-layer reinforcement learning, characterized by: The steps include: S1. Collect the operating data of the source-load-storage system, construct an operating data set, standardize the operating data set, and construct a standardized operating data set; S2. Based on a standardized operational data set, a two-layer reinforcement learning model is constructed. The upper-layer policy network uses a gated recurrent unit to extract the dynamic evolution characteristics of the source, load, and storage to generate a global scheduling strategy. The lower-layer policy network uses a multi-layer perceptron to receive the global scheduling strategy and output an action decision strategy. S3. Use the policy gradient optimization method to jointly train the two-layer reinforcement learning model and output the scheduling strategy; S4. Introduce the integral gradient method to perform attribution analysis on the scheduling strategy, generate feature contribution vectors, and construct a set of key scheduling state nodes; S5. Optimize the generation logic of the global scheduling strategy of the upper-layer strategy network based on the key scheduling state nodes to obtain an optimized two-layer reinforcement learning model; S6. Construct multiple source-load-storage system operation scenarios to form a set of typical operation scenarios; S7. Deploy the optimized two-layer reinforcement learning model in a set of typical operating scenarios, execute scheduling actions, and output policy verification results.

2. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The operating data set includes electricity price data, load data, energy storage charge state data, renewable energy output data and frequency data.

3. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The S2 specifically includes: S21. Construct a two-layer reinforcement learning model based on the standardized operation data set. The input data of the upper policy network is the system state vector s of the source-load-storage system. t =[p t ,l t ,soc t ,r t ,f t ], where p t is the electricity price data at time t, l t is the load data at time t, soc t is the energy storage charge state at time t, r t is the renewable energy output data at time t, f t is the frequency data at time t; S22, the first layer strategy network uses gated recurrent units to extract s1, s2, ..., s t The dynamic evolution characteristics of source, load and storage are used to generate sub-target states and build a global scheduling strategy: Among them, π H (g t ) is the global scheduling strategy at time t, t is the time step index, g t is the sub-goal state at time t, To select the state that maximizes the objective function among all possible states, R k is the strategic benefit obtained under the guidance of the sub-goal at time t, K is the length of time considered in the future, k is the index of the future time step rolling forward from the current time t, and γ is the discount factor; S23, merging the global scheduling strategy with the system state vector in the standardized operation data set to form a joint vector, and inputting the joint vector into the lower-layer strategy network; S24, the lower layer strategy network adopts a multi-layer perceptron structure, outputs the action decision strategy according to the input joint vector, and generates the local action decision strategy at time t: Among them, π L (a t ) is the action decision strategy at time t, a t is the scheduling action selected at time t, To select the action that maximizes the objective function among all possible actions, M is the number of future time steps considered by the lower-level strategy when making decisions, j is the index of the future time step rolling forward from the current time t, r j is the instant reward obtained at the jth moment, λ1 is the weight coefficient of the energy storage charge state error penalty term, Δ s oc is the deviation between the target state of charge and the actual state of charge of the energy storage system, Δ l oad is the deviation between the system expected load and the actual load after scheduling.

4. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The optimization method of the policy gradient includes constructing the optimization target of the upper policy network, constructing the optimization target of the lower policy network and joint training; The optimization goal of the upper-layer strategy network is to maximize the expected weighted sum of the strategy benefits at all moments from the current moment to the Kth moment in the future under the action of the discount factor under the upper-layer strategy network parameters; The optimization goal of the lower-level strategy network is to maximize the value of the weighted sum of the instantaneous rewards at all moments from the current moment to the Mth moment in the future under the parameters of the lower-level strategy network, minus the weighted square term of the energy storage state of charge deviation and the weighted square term of the load change deviation. The joint training is to jointly train the parameters of the upper-layer policy network and the lower-layer policy network through a policy gradient update algorithm to obtain a scheduling strategy.

5. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The S4 specifically includes: S41, the system state vector s in the standardized operation data set t =[p t ,l t ,soc t ,r t ,f t ] is the target input, with all zero vector s′ t =[0,0,0,0,0] is the reference input, and the state interpolation path is constructed: in, is the interpolation path from the reference state to the current state, t is the time step index, s′ t is an all-zero vector, s t is the system state vector, α is the interpolation coefficient; S42. Perform gradient integration on the interpolation path for the output function in the upper or lower policy network to obtain the integrated gradient of the i-th input feature: Among them, IG i (s t ) is the i-th input feature in state s t The integral gradient value under , i is the index of the i-th input feature in the system state vector, s t,i is the value of the i-th input feature in the system state vector at the current time t, t is the time step index, s′ t,i is the i-th input feature value in the all-zero vector, is the output result of the selected strategy network in the interpolation state; S43. The integrated gradient results of all dimensions are used to form a feature contribution vector, and the contribution of each dimensional input feature in multiple consecutive time steps is analyzed to obtain a feature contribution sequence; S44, calculate the change rate or fluctuation degree of each feature contribution sequence, and determine the time step t by detecting the local maximum value, mutation point or the position where the change rate exceeds the preset threshold. * is a candidate key node; S45. Combine the contribution change trends of multiple feature dimensions, set key node determination rules, screen the candidate time step set, and finally output the key scheduling state node set.

6. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 5 is characterized in that: The key node determination rules include identifying the time step with the local maximum value, identifying the moment when the change of the feature contribution degree per unit time exceeds the set threshold, and identifying the moment when the weighted sum of the absolute value of the full-dimensional contribution degree exceeds the overall scheduling sensitivity threshold.

7. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The S5 specifically includes: S51, based on key scheduling state node set In chronological order, the time period between any two adjacent key nodes is divided into a sub-goal boundary interval to form a sub-goal boundary interval set; S52. Extracting a subsequence of system state vectors within each sub-goal boundary interval, and generating a new global scheduling strategy using an upper-layer strategy network based on the subsequence of system state vectors; S53, arranging all new global scheduling strategies in chronological order to form an optimized global scheduling strategy set; S54. Under the guidance of the optimized global scheduling strategy set, the input structure of the lower-level strategy is updated to obtain an optimized two-layer reinforcement learning model.

8. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 7 is characterized in that: The input structure of the updated lower-level strategy is that within each sub-goal boundary interval, if the current moment belongs to this interval, the input of the lower-level strategy network is composed of the current system state vector and the corresponding global scheduling strategy, thereby generating the action decision strategy corresponding to the moment.

9. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The S6 specifically includes: S61. Construct a source-load-storage system simulation model and set basic boundary parameters, including load curve type, energy storage system rated capacity, initial state of charge, electricity price curve model, renewable energy output model, and time resolution; S62. Set a load disturbance function, construct a load mutation scenario by setting the mutation time, amplitude variation range, and disturbance probability, and output a set of load mutation scenarios including sudden jump load disturbance, multi-step disturbance, and high-frequency disturbance load; S63. Set a renewable energy disturbance function, combine historical wind speed and irradiation data, superimpose artificial shading probability and instantaneous fluctuation factor, and output a set of renewable energy fluctuation scenarios including photovoltaic shading fluctuation, wind power instantaneous fluctuation, and multi-energy coordinated disturbance; S64. Define a family of electricity price disturbance curves, construct three types of drastic change patterns: linear jump type, high-low interval reversal type, and random step disturbance type, and output a set of electricity price disturbance scenarios; S65: Construct an energy storage operation restricted scenario, set the initial state of charge to full load or near zero, set the maximum charging power limit, the minimum discharge power lower limit, and the state switching delay constraint, and output the energy storage constraint scenario set; S66: Combine and cross the load mutation scenario set, renewable energy fluctuation scenario set, electricity price disturbance scenario set, and energy storage limited scenario set to generate a typical operation scenario set.

10. The source-load-storage dynamic strategy verification method based on double-layer reinforcement learning according to claim 1 is characterized in that: The S7 specifically includes: S71. Deploy the optimized two-layer reinforcement learning model in the typical operation scenario set, perform a simulation of a complete scheduling cycle, and record the state trajectory, action instructions, energy storage power, electricity price data, and strategy output records in each scenario; S72. Calculate the total system scheduling cost: Among them, C total is the total cost of system scheduling, T is the total duration of the scheduling cycle, t is the time step index, P grid,t is the actual power purchased from the grid at time t, p t is the market electricity price at time t, C switch,t is the scheduling switching cost at time t; S73. Calculate the energy storage utilization efficiency based on the ratio between the charge and discharge time and the maximum power capacity of the energy storage system within the scheduling period; S74, calculating the ratio of the number of times the upper-layer strategy network output sub-goal state and the lower-layer strategy network output scheduling action switch occurs to the time during the scheduling process to obtain the strategy switching frequency; S75. Record the system frequency at each moment, calculate the difference between the frequency at all moments and the average frequency, and obtain the root mean square value based on the sum of the squares of the differences to obtain the fluctuation degree of the system frequency; S76. Use the total system scheduling cost, energy storage utilization efficiency, strategy switching frequency, and system frequency fluctuation as scheduling strategy verification indicators, and output strategy verification results.

Images