Power system load dynamic optimization method based on reinforcement learning

Abstract

The application discloses a power system load dynamic optimization method based on reinforcement learning, comprising the following steps: S1, collecting power system data to construct a state space; S2, constructing a hierarchical reinforcement learning model based on the state space, and dividing high-level decision and low-level execution tasks; S3, training a high-level decision model to output scheduling task target category instruction with high-level state; S4, training a low-level execution model, and outputting control actions in combination with the current node state and high-level instruction; S5, introducing an evolution mechanism to generate a strategy population and optimize the low-level execution model; S6, fusing the evolution mechanism and strategy gradient to synchronously optimize good individuals; S7, deploying the trained model to a power dispatching system; S8, fine-tuning model parameters based on scheduling feedback; and S9, continuously applying the fine-tuned model to load scheduling control. The application realizes accurate power load scheduling and efficient adaptive optimization of strategies, and improves system responsiveness and operation stability.

Description

Technical Field

[0001] The present invention relates to the technical field of power load optimization, and in particular to a method for dynamic optimization of power system load based on reinforcement learning. Background Art

[0002] With the increasing penetration of renewable energy in power systems and the increasing complexity of user-side electricity consumption, traditional static load scheduling methods are struggling to cope with increasingly dynamic and volatile operating environments. In particular, in scenarios such as demand response, energy storage control, and load migration, scheduling tasks face challenges such as high-dimensional state spaces, continuous action outputs, and system uncertainty. To address these challenges, researchers have widely introduced reinforcement learning methods to optimize power system load scheduling. Among them, policy gradient algorithms have attracted attention for their performance in continuous control problems. Representative methods include Advantage Actor-Critic (A2C), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC).

[0003] Most existing load optimization methods employ a single-layer reinforcement learning architecture, directly mapping system states to dispatch actions within a unified policy model. These methods often exhibit slow convergence, prone to local optimality, and unclear control granularity when applied to multi-level, multi-objective grid dispatch scenarios. Furthermore, some research has attempted to improve model performance through policy gradient methods, but these approaches are often limited to standard forms of A2C or PPO, and still suffer from poor generalization and instability when dealing with complex power system state structures and high-dimensional decision actions.

[0004] In the study of evolutionary methods for power load optimization, traditional genetic algorithms have been used to optimize control strategies. However, most methods rely solely on fixed fitness evaluation and parameter mutation, lacking deep integration with the policy network structure and struggling to capture structural differences or dynamic characteristics between individuals. Furthermore, current reinforcement learning methods often overlook the adaptive limitations of policy models in terms of transfer and generalization capabilities, making rapid deployment particularly challenging in cross-regional power grids or fluctuating load environments.

[0005] Based on this, existing technologies still face the following problems: First, reinforcement learning models fail to effectively combine multi-layer structures and task division, lacking a highly controllable and responsive dispatch process; second, evolutionary optimization and policy gradient optimization paths do not form a synergistic mechanism, resulting in information fragmentation and isolated optimization processes; third, while policy gradient methods such as SAC are robust, their specific structures in power scenarios have not been improved to address state clustering, key node perception, and policy distribution control, limiting their performance ceiling. Therefore, a power system load dynamic optimization solution that integrates hierarchical modeling, evolutionary optimization, and an improved policy gradient method is urgently needed to improve dispatch efficiency, model stability, and adaptability.

[0006] Therefore, how to provide a dynamic optimization method for power system load based on reinforcement learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0007] One objective of the present invention is to propose a method for dynamic optimization of power system loads based on reinforcement learning. The present invention integrates hierarchical reinforcement learning, evolutionary optimization algorithm and improved soft actor-critic policy gradient method, describes in detail the hierarchical decision-making structure in the power system from high-level dispatching strategy generation to low-level control action output, and introduces a structural optimization mechanism to enhance the policy network. It has the advantages of high control accuracy, fast policy convergence and strong system adaptability.

[0008] A method for dynamic optimization of power system load based on reinforcement learning according to an embodiment of the present invention includes the following steps:

[0009] S1, collect data of the power system and construct the state space;

[0010] S2. Construct a hierarchical reinforcement learning model based on the state space, and divide the hierarchical reinforcement learning model into a high-level decision model and a low-level execution model;

[0011] S3. Training a high-level decision model. The high-level decision model takes the high-level state in the state space as input and outputs the scheduling task target category instruction;

[0012] S4. Training a low-level execution model. The low-level execution model takes the current node-level state and the scheduling task target category instructions output by the high-level decision model as input, and outputs a control action sequence including the load migration amount of each node, energy storage charging and discharging instructions, and user response incentive coefficients.

[0013] S5. When training the low-level execution model, an evolutionary reinforcement learning mechanism is introduced to construct several strategic individuals. The reward value of each strategic individual is calculated through interactive simulation of the environment. An improved genetic algorithm is used to perform selection, crossover, and mutation operations on the population of strategic individuals to obtain a new generation of strategic individuals.

[0014] S6. Combining the evolutionary reinforcement learning mechanism with the policy gradient optimization method, while the individual policies are evolving and updating, the policy gradient-based optimization method is used to further update the parameters of the policy individuals with better performance;

[0015] S7. Deploy the hierarchical reinforcement learning model to the power system load dispatch module;

[0016] S8. During deployment and operation, continuously collect scheduling feedback data and fine-tune the parameters of the deployed hierarchical reinforcement learning model;

[0017] S9. Continue to apply the migrated and fine-tuned high-level decision-making model and low-level execution model to the load scheduling process of the power system.

[0018] Optionally, the S2 specifically includes:

[0019] S21. Divide the state space into a high-level state space and a low-level state space, wherein the high-level state space includes the global load trend of the power system, the range of electricity price fluctuations, and renewable energy output forecast data, and the low-level state space includes the real-time load value of each node, energy storage status information, and controllable load regulation capability parameters;

[0020] S22. Constructing a high-level decision model based on the high-level state space, wherein the high-level decision model uses the high-level state as input and outputs a scheduling task target category instruction, wherein the scheduling task target category instruction includes a peak shaving and valley filling scheduling instruction, an energy storage priority regulation instruction, or a demand response control instruction;

[0021] S23: Constructing a low-level execution model based on the low-level state space and the dispatch task target category instructions output by the high-level decision model, wherein the low-level execution model outputs a control action sequence including the load migration amount, energy storage charge and discharge control instructions, and user response incentive coefficients of each grid node;

[0022] S24. Establish a mapping relationship between the high-level decision model and the low-level execution model, so that the scheduling task target category instruction serves as an input condition of the low-level execution model, forming a hierarchical linkage scheduling control structure.

[0023] Optionally, the S3 specifically includes:

[0024] S31, obtaining a high-level state space as input data of a training sample;

[0025] S32. Build high-level decision-making strategy functions ,in, represents the high-level decision-making strategy function, is the high-level state input of the input, The output scheduling task target category instruction;

[0026] S33. Define the immediate reward function of the high-level decision model ,in, Represents the time step, evaluating the output of the high-level decision strategy function The performance of the guided low-level execution results in terms of load balancing, operating cost control and scheduling volatility;

[0027] S34, using policy gradient-based reinforcement learning method to improve the high-level decision-making policy function Training is performed by maximizing the immediate reward function The expected value of the high-level decision model is updated, and the high-level state of the input is Output the optimal scheduling task target category instruction .

[0028] Optionally, the policy gradient-based reinforcement learning method in S34 is a Soft Actor-Critic method based on maximum entropy policy optimization, specifically including:

[0029] S341. Construct two high-level Q-value function networks and , in a high-level state The corresponding output scheduling task target category instruction As input, output Q value estimation and , use the smaller value to participate in strategy optimization;

[0030] S342, introduce the objective function containing the policy entropy term as the training high-level decision-making policy function The optimization goal is to guide the optimization of the given high-level state The parameter update process under ;

[0031] S343, high-level decision strategy function and high-level Q-value function network 、 Alternating updates are performed with the following structural improvements: a residual connection structure is added to the high-level decision strategy function, a shared state encoder is used in the Q-value function network, and a dynamic entropy coefficient adjustment mechanism based on the target Q-network fluctuation amplitude is introduced to replace the static or predefined entropy coefficient update method.

[0032] Optionally, the S4 specifically includes:

[0033] S41. Low-level state input is defined as , including the current load value of each node in the power grid, the current state of charge of the energy storage device corresponding to the node, and the adjustable load capacity:

[0034] S42, the scheduling task target category instruction output by the high-level decision model With low-level state input The combination is used as the joint input of the low-level execution model to construct the low-level policy function;

[0035] S43. Constructing low-level immediate reward functions , evaluate the input in the low-level state and the scheduling task target category instructions output by the high-level decision model Lower level control actions the resulting system performance;

[0036] S44, using the low-level immediate reward function To optimize the goal, train the low-level policy function , given the low-level state input and the scheduling task target category instructions output by the high-level decision model Output the optimal control action under the condition .

[0037] Optionally, the S5 specifically includes:

[0038] S51, initialize the low-level strategy function, build Low-level policy function individuals , each low-level policy function individual is a set of independent low-level policy network parameters, corresponding to the control action , For the Individual control actions;

[0039] S52, record each strategy individual in the continuous The cumulative reward value in the training cycle , calculate the comprehensive fitness value of the strategy individual with historical smoothing weight;

[0040] S53. In the screening operation of the genetic algorithm, the selection method of the elite retention strategy is adopted, giving priority to retaining the top The strategy individual with the largest comprehensive fitness value is selected, and a roulette wheel selection is performed based on the ranking from the remaining individuals to generate a candidate parent set;

[0041] S54. Improve the offspring crossover operation to determine the similarity of the performance of two parent strategy individuals in the strategy parameter distribution space. If the Euclidean distance or KL divergence is less than the set threshold, crossover is allowed; otherwise, the combination is discarded.

[0042] S55, perform mutation operation on the selected offspring parameter set, introduce target guidance mechanism, and calculate mutation probability based on the current target residual of the strategy individual ;

[0043] S56. Take the new generation of strategy individuals after screening, crossover and target-guided mutation as the current strategy population, repeat S52 to S55, iterate and update until convergence, and select strategy individuals as the output strategy structure of the low-level execution model according to the final comprehensive fitness value from high to low.

[0044] Optionally, the S6 specifically includes:

[0045] S61. After each generation of evolutionary operations, select the fitness value from the current strategy population. strategic individuals As the candidate subset for policy gradient optimization, Indicates the A low-level strategy individual, ;

[0046] S62. For each individual strategy , construct a set of parameter-sharing dual Q-value function networks 、 , input in low-level state and the scheduling task target category instructions output by the high-level decision model And corresponding control actions For input;

[0047] S63. Construct low-level strategy objective function ;

[0048] S64, will As the optimization target, update the low-level strategy individuals separately And the corresponding Q value function network The parameters of the Q network are updated using an asynchronous alternating training method, and the target Q network parameters are adjusted based on the soft update rule;

[0049] S65. While updating the evolution of individual strategies, the optimization method based on policy gradient is used to further update the parameters of the strategy individuals with better performance. Among them, the optimization method based on policy gradient is the improved SoftActor-Critic method. The SoftActor-Critic method makes the following structural improvements: introduce a feature gating mechanism based on the input state distribution clustering label in the low-level dual Q-value function network, adjust the Q-value channel response strength, introduce a collaborative attention mechanism in the strategy network, adopt a dynamic target KL divergence adjustment strategy in the entropy coefficient update process, and adaptively adjust the strategy distribution offset amplitude during the training process. ;

[0050] S66, using the strategy individual trained jointly by the genetic algorithm and the improved SoftActor-Critic method as the low-level execution model target output strategy.

[0051] Optionally, the structural improvement of the low-level SoftActor-Critic method in S65 specifically includes:

[0052] S651, in the low-level dual Q-value function network and The state distribution gating mechanism is introduced to input the low-level state Perform clustering to obtain state category labels , label the state category As the input embedded in the gating module, in the low-level double Q value function network feature extraction layer, according to Activate the corresponding channel control factor and perform gate adjustment on the intermediate features to generate category-sensitive Q-value estimation;

[0053] S652, at the lower level strategy individuals The cross-node state collaborative attention mechanism is introduced to form the low-level state The state features of each node are used as attention encoding input, and the attention weight distribution based on the dependency relationship between nodes is constructed. The features of each node are weighted and aggregated as the input of the strategy distribution;

[0054] S653, Entropy coefficient in low-level strategy network Introduce a dynamic adjustment mechanism to set the target strategy distribution offset value , calculate the current low-level strategy individual in each round of training Compared with the low-level strategy individuals in the previous round KL divergence of ,based on and The deviation between the adjusted entropy coefficient , the bias-adjusted entropy coefficient The adjustment rules are:

[0055] like , then increase ;

[0056] like , then reduce ;

[0057] like , then keep the current constant;

[0058] Dynamically control the exploration level and training stability of the policy.

[0059] The beneficial effects of the present invention are:

[0060] (1) The present invention divides the power load dispatching task into two subtasks: high-level dispatching target decision-making and low-level specific execution control by constructing a hierarchical reinforcement learning model. This solves the problems of task aliasing and control granularity ambiguity existing in existing reinforcement learning methods when facing complex multi-level dispatching tasks, realizes hierarchical response from global strategy planning to local control execution, and effectively improves the clarity and decision-making efficiency of the dispatching process.

[0061] (2) The present invention introduces an evolutionary reinforcement learning mechanism into the training of the low-level execution model, and combines it with the policy gradient optimization method based on the SoftActor-Critic method to perform two-stage hybrid training on the high-quality individuals in the policy population. This not only retains the diversity advantage of the evolutionary algorithm in global search, but also integrates the high-precision local update capability of the policy gradient, solving the problems of slow convergence and discontinuous policy updates in existing evolutionary optimization methods, and significantly enhancing the policy performance and robustness.

[0062] (3) Based on the Soft Actor-Critic method, this paper proposes low-level structural improvements, including a state distribution gating mechanism, a node attention mechanism, and a dynamic entropy adjustment strategy based on KL divergence. This enhances the model's ability to model the response differences of different types of node states and its strategy distribution control capabilities, effectively solving the problems of insufficient generalization ability and unstable strategy of traditional SAC algorithms in power dispatching scenarios, and improving the model's adaptability and deployment efficiency in multi-region and multi-type power grid scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0064] Figure 1 This is the overall flow chart of the power system load dynamic optimization method based on reinforcement learning proposed by the present invention. DETAILED DESCRIPTION

[0065] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0066] refer to Figure 1 , a power system load dynamic optimization method based on reinforcement learning, includes the following steps:

[0067] S1. Collect data from the power system and construct a state space. Specifically, this includes collecting key operating data such as the real-time load of each node, energy storage equipment status, user response capability, renewable energy output, electricity price information, and historical dispatch records from the power dispatch system in the target area. The state space includes high-level state space and low-level state space. The high-level state space is used to describe the overall operating status of the system, including the total load of the entire network, energy storage margin, electricity price trends, etc. Global features are uniformly input into the high-level decision model in vector form, and the low-level state space is constructed as a set of local states of each node, including features such as the node's current load, energy storage charge state, adjustable load capacity, and node category.

[0068] S2. Construct a hierarchical reinforcement learning model based on the state space, and divide the hierarchical reinforcement learning model into a high-level decision model and a low-level execution model;

[0069] S3. Training a high-level decision model. The high-level decision model takes the high-level state in the state space as input and outputs the scheduling task target category instruction;

[0070] S4. Training a low-level execution model. The low-level execution model takes the current node-level state and the scheduling task target category instructions output by the high-level decision model as input, and outputs a control action sequence including the load migration amount of each node, energy storage charging and discharging instructions, and user response incentive coefficients.

[0071] S5. When training the low-level execution model, an evolutionary reinforcement learning mechanism is introduced to construct several strategic individuals. The reward value of each strategic individual is calculated through interactive simulation of the environment. An improved genetic algorithm is used to perform selection, crossover, and mutation operations on the population of strategic individuals to obtain a new generation of strategic individuals.

[0072] S6. Combining the evolutionary reinforcement learning mechanism with the policy gradient optimization method, while the individual policies are evolving and updating, the policy gradient-based optimization method is used to further update the parameters of the policy individuals with better performance;

[0073] S7. Deploy the hierarchical reinforcement learning model to the power system load dispatch module. Specifically, after completing the training of the high-level decision model and the low-level execution model, integrate them into the power system load dispatch platform as the dispatch instruction generation core. During the deployment process, first configure the model reasoning module in the master station control center to receive the real-time collected status data as input. The high-level model first generates the current dispatch task target category instruction. Then, the low-level model outputs specific control actions based on the high-level instructions and node status, including load migration, energy storage charging and discharging, and user response incentives. The model sends the actions to each execution unit through the edge computing node or control terminal to achieve distributed execution. The deployment solution supports modular docking with the existing dispatch platform and can be loaded in different levels by region to achieve rapid response and scalable operation.

[0074] S8. During deployment and operation, continuously collect dispatch feedback data and fine-tune the parameters of the deployed hierarchical reinforcement learning model. Specifically, after the hierarchical reinforcement learning model is put into operation, the system continuously collects feedback information from the real-time dispatch process, including the results of dispatch instruction execution, actual load changes, energy storage response behavior, user participation, and system volatility indicators. The above feedback information is compared with the expected output results of the model to evaluate the performance deviation of the strategy in the real environment. If there is a significant deviation or environmental change, such as a change in user behavior pattern, adjustment of the electricity price mechanism, or change in the load structure, the fine-tuning mechanism is activated. The fine-tuning process adopts an incremental training method. On the basis of retaining the original model structure, the newly collected data is introduced to update the strategy parameters, especially the dynamic adjustment of the strategy weights related to key nodes in the low-level model;

[0075] S9. The high-level decision model and low-level execution model that have been migrated and fine-tuned are continuously applied to the load dispatching process of the power system. Specifically, after the fine-tuning optimization is completed, the updated high-level decision model and low-level execution model are reloaded into the dispatching system and continue to run as the main control strategy. The high-level model dynamically generates dispatching targets based on the overall state of the system, such as peak shaving, energy storage priority, load shifting and other task instructions. The low-level model combines the local state of each node with the high-level instructions to output refined control actions to achieve intelligent dispatching and control of node loads, energy storage equipment and user behavior. The dispatching process runs in real time in an online environment. The model can adaptively adjust the strategy output according to system load changes, grid topology adjustments and user behavior differences.

[0076] The present invention proposes a method for dynamic optimization of power system loads that integrates hierarchical reinforcement learning and evolutionary optimization. It realizes hierarchical response of scheduling tasks based on high- and low-level strategy division, effectively solving the problems of unstable strategy and unclear control in existing single-layer reinforcement learning structures when dealing with multi-objective and multi-granularity scheduling problems, and improves the optimization efficiency and controllability of strategy decision-making.

[0077] In this embodiment, S2 specifically includes:

[0078] S21. Divide the state space into a high-level state space and a low-level state space, wherein the high-level state space includes the global load trend of the power system, the range of electricity price fluctuations, and renewable energy output forecast data, and the low-level state space includes the real-time load value of each node, energy storage status information, and controllable load regulation capability parameters;

[0079] S22. Constructing a high-level decision model based on the high-level state space. The high-level decision model is responsible for capturing the overall operating trend of the system, formulating macro-dispatching directions, and serving as a guiding signal for low-level control. The high-level decision model uses the high-level state as input and outputs dispatch task target category instructions. The dispatch task target category instructions include peak shaving and valley filling dispatch instructions, energy storage priority regulation instructions, or demand response control instructions.

[0080] S23. Constructing a low-level execution model based on the low-level state space and the dispatch task target category instructions output by the high-level decision model. The low-level execution model uses the local state information of each node and the dispatch task target instructions output by the high-level model as input to generate a specific control action sequence, including node load adjustment, energy storage charging and discharging plan, and user incentive strategy. The low-level execution model is responsible for implementing the refined execution of the high-level strategy at specific nodes. The low-level execution model outputs a control action sequence including the load migration amount, energy storage charging and discharging control instructions, and user response incentive coefficient for each grid node.

[0081] S24. Establish a mapping relationship between the high-level decision model and the low-level execution model, using the scheduling task target category instructions as input conditions for the low-level execution model to form a hierarchical and coordinated scheduling control structure. Specifically, the scheduling task target category instructions output by the high-level decision model are directly passed into the low-level execution model as an auxiliary input. This allows the low-level execution model to not only consider local states when generating control actions for each node, but also combine high-level instructions for policy guidance. This mapping relationship effectively connects the global scheduling goal to the local control execution, allowing the two-layer model to form a unified input-output chain in structure, ensuring the consistency of scheduling strategies and the accuracy of execution.

[0082] In response to the complexity of power load scheduling tasks, the present invention proposes to divide the state space into high-level and low-level layers, and construct corresponding high-level decision models and low-level execution models to achieve a clearly structured task modeling process. Compared with traditional integrated strategy models, this model has stronger interpretability and scalability, and is suitable for large-scale scheduling system deployment.

[0083] In this embodiment, S3 specifically includes:

[0084] S31, obtaining a high-level state space as input data of a training sample;

[0085] S32. Build high-level decision-making strategy functions ,in, represents the high-level decision-making strategy function, is the high-level state of the input, The output scheduling task target category instruction;

[0086] S33. Define the immediate reward function of the high-level decision model ,in, Represents the time step, evaluating the output of the high-level decision strategy function The performance of the guided low-level execution results in terms of load balancing, operating cost control and scheduling volatility. The reward function is:

[0087] ;

[0088] in, Represents the high-level strategy at time step The immediate reward value, represents the system load deviation cost under the current scheduling instruction, represents the scheduling resource cost, represents the volatility cost of scheduling actions, 、 、 is the weighting coefficient;

[0089] S34, using policy gradient-based reinforcement learning method to improve the high-level decision-making policy function Training is performed by maximizing the immediate reward function The expected value of the high-level decision model is updated, and the high-level state input is input Output the optimal scheduling task target category instruction .

[0090] By introducing the advantage function and the instant reward construction mechanism, the present invention establishes a high-level policy training process, and combines the policy gradient method to achieve fine optimization of target category instructions. This is different from the existing methods that have vague or non-discriminatory designs for policy training objectives, and improves the rationality and stability of the scheduling strategy.

[0091] In this embodiment, the policy gradient-based reinforcement learning method in S34 is a SoftActor-Critic method based on maximum entropy policy optimization, which specifically includes:

[0092] S341. Construct two high-level Q-value function networks and , in a high-level state The corresponding output scheduling task target category instruction As input, output Q value estimation and , use the smaller value to participate in strategy optimization;

[0093] S342, introduce the objective function containing the policy entropy term as the training high-level decision-making policy function The optimization goal is to guide the optimization of the given high-level state The parameter update process under ;

[0094] S343, high-level decision strategy function and high-level Q-value function network 、 Perform alternating updates, specifically: during model training, do not update the parameters of the policy network and the Q-value network at the same time. Instead, first fix one, update the other, and then exchange them alternately. The following structural improvements are adopted: add a residual connection structure to the high-level decision policy function to improve training stability, use a shared state encoder in the Q-value function network to reduce parameter redundancy, and introduce a dynamic entropy coefficient adjustment mechanism based on the fluctuation amplitude of the target Q network to replace the static or predefined entropy coefficient update method.

[0095] This paper explicitly defines the high-level policy training method as Soft Actor-Critic and introduces structural improvements such as residual connection and dynamic entropy regulation mechanism, so that the high-level model exhibits stronger convergence stability and policy distribution control capabilities when dealing with electricity price fluctuations and load uncertainty, which is superior to traditional A2C or PPO methods.

[0096] In this embodiment, the S4 specifically includes:

[0097] S41. Low-level state input is defined as , including the current load value of each node in the power grid, the current state of charge of the energy storage device corresponding to the node, and the adjustable load capacity:

[0098] S42, the scheduling task target category instruction output by the high-level decision model With low-level state input The combination is used as the joint input of the low-level execution model to build the low-level policy function:

[0099] ;

[0100] in, Represents the control actions output by the low-level execution model, including load migration instructions for each node, charge and discharge operation instructions for energy storage devices, and user response incentive coefficients;

[0101] S43. Constructing low-level immediate reward functions , evaluate the input in the low-level state and the scheduling task target category instructions output by the high-level decision model Lower level control actions The resulting system performance, the reward function is defined as:

[0102] ;

[0103] in, represents the scheduling loss cost caused by load migration, represents the operating cost of the energy storage equipment at this moment, represents the incentive cost generated by motivating user response, is the weighting coefficient of each cost item;

[0104] S44, using the low-level immediate reward function To optimize the goal, train the low-level policy function The training is to optimize the low-level policy function so that the control action output by the low-level policy function can achieve tasks such as load adjustment, energy storage operation and user response at each node, thereby maximizing the overall scheduling effect; given the low-level state input and the scheduling task target category instructions output by the high-level decision model Output the optimal control action under the condition .

[0105] The present invention defines in detail the input structure and control action output form of the low-level execution model, enabling the model to simultaneously perceive the node load status, energy storage information, and user response capabilities, and generate multiple types of feasible scheduling instructions. Compared with existing control models that use a single load as the optimization target, this model has more practical scheduling deployment value.

[0106] In this embodiment, the S5 specifically includes:

[0107] S51, initialize the low-level strategy function, build Low-level policy function individuals , each low-level policy function individual is a set of independent low-level policy network parameters, corresponding to the control action , For the Individual control actions;

[0108] S52, record each strategy individual in the continuous The cumulative reward value in the training cycle , calculate the comprehensive fitness value of the strategy individual with historical smoothing weight:

[0109] ;

[0110] in, is the comprehensive fitness value, is the time step of each training cycle, For the Individuals in The first The low-level immediate reward function of the step;

[0111] S53. In the screening operation of the genetic algorithm, the selection method of the elite retention strategy is adopted, giving priority to retaining the top The strategy individual with the largest comprehensive fitness value is selected, and a roulette wheel selection is performed based on the ranking from the remaining individuals to generate a candidate parent set;

[0112] S54. Improve the offspring crossover operation. Specifically, based on the traditional strategy parameter crossover, a strategy similarity judgment mechanism is introduced to avoid invalid combinations. That is, before performing the crossover, the similarity of the two parent strategy individuals in the parameter space is calculated. If the similarity is higher than the set threshold, it is considered that the strategy difference is insufficient, and no crossover is performed. The individuals are directly copied or re-matched. If the similarity is within a reasonable range, the crossover operation is performed. The performance similarity between the two parent strategy individuals in the strategy parameter distribution space is judged. If their Euclidean distance or KL divergence is lower than the set threshold, crossover is allowed. Otherwise, the combination is discarded to avoid invalid or degenerate strategy fusion.

[0113] S55. Perform mutation operation on the selected offspring parameter set. Specifically, after completing the crossover of the strategy parameters, a certain probability of perturbation is applied to the generated offspring strategy individual parameters to introduce new strategy features and enhance the diversity of the strategy population. Different from the traditional method of fixed mutation rate, the present invention adopts a guidance mechanism based on target residual. For each offspring individual, the strategy target residual between it and the current optimal strategy is calculated. The mutation probability is dynamically determined according to the size of the residual. The larger the residual, the higher the mutation probability, and vice versa. The target guidance mechanism is introduced to calculate the mutation probability according to the current target residual of the strategy individual. , specifically defined as:

[0114] ;

[0115] in, is the coefficient of variation intensity, Indicates the gap indicator between the current target output of the strategy individual and the optimal individual;

[0116] S56. Take the new generation of strategy individuals after screening, crossover and target-guided mutation as the current strategy population, repeat S52 to S55, iterate and update until convergence, and select strategy individuals as the output strategy structure of the low-level execution model according to the final comprehensive fitness value from high to low.

[0117] The present invention introduces an evolutionary reinforcement learning mechanism, generates strategy individuals through evolutionary processes such as population evaluation and selection, crossover, and mutation, and deeply integrates it with the reinforcement learning process, which makes up for the defects of strong initial strategy dependence and insufficient global exploration ability in traditional reinforcement learning, and improves the diversity and search ability of scheduling strategies.

[0118] In this embodiment, S6 specifically includes:

[0119] S61. After each generation of evolutionary operations, select the fitness value from the current strategy population. strategic individuals As the candidate subset for policy gradient optimization, Indicates the A low-level strategy individual, ;

[0120] S62. For each individual strategy , construct a set of parameter-sharing dual Q-value function networks 、 , input in low-level state and the scheduling task target category instructions output by the high-level decision model And corresponding control actions For input;

[0121] S63. Construct the low-level strategy objective function as:

[0122] ;

[0123] in, For the lower state and the scheduling task target category instructions output by the high-level decision model Find the expected joint distribution of represents the minimum function, is the entropy weight coefficient, The logarithm of the probability generated by the policy network for the current action;

[0124] S64, will As the optimization target, update the low-level strategy individuals separately And the corresponding Q value function network The parameters of the Q network are updated using an asynchronous alternating training method, adjusting the target Q network parameters based on a soft update rule. Specifically, during training, the policy network and the Q-value function network are not updated simultaneously, but are updated alternately: first, the Q network is fixed, the policy network is updated, and then the policy network is fixed again, and the Q network parameters are updated. This is called asynchronous alternating training. In addition, to improve training stability, the Q network uses a soft update mechanism, that is, the target Q network parameters are not completely replaced by the current network parameters, but are updated smoothly according to a certain ratio.

[0125] S65. While updating the evolution of individual strategies, the optimization method based on policy gradient is used to further update the parameters of the strategy individuals with better performance. Among them, the optimization method based on policy gradient is the improved SoftActor-Critic method. The SoftActor-Critic method makes the following structural improvements: introduce a feature gating mechanism based on the input state distribution clustering label in the low-level dual Q-value function network, adjust the Q-value channel response strength, introduce a collaborative attention mechanism in the strategy network, adopt a dynamic target KL divergence adjustment strategy in the entropy coefficient update process, and adaptively adjust the strategy distribution offset amplitude during the training process. ;

[0126] S66, using the strategy individual trained jointly by the genetic algorithm and the improved SoftActor-Critic method as the low-level execution model target output strategy.

[0127] This paper combines evolutionary optimization with soft actor-critic policy gradient training to further optimize outstanding low-level policy individuals and construct an evolution-gradient fusion learning framework. Compared with evolution or gradient optimization schemes alone, this framework balances policy accuracy and exploration efficiency to achieve better performance.

[0128] In this embodiment, the structural improvement of the low-level SoftActor-Critic method in S65 specifically includes:

[0129] S651, in the low-level dual Q-value function network and The state distribution gating mechanism is introduced to input the low-level state Perform clustering to obtain state category labels , label the state category As the input embedded in the gating module, in the low-level double Q value function network feature extraction layer, according to Activate the corresponding channel control factor and perform gate adjustment on the intermediate features to generate category-sensitive Q-value estimation;

[0130] S652, at the lower level strategy individuals The cross-node state collaborative attention mechanism is introduced to form the low-level state The state features of each node are used as attention encoding input, and the attention weight distribution based on the dependency relationship between nodes is constructed. The features of each node are weighted and aggregated as the input of the strategy distribution;

[0131] S653, Entropy coefficient in low-level strategy network Introduce a dynamic adjustment mechanism to set the target strategy distribution offset value , calculate the current low-level strategy individual in each round of training Compared with the low-level strategy individuals in the previous round KL divergence of ,based on and The deviation between the adjusted entropy coefficient , the bias-adjusted entropy coefficient The adjustment rules are:

[0132] like , then increase ;

[0133] like , then reduce ;

[0134] like , then keep the current constant;

[0135] Dynamically control the exploration level and training stability of the policy.

[0136] The present invention structurally improves the low-level Soft Actor-Critic method by introducing a state gating mechanism, a node attention mechanism, and an entropy regulation strategy based on KL divergence, forming a strategy optimization path customized for the characteristics of power grid dispatching. Different from the traditional SAC structure, it improves the model's ability to express complex states and output stability.

[0137] Example 1:

[0138] To verify the feasibility of this invention, we applied it to the real-time load optimization scheduling of a regional power distribution system within the jurisdiction of a provincial power grid control center. This region, encompassing 21 grid dispatching nodes, covers urban residential areas, commercial districts, and some light industrial areas. This region exhibits typical characteristics of high power consumption volatility, uneven user load response capabilities, and unstable renewable energy output. Furthermore, the current dispatch model, which combines static priority rules with empirical models, suffers from response delays, low dispatch efficiency, and poor load redistribution accuracy.

[0139] The main challenges facing the power system scheduling task include: during the two peak hours of 12:00-14:00 and 18:00-20:00 daily, electricity load fluctuates violently, and local nodes are prone to overload alarms; the large number of household energy storage devices and user air conditioning equipment connected to the system causes high uncertainty in load forecasting and scheduling response; the system's scheduling response lags when responding to sudden drops in renewable energy output (such as photovoltaic shading or sudden changes in wind speed), and it is unable to timely control system stability. The method proposed in this invention uses reinforcement learning as its core to construct a hierarchical scheduling structure. The high-level model is responsible for identifying the global state and formulating the scheduling target type, such as peak shaving, energy storage priority, or load reduction; the low-level model generates specific scheduling behaviors for each node based on the node state and high-level instructions, including energy storage instructions, load migration plans, and strategies to encourage users to participate in demand response.

[0140] During this project, electricity load data, electricity price signals, user energy storage operating status, and photovoltaic output data for the region from July 2024 to January 2025 were collected to construct a state space and perform cluster modeling for different nodes. The high-level decision-making model was trained using an improved Soft Actor-Critic algorithm, improving scheduling task classification accuracy by approximately 14.2% compared to the traditional K-means priority dispatching model. The low-level control model integrates evolutionary optimization with improved SAC methods, dynamically adapting to the response capabilities of different nodes during policy iteration. Gating and attention mechanisms were introduced to strengthen policy weight control for key nodes. After deployment, the response speed to load mutations was reduced from an average of 68 seconds to 24 seconds, and the error in load redistribution at each node in peak shaving scenarios was reduced by 38%.

[0141] During the scheduling operation, November 16 to 30, 2024, was selected as a continuous verification window. The system automatically collected node data and executed the trained model for scheduling control. During the peak period of 6:00 PM to 8:00 PM on November 22, when load fluctuations were severe, the system processed 17 energy storage call events and 43 load migration operations, without any node exceeding the limit. Compared with the original baseline model (linear programming + expert rule), energy savings increased by 7.9%, user response rate increased by approximately 12.6%, and the volatility of the daily average load balance indicator decreased by 23%. Further observation showed that during the extreme weather event in early December that significantly reduced photovoltaic production, the proposed model was able to complete strategy adaptive fine-tuning and restore scheduling stability within 4 minutes, while the traditional model required over 10 minutes.

[0142] Table 1: Performance comparison of the present invention and traditional methods on a typical dispatch day

[0143]

[0144] Analyzing the data results in Table 1 "Performance comparison of the present invention and traditional methods on a typical scheduling day", it can be clearly seen that the present invention has significant advantages over traditional scheduling systems in multiple key performance indicators, reflecting its comprehensive improvement effects in response efficiency, control accuracy, user participation, and system stability.

[0145] In terms of load dispatch response, the system's average policy execution response time is only 24 seconds, a 64.7% reduction compared to the 68 seconds of the traditional dispatch system. The maximum single-node response delay is reduced from 112 seconds to 39 seconds, a reduction of 65.2%. This demonstrates that the system significantly improves the efficiency of policy triggering and action delivery. This high responsiveness is particularly critical when dealing with sudden load fluctuations during peak hours on the power grid, effectively reducing the risk of overloads.

[0146] In terms of load balancing performance, the optimized strategy of this invention demonstrates more accurate peak-shaving and valley-filling control. The residual deviation after regulation was reduced from 11.4% to 6.2%, and the mean redistribution error was reduced from 9.3kW to 5.7kW, a 38.7% reduction. This demonstrates that the low-level execution model of this invention can more rationally distribute load tasks, improving system balance and control effectiveness.

[0147] In terms of user behavior response, this invention enhances user engagement with scheduling through its user incentive mechanism and node attention strategy. This increased user participation from 54.1% to 60.9%, a 12.6% improvement. Furthermore, the average time it takes to adopt incentive signals was shortened from 36 seconds to 22 seconds, demonstrating improved timeliness of user response.

[0148] In terms of energy storage control effectiveness, the proposed strategy can more efficiently utilize energy storage resources, increasing daily energy storage utilization from 73.5% to 82.4%, and increasing the coverage of energy storage for peak load reduction from 48.2% to 56.8%. This means that the proposed strategy can more fully mobilize energy storage units to participate in power optimization and dispatch, reducing reliance on external power supply from the grid.

[0149] In terms of energy economy, the electricity procurement cost during peak hours dropped from 468,000 yuan to 421,000 yuan, a 10% reduction in cost. This shows that the present invention better achieves resource coordination on the electricity consumption side during peak hours, reduces peak power procurement, and has significant economic benefits.

[0150] Finally, in terms of system stability, the present invention reduced the number of abnormal fluctuations during scheduling strategy switching from 8 to 2 per day, a 75% reduction. This demonstrates that the present invention offers advantages in terms of continuity during strategy adjustment and smooth system control, effectively preventing local instability caused by frequent switching.

[0151] In summary, the present invention significantly improves the real-time, accuracy, robustness, and economy of the power system load dynamic optimization process through a clearly structured hierarchical reinforcement learning model, an evolutionary-gradient fusion optimization mechanism, and an improved SAC strategy structure, and has system-level performance that is superior to existing scheduling strategies.

[0152] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for dynamic optimization of power system load based on reinforcement learning, characterized in that: The steps include: S1, collect data of the power system and construct the state space; S2. Construct a hierarchical reinforcement learning model based on the state space, and divide the hierarchical reinforcement learning model into a high-level decision model and a low-level execution model; S3. Training a high-level decision model. The high-level decision model takes the high-level state in the state space as input and outputs the scheduling task target category instruction; S4. Training a low-level execution model. The low-level execution model takes the current node-level state and the scheduling task target category instructions output by the high-level decision model as input, and outputs a control action sequence including the load migration amount of each node, energy storage charging and discharging instructions, and user response incentive coefficients. S5. When training the low-level execution model, an evolutionary reinforcement learning mechanism is introduced to construct several strategic individuals. The reward value of each strategic individual is calculated through interactive simulation of the environment. An improved genetic algorithm is used to perform selection, crossover, and mutation operations on the population of strategic individuals to obtain a new generation of strategic individuals. Wherein, the S5 specifically includes: S51, initialize the low-level strategy function, build Low-level policy function individuals ; S52, record each strategy individual in the continuous The cumulative reward value in the training cycle , calculate the comprehensive fitness value of the strategy individual with historical smoothing weight; S53. In the screening operation of the genetic algorithm, the selection method of the elite retention strategy is adopted, giving priority to retaining the top The strategy individual with the largest comprehensive fitness value is selected, and a roulette wheel selection is performed based on the ranking from the remaining individuals to generate a candidate parent set; S54. Improve the offspring crossover operation to determine the similarity between the performance of two parent strategy individuals in the strategy parameter distribution space. If the Euclidean distance or KL divergence is less than the set threshold, crossover is allowed; otherwise, the combination is discarded. S55, perform mutation operation on the selected offspring parameter set, introduce target guidance mechanism, and calculate mutation probability based on the current target residual of the strategy individual ; S6. Combining the evolutionary reinforcement learning mechanism with the policy gradient optimization method, while the individual policies are evolving and updating, the policy gradient-based optimization method is used to further update the parameters of the policy individuals with better performance; Wherein, the S6 specifically includes: S61. After each generation of evolutionary operations, select the fitness value from the current strategy population. strategic individuals As the candidate subset for policy gradient optimization, Indicates the A low-level strategy individual, ; S62. For each individual strategy , construct a set of parameter-sharing dual Q-value function networks 、 , S63, construct the low-level strategy objective function as ; S64, will As the optimization target, update the low-level strategy individuals separately And the corresponding Q value function network The parameters of the Q network are updated using an asynchronous alternating training method, and the target Q network parameters are adjusted based on the soft update rule; S65. While the individual policies are evolving and updating, the policy gradient-based optimization method is used to further update the parameters of the policy individuals with better performance. The policy gradient-based optimization method is the improved Soft Actor-Critic method. S651, in the low-level dual Q-value function network and The state distribution gating mechanism is introduced to input the low-level state Perform clustering to obtain state category labels , label the state category As the input embedded in the gating module, in the low-level double Q value function network feature extraction layer, according to Activate the corresponding channel control factor and perform gate adjustment on the intermediate features to generate category-sensitive Q-value estimation; S652, at the lower level strategy individuals The cross-node state collaborative attention mechanism is introduced to form the low-level state The state features of each node are used as attention encoding input, and the attention weight distribution based on the dependency relationship between nodes is constructed. The features of each node are weighted and aggregated as the input of the strategy distribution; S653, Entropy coefficient in low-level strategy network Introduce a dynamic adjustment mechanism to set the target strategy distribution offset value , calculate the current low-level strategy individual in each round of training Compared with the low-level strategy individuals in the previous round KL divergence of ,based on and The deviation between the adjusted entropy coefficient ; S7. Deploy the hierarchical reinforcement learning model to the power system load dispatch module; S8. During deployment and operation, continuously collect scheduling feedback data and fine-tune the parameters of the deployed hierarchical reinforcement learning model; S9. Continue to apply the migrated and fine-tuned high-level decision-making model and low-level execution model to the load scheduling process of the power system.

2. The method for dynamic optimization of power system load based on reinforcement learning according to claim 1, characterized in that: The S2 specifically includes: S21. Divide the state space into a high-level state space and a low-level state space, wherein the high-level state space includes the global load trend of the power system, the range of electricity price fluctuations, and renewable energy output forecast data, and the low-level state space includes the real-time load value of each node, energy storage status information, and controllable load regulation capability parameters; S22. Constructing a high-level decision model based on the high-level state space, wherein the high-level decision model uses the high-level state as input and outputs a scheduling task target category instruction, wherein the scheduling task target category instruction includes a peak shaving and valley filling scheduling instruction, an energy storage priority regulation instruction, or a demand response control instruction; S23: Constructing a low-level execution model based on the low-level state space and the dispatch task target category instructions output by the high-level decision model, wherein the low-level execution model outputs a control action sequence including the load migration amount, energy storage charge and discharge control instructions, and user response incentive coefficients of each grid node; S24. Establish a mapping relationship between the high-level decision model and the low-level execution model, so that the scheduling task target category instruction serves as an input condition of the low-level execution model, forming a hierarchical linkage scheduling control structure.

3. The method for dynamic optimization of power system load based on reinforcement learning according to claim 2, characterized in that: The S3 specifically includes: S31, obtaining a high-level state space as input data of a training sample; S32. Build high-level decision-making strategy functions ,in, represents the high-level decision-making strategy function, is the high-level state of the input, The output scheduling task target category instruction; S33. Define the immediate reward function of the high-level decision model ,in, Represents the time step, evaluating the output of the high-level decision strategy function The performance of the guided low-level execution results in terms of load balancing, operating cost control and scheduling volatility; S34, using policy gradient-based reinforcement learning method to improve the high-level decision-making policy function Training is performed by maximizing the immediate reward function The expected value of the high-level decision model is updated, and the high-level state of the input is Output the optimal scheduling task target category instruction .

4. The method for dynamic optimization of power system load based on reinforcement learning according to claim 3, characterized in that: The policy gradient-based reinforcement learning method in S34 is a SoftActor-Critic method based on maximum entropy policy optimization, specifically including: S341. Construct two high-level Q-value function networks and , in a high-level state The corresponding output scheduling task target category instruction As input, output Q value estimation and , use the smaller value to participate in strategy optimization; S342, introduce the objective function containing the policy entropy term as the training high-level decision-making policy function The optimization goal is to guide the optimization of the given high-level state The parameter update process under ; S343, high-level decision strategy function and high-level Q-value function network 、 Alternating updates are performed with the following structural improvements: a residual connection structure is added to the high-level decision strategy function, a shared state encoder is used in the Q-value function network, and a dynamic entropy coefficient adjustment mechanism based on the target Q-network fluctuation amplitude is introduced to replace the static or predefined entropy coefficient update method.

5. The method for dynamic optimization of power system load based on reinforcement learning according to claim 4, characterized in that: The S4 specifically includes: S41. Low-level state input is defined as , including the current load value of each node in the power grid, the current state of charge of the energy storage device corresponding to the node, and the adjustable load capacity: S42, the scheduling task target category instruction output by the high-level decision model With low-level state input The combination is used as the joint input of the low-level execution model to construct the low-level policy function; S43. Constructing low-level immediate reward functions , evaluate the input in the low-level state and the scheduling task target category instructions output by the high-level decision model Lower level control actions the resulting system performance; S44, using the low-level immediate reward function To optimize the goal, train the low-level policy function , given the low-level state input and the scheduling task target category instructions output by the high-level decision model Output the optimal control action under the condition .

6. The method for dynamic optimization of power system load based on reinforcement learning according to claim 5, characterized in that: The S5 specifically further includes: Each low-level policy function individual is a set of independent low-level policy network parameters corresponding to the control action , For the Individual control actions; The new generation of strategy individuals after screening, crossover and target-guided mutation is used as the current strategy population. S52 to S55 are repeatedly executed, and iterative updates are made until convergence. According to the final comprehensive fitness value from high to low, the strategy individuals are selected as the output strategy structure of the low-level execution model.

7. The method for dynamic optimization of power system load based on reinforcement learning according to claim 6, characterized in that: The S6 specifically further includes: Input in low level state and the scheduling task target category instructions output by the high-level decision model And corresponding control actions For input; The SoftActor-Critic method makes the following structural improvements: introduces a feature gating mechanism based on the input state distribution cluster label in the low-level dual Q-value function network, adjusts the Q-value channel response strength, introduces a collaborative attention mechanism in the policy network, adopts a dynamic target KL divergence adjustment strategy in the entropy coefficient update process, and adaptively adjusts the policy distribution offset amplitude during training. ; The strategy individuals trained jointly by the genetic algorithm and the improved SoftActor-Critic method are used as the low-level execution model target output strategy.

8. The method for dynamic optimization of power system load based on reinforcement learning according to claim 7, characterized in that: The structural improvements made by S65 to the low-level SoftActor-Critic method specifically include: The bias-adjusted entropy coefficient The adjustment rules are: like , then increase ; like , then reduce ; like , then keep the current constant; Dynamically control the exploration level and training stability of the policy.

Images