Load management and control method and system based on reinforcement learning

Abstract

The invention relates to a load regulation and control method and system based on reinforcement learning, and the method comprises the steps: 1, decomposing a multi-time scale optimization problem into a plurality of layers in a source network load storage management and control scene through employing a layered optimization frame, enabling each layer to correspond to a different time scale, and carrying out the optimization; a model prediction control algorithm is adopted in the short-term optimization layer, a mixed integer linear programming algorithm is adopted in the medium-term optimization layer, and a reinforcement learning method is adopted in the long-term optimization layer to solve a multi-objective optimization problem; and 2, long-term load regulation and control decision making: controlling the power grid to make different load regulation and control decisions through an actually adjustable reward mechanism based on reinforcement learning, so that the intelligent power grid load regulation and control system adapts to continuously changing environments and targets, and the flexibility and adaptability of the system are improved. According to the invention, the reliability and stability of power supply can be ensured, fine management and regulation of power loads are realized, and the operation efficiency, reliability and safety of a power system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart grids, and in particular relates to a load control method and system based on reinforcement learning. Background Art

[0002] Smart grid power load control system is an important part of smart grid construction, and its market prospects are broad and have a good development trend. Smart grid power load control system uses advanced information technology, communication technology and control technology to achieve refined management and control of power load, aiming to improve the operating efficiency, reliability and safety of the power system.

[0003] At present, the smart grid power load control system is mainly based on advanced information and communication technology, energy management technology and intelligent control technology, integrating various types of power system data, including multi-source data from the power supply side, power consumption side, and environmental side, and processing and comprehensive analysis of these data through intelligent analysis technology to achieve accurate prediction and dynamic control of power load. Although the construction of smart grid power load control system has achieved remarkable results, there are still problems such as the difficulty in controlling massive demand-side resources, insufficient interaction between levels, difficulty in multi-objective optimization, and conflicts between long-term and short-term goals. Summary of the invention

[0004] The purpose of the present invention is to provide a load control method and system based on reinforcement learning, to subdivide and manage massive demand-side resources, to introduce multi-time-scale multi-objective reward functions to solve the long-term and short-term goal optimization problems, and to combine multi-agent reinforcement learning and hierarchical reinforcement learning methods. Through cross-level communication protocols, the interaction between load control levels is enhanced, and multi-level load control is realized by integrating reinforcement learning and traditional optimization methods.

[0005] The present invention provides a load control method based on reinforcement learning, comprising the following steps:

[0006] Step 1: In the source-grid-load-storage control scenario, a hierarchical optimization framework is used to decompose the multi-time-scale optimization problem into multiple levels. Each level corresponds to a different time scale and is optimized separately. The model predictive control algorithm is used in the short-term optimization layer, the mixed integer linear programming algorithm is used in the medium-term optimization layer, and the reinforcement learning method is used in the long-term optimization layer to solve the multi-objective optimization problem.

[0007] Step 2, long-term load control decision-making, based on reinforcement learning through a practically adjustable reward mechanism, controls the power grid to make different load control decisions, so that the smart grid load control system can adapt to the changing environment and goals, and improve the flexibility and adaptability of the system.

[0008] Furthermore, the step 1 comprises:

[0009] Step 1.1, divide the time scale:

[0010] Short-term optimization is based on hours or days, and the control targets include real-time scheduling, load balancing, and instantaneous response. Medium-term optimization is based on weeks or months, and the control targets include periodic scheduling and energy storage management. Long-term optimization is based on years, and the control targets include equipment investment, long-term planning, and strategic decision-making.

[0011] Step 1.2, define the objective function:

[0012] Determine the minimum operating cost, benefit target, and reliability target expected by the system, define different objective functions at each time scale, and ensure that the objective functions are consistent and coordinated between different time scales; the minimum operating cost includes power generation cost, energy storage system operating cost, and load management cost; the benefit target includes renewable energy utilization rate and user satisfaction; the reliability target includes the power supply fluctuation range allowed by the system;

[0013] Step 1.3, define constraints:

[0014] Ensure the consistency and coordination between the objective functions and constraints at each time scale based on the capacity limitations of the energy storage system, the charging and discharging rate limitations, and the voltage and frequency stability requirements of the power system in the specific scenario;

[0015] The constraints of distributed photovoltaic output are:

[0016] (P PV,i (t)) 2 +(Q PV,i (t)) 2 ≤(S pV,i,max ) 2

[0017] Where P PV,i and Q Pv,i are the photovoltaic active and reactive outputs of node i at time t, S PV,i,max is the apparent power upper limit of the photovoltaic node i;

[0018] The energy storage SOC constraint is:

[0019] SOC min <SOC(t)<SOC max

[0020] In the formula, SOC min , SOC max They represent the minimum and maximum charge of the energy storage device respectively, and SOC(t) represents the remaining charge of the energy storage device at time t;

[0021] When a power grid fails, each virtual power plant in each zone enters island operation. At this time, the power supply rate of each zone is constrained to ensure that the demand of important loads is met. The constraints are as follows:

[0022] I k (t)≥I k,min ,k=1,2,…,n p

[0023] In the formula, I k,min I is the proportion of the primary important load of the k-th zone photovoltaic storage virtual power plant; k is the power supply rate of the k-th zone photovoltaic storage virtual power plant; n p is the number of partitions;

[0024] The stable operation of the power grid also needs to meet the following power flow constraints:

[0025]

[0026] U i,min ≤U i (t)≤U i,max

[0027] |I ij |≤I ij,max

[0028] Where P mi (t), Q mi (t) and P ik (t), Q ik (t) are the active and reactive power of branch mi and branch ik at time t respectively; R mi is the resistance of branch mi; X mi is the reactance of branch mi; P i (t) and Q i (t) are the active power and reactive power injected by node i at time t; u(i) is the set of head nodes of the branch with node i as the end node; d(i) is the set of end nodes of the branch with node i as the head node; U i (t) is the voltage of node i; U i,min and U i,max are the minimum and maximum voltages of node i respectively; I ij and I ij,max are the current flowing through the branch ij and the maximum value of the current allowed to flow through it; N is the number of system nodes; m is the index number of the head node; k is the index number of the end node;

[0029] Step 1.4: Use the MPC method for real-time scheduling in short-term optimization. At each control moment, based on the current state and the prediction model, predict the behavior of the system in the future. Through the optimization algorithm, find the optimal control sequence in the prediction time domain. According to the optimal control sequence, implement the first control action in the optimized control sequence. At the next moment, roll the window, update the system state, and repeat the prediction and optimization process.

[0030] In the mid-term optimization, the MILP method is used for monthly scheduling, and the mixed integer linear programming solver is used to solve the problem. The optimal solution that satisfies all constraints is found according to the objective function. The mid-term scheduling plan is generated according to the solution results, including the generator switching state and the energy storage system charging and discharging plan.

[0031] Long-term optimization uses the long-term load control decision based on reinforcement learning in step 2;

[0032] Step 1.5, integrated optimization and result analysis:

[0033] The long-term optimization results are used as constraints for medium-term optimization, and the medium-term optimization results are used as constraints for short-term optimization. Coordinated optimization is performed on multiple time scales. The Pareto frontier of each time scale is generated and visualized. The trade-off between different objectives is analyzed. The appropriate Pareto optimal solution is selected according to the actual operating conditions for scheduling and optimization.

[0034] Furthermore, in step 1.2, the short-term goal is to minimize the real-time operating cost C short-term , maximize the use of renewable energy RE , ensure the stability of the power system power :

[0035]

[0036] Subject to S power (x t )≤Δ

[0037] Where, MinC short-term To minimize the real-time operating cost goal; MaxU RE To maximize the utilization rate of renewable energy; Subject to S power is the power grid power stability fluctuation range constraint; x t It is the short-term load regulation decision in period t;

[0038] Mid-term goal is to minimize monthly operating costs C mid-term , maximize the utilization rate of energy storage system U storage , ensure power supply reliability supply :

[0039]

[0040] Subject to R supply (y t )≤Δ

[0041] Where, Min C mid-term To minimize mid-term operating costs; Max U storage To maximize the utilization of energy storage system; Subject to R supply is the power supply stability range constraint of the power grid; t is the mid-term load control decision for period t; T is the total event segment length of the mid-term target; t is the basic time period index number within the total mid-term optimization duration; c t is the mid-term unit operating cost in period t; Δ is the upper limit of the fluctuation range of power grid stability;

[0042] Long-term goal is to minimize the overall investment cost C long- , maximize system benefits B system , ensuring long-term sustainable development long-ter :

[0043]

[0044] Subject to S long-ter (z t )≤Δ

[0045] Where Min is the goal of minimizing the overall investment cost; MaxB system To maximize the system benefit goal; Subject to S long-term To constrain the volatility range of sustainable development; t For long-term load regulation decisions.

[0046] Furthermore, the step 2 comprises:

[0047] Step 2.1, data collection and preprocessing:

[0048] Collect multimodal data, increase the robustness of the data through data interpolation and smoothing, and convert the data into a format suitable for model input; the multimodal data includes historical load data, sensor data, user behavior data, weather data, and electricity price data;

[0049] Step 2.2, feature engineering and feature extraction:

[0050] Use machine learning methods to calculate the importance of each feature, select key features, and use contrastive learning methods to construct a high-dimensional mapping relationship between multi-time scale, multi-modal data and features related to user internal characteristics;

[0051] Step 2.3, define the state and action space and build a reinforcement learning model:

[0052] Determine the state space and possible actions of the power load system, wherein the state includes the current power demand, the power generation of renewable energy, the predicted load change, and the power reserve in the energy storage system, and the actions include adjusting the output of the generator set, adjusting the operating power of the equipment, and activating the reserve power;

[0053] Assume that there is a state space S, where each state represents a certain situation in the environment; an action space A, where each action represents an action that can be taken in the current state; and a reward function R(s,a), which represents the reward obtained by taking action a in state s. The goal is to find a strategy π(a|s), that is, the probability of choosing action a in state s, to maximize the expected cumulative reward. The expected value of the cumulative reward is expressed as the return value G. t :

[0054]

[0055] Among them, γ is the discount factor, which indicates the importance of future rewards, and r t+k+1 It is the immediate reward obtained at time t+k+1;

[0056] Dynamically adjust the reward function R(s,a) and use the policy gradient method or Q learning method to optimize it according to changes in the environment and improvements in the strategy;

[0057] Step 2.4, design reward function:

[0058] When the system successfully balances the real-time load and avoids overload, positive rewards are given; conversely, if it causes insufficient power supply or waste of resources, negative penalties are given;

[0059] Provide positive rewards for behaviors that encourage the use of electricity when prices are low and encourage the use of less electricity when prices are high;

[0060] During peak load periods: Increase incentives for reducing load to encourage users to reduce peak loads;

[0061] In the event of equipment failure or natural disasters, the reward function is dynamically adjusted to prioritize grid stability and security;

[0062] In the dynamic reward mechanism, the reward function R(s,a) is dynamically adjusted as the environment changes and the strategy improves. The reward adjustment function ΔR(s,a,t) is introduced to represent the adjustment amount of the reward at time t:

[0063] R(s,a,t)=R(s,a)+ΔR(s,a,t)

[0064] Among them, the reward adjustment function ΔR(s,a,t) is:

[0065]

[0066] Where β is the adjustment coefficient, G t is the actual return at time t, is the average return over the past period of time;

[0067] When the strategy performs better than average, the reward increases; otherwise, the reward decreases.

[0068] Furthermore, in step 2.2, the load curve characteristics are introduced from the perspective of time series data mining:

[0069] The values ​​of the load sequence X are binned, and the interval [min(X), max(X)] is equally divided into k small intervals. The values ​​of the load sequence are dispersed in k bins.

[0070] According to the situation of equally spaced bins, the entropy of the probability distribution is calculated:

[0071]

[0072] Where: Indicates the proportion of the load sequence values ​​that fall into the kth bucket; max bins indicates the number of buckets; len(X) indicates the length of the time series:

[0073] The entropy of the probability distribution is embedded into the reinforcement learning model as a state.

[0074] Furthermore, in step 2.3, the Q learning method selects the optimal action by estimating the action-value function Q(s,a), with the goal of making Q(s,a) satisfy the Bellman equation:

[0075]

[0076] Approach the optimal Q value by iteratively updating the Q value;

[0077] The update formula of Q value is:

[0078]

[0079] where α is the learning rate and s′ is the new state that state s transitions to after taking action a.

[0080] The present invention also provides a load control system based on reinforcement learning, including a load control module, and the load control module executes the load control method based on reinforcement learning.

[0081] The present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the load control method based on reinforcement learning is implemented.

[0082] The present invention also provides an electronic device, comprising:

[0083] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the load control method based on reinforcement learning by executing the computer instructions.

[0084] Through the above scheme, the load control method and system based on reinforcement learning have the following technical effects:

[0085] 1) The present invention dynamically adjusts the reward mechanism so that the load control system can adapt to the ever-changing environment and goals, thereby improving the flexibility and adaptability of the system. In actual operation, the system can improve the robustness of data by collecting and processing multimodal data such as historical load data, user behavior data, and weather data. Based on the reinforcement learning method, a high-dimensional mapping relationship between multi-time scales, multimodal data and user internal characteristics is constructed, and a reasonable reward function is set to achieve real-time load balancing in the system, avoid overload, and improve the economy, stability, and environmental protection of the power grid.

[0086] 2) The multi-objective optimization algorithm under multiple time scales of the present invention adopts a hierarchical optimization strategy to decompose the optimization problem of multiple time scales into multiple levels, each level corresponds to a different time scale, and is optimized separately. Among them, short-term optimization minimizes short-term operating costs to ensure the stability of the power system; mid-term optimization maximizes the utilization of the energy storage system to ensure power supply reliability; long-term optimization comprehensively considers the long-term service life and maintenance cost of the energy storage equipment in the system to ensure the long-term sustainable development of the system.

[0087] 3) This invention uses the subdivision and management method of massive demand-side resources to improve the refinement level of load control through subdivision and management, aiming at the problem that massive demand-side resources are difficult to control in current load control methods. Through the dynamic incentive mechanism based on reinforcement learning and combined with the multi-agent reinforcement learning method, the interaction between load control levels is enhanced.

[0088] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1It is a flow chart of the load control method based on reinforcement learning of the present invention;

[0090] Figure 2 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0091] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0092] Ginseng Figure 1 The present embodiment provides a load control method based on reinforcement learning, comprising the following steps:

[0093] Step S1: In the source-grid-load-storage control scenario, a hierarchical optimization framework is used to decompose the multi-time-scale optimization problem into multiple levels, each level corresponds to a different time scale, and is optimized separately; wherein, a model predictive control algorithm is used in the short-term optimization layer, a mixed integer linear programming algorithm is used in the medium-term optimization layer, and a reinforcement learning method is used in the long-term optimization layer to solve the multi-objective optimization problem;

[0094] Step S2, long-term load control decision, controls the power grid to make different load control decisions based on reinforcement learning through a practically adjustable reward mechanism, so that the smart grid load control system can adapt to the changing environment and goals and improve the flexibility and adaptability of the system.

[0095] The present invention is described in further detail below.

[0096] Considering the massive resource regulation in the energy consumption scenario, the insufficient interaction between the various levels in the system, and the existing power load regulation methods, it is difficult to achieve multi-objective optimization and conflicts between long-term and short-term load regulation objectives. This embodiment proposes a load control method based on hierarchical reinforcement learning, which can reasonably arrange the power supply plan and make preparations for power dispatch in advance to ensure the reliability and stability of power supply, realize the refined management and regulation of power loads, and improve the operating efficiency, reliability and safety of the power system. The load regulation method based on reinforcement learning mainly includes a multi-objective optimization framework and target design under multiple time scales, as well as dynamic intelligent load regulation based on reinforcement learning. The specific plan is as follows:

[0097] a) Multi-objective optimization framework at multiple time scales

[0098] Multi-objective optimization is an effective tool to solve the long-term and short-term optimization problems of source, grid, load and storage control needs. Multi-objective optimization not only considers the optimal solution of a single objective, but also considers multiple objectives at the same time to find a balanced solution. In the source, grid, load and storage control scenario, the system needs to simultaneously optimize cost minimization and system reliability maximization, and optimize the two conflicting objective functions of long-term load regulation and short-term load regulation. In this case, there is no single optimal solution, but there is a Pareto optimal solution. The Pareto optimal solution means that it is impossible to optimize better on one objective without making another objective worse while being no worse than any other solution.

[0099] In this embodiment, a hierarchical optimization framework is used to decompose the multi-time scale optimization problem into multiple levels, each level corresponds to a different time scale, and is optimized separately. The model predictive control algorithm (MPC) is used in the short-term optimization layer, the mixed integer linear programming algorithm (MILP) is used in the medium-term optimization layer, and the reinforcement learning method is used in the long-term optimization layer to solve the multi-objective optimization problem.

[0100] The specific operations and steps of this method are as follows:

[0101] Step 1.1 Time scale division: short-term optimization is based on hours or days, focusing on control objectives such as real-time scheduling, load balance and instantaneous response; medium-term optimization is based on weeks or months, focusing on scheduling objectives such as periodic scheduling and energy storage management; long-term optimization is based on years, focusing on control objectives such as equipment investment, long-term planning and strategic decision-making.

[0102] Step 1.2 Objective function definition, clarify the system's expected minimum operating cost, including power generation cost, energy storage system operating cost, load management cost, etc., clarify the system's expected benefit goals such as renewable energy utilization rate and user satisfaction, and clarify the system's reliability goals such as the allowable power supply fluctuation range. Define different objective functions at each time scale and ensure that these objective functions are consistent and coordinated between different time scales. Short-term goals seek to minimize (Min) real-time operating cost C short-ter , Maximize the utilization of renewable energy RE , ensure the stability of the power system power :

[0103]

[0104] Subject to S power (x t )≤Δ

[0105] Where, Min C short-ter To minimize the real-time operating cost objective; Max U RETo maximize the utilization rate of renewable energy; Subject to S power is the power grid power stability fluctuation range constraint; x t It is the short-term load control decision in period t.

[0106] Mid-term goal is to minimize monthly operating costs C mid-term , maximize the utilization rate of energy storage system U storage , ensure power supply reliability supply :

[0107]

[0108] Subject to R supply (y t )≤Δ

[0109] Where, Min C mid-term To minimize mid-term operating costs; Max U storage To maximize the utilization of energy storage system; Subject to R supply is the power supply stability range constraint of the power grid; t is the mid-term load control decision for period t; T is the total event segment length of the mid-term target; t is the basic time period index number within the total mid-term optimization duration; c t is the mid-term unit operating cost in period t; Δ is the upper limit of the fluctuation range of power grid stability.

[0110] Long-term goal is to minimize the overall investment cost C long-ter , maximize system benefits B system , ensuring long-term sustainable development long-term :

[0111]

[0112] Subject to S long-term (z t )≤Δ

[0113] Where, Min is the goal of minimizing the overall investment cost; Max B system To maximize the system benefit goal; Subject to S long-term To constrain the volatility range of sustainable development; t For long-term load regulation decisions.

[0114] Step 1.3 Constraint definition: according to the capacity limitation, charge and discharge rate and other limitations of the energy storage system in the specific scenario and the voltage and frequency stability requirements of the power system, ensure the consistency and coordination between the objective functions and constraints of each time scale. For example, the constraint conditions of distributed photovoltaic output are:

[0115] (P PV,i (t)) 2 +(Q PV,i (t)) 2 ≤(S PV,i,max ) 2

[0116] Where P PV,i and Q PV,i are the photovoltaic active and reactive outputs of node i at time t, S PV,i,max is the apparent power upper limit of the photovoltaic node i.

[0117] In order to ensure that the energy storage can work normally and extend the life of the energy storage, the state of charge (SOC) of the energy storage device needs to be subject to certain constraints to avoid overcharging and over-discharging of the energy storage device. The energy storage SOC constraint is:

[0118] SOC min <SOC(t)<SOC max

[0119] Where SOC min , SOC max They represent the minimum and maximum charge of the energy storage device respectively, and SOC(t) represents the remaining charge of the energy storage device at time t.

[0120] When the power grid fails, each virtual power plant in each zone enters island operation. At this time, the power supply rate of each zone should be constrained to ensure that the demand of important loads is met. The constraints are as follows:

[0121] I k (t)≥I k,min ,k=1,2,…,n p

[0122] Where: I k,min I is the proportion of the primary important load of the K-th zone photovoltaic storage virtual power plant; k is the power supply rate of the k-th zone photovoltaic storage virtual power plant; n p is the number of partitions.

[0123] The stable operation of the power grid also needs to meet the following power flow constraints:

[0124]

[0125] U i,min ≤U i (t)≤U i,max

[0126] |I ij |≤I ij,mac

[0127] In the above formula, P mi (t), Q mi (t) and P ik (t), Q ik (t) are the active power of branch mi and branch ik at time t. Reactive power; R mi is the resistance of branch mi; X mi is the reactance of branch mi; P i (t) and Q i (t) are the active power and reactive power injected by node i at time t. u(i) is the set of head nodes of the branch with node i as the end node; d(i) is the set of end nodes of the branch with node i as the head node. i (t) is the voltage of node i; U i,min and U i,max are the minimum and maximum voltages of node i respectively; I ij and I ij,max They are respectively the current flowing through branch ij and the maximum value of the current allowed to flow; N is the number of system nodes; m is the index number of the head node; k is the index number of the end node.

[0128] Step 1.4 Use the MPC method for real-time scheduling in short-term optimization. At each control moment, based on the current state and prediction model, predict the behavior of the system in the future. Use the optimization algorithm to find the optimal control sequence {u 0 ,u 1 ,…,u N-1}, according to the control sequence, implement the first control action in the optimized control sequence, roll the window and update the system status at the next moment, and repeat the prediction and optimization process; use the MILP method for monthly scheduling in the medium-term optimization, and use a mixed integer linear programming solver (such as CPLEX, Gurobi) to solve and find the optimal solution that meets all constraints.

[0129] A medium-term dispatch plan is generated based on the solution results, including the generator switching status, energy storage system charging and discharging plan, etc.; long-term optimization uses the long-term load control decision based on reinforcement learning in the following b).

[0130] Step 1.5 integrates optimization and result analysis, taking the long-term optimization results as the constraints of the mid-term optimization, and the mid-term optimization results as the constraints of the short-term optimization, to achieve coordinated optimization of multiple time scales. Generate and visualize the Pareto frontier of each time scale, analyze the trade-off relationship between different objectives, select the appropriate Pareto optimal solution according to the actual operation situation, and perform scheduling and optimization.

[0131] b) Long-term load control decision-making based on reinforcement learning

[0132] The algorithm controls the power grid to make different load control decisions by adjusting the reward mechanism according to actual conditions, so that the smart grid load control system can better adapt to the changing environment and goals, improve the flexibility and adaptability of the system, and thus optimize its behavior strategy.

[0133] The specific operations and steps are as follows:

[0134] Step 1.1 Data collection and preprocessing: collect multimodal data including historical load data, sensor data, user behavior data, weather data, electricity price data, etc., increase the robustness of the data through data interpolation, smoothing and other methods, and convert the data into a format suitable for model input.

[0135] Step 1.2 Feature engineering and feature extraction, use machine learning methods to calculate the importance of each feature and select key features. Use contrastive learning methods to construct a high-dimensional mapping relationship between multi-time scale, multi-modal data and user internal characteristics related features. Conventional load feature extraction cannot reflect the complexity and volatility of the load curve. This embodiment introduces load curve features from the following time series data mining perspectives:

[0136] By binning the values ​​of the load sequence X, the interval [min(X), max(X)] can be equally divided into k small intervals, and the values ​​of the load sequence will be dispersed in these k bins. Based on this equally spaced binning, the entropy of the probability distribution can be calculated.

[0137]

[0138] Where: represents the proportion (probability) of the load sequence values ​​falling into the kth bucket; max bins represents the number of buckets; len(X) represents the length of the time series. This probability distribution is embedded as a state in the following reinforcement learning model.

[0139] Step 1.3 Reinforcement learning model construction, definition of state and action space, first of all, it is necessary to clarify the state space and possible actions of the power load system. The state can include the current power demand, the power generation of renewable energy, the predicted load changes, the amount of power reserves in the energy storage system, etc., while the action can be to adjust the output of the generator set, adjust the operating power of the equipment, or enable reserve power, etc. Suppose there is now a state space S, in which each state represents a certain situation of the environment; an action space A, in which each action represents an action that can be taken in the current state; and a reward function R(s,a), which represents the reward obtained by taking action a in state s. The goal is to find a strategy π(a|s), that is, the probability of choosing action a in state s, so as to maximize the expected cumulative reward. The expected value of the cumulative reward can be expressed as the return value Gt :

[0140]

[0141] Among them, γ is the discount factor, which indicates the importance of future rewards, and r t+k+1 is the instant reward obtained at time t+k+1. In order to realize the dynamic reward mechanism, the reward function R(s,a) can be dynamically adjusted, and the policy gradient method (PolicyGradient) or Q-Learning method can be used to optimize it according to changes in the environment and improvements in the strategy. For example, in the Q-learning method, it is necessary to select the optimal action by estimating the action value function Q(s,a). The goal is to make Q(s,a) satisfy the Bellman equation:

[0142]

[0143] The optimal Q value is approached by iteratively updating the Q value. The update formula of the Q value is:

[0144]

[0145] where α is the learning rate and s′ is the new state that state s transitions to after taking action a.

[0146] Step 1.4 Design a reward function. When the system successfully balances the real-time load and avoids overload, a positive reward is given; conversely, if it causes insufficient power supply or waste of resources, a negative penalty is given. The reward function also needs to consider the economic, stable, and environmentally friendly nature of the action, and consider the grid regulation actions under special circumstances. Positive rewards are given to encourage the use of electricity when the electricity price is low and to encourage the reduction of electricity consumption when the electricity price is high; during peak load periods: increase the reward for reducing the load to encourage users to reduce the peak load; in the event of equipment failure or natural disasters, the reward function is dynamically adjusted to prioritize the stability and security of the grid.

[0147] In the dynamic reward mechanism, the reward function R(s,a) will be dynamically adjusted as the environment changes and the strategy improves. The reward adjustment function ΔR(s,a,t) is introduced to represent the adjustment amount of the reward at time t:

[0148] R(s,a,t)=R(s,a)+ΔR(s,a,t)

[0149] The reward adjustment function ΔR(s,a,t) is designed based on multiple factors. Here, ΔR(s,a,t) is defined as:

[0150]

[0151] Where β is the adjustment coefficient, G tis the actual return at time t, is the average return over the past period of time. In this way, when the strategy performs better than the average, the reward increases; otherwise, the reward decreases.

[0152] The load control method based on reinforcement learning has the following technical effects:

[0153] 1) By dynamically adjusting the reward mechanism, the load control system can adapt to the changing environment and goals, and improve the flexibility and adaptability of the system. In actual operation, the system can improve the robustness of data by collecting and processing multimodal data such as historical load data, user behavior data, and weather data. Based on the reinforcement learning method, a high-dimensional mapping relationship between multi-time scale, multimodal data and user internal characteristics is constructed, and a reasonable reward function is set to achieve real-time load balancing in the system, avoid overload, and improve the economy, stability, and environmental protection of the power grid.

[0154] 2) The multi-objective optimization algorithm under multiple time scales adopts a hierarchical optimization strategy to decompose the optimization problem of multiple time scales into multiple levels, each level corresponds to a different time scale, and is optimized separately. Among them, short-term optimization minimizes short-term operating costs to ensure the stability of the power system; mid-term optimization maximizes the utilization of the energy storage system to ensure power supply reliability; long-term optimization comprehensively considers the long-term service life and maintenance cost of the energy storage equipment in the system to ensure the long-term sustainable development of the system.

[0155] 3) By using the subdivision and management methods of massive demand-side resources, the current load control method has the problem that massive demand-side resources are difficult to control. Through subdivision and management, the level of refinement of load control has been improved. Through the dynamic incentive mechanism based on reinforcement learning and combined with the multi-agent reinforcement learning method, the interaction between load control levels has been enhanced.

[0156] This embodiment also provides a load control system based on reinforcement learning, including a load control module, and the load control module executes the load control method based on reinforcement learning.

[0157] This embodiment also provides a non-transitory computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the load control method based on reinforcement learning is implemented.

[0158] Ginseng Figure 2 As shown, this embodiment also provides an electronic device, including:

[0159] The memory 201 and the processor 202 are communicatively connected with each other, the memory 201 stores computer instructions, and the processor 202 executes the load control method based on reinforcement learning by executing the computer instructions.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A load control method based on reinforcement learning, characterized in that: The steps include: Step 1: In the source-grid-load-storage control scenario, a hierarchical optimization framework is used to decompose the multi-time-scale optimization problem into multiple levels. Each level corresponds to a different time scale and is optimized separately. The model predictive control algorithm is used in the short-term optimization layer, the mixed integer linear programming algorithm is used in the medium-term optimization layer, and the reinforcement learning method is used in the long-term optimization layer to solve the multi-objective optimization problem. Step 2, long-term load control decision-making, based on reinforcement learning through a practically adjustable reward mechanism, controls the power grid to make different load control decisions, so that the smart grid load control system can adapt to the changing environment and goals, and improve the flexibility and adaptability of the system.

2. The load control method based on reinforcement learning according to claim 1 is characterized in that: The step 1 comprises: Step 1.1, divide the time scale: Short-term optimization is based on hours or days, and the control targets include real-time scheduling, load balancing, and instantaneous response. Medium-term optimization is based on weeks or months, and the control targets include periodic scheduling and energy storage management. Long-term optimization is based on years, and the control targets include equipment investment, long-term planning, and strategic decision-making. Step 1.2, define the objective function: Determine the minimum operating cost, benefit target, and reliability target expected by the system, define different objective functions at each time scale, and ensure that the objective functions are consistent and coordinated between different time scales; the minimum operating cost includes power generation cost, energy storage system operating cost, and load management cost; the benefit target includes renewable energy utilization rate and user satisfaction; the reliability target includes the power supply fluctuation range allowed by the system; Step 1.3, define constraints: Ensure the consistency and coordination between the objective functions and constraints at each time scale based on the capacity limitations of the energy storage system, the charging and discharging rate limitations, and the voltage and frequency stability requirements of the power system in the specific scenario; The constraints of distributed photovoltaic output are: (P Pv,i (t)) 2 +(Q Pv,i (t)) 2 ≤(S pv,i,max ) 2 Where P PV,i and Q PV,i are the photovoltaic active and reactive outputs of node i at time t, S PV,i,max is the apparent power upper limit of the photovoltaic node i; The energy storage SOC constraint is: SOC min <SOC(t)<SOC max In the formula, SOC min , SOC max They represent the minimum and maximum charge of the energy storage device respectively, and SOC(t) represents the remaining charge of the energy storage device at time t; When a power grid fails, each virtual power plant in each zone enters island operation. At this time, the power supply rate of each zone is constrained to ensure that the demand of important loads is met. The constraints are as follows: I k (t)≥I k,min ,k=1,2,…,n p In the formula, I k,min I is the proportion of the primary important load of the k-th zone photovoltaic storage virtual power plant; k is the power supply rate of the k-th zone photovoltaic storage virtual power plant; n p is the number of partitions; The stable operation of the power grid also needs to meet the following power flow constraints: U i,min ≤U i (t)≤U i,max |I ij |≤I ij,max Where P mi (t), Q mi (t) and P ik (t), Q ik (t) are the active and reactive power of branch mi and branch ik at time t respectively; R mi is the resistance of branch mi; X mi is the reactance of branch mi; P i (t) and Q i (t) are the active power and reactive power injected by node i at time t; u(i) is the set of head nodes of the branch with node i as the end node; d(i) is the set of end nodes of the branch with node i as the head node; U i (t) is the voltage of node i; U i,min and U i,max are the minimum and maximum voltage of node i respectively; ij and I ij,max are the current flowing through the branch ij and the maximum value of the current allowed to flow through it; N is the number of system nodes; m is the index number of the head node; k is the index number of the end node; Step 1.4: Use the MPC method for real-time scheduling in short-term optimization. At each control moment, based on the current state and the prediction model, predict the behavior of the system in the future. Through the optimization algorithm, find the optimal control sequence in the prediction time domain. According to the optimal control sequence, implement the first control action in the optimized control sequence. At the next moment, roll the window, update the system state, and repeat the prediction and optimization process. In the mid-term optimization, the MILP method is used for monthly scheduling, and the mixed integer linear programming solver is used to solve the problem. The optimal solution that satisfies all constraints is found according to the objective function. The mid-term scheduling plan is generated according to the solution results, including the generator switching state and the energy storage system charging and discharging plan. Long-term optimization uses the long-term load control decision based on reinforcement learning in step 2; Step 1.5, integrated optimization and result analysis: The long-term optimization results are used as constraints for medium-term optimization, and the medium-term optimization results are used as constraints for short-term optimization. Coordinated optimization is performed on multiple time scales. The Pareto frontier of each time scale is generated and visualized. The trade-off between different objectives is analyzed. The appropriate Pareto optimal solution is selected according to the actual operating conditions for scheduling and optimization.

3. The load control method based on reinforcement learning according to claim 2 is characterized in that: In step 1.2, the short-term goal is to minimize the real-time operating cost C short-term , maximize the use of renewable energy RE , ensure the stability of the power system power : Subject to S power (x t )≤Δ Where, Min C short-term To minimize the real-time operating cost objective; Max U RE To maximize the utilization rate of renewable energy; Subject to S power is the power grid power stability fluctuation range constraint; x t It is the short-term load regulation decision in period t; Mid-term goal is to minimize monthly operating costs C mid-term , maximize the utilization rate of energy storage system U storage , ensure power supply reliability supply : Subject to R supply (y t )≤Δ Where, Min C mid-term To minimize mid-term operating costs; Max U storage To maximize the utilization of energy storage system; Subject to R supply is the power supply stability range constraint of the power grid; t is the mid-term load regulation decision for period t; T is the total event length of the mid-term target; t is the basic time period index number within the total duration of the mid-term optimization; c t is the mid-term unit operating cost in period t; Δ is the upper limit of the fluctuation range of power grid stability; Long-term goal is to minimize the overall investment cost C long-ter , maximize system benefits B system , ensuring long-term sustainable development long-term : Subject to S long-ter (z t )≤Δ Where, Min is the goal of minimizing the overall investment cost; Max B system To maximize the system benefit goal; Subject toS long-ter To constrain the volatility range for sustainable development; t For long-term load regulation decisions.

4. The load control method based on reinforcement learning according to claim 3 is characterized in that: The step 2 comprises: Step 2.1, data collection and preprocessing: Collect multimodal data, increase the robustness of the data through data interpolation and smoothing, and convert the data into a format suitable for model input; the multimodal data includes historical load data, sensor data, user behavior data, weather data, and electricity price data; Step 2.2, feature engineering and feature extraction: Use machine learning methods to calculate the importance of each feature, select key features, and use contrastive learning methods to construct a high-dimensional mapping relationship between multi-time scale, multi-modal data and features related to user internal characteristics; Step 2.3, define the state and action space and build a reinforcement learning model: Determine the state space and possible actions of the power load system, wherein the state includes the current power demand, the power generation of renewable energy, the predicted load change, and the power reserve in the energy storage system, and the actions include adjusting the output of the generator set, adjusting the operating power of the equipment, and activating the reserve power; Assume that there is a state space S, where each state represents a certain situation in the environment; an action space A, where each action represents an action that can be taken in the current state; and a reward function R(s,a), which represents the reward obtained by taking action a in state s. The goal is to find a strategy π(a|s), that is, the probability of choosing action a in state s, to maximize the expected cumulative reward. The expected value of the cumulative reward is expressed as the return value G. t : Among them, γ is the discount factor, which indicates the importance of future rewards, and r t+k+1 It is the immediate reward obtained at time t+k+1; Dynamically adjust the reward function R(s,a) and use the policy gradient method or Q learning method to optimize it according to changes in the environment and improvements in the strategy; Step 2.4, design reward function: When the system successfully balances the real-time load and avoids overload, positive rewards are given; conversely, if it causes insufficient power supply or waste of resources, negative penalties are given; Provide positive rewards for behaviors that encourage the use of electricity when prices are low and encourage the use of less electricity when prices are high; During peak load periods: Increase incentives for reducing load to encourage users to reduce peak loads; In the event of equipment failure or natural disasters, the reward function is dynamically adjusted to prioritize grid stability and security; In the dynamic reward mechanism, the reward function R(s,a) is dynamically adjusted as the environment changes and the strategy improves. The reward adjustment function ΔR(s,a,t) is introduced to represent the adjustment amount of the reward at time t: R(s,a,t)=R(s,a)+ΔR(s,a,t) Among them, the reward adjustment function ΔR(s,a,t) is: Where β is the adjustment coefficient, G t is the actual return at time t, is the average return over the past period of time; When the strategy performs better than average, the reward increases; otherwise, the reward decreases.

5. The load control method based on reinforcement learning according to claim 4 is characterized in that: In step 2.2, the load curve characteristics are introduced from the perspective of time series data mining: The values ​​of the load sequence X are binned, and the interval [min(X), max(X)] is equally divided into k small intervals. The values ​​of the load sequence are dispersed in k bins. According to the situation of equally spaced bins, the entropy of the probability distribution is calculated: Where: Indicates the proportion of the load sequence values ​​that fall into the kth bucket; max bins indicates the number of buckets; len(X) indicates the length of the time series: The entropy of the probability distribution is embedded into the reinforcement learning model as a state.

6. The load control method based on reinforcement learning according to claim 5 is characterized in that: In step 2.3, the Q learning method selects the optimal action by estimating the action-value function Q(s,a), with the goal of making Q(s,a) satisfy the Bellman equation: Approach the optimal Q value by iteratively updating the Q value; The update formula of Q value is: where α is the learning rate and s′ is the new state that state s transitions to after taking action a.

7. A load control system based on reinforcement learning, characterized in that: It includes a load control module, which executes a load control method based on reinforcement learning as described in any one of claims 1-6.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, a load control method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the load control method based on reinforcement learning as described in any one of claims 1 to 6 by executing the computer instructions.

Images