Multi-time-scale adaptive control method and system for power distribution network

Abstract

The invention discloses a power distribution network multi-time scale adaptive control method and system, and the method comprises the steps: obtaining the initial state information of a power distribution network, and decomposing a power distribution network control problem into a day-ahead planning layer, an hour-level scheduling layer and a real-time control layer; establishing a layered reinforcement learning framework which is respectively corresponding to a day-ahead planning layer, an hour-level scheduling layer and a real-time control layer; an Actor-Critic architecture is adopted, an action is generated according to the current state, and a state value is evaluated; the method comprises the following steps: based on an Actor-Critic architecture, introducing an experience playback and priority sampling mechanism, and storing and sampling an experience sample; and performing pre-training and fine tuning operation by adopting a parameter migration technology to obtain a multi-level control strategy. According to the method, multi-time-scale coordinated optimization is realized, all levels are coordinated through information transmission and target decomposition, and the consistency and effectiveness of an overall control strategy are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of distribution network control, and in particular to a distribution network multi-time scale adaptive control method and system. Background Art

[0002] With the large-scale access of new power equipment such as distributed energy, electric vehicles and flexible loads, the operating characteristics of distribution networks have changed significantly. Traditional distribution network control methods are mainly based on deterministic models and predictions, which are difficult to effectively deal with the uncertainty and volatility challenges brought by high-penetration distributed resources. In addition, existing control methods often optimize for a single time scale, making it difficult to achieve coordinated optimization of multiple time scales such as day-ahead planning, hourly scheduling and real-time control. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a multi-time scale adaptive control method and system for a distribution network to solve the problem that existing control methods are difficult to achieve coordinated optimization of multiple time scales.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a multi-time scale adaptive control method for a distribution network, comprising:

[0008] Obtain the initial state information of the distribution network and decompose the distribution network control problem into the day-ahead planning layer, the hourly scheduling layer and the real-time control layer;

[0009] Establish a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, hourly scheduling layer, and real-time control layer;

[0010] Based on the reinforcement learning framework, the Actor-Critic architecture is adopted to generate actions and evaluate state values ​​according to the current state;

[0011] Based on the Actor-Critic architecture, the experience replay and priority sampling mechanisms are introduced to store and sample experience samples;

[0012] Based on the experience replay and priority sampling mechanism, parameter migration technology is used to perform pre-training and fine-tuning operations to obtain a multi-level control strategy.

[0013] As a preferred solution of the multi-time scale adaptive control method for distribution network of the present invention, wherein:

[0014] The day-ahead planning layer includes using the day-ahead forecast information to formulate a distribution network operation plan within 24 hours. The decision cycle of the day-ahead planning layer is 24 hours, and the decision interval is 1 hour;

[0015] The hourly scheduling layer includes using short-term forecast information to adjust the day-ahead planning scheme. The decision cycle of the hourly scheduling layer is 1 hour and the decision interval is 15 minutes.

[0016] The real-time control layer includes sending control instructions to each controllable resource in the distribution network based on the hourly scheduling results and executing control actions. The decision cycle of the real-time control layer is 5 minutes and the decision interval is 1 minute.

[0017] As a preferred solution of the multi-time scale adaptive control method for distribution network of the present invention, wherein:

[0018] The hierarchical reinforcement learning framework is established, corresponding to the day-ahead planning layer, the hourly scheduling layer and the real-time control layer, respectively, including the first agent corresponding to the day-ahead planning layer, the second agent corresponding to the hourly scheduling layer, and the third agent corresponding to the real-time control layer, and each agent is designed with a corresponding state space, action space and reward function;

[0019] The state space of the first agent is expressed as:

[0020] S d ={L d , G d , P d , Q d , C d}

[0021] Among them, L d is the day-ahead load forecast, G d is the forecast of renewable energy generation on the day before, P d , Q d are the day-ahead active and reactive power exchange predictions, C d Forecast of day-ahead electricity prices;

[0022] The state space of the second agent is expressed as:

[0023] S h ={L h , G h , P h , Q h , V h , SOC h}

[0024] Among them, L h is the hourly load, G h For hourly renewable energy generation, P h , Q h are hourly active and reactive power exchange, V h is the voltage state, SOC h The state of charge of the energy storage system;

[0025] The state space of the third agent is expressed as:

[0026] S r ={L r , G r , P r , Q r , V r , f r}

[0027] Among them, L r is the real-time load, G r For real-time renewable energy generation, P r , Q r are real-time active and reactive power exchange, V r is the real-time voltage status, f r is the real-time frequency;

[0028] The action space of the first agent is expressed as:

[0029]

[0030] in, are the day-ahead active and reactive power exchange plans, Day-ahead dispatch plan for energy storage system;

[0031] The action space of the second agent is expressed as:

[0032]

[0033] in, They are hourly active and reactive power regulation, is the voltage regulation instruction, Provides charging and discharging instructions for energy storage systems;

[0034] The action space of the third agent is expressed as:

[0035]

[0036] in, They are real-time active and reactive power regulation instructions, It is the real-time voltage regulation instruction;

[0037] The reward function of the second agent is expressed as:

[0038] R h =w1R recon +w2R reli +w3R env

[0039] Among them, P recon , R reli , R env They represent economic incentives, reliability incentives and environmental incentives respectively, and w1, w2, w3 are the corresponding weight coefficients.

[0040] As a preferred solution of the multi-time scale adaptive control method for distribution network of the present invention, wherein:

[0041] The Actor-Critic architecture is used to generate actions and evaluate state values ​​according to the current state, including the following steps:

[0042] Observe the new state of the distribution network and calculate the reward value;

[0043] Use the Actor network to generate actions based on the current state;

[0044] Use the Critic network to evaluate the current state value.

[0045] As a preferred solution of the multi-time scale adaptive control method for distribution network of the present invention, wherein:

[0046] The introduction of the experience playback and priority sampling mechanism to store and sample the experience samples includes the following steps:

[0047] Store the experience samples generated by each interaction into the experience pool;

[0048] Sample a batch of samples from the experience pool based on priority;

[0049] Update the Actor-Critic network parameters;

[0050] Update the priority of experience samples.

[0051] As a preferred solution of the multi-time scale adaptive control method for distribution network of the present invention, wherein:

[0052] The parameter migration technology is used to perform pre-training and fine-tuning operations, including the following steps:

[0053] Select a source task that has some relevance to the target distribution network scenario and pre-train a deep learning model for this task on a standard IEEE test system;

[0054] After pre-training, the learned network parameters are transferred to the model under the target distribution network scenario;

[0055] Continue to use experience replay and priority sampling mechanisms in the target distribution network scenario to fine-tune the model and optimize network parameters.

[0056] As a preferred solution of the multi-time scale adaptive control method for distribution network of the present invention, wherein:

[0057] The fine-tuning of the model includes defining a total loss function;

[0058] The total loss function of the fine-tuning stage is expressed as:

[0059] L=L task +λL transfer

[0060] Among them, L task is the loss function of the target task, L transfer is the migration regularization term, and λ is the trade-off coefficient.

[0061] In a second aspect, the present invention provides a distribution network multi-time scale adaptive control system, comprising:

[0062] The multi-time scale control problem decomposition module is used to obtain the initial state information of the distribution network and decompose the distribution network control problem into the day-ahead planning layer, the hourly scheduling layer and the real-time control layer;

[0063] Hierarchical reinforcement learning framework building module, used to establish a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, hourly scheduling layer and real-time control layer;

[0064] Actor-Critic architecture implementation module, which is used to generate actions and evaluate state values ​​based on the current state using the Actor-Critic architecture based on the reinforcement learning framework;

[0065] The experience replay and priority sampling module is used to introduce the experience replay and priority sampling mechanism based on the Actor-Critic architecture to store and sample experience samples;

[0066] The parameter migration module is used to perform pre-training and fine-tuning operations based on experience replay and priority sampling mechanism, and obtain a multi-level control strategy by adopting parameter migration technology.

[0067] In a third aspect, the present invention provides a computing device, comprising:

[0068] Memory, used to store programs;

[0069] A processor is used to execute the computer executable instructions, which, when executed by the processor, implement the steps of the distribution network multi-time scale adaptive control method.

[0070] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising: when the program is executed by a processor, the steps of implementing the multi-time-scale adaptive control method for the distribution network are implemented.

[0071] Beneficial effects of the present invention: The present invention adopts a hierarchical reinforcement learning framework to decompose the distribution network control problem into three levels: day-ahead planning, hourly scheduling, and real-time control. By designing the corresponding state space, action space, and reward function, the coordinated optimization of multiple time scales is achieved. The coordination between the various levels is achieved through information transmission and target decomposition, ensuring the consistency and effectiveness of the overall control strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0073] Figure 1 A schematic diagram of the basic flow of a multi-time-scale adaptive control method for a distribution network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0075] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0076] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0077] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0078] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0079] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0080] Example 1

[0081] Reference Figure 1 , is an embodiment of the present invention, and provides a multi-time scale adaptive control method for a distribution network, such as Figure 1 As shown, the following steps are included:

[0082] S1: Obtain the initial state information of the distribution network and decompose the distribution network control problem into the day-ahead planning layer, the hourly scheduling layer and the real-time control layer;

[0083] In the embodiment of the present application, the distribution network control problem is decomposed into three levels: day-ahead planning, hourly scheduling, and real-time control:

[0084] (1) Day-ahead planning layer: Based on the day-ahead forecast information, the 24-hour operation plan of the distribution network is planned, including the distributed generation output plan, energy storage charging and discharging plan, etc. The decision cycle is 24 hours and the decision interval is 1 hour.

[0085] (2) Hourly dispatching layer: Based on short-term forecast information, the day-ahead planning scheme is adjusted to achieve coordinated optimization of various resources in the distribution network. The decision cycle is 1 hour and the decision interval is 15 minutes.

[0086] (3) Real-time control layer: Based on real-time measurement data, various controllable resources of the distribution network are controlled in real time to ensure safe and stable operation of the system. The decision cycle is 5 minutes and the decision interval is 1 minute.

[0087] S2: Establish a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, hourly scheduling layer, and real-time control layer;

[0088] In the embodiment of the present application, a hierarchical reinforcement learning framework is established, which includes three levels of reinforcement learning agents, corresponding to the day-ahead planning layer, the hourly scheduling layer and the real-time control layer, respectively, including a first agent corresponding to the day-ahead planning layer, a second agent corresponding to the hourly scheduling layer, and a third agent corresponding to the real-time control layer, and each agent is designed with a corresponding state space, action space and reward function;

[0089] S2.1: State-space design

[0090] (1) The state space of the first agent is expressed as:

[0091] S d ={L d ,G d ,P d ,Q d ,C d}

[0092] Among them, L d is the day-ahead load forecast, G d is the forecast of renewable energy generation on the day before, P d ,Q d are the day-ahead active and reactive power exchange predictions, C d The day-ahead electricity price forecast.

[0093] (2) The state space of the second agent is expressed as:

[0094] S h ={L h ,G h ,P h ,Q h ,V h ,SOC h}

[0095] Among them, L h is the hourly load, G h For hourly renewable energy generation, P h ,Q h are hourly active and reactive power exchange, V h is the voltage state, SOC h It is the charge state of the energy storage system.

[0096] (3) The state space of the third agent is expressed as:

[0097] S r ={L r ,G r ,P r ,Q r ,V r ,f r}

[0098] Among them, L r is the real-time load, G r For real-time renewable energy generation, P r ,Q r are real-time active and reactive power exchange, V r is the real-time voltage status, f r is the real-time frequency.

[0099] S2.2: Action Space Design

[0100] (1) The action space of the first agent is expressed as:

[0101]

[0102] in, are the day-ahead active and reactive power exchange plans, It is the day-ahead dispatch plan for the energy storage system.

[0103] (2) The action space of the second agent is expressed as:

[0104]

[0105] in, They are hourly active and reactive power regulation, is the voltage regulation instruction, Provides charging and discharging instructions for the energy storage system.

[0106] (3) The action space of the third agent is expressed as:

[0107]

[0108] in, They are real-time active and reactive power regulation instructions, It is the real-time voltage regulation instruction.

[0109] S2.3: Reward function design

[0110] In the embodiment of the present application, in order to achieve multi-objective optimization, a comprehensive reward function that considers economy, reliability and environmental protection is designed. Taking the hourly scheduling layer as an example, the reward function of the first agent is defined as follows:

[0111] R h =w1R recon +w2R reli +w3R env

[0112] Among them, P recon , R reli , R env They represent economic incentives, reliability incentives and environmental incentives respectively, and w1, w2, w3 are the corresponding weight coefficients.

[0113] In the embodiment of the present application, the economic reward R recon Defined as:

[0114]

[0115] Among them, C e is the electricity price, P h is the actual power exchange, C l is the deviation penalty coefficient, Plan power exchanges for the day ahead.

[0116] In the embodiment of the present application, the reliability reward R reli Defined as:

[0117] R reli =-∑(α|V h -V ref | 2 +β|f h -f ref | 2 )

[0118] Among them, V h and f h are the actual voltage and frequency, V ref and f ref reference value, α and β are weight coefficients.

[0119] In the embodiment of the present application, the environmental protection reward R env Defined as:

[0120] R env =-∑(γE h )

[0121] Among them, E h is the carbon emission, and γ is the carbon emission penalty coefficient.

[0122] S3: Based on the reinforcement learning framework, it adopts the Actor-Critic architecture to generate actions and evaluate state values ​​according to the current state;

[0123] In an embodiment of the present application, a deep neural network is used to approximate a value function or a policy function, that is, an Actor-Critic architecture is used.

[0124] It should be noted that the Actor-Critic network is an algorithm framework commonly used in reinforcement learning (RL). It combines two types of learning strategies: policy learning (Actor) and value function learning (Critic).

[0125] It should be noted that in the Actor-Critic framework, the Actor is responsible for generating actions. It is a policy network that directly maps from the state space to the action space, that is, given the current state, the Actor decides which action to take, and the Actor's goal is to maximize the cumulative reward.

[0126] It should be noted that the Critic is responsible for evaluating the actions taken by the Actor. It is a value network that estimates the expected return of taking an action in a given state, that is, the value function. The Critic provides feedback on the quality of the current strategy to help the Actor make adjustments.

[0127] It should be noted that the basic idea of ​​the Actor-Critic algorithm is:

[0128] Actor learning policy: Actor network learns a policy that maximizes the cumulative reward by trying different actions and observing the feedback from the environment.

[0129] Critic learns value function: The critic network learns a value function that estimates the long-term reward of taking an action in any given state.

[0130] Interaction between strategy and value function: Actor and Critic networks are usually trained simultaneously. The value estimate provided by Critic is used to update Actor to improve the strategy. At the same time, the output of Actor (i.e. action) is used to update Critic to estimate the value function more accurately.

[0131] In the embodiment of the present application, an Actor-Critic architecture is adopted to generate an action according to the current state and evaluate the state value, including the following steps:

[0132] Observe the new state of the distribution network and calculate the reward value;

[0133] Use the Actor network to generate actions based on the current state;

[0134] Use the Critic network to evaluate the current state value.

[0135] In the embodiment of the present application, the Actor network is used to generate actions. Taking the hourly scheduling layer as an example, its structure is as follows:

[0136] Input layer: state vector S h ;

[0137] Hidden layer 1: 64 neurons, ReLU activation function;

[0138] Hidden layer 2: 32 neurons, ReLU activation function;

[0139] Output layer: action vector A h , tanh activation function;

[0140] The Critic network is used to evaluate the state value, and its structure is as follows:

[0141] Input layer: state vector S h and the action vector A h splicing;

[0142] Hidden layer 1: 64 neurons, ReLU activation function;

[0143] Hidden layer 2: 32 neurons, ReLU activation function;

[0144] Output layer: scalar Q value.

[0145] S4: Based on the Actor-Critic architecture, the experience replay and priority sampling mechanisms are introduced to store and sample experience samples;

[0146] In the embodiments of the present application, in order to improve learning efficiency, an experience replay mechanism and a priority sampling method are introduced.

[0147] In the embodiment of the present application, an experience playback and priority sampling mechanism is introduced to store and sample experience samples, including the following steps:

[0148] Store the experience samples generated by each interaction into the experience pool;

[0149] Sample a batch of samples from the experience pool based on priority;

[0150] Update the Actor-Critic network parameters;

[0151] Update the priority of experience samples.

[0152] It should be noted that the experience replay mechanism is a technology used for offline learning, which improves learning effects by storing and reusing historical data. The idea of ​​experience replay is to store these experiences in an experience pool (replay buffer), and then randomly extract a part of the samples for reuse during the training process.

[0153] It should be noted that the priority sampling method is an improvement on the experience replay mechanism, which not only considers random sampling, but also the importance of different experiences. In standard experience replay, samples are randomly drawn from the experience pool, while in priority sampling, the probability of drawing a sample is related to its importance.

[0154] It should be noted that importance is usually measured by the TD error (Temporal Difference Error) of the experience. The larger the TD error, the greater the difference between the experience and the current model's prediction, and therefore it is more worth learning. The priority sampling method weights the samples according to the TD error, so that those experiences with larger errors are more likely to be selected.

[0155] The experience replay buffer D stores the transfer samples (s t ,a t ,r t ,s t+1 ). The probability of priority sampling is defined as:

[0156]

[0157] Among them, δ i is the TD error, ε is a small positive number, and α is the priority index.

[0158] S5: Based on the experience replay and priority sampling mechanism, parameter migration technology is used to perform pre-training and fine-tuning operations to obtain a multi-level control strategy.

[0159] In the embodiments of the present application, transfer learning technology is used to improve the generalization ability of the model.

[0160] It should be noted that transfer learning is a machine learning technique that aims to transfer learned knowledge from one or more source tasks to a related but different target task. The basic idea of ​​transfer learning is to use the knowledge in the source task to accelerate the learning of the target task. This usually involves the following aspects:

[0161] Feature transfer: Features extracted from the source task can be directly used in the target task.

[0162] Parameter migration: The model parameters in the source task can be used to initialize the model in the target task, thus avoiding training from scratch.

[0163] Representation transfer: The intermediate layer representation in the source task can be used in the target task to help the target task better understand the input data.

[0164] In the embodiment of the present application, the parameter migration technology is used to perform pre-training and fine-tuning operations, including the following steps:

[0165] Select a source task that has some relevance to the target distribution network scenario and pre-train a deep learning model for this task on a standard IEEE test system;

[0166] After pre-training, the learned network parameters are transferred to the model under the target distribution network scenario;

[0167] Continue to use experience replay and priority sampling mechanisms in the target distribution network scenario to fine-tune the model and optimize network parameters.

[0168] In the embodiment of the present application, the loss function of transfer learning is defined as:

[0169] L=L tas k+λL transfer

[0170] Among them, L task is the loss function of the target task, L transfer is the migration regularization term, and λ is the trade-off coefficient.

[0171] In an embodiment of the present application, the iterative process of controlling the entire reinforcement learning algorithm includes determining whether a termination condition is met. If not, returning to execute the day-ahead planning layer decision; otherwise, terminating the algorithm.

[0172] This embodiment also provides a distribution network multi-time scale adaptive control system, including:

[0173] The multi-time scale control problem decomposition module is used to obtain the initial state information of the distribution network and decompose the distribution network control problem into the day-ahead planning layer, the hourly scheduling layer and the real-time control layer;

[0174] Hierarchical reinforcement learning framework building module, used to establish a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, hourly scheduling layer and real-time control layer;

[0175] Actor-Critic architecture implementation module, which is used to generate actions and evaluate state values ​​based on the current state using the Actor-Critic architecture based on the reinforcement learning framework;

[0176] The experience replay and priority sampling module is used to introduce the experience replay and priority sampling mechanism based on the Actor-Critic architecture to store and sample experience samples;

[0177] The parameter migration module is used to perform pre-training and fine-tuning operations based on experience replay and priority sampling mechanism, and obtain a multi-level control strategy by adopting parameter migration technology.

[0178] Furthermore, it also includes:

[0179] Memory, used to store programs;

[0180] A processor is used to load the program to execute the distribution network multi-time scale adaptive control method.

[0181] This embodiment also provides a computer-readable storage medium storing a program, and when the program is executed by a processor, the distribution network multi-time scale adaptive control method is implemented.

[0182] The storage medium proposed in this embodiment and the distribution network multi-time scale adaptive control method proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0183] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0184] Example 2

[0185] This is an embodiment of the present invention, which provides a multi-time scale adaptive control system for a distribution network, including a multi-time scale control problem decomposition module, a hierarchical reinforcement learning framework construction module, an Actor-Critic architecture implementation module, an experience playback and priority sampling module, and a parameter migration module;

[0186] In the embodiment of the present application, the multi-time scale control problem decomposition module includes obtaining the initial state information of the distribution network, and decomposing the distribution network control problem into a day-ahead planning layer, an hourly scheduling layer, and a real-time control layer;

[0187] In an embodiment of the present application, the multi-time scale control problem decomposition module also includes a module responsible for collecting initial state information of the distribution network, including but not limited to the grid structure, load demand, power supply status, etc., to provide basic data support for subsequent control decisions.

[0188] In an embodiment of the present application, the multi-time scale control problem decomposition module also includes decomposing the complex distribution network control problem into a day-ahead planning layer, an hourly scheduling layer and a real-time control layer according to the time scale, each layer is responsible for optimizing decisions within different time ranges, thereby realizing refined management of the distribution network.

[0189] In the embodiment of the present application, the hierarchical reinforcement learning framework construction module includes establishing a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, the hourly scheduling layer, and the real-time control layer respectively;

[0190] In an embodiment of the present application, the hierarchical reinforcement learning framework building module also includes establishing a hierarchical reinforcement learning (HRL) framework, which consists of multiple levels, each layer corresponds to a control layer defined in the previous step, and works together by guiding the lower levels through the high-level level to jointly solve the distribution network control problem.

[0191] In the embodiment of the present application, the Actor-Critic architecture implementation module includes generating actions and evaluating state values ​​according to the current state based on the reinforcement learning framework and using the Actor-Critic architecture;

[0192] In the embodiment of the present application, the Actor-Critic architecture implementation module is also included in each control layer, using the Actor-Critic architecture to generate actions and evaluate state values. Among them, the Actor part is responsible for outputting the optimal action based on the currently observed state, and the Critic part is responsible for evaluating the quality of the current strategy, that is, giving an estimate of the value of taking a certain action under the current state.

[0193] In the embodiment of the present application, the experience replay and priority sampling module includes an experience replay and priority sampling mechanism based on the Actor-Critic architecture to store and sample experience samples;

[0194] In the embodiment of the present application, the experience replay and priority sampling module also includes the introduction of experience replay and priority sampling mechanisms, which are used to store and efficiently reuse historical experience data to improve learning efficiency and model generalization ability. This module can ensure that the model pays more attention to important experience samples during the learning process.

[0195] In an embodiment of the present application, the parameter migration module includes a parameter migration technology based on experience replay and priority sampling mechanism, and performs pre-training and fine-tuning operations to obtain a multi-level control strategy.

[0196] In the embodiment of the present application, the parameter migration module also includes pre-training and fine-tuning the model in combination with parameter migration technology, experience playback, and priority sampling mechanism, especially fine-tuning in the target distribution network scenario to obtain an efficient solution suitable for multi-level control strategies. This module adjusts the model parameters to enable it to quickly adapt to the new task environment while maintaining a good inheritance of existing knowledge.

[0197] In the embodiment of the present application, through the synergy of these modules, a powerful distribution network intelligent control system can be constructed to effectively cope with the complexity and uncertainty challenges of the power system.

[0198] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-time scale adaptive control method for a distribution network, characterized in that: include: Obtain the initial state information of the distribution network and decompose the distribution network control problem into the day-ahead planning layer, the hourly scheduling layer and the real-time control layer; Establish a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, hourly scheduling layer, and real-time control layer; Based on the reinforcement learning framework, the Actor-Critic architecture is adopted to generate actions and evaluate state values ​​according to the current state; Based on the Actor-Critic architecture, the experience replay and priority sampling mechanisms are introduced to store and sample experience samples; Based on the experience replay and priority sampling mechanism, parameter migration technology is used to perform pre-training and fine-tuning operations to obtain a multi-level control strategy.

2. The multi-time scale adaptive control method for distribution network according to claim 1, characterized in that: The day-ahead planning layer includes using the day-ahead forecast information to formulate a distribution network operation plan within 24 hours. The decision cycle of the day-ahead planning layer is 24 hours, and the decision interval is 1 hour; The hourly scheduling layer includes using short-term forecast information to adjust the day-ahead planning scheme. The decision cycle of the hourly scheduling layer is 1 hour and the decision interval is 15 minutes. The real-time control layer includes sending control instructions to each controllable resource in the distribution network based on the hourly scheduling results and executing control actions. The decision cycle of the real-time control layer is 5 minutes and the decision interval is 1 minute.

3. The multi-time scale adaptive control method for distribution network according to claim 1 or 2, characterized in that: The hierarchical reinforcement learning framework is established, corresponding to the day-ahead planning layer, the hourly scheduling layer and the real-time control layer, respectively, including the first agent corresponding to the day-ahead planning layer, the second agent corresponding to the hourly scheduling layer, and the third agent corresponding to the real-time control layer, and each agent is designed with a corresponding state space, action space and reward function; The state space of the first agent is expressed as: S d ={L d ,G d ,P d ,Q d ,C d } Among them, L d is the day-ahead load forecast, G d is the forecast of renewable energy generation on the day before, P d , Q d are the day-ahead active and reactive power exchange predictions, C d Forecast of day-ahead electricity prices; The state space of the second agent is expressed as: S h ={L h ,G h ,P h ,Q h ,V h , SOC h } Among them, L h is the hourly load, G h For hourly renewable energy generation, P h , Q h are hourly active and reactive power exchange, V h is the voltage state, SOC h is the charge state of the energy storage system; The state space of the third agent is expressed as: S r ={L r ,G r ,P r ,Q r ,V r ,f r } Among them, L r is the real-time load, G r For real-time renewable energy generation, P r , Q r are real-time active and reactive power exchange, V r is the real-time voltage status, f r is the real-time frequency; The action space of the first agent is expressed as: in, are the day-ahead active and reactive power exchange plans, Day-ahead dispatch plan for energy storage system; The action space of the second agent is expressed as: in, They are hourly active and reactive power regulation, is the voltage regulation instruction, Provides charging and discharging instructions for energy storage systems; The action space of the third agent is expressed as: in, They are real-time active and reactive power regulation instructions, It is the real-time voltage regulation instruction; The reward function of the second agent is expressed as: R h =w1R recon +w2R reli +w3R env Among them, R recon , R reli , R env They represent economic incentives, reliability incentives and environmental incentives respectively, and w1, w2, w3 are the corresponding weight coefficients.

4. The multi-time scale adaptive control method for distribution network according to claim 3, characterized in that: The Actor-Critic architecture is used to generate actions and evaluate state values ​​according to the current state, including the following steps: Observe the new state of the distribution network and calculate the reward value; Use the Actor network to generate actions based on the current state; Use the Critic network to evaluate the current state value.

5. The multi-time scale adaptive control method for distribution network according to claim 4, characterized in that: The introduction of the experience playback and priority sampling mechanism to store and sample the experience samples includes the following steps: Store the experience samples generated by each interaction into the experience pool; Sample a batch of samples from the experience pool based on priority; Update the Actor-Critic network parameters; Update the priority of experience samples.

6. The multi-time scale adaptive control method for distribution network according to claim 5, characterized in that: The parameter migration technology is used to perform pre-training and fine-tuning operations, including the following steps: Select a source task that has some relevance to the target distribution network scenario and pre-train a deep learning model for this task on a standard IEEE test system; After pre-training, the learned network parameters are transferred to the model under the target distribution network scenario; Continue to use experience replay and priority sampling mechanisms in the target distribution network scenario to fine-tune the model and optimize network parameters.

7. The multi-time scale adaptive control method for distribution network according to claim 6, characterized in that: The fine-tuning of the model includes defining a total loss function; The total loss function of the fine-tuning stage is expressed as: L=L task +λL transfer Among them, L task is the loss function of the target task, L transfer is the migration regularization term, and λ is the trade-off coefficient.

8. A system based on the multi-time scale adaptive control method for distribution network according to claim 1, characterized in that: The multi-time scale control problem decomposition module is used to obtain the initial state information of the distribution network and decompose the distribution network control problem into the day-ahead planning layer, the hourly scheduling layer and the real-time control layer; Hierarchical reinforcement learning framework building module, used to establish a hierarchical reinforcement learning framework, corresponding to the day-ahead planning layer, hourly scheduling layer and real-time control layer; Actor-Critic architecture implementation module, which is used to generate actions and evaluate state values ​​based on the current state using the Actor-Critic architecture based on the reinforcement learning framework; The experience replay and priority sampling module is used to introduce the experience replay and priority sampling mechanism based on the Actor-Critic architecture to store and sample experience samples; The parameter migration module is used to perform pre-training and fine-tuning operations based on experience replay and priority sampling mechanism, and obtain a multi-level control strategy by adopting parameter migration technology.

9. An electronic device, characterized in that: include: Memory, used to store programs; A processor, used to load the program to execute the steps of the distribution network multi-time scale adaptive control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the steps of the distribution network multi-time scale adaptive control method as described in any one of claims 1 to 7 are implemented.

Images