A distribution network scheduling method and system based on deep reinforcement learning

Abstract

The present invention relates to the field of distribution network technology, and discloses a distribution network dispatching method and system based on deep reinforcement learning. The method comprises: clustering and partitioning a target distribution network to obtain a plurality of distribution network sub-areas; performing numerical operations on operating status monitoring data to obtain a state vector of the corresponding distribution network sub-area; training a pre-constructed first deep reinforcement learning model to obtain a second deep reinforcement learning model of the corresponding distribution network sub-area; evaluating each second deep reinforcement learning model to obtain an evaluation result of the corresponding second deep reinforcement learning model; updating each second deep reinforcement learning model according to a global aggregation parameter to obtain a third deep reinforcement learning model of the corresponding distribution network sub-area; and inputting each real-time state vector into the corresponding third deep reinforcement learning model for strategy mapping to obtain a dispatching strategy for each distribution network sub-area. The present invention achieves a dual improvement in the reliability and real-time performance of distribution network dispatching.

Description

Technical Field

[0001] The present invention relates to the technical field of distribution network technology, and in particular to a distribution network scheduling method and system based on deep reinforcement learning. Background Art

[0002] With the increase in the proportion of distributed energy grid connection in distribution networks and the complexity of load structure, traditional rule-based scheduling methods are difficult to adapt to the dynamics and uncertainty of distribution network operation.

[0003] At present, the main problems in distribution network scheduling are as follows: 1) Traditional clustering and partitioning methods are mostly based on static load distribution and fail to fully integrate historical fault data for reliability optimization, resulting in insufficient regional mutual assistance capabilities during faults; 2) Single reinforcement learning models experience a dramatic increase in dimensions in multi-region scheduling, resulting in excessive decision-making delays; 3) Centralized training models require unified iteration after collecting data from the entire network. In scenarios where renewable energy output fluctuates, model parameters lag behind changes in grid status.

[0004] To address these issues, existing technologies attempt to implement zoned scheduling through distributed reinforcement learning. However, these technologies suffer from shortcomings such as inconsistent state vector dimensions and insufficient cross-regional strategy coordination. Therefore, an innovative scheduling method is urgently needed to improve the reliability and real-time performance of distribution network scheduling. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a distribution network scheduling method and system based on deep reinforcement learning.

[0006] In a first aspect, an embodiment of the present invention provides a distribution network scheduling method based on deep reinforcement learning, comprising:

[0007] Acquire historical fault distribution data and regional load peak data of a target distribution network, and cluster and partition the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions;

[0008] Acquiring operation status monitoring data of each of the distribution network sub-areas, and performing numerical calculations on the operation status monitoring data to obtain a state vector corresponding to the distribution network sub-area;

[0009] Based on each of the state vectors, a pre-built first deep reinforcement learning model is trained to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area;

[0010] Evaluate each of the second deep reinforcement learning models based on a pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model;

[0011] Based on each of the evaluation results, parameters of each of the second deep reinforcement learning models are aggregated to obtain a global aggregate parameter, and each of the second deep reinforcement learning models is updated according to the global aggregate parameter to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area;

[0012] Obtain a real-time state vector of each of the distribution network sub-areas, and input each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a scheduling strategy for each of the distribution network sub-areas.

[0013] Preferably, the acquiring of historical fault distribution data and regional load peak data of the target distribution network, and clustering and partitioning the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions, includes:

[0014] Obtaining historical fault distribution data of the target distribution network, and calculating the fault distribution density index based on the historical fault distribution data;

[0015] Acquiring regional load peak data of the target distribution network, and calculating the regional load peak data to obtain an offset characteristic parameter;

[0016] Based on the fault distribution density index and in combination with the topological partitioning constraints of the target distribution network, clustering the target distribution network to obtain an initial partitioning result;

[0017] The initial partitioning result is modified based on the offset characteristic parameter to obtain a plurality of distribution network sub-areas.

[0018] Preferably, the initial partitioning result is modified based on the offset characteristic parameter to obtain several distribution network sub-areas, including:

[0019] If the offset characteristic parameter is greater than a preset threshold, the boundaries of the initial partitioning result are adjusted to obtain a plurality of distribution network sub-areas.

[0020] Preferably, the step of training a pre-built first deep reinforcement learning model based on each state vector to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area includes:

[0021] Constructing a first deep reinforcement learning model based on a deep deterministic policy gradient algorithm, wherein the first deep reinforcement learning model includes an actor network and a critic network, the actor network is used to generate actions, and the critic network is used to evaluate the value of the actions;

[0022] Input each of the state vectors into the actor network for strategy mapping, and output a corresponding energy control action;

[0023] Input each of the energy control actions into the critic network for value evaluation, and output a corresponding value evaluation result;

[0024] Based on each of the value assessment results, the parameters of the first deep reinforcement learning model are updated respectively to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area.

[0025] Preferably, the evaluating each second deep reinforcement learning model based on the pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model includes:

[0026] A validation dataset containing extreme scenarios of the target distribution network is constructed based on a generative adversarial network.

[0027] Performing a state vector validity evaluation on the verification data set to obtain a first evaluation result of each of the second deep reinforcement learning models;

[0028] Performing a dynamic action response evaluation on each of the second deep reinforcement learning models based on the validation data set to obtain a second evaluation result corresponding to the second deep reinforcement learning model;

[0029] Perform weighted fusion on each of the first evaluation results and each of the second evaluation results to obtain an evaluation result corresponding to the second deep reinforcement learning model.

[0030] Preferably, the performing a dynamic action response evaluation on each of the second deep reinforcement learning models based on the validation data set to obtain a second evaluation result corresponding to the second deep reinforcement learning model includes:

[0031] Based on the verification data set, Markov decision process theory is used to perform dynamic action response evaluation on each of the second deep reinforcement learning models to obtain a second evaluation result corresponding to the second deep reinforcement learning model.

[0032] Preferably, the step of performing parameter aggregation on each of the second deep reinforcement learning models based on each of the evaluation results to obtain a global aggregation parameter, and updating each of the second deep reinforcement learning models according to the global aggregation parameter to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area, including:

[0033] Using the softmax function to convert each of the evaluation results into a corresponding aggregation weight;

[0034] Performing parameter weighted aggregation on each of the second deep reinforcement learning models based on each of the aggregation weights to obtain a global aggregation parameter;

[0035] Based on the global aggregation parameters, parameters of each of the second deep reinforcement learning models are fused to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area.

[0036] Preferably, obtaining the real-time state vector of each of the distribution network sub-areas and inputting each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain the scheduling strategy of each of the distribution network sub-areas includes:

[0037] Obtaining a real-time state vector of each of the distribution network sub-areas, and inputting each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a first energy control action for each of the distribution network sub-areas;

[0038] Acquire real-time topology information of each of the distribution network sub-areas, and adjust each of the first energy control actions based on each of the real-time topology information to obtain a second energy control action corresponding to the distribution network sub-area;

[0039] Based on each of the second energy regulation actions, a scheduling strategy corresponding to the distribution network sub-area is generated.

[0040] Preferably, the acquiring of the real-time topology information of each of the distribution network sub-areas, and adjusting each of the first energy control actions based on each of the real-time topology information to obtain a second energy control action corresponding to the distribution network sub-area, includes:

[0041] Acquiring real-time topological information of each of the distribution network sub-areas;

[0042] A feasibility analysis is performed on each first energy regulation action based on each piece of real-time topology structure information, and infeasible first energy regulation actions are adjusted to obtain second energy regulation actions corresponding to the distribution network sub-area.

[0043] In a second aspect, an embodiment of the present invention provides a distribution network dispatching system based on deep reinforcement learning, including:

[0044] an acquisition module, configured to acquire historical fault distribution data and regional load peak data of a target distribution network, and cluster and partition the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions;

[0045] a calculation module, configured to obtain operation status monitoring data of each of the distribution network sub-areas, and perform numerical calculations on the operation status monitoring data to obtain a state vector corresponding to the distribution network sub-area;

[0046] A training module, configured to train the pre-built first deep reinforcement learning model based on each of the state vectors to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area;

[0047] An evaluation module, configured to evaluate each of the second deep reinforcement learning models based on a pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model;

[0048] an updating module, configured to perform parameter aggregation on each of the second deep reinforcement learning models based on each of the evaluation results to obtain a global aggregation parameter, and to update each of the second deep reinforcement learning models according to the global aggregation parameter to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area;

[0049] A scheduling module is used to obtain a real-time state vector of each of the distribution network sub-areas, and input each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a scheduling strategy for each of the distribution network sub-areas.

[0050] Compared with the prior art, the distribution network dispatching method and system based on deep reinforcement learning in the embodiment of the present invention has the following beneficial effects: clustering and partitioning are performed based on historical fault distribution data, so that the sub-region division of the distribution network matches the distribution of high-incidence fault areas. Compared with the traditional static partitioning method, it effectively improves the regional mutual assistance ability in the event of a fault and avoids the chain power outage phenomenon; numerical calculations are performed on the operating status monitoring data to generate state vectors, and combined with the distributed training framework, it solves the problem of dimensionality increase in multi-region scheduling, greatly reduces the action space dimension, shortens the decision delay time, and meets the real-time scheduling requirements of the distribution network; through pre-constructed model training and parameter aggregation mechanism, the model update cycle is greatly shortened, and the model's adaptability to dynamic changes in the distribution network is significantly improved. At the same time, the verification data set evaluation and cross-region parameter aggregation strategy improve the generalization performance of the model. The present invention achieves a dual improvement in the reliability and real-time performance of distribution network dispatching by integrating the clustering and partitioning strategy of historical fault distribution and load peak data, and the parameter aggregation mechanism of distributed reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a distribution network scheduling method based on deep reinforcement learning according to an embodiment of the present invention;

[0052] Figure 2 This is a structural diagram of a distribution network dispatching system based on deep reinforcement learning according to an embodiment of the present invention;

[0053] Reference numerals:

[0054] 1. Acquisition module; 2. Calculation module; 3. Training module; 4. Evaluation module; 5. Update module; 6. Scheduling module. DETAILED DESCRIPTION

[0055] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0056] In the description of the present invention, it should be understood that the terms "first" and "second" etc. are used in the present invention to distinguish different objects rather than to describe a specific order.

[0057] In describing the present invention, it should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Those skilled in the art will understand the specific meanings of the above terms in the present invention in specific circumstances.

[0058] like Figure 1 As shown, an embodiment of the present invention provides a distribution network scheduling method based on deep reinforcement learning, comprising the steps of:

[0059] S1. Obtain historical fault distribution data and regional load peak data of the target distribution network, and cluster and partition the target distribution network based on the historical fault distribution data and regional load peak data to obtain several distribution network sub-regions;

[0060] Specifically, step S1 includes:

[0061] 1) Obtain historical fault distribution data of the target distribution network and calculate the fault distribution density index based on the historical fault distribution data;

[0062] Historical fault distribution data is used to characterize the spatial distribution of high-fault-incidence areas in the target distribution network. By calculating the fault distribution density index, this data is converted into quantifiable characteristic parameters, providing data support for subsequent zoning. The fault distribution density index is a normalized value of the frequency of faults per unit area.

[0063] 2) Obtain regional load peak data of the target distribution network and calculate the regional load peak data to obtain the offset characteristic parameters;

[0064] Regional load peak data reflects the time offset characteristics of each regional load peak relative to the global load peak. By calculating the offset characteristic parameter, the regional load peak data is converted into quantifiable characteristic parameters, providing data support for subsequent partitioning. The offset characteristic parameter is the difference between the load peak time and the global average.

[0065] 3) Based on the fault distribution density index and the topological partitioning constraints of the target distribution network, the target distribution network is clustered to obtain the initial partitioning results;

[0066] Using the fault distribution density index as input, the K-means algorithm clusters regions with similar fault characteristics into one category, ensuring homogeneity in the fault distribution density within the same sub-region. This facilitates the subsequent development of dispatch strategies tailored to regional fault characteristics. Furthermore, topological partitioning constraints are introduced as hard conditions to ensure that the clustering results do not violate the electrical connection boundaries of the target distribution network's physical topology, thus avoiding unreasonable partitioning across physical nodes.

[0067] The initial partitioning result obtained by clustering is the product of the combined effect of fault feature clustering and topological constraints. It not only ensures the similarity of fault distribution in sub-areas, but also conforms to the objective limitations of the physical structure of the distribution network.

[0068] 4) Based on the offset characteristic parameters, the initial partitioning results are modified to obtain several distribution network sub-areas.

[0069] Specifically, if the offset characteristic parameter exceeds a preset threshold, the boundaries of the initial partitioning results are adjusted to obtain several distribution network sub-regions. When the load peak offset characteristic is significant, the initial partitioning results may only consider fault characteristics and ignore the differences in load characteristics, resulting in insufficient adaptability of the scheduling strategy within the sub-region. Therefore, it is necessary to adjust the boundaries of the initial partitioning results based on the offset characteristic parameter so that the adjusted sub-regions meet both fault distribution similarity and load characteristic consistency, thereby improving the response to regional load fluctuations.

[0070] S2. Obtaining the operating status monitoring data of each distribution network sub-area, and performing numerical calculations on the operating status monitoring data to obtain a state vector of the corresponding distribution network sub-area;

[0071] Specifically, step S2 includes:

[0072] 1) Obtain historical load data for each distribution network sub-area, and analyze and calculate the historical load data to obtain the load peak offset of the corresponding distribution network sub-area;

[0073] Peak load offset = (Current load value - Historical load peak) / Historical load peak. Peak load offset reflects the degree of deviation between the current load and the historical maximum load level for the area, helping to determine load trends and whether the load capacity of the area is approaching or exceeding it. If the peak load offset approaches or exceeds 1, it indicates that the current load has reached or exceeded the historical peak, potentially indicating power supply pressure.

[0074] 2) Obtaining real-time monitoring data of energy storage devices in each distribution network sub-area and calculating and processing the real-time monitoring data to obtain the remaining energy storage capacity ratio of the corresponding distribution network sub-area;

[0075] Real-time monitoring data for energy storage devices includes the device's rated capacity and current charge and discharge status. The remaining capacity ratio (current stored charge / rated capacity) determines the energy storage device's ability to regulate energy over subsequent periods. A low remaining capacity ratio may not effectively reduce peak loads during peak periods. Furthermore, during low-load periods, if the energy storage is full, no further excess energy can be stored.

[0076] 3) Obtain the predicted power and actual power of renewable energy generation equipment in each distribution network sub-area, and calculate the difference between the predicted power and actual power of renewable energy generation equipment to obtain the renewable energy prediction error value of the corresponding distribution network sub-area;

[0077] A positive forecast error for renewable energy indicates that the predicted power is higher than the actual power; a negative error indicates that the actual power is higher than the predicted power. Due to the intermittent and uncertain nature of renewable energy, the forecast error provides a direct indicator of forecast accuracy. For distribution network dispatchers, accurately understanding the forecast error helps rationally schedule the output of other power generation equipment, thereby balancing power supply and demand and avoiding power imbalances caused by fluctuations in renewable energy generation.

[0078] 4) Obtaining node voltage monitoring data and node voltage rated data for each distribution network sub-area, and performing deviation calculation on the node voltage monitoring data and the node voltage rated data to obtain the node voltage deviation value for the corresponding distribution network sub-area;

[0079] Node voltage deviation = (actual voltage - rated voltage) / rated voltage. The node voltage deviation, expressed as a percentage, represents the degree to which the node voltage deviates from the rated value and is a key indicator of power quality in distribution networks. Excessive node voltage deviation can affect the normal operation of electrical equipment or even damage it. During distribution network scheduling, node voltage deviations must be closely monitored so that timely voltage regulation measures can be implemented, such as adjusting transformer taps and switching capacitors.

[0080] 5) Obtaining the line current monitoring data and line current rated data of each distribution network sub-area, and performing load rate calculation on the line current monitoring data and the line current rated data to obtain the line load value of the corresponding distribution network sub-area;

[0081] Line load value = actual current value / rated current value. The line load value reflects the ratio of the actual line-carrying current to the rated line-carrying current and is used to assess the line's operating status and load capacity. When the line load value approaches or exceeds 1, it indicates that the line is heavily loaded or overloaded, which may cause problems such as overheating, increased voltage drop, and even line failure. Based on the line load value, power flow can be rationally distributed to avoid line overload.

[0082] 6) Based on the load peak offset, energy storage remaining capacity ratio, renewable energy prediction error value, node voltage deviation value and line load value of each distribution network sub-region, a state vector of the corresponding distribution network sub-region is constructed.

[0083] The load peak offset, energy storage remaining capacity ratio, renewable energy prediction error, node voltage deviation and line load value of each distribution network sub-area are combined to obtain the state vector of the corresponding distribution network sub-area.

[0084] S3. Based on each state vector, the pre-built first deep reinforcement learning model is trained to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area;

[0085] Specifically, step S3 includes:

[0086] 1) Build the first deep reinforcement learning model based on the deep deterministic policy gradient algorithm;

[0087] The first deep reinforcement learning model includes an actor network and a critic network. The actor network is used to generate actions, and the critic network is used to evaluate the value of actions. This embodiment uses a multi-layer perceptron to construct the actor network and the critic network respectively.

[0088] 2) Each state vector is input into the actor network for strategy mapping, and the corresponding energy control action is output;

[0089] For example, if the state vector of a distribution network sub-region is [s1, s2, s3, s4, s5], after passing through the input layer of the actor network, linear transformation and activation operations are performed in each hidden layer in turn, and the final output layer is [a1, a2, a3, a4, a5]. This is mapped to the interval [-1, 1] using the tanh function, and then linearly transformed according to the value range of the actual energy control action to obtain the actual energy control action. The energy control action includes power generation regulation action and energy storage control action.

[0090] 3) Each energy control action is input into the critic network for value evaluation, and the corresponding value evaluation result is output;

[0091] Each energy control action and the state vector corresponding to each energy control action are input into the critic network respectively. After calculation in the hidden layer, a scalar value Q is obtained in the output layer. The Q value is the value evaluation result of taking the corresponding energy control action in the current state.

[0092] 4) Based on each value assessment result, the parameters of the first deep reinforcement learning model are updated to obtain a second deep reinforcement learning model for the corresponding distribution network sub-area.

[0093] The parameters of the first deep reinforcement learning model are iteratively updated based on each value assessment result, resulting in a second deep reinforcement learning model for the corresponding distribution network subregion. The core idea of the deep deterministic policy gradient algorithm is to update the critic network parameters by minimizing the mean squared error between the critic network's output and the target value. Simultaneously, the actor network parameters are updated based on the value assessed by the critic network. Using stochastic gradient descent, the weights and biases of the actor and critic networks are continuously adjusted, allowing the model to gradually learn the optimal policy.

[0094] S4. Evaluate each second deep reinforcement learning model based on the pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model;

[0095] Specifically, step S4 includes:

[0096] 1) Construct a validation dataset containing extreme scenarios of the target distribution network based on a generative adversarial network;

[0097] A generative adversarial network consists of a generator and a discriminator. When constructing a validation dataset, the generator's task is to generate simulated target distribution network operating data, including historical fault data, historical load data, grid topology data, and state vectors. This data must be as realistic as possible and cover a variety of extreme scenarios for the target distribution network, such as rare large-scale faults, extreme load fluctuations, and abnormal renewable energy generation in severe weather. The discriminator is responsible for determining whether the data generated by the generator is real or false. The generator continuously adjusts its own parameters so that the generated data can mislead the discriminator, while the discriminator is continuously optimized to better distinguish between real and false data. The two interact in a game, and ultimately the generator is able to generate a validation dataset that is close to the true distribution and contains extreme scenarios of the target distribution network.

[0098] 2) Evaluate the validity of the state vector on the validation dataset to obtain the first evaluation result of each second deep reinforcement learning model;

[0099] To evaluate the effectiveness of the state vector, a series of relevant evaluation indicators need to be determined. This embodiment constructs a three-dimensional evaluation system consisting of root mean square error (RMS), cosine similarity, and information entropy. RMS is used to measure the error between the predicted state vector and the true state vector, cosine similarity is used to assess the similarity between the predicted state vector and the true state vector, and information entropy is used to measure the amount of information contained in the state vector.

[0100] For each evaluation metric, different fuzzy levels are assigned based on their numerical range. For each state vector in the validation dataset, its numerical value for each evaluation metric is calculated, and its degree of membership in each fuzzy level for each evaluation metric is determined based on the fuzzy level. In this way, a fuzzy evaluation matrix for each state vector for each evaluation metric is constructed.

[0101] The weighted average operator of the fuzzy comprehensive evaluation method is used to integrate the three-dimensional indicators into a state evaluation value between 0 and 1. Specifically, for a certain state vector, its state vector validity evaluation value is:

[0102]

[0103] in, represents the validity evaluation value of the state vector, Indicates the The weight of the evaluation index, Indicates the state vector in By statistically analyzing the validity evaluation values of all state vectors in the validation data set, the first evaluation result reflecting the validity of the state vector is obtained.

[0104] 3) performing a dynamic action response evaluation on each second deep reinforcement learning model based on the validation dataset to obtain a second evaluation result corresponding to the second deep reinforcement learning model;

[0105] Based on the verification data set, the Markov decision process theory is used to perform dynamic evaluation of the action response of each second deep reinforcement learning model to obtain the second evaluation result corresponding to the second deep reinforcement learning model.

[0106] Specifically, step 3) includes:

[0107] 31) Constructing a Markov decision process quintuple ;

[0108] represents the state space, i.e., the state vector, Represents the action space, i.e., power generation regulation action and energy storage control action, represents the state transition probability, represents the immediate reward function, Represents the discount factor.

[0109] 32) Calculate the long-term cumulative reward of each second deep reinforcement learning model’s output action sequence based on the Markov decision process quintuple;

[0110] The second deep reinforcement learning model generates actions for each state vector in the validation dataset. Based on the defined state transition probabilities and immediate reward function, the model calculates the next possible state and reward after taking an action from the current state. Through continuous iteration, the model calculates the long-term cumulative reward for a series of actions.

[0111] 33) Perform real-time constraint verification on the output action sequence and calculate the feasibility of action execution;

[0112] Action feasibility refers to the percentage of actions in the output action sequence that violate device constraints. Output actions must meet safety constraints. For example, power generation regulation actions must not exceed the rated power of the power generation equipment, and energy storage control actions must ensure the safe charging and discharging of the energy storage equipment. If an action violates these constraints, a large negative reward is given or the action sequence is deemed infeasible.

[0113] 34) Based on a linear weighted function, the long-term cumulative reward and the action execution feasibility are integrated to obtain a second evaluation result corresponding to the second deep reinforcement learning model.

[0114] The long-term cumulative reward and the action execution feasibility are linearly weighted to obtain a second evaluation result corresponding to the second deep reinforcement learning model.

[0115] 4) Perform weighted fusion on each first evaluation result and each second evaluation result to obtain an evaluation result corresponding to the second deep reinforcement learning model.

[0116] This embodiment performs weighted fusion with a weight of 60% for each first evaluation result and a weight of 40% for each second evaluation result to obtain the evaluation result of the corresponding second deep reinforcement learning model. The evaluation result reflects the comprehensive performance of the model in extreme scenarios and can be used for horizontal comparison of different second deep reinforcement learning models.

[0117] S5. Based on each evaluation result, perform parameter aggregation on each second deep reinforcement learning model to obtain a global aggregation parameter, and update each second deep reinforcement learning model according to the global aggregation parameter to obtain a third deep reinforcement learning model for the corresponding distribution network sub-area;

[0118] Specifically, step S5 includes:

[0119] 1) Use the softmax function to convert each evaluation result into the corresponding aggregation weight;

[0120] First, each evaluation result is standardized to eliminate the dimension effect. The standardized value reflects the quality of the corresponding second deep reinforcement learning model relative to the global average level. Then, the softmax function is used to convert each evaluation result into a corresponding aggregation weight. The weight has a probabilistic attribute to ensure that the sum of all weights is 1, and the model with a higher evaluation result obtains a greater weight.

[0121] 2) Performing weighted aggregation on the parameters of each second deep reinforcement learning model based on each aggregation weight to obtain a global aggregation parameter;

[0122] The parameters of each second deep reinforcement learning model are weighted and aggregated based on each aggregation weight, and the parameters of each second deep reinforcement learning model are integrated to generate global aggregation parameters that can reflect the characteristics and needs of the entire target distribution network.

[0123] 3) Based on the global aggregation parameters, the parameters of each second deep reinforcement learning model are fused separately to obtain the third deep reinforcement learning model of the corresponding distribution network sub-area.

[0124] For each second deep reinforcement learning model, its original parameters are fused with the global aggregate parameters. Through this updating process, each second deep reinforcement learning model absorbs global information and obtains a more adaptable and robust third deep reinforcement learning model, which can better cope with the complex operation of the entire target distribution network.

[0125] S6. Obtain the real-time state vector of each distribution network sub-area, and input each real-time state vector into the corresponding third deep reinforcement learning model for strategy mapping to obtain the scheduling strategy of each distribution network sub-area.

[0126] Specifically, step S6 includes:

[0127] 1) Obtaining the real-time state vector of each distribution network sub-area and inputting each real-time state vector into the corresponding third deep reinforcement learning model for strategy mapping to obtain the first energy control action for each distribution network sub-area;

[0128] If the real-time state vector of a distribution network sub-region is [s11, s22, s33, s44, s55], after passing through the input layer of the actor network, linear transformation and activation operations are performed in each hidden layer in turn, and the final output layer is [a11, a22, a33, a44, a55]. This is mapped to the interval [-1, 1] using the tanh function, and then linearly transformed according to the value range of the actual first energy control action to obtain the actual first energy control action. The first energy control action includes the first power generation regulation action and the first energy storage control action.

[0129] 2) obtaining real-time topology information of each distribution network sub-area, and adjusting each first energy control action based on each real-time topology information to obtain a second energy control action for the corresponding distribution network sub-area;

[0130] Specifically, step 2) includes:

[0131] 21) Obtain real-time topological information of each distribution network sub-area;

[0132] Through the target distribution network's automated monitoring system, real-time topological information for each distribution network sub-area is obtained, including line connection status, switch on / off status, etc. This information is crucial for determining the feasibility of the first energy regulation action.

[0133] 22) Performing a feasibility analysis on each first energy control action based on each real-time topology information, and adjusting the infeasible first energy control action to obtain a second energy control action for the corresponding distribution network sub-area.

[0134] Based on the real-time topology information, a feasibility analysis is performed on the first energy control action. For example, the feasibility of the first power generation control action is checked to see whether it will cause certain lines to overload, or whether the first energy storage control action will exceed the charge and discharge limits of the energy storage device. If executing a first power generation control action in the first energy control action causes the current of the related line to exceed its rated current, the first energy control action is considered infeasible.

[0135] For any infeasible first energy control actions, they are optimized and adjusted based on real-time topology information and the target distribution network's operational constraints. For example, if a first generation adjustment action becomes infeasible due to increasing the power of a generating device, the system considers adjusting the power of other generating devices or combining the charge and discharge power of energy storage devices to balance power demand. After optimization and adjustment, the second energy control action to be executed is obtained.

[0136] 3) Based on each second energy control action, a dispatching strategy for the corresponding distribution network sub-area is generated.

[0137] The dispatching strategy of each distribution network sub-area includes a second energy regulation action to be executed.

[0138] The embodiment of the present invention is a distribution network dispatching method based on deep reinforcement learning. It performs clustering and partitioning based on historical fault distribution data, so that the sub-region division of the distribution network matches the distribution of high-incidence fault areas. Compared with the traditional static partitioning method, it effectively improves the regional mutual assistance ability in the event of a fault and avoids the chain power outage phenomenon; it performs numerical calculations on the operating status monitoring data to generate a state vector, and combines it with a distributed training framework to solve the problem of a sharp increase in dimensions in multi-region scheduling, greatly reduces the dimension of the action space, shortens the decision delay time, and meets the real-time scheduling requirements of the distribution network; through pre-constructed model training and parameter aggregation mechanism, it greatly shortens the model update cycle and significantly improves the model's adaptability to dynamic changes in the distribution network. At the same time, the verification data set evaluation and cross-region parameter aggregation strategy improve the generalization performance of the model. The present invention achieves a dual improvement in the reliability and real-time performance of distribution network dispatching by integrating the clustering and partitioning strategy of historical fault distribution and load peak data, and the parameter aggregation mechanism of distributed reinforcement learning.

[0139] Based on the above-mentioned distribution network scheduling method based on deep reinforcement learning, Figure 2 As shown, an embodiment of the present invention provides a distribution network dispatching system based on deep reinforcement learning, including:

[0140] Acquisition module 1 is used to obtain historical fault distribution data and regional load peak data of the target distribution network, and cluster and partition the target distribution network based on the historical fault distribution data and regional load peak data to obtain several distribution network sub-regions;

[0141] Operation module 2 is used to obtain the operation status monitoring data of each distribution network sub-area and perform numerical operations on the operation status monitoring data to obtain the state vector of the corresponding distribution network sub-area;

[0142] Training module 3, configured to train the pre-built first deep reinforcement learning model based on each state vector to obtain a second deep reinforcement learning model for the corresponding distribution network sub-area;

[0143] Evaluation module 4, configured to evaluate each second deep reinforcement learning model based on a pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model;

[0144] An updating module 5 is configured to aggregate parameters of each second deep reinforcement learning model based on each evaluation result to obtain a global aggregate parameter, and to update each second deep reinforcement learning model according to the global aggregate parameter to obtain a third deep reinforcement learning model for the corresponding distribution network sub-area;

[0145] The scheduling module 6 is used to obtain the real-time state vector of each distribution network sub-area, and input each real-time state vector into the corresponding third deep reinforcement learning model for strategy mapping to obtain the scheduling strategy of each distribution network sub-area.

[0146] It should be noted that the various modules in the above-mentioned distribution network dispatching system based on deep reinforcement learning can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. For the specific definition of a distribution network dispatching system based on deep reinforcement learning, please refer to the definition of a distribution network dispatching method based on deep reinforcement learning above. The two have the same functions and effects and will not be repeated here.

[0147] In summary, the embodiment of the present invention is a distribution network dispatching method and system based on deep reinforcement learning. Clustering and partitioning are performed based on historical fault distribution data, so that the sub-region division of the distribution network matches the distribution of high-incidence fault areas. Compared with the traditional static partitioning method, it effectively improves the regional mutual assistance ability in the event of a fault and avoids the chain power outage phenomenon; the operation status monitoring data is numerically calculated to generate a state vector, and combined with a distributed training framework, it solves the problem of a sharp increase in dimensions in multi-region scheduling, greatly reduces the dimension of the action space, shortens the decision delay time, and meets the real-time scheduling requirements of the distribution network; through pre-constructed model training and parameter aggregation mechanism, the model update cycle is greatly shortened, and the model's adaptability to dynamic changes in the distribution network is significantly improved. At the same time, the verification data set evaluation and cross-region parameter aggregation strategy improve the generalization performance of the model. The present invention achieves a dual improvement in the reliability and real-time performance of distribution network dispatching by integrating the clustering and partitioning strategy of historical fault distribution and load peak data, and the parameter aggregation mechanism of distributed reinforcement learning.

[0148] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the various technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A distribution network scheduling method based on deep reinforcement learning, characterized in that: include: Acquire historical fault distribution data and regional load peak data of a target distribution network, and cluster and partition the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions; Acquiring operation status monitoring data of each of the distribution network sub-areas, and performing numerical calculations on the operation status monitoring data to obtain a state vector corresponding to the distribution network sub-area; Based on each of the state vectors, a pre-built first deep reinforcement learning model is trained to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area; Evaluate each of the second deep reinforcement learning models based on a pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model; Based on each of the evaluation results, parameters of each of the second deep reinforcement learning models are aggregated to obtain a global aggregate parameter, and each of the second deep reinforcement learning models is updated according to the global aggregate parameter to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area; Obtaining a real-time state vector of each of the distribution network sub-areas, and inputting each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a scheduling strategy for each of the distribution network sub-areas; The acquiring of historical fault distribution data and regional load peak data of the target distribution network, and clustering and partitioning the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions, includes: Obtaining historical fault distribution data of the target distribution network, and calculating the fault distribution density index based on the historical fault distribution data; Acquiring regional load peak data of the target distribution network, and calculating the regional load peak data to obtain an offset characteristic parameter; Based on the fault distribution density index and in combination with the topological partitioning constraints of the target distribution network, clustering the target distribution network to obtain an initial partitioning result; Modifying the initial partitioning result based on the offset characteristic parameter to obtain a plurality of distribution network sub-areas; The evaluating each of the second deep reinforcement learning models based on the pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model includes: A validation dataset containing extreme scenarios of the target distribution network is constructed based on a generative adversarial network. Performing a state vector validity evaluation on the verification data set to obtain a first evaluation result of each of the second deep reinforcement learning models; Performing a dynamic action response evaluation on each of the second deep reinforcement learning models based on the validation data set to obtain a second evaluation result corresponding to the second deep reinforcement learning model; Perform weighted fusion on each of the first evaluation results and each of the second evaluation results to obtain an evaluation result corresponding to the second deep reinforcement learning model.

2. The distribution network scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The initial partitioning result is modified based on the offset characteristic parameter to obtain a plurality of distribution network sub-areas, including: If the offset characteristic parameter is greater than a preset threshold, the boundaries of the initial partitioning result are adjusted to obtain a plurality of distribution network sub-areas.

3. The distribution network scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The method of training the pre-built first deep reinforcement learning model based on each state vector to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area includes: Constructing a first deep reinforcement learning model based on a deep deterministic policy gradient algorithm, wherein the first deep reinforcement learning model includes an actor network and a critic network, the actor network is used to generate actions, and the critic network is used to evaluate the value of the actions; Input each of the state vectors into the actor network for strategy mapping, and output a corresponding energy control action; Input each of the energy control actions into the critic network for value evaluation, and output a corresponding value evaluation result; Based on each of the value assessment results, the parameters of the first deep reinforcement learning model are updated respectively to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area.

4. The distribution network scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The dynamically evaluating the action response of each second deep reinforcement learning model based on the verification data set to obtain a second evaluation result corresponding to the second deep reinforcement learning model includes: Based on the verification data set, Markov decision process theory is used to perform dynamic action response evaluation on each of the second deep reinforcement learning models to obtain a second evaluation result corresponding to the second deep reinforcement learning model.

5. The distribution network scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The step of aggregating parameters of each of the second deep reinforcement learning models based on each of the evaluation results to obtain a global aggregation parameter, and updating each of the second deep reinforcement learning models according to the global aggregation parameter to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area, including: Using the softmax function to convert each of the evaluation results into a corresponding aggregation weight; Performing parameter weighted aggregation on each of the second deep reinforcement learning models based on each of the aggregation weights to obtain a global aggregation parameter; Based on the global aggregation parameters, parameters of each of the second deep reinforcement learning models are fused to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area.

6. The distribution network scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The obtaining of a real-time state vector of each of the distribution network sub-areas and inputting each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a scheduling strategy for each of the distribution network sub-areas includes: Obtaining a real-time state vector of each of the distribution network sub-areas, and inputting each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a first energy control action for each of the distribution network sub-areas; Acquire real-time topology information of each of the distribution network sub-areas, and adjust each of the first energy control actions based on each of the real-time topology information to obtain a second energy control action corresponding to the distribution network sub-area; Based on each of the second energy regulation actions, a scheduling strategy corresponding to the distribution network sub-area is generated.

7. The distribution network scheduling method based on deep reinforcement learning according to claim 6, characterized in that: The acquiring of the real-time topology information of each of the distribution network sub-areas, and adjusting each of the first energy control actions based on each of the real-time topology information to obtain a second energy control action corresponding to the distribution network sub-area, includes: Acquiring real-time topological information of each of the distribution network sub-areas; A feasibility analysis is performed on each first energy regulation action based on each piece of real-time topology structure information, and infeasible first energy regulation actions are adjusted to obtain second energy regulation actions corresponding to the distribution network sub-area.

8. A distribution network dispatching system based on deep reinforcement learning, characterized in that: include: an acquisition module, configured to acquire historical fault distribution data and regional load peak data of a target distribution network, and cluster and partition the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions; a calculation module, configured to obtain operation status monitoring data of each of the distribution network sub-areas, and perform numerical calculations on the operation status monitoring data to obtain a state vector corresponding to the distribution network sub-area; A training module, configured to train the pre-built first deep reinforcement learning model based on each of the state vectors to obtain a second deep reinforcement learning model corresponding to the distribution network sub-area; An evaluation module, configured to evaluate each of the second deep reinforcement learning models based on a pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model; an updating module, configured to perform parameter aggregation on each of the second deep reinforcement learning models based on each of the evaluation results to obtain a global aggregation parameter, and to update each of the second deep reinforcement learning models according to the global aggregation parameter to obtain a third deep reinforcement learning model corresponding to the distribution network sub-area; A scheduling module is configured to obtain a real-time state vector of each of the distribution network sub-areas, and input each of the real-time state vectors into the corresponding third deep reinforcement learning model for strategy mapping to obtain a scheduling strategy for each of the distribution network sub-areas; The acquiring of historical fault distribution data and regional load peak data of the target distribution network, and clustering and partitioning the target distribution network based on the historical fault distribution data and the regional load peak data to obtain a plurality of distribution network sub-regions, includes: Obtaining historical fault distribution data of the target distribution network, and calculating the fault distribution density index based on the historical fault distribution data; Acquiring regional load peak data of the target distribution network, and calculating the regional load peak data to obtain an offset characteristic parameter; Based on the fault distribution density index and in combination with the topological partitioning constraints of the target distribution network, clustering the target distribution network to obtain an initial partitioning result; Modifying the initial partitioning result based on the offset characteristic parameter to obtain a plurality of distribution network sub-areas; The evaluating each of the second deep reinforcement learning models based on the pre-constructed validation dataset to obtain an evaluation result corresponding to the second deep reinforcement learning model includes: A validation dataset containing extreme scenarios of the target distribution network is constructed based on a generative adversarial network. Performing a state vector validity evaluation on the verification data set to obtain a first evaluation result of each of the second deep reinforcement learning models; Performing a dynamic action response evaluation on each of the second deep reinforcement learning models based on the validation data set to obtain a second evaluation result corresponding to the second deep reinforcement learning model; Perform weighted fusion on each of the first evaluation results and each of the second evaluation results to obtain an evaluation result corresponding to the second deep reinforcement learning model.

Images