Ring main unit intelligent regulation and control method based on deep reinforcement learning

Abstract

The invention discloses a ring main unit intelligent regulation and control method based on deep reinforcement learning, and the method comprises the following steps: building an electrical topological graph model through collecting the real-time operation data of a ring main unit and a power distribution network, extracting the characteristics of nodes and branches, and generating an initial state vector; a comparison strategy embedding layer is arranged in front of the strategy network, and state-action representation is optimized through comparison learning; constructing a reinforcement learning decision structure by adopting a trust region strategy optimization algorithm, and outputting a ring main unit control action sequence; before execution, an out-of-limit action is judged and corrected by a control barrier function convoy layer; and closing and opening control is completed according to the corrected action, rewards are recorded, and incremental learning and parameter fine adjustment are performed by using an experience playback buffer area. According to the invention, by fusing the graph neural network and deep reinforcement learning and introducing a comparison strategy embedding layer and a control barrier function escorting layer, safe, adaptive and efficient intelligent regulation and control of the ring main unit in a complex power distribution network are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system automation, and particularly relates to an intelligent regulation and control method for ring network cabinets based on deep reinforcement learning. BACKGROUND

[0002] With the continuous increase of urban power grid load density and the large number of access of distributed power sources, the distribution network structure presents the characteristics of meshing, dynamic and high coupling. As the key equipment for realizing power supply segmentation, connection and fault isolation in the distribution network, the switch state of the ring network cabinet directly affects the power supply reliability and power quality. The traditional regulation and control of the ring network cabinet mainly relies on manual duty or sequential control strategy based on fixed logic, which is difficult to respond to the changes of power flow and equipment failures in complex operating scenarios, and is prone to cause local overload, over-limit operation or reverse power at the connection point.

[0003] In recent years, with the development of intelligent distribution network and edge computing technology, some researches attempt to introduce machine learning and expert system to realize autonomous control of the ring network cabinet, such as switch decision models based on rule-based reasoning, fuzzy logic or shallow neural network. However, such methods generally rely on preset thresholds and artificial experience, lack adaptive modeling capability for dynamic topology and multi-source uncertainty of the power grid, and are prone to strategy lag, oscillation control or action conflict under sudden failures, load mutations or fluctuations of distributed power sources, which is difficult to meet the requirements of real-time, safety and self-learning ability in intelligent distribution network.

[0004] Therefore, how to provide an intelligent regulation and control method for ring network cabinets based on deep reinforcement learning is a problem that those skilled in the art need to solve. SUMMARY

[0005] One object of the present application is to provide an intelligent regulation and control method for ring network cabinets based on deep reinforcement learning. The present application combines graph neural network with deep reinforcement learning to realize self-learning and safety optimization of the regulation and control strategy of the ring network cabinet. By establishing an electrical topology data structure containing node feature matrix and edge feature matrix, the electrical connection relationship of the distribution network can be accurately represented. A contrast policy embedding layer is introduced in the front end of the reinforcement learning strategy network to improve the discriminability of the state-action mapping through sample sparsity regularization and inter-class separation constraint. A control barrier function escort layer is set to establish physical safety constraints for node voltage, branch current and power flow direction, and to realize action safety clipping through feasible region projection correction based on quadratic programming. A trust region policy optimization algorithm is used to limit the policy update amplitude to maintain training stability, and an experience replay and low learning rate fine-tuning mechanism is combined to realize online adaptive optimization of the model. Therefore, the present application can realize safe, efficient and stable regulation and control of the ring network cabinet under complex distribution network operating environment, and significantly improve the power supply reliability and system intelligence level.

[0006] According to an embodiment of the present application, a method for intelligent regulation and control of a ring main unit based on deep reinforcement learning comprises the following steps: Real-time operation data of the ring main unit and the power distribution network to which the ring main unit belongs are collected, a power distribution network topology graph model is established according to the real-time operation data, and an electrical topology data structure comprising a node feature matrix and an edge feature matrix is generated; The collected data is subjected to time synchronization and normalization processing within a preset time window, is input into a graph neural network encoder, and the associated features between nodes and branches are extracted, and an initial state vector is output; A comparative strategy embedding layer is arranged at the input end of the reinforcement learning strategy network, state-action samples under different operating conditions are input into the comparative strategy embedding layer for comparative learning and training, and an optimized strategy input vector is obtained; The optimized strategy input vector is input into a deep reinforcement learning network, the deep reinforcement learning network adopts a trust region policy optimization algorithm to construct a decision structure, and a control action sequence of the ring main unit is output; Before the control action sequence is executed, a control barrier function escort layer is arranged to perform physical constraint judgment and correction on the action, and when any barrier condition is triggered by the output action of the strategy, feasible region projection correction is performed on the output action of the strategy; According to the action sequence corrected by the escort layer, the ring main unit is controlled to perform closing and opening operations, the operating state of the ring main unit is updated, and a reward value is recorded; The latest state, action and reward are combined to form a new sample, which is written into an experience replay buffer, and the reinforcement learning network is periodically updated incrementally; During operation, the parameters of the reinforcement learning network are periodically fine-tuned, and the comparative strategy embedding layer is updated at a low learning rate.

[0007] Further, the real-time operation data comprises node voltage amplitude and phase angle, branch current, active power, reactive power, tie point state, switch bit number, on-off signal, protection action flag and branch impedance parameter; When establishing the power distribution network topology graph model, busbars, feeder ends and load nodes are taken as graph nodes, the conduction relationship corresponding to lines and ring main unit switches is taken as graph edges, and impedance parameters, rated currents, real-time on-off states and power flow direction attributes are set for each graph edge; Node feature matrices and edge feature matrices are constructed according to the corresponding relationship between nodes and edges, and the node feature matrices and the edge feature matrices jointly form an electrical topology data structure.

[0008] Further, the initial state vector generation step comprises: The preset time window is defined as a time range covering a plurality of continuous sampling periods, and the operating data in the time range constitutes an operating data sequence; Time synchronization is completed by time alignment and missing interpolation, and the running data sequence is normalized to convert the voltage, current and power data into a unified dimension form; In the extraction of the correlation features between the nodes and branches, the local neighborhood set of each node is calculated according to the adjacency relationship of the topological graph, the features of the neighborhood nodes and the connected edges are weighted and aggregated, and the node-level aggregated features are obtained; The node-level aggregated features are updated by a nonlinear transformation layer to form a state representation vector; The state representation vector is output as an initial state vector after dimension reduction by a linear mapping layer.

[0009] Further, the contrastive learning training includes: The contrastive strategy embedding layer is used for contrastive learning training of the state-action samples under different operating conditions, including an encoding sublayer, a sample construction module and a similarity calculation module; The encoding sublayer maps the input state vector into a low-dimensional embedding representation; The sample construction module takes samples satisfying the voltage and current limits as positive samples and takes out-of-limit or abnormal samples as negative samples; The similarity calculation module uses temperature-calibrated cosine similarity to measure to distinguish positive and negative sample pairs, and optimizes the embedding space distribution by minimizing the contrastive loss function; The embedding result output after training is used as the optimized strategy input vector.

[0010] Further, the contrastive strategy embedding layer further includes: In the training process, a sample distribution sparsity regularization term and an inter-class separation constraint term are introduced to maximize the center distance between the safe sample cluster and the out-of-limit sample cluster and minimize the intra-cluster variance; After each policy network update, the sampling probability of the experience replay samples is reweighted based on the sample similarity in the embedding space, and the samples near the safety boundary are preferentially learned.

[0011] Further, the generation step of the control action sequence of the ring network cabinet includes: The deep reinforcement learning network adopts a trust region policy optimization algorithm to construct a decision structure, including a policy network and a value network; The policy network receives the input vector optimized by the contrastive strategy embedding layer, and outputs the action probability distribution of each operable switch according to the current state, the actions including closing, opening and maintaining state; The value network is used to evaluate the expected return corresponding to the current state, and sets a trust region constraint on the KL divergence between the policy distribution and the old policy distribution when the policy is updated, to limit the update step; The power recovery rate is taken as a positive reward in the training process, and the voltage out-of-limit, current saturation and frequent switching are taken as penalty terms, and the control action sequence for the ring main unit regulation and control is obtained through weighted optimization.

[0012] Further, the control barrier function escort layer includes a voltage barrier function, a current barrier function and a power flow direction barrier function: The voltage barrier function is used to constrain the node voltage value to remain within the upper and lower limits of the rated voltage; The current barrier function is used to constrain the branch current not to exceed the rated thermal stability current value; The power flow direction barrier function is used to constrain the tie point power flow direction to be consistent with the intended power supply direction; When it is detected that the policy output action makes any barrier function value exceed a preset threshold, a feasible region projection correction based on quadratic programming is performed, taking the action vector output by the original policy as a reference, and a constraint optimization problem is constructed under the constraint conditions defined by each barrier function; In the feasible region projection correction process based on quadratic programming, the gradient information of each barrier function and the constraint active set are calculated in real time, and the nonlinear barrier constraint is converted into a local linear inequality constraint using an incremental linearization approximation method; And set a safety margin coefficient in the solving process, and dynamically adjust the constraint boundary according to the measurement uncertainty of each electrical quantity; Taking the minimization of action adjustment as the target, a feasible solution satisfying the constraints is obtained, and the feasible solution is taken as the corrected control instruction.

[0013] Further, the specific steps of updating the running state of the ring main unit and recording the reward value include: When the control operation is performed according to the action sequence corrected by the escort layer, the interlocking state of the ring main unit switch and the minimum dead time are checked, and when the adjacent switches are in an interlocking relationship, simultaneous action is prohibited; The minimum dead time is set to be no less than 200 milliseconds to avoid repeated triggering; After performing the closing and opening operations, the node voltage, branch current and tie point power flow are quickly recovered, And the recovery results are compared with the predicted values of the strategy to generate a result confirmation flag; The result confirmation flag and the corresponding immediate reward value are recorded together.

[0014] Further, the step of incremental updating includes: The experience replay buffer is used to store the state, action and reward samples formed during the operation of the ring main unit, and a priority sampling mechanism based on time difference error is used for sample extraction; In the incremental update, the double network structure of the target network and the current network is used, the parameters are replaced by a soft update method, and the soft update coefficient is between 0.005 and 0.01. The batch size of each incremental update is set to 64 to 128 samples, and the network loss function is normalized and smoothed after a plurality of continuous updates.

[0015] Further, the low learning rate update includes: The reinforcement learning network is periodically fine-tuned at a preset time interval during operation, and the fine-tuning period is between 30 seconds and 60 seconds. The policy network and the value network use independent learning rates for parameter updating. The contrast policy embedding layer is updated at a low learning rate, and when the validation set reward value does not improve for a plurality of continuous periods, the embedding layer parameter updating is stopped.

[0016] The beneficial effects of the present application are: The present application can explicitly express the electrical relationship between busbars, cables and tie switches in the model by establishing an electrical topology data structure containing node feature matrix and edge feature matrix, realize the structured representation of distribution network topology characteristics, and provide high-dimensional input basis for subsequent intelligent decision-making. The contrast policy embedding layer is introduced in the front end of the reinforcement learning policy network, the separability of the state-action mapping space is improved through sample distribution sparsity regularization and inter-class separation constraint, the problems of policy collapse and sample redundancy of traditional reinforcement learning under complex working conditions are avoided, and the convergence speed and generalization ability of the policy are significantly improved.

[0017] The present application sets a control barrier function escort layer, establishes physical safety constraints for node voltage, branch current and power flow direction, and introduces safety margin and dynamic linearization solving mechanism in the feasible domain projection correction process, so that the policy output action has physical feasibility and electrical safety while maintaining response speed, which can effectively reduce the risk of out-of-limit action and misoperation. The reinforcement learning decision structure is constructed by using the trust region policy optimization algorithm (TRPO), the policy update amplitude is limited by the KL divergence constraint, the network maintains training stability in the high-dimensional action space, and the policy oscillation phenomenon caused by excessive update is avoided.

[0018] The present application combines the experience replay buffer and the periodic fine-tuning mechanism to realize the continuous online optimization and parameter adaptive adjustment of the ring network cabinet control strategy, and the contrast policy embedding layer is updated at a low learning rate, which can maintain the stability of the policy while maintaining the sensitive response to changes in the operating environment. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0020] Figure 1 A flow chart of an intelligent regulation and control method for ring network cabinets based on deep reinforcement learning is proposed for the present application. Figure 2 A power distribution network topology structure schematic diagram of an intelligent regulation and control method for ring network cabinets based on deep reinforcement learning is proposed for the present application. Figure 3 A reinforcement learning decision structure schematic diagram of an intelligent regulation and control method for ring network cabinets based on deep reinforcement learning is proposed for the present application. DETAILED DESCRIPTION

[0021] The application will now be described in further detail with reference to the drawings. These drawings show only the essential features of the application and are therefore to be regarded only as a schematic illustration of the basic structure of the application. In the drawings:

[0022] REFERENCE Figure 1 - Figure 3 An intelligent regulation and control method for ring network cabinets based on deep reinforcement learning, comprising the following steps: Real-time operation data of the ring network cabinet and the power distribution network to which it belongs are collected, including bus voltage, current, active power, reactive power, switch position number, branch power flow direction and node connection relationship; a power distribution network topology graph model is established according to the data, the bus and load nodes are taken as graph nodes, the ring network cabinet switches and lines are taken as graph edges, and impedance parameters, rated current and real-time on-off state attributes are assigned to each graph edge; an electrical topology data structure containing node feature matrix and edge feature matrix is generated.

[0023] The collected data is subjected to time synchronization and normalization processing within a preset time window, input into a graph neural network encoder, the associated features between nodes and branches are extracted, and an initial state vector representing the current operating state is output; wherein the graph neural network is a stacked graph convolution structure, which is used to fuse the electrical features of adjacent nodes and realize the representation of spatial topology association.

[0024] The contrast policy embedding layer is arranged at the input end of the reinforcement learning strategy network, and state-action samples under different operating conditions are input into the embedding layer for contrast learning training; the contrast policy embedding layer comprises an encoding subnetwork and a contrast loss function module, wherein the encoding subnetwork is used for mapping a state vector into a low-dimensional embedding representation, the contrast loss function adopts a temperature-calibrated cosine similarity, and "safe operation samples" and "over-limit operation samples" are used as positive and negative sample pairs; by minimizing the similarity difference, the encoding weights are optimized, so that the safe states are clustered and the dangerous states are separated in the embedding space, and an optimized strategy input vector is obtained; the contrast policy embedding layer can enhance the distinguishing ability and learning efficiency of the strategy network for abnormal states.

[0025] The optimized strategy input vector is input into a deep reinforcement learning network, and a control action sequence of the ring main unit is output; the deep reinforcement learning network comprises a strategy network and a value network, and the training target of the deep reinforcement learning network is to maximize an expected return function; the return function comprises the following weighted items: (1) a power supply recovery rate reward; (2) a voltage deviation and a power flow over-limit penalty; (3) a switch action cost penalty; and (4) an opening and closing failure penalty.

[0026] Before the control action sequence is executed, a control barrier function escort layer is arranged to perform physical constraint judgment and correction on the action; the escort layer comprises a plurality of barrier functions: a. a voltage over-limit barrier function for limiting the node voltage to be within a safe upper and lower limit; b. a branch current barrier function for limiting the current to be within a thermal stability limit; and c. a power flow direction barrier function for preventing the power flow at a tie point from being reversed; when the strategy output action causes any barrier condition to be triggered, feasible region projection correction is performed on the action to ensure that the corrected action meets the safety constraints.

[0027] According to the action sequence corrected by the escort layer, the ring main unit is controlled to perform opening and closing operations, the operating state of the ring main unit is updated, and a reward value is recorded; a new sample composed of the latest state, action and reward is written into an experience replay buffer, and the reinforcement learning network is periodically updated incrementally.

[0028] During operation, the reinforcement learning network parameters are periodically fine-tuned, and the contrast policy embedding layer is updated with a low learning rate to maintain feature stability.

[0029] In the embodiment, the real-time operating data comprises node voltage amplitude and phase angle, branch current, active power, reactive power, tie point state, switch bit number, on-off signal, protection action flag and branch impedance parameter; The real-time operating data is collected by monitoring units arranged on the ring main unit body and the power distribution branch to which the ring main unit body belongs; the monitoring units comprise voltage sensors, current transformers and state collection modules, and can continuously monitor electrical quantities and switch states and upload them to an edge computing node; In establishing the power distribution network topology model, the bus, feeder end and load node are taken as graph nodes, and the conduction relationship corresponding to the line and ring network cabinet switch is taken as graph edge. For each graph edge, the connection relationship is determined according to the conduction state of the ring network cabinet switch. When the switch is in the closed state, the corresponding graph edge is in the conduction state. When the switch is in the open state, the corresponding graph edge is in the disconnected state. The impedance parameter, rated current, real-time on-off state and power flow direction attribute are set for each graph edge. According to the corresponding relationship between the nodes and the edges, a node feature matrix and an edge feature matrix are constructed. The node feature matrix is used to represent the voltage, current and power characteristics of each node, and the edge feature matrix is used to represent the impedance attribute, on-off state and power flow direction of the branch. The node feature matrix and the edge feature matrix together form an electrical topology data structure.

[0030] In the embodiment, the initial state vector generation step includes: The preset time window is defined as a time range covering a plurality of consecutive sampling periods, and is used to extract representative samples from the continuously collected ring network cabinet operation data. The voltage, current and power data obtained in the time range form an operation data sequence in chronological order; Time synchronization is completed by time alignment and missing interpolation. The missing interpolation is supplemented by a sliding average method. The operation data sequence is normalized and proportionally transformed according to the respective rated values. The voltage, current and power data are converted into a unified dimension form; In extracting the associated features between nodes and branches, the local neighborhood set of each node is calculated according to the adjacency relationship of the topology graph. The local neighborhood set is composed of other nodes directly connected to the node and corresponding branches. The features of the neighborhood nodes and connected edges are weighted and aggregated to obtain node-level aggregated features; The node-level aggregated features are updated through a nonlinear transformation layer. The nonlinear transformation layer re-encodes the node features through a learnable mapping function to form a state representation vector; The state representation vector is dimensionally reduced by a linear mapping layer to obtain a structured initial state vector.

[0031] In the embodiment, the contrastive learning training includes: The contrast strategy embedding layer is used for contrastive learning training of state-action samples under different operating conditions, including an encoding sublayer, a sample construction module and a similarity calculation module. Further, the encoding sublayer receives the input state vector and the corresponding action instruction, jointly maps the state information and the action information, extracts comprehensive features through multi-layer nonlinear transformation, obtains a low-dimensional embedding representation with distinguishing ability, and the low-dimensional embedding representation retains the main electrical characteristics of the power distribution network operating state and includes the difference characteristics before and after the action execution in the same representation space. The sample construction module generates training samples according to historical operation records and real-time data, and labels the samples according to whether the operating state meets the voltage, current and power safety range. The sample whose operating parameters are within the normal range is defined as a positive sample, and the sample whose operating parameters exceed the safety range or abnormal switch action occurs is defined as a negative sample. The sample construction module combines the state-action pairs collected in the same time period into a sample pair, and continuously updates the sample set during the training process. The similarity calculation module measures the similarity of the embedding results of the positive and negative sample pairs, and adopts a temperature calibrated cosine similarity calculation method to control the distinguishing scale between samples. The similarity calculation module minimizes the embedding distance between positive sample pairs and maximizes the embedding distance between negative sample pairs during the training process, thereby forming an embedding space with clustering and separation characteristics. After completing the contrast learning training, the embedding result output by the encoding sublayer is input into the reinforcement learning network as the optimized strategy input vector.

[0032] After completing the basic training of the contrast strategy embedding layer, enter the embedding space structure optimization phase. In this phase, a sample distribution sparsity regularization term is introduced at the sample input end. When the new state-action sample is mapped to the embedding representation by the encoding sublayer, the feature activation distribution of each dimension is calculated, and the embedding features are constrained according to the activation degree, so that the excessively concentrated dimensions are suppressed, and only the features related to the sample category are retained. After this constraint, the dimensions of the embedding representation are in a dispersed distribution state, and the safe operation samples and the out-of-limit samples form separable sparse feature distributions in the embedding space.

[0033] In the embedding optimization phase, an inter-class separation constraint term is introduced. According to the embedding representation generated in the previous phase, the center positions of the safe sample cluster and the out-of-limit sample cluster are calculated, and the parameters of the embedding mapping are adjusted in each iteration to gradually increase the distance between the centers of the two classes, while compressing the distance between the same class samples, so that the model can distinguish the feature hierarchy of the safe boundary and the abnormal boundary in the subsequent strategy update process.

[0034] After the contrastive policy embedding layer is updated, the latest embedding space structure is used to reevaluate the historical samples in the experience replay buffer. In this step, the similarity distance of each sample to the safety boundary is calculated, and the sample weight is redistributed according to the distance. The samples closer to the safety boundary are given higher sampling priority and are preferentially put into the training batch in the subsequent incremental update process of the reinforcement learning network, so that the model pays more attention to the samples in the safety critical state during learning.

[0035] In this process, the sample distribution sparsity regularization term ensures the compactness of feature expression, the inter-class separation constraint term strengthens the boundary difference between safe and out-of-limit samples, and the sample reweighting mechanism guides the policy network to learn the control law in the critical state. Finally, the collaborative optimization between the contrastive policy embedding layer and the reinforcement learning network is realized.

[0036] After the contrastive policy embedding layer is trained, the optimized policy input vector is input into the deep reinforcement learning network. The network consists of a policy network and a value network, which is used to generate the control action sequence of the ring main unit and is trained and updated using the trust region policy optimization algorithm.

[0037] In the policy network, the input vector is mapped through multiple layers to output a set of action probability distributions. Each operable switch corresponds to an action distribution node, and its output probability represents the optional weight of the three actions of "closing", "opening" and "keeping". According to the probability distribution, the action combination of the current round is generated by sampling each switch.

[0038] In the value network, the action combination output by the policy network is evaluated in the state, and the expected return in the state is calculated with the current state and reward signal as input. The action distribution of the current policy is recorded as the "old policy distribution" and stored in the experience replay buffer.

[0039] In the next round of training update, the system re-enters the latest state to obtain the "new policy distribution".

[0040] To prevent the training from diverging due to excessive policy changes, the trust region policy optimization algorithm introduces a constraint mechanism in the policy update process: the probability distance between the new and old policy distributions is calculated to measure the magnitude of policy change, and when the distance exceeds the set threshold, the parameter update step is reduced, ensuring that each update is within the preset trust region. The trust region constraint is embedded in the optimization objective in the form of a constraint, so that the parameter update direction always moves in the direction of the approximate optimal gradient, but does not exceed the learning stable interval of the model for the current electrical topology.

[0041] In the strategy network updating process, the expected return is maximized, and multiple weighted items in the reward function are optimized. Among them, the power supply recovery rate as a positive reward item encourages the ring network cabinet to achieve fast power supply recovery; voltage deviation and power flow limit as a penalty item is used to suppress overload and limit behavior; switch action cost as a trade-off item prevents frequent action switching of the strategy; when abnormal actions such as open failure occur, an additional penalty signal is applied.

[0042] The value network is adjusted after each strategy update, providing a reference signal for the direction of strategy update by fitting the cumulative return estimate under the current state, so that the training continues to converge within a stable trust domain.

[0043] After multiple rounds of optimization and update under the trust domain constraint, the strategy network gradually forms a stable mapping of the operation law of the distribution network, and can output a convergent and safe control action distribution under any operating state, ultimately generating a control action sequence for the ring network cabinet, providing an input basis for subsequent safety constraint judgment.

[0044] In this embodiment, after the reinforcement learning network generates the control action sequence, it enters the constraint judgment stage of the control barrier function escort layer. In this stage, the action sequence output by the strategy network is received, where each action corresponds to a specific switch operation instruction in the ring network cabinet. According to the preset electrical safety constraints, each action is screened and judged.

[0045] Perform voltage barrier function judgment. Extract the node voltage values from the latest node voltage monitoring data and compare them with the upper and lower limits of the rated voltage. If any node voltage is detected to be outside the allowed range, the action is determined to be an out-of-limit action, which is prohibited from being directly issued, and the node number and corresponding action are recorded in the out-of-limit set.

[0046] Perform current barrier function judgment. According to the real-time monitoring results of branch current, calculate the current amplitude of each branch. When the current of any branch exceeds its rated thermal stability current, the corresponding action is marked as an overload action and added to the set of actions to be corrected, waiting for subsequent feasible domain projection correction.

[0047] Perform the power flow direction barrier function judgment. By comparing the voltage phase angle and power sign at the tie point, it is determined whether the power flow direction is consistent with the established power supply direction. If the power flow is reversed, it is determined that the action may cause reverse power supply risk, and is also included in the correction queue.

[0048] When any barrier function is triggered, a feasible region projection correction process is started. In this process, the action vector of the original output of the strategy network is taken as a reference, all triggered barrier conditions are read and converted into constraint conditions, and then input into the constraint solving step. In the constraint solving process, an optimization problem containing multiple barrier function constraint terms is constructed, and each barrier condition is expressed in the form of an inequality to ensure that the corrected action does not cause voltage out-of-limit, current saturation or reverse power flow.

[0049] Each constraint condition is expanded in a linear approximation manner, and the constraint boundary is dynamically updated according to the current monitoring data to form a local feasible region corresponding to the current operating state. A projection operation based on constraint optimization is performed to minimize the adjustment amount between the original action and the corrected action, and a feasible solution within the feasible region is calculated.

[0050] After the corrected action is calculated, all action instructions are recombined to form a final control sequence that satisfies the safety constraints. The corrected control sequence is used as the input instruction for the execution of the ring network cabinet operation, and after sequential execution, the safe closing and opening operation is realized, which ensures the safe operation of the entire distribution network while maintaining the continuity of the control.

[0051] In the embodiment, the specific steps of updating the ring network cabinet operating state and recording the reward value include: When the control operation is performed according to the action sequence corrected by the escort layer, the interlocking state of the ring network cabinet switch and the minimum dead time are checked, and when the adjacent switches are in interlocking relationship, simultaneous action is prohibited; The minimum dead time is set to be not less than 200 milliseconds to avoid repeated triggering; After the closing and opening operation is performed, the node voltage, branch current and tie point power flow are quickly recovered, and the recovery result is compared with the strategy prediction value to generate a result confirmation flag; The result confirmation flag is recorded together with the corresponding immediate reward value.

[0052] In the embodiment, the step of incremental updating includes: The experience replay buffer is used to store state, action and reward samples formed during the operation of the ring network cabinet, and a priority sampling mechanism based on time difference error is used for sample extraction; When performing incremental updating, a double-network structure of the target network and the current network is used, and the parameters are replaced by soft updating, and the soft updating coefficient is between 0.005 and 0.01; The batch size of each incremental update is set to 64 to 128 samples, and the network loss function is normalized and smoothed after a plurality of consecutive updates.

[0053] In the embodiment, the low learning rate update includes: The reinforcement learning network is periodically fine-tuned at preset time intervals during operation, and the fine-tuning period is between 30 seconds and 60 seconds; The policy network and the value network respectively use independent learning rates for parameter updating; The contrast policy embedding layer uses a low learning rate for updating, and when the validation set reward value does not improve for a plurality of consecutive periods, the parameter updating of the embedding layer is stopped.

[0054] Embodiment 1 To verify the feasibility and effectiveness of the method in the intelligent regulation and control of the power distribution network, the application is applied to test the actual working conditions of a 10-kilovolt power distribution loop network. The test object includes four loop cabinet nodes, eight branches and one tie point. The power distribution network is mainly industrial load and is connected to a distributed photovoltaic device.

[0055] Voltage sensors, current transformers and state acquisition terminals are installed at the loop cabinet and each branch node to collect node voltage amplitude, phase angle, branch current, active power, reactive power, switch position number, on-off signal and protection action flag. The sampling period is set to 1 second, and the running data is continuously collected for 72 hours. After time synchronization, missing interpolation and normalization processing, the collected data is input into the graph neural network encoder to extract the electrical correlation features between the nodes and the branches, forming the node feature matrix and the edge feature matrix, which are used to construct the electrical topology data structure.

[0056] From the historical running samples, the working conditions with voltage and current within the rated range are selected as safe samples, and the samples with over-limit, reverse power supply or protection tripping are selected as abnormal samples. In the training process, the temperature calibrated cosine similarity is used to measure the contrast learning of positive and negative sample pairs, and the sample sparsity regular term and the inter-class separation constraint term are used to optimize the embedding space distribution, so that the safe samples are clustered and the abnormal samples are separated. After the training converges, the policy input vector output by the embedding layer can stably reflect the current power distribution network operating state.

[0057] The trust region policy optimization algorithm is used to construct the reinforcement learning network, and the state vector optimized by the embedding layer is input to output the loop cabinet control action sequence. In the training process, the power supply recovery rate is used as the positive reward, the voltage deviation and the power flow over-limit are used as the penalty term, and the frequent action of the switch and the failure of opening and closing are used as the additional penalty term. Through multiple rounds of policy iteration, a converged control policy is formed. To ensure the stability of the update, the distance between the new and old policy distributions is calculated when the policy is updated, and when the distance exceeds the trust region threshold, the update step is automatically reduced to avoid training divergence.

[0058] Before executing the control action, the control barrier function escort layer is started to constrain and determine the node voltage, branch current and power flow direction. If any barrier function triggers, the feasible region projection correction is executed, and the feasible solution satisfying the constraint is solved by minimizing the action adjustment amount. The corrected action instruction is issued to the ring network cabinet control terminal to execute the safe closing and opening operation.

[0059] To verify the advantages of the present application, the traditional fixed logic control method is used as a control scheme. In the traditional method, the ring network cabinet control logic is determined according to the static threshold value: when the node voltage is lower than 0.9 times the rated value, the closing operation is executed; when the node voltage is higher than 1.1 times the rated value or the branch current exceeds the rated value, the opening operation is executed; when the protection action signal is detected, the reclosing is triggered, and the power flow direction and multi-node linkage are not considered. The method does not have learning and adaptive ability, and is easy to produce misoperation or delay recovery under the conditions of distributed access, load fluctuation or topology transformation.

[0060] Under the same network condition, the traditional fixed logic control and the method of the present application are respectively used for operation test.

[0061] Table 1 test result table

[0062] As can be seen from Table 1, the method of the present application is obviously superior to the traditional scheme in voltage control, overload suppression and action stability. The introduction of the comparison strategy embedding layer enables the strategy network to identify the abnormal state boundary in the training stage, the reliability domain strategy optimization algorithm ensures the stable convergence of the training process, and the control barrier function escort layer ensures the safe execution of the action. In 72 hours of continuous test, the method of the present application realizes the adaptive, efficient and safe regulation and control of the ring network cabinet, and significantly improves the operation reliability and intelligent level of the distribution system.

[0063] The above describes only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for intelligent control of ring main units based on deep reinforcement learning, characterized in that, Includes the following steps: Collect real-time operation data of the ring main unit and its associated distribution network, establish a distribution network topology model based on the real-time operation data, and generate an electrical topology data structure containing node feature matrices and edge feature matrices; Within a preset time window, the collected data is synchronized and normalized, input into the graph neural network encoder, the correlation features between nodes and branches are extracted, and the initial state vector is output. A contrastive policy embedding layer is set at the input end of the reinforcement learning policy network. State-action samples under different operating conditions are input into the contrastive policy embedding layer for contrastive learning training to obtain the optimized policy input vector. The optimized policy input vector is input into a deep reinforcement learning network, which uses a trust region policy optimization algorithm to construct a decision structure and outputs the control action sequence of the ring network cabinet. Before the execution of the control action sequence, a control barrier function guard layer is set to perform physical constraint judgment and correction on the action. When the strategy output action causes any barrier condition to be triggered, feasible domain projection correction is performed on the strategy output action. Based on the action sequence corrected by the escort layer, control the ring main unit to perform closing and opening operations, update the ring main unit's operating status, and record the reward value; The latest state, actions, and rewards are combined into new samples and written into the experience replay buffer, and the reinforcement learning network is periodically updated incrementally. During operation, the parameters of the reinforcement learning network are periodically fine-tuned, and the contrast policy embedding layer is updated with a low learning rate.

2. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The real-time operating data includes node voltage amplitude and phase angle, branch current, active power, reactive power, tie point status, switch tag number, on / off signal, protection action flag, and branch impedance parameters. When establishing the power distribution network topology model, the bus, feeder end and load node are used as graph nodes, the conduction relationship of the line and ring network switch are used as graph edges, and impedance parameters, rated current, real-time on / off status and power flow direction attributes are set for each graph edge. Based on the correspondence between nodes and edges, node feature matrices and edge feature matrices are constructed. Together, these matrixes form the electrical topology data structure.

3. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The steps for generating the initial state vector include: The preset time window is defined as a time range covering several consecutive sampling periods, and the running data within the time range constitutes a running data sequence. Time synchronization is achieved through time-scale alignment and missing interpolation, and the running data sequence is normalized to convert voltage, current and power data into a unified dimensional form. When extracting the association features between nodes and branches, the local neighborhood set of each node is calculated based on the adjacency relationship of the topology graph. The features of the neighborhood nodes and connected edges are weighted and aggregated to obtain node-level aggregated features. The node-level aggregated features are updated through a nonlinear transformation layer to form a state representation vector. The state representation vector is reduced in dimension by a linear mapping layer and then output as the initial state vector.

4. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The comparative learning training includes: The contrast strategy embedding layer is used to perform contrast learning training on state-action samples under different operating conditions, including an encoding sub-layer, a sample construction module, and a similarity calculation module; The encoding sublayer maps the input state vector to a low-dimensional embedding representation. The sample construction module uses samples that meet the voltage and current limits as positive samples and samples that exceed the limits or are abnormal as negative samples. The similarity calculation module uses temperature-calibrated cosine similarity to distinguish between positive and negative sample pairs, and optimizes the embedding space distribution by minimizing the contrast loss function; The embedded result output after training is used as the optimized policy input vector.

5. The intelligent control method for ring main units based on deep reinforcement learning according to claim 4, characterized in that, The comparison strategy embedding layer also includes: During training, a sample distribution sparsity regularization term and an inter-class separation constraint term are introduced to maximize the center distance between safe sample clusters and over-limit sample clusters and minimize intra-cluster variance. After each policy network update, the sampling probability of empirical replay samples is reweighted based on the sample similarity in the embedding space, prioritizing the learning of samples near the safety boundary.

6. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The steps for generating the control action sequence of the ring main unit include: The deep reinforcement learning network uses a trust-domain policy optimization algorithm to construct a decision structure, including a policy network and a value network. The policy network receives the input vector optimized by the comparison policy embedding layer and outputs the action probability distribution of each operable switch according to the current state. The actions include closing, opening and holding states. The value network is used to evaluate the expected return corresponding to the current state, and sets a trust region constraint on the KL divergence between the policy distribution and the old policy distribution during policy updates to limit the update step size; During the training process, the power supply recovery rate is used as a positive reward, and voltage exceeding the limit, current saturation, and frequent switching are used as penalties. The control action sequence for ring main unit regulation is obtained through weighted optimization.

7. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The control barrier function protection layer includes a voltage barrier function, a current barrier function, and a power flow direction barrier function: The voltage barrier function is used to constrain node voltage values ​​to remain within the upper and lower limits of the rated voltage; The current barrier function is used to constrain the branch current from exceeding its rated thermal stability current value. The power flow direction barrier function is used to constrain the power flow direction at the tie point to be consistent with the predetermined power supply direction. When it is detected that the policy output action causes any barrier function value to exceed the preset threshold, a feasible region projection correction based on quadratic programming is performed. The action vector output by the original policy is used as a reference to construct a constrained optimization problem under the constraints defined by each barrier function. In the feasible region projection correction process based on quadratic programming, the gradient information of each barrier function and the active set of constraints are calculated in real time, and the nonlinear barrier constraints are transformed into local linear inequality constraints by incremental linearization approximation. During the solution process, a safety margin coefficient is set, and the constraint boundary is dynamically adjusted according to the measurement uncertainty of each electrical quantity. With the goal of minimizing the action adjustment amount, a feasible solution that satisfies the constraints is obtained, and the feasible solution is used as the corrected control command.

8. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The specific steps for updating the operating status of the ring main unit and recording the reward value include: When performing control operations based on the action sequence corrected by the escort layer, the interlock status and minimum dead time of the ring main unit switches are checked. When adjacent switches are in an interlock relationship, simultaneous operation is prohibited. The minimum dead time is set to no less than 200 milliseconds to avoid repeated triggering; After performing the closing and opening operations, the node voltage, branch current, and power flow at the tie point are quickly sampled. The collected data is then compared with the predicted values ​​from the strategy to generate a result confirmation flag. Record the result confirmation flag along with the corresponding instant reward value.

9. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The incremental update steps include: The experience playback buffer is used to store the status, action and reward samples formed during the operation of the ring network cabinet, and the sample extraction is performed using a priority sampling mechanism based on time-series difference error; When performing incremental updates, a dual-network structure of the target network and the current network is used, and parameters are replaced by a soft update method with a soft update coefficient between 0.005 and 0.

01. The batch size for each incremental update is set to 64 to 128 samples, and the network loss function is normalized and smoothed after several consecutive updates.

10. The intelligent control method for ring main units based on deep reinforcement learning according to claim 1, characterized in that, The low learning rate update includes: The reinforcement learning network is periodically fine-tuned at preset time intervals during operation, with the fine-tuning period ranging from 30 to 60 seconds. The policy network and the value network are updated using independent learning rates. The contrast strategy embedding layer is updated with a low learning rate, and the embedding layer parameter updates are stopped when the validation set reward value does not improve within a certain number of consecutive periods.

Images