Microgrid voltage active control method and system based on fusion experience safety reinforcement learning

Abstract

The application relates to a micro-grid voltage active control method and system based on fusion experience safety reinforcement learning, which comprises the following steps: constructing a micro-grid constraint Markov decision process model; constructing an Actor-Critic continuous control network and an original-dual safety reinforcement learning optimization framework; connecting a safety action correction layer to the back end of a strategy network to correct an original action to obtain an actual execution action; calculating a correction deviation between the original action and the actual execution action to generate a single-step real experience; identifying a continuous correction event, dividing normal and event samples, and respectively storing the normal and event samples into normal and event experience pools; adaptively extracting samples from the experience pools to train the strategy network; based on the trained strategy network, performing micro-grid voltage active control, storing single-step real experiences in the control process into the experience pools, and periodically updating network parameters. The application can improve inverter reactive power regulation resource allocation, improve the training efficiency of safety boundary working conditions and the micro-grid voltage control capability.

Description

Technical Field

[0001] This invention belongs to the field of voltage control technology for microgrids and active distribution networks, and in particular relates to a method and system for active voltage control of microgrids based on fusion of empirical security reinforcement learning. Background Technology

[0002] As the integration of distributed photovoltaic (PV), energy storage, and other power electronic interface devices into microgrids continues to increase, random variations in renewable energy output and load power can easily lead to node voltage exceeding limits and line current overload. Distributed power inverters, with their ability to rapidly regulate reactive power, can be used to achieve real-time regulation of microgrid node voltages.

[0003] Reinforcement learning methods can autonomously generate reactive power regulation actions for distributed generation inverters based on the microgrid's operating state. However, in the early stages of strategy training, under conditions of sudden changes in source and load, or in operating regions where learning is insufficient, the strategy network may generate actions that do not meet node voltage constraints, line current constraints, or inverter non-functional capacity constraints. To reduce operational risks, existing methods typically set up a safety action correction layer after the strategy network outputs the original actions. Based on the voltage-reactive power sensitivity matrix, the original actions are corrected to actual execution actions that meet the predicted operating constraints. Existing methods usually evaluate strategy risk based on whether a safety constraint violation occurs in the current control cycle, making it difficult to identify whether the strategy network will output actions requiring significant correction by the safety correction layer for multiple consecutive control cycles before an explicit violation occurs. The action correction deviation in a single control cycle is easily affected by random source and load disturbances and measurement noise. Directly adjusting the safety constraint penalty or empirical sampling probability based on the instantaneous correction deviation can easily lead to frequent changes in training parameters. By performing sliding window statistics on the safety action correction deviation, it is possible to reflect whether the inconsistency between the strategy network's output actions and safe executable actions is persistent. However, simply using the window average correction deviation as an additional parameter in the original-dual update formula or empirical priority formula still cannot determine the specific node voltage constraint or line current constraint that causes the action correction, nor can it determine which inverter and which reactive power regulation direction should bear the safety correction amount. Furthermore, fixed safety action correction targets typically use Euclidean distances with equal weights for each action component. When different inverters have different reactive power regulation margins, historical correction frequencies, and actual regulation effects, equal-weight correction can easily cause some high-sensitivity inverters to be repeatedly called within continuous control cycles, leading to these inverters gradually approaching their reactive power limits, while the adjustable capabilities of other inverters are not fully utilized. Existing empirical playback methods typically calculate sampling priorities based on single-step timing differential errors or constraint violations, making it difficult to retain the continuous state evolution process before and after changes in the window average correction deviation from low to high levels, switching of activation constraints, and changes in the main correction inverter. Directly weighting the states, actions, and next states of different control cycles may also result in empirical samples that do not correspond to the actual power flow state transitions.

[0004] Chinese patent application CN121436615A discloses a distributed photovoltaic scheduling method based on multi-agent consensus optimization. It models the real-time reactive power optimization decision of photovoltaic inverters as a networked multi-agent Markov decision process, maps minimizing voltage deviation and minimizing network loss to reward functions, employs a multi-agent soft actor-critic method, and constrains the local policies and value function parameters of each agent to converge by adding a consensus penalty term, and introduces a constraint violation priority experience replay mechanism. However, it still has the following shortcomings: Although the scheme introduces a constraint violation priority experience replay mechanism, its experience sample priority is mainly determined based on the degree of constraint violation or value function error information, and it does not use the deviation between the original action output by the policy network and the actual executed action after safety correction as an operational risk feature in the experience sample selection. Therefore, for boundary operation samples where there has been no explicit voltage or line current exceedance, but the original control action has been significantly corrected by the safety action correction layer, the existing method cannot promptly increase its training weight, resulting in insufficient learning of executable actions near the safety boundary by the policy network, and the online operation phase may still rely on post-processing safety correction for a long time.

[0005] Chinese patent application CN118763705A discloses a method for determining distribution network control strategies. It determines an initial optimization model of the distribution network using historical operating data and a pre-established mathematical model of charging stations. The initial optimization model is then trained using a pre-defined actor network, critic network, and voltage sensitivity matrix to obtain a target optimization model. This target optimization model is used to determine the collaborative optimization control strategy for the distribution network. However, this method still has the following shortcomings: Although it utilizes the voltage sensitivity matrix for optimization training of the control strategy, the safety correction process is mainly treated as an independent processing step before and after strategy execution. It does not feed back the deviation between the safety correction action and the original action into the reinforcement learning training process, nor does it dynamically adjust the dual variables or safety penalty intensity in the original-dual optimization process based on this deviation. Therefore, when the strategy network continuously outputs control actions requiring significant correction, the existing method struggles to identify inconsistencies between the strategy output action and the safe executable action in a timely manner. This can easily lead to a disconnect between the training objective and the online execution action, affecting the learning effect of the control strategy near the safety boundary.

[0006] Therefore, to address the problem that existing microgrid voltage reinforcement learning control methods suffer from the disconnect between the safety action correction module, the primal-dual constraint optimization module, and the experience playback module, which makes it difficult for the policy network to learn executable actions near the safety boundary in a timely manner and results in insufficient representation of potential risk conditions by training samples, it is necessary to propose a microgrid voltage active control method that retains the instantaneous safety action correction deviation and the sliding window average safety action correction deviation, and combines the window average safety action correction deviation with specific constraint identification, correction resource allocation, policy training, and continuous event experience playback. Summary of the Invention

[0007] To address the challenges of existing microgrid reinforcement learning-based voltage control methods in identifying the continuous dependence of the policy network on the safety action correction layer and in rationally allocating safety correction amounts based on the directional residual regulation capability, historical correction burden, and actual closed-loop regulation effect of different inverters, this invention provides a microgrid active voltage control method and system based on fused empirical safety reinforcement learning.

[0008] The present invention adopts the following technical solution.

[0009] In a first aspect, the present invention discloses a microgrid voltage active control method based on fusion empirical security reinforcement learning, the method comprising the following steps: Step 1: Construct a constrained Markov decision process model for the microgrid, defining the continuous action space consisting of the reactive power regulation of each distributed power inverter, and the state space consisting of the real-time operating status data of the microgrid. Step 2: Based on the microgrid constrained Markov decision process model, construct an Actor-Critic continuous control network including a policy network and an evaluation network, and establish a primitive-dual security reinforcement learning optimization framework. Step 3: Connect a safety action correction layer in series at the back end of the strategy network to perform safety correction on the original reactive power regulation action output by the strategy network to obtain the actual executed action; calculate the correction deviation between the original reactive power regulation action and the actual executed action, and generate single-step real experience; the original reactive power regulation action is the original reactive power regulation amount of the distributed power inverter. Step 4: Based on the single-step real experience, identify the continuous correction event, divide it into regular samples and event samples, store the regular samples in the regular experience pool, and store the event samples in the event experience pool. Step 5: Adaptively draw samples from the regular experience pool and the event experience pool to train the policy network; wherein, the policy network loss function is constructed based on the correction bias and the average residual correction bias obtained by replaying and correcting the event samples after the policy network is updated; Step 6: Input the real-time operating status data of the microgrid into the trained strategy network, output the original reactive power regulation action, send it to the safety action correction layer to obtain the actual execution action and send it to each distributed power inverter to perform voltage regulation; and store the single-step real experience generated during the voltage regulation process into the regular experience pool and the event experience pool respectively, and periodically update the network parameters of the strategy network and the evaluation network.

[0010] More preferably, In step 3, in the safety action correction layer, an optimization model for the safety action correction amount is constructed and solved. The solved safety action correction amount is superimposed with the original reactive power adjustment amount as the actual action to be executed. The safety action correction amount includes reactive power increase direction correction amount and reactive power decrease direction correction amount. The objective function of the optimization model is:

[0011] in, , They are respectively t The set vector of reactive power directional corrections for all distributed power inverters at any given moment; , They are respectively t The reactive power directional correction of the j-th distributed power inverter at time j, and satisfying the following conditions: , ; , They are respectively t Time of the first j The correction weights corresponding to the increase and decrease of reactive power direction correction in the distributed power inverter. This represents the total number of distributed power inverters.

[0012] More preferably, The correction weights corresponding to the reactive power increase and reactive power decrease corrections of the distributed power inverter are determined as follows:

[0013]

[0014] in, and They are respectively t Time of the first j The normalized reactive power direction residual regulation margin and reactive power direction residual regulation margin of the distributed power inverter. and They are respectively t Time of the first The historical cumulative correction burden of the inverter in the reactive power increase and decrease directions after normalization; For cumulative correction burden factor; To prevent positive numbers with a denominator of zero; , These are the lower and upper limits of the correction weights, respectively. This means that the calculation results will be limited to the preset upper and lower limits.

[0015] More preferably, The historical cumulative correction load of the inverter in the reactive power increase and reactive power decrease directions is determined as follows:

[0016]

[0017] in, , for t +1 、t Time of the first Taiwan inverters increase the historical cumulative correction burden of reactive power direction; , for t +1 、t Time of the first Taiwan inverters reduce the historical cumulative correction burden of reactive power direction; For the burden of forgetting coefficient, and .

[0018] More preferably, In step 4, the specific method for identifying the continuous correction event is as follows: The average value of the correction deviation between the original reactive power adjustment action and the actual executed action within the sliding window is calculated as the window average safe action correction deviation. When the window average safe action correction deviation is greater than the upper limit of the preset correction deviation threshold, it indicates that the strategy network is in a high continuous correction state; when the window average safe action correction deviation is less than the lower limit of the preset correction deviation threshold, it indicates that the strategy network is in a low continuous correction state. Based on the predicted node voltage and predicted line current after safety correction, the relaxation amount of each safety constraint is solved, and a safety constraint activation identifier vector is generated by combining the preset activation judgment tolerance; the safety constraints include the upper and lower limit safety constraints of node voltage and the upper limit constraint of line current. Normalize the increase and decrease of reactive power directional correction for each distributed power inverter to obtain the directional correction responsibility vector; A continuous correction event is triggered when at least one of the following conditions is met: 1) The continuous calibration state switches from low continuous calibration state to high continuous calibration state, or from high continuous calibration state to low continuous calibration state; 2) The safety constraint activation flag vector changes relative to the previous control cycle; 3) The average safety action correction deviation of the window is greater than the lower limit of the preset correction deviation threshold, and the top M elements with the largest values ​​in the direction correction responsibility vector have changed relative to the previous control cycle, where M is the preset number of direction correction elements, and the following conditions are met: .

[0019] More preferably, The safety constraint activation identifier vector is composed of the node voltage lower limit constraint activation identifier vector, the node voltage upper limit constraint activation identifier vector, and the line current upper limit constraint activation identifier vector; the node voltage lower limit constraint activation identifier vector Node voltage upper limit constraint activation identifier vector Line current upper limit constraint activation flag vector ,in, for t Time Node i Predicted node voltage, for t Timetable l Predicted line current, , They are nodes i Under the voltage, upper limit, For the line l The upper limit of current, , These are the preset activation judgment tolerances for node voltage constraints and line current constraints, respectively. This is an indicator function; it takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

[0020] More preferably, The normalized reactive power directional correction values ​​for each distributed power inverter are adjusted to obtain a directional correction responsibility vector, which specifically includes: For any distributed generation inverter, its correction responsibility in the reactive power increase direction is calculated as follows:

[0021] Its correction responsibility in the reactive power reduction direction is calculated as follows:

[0022] After calculating the reactive power directional correction responsibility degree of all distributed power inverters, they are sequentially arranged and concatenated into a directional correction responsibility degree vector. ;in, , They are respectively t Time of the first j The responsibility for reactive power direction correction of distributed power inverters in Taiwan. A positive number set to prevent the denominator from being zero; , They are respectively t Time of the first m The reactive power direction correction amount of the distributed power inverter; , They are respectively t Time of the first mThe correction weights corresponding to the increase and decrease of reactive power direction correction in the distributed power inverter.

[0023] More preferably, When a continuous correction event is triggered, the data before the continuous correction event occurs is captured. One control cycle until the event occurs. Continuous real-world experience from each control cycle forms an event experience sequence, which is then stored as an event sample in an event experience pool; wherein... To continuously correct the number of control cycles intercepted before the event is triggered, To continuously correct the number of control cycles captured after the event is triggered.

[0024] More preferably, In step 5, the average residual correction bias obtained by replaying and recalibrating the event samples after the policy network update is calculated as follows: First, calculate the residual correction bias corresponding to a single event experience sequence. The calculation formula is as follows:

[0025] For all event experience sequences in the event experience pool, the mean of the residual correction bias after the most recent training update for each sequence is calculated to obtain the average residual correction bias. The calculation formula is as follows:

[0026] in, For the first n The event experience sequence in the th ... r Residual correction bias after the next strategy update; In the first r The policy network after the second parameter update, input to the first parameter... n Within the empirical sequence of continuous correction events, the [number]th The original reactive power regulation action is re-verified after the historical microgrid operating status is determined. To re-verify the actual execution action obtained by re-executing the safety correction mapping on the original reactive power regulation action; The length of the event experience sequence; For the first n A continuous sequence of corrective events; for t The average residual correction bias is obtained by replaying and recalibrating the event samples at each time point after the policy network update. This represents the total number of continuously correcting event experience sequences in the current event experience pool; For the first n A sequence of events in its most recent training iteration The corresponding residual correction deviation.

[0027] More preferably, In step 5, the adaptive sampling of samples from the regular experience pool and the event experience pool to train the policy network specifically includes: Based on the average safety action correction deviation of the window and the average residual correction deviation of the event samples obtained by replaying and correcting after the policy network update, the event experience extraction ratio in the training batch is calculated; samples are extracted from the regular experience pool and the event experience pool for training according to the event experience ratio. The event experience extraction ratio is as follows:

[0028] in, The proportion of basic event experience; Adjustment coefficient for average safety action correction deviation of the window; This is the adjustment coefficient for the average residual correction deviation in the event pool; for t Average safety action correction deviation within the time window; and These are the lower and upper limits of the event experience ratio, respectively.

[0029] More preferably, In step 5, the policy network loss function is specifically as follows:

[0030] in, This represents the total number of samples in a single training session. This represents the microgrid operating state corresponding to the nth empirical sample. , is the original reactive power regulation action of the j-th distributed power inverter output by the strategy network for the n-th empirical sample; The actual execution action of the j-th distributed power inverter is obtained after the original reactive power regulation action has been safety-corrected. The comprehensive correction responsibility of the j-th distributed power inverter in the n-th empirical sample is the sum of the reactive power direction correction responsibilities of the distributed power inverter. For parameters Evaluation network; To improve the intensity of movement consistency training; The intensity of the motion consistency training Specifically:

[0031] in, Intensity of basic movement consistency training; and These are the lower and upper limits of the training intensity for motion consistency, respectively, and satisfy the following conditions: ; The adjustment coefficient for the average safe action correction deviation of the online window. Let be the average residual correction deviation adjustment coefficient in the event pool, and satisfy . , .

[0032] Secondly, this invention discloses a microgrid voltage active control system based on the aforementioned method and fusion experience-based security reinforcement learning, including a microgrid constrained Markov decision process model construction module, an Actor-Critic network construction module, a single-step real experience generation module, an event recognition and dual-experience pool classification and storage module, a dual-pool adaptive sampling and policy network training module, and an online control and network periodic update module. The microgrid constrained Markov decision process model construction module constructs a microgrid constrained Markov decision process model, defines the continuous action space consisting of the reactive power regulation of each distributed power inverter, and the state space consisting of the real-time operating status data of the microgrid. The Actor-Critic network construction module, based on the microgrid constrained Markov decision process model, constructs an Actor-Critic continuous control network including a policy network and an evaluation network, and establishes a primitive-dual security reinforcement learning optimization framework. The single-step real experience generation module connects a safety action correction layer in series at the back end of the strategy network to perform safety correction on the original reactive power adjustment action output by the strategy network to obtain the actual execution action; calculates the correction deviation between the original reactive power adjustment action and the actual execution action, and generates single-step real experience; the original reactive power adjustment action is the original reactive power adjustment amount of the distributed power inverter. The event recognition and dual experience pool classification and storage module continuously corrects events based on the single-step real experience recognition, divides regular samples and event samples, stores regular samples in the regular experience pool, and stores event samples in the event experience pool. The dual-pool adaptive sampling and policy network training module adaptively extracts samples from the regular experience pool and the event experience pool to train the policy network; wherein, the policy network loss function is constructed based on the correction bias and the average residual correction bias obtained by replaying and recorrecting the event samples after the policy network is updated; The online control and network periodic update module inputs real-time operating status data of the microgrid into the trained strategy network, outputs the original reactive power regulation action, sends it to the safety action correction layer to obtain the actual execution action, and sends it to each distributed power inverter to perform voltage regulation; and stores the single-step real experience generated during the voltage regulation process into the regular experience pool and the event experience pool respectively, and periodically updates the network parameters of the strategy network and the evaluation network.

[0033] Thirdly, the present invention provides a terminal, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of the first aspects of the present invention.

[0034] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects of the present invention.

[0035] The beneficial effects of this invention are compared with those of the prior art: This invention calculates the instantaneous safety action correction deviation between the original reactive power adjustment action output by the calculation strategy network and the actual execution action output by the safety action correction layer, and further calculates the average safety action correction deviation of the sliding window. This can reduce the impact of source load disturbance and measurement noise on the judgment of the strategy safety status within a single control cycle, thereby identifying the degree of continuous dependence of the strategy network on the safety action correction layer.

[0036] This invention determines the directional correction weight based on the remaining regulation margin of each distributed power inverter in the reactive power increase and decrease directions and the historical cumulative correction burden. This allows the regulation directions with larger remaining regulation margins and smaller recent correction loads to participate in safety operation corrections first, reducing the situation where some inverters are repeatedly called up in continuous control cycles and improving the utilization balance of inverter reactive power regulation resources.

[0037] This invention identifies continuous correction events based on changes in continuous correction state, changes in safety constraint activation state, and changes in the main direction correction path, and stores the continuous real experience before and after the event in the event experience pool, which can preserve the continuous change process of operating state, activation constraints, and main correction direction under safety boundary conditions.

[0038] This invention involves updating the policy network after the event experience is involved, then using the updated policy network to replay and recalibrate the historical event states, and adjusting the sampling priority, sampling ratio, and action consistency training intensity of the event experience based on the residual calibration deviation. This creates a closed loop between the safety action calibration results, the event experience replay, and the policy network training, thereby reducing the probability that the policy network will output actions requiring significant safety calibration again under similar operating conditions. Attached Figure Description

[0039] Figure 1 This is a flowchart of the microgrid voltage active control method based on fusion of empirical security reinforcement learning, as described in this invention. Figure 2 The voltage fluctuation diagram is from the effectiveness test of Embodiment 1 of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The embodiments described in this application are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this invention.

[0041] like Figure 1 As shown, this invention discloses a microgrid voltage active control method based on fusion empirical security reinforcement learning, the method comprising the following steps: Step 1: Construct a constrained Markov decision process model for the microgrid, defining the continuous action space consisting of the reactive power regulation of each distributed power inverter, and the state space consisting of the real-time operating status data of the microgrid. The real-time operating status data of the microgrid includes real-time operating data such as the voltage of each node, line current, and load of the microgrid. Specifically, the microgrid constrained Markov decision process model includes the following components: 1) State Space This includes real-time operating data such as voltage at each node of the microgrid, line current, and load, used to indicate the current state of the system.

[0042] 2) Action Space This includes the reactive power regulation of each distributed generation (DG), used to indicate the control operations that the system can take.

[0043] 3) State transition probability : Describes how the microgrid state transitions to the next state based on control operations.

[0044] 4) Reward Function This function is defined based on voltage deviation, network loss, and system stability, and the reward value reflects the effectiveness of the control strategy in the current state.

[0045] 5) Constraint cost function : Used to represent voltage and current constraints, ensuring that the system always meets physical limitations such as voltage and current during operation.

[0046] 6) Discount Factor Used to control the impact of future states, ensuring that the learning algorithm prioritizes the optimization of recent states.

[0047] By constructing a CMDP model, this invention can effectively evaluate the effects of different control strategies in dynamic power grids and ensure that these strategies meet safety constraints.

[0048] Specifically, the microgrid voltage control problem is modeled as a constrained Markov decision process, and the model expression is as follows:

[0049] in, For state space; For continuous action space; The state transition probability; For the reward function; For the constraint cost function; This is the discount factor.

[0050] The state space Inside t The running state vector corresponding to time step is defined as:

[0051] in For node voltage vectors, For line current vector, The active power generated by the distributed generation source, and the current reactive power vector of the distributed generation inverter. The data is collected in real time by the controller during each control cycle; , These are the active and reactive power vectors of the load, respectively.

[0052] The action space Inside t The action vector corresponding to time step is represented as:

[0053] in for t Action vector at time step, for tThe reactive power regulation vector of the distributed power inverter at any given time. for t Time of the first i Reactive power regulation of a distributed power inverter This represents the number of distributed power inverters.

[0054] Design voltage and line current constraints to ensure that the voltage and current values ​​in the model do not exceed the set safety limits: 1) Node voltage constraints:

[0055] in: Indicates at time t No. i The voltage of each node; and They are the first i The lower and upper voltage limits for each node.

[0056] 2) Line current constraints:

[0057] in: Indicates at time t No. l The current in the line; is the maximum safe current for the i-th line.

[0058] Construct a reward function that includes voltage deviation and network loss metrics, and define the reward function as follows:

[0059] in: Reference voltage; Network active power loss; These are the weighting coefficients; This represents the total number of nodes within the microgrid.

[0060] A safety constraint cost function is constructed to characterize voltage and line current overruns, ensuring that the system does not violate voltage and current safety constraints during operation; this function can be expressed as:

[0061] in: , These represent the deviations when the voltage exceeds the upper limit or falls below the lower limit, respectively. This indicates the deviation when the line current exceeds the maximum limit; This represents the total number of current lines within the microgrid.

[0062] Step 2: Based on the microgrid constrained Markov decision process model, construct an Actor-Critic continuous control network including a policy network and an evaluation network, and establish a primitive-dual security reinforcement learning optimization framework. The Actor-Critic continuous control network includes a policy network and an evaluation network. The policy network takes the state space data defined in step 1 as input and outputs the original reactive power adjustment actions corresponding to the continuous action space dimension. The evaluation network takes the joint information of state and reactive power actions as input and outputs the state action value to guide the policy network update. The original-dual secure reinforcement learning optimization framework is a top-level constrained optimization solution system with the Lagrangian function as the optimization objective. The Actor-Critic continuous control network is the computational carrier for iteratively solving the optimization objective under this framework. In the aforementioned primitive-dual secure reinforcement learning optimization framework, the objective function is specifically:

[0063] in: For policy network parameters; As dual variables; This is the maximum average security cost threshold allowed by the system. Indicating in strategy Under the sampled trajectory, the long-term expectation of the index within parentheses is calculated.

[0064] The parameters of the policy network are updated using the gradient ascent algorithm. Specifically, the optimized grid control strategy parameters are updated as follows:

[0065] in, The step size for updating the policy network parameters is also called the policy network learning rate; For Lagrange functions Regarding policy network parameters The gradient.

[0066] Step 3: Connect a safety action correction layer in series at the back end of the policy network to perform safety correction on the original reactive power adjustment action output by the policy network to obtain the actual execution action; calculate the correction deviation between the original reactive power adjustment action and the actual execution action, and generate single-step real experience. The original reactive power regulation action is the original reactive power regulation amount of the distributed power inverter. In the safety action correction layer, an optimization model for the safety action correction amount is constructed and solved. The solved safety action correction amount is superimposed with the original reactive power adjustment amount as the actual action to be executed. The safety action correction amount includes reactive power increase direction correction amount and reactive power decrease direction correction amount. Specifically, in the safety action correction layer, the calculation of the actual executed action includes the following steps: Step 3.1: Obtain the raw reactive power control actions output by the policy network; The original action output by the policy network is represented as:

[0067] in, For the policy network in state The original reactive power regulation action of the output is as follows: For parameters The strategy network.

[0068] Step 3.2: Divide the bidirectional correction amount and define the actual execution action after correction; Specifically, the safety action correction amount is decomposed into reactive power increase direction correction amount and reactive power decrease direction correction amount, and the actual action is executed. Represented as:

[0069] in, To increase the reactive power directional correction, To reduce reactive power directional correction.

[0070] Step 3.3: Calculate the remaining adjustment margin of reactive power direction correction for each inverter; Define the directional residual reactive power adjustment margin:

[0071]

[0072] in, and They are time points No. The residual regulation margin in the reactive power direction and the residual regulation margin in the reactive power reduction direction of the distributed power inverter; Before executing the current control action, the first The reactive power of a distributed power inverter; The original reactive power regulation action output by the policy network; and The first The upper and lower limits of reactive power for the distributed power inverter are defined. A maximum value function is used to ensure that the directional residual regulation margin is not less than zero.

[0073] Step 3.4: Calculate the historical cumulative correction load for each inverter in the direction of reactive power increase and decrease;

[0074]

[0075] in, , for t +1 、t Time of the first Taiwan inverters increase the historical cumulative correction burden of reactive power direction; , for t +1 、t Time of the first Taiwan inverters reduce the historical cumulative correction burden of reactive power direction; For the burden of forgetting coefficient, and When an inverter repeatedly undertakes the same direction's motion correction within multiple consecutive control cycles, its historical cumulative correction burden in the corresponding direction increases.

[0076] Step 3.5: Based on the remaining adjustment margin of the reactive power correction in the increasing and decreasing directions of each inverter and the historical cumulative correction burden, calculate the correction weight of each inverter in the reactive power increasing and decreasing directions; The correction weights are determined in the following manner:

[0077]

[0078] in, and They are time points No. The correction weights of the distributed power inverter in the reactive power increase and reactive power decrease directions; and They are respectively t Time of the first j The normalized reactive power direction residual regulation margin and reactive power direction residual regulation margin of the distributed power inverter. and The first The historical cumulative correction burden of the inverter in the reactive power increase and decrease directions after normalization; The cumulative correction burden coefficient is used to adjust the influence of historical cumulative correction burden on the direction correction weight. The preferred value range is 0.1 to 5, and in this embodiment, it is preferably 1. To prevent positive numbers with a denominator of zero, the preferred value range is 10. -6 ~10-2 In this embodiment, 10 is preferred. -3 ; and These are the lower and upper limits of the direction correction weight, respectively, with preferred values ​​ranging from 0.1 to 1 and from 2 to 20, respectively, and satisfying that the lower limit of the direction correction weight is less than the upper limit of the direction correction weight. In a preferred embodiment, the lower limit of the direction correction weight is 0.2 and the upper limit of the direction correction weight is 10. This means that the calculation results are limited to the preset upper and lower limits; the preferred value range for the burden-forgetting coefficient is 0.8 to 0.99, and more preferably 0.9 to 0.95.

[0079] When the residual reactive power regulation margin of a certain inverter in a certain direction is large and the cumulative correction amount undertaken recently is small, the correction weight of its corresponding direction is reduced, so that the safety action correction optimization prioritizes the use of regulation resources in that direction; when the corresponding direction is close to the reactive power limit or has been continuously called recently, the correction weight of its corresponding direction is increased to reduce the continued frequent calling of that direction.

[0080] Step 3.6: Construct and solve the optimization model for the safety action correction amount, and superimpose the solved safety action correction amount with the original reactive power regulation amount as the actual action to be executed; Based on the correction weights of each distributed power inverter in the reactive power increase and decrease directions, and with the goal of minimizing the weighted sum of squares of the bidirectional corrections, a safety action correction optimization problem is constructed:

[0081] in, , They are respectively t The set vector of reactive power correction values ​​for all distributed power inverters at any given moment; , They are respectively t The reactive power directional correction of the j-th distributed power inverter at time j, and satisfying the following conditions: , ; , They are respectively t Time of the first j The correction weights corresponding to the increase and decrease of reactive power direction correction in the distributed power inverter. Corrected actual execution action for:

[0082] The actual actions performed satisfy the node voltage constraints:

[0083] Meets line current constraints:

[0084] And it satisfies the no-function constraint of distributed power inverters:

[0085] in, The original reactive power regulation action output by the policy network; The actual actions output by the safety action correction layer and actually sent to the distributed power inverters are vectors composed of the actual actions executed by each distributed power inverter. For the first j The actual actions performed by the distributed power inverter; The reactive power of the j-th distributed power inverter before the current control cycle action is executed; and These are the voltage-reactive power sensitivity matrix and the line current-reactive power sensitivity matrix at time t, respectively. and These are the node voltage vector and the line current vector at time t, respectively; , These are the lower and upper bound vectors of the node voltage, respectively. This is a vector of upper limits for line current, consisting of the maximum allowable current for each line. , The first j The reactive power lower and upper limits of a distributed power inverter.

[0086] Furthermore, the voltage-reactive power sensitivity matrix is ​​defined as follows:

[0087] in, For a moment The voltage-reactive power sensitivity matrix; For a moment The node voltage vector; For a moment The reactive power vector of the distributed power inverter; For a moment The operating status of the microgrid; Indicates the current running state Nearby, the partial derivative of the node voltage with respect to the reactive power regulation of the distributed power inverter.

[0088] Furthermore, the line current-reactive power sensitivity matrix is ​​defined as follows:

[0089] in, For a moment Line current-reactive power sensitivity matrix; For a moment The line current vector; Indicates the current running state Nearby, the partial derivative of the line current with respect to the reactive power regulation of the distributed power inverter.

[0090] Expanding on this, the voltage-reactive power sensitivity matrix can be expressed as:

[0091] in, For the number of microgrid nodes, The number of distributed power inverters. Indicates the first The reactive power change of the distributed power inverter affects the first The degree of influence of node voltage.

[0092] The line current-reactive power sensitivity matrix can be expressed as:

[0093] in, For the number of microgrid lines, Indicates the first The reactive power change of the distributed power inverter affects the first The degree of influence of the line current.

[0094] The sensitivity matrix is ​​calculated based on the power flow Jacobian matrix. Assume the current operating state... The following trend Jacobian matrix is:

[0095] in, Let be the partial derivative matrix of the node active power with respect to the voltage phase angle. Let be the partial derivative matrix of the node active power with respect to the voltage magnitude. Let be the partial derivative matrix of nodal reactive power with respect to voltage phase angle. Let be the partial derivative matrix of the node reactive power with respect to the voltage amplitude.

[0096] When the reactive power of a distributed generation inverter changes slightly, the change in node power and the changes in node voltage phase angle and amplitude satisfy the following:

[0097] During reactive voltage control, active disturbances can be controlled. And the reactive power regulation of the distributed power inverter. If the reactive power injection at the node changes, then:

[0098] in, This is a mapping matrix representing the change in reactive power of a distributed power inverter to the change in reactive power injection at a node.

[0099] Therefore, the voltage-reactive power sensitivity matrix can be obtained from the following equation:

[0100] in, Extract the voltage amplitude component matrix, used to extract from Extracting the node voltage amplitude change .

[0101] Furthermore, suppose the line current can be expressed as a function of the node voltage magnitude and phase angle:

[0102] The line current-reactive power sensitivity matrix can then be obtained by chain rule differentiation:

[0103] in, It represents a function that calculates line current from node voltage phase angle and node voltage magnitude.

[0104] In another implementation, when it is inconvenient to directly obtain the power flow Jacobian matrix, the sensitivity matrix can be calculated using the finite difference method. For the first... A distributed power inverter, while keeping the reactive power output of other distributed power inverters constant, [is used to address the issue of]... Apply small perturbation Then we have:

[0105] in, The voltage-reactive power sensitivity matrix is ​​the first... Line 1 Column element, representing the first The reactive power disturbance of the distributed power inverter affects the first The impact of node voltage.

[0106] Similarly, in the line current-reactive power sensitivity matrix, the first... Line 1 Column elements can be represented as:

[0107] in, Indicates the first The reactive power disturbance of the distributed power inverter affects the first The influence of line current.

[0108] Furthermore, Based on the actual execution actions obtained from the above calculations, the correction deviation between the original reactive power adjustment actions and the actual execution actions is calculated, and single-step real experience is generated. The correction deviation is:

[0109] in, Indicates time Safety action correction deviation; The original action output by the policy network; The actual actions performed are those corrected by the safety action correction layer.

[0110] Introducing window average safety action correction deviation:

[0111] in, Indicates time The window average safety action correction deviation serves as a feedback indicator for the policy network's continued reliance on the safety action correction layer. This is the length of the sliding window.

[0112] The aforementioned real-world experience in a single step is:

[0113] in, and For a moment and the microgrid's operating status at the next moment; The original reactive power regulation action output by the policy network; The actual executed action output by the safety action correction layer; This is the reward value; The cost of safety constraints; To correct deviations for instantaneous safety actions; Correct deviations for average safety actions of the window; This is the safety constraint activation identifier vector calculated based on the predicted node voltage, predicted line current, and activation judgment tolerance after the safety action is executed; This is the directional correction responsibility vector calculated based on the bidirectional safety correction amount and the corresponding correction weight; This provides a safety correction context for reproducing the safety action correction optimization problem in the current control cycle.

[0114] Furthermore, The safety constraint activation flag vector, calculated based on the predicted node voltage, predicted line current, and activation judgment tolerance after the safety action is executed, is specifically as follows: First, the relaxation amount of each constraint is calculated based on the predicted node voltage and predicted line current after safety correction, and then the activation status of each constraint is determined based on the preset activation judgment tolerance.

[0115] For any candidate reactive power regulation action The predicted node voltage is expressed as:

[0116] The predicted line current is expressed as follows:

[0117] in, and These are the node voltage vector and the line current vector at time t, respectively; and These are the voltage-reactive power sensitivity matrix and the line current-reactive power sensitivity matrix at time t, respectively. These are the candidate reactive power adjustment actions to be verified.

[0118] The safety constraint activation flag vector is defined as follows:

[0119] The activation flag for the node voltage lower limit constraint is:

[0120] The node voltage upper limit constraint activation flag is:

[0121] The activation flag for the line current upper limit constraint is:

[0122] in, and To perform actual safety actions The predicted node voltage and predicted line current are then calculated. and These are the activation judgment tolerances for node voltage constraints and line current constraints, respectively. This is an indicator function; it takes the value 1 when the condition within the parentheses is true, and 0 otherwise. Using activation-based tolerance instead of directly determining whether the constraint strictly adheres to the equality sign reduces frequent switching of activation states caused by numerical calculation errors and measurement fluctuations.

[0123] Furthermore, The direction correction responsibility vector, calculated based on the bidirectional safety correction amount and corresponding correction weight, is specifically calculated as follows: The contribution of each distributed power inverter to the objective function of this safety action correction in the reactive power increase and decrease directions is normalized to obtain the directional correction responsibility degree of each inverter. Specifically, for the j-th distributed power inverter, its correction responsibility in the reactive power increase direction is calculated as follows:

[0124] Its correction responsibility in the reactive power reduction direction is calculated as follows:

[0125] After calculating the reactive power directional correction responsibility degree of all distributed power inverters, they are arranged and concatenated in sequence to form a directional correction responsibility degree vector:

[0126] in, A positive number is set to prevent the denominator of the formula from being zero when all reactive power directional corrections are zero. When reactive power adjustment is expressed in per-unit values, its preferred range is 10. -8 ~10 -4 More preferably 10 -6 ; , They are respectively t Time of the first m The reactive power direction correction amount of the distributed power inverter; , They are respectively t Time of the first m The correction weights corresponding to the increase and decrease of reactive power direction correction in the distributed power inverter.

[0127] When the sum of all increases and decreases in reactive power directional corrections is less than the preset zero correction tolerance, the current primary directional correction path is defined as an empty set, and the primary directional correction path change event is not triggered by a small change in the responsibility value under the zero correction state.

[0128] Furthermore, the security correction context is used to ensure that the event experience can be re-verified in subsequent playback processes according to the same physical operating boundaries and security correction parameters when the event occurred; The data required to solve the safety action correction optimization problem in the current control cycle are used to form a safety correction context; the safety correction context used to reproduce the safety action correction optimization problem in the current control cycle is specifically:

[0129] in, This is the reactive power vector of each distributed power inverter before executing the current control cycle action; and These are the safety action correction weight vectors for each distributed power inverter in the reactive power increase and decrease directions, respectively. When the safety action correction layer also uses activation judgment tolerance, optimization solution tolerance, safety margin, or other operating boundary parameters, these parameters are also stored in the safety correction context. middle.

[0130] Step 4: Based on the single-step real experience, identify the continuous correction event, divide it into regular samples and event samples, store the regular samples in the regular experience pool, and store the event samples in the event experience pool. The specific identification method for the continuous correction event is as follows: 1) Determine the continuous correction status based on the average safety action correction deviation of the window and the high and low correction deviation thresholds:

[0131] in, This indicates that the policy network is in a state of high sustained correction; This indicates that the policy network is in a low continuous correction state; when the window average safety action correction deviation is between the low threshold and the high threshold, the continuous correction state of the previous control cycle is maintained to avoid frequent state switching near the threshold.

[0132] 2) Based on the predicted node voltage and predicted line current after safety correction, solve for the relaxation amount of each safety constraint, and generate a safety constraint activation flag vector by combining the preset activation judgment tolerance; the safety constraints include the upper and lower limit safety constraints of node voltage and the upper limit constraint of line current. 3) Arrange the correction responsibility of all inverters in the reactive power increase and decrease directions into a directional correction responsibility vector:

[0133] The inverter numbers and their reactive power adjustment directions corresponding to the M elements with the largest values ​​from the directional correction responsibility vector are selected to form the time... Main directional correction path:

[0134] in, This is used to characterize which inverters and which reactive power regulation directions are primarily responsible for the current safety correction; M is a preset number of directional correction elements used to characterize the main safety correction path. M is predetermined based on the number of controllable distributed power inverters and the concentration of directional correction responsibility, and remains unchanged during online control. The directional correction responsibility is arranged in descending order, and each directional correction responsibility is accumulated sequentially. When the accumulated directional correction responsibility is not less than 80% of the sum of all directional correction responsibility, the corresponding minimum number of elements is determined as M. Its preferred value range is 1 to 5, and it satisfies: When the number of controllable distributed power inverters is small or historical operating data is lacking, the preferred option is: .

[0135] A continuous correction event is triggered when at least one of the following conditions is met:

[0136] in, To continuously correct the event trigger flag; Indicator function; symbol It represents the logical "OR".

[0137] The continuous correction events include: 1) The continuous calibration state switches from low continuous calibration state to high continuous calibration state, or from high continuous calibration state to low continuous calibration state; 2) The safety constraint activation flag vector changes relative to the previous control cycle; 3) The average safety action correction deviation of the window is greater than the lower limit of the preset correction deviation threshold, and the top M elements with the largest values ​​in the direction correction responsibility vector have changed relative to the previous control cycle, where M is the preset number of direction correction elements, and the following conditions are met: .

[0138] The top M elements with the largest values ​​in the direction correction responsibility vector have changed relative to the previous control cycle, including the corresponding inverter number and the change in reactive power regulation direction.

[0139] Furthermore, based on the continuous correction events, regular samples and event samples are divided. Regular samples are stored in the regular experience pool, and event samples are stored in the event experience pool; specifically: When the continuous correction event is triggered, the event before it occurs is captured. One control cycle until the event occurs. Continuous real-world experience across control cycles, where... To continuously correct the number of control cycles intercepted before the event is triggered, The number of control cycles captured after a continuous correction event is triggered; forming an empirical sequence of continuous correction events:

[0140] in, This is the empirical sequence of the nth continuous correction event. The sliding window length W is determined based on the average safety action correction deviation of the window, and is preferably selected as follows: To preserve the historical control process that caused the window average safety action correction deviation to switch states; The value is determined based on the state change process that needs to be observed after the continuous correction event occurs, and the preferred value range is 2 to 6 control cycles.

[0141] The event experience sequence is the sequence of events after the event occurs. Write to the event experience pool at the end of each control cycle. Real-world single-step experiences not included in the continuous correction event experience sequence are written into the regular experience pool. .

[0142] Step 5: Adaptively draw samples from the regular experience pool and the event experience pool to train the policy network; wherein, the policy network loss function is constructed based on the correction bias and the average residual correction bias obtained by replaying and correcting the event samples after the policy network is updated; The adaptive training of the policy network by drawing samples from the regular experience pool and the event experience pool specifically includes the following steps: 1) Re-calibrate and verify after playback, and calculate the residual correction bias of a single sequence: After updating the policy network using the event experience sequence for the rth time, the historical states from the event experience sequence are re-inputted into the updated policy network. For the rth event experience sequence... Given a historical state, the updated policy network output verifies the original action as follows:

[0143] in, In the first r The policy network after the second parameter update, input to the first parameter... n Within the empirical sequence of continuous correction events, the [number]th The original reactive power regulation action is re-verified after the historical microgrid operating status is determined. These are the updated policy network parameters after the r-th training iteration; It serves as the internal time-series index for the event time sequence, used to traverse all consecutive single-step historical states within the sequence.

[0144] Invoke the security correction context corresponding to this historical state. The original motion input will be re-verified to be the same as the safety motion correction layer in the online control stage, and the actual executed motion will be re-verified. :

[0145] in, Indicates in the context of security correction Next, the safety action correction solution is completed according to the optimization model of node voltage and current constraints and safety action correction amount in step 3.

[0146] The residual correction bias of the experience sequence of the nth event after the rth policy update is:

[0147] in, The length of the event experience sequence; This is used to characterize the extent to which the current policy network still needs to rely on the security action correction layer when facing the same historical security boundary event again.

[0148] When the residual correction deviation is large, it indicates that the current policy network has not yet fully learned the safe and executable actions in this event; when the residual correction deviation is small, it indicates that the current policy network has a high degree of learning of the safe actions in this event.

[0149] By traversing all sequences in the event experience pool and calculating the mean, the average residual correction bias of the event experience pool at time t after the most recent recalibration verification is obtained as follows:

[0150] in, This represents the number of event experience sequences in the current event experience pool. This represents the number of times the experience sequence of the nth event was most recently involved in policy training.

[0151] 2) Construct dynamic sampling priority for events based on timing errors and residual correction deviations; Dynamic sampling priority of the nth event experience sequence for:

[0152] in, Let be the average temporal difference error of the empirical sequence of the nth event after the rth training iteration; For time-series difference error weights; Weights for residual correction bias; To prevent positive numbers with a priority of zero; This is the priority adjustment index. The residual correction bias in this formula is not the fixed correction bias when the event first occurs, but the current residual value obtained after each policy update by passing through the safety action correction layer.

[0153] When the residual correction deviation of the event experience sequence is below the low threshold of correction deviation for K consecutive times. When the event experience sequence is transferred from the event experience pool to the regular experience pool or its retention priority is reduced; when the residual correction bias is still higher than the high threshold of correction bias. When this happens, the event experience sequence is retained in the event experience pool and its subsequent sampling probability is increased.

[0154] 3) Perform dual-pool adaptive combined sampling to dynamically adjust the proportion of event samples within a batch; Based on the average safety action correction deviation of the online window and the average residual correction deviation of the event pool, the event experience extraction ratio in a single training batch is adaptively determined. :

[0155] in, The preferred value range for the basic event experience ratio is 0.1 to 0.3. The online window average safety action correction deviation adjustment coefficient is preferably set within the range of 0.1 to 0.5. The average residual correction deviation adjustment coefficient for the event pool is preferably in the range of 0.1 to 0.5. and These are the lower and upper limits of the event experience ratio, respectively, with preferred value ranges of 0.05–0.2 and 0.5–0.8, respectively, and satisfying the following: In a preferred embodiment, the basic event experience ratio is set to 0.2, the lower limit of the event experience ratio is set to 0.1, the upper limit of the event experience ratio is set to 0.6, and the adjustment coefficients for the average safety action correction deviation of the window and the average residual correction deviation of the event experience pool are both set to 0.2.

[0156] Let the total number of samples in each training batch be... Training batches are constructed by drawing samples from the event experience pool and the regular experience pool according to an adaptive ratio:

[0157]

[0158] in, For event experience pool; This is a standard experience pool; This indicates drawing from the event experience pool. Sample; This indicates that it is drawn from the regular experience pool. Sample; This indicates rounding up to the nearest integer.

[0159] The event experience pool is configured according to the aforementioned dynamic sampling priority. Sampling is performed, and the regular experience pool uses random sampling or regular time-series difference error priority sampling.

[0160] When the average safety action correction deviation of the online window or the average residual correction deviation of the event pool increases, the proportion of event experience in the training batch is increased; when both decrease, the proportion of event experience is reduced to maintain training coverage under normal operating conditions.

[0161] Different event experiences are not numerically weighted and integrated based on state, action, reward, or next state.

[0162] Furthermore, after completing adaptive dual-pool sample extraction, a deterministic policy gradient update mechanism for primal-dual safe reinforcement learning is constructed based on the extracted training batches. The policy network loss function integrates two types of indicators: single-step correction bias and event pool average residual correction bias obtained from replay and recorrection, to achieve collaborative training of safe and feasible actions and voltage regulation optimization objectives; specifically including: 1) Construct a deterministic policy gradient update mechanism; The policy gradient update formula is:

[0163] in, The Lagrangian function described in equation (8) is used to simultaneously characterize the reward maximization objective and the security constraint penalty term. For policy network parameters, As dual variables; Representing the Lagrange function Regarding policy network parameters The gradient is used to guide the update of policy network parameters; This represents the system state as observed by the agent; This represents the experience replay pool. Representing state From the experience replay pool Obtained by sampling from the middle; This represents the expectation of the gradient term corresponding to the sampled state in the experience replay pool; It is an action value function that evaluates the network output, used to evaluate the value of actions in a given state. Next action The expected return that can be obtained; Action value function Regarding the action The gradient; This indicates that the policy network output is related to the policy network parameters. The gradient.

[0164] During the evaluation phase of network training, the action terms in the empirical samples are replaced by the actual actions performed. During the policy network update phase, the original action output by the policy network... After being mapped and corrected for safety actions, the samples participate in environmental interactions to ensure consistency between training samples and online execution actions.

[0165] 2) Construct adaptive motion consistency training intensity; Define the action consistency training strength as jointly adjusted by the window-average safe action correction bias and the event experience pool-average residual correction bias:

[0166] in, For a moment The intensity of the motion consistency training used; Intensity of basic movement consistency training; and These are the lower and upper limits of the training intensity for motion consistency, respectively, and they satisfy: .

[0167] Correcting deviations in the average safe action of windows generated during online operation; The average residual correction bias of the event experience pool is obtained after the event experience has been re-verified by the current policy network and the security action correction layer. Adjustment coefficient for average safe action correction deviation in online windows; Let be the residual correction bias adjustment coefficient for the event experience, and satisfy: , .

[0168] This indicates that the calculation result is restricted to an interval. Internally, when the event experience pool has not yet been established or there is no valid event experience in the event experience pool, let: .

[0169] 3) Construct the policy network loss function; The loss function for constructing the policy network is:

[0170] in, The total number of samples in a single training run; n∈{1,…,B} is the sample index in the current training mini-batch; j∈{1,…,N DG} represents the index for controllable distributed power inverters; This represents the microgrid operating state corresponding to the nth empirical sample. The j-th dimension of the original reactive power regulation action output by the policy network for the nth empirical sample; The j-th dimension of the actual executed action is obtained after the original action passes through the safety action correction layer; The comprehensive correction responsibility undertaken by the j-th distributed power inverter in the n-th empirical sample; For parameters Evaluation network; Intensity for training to improve movement consistency.

[0171] For the j-th distributed power inverter, its comprehensive correction responsibility can be expressed as:

[0172] in, and These represent the correction responsibility of the j-th distributed power inverter in the reactive power increase and reactive power decrease directions, respectively.

[0173] When the average window safety action correction deviation is high, the training intensity for action consistency increases, causing the policy network to focus more on learning the actual execution actions output by the safety action correction layer. Conversely, when the average window safety action correction deviation is low, the training intensity for action consistency decreases, causing the policy network to optimize node voltage deviation and network losses more. Inverter action dimensions with larger correction ratios have higher weights in the action consistency loss.

[0174] 4) Evaluate the target network's soft updates; To stabilize the value function training process, a soft update method is used to periodically synchronize the target evaluation network parameters; the update formula is:

[0175] in, This is the soft update coefficient. This represents the model parameters of the target evaluation network. These are the model parameters for the current online evaluation network.

[0176] 5) Based on the correction bias and the cost of exceeding the limit, the dual variable is updated adaptively and iteratively; The penalty intensity of safety constraints during policy training is dynamically adjusted using the dual variable. To avoid frequent increases and decreases in the dual variable due to fluctuations in the window-average safety action correction bias around a preset threshold, a low threshold for the correction bias is set. and correction deviation high threshold ,in .

[0177] Define the hysteresis feedback function for the window average safety action correction deviation as follows:

[0178] The dual variable is updated as follows:

[0179] in, Let be the input variable of the hysteresis feedback function; during the update of the dual variable, let , Correct the deviation for the average safe action of the window; For a moment The dual variable; The upper limit of the dual variable; This indicates that the calculation results are projected onto the interval. ; This is the estimated safety constraint cost corresponding to the current control cycle or the current training batch. To preset a safety constraint cost threshold; Safety constraint cost feedback step size The feedback step size for the average safety action correction deviation of the window.

[0180] When the average safety action correction deviation of the window is higher than the high correction deviation threshold At this time, increasing the dual variable makes the policy network pay more attention to safety constraints in subsequent training; when the window average safety action correction deviation is lower than the low threshold of correction deviation. When the dual variable is appropriately reduced, the policy network optimizes voltage deviation and network loss more when the original action has less dependence on the safety correction layer; when the window average safety action correction deviation is between the low threshold and the high threshold, the dual variable is not changed according to the correction deviation, thereby reducing the frequent oscillation of the dual variable near the threshold.

[0181] Step 6: Input the real-time operating status data of the microgrid into the trained strategy network, output the original reactive power regulation action, send it to the safety action correction layer to obtain the actual execution action and send it to each distributed power inverter to perform voltage regulation; and store the single-step real experience generated during the voltage regulation process into the regular experience pool and the event experience pool respectively, and periodically update the network parameters of the strategy network and the evaluation network.

[0182] The trained policy network, safety action correction layer, continuous correction event recognition module, and dual-pool experience playback module are deployed in the microgrid central control device to perform original action generation, safety action correction, correction deviation update, event experience storage, and policy incremental training in each control cycle.

[0183] Specifically, the following steps are included: At any moment Collect data on microgrid node voltages, line currents, active power of distributed generation sources, load power, and current reactive power of inverters to construct the microgrid's operating status. .

[0184] The microgrid operating status is input into the trained policy network to obtain the original reactive power regulation actions:

[0185] in, These are the parameters of the currently trained policy network. During the online control phase, random exploration noise is not added to the original reactive power regulation actions.

[0186] Based on the current reactive power, original reactive power regulation actions, historical cumulative directional correction load, and the remaining reactive power regulation margin in the corresponding direction of each inverter, update the reactive power increase directional correction weight and reactive power decrease directional correction weight of each inverter.

[0187] The original reactive power adjustment action is input into the safety action correction layer to obtain the actual action to be executed:

[0188] in, This serves as the safety correction context for the current control cycle; the actual actions performed satisfy the aforementioned node voltage constraints, line current constraints, and inverter no-function constraints under the current sensitivity model.

[0189] Calculate the instantaneous safety action correction deviation between the original reactive power regulation action and the actual executed action. The instantaneous safety action correction deviation is written into a length of [length missing]. The sliding window updates the window's average safety action correction deviation. .

[0190] The dual variable is updated based on the window average safe action correction deviation and the hysteresis feedback function, and the action consistency training intensity is also updated.

[0191] The actual actions are sent to the corresponding distributed power inverters, and the node voltage, line current, and inverter reactive power are collected after the actions are executed to form the operating status at the next moment. And calculate the reward value and the cost of security constraints.

[0192] The historical cumulative direction correction burden of each inverter is updated based on the direction correction amount of the current control cycle. The activation safety constraint is determined based on the constraint multiplier of the safety action correction optimization problem, and the correction responsibility of each inverter in different reactive power regulation directions is calculated.

[0193] Construct real single-step experience and determine whether a continuous correction event is triggered; when a continuous correction event is triggered, cache real experience of the continuous control cycle before and after the event occurs, and form an event experience sequence after the event ends.

[0194] According to the preset training period, the event experience ratio is determined, and training batches are constructed from the event experience pool and the regular experience pool. The policy network and evaluation network are updated using the training batches.

[0195] After the policy network is updated, the replay and recalibration verification is performed on the event experience that participated in the training, and the residual calibration deviation of each event experience sequence is calculated; the dynamic sampling priority of the event experience and the event pool retention status are updated according to the residual calibration deviation.

[0196] The action consistency training intensity and event experience sampling ratio for the next training cycle are updated based on the average residual correction bias of the event pool. The updated policy network then executes the original action generation for the next control cycle.

[0197] Therefore, the safety action correction results of the current control cycle determine the generation and training weight of event experience. The event experience training results are fed back to the safety action correction evaluation and subsequent strategy training through playback and correction verification, so that safety action correction, strategy network training and experience playback form a closed loop.

[0198] Example 1: To more clearly illustrate the outstanding substantive features of this invention and the significant progress it brings to the prior art, an application example of implementing this invention is described below.

[0199] Specifically, in the IEEE 33-bus test system, a microgrid system and parameters were set up, and comparative tests demonstrated the feasibility and effectiveness of the enhanced experience fusion method. The algorithm performance test results are shown in the table below: Table 1 Performance of different algorithms

[0200] Analyze nodes with relatively large voltage fluctuations, specifically the end-of-feed node, by selecting it for daily voltage variation analysis. Figure 2 As shown, the TRPO algorithm and SAC algorithm experience voltage overruns during certain periods, the original-dual PPO algorithm exhibits large voltage fluctuations, while the method of this invention maintains the voltage within the safe range of [0.95, 1.05] pu with minimal fluctuations and optimal stability.

[0201] The specific experimental setup is as follows: In the IEEE 33-bus test system, based on the reactive power balance index, the system is divided into four regional agents, with photovoltaic (PV) generators connected to each region. The system's base voltage is 12.66 kV, and the per-unit safe range for node voltages is [0.95, 1.05] pu. To construct the microgrid simulation environment, Python is used for modeling. Power flow calculations are performed based on the control actions of each agent, and rewards and costs are evaluated in real time based on the power flow calculation results. The optimization cycle is 24 hours, and the decision interval for the agents is 5 minutes. The system can meet the requirements of real-time voltage control. During actual operation, the agents can obtain the current grid status information in real time and make decisions only based on their own status, thereby realizing real-time scheduling of distributed power inverters. During the training phase, the PV penetration rate is set to 60%, meaning that the maximum PV output is 60% of the maximum load power. The training data is based on one year of load and PV data from a power grid in a certain region of China. Thirty typical dates were selected for data processing, and some noise was added during the process. The dataset is constructed as follows: 20 days of data are used as the training set, and the remaining 10 days of data are used as the test set.

[0202] Design the following 5 comparison algorithms: 1) TRPO algorithm: Based on policy optimization, it constrains the range of each policy update; 2) SAC algorithm: Explores the policy space using a soft maximization method; 3) Primitive-Dual DDPG Algorithm: Uses a primal-dual method based on CMDP; 4) Primitive-Dual PPO Algorithm: Uses a primal-dual method based on CMDP; 5) The method of this invention: The original-dual DDPG algorithm adopts a safety action correction layer and an operational risk feature fusion mechanism.

[0203] Furthermore, five typical daily load and photovoltaic data sets were selected as test datasets to test the trained model. Test metrics included: average grid loss, average voltage deviation, and voltage compliance rate. Average grid loss is the average power loss of all branches across all time periods within a day. Average voltage deviation is the average deviation of the voltage at each node from the reference voltage across all time periods within a day. Voltage compliance rate is the proportion of time periods within a day where the voltage is within the safe constraint range. The algorithm results are shown in Table 1.

[0204] Depend on Figure 2 It is known that voltage exceedance occurs under the TRPO algorithm and SAC algorithm models, while the voltage remains within a safe range and fluctuates less under the method model of this application.

[0205] The method of this invention can adapt well to changes in the microgrid environment and can still generate a better voltage control strategy in real time when the photovoltaic penetration rate changes. It has strong robustness and strong generalization ability.

[0206] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0207] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0208] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0209] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A microgrid voltage active control method based on fusion of empirical security reinforcement learning, characterized in that, The method includes the following steps: Step 1: Construct a constrained Markov decision process model for the microgrid, defining the continuous action space consisting of the reactive power regulation of each distributed power inverter, and the state space consisting of the real-time operating status data of the microgrid. Step 2: Based on the microgrid constrained Markov decision process model, construct an Actor-Critic continuous control network including a policy network and an evaluation network, and establish a primitive-dual security reinforcement learning optimization framework. Step 3: Connect a safety action correction layer in series at the back end of the strategy network to perform safety correction on the original reactive power regulation action output by the strategy network to obtain the actual executed action; calculate the correction deviation between the original reactive power regulation action and the actual executed action, and generate single-step real experience; the original reactive power regulation action is the original reactive power regulation amount of the distributed power inverter. Step 4: Based on the single-step real experience, identify the continuous correction event, divide it into regular samples and event samples, store the regular samples in the regular experience pool, and store the event samples in the event experience pool. Step 5: Adaptively draw samples from the regular experience pool and the event experience pool to train the policy network; wherein, the policy network loss function is constructed based on the correction bias and the average residual correction bias obtained by replaying and correcting the event samples after the policy network is updated; Step 6: Input the real-time operating status data of the microgrid into the trained strategy network, output the original reactive power regulation action, send it to the safety action correction layer to obtain the actual execution action and send it to each distributed power inverter to perform voltage regulation; and store the single-step real experience generated during the voltage regulation process into the regular experience pool and the event experience pool respectively, and periodically update the network parameters of the strategy network and the evaluation network.

2. The microgrid voltage active control method according to claim 1, characterized in that: In step 3, in the safety action correction layer, an optimization model for the safety action correction amount is constructed and solved. The solved safety action correction amount is superimposed with the original reactive power adjustment amount as the actual action to be executed. The safety action correction amount includes reactive power increase direction correction amount and reactive power decrease direction correction amount. The objective function of the optimization model is: in, , They are respectively t The set vector of reactive power directional corrections for all distributed power inverters at any given moment; , They are respectively t The reactive power direction correction of the j-th distributed power inverter at time j, and satisfying the following conditions: , ; , They are respectively t Time of the first j The correction weights corresponding to the increase and decrease of reactive power direction correction in the distributed power inverter. This represents the total number of distributed power inverters.

3. The microgrid voltage active control method according to claim 2, characterized in that: The correction weights corresponding to the reactive power increase and reactive power decrease corrections of the distributed power inverter are determined as follows: in, and They are respectively t Time of the first j The normalized reactive power direction residual regulation margin and reactive power direction residual regulation margin of the distributed power inverter. and They are respectively t Time of the first The historical cumulative correction burden of the inverter in the reactive power increase and decrease directions after normalization; For cumulative correction burden factor; To prevent positive numbers with a denominator of zero; , These are the lower and upper limits of the correction weights, respectively. This means that the calculation results will be limited to the preset upper and lower limits.

4. The microgrid voltage active control method according to claim 3, characterized in that: The historical cumulative correction load of the inverter in the reactive power increase and reactive power decrease directions is determined as follows: in, , for t +1 、t Time of the first Taiwan inverters increase the historical cumulative correction burden of reactive power direction; , for t +1 、t Time of the first Taiwan inverters reduce the historical cumulative correction burden of reactive power direction; For the burden of forgetting coefficient, and .

5. The microgrid voltage active control method according to claim 2, characterized in that: In step 4, the specific method for identifying the continuous correction event is as follows: The average value of the correction deviation between the original reactive power adjustment action and the actual executed action within the sliding window is calculated as the window average safe action correction deviation. When the window average safe action correction deviation is greater than the upper limit of the preset correction deviation threshold, it indicates that the strategy network is in a high continuous correction state; when the window average safe action correction deviation is less than the lower limit of the preset correction deviation threshold, it indicates that the strategy network is in a low continuous correction state. Based on the predicted node voltage and predicted line current after safety correction, the relaxation amount of each safety constraint is solved, and a safety constraint activation identifier vector is generated by combining the preset activation judgment tolerance; the safety constraints include the upper and lower limit safety constraints of node voltage and the upper limit constraint of line current. Normalize the increase and decrease of reactive power directional correction for each distributed power inverter to obtain the directional correction responsibility vector; A continuous correction event is triggered when at least one of the following conditions is met: 1) The continuous calibration state switches from low continuous calibration state to high continuous calibration state, or from high continuous calibration state to low continuous calibration state; 2) The safety constraint activation flag vector changes relative to the previous control cycle; 3) The average safety action correction deviation of the window is greater than the lower limit of the preset correction deviation threshold, and the top M elements with the largest values ​​in the direction correction responsibility vector have changed relative to the previous control cycle, where M is the preset number of direction correction elements, and the following conditions are met: .

6. The microgrid voltage active control method according to claim 5, characterized in that: The safety constraint activation identifier vector is composed of the node voltage lower limit constraint activation identifier vector, the node voltage upper limit constraint activation identifier vector, and the line current upper limit constraint activation identifier vector; the node voltage lower limit constraint activation identifier vector Node voltage upper limit constraint activation identifier vector Line current upper limit constraint activation flag vector ,in, for t Time Node i Predicted node voltage, for t Timetable l Predicted line current, , They are nodes i Under the voltage, upper limit, For the line l The upper limit of current, , These are the preset activation judgment tolerances for node voltage constraints and line current constraints, respectively. This is an indicator function; it takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

7. The microgrid voltage active control method according to claim 5, characterized in that: The normalized reactive power directional correction values ​​for each distributed power inverter are adjusted to obtain a directional correction responsibility vector, which specifically includes: For any distributed generation inverter, its correction responsibility in the reactive power increase direction is calculated as follows: Its correction responsibility in the reactive power reduction direction is calculated as follows: After calculating the reactive power directional correction responsibility degree of all distributed power inverters, they are sequentially arranged and concatenated into a directional correction responsibility degree vector. ;in, , They are respectively t Time of the first j The responsibility for reactive power direction correction of distributed power inverters in Taiwan. A positive number set to prevent the denominator from being zero; , They are respectively t Time of the first m The reactive power direction correction amount of the distributed power inverter; , They are respectively t Time of the first m The correction weights corresponding to the increase and decrease of reactive power direction correction in the distributed power inverter.

8. The microgrid voltage active control method according to claim 5, characterized in that: When a continuous correction event is triggered, the data before the continuous correction event occurs is captured. One control cycle until the event occurs. Continuous real-world experience from each control cycle forms an event experience sequence, which is then stored as an event sample in an event experience pool; wherein... To continuously correct the number of control cycles intercepted before the event is triggered, To continuously correct the number of control cycles captured after the event is triggered.

9. The microgrid voltage active control method according to claim 8, characterized in that: In step 5, the average residual correction bias obtained by replaying and recalibrating the event samples after the policy network update is calculated as follows: First, calculate the residual correction bias corresponding to a single event experience sequence. The calculation formula is as follows: For all event experience sequences in the event experience pool, the mean of the residual correction bias after the most recent training update for each sequence is calculated to obtain the average residual correction bias. The calculation formula is as follows: in, For the first n The event experience sequence in the th ... r Residual correction bias after the next strategy update; In the first r The policy network after the second parameter update, input to the first parameter... n Within the empirical sequence of continuous correction events, the [number]th [event]... The original reactive power regulation action is re-verified after the historical microgrid operating status is determined. To re-verify the actual execution action obtained by re-executing the safety correction mapping on the original reactive power regulation action; The length of the event experience sequence; For the first n A continuous sequence of corrective events; for t The average residual correction bias is obtained by replaying and recalibrating the event samples at each time point after the policy network update. This represents the total number of continuously correcting event experience sequences in the current event experience pool; For the first n A sequence of events in its most recent training iteration The corresponding residual correction deviation.

10. The microgrid voltage active control method according to claim 9, characterized in that: In step 5, the adaptive sampling of samples from the regular experience pool and the event experience pool to train the policy network specifically includes: Based on the average safety action correction deviation of the window and the average residual correction deviation of the event samples obtained by replaying and correcting after the policy network update, the event experience extraction ratio in the training batch is calculated; samples are extracted from the regular experience pool and the event experience pool for training according to the event experience ratio. The event experience extraction ratio is as follows: in, The proportion of basic event experience; Adjustment coefficient for average safety action correction deviation of the window; This is the adjustment coefficient for the average residual correction deviation in the event pool; for t Average safety action correction deviation within the time window; and These are the lower and upper limits of the event experience ratio, respectively. Indicates the variable Limited to the range within; when At that time, take ;when At that time, take In other cases, take .

11. The microgrid voltage active control method according to claim 10, characterized in that: In step 5, the policy network loss function is specifically as follows: in, is the total number of samples in a single training session; n is the sample index in the current training mini-batch; j is the index of the controllable distributed power inverter. This represents the microgrid operating state corresponding to the nth empirical sample. , is the original reactive power regulation action of the j-th distributed power inverter output by the strategy network for the n-th empirical sample; The actual execution action of the j-th distributed power inverter is obtained after the original reactive power regulation action has been safety-corrected. The comprehensive correction responsibility of the j-th distributed power inverter in the n-th empirical sample is the sum of the reactive power direction correction responsibilities of the distributed power inverter. For parameters Evaluation network; To improve the intensity of movement consistency training; The intensity of the motion consistency training Specifically: in, Intensity of basic movement consistency training; and These are the lower and upper limits of the training intensity for motion consistency, respectively, and satisfy the following conditions: ; The adjustment coefficient for the average safe action correction deviation of the online window. Let be the average residual correction deviation adjustment coefficient in the event pool, and satisfy . , .

12. A microgrid voltage active control system based on fusion empirical safety reinforcement learning according to any one of claims 1-11, comprising a microgrid constrained Markov decision process model construction module, an Actor-Critic network construction module, a single-step real experience generation module, an event recognition and dual-experience pool classification and storage module, a dual-pool adaptive sampling and policy network training module, and an online control and network periodic update module, characterized in that: The microgrid constrained Markov decision process model construction module constructs a microgrid constrained Markov decision process model, defines the continuous action space consisting of the reactive power regulation of each distributed power inverter, and the state space consisting of the real-time operating status data of the microgrid. The Actor-Critic network construction module, based on the microgrid constrained Markov decision process model, constructs an Actor-Critic continuous control network including a policy network and an evaluation network, and establishes a primitive-dual security reinforcement learning optimization framework. The single-step real experience generation module connects a safety action correction layer in series at the back end of the strategy network to perform safety correction on the original reactive power adjustment action output by the strategy network to obtain the actual execution action; calculates the correction deviation between the original reactive power adjustment action and the actual execution action, and generates single-step real experience; the original reactive power adjustment action is the original reactive power adjustment amount of the distributed power inverter. The event recognition and dual experience pool classification and storage module continuously corrects events based on the single-step real experience recognition, divides regular samples and event samples, stores regular samples in the regular experience pool, and stores event samples in the event experience pool. The dual-pool adaptive sampling and policy network training module adaptively extracts samples from the regular experience pool and the event experience pool to train the policy network; wherein, the policy network loss function is constructed based on the correction bias and the average residual correction bias obtained by replaying and recorrecting the event samples after the policy network is updated; The online control and network periodic update module inputs real-time operating status data of the microgrid into the trained strategy network, outputs the original reactive power regulation action, sends it to the safety action correction layer to obtain the actual execution action, and sends it to each distributed power inverter to perform voltage regulation; and stores the single-step real experience generated during the voltage regulation process into the regular experience pool and the event experience pool respectively, and periodically updates the network parameters of the strategy network and the evaluation network.

13. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-11.

Images