Micro-grid voltage and power self-balancing cooperative control method and system

Abstract

The present application relates to the technical field of micro-grid autonomous control and micro-grid coordination control, and particularly relates to a micro-grid voltage and power self-balancing coordination control method and system, which is based on a double-delay deep deterministic policy gradient (TD3) and introduces a squeeze-and-excitation (SE) channel attention mechanism at a state perception end to adaptively calibrate weights of observation features including node voltage, renewable available output, load coefficient, energy storage SOC, charging station demand pool, PCC interactive power deviation and time coding; a master-slave cascaded decoupling strategy structure is constructed at an action decision end to decompose multi-dimensional continuous actions into master branch output and slave branch compensation output conditioned on the master branch output, and to generate active and reactive control instructions of photovoltaic, wind power, diesel generator, energy storage and charging station according to fixed mapping.

Description

Technical Field

[0001] This invention relates to the field of microgrid autonomous control and distribution microgrid collaborative control technology, and in particular to a microgrid voltage and power self-balancing collaborative control method and system. Background Technology

[0002] As a local power system containing distributed power sources, energy storage, and controllable loads, microgrids are affected by multiple constraints, including fluctuations in renewable energy output, random load changes, equipment capacity, and SOC boundaries. Existing model-based centralized optimization methods are prone to model mismatch and computational burden in scenarios with uncertain topology and parameters, strong disturbances, and high real-time requirements. Traditional rule-based control or hierarchical control struggles to balance and maintain stability in multi-objective coordination (voltage, network loss, absorption capacity, PCC interactive power, quality of service, etc.).

[0003] While end-to-end deep reinforcement learning (such as TD3) has online decision-making capabilities, it often suffers from slow convergence, reward oscillation, and policy oscillation in high-dimensional mixed states and strongly coupled multidimensional continuous action scenarios. This is partly due to the processing of a large number of redundant features. Therefore, there is an urgent need for an improved policy structure that can balance stability, convergence efficiency, and interpretability. Summary of the Invention

[0004] Technical Problem: In order to solve the technical problems existing in the prior art, the present invention provides a microgrid voltage and power self-balancing coordinated control method. In addition, the present application also provides a microgrid voltage and power self-balancing coordinated control system.

[0005] Technical Solution: According to the technical solution provided by this invention, on one hand, a method for coordinated control of voltage and power self-balancing in a microgrid is provided, the method comprising: The control input features of the microgrid at various times are obtained, and these features are categorized into main control objects and other objects. The main control objects are the features corresponding to resources or devices that affect the stability and power balance of the microgrid, while the other objects are the features corresponding to devices that work in conjunction with the main control objects or other loads or resources that are responsible for assisting in the regulation of the microgrid. The control input features are then normalized and time-encoded to obtain the state feature vector for the policy network input. The state feature vector is input to the state representation layer of the policy network. The state representation layer is used to represent the state feature vector and obtain an enhanced feature vector. Based on the enhanced feature vector and the distinguished objects, master-slave cascade decoupling is performed to obtain a first action sub-vector and a second action sub-vector. The master-slave cascade decoupling is used to generate the action decision of the master object, i.e., the first action sub-vector, through the main branch of the policy network, and also to generate the collaborative decision of the remaining objects, i.e., the second action sub-vector, through the slave branch of the policy network. The enhanced feature vector, the first action sub-vector, and the second action sub-vector are concatenated and input into the compensation branch of the policy network to generate a compensation action vector. The compensation action vector is the active power regulation compensation action of the energy storage battery, thus forming the final action vector. The final action vector is made feasible according to the constraints of the microgrid to obtain an executable action vector. The executable action vector is executed and calculated using numerical calculation methods to obtain node voltage, network loss and PCC interaction power. After determining the reward function, the interaction samples are stored in the experience replay pool. Under the TD3 framework, dual Critic network update, Actor delayed update and target network soft update are executed. The trained control policy parameters are stored for online inference calls.

[0006] Furthermore, including: The state representation layer is used to represent the state feature vector and obtain an enhanced feature vector, including: The state feature vector is input to the state representation layer of the policy network to obtain the state representation feature vector. , is represented as: ;in, and These are the weight matrix and bias vector of the state representation layer, respectively. It is a non-linear activation function. This is the current state feature vector.

[0007] Channel weight vectors are generated through the channel attention module, which represents the state feature vectors. We perform weighting to obtain the enhanced feature vector. .

[0008] Furthermore, including: The channel weight vector is generated through the channel attention module, which is used to represent the state feature vector. We perform weighting to obtain the enhanced feature vector. ,include: The state representation feature vector After performing compression mapping, the bottleneck vector is obtained, represented as: ; The channel weight vector is then generated by the activation mapping, and is expressed as: ; Enhanced features are obtained by channel recalibrating the state representation feature vector, as follows: .

[0009] in: ReLU represents the nonlinear activation function; This represents the Sigmoid activation function; , These are the weight matrices of the fully connected layer. , These are the bias vectors; For channel weight vectors, This represents element-wise multiplication; These are the state representation features enhanced by attention, used for subsequent policy output or value evaluation.

[0010] Furthermore, including: The process of combining the enhanced feature vectors with the differentiated objects to perform master-slave cascade decoupling, thereby obtaining the first action sub-vector and the second action sub-vector, includes: Represent the first action subvector and the second action subvector: ,in, This represents the hyperbolic tangent activation function, which restricts the action output to... , This represents the strategy mapping corresponding to the main branch, resulting in the decision used to control the main control object. This indicates that the strategy mapping corresponding to the branch is used to coordinate and adjust other devices or resources. and These are all nonlinear mappings on different branches within the Actor network.

[0011] Furthermore, including: The step of storing the interaction samples into the experience replay pool after determining the reward function includes: The reward function is a weighted combination of at least two of the following penalty items, and includes at least the following: voltage limit violation penalty, network loss penalty, PCC interactive power deviation penalty, renewable energy curtailment penalty, fuel cost penalty, charging demand unmet penalty, and SOC maintenance penalty: ; in, This refers to voltage exceeding limits or deviation indicators; For network loss indicators; PCC interactive power deviation index; For renewable energy curtailment quotas; For fuel cost indicators; The charging demand was not met. These are the maintenance indicators for energy storage SOC deviation from the target; the voltage over-limit or deviation indicator is obtained from the node voltage, the network loss indicator is obtained from the network loss, and the PCC interactive power deviation indicator is obtained from the PCC interactive power. These are the weighting coefficients for the following indicators: voltage over-limit or deviation, network loss, PCC interactive power deviation, new energy curtailment, fuel cost, unmet charging demand, and maintenance of energy storage SOC deviating from the target. All of these are non-negative real numbers.

[0012] On the other hand, the present invention also provides a microgrid voltage and power self-balancing coordinated control system, the system comprising: The state feature vector acquisition module is used to acquire the control input feature quantities of the microgrid at various times, and to distinguish the control input feature quantities into the main control object and other objects. The main control object is the feature quantity corresponding to the resource or equipment that affects the stability and power balance of the microgrid, and the other objects are the feature quantities corresponding to the equipment that works in cooperation with the main control object or other loads or resources that are responsible for assisting in the regulation of the microgrid. The control input feature quantities are normalized and time-encoded to obtain the state feature vector for input to the policy network. The sub-vector determination module is used to input the state feature vector into the state representation layer of the policy network. The state representation layer is used to perform state representation on the state feature vector and obtain an enhanced feature vector. Based on the enhanced feature vector and the distinguished objects, master-slave cascade decoupling is performed to obtain a first action sub-vector and a second action sub-vector. The master-slave cascade decoupling is used to generate the action decision of the master object, i.e., the first action sub-vector, through the main branch of the policy network, and also to generate the collaborative decision of the remaining objects, i.e., the second action sub-vector, through the slave branch of the policy network. The action vector generation module is used to concatenate the enhanced feature vector, the first action sub-vector, and the second action sub-vector and input them into the compensation branch of the policy network to generate a compensation action vector. The compensation action vector is the active power regulation compensation action of the energy storage battery, thereby forming the final action vector. The final action vector is made feasible according to the constraints of the microgrid to obtain an executable action vector. The control strategy training module is used to execute and calculate the executable action vectors according to numerical calculation methods to obtain node voltage, network loss and PCC interaction power. After determining the reward function, the interaction samples are stored in the experience replay pool. Under the TD3 framework, dual Critic network update, Actor delayed update and target network soft update are performed, and the trained control strategy parameters are stored for online inference calls.

[0013] According to the technical solution provided by the present invention, in a third aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the above-mentioned embodiments.

[0014] Beneficial effects: This invention partitions the control input features and uses the main branch and slave branch networks in the strategy network to process the features corresponding to the master control object and other objects respectively. In other words, it decomposes the multidimensional continuous control actions into more refined sub-tasks and decouples them through the master-slave relationship, so that each branch can independently optimize the part it is responsible for, thereby improving the efficiency and stability of the control system.

[0015] In reinforcement learning for microgrid control systems, this invention further enhances the effectiveness of input features through a state representation layer. This state representation layer, a crucial layer in the policy network, aims to enhance the input state observation vector. Utilizing mechanisms such as SE channel attention, it assigns appropriate weights to each input feature, highlighting those more critical for decision-making. The output of this layer is the enhanced state features, which are then fed into the policy network to generate the final control commands. This allows the network to better capture key features, effectively reducing voltage over-limit rates and PCC power deviations under renewable energy fluctuations and load disturbances, minimizing solar and wind curtailment, and improving operational stability. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the working principle of the microgrid collaborative control method and system based on dual-end augmented deep reinforcement learning as described in an embodiment of the present invention. Figure 2 The flowchart is a microgrid cooperative control method and system based on dual-end augmented deep reinforcement learning as described in the embodiments of the present invention; Figure 3 This is a comparison diagram of the spatiotemporal distribution of the voltages at the TD3 and DE-TD3 nodes as described in an embodiment of the present invention; Figure 4 This is a comparison chart of the PCC tie-line power interaction stability analysis of TD3 and DE-TD3 as described in the embodiments of the present invention; Figure 5This is a comparison chart of voltage, network loss, and light curtailment between TD3 and DE-TD3 as described in this embodiment of the invention; Figure 6 This is a comparison chart of the photovoltaic and wind power consumption of TD3 and DE-TD3 as described in the embodiments of the present invention; Figure 7 This is a schematic diagram of the structure of the policy network state representation layer according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0019] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0020] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0021] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0022] Please see Figure 1 and Figure 2 This embodiment provides a microgrid voltage and power self-balancing collaborative control method. By introducing an SE channel attention mechanism at the state sensing end and adopting a master-slave cascaded decoupling structure at the action decision end, it improves the stability and convergence efficiency of strategy training, reduces voltage over-limit and power deviation, reduces solar and wind curtailment, and enhances robustness under complex disturbance scenarios. The method includes the following steps: Step S1: Obtain the control input feature quantities of the microgrid at various times, and classify the control input feature quantities into the main control object and other objects. The main control object is the feature quantity corresponding to the resource or equipment that affects the stability and power balance of the microgrid, and the other objects are the feature quantities corresponding to the equipment that works in cooperation with the main control object or other loads or resources that are responsible for assisting in the regulation of the microgrid; normalize and time-encode the control input feature quantities to obtain the state feature vector for the policy network input.

[0023] In this embodiment, firstly, the microgrid at time [time] is obtained. t Control input feature quantity s t The control input characteristics include at least: node voltage information, renewable energy available output information, load factor information, energy storage status information, charging station demand status information, PCC interaction power deviation information, and time coding information. PCC interaction power refers to the active and / or reactive power exchanged between the microgrid (or distributed energy system) and the upstream distribution network through the Point of Common Coupling (PCC). It is a key control variable for achieving coordinated operation, energy interaction, and power regulation between the microgrid and the distribution network, as well as for collecting or calculating microgrid node voltages. PCC Interaction Power Available output of renewable energy Load factor Energy storage status and the demand status of charging stations .

[0024] Secondly, the collected / computed data is assembled into control input features. The control input features are then normalized and time-encoded to obtain a state feature vector for the policy network input; wherein the time encoding includes at least... At least one of them, where T is the period length.

[0025] Step S2: Input the state feature vector into the state representation layer of the policy network. The state representation layer is used to represent the state feature vector and obtain an enhanced feature vector. Based on the enhanced feature vector and the distinguished objects, master-slave cascade decoupling is performed to obtain a first action sub-vector and a second action sub-vector. The master-slave cascade decoupling is used to generate the action decision of the master object, i.e., the first action sub-vector, through the main branch of the policy network, and also to generate the collaborative decision of the remaining objects, i.e., the second action sub-vector, through the slave branch of the policy network.

[0026] In a preferred embodiment of this invention, the policy network includes not only an Actor network but also modules for enhancing features and decoupling action decisions. Both the feature enhancement module and the action decision decoupling module are implemented within the Actor network. This means that this embodiment technically improves the Actor network to address the complexities of microgrid data. Traditional Actor networks in TD3 are primarily used to receive states and generate corresponding actions, employing a twin-delayed deep deterministic policy gradient algorithm. TD3 is an algorithm in reinforcement learning specifically designed to solve the problem of continuous action spaces.

[0027] In this embodiment, the state feature vector is input into the state representation layer of the policy network, and a channel weight vector is generated. Finally, the state representation features are weighted to obtain the enhanced features. In the microgrid control system of this embodiment, the policy network performs the following tasks: State representation: Enhancing the state features of the input enables the network to better capture key features. In this embodiment, the input state features can be enhanced using the SE channel attention mechanism. The SE attention module is a channel attention module that is plug-and-play. It can enhance the channel features of the input feature map, and the final output of the SE module does not change the size of the input feature map.

[0028] Multi-branch output: The master-slave cascaded decoupling strategy structure decomposes actions into multiple sub-vectors for the coordinated control of different resources.

[0029] Vector output: The final output is based on the state feature vector. Output multidimensional continuous action vector The action vector is used to represent active / reactive power control commands or output coefficients for photovoltaic, wind power, energy storage, diesel generators and charging stations.

[0030] Specifically, in one implementation of this embodiment, the state feature vector is... The state representation layer of the input policy network is processed through at least one fully connected layer and nonlinear activation to obtain the state representation feature vector. Then, the SE channel attention module generates channel weight vectors, which are used to represent the state feature vectors. Weighting is performed to obtain enhanced features. ,therefore, ; in, and These are the weight matrix and bias vector of the state representation layer, respectively. It is a non-linear activation function.

[0031] The SE channel attention module includes the following calculations: State representation feature vector Input compression mapping yields the bottleneck vector: ; Then, the channel weight vector is generated by the activation mapping: ; The enhanced features are obtained by channel recalibrating the state representation feature vector. ; in: ReLU represents the nonlinear activation function; This represents the Sigmoid activation function; , These are the weight matrices of the fully connected layer. , These are the bias vectors; For channel weight vectors, This represents element-wise multiplication; This is the state representation feature vector after attention enhancement, used for subsequent policy output or value evaluation.

[0032] Step S3: The enhanced feature vector, the first action sub-vector, and the second action sub-vector are concatenated and input into the compensation branch of the policy network to generate a compensation action vector. The compensation action vector is the active power regulation compensation action of the energy storage battery, thus forming the final action vector. The final action vector is made feasible according to the constraints of the microgrid to obtain an executable action vector.

[0033] In a preferred embodiment of this invention, such as Figure 7 As shown, the enhanced feature vector The main branch outputs the first action vector. With the second action vector In this embodiment, the first action sub-vector represents the initial decision of a group of master control objects, and the second action sub-vector represents the collaborative decision of the remaining objects. In this embodiment, the master control objects refer to resources or equipment that play a key role in the stability and power balance of the microgrid, including photovoltaic, wind power, and energy storage. The remaining objects are equipment that works in collaboration with the master control objects and are responsible for assisting in the regulation of other loads or resources in the microgrid, such as diesel generators and non-critical loads. This embodiment classifies the enhanced feature vectors according to the master control objects and the remaining objects, and processes them separately in the actor network. This avoids problems such as slow convergence, reward oscillation, and policy oscillation that occur in high-dimensional mixed states and strongly coupled multi-dimensional continuous action scenarios. Furthermore, the master-slave cascade decoupling strategy in this embodiment is based on a master-branch and slave-branch network. Specifically, it decomposes multi-dimensional continuous control actions into more refined sub-tasks and decouples them through master-slave relationships, allowing each branch to independently optimize its responsible part, thereby improving the efficiency and stability of the control system. Finally, master-slave cascade decoupling reduces action coupling, reduces the mutual cancellation of multiple resources, and improves the convergence speed.

[0034] In a preferred embodiment of this invention, the enhanced features are... The input to the compensation branch is concatenated with the first action sub-vector and the second action sub-vector to generate the compensation action for key adjustment resources. ;According to the preset action mapping order , and Assemble into a complete action vector and to The system performs physical constraint processing, including considerations such as apparent power margin, energy storage SOC boundary, and charging acceptance capability, to obtain executable actions. In other words, in this embodiment, the constraint processing of the action vector is based on the microgrid's state observation vector. Specifically, the system first generates a state observation vector based on collected or calculated microgrid-related data (such as node voltage, PCC interaction power, renewable energy output, load factor, energy storage status, and charging station demand). Then, through the policy network, it uses this observation vector to generate action vectors.

[0035] In a preferred embodiment, the policy network is based on enhanced features. Output the first action subvector and the second action subvector respectively, as follows: ; And enhance features After concatenation with the first and second action sub-vectors, the input is fed into the branch network, which outputs compensation actions for key adjustment resources: ; The final action vectors are assembled in a preset order as follows: ; in: These are the neural network mappings consisting of fully connected layers and non-linear activation layers, respectively; that is... This part is the network corresponding to the main branch, responsible for generating the first action sub-vector, which represents the initial decision of the master object; This part corresponds to the branch network, which is used to generate the second action sub-vector, namely the cooperative control decision part, which mainly regulates other devices and resources in the microgrid; This part is mainly used to generate compensation actions related to key regulating resources (such as energy storage systems) to adjust the resource scheduling of the entire system. The key regulating resources are preferably the active power regulation actions of energy storage batteries, and the first action sub-vector and the second action sub-vector preferably correspond to the collaborative decision-making of renewable resources and other resources, respectively.

[0036] Specifically, in a preferred embodiment of this invention, The policy mapping corresponding to the main branch takes the enhanced features as input and outputs the first action sub-vector of the main control object. Corresponding to the policy mapping from the branch, the input is also the enhanced feature, and the output is the second action vector of the other objects; both are essentially non-linear mapping modules on different branches within the Actor network, rather than abstract mathematical functions. This indicates vector concatenation; This represents the hyperbolic tangent activation function, which restricts the action output to... .

[0037] Step S4: Execute and calculate the executable action vector according to the numerical calculation method to obtain node voltage, network loss and PCC interaction power. After determining the reward function, store the interaction sample in the experience replay pool, and perform dual Critic network update, Actor delayed update and target network soft update under the TD3 framework. Store the trained control policy parameters for online inference.

[0038] In this embodiment, firstly, the action vector This includes active and reactive power control quantities for photovoltaic generators, wind turbines, diesel generators, energy storage, and charging stations, with the reactive power output of each device meeting the apparent power margin constraint. ; in: k The equipment index should include at least photovoltaic, wind power, diesel generator sets, energy storage, and charging stations; For the first k The rated apparent power of each device; They are time points tNo. k The active and reactive power output of each device; when near At this time, the upper limit of reactive power output is reduced accordingly to ensure that the equipment output does not exceed the rated capacity. In this embodiment, this constraint and action vector... It is relevant because it involves power control and regulation in microgrids, particularly the active and reactive power output of individual devices within the microgrid. For example, in the reinforcement learning control framework of the embodiment, the action vector... This constraint is used to represent control commands, and it ensures that the output power does not exceed the device's rated capacity, thus preventing device overload.

[0039] Then, the action vector is adjusted according to the apparent power margin constraint, the energy storage SOC boundary constraint, and the charge acceptance capability constraint. Perform feasibility analysis to obtain executable actions. In this embodiment, the energy storage SOC boundary constraints are expressed as follows: In the formula, These are the upper and lower limits of the energy storage state of charge; Let i be the state of charge of energy storage at time t; For the safety margin of energy storage, in this embodiment, the charging acceptance capability represents the maximum charging power that the energy storage battery pack can accept under the current state, in kilowatts.

[0040] In this embodiment, relevant constraints are applied to the action vector. Feasibility processing includes: eliminating candidate actions that violate constraints or mapping them within the set of feasible actions based on the principle of minimum modification to obtain alternative actions that satisfy the constraints, and performing consistent constraint checks and limiting on the parameters contained in the candidate actions to ensure that they are consistent with the relevant constraints in the set of feasible actions.

[0041] Secondly, the executable actions are based on the steady-state power flow model. Perform the execution and calculation to obtain the node voltage. Network loss and PCC interaction power Rewards are calculated based on indicators such as voltage over-limit / deviation, grid loss, PCC deviation, renewable energy curtailment, fuel cost, and service quality. With termination mark .

[0042] Specifically, in a preferred embodiment, the executable action is performed in the microgrid steady-state power flow model to obtain the node voltage. Network loss and PCC interaction power And calculate the reward function. With termination mark The termination flag is triggered based on the calculation results of the reward function, combined with the satisfaction of a series of key indicators in the microgrid system (such as voltage, power interaction, and state of charge). When these indicators meet the predetermined standards and the value of the reward function reaches the required level, the termination flag is activated, marking the completion of the training or operation process.

[0043] In this embodiment, the steady-state power flow model is mainly used to perform physical-level execution and calculation of executable actions, obtaining operational results such as node voltage, network loss, and PCC interactive power; reward function With termination mark This is calculated based on the above operational results, according to preset indicators such as voltage deviation, network loss, PCC deviation, renewable energy curtailment, fuel cost, and service quality, rather than being directly implemented by the Critic network. The role of the Critic network is mainly reflected in step seven, namely, value assessment and TD3 parameter updates based on interactive samples. Specifically: Input executable actions The control input characteristics include various control parameters of the power grid, such as power dispatch and load requirements.

[0044] Perform steady-state power flow calculations: Based on the input control quantities, numerical calculation methods, such as the Newton-Raphson method, are used to calculate node voltages, network losses, and PCC interactive power.

[0045] In this embodiment, the reward function is calculated. Based on the power grid calculation results, such as node voltage and network losses, a reward value is calculated using a reward function design formula. Critic network evaluation: The Critic network evaluates the value of the current state and action combination and feeds this evaluation back to the Actor network for policy optimization.

[0046] Preferably, the reward function is a weighted combination of at least two of the following penalty items, and includes at least the following: voltage limit violation penalty, network loss penalty, PCC interactive power deviation penalty, renewable energy curtailment penalty, fuel cost penalty, charging demand unmet penalty, and SOC maintenance penalty:

[0047] in: Voltage over-limit / deviation indicator; For network loss indicators; This refers to the PCC interactive power deviation index. For renewable energy curtailment quotas; For fuel cost indicators; The charging demand was not met. This serves as a benchmark for maintaining the energy storage SOC within a specified range. represents the weight coefficients of each penalty term, and is a non-negative real number. In this embodiment, these indicators are indirectly reflected in the reward function through several penalty terms. The representative network loss indicator is derived by calculating the network loss in a microgrid. and The calculation results for voltage deviation and power interaction deviation will affect the final reward value, thereby adjusting the system's control strategy and optimizing the stability and efficiency of the microgrid.

[0048] Interactive Samples The data is stored in the experience replay pool and subjected to dual Critic updates, delayed Actor updates, and soft updates of the target network within the TD3 framework. The trained control policy parameters are also stored for online inference. The data is stored in the experience replay pool, and the parameters of the dual Critic network and Actor network are updated under the TD3 framework to obtain the trained control strategy. This strategy is used to output active and reactive power control commands for each distributed resource of the microgrid, such as... Figure 1 As shown.

[0049] TD3 training updates include: minimum objective with dual criters, objective policy smoothing, delayed policy updates, and soft updates, and the objective value... It is calculated through the Critic network and represents the active and reactive power control commands for each distributed resource in the power grid.

[0050] The target value is closely related to the regulation of microgrid resources (such as photovoltaic, wind power, energy storage, etc.), and corresponding control actions are output through the Actor network to ensure the efficient and stable operation of the microgrid and meet the following requirements: ; in: Discount factor; This is the termination marker, used when the round ends. ,otherwise, ; , For the target Critic network; The target action after the target policy is smoothed satisfies:

[0051] in, For the target Actor network, To cut off noise, Indicates the amplitude limiting operator. The upper and lower bounds for the actions are set in this embodiment based on actual operational requirements and the physical limitations of the equipment. These upper and lower bounds are derived from the action vectors obtained above. Decision made Gaussian exploration noise is used to execute actions during the training phase. ; in: The deterministic policy output by the current Actor network; It is zero-mean Gaussian noise. The noise covariance matrix or the diagonal variance matrix; The decay occurs as training progresses, in order to achieve sufficient exploration in the early stage and stable convergence in the later stage; Used to restrict the execution of an action to its upper and lower bounds. Inside.

[0052] After training, exploration noise is turned off and a deterministic policy is adopted to output the state observation vector based on time t. Action vectors are calculated online. This allows for the output of active and reactive power control commands for each distributed resource in the microgrid, enabling coordinated control of voltage qualification and power self-balancing within the microgrid.

[0053] The systems identified based on the above methods include: The state feature vector acquisition module is used to acquire the control input feature quantities of the microgrid at various times, and to distinguish the control input feature quantities into the main control object and other objects. The main control object is the feature quantity corresponding to the resource or equipment that affects the stability and power balance of the microgrid, and the other objects are the feature quantities corresponding to the equipment that works in cooperation with the main control object or other loads or resources that are responsible for assisting in the regulation of the microgrid. The control input feature quantities are normalized and time-encoded to obtain the state feature vector for input to the policy network. The sub-vector determination module is used to input the state feature vector into the state representation layer of the policy network. The state representation layer is used to perform state representation on the state feature vector and obtain an enhanced feature vector. Based on the enhanced feature vector and the distinguished objects, master-slave cascade decoupling is performed to obtain a first action sub-vector and a second action sub-vector. The master-slave cascade decoupling is used to generate the action decision of the master object, i.e., the first action sub-vector, through the main branch of the policy network, and also to generate the collaborative decision of the remaining objects, i.e., the second action sub-vector, through the slave branch of the policy network. The action vector generation module is used to concatenate the enhanced feature vector, the first action sub-vector, and the second action sub-vector and input them into the compensation branch of the policy network to generate a compensation action vector. The compensation action vector is the active power regulation compensation action of the energy storage battery, thereby forming the final action vector. The final action vector is made feasible according to the constraints of the microgrid to obtain an executable action vector. The control strategy training module is used to execute and calculate the executable action vectors according to numerical calculation methods to obtain node voltage, network loss and PCC interaction power. After determining the reward function, the interaction samples are stored in the experience replay pool. Under the TD3 framework, dual Critic network update, Actor delayed update and target network soft update are performed, and the trained control strategy parameters are stored for online inference calls.

[0054] The other technical features of the microgrid voltage and power self-balancing cooperative control system based on two-end augmented deep reinforcement learning described in this embodiment are similar to the corresponding microgrid voltage and power self-balancing cooperative control method based on two-end augmented deep reinforcement learning, and will not be repeated here.

[0055] The improved TD3 framework node obtained based on this embodiment is defined as a DE-TD3 node, such as... Figure 3 The node voltage and transient overvoltage analysis shown are as follows: Figure 3 The average maximum voltage deviation of TD3 shown in (a) is 0.026286 pu; in this embodiment... Figure 3 (b) shows that the average maximum voltage deviation of DE-TD3 is 0.021464 pu. The node voltage deviation is smaller and the voltage stability is better in the DE-TD3 scenario.

[0056] like Figure 4 The PCC tie-line power interaction analysis is shown below. Figure 4 As shown in (a), the average daily interaction volume of TD3 is 0.011766; Figure 4 (b) shows that the average daily interaction volume of DE-TD3 is 0.010396. In the DE-TD3 scenario, the PCC tie-line power fluctuation is smaller and the interaction with the grid is more stable.

[0057] like Figure 5 The voltage, network loss, and light curtailment information are shown. Figure 5 The average daily network loss of TD3 shown in (a) is 0.096405; Figure 5 (b) shows that the average daily network loss for DE-TD3 is 0.038137. The system network loss is significantly reduced in the DE-TD3 scenario, and the energy utilization efficiency is higher.

[0058] Therefore, in microgrid optimization operation scenarios, the DE-TD3 control strategy outperforms the traditional TD3 strategy in the following aspects: 1) Voltage stability: smaller node voltage deviation and smoother system operation; 2) Power interaction: smaller power fluctuations in PCC tie lines and lower impact on the grid; 3) Energy efficiency: significantly reduced daily grid loss and higher energy utilization efficiency; 4) Wind and solar integration: a significant decrease in wind curtailment rate and a significant increase in renewable energy utilization.

[0059] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0060] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0061] The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform a microgrid voltage and power self-balancing cooperative control method based on dual-end augmented deep reinforcement learning as provided in any one or more of the above embodiments. The electronic device includes: one or more central processing units (CPUs), and interfaces for connecting various components, such as displays, infrared sensors, and cameras. That is, the various components are interconnected using different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory sets, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown in this embodiment are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0062] In a preferred embodiment of this invention, the electronic device may further include an input device and an output device. The processing unit, memory, input device, and output device may be connected via a bus or other means.

[0063] The input device can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. The output device may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays, light-emitting diode displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

[0064] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0065] In this embodiment, a computer-readable medium stores a computer program / instructions, which, when executed by a processor, implement a method for constructing insulation degradation characteristics of a generator motor provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The computer-readable medium carries one or more computer-readable instructions.

[0066] Memory can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The central processing unit executes various server functions and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0067] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0069] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0070] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0071] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0072] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0073] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0074] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0075] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A micro-grid voltage and power self-balancing cooperative control method, characterized in that, The method includes: The control input features of the microgrid at various times are obtained, and these features are categorized into main control objects and other objects. The main control objects are the features corresponding to resources or devices that affect the stability and power balance of the microgrid, while the other objects are the features corresponding to devices that work in conjunction with the main control objects or other loads or resources that are responsible for assisting in the regulation of the microgrid. The control input features are then normalized and time-encoded to obtain the state feature vector for the policy network input. The state feature vector is input to the state representation layer of the policy network. The state representation layer is used to represent the state feature vector and obtain an enhanced feature vector. Based on the enhanced feature vector and the distinguished objects, master-slave cascade decoupling is performed to obtain a first action sub-vector and a second action sub-vector. The master-slave cascade decoupling is used to generate the action decision of the master object, i.e., the first action sub-vector, through the main branch of the policy network, and also to generate the collaborative decision of the remaining objects, i.e., the second action sub-vector, through the slave branch of the policy network. The enhanced feature vector, the first action sub-vector, and the second action sub-vector are concatenated and input into the compensation branch of the policy network to generate a compensation action vector. The compensation action vector is the active power regulation compensation action of the energy storage battery, thus forming the final action vector. The final action vector is made feasible according to the constraints of the microgrid to obtain an executable action vector. The executable action vector is executed and calculated using numerical calculation methods to obtain node voltage, network loss and PCC interaction power. After determining the reward function, the interaction samples are stored in the experience replay pool. Under the TD3 framework, dual Critic network update, Actor delayed update and target network soft update are executed. The trained control policy parameters are stored for online inference calls.

2. The microgrid voltage and power self-balancing coordinated control method according to claim 1, characterized in that, The state representation layer is used to represent the state feature vector and obtain an enhanced feature vector, including: The state feature vector is input to the state representation layer of the policy network to obtain the state representation feature vector. , is represented as: ;in, and These are the weight matrix and bias vector of the state representation layer, respectively. It is a non-linear activation function. This is the current state feature vector. Channel weight vectors are generated through the channel attention module to represent the state feature vector. We perform weighting to obtain the enhanced feature vector. .

3. The microgrid voltage and power self-balancing coordinated control method according to claim 2, characterized in that, The channel weight vector is generated through the channel attention module, which is used to represent the state feature vector. We perform weighting to obtain the enhanced feature vector. This includes: representing the state feature vector After performing compression mapping, the bottleneck vector is obtained, represented as: Then, the channel weight vector is generated by the activation mapping, which is expressed as: The enhanced features are obtained by channel recalibrating the state representation feature vector, as follows: ; in, ReLU represents the nonlinear activation function; This represents the Sigmoid activation function; These are the weight matrices of the fully connected layer. These are the bias vectors; For channel weight vectors, This represents element-wise multiplication; These are the state representation features enhanced by attention, used for subsequent policy output or value evaluation.

4. The microgrid voltage and power self-balancing coordinated control method according to claim 3, characterized in that, The process of combining the enhanced feature vectors with the differentiated objects to perform master-slave cascade decoupling, thereby obtaining the first action sub-vector and the second action sub-vector, includes: Represent the first action subvector and the second action subvector: ,in, This represents the hyperbolic tangent activation function, which restricts the action output to... , This represents the strategy mapping corresponding to the main branch, resulting in the decision used to control the main control object. This indicates that the strategy mapping corresponding to the branch is used to coordinate and adjust other devices or resources. These are all nonlinear mappings on different branches within the Actor network.

5. The microgrid voltage and power self-balancing coordinated control method according to claim 4, characterized in that, The step of storing the interaction samples into the experience replay pool after determining the reward function includes: The reward function is a weighted combination of at least two of the following penalty items, and includes at least the following: voltage limit violation penalty, network loss penalty, PCC interactive power deviation penalty, renewable energy curtailment penalty, fuel cost penalty, charging demand unmet penalty, and SOC maintenance penalty: in, This refers to voltage exceeding limits or deviation indicators; For network loss indicators; PCC interactive power deviation index; For renewable energy curtailment quotas; For fuel cost indicators; The charging demand was not met. These are the maintenance indicators for energy storage SOC deviation from the target; the voltage over-limit or deviation indicator is obtained from the node voltage, the network loss indicator is obtained from the network loss, and the PCC interactive power deviation indicator is obtained from the PCC interactive power. These are the weighting coefficients for the following indicators: voltage over-limit or deviation, network loss, PCC interactive power deviation, new energy curtailment, fuel cost, unmet charging demand, and maintenance of energy storage SOC deviating from the target. All of these are non-negative real numbers.

6. A microgrid voltage and power self-balancing coordinated control system, characterized in that, The system includes: The state feature vector acquisition module is used to acquire the control input feature quantities of the microgrid at various times, and to distinguish the control input feature quantities into the main control object and other objects. The main control object is the feature quantity corresponding to the resource or equipment that affects the stability and power balance of the microgrid, and the other objects are the feature quantities corresponding to the equipment that works in cooperation with the main control object or other loads or resources that are responsible for assisting in the regulation of the microgrid. The control input feature quantities are normalized and time-encoded to obtain the state feature vector for input to the policy network. The sub-vector determination module is used to input the state feature vector into the state representation layer of the policy network. The state representation layer is used to perform state representation on the state feature vector and obtain an enhanced feature vector. Based on the enhanced feature vector and the distinguished objects, master-slave cascade decoupling is performed to obtain a first action sub-vector and a second action sub-vector. The master-slave cascade decoupling is used to generate the action decision of the master object, i.e., the first action sub-vector, through the main branch of the policy network, and also to generate the collaborative decision of the remaining objects, i.e., the second action sub-vector, through the slave branch of the policy network. The action vector generation module is used to concatenate the enhanced feature vector, the first action sub-vector, and the second action sub-vector and input them into the compensation branch of the policy network to generate a compensation action vector. The compensation action vector is the active power regulation compensation action of the energy storage battery, thereby forming the final action vector. The final action vector is made feasible according to the constraints of the microgrid to obtain an executable action vector. The control strategy training module is used to execute and calculate the executable action vectors according to numerical calculation methods to obtain node voltage, network loss and PCC interaction power. After determining the reward function, the interaction samples are stored in the experience replay pool. Under the TD3 framework, dual Critic network update, Actor delayed update and target network soft update are performed, and the trained control strategy parameters are stored for online inference calls.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

Images