Method and device for generating control parameters of voltage regulating equipment based on deep reinforcement learning

Abstract

The present application discloses a method and device for generating control parameters of voltage regulating equipment based on deep reinforcement learning, which relates to the field of power grid operation technology, and its main purpose is to improve the problem that the existing power grid cannot meet the needs of new power grids in reactive power optimization, and thus cannot generate control parameters of voltage regulating equipment that meet the needs. It includes: obtaining the state parameters of the target power grid at a preset time; generating a control parameter generation model based on the completed model training, generating control parameters of the voltage regulating equipment according to the state parameters, the control parameter generation model is constructed based on a reinforcement learning network, and is obtained by model training based on a deep deterministic policy gradient algorithm and the interaction state with the target power grid; controlling each voltage regulating equipment in the target power grid according to the control parameters of the voltage regulating equipment.

Description

Technical Field

[0001] The present application relates to the technical field of power grid operation, and in particular to a method and device for generating control parameters of a voltage regulating device based on deep reinforcement learning. Background Art

[0002] As the development of new energy continues to gain attention, the proportion of new energy power generation in new power grids is also increasing. Due to the natural randomness and volatility of new energy power generation, it is impossible to output stably; at the same time, its load demand is also uncertain. Under the combined influence of the unstable output of new energy power generation and the uncertainty of load demand, the power grid is prone to large voltage fluctuations and even frequent over-limits during actual operation, which will lead to the inability of the power grid to operate normally and reduce the safety of power grid operation. At the same time, the large voltage fluctuations will also lead to an increase in power grid losses, thereby increasing the cost of power grid operation.

[0003] In order to ensure the normal operation of the power grid, the reactive power compensation equipment, on-load tap-changing transformer tap changers and other voltage regulating equipment can be controlled to optimize the power grid flow distribution, reduce the power grid loss and achieve the normal operation of the power grid under the premise of ensuring the stability of the power grid voltage, which is the reactive power optimization problem. The reactive power optimization problem is a high-dimensional, nonlinear, constrained hybrid optimization problem. At present, the solution methods for reactive power optimization problems include traditional analytical methods such as linear programming, dynamic programming, cone programming relaxation, decoupled interior point method, mixed integer programming, and heuristic algorithms such as genetic algorithm, particle swarm algorithm, and gray wolf algorithm.

[0004] However, in actual operation, the traditional analytical methods require high-precision physical modeling and initial values, and are prone to non-convergence when the power grid scenario changes; and the heuristic algorithm has a slow calculation speed and is prone to falling into the local optimal solution. Therefore, the above methods cannot meet the needs of the new power grid for reactive power optimization problems, and thus cannot generate voltage regulating equipment control parameters that meet the needs. Summary of the invention

[0005] In view of this, the present application provides a method and device for generating control parameters of a voltage regulating device based on deep reinforcement learning. The main purpose is to improve the problem that the existing system cannot meet the needs of new power grids in reactive power optimization, and thus cannot generate control parameters of voltage regulating equipment that meet the needs.

[0006] According to one aspect of the present application, a method for generating control parameters of a voltage regulating device based on deep reinforcement learning is provided, comprising:

[0007] Obtaining state parameters of the target power grid at a preset time;

[0008] Based on the control parameter generation model of the completed model training, the voltage regulating device control parameters are generated according to the state parameters, wherein the control parameter generation model is constructed based on a reinforcement learning network and is obtained by model training based on a deep deterministic policy gradient algorithm and combined with the interaction state between the target power grid;

[0009] Each voltage regulating device in the target power grid is controlled according to the voltage regulating device control parameter.

[0010] Preferably, before generating the control parameter generation model based on the completed model training and generating the voltage regulating device control parameter according to the state parameter, the method further comprises:

[0011] Construct the initial control parameter generation model, initial control parameter evaluation model, initial target control parameter generation model, and initial target control parameter evaluation model based on the reinforcement learning network;

[0012] Acquire a training data set, wherein the training data set includes multiple training data groups, and the training data groups include current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters;

[0013] Based on the initial control parameter evaluation model, generating an action value of the control parameter at the current moment according to the current state parameter and the current control parameter;

[0014] Based on the initial target control parameter evaluation model, the target action value of the control parameter at the next moment is generated according to the next moment state parameter and the next moment control parameter, wherein the next moment control parameter is predicted based on the initial target control parameter generation model according to the next moment state parameter;

[0015] updating the parameters of the initial control parameter evaluation model based on a mean square error loss function between the action value and the target action value to obtain an updated control parameter evaluation model;

[0016] Based on the initial control parameter generation model, a predicted control parameter is generated according to the current state parameter, and based on the updated control parameter evaluation model, a predicted action value of the predicted control parameter is generated according to the current state parameter and the predicted control parameter;

[0017] Calculating a deep deterministic policy gradient based on a gradient of the predicted action value to the predicted control parameter and a gradient of the predicted control parameter to a parameter of the initial control parameter generation model;

[0018] Updating the parameters of the initial control parameter generation model based on the deep deterministic policy gradient to obtain an updated control parameter generation model;

[0019] Copying the parameters of the updated control parameter evaluation model to the initial target control parameter evaluation model to obtain an updated target control parameter evaluation model, so as to generate a target action value based on the updated target control parameter evaluation model;

[0020] Copying the parameters of the updated control parameter generation model to the initial target control parameter generation model to obtain an updated target control parameter generation model, so as to generate a prediction control parameter based on the updated target control parameter generation model;

[0021] Repeat the above steps until the preset model training stop condition is met, and obtain the control parameter generation model and the control parameter evaluation model that have completed the model training, so as to generate the control parameters of the voltage regulating device based on the control parameter generation model that has completed the model training.

[0022] Preferably, the obtaining of the training data set includes:

[0023] Based on the initial control parameter generation model, the current moment control parameters are generated according to the current moment state parameters;

[0024] Calculate the current control parameter reward value according to the following formula:

[0025]

[0026] Among them, R represents the current control parameter reward value, c loss Represents the weight factor of network loss, P loss represents the grid loss, c u represents the weight factor of node voltage deviation, ΔU represents voltage deviation, c m Represents the weight factor of reactive power margin, Q margin represents reactive power margin, n represents the number of network nodes, U i represents the voltage amplitude of node i, U ref represents the reference voltage, m1 represents the number of capacitor banks, m2 represents the number of SVC reactive compensation devices, Indicates the maximum reactive power output of the capacitor bank, Q CB，i Indicates the reactive output of the capacitor bank. Indicates the maximum reactive power output of the SVC reactive power compensation device, Q SVC，i Indicates the reactive power output of the SVC reactive power compensation device;

[0027] The next moment state parameters are generated based on the following formula:

[0028]

[0029] Among them, s t+1 Indicates the state parameters at the next moment, represents the active power output of the photovoltaic node i at the current moment, represents the power factor of the photovoltaic node i at the current moment, represents the reactive power output of the capacitor bank at node i at the current moment, represents the reactive power output of the SVC reactive power compensation device at node i at the current moment, T i,t Indicates the transformer tap position value. represents the active output of the load at node i at the next moment, represents the reactive power output of node i load at the next moment, U i,t+1 Indicates the voltage value of node i at the next moment;

[0030] Combine the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data set;

[0031] Repeat the above steps to generate multiple sets of training data sets;

[0032] A plurality of the training data sets are combined to generate a training data set.

[0033] Preferably, before calculating the current moment control parameter reward value, the method further includes:

[0034] The renewable energy output constraint is expressed as follows:

[0035]

[0036] Among them, P DG,i Represents the active output of renewable distributed energy, P DG,min Represents the lower limit of the active output of renewable distributed energy, P DG,max represents the upper limit of the active output of renewable distributed energy, Q DG,i represents the reactive power output of renewable distributed energy, Q DG,min represents the lower limit of the reactive power output of renewable distributed energy, Q DG,max It represents the upper limit of the reactive power output of renewable distributed energy;

[0037] The reactive power processing constraint condition of reactive power generating compensation equipment is expressed as the following formula:

[0038]

[0039] Among them, Q SVC,iIndicates the reactive power output of the SVC reactive power compensation device, Q SVC,min Indicates the lower limit of reactive output regulation of SVC reactive compensation equipment, Q SVC,max Indicates the upper limit of reactive output regulation of SVC reactive compensation equipment, Q CB,i Indicates the reactive output of the capacitor bank, Q CB,min Indicates the lower limit of reactive output regulation of capacitor bank, Q CB,max Indicates the upper limit of reactive output regulation of capacitor bank;

[0040] If the current control parameter satisfies both the renewable energy output constraint and the reactive power processing constraint of the reactive power generating compensation equipment, the current control parameter is added to the training data set to calculate the reward value of the current control parameter in the training data set.

[0041] Preferably, before combining the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data set, the method further includes:

[0042] The power flow constraint condition is expressed as follows:

[0043]

[0044] Among them, P G,i represents the active power of the generator at node i, P DG,i represents the active power of distributed generation at node i, P L,i represents the active power of the load at node i, Q G,i represents the reactive power of the generator at node i, Q DG,i represents the reactive power of distributed generation at node i, Q C,i represents the reactive power of the capacitor bank at node i, Q L,i represents the reactive power of the load at node i, U i represents the voltage amplitude of node i, U j represents the voltage amplitude of node j, G ij and B ij represents the line admittance value between node i and node j, θ ij represents the voltage phase difference between node i and node j, P DG,i represents the active output of renewable distributed energy, Q DG,i Represents the reactive power output of renewable distributed energy, S DG,i Represents the apparent power of renewable distributed energy;

[0045] The node voltage constraint is expressed as follows:

[0046] Uimin ≤U i ≤U imax

[0047] Among them, U i represents the voltage amplitude of node i, U imin Indicates the minimum voltage at node i, U imax represents the maximum value of the voltage at node i;

[0048] The current safety constraint condition is expressed as follows:

[0049] I ij ≤I ij,max

[0050] Among them, I ij represents the current in the branch between node i and node j, I ij,max represents the maximum allowable current in the branch between node i and node j;

[0051] If the state parameter at the next moment satisfies the power flow constraint condition, the node voltage constraint condition, and the current safety constraint condition at the same time, the state parameter at the next moment is added to the training data set.

[0052] Preferably, before calculating the current moment control parameter reward value, the method further includes:

[0053] Gaussian noise is added to the current moment control parameter to calculate the current moment control parameter reward value according to the current moment control parameter after the Gaussian noise is added.

[0054] Preferably, the state parameters include: active output and power factor of the distributed power source, reactive output of the SVC reactive compensation device, reactive output of the capacitor bank, and transformer tap position value.

[0055] According to another aspect of the present application, a device for generating control parameters of a voltage regulating device based on deep reinforcement learning is provided, comprising:

[0056] A state parameter acquisition module is used to obtain the state parameters of the target power grid at a preset time;

[0057] A control parameter generation module, which is used to generate a control parameter generation model based on a completed model training, and to generate a voltage regulating device control parameter according to the state parameter, wherein the control parameter generation model is obtained by model training based on a deep deterministic policy gradient algorithm and in combination with the interaction state with the target power grid;

[0058] The voltage regulating device control module is used to control each voltage regulating device in the target power grid according to the voltage regulating device control parameters.

[0059] Preferably, before the control parameter generation module, the device further includes a model training module, including:

[0060] A model building unit, used to build an initial control parameter generation model, an initial control parameter evaluation model, an initial target control parameter generation model, and an initial target control parameter evaluation model based on a reinforcement learning network;

[0061] A training data set acquisition unit, used to acquire a training data set, wherein the training data set includes a plurality of training data groups, wherein the training data groups include current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters;

[0062] A control parameter evaluation model training unit, configured to generate an action value of the control parameter at the current moment based on the initial control parameter evaluation model, according to the current state parameter and the current control parameter;

[0063] The control parameter evaluation model training unit is further used to generate a target action value of the control parameter at the next moment based on the initial target control parameter evaluation model, according to the next moment state parameter and the next moment control parameter, wherein the next moment control parameter is predicted based on the initial target control parameter generation model according to the next moment state parameter;

[0064] The control parameter evaluation model training unit is further used to update the parameters of the initial control parameter evaluation model based on the mean square error loss function between the action value and the target action value to obtain an updated control parameter evaluation model;

[0065] A control parameter generation model training unit, configured to generate a predicted control parameter based on the initial control parameter generation model and the current state parameter, and to generate a predicted action value of the predicted control parameter based on the updated control parameter evaluation model and the current state parameter and the predicted control parameter;

[0066] The control parameter generation model training unit is further used to calculate the deep deterministic policy gradient according to the gradient of the predicted action value to the predicted control parameter and the gradient of the predicted control parameter to the parameter of the initial control parameter generation model;

[0067] The control parameter generation model training unit is further used to update the parameters of the initial control parameter generation model based on the deep deterministic policy gradient to obtain an updated control parameter generation model;

[0068] A target model updating unit, used for copying the parameters of the updated control parameter evaluation model to the initial target control parameter evaluation model to obtain an updated target control parameter evaluation model, so as to generate a target action value based on the updated target control parameter evaluation model;

[0069] The target model updating unit is further used to copy the parameters of the updated control parameter generation model to the initial target control parameter generation model to obtain an updated target control parameter generation model, so as to generate a prediction control parameter based on the updated target control parameter generation model;

[0070] The model training stopping unit is used to repeat the above steps until the preset model training stopping condition is met, so as to obtain the control parameter generation model and the control parameter evaluation model that have completed the model training, so as to generate the control parameters of the voltage regulating device based on the control parameter generation model that has completed the model training.

[0071] Preferably, the training data set acquisition unit includes:

[0072] A control parameter generation subunit, configured to generate a current control parameter based on the initial control parameter generation model and according to the current state parameter;

[0073] The reward value calculation subunit is used to calculate the current control parameter reward value according to the following formula:

[0074]

[0075] Among them, R represents the current control parameter reward value, c loss Represents the weight factor of network loss, P loss represents the grid loss, c u represents the weight factor of node voltage deviation, ΔU represents voltage deviation, c m Represents the weight factor of reactive power margin, Q margin represents reactive power margin, n represents the number of network nodes, U i represents the voltage amplitude of node i, U ref represents the reference voltage, m1 represents the number of capacitor banks, m2 represents the number of SVC reactive compensation devices, Indicates the maximum reactive power output of the capacitor bank, Q CB，i Indicates the reactive output of the capacitor bank. Indicates the maximum reactive power output of the SVC reactive power compensation device, Q SVC，i Indicates the reactive power output of the SVC reactive power compensation device;

[0076] The next moment state parameter generation subunit is used to generate the next moment state parameter based on the following formula:

[0077]

[0078] Among them, s t+1 Indicates the state parameters at the next moment, represents the active power output of the photovoltaic node i at the current moment, represents the power factor of the photovoltaic node i at the current moment, represents the reactive power output of the capacitor bank at node i at the current moment, represents the reactive power output of the SVC reactive power compensation device at node i at the current moment, T i,t Indicates the transformer tap position value. represents the active output of the load at node i at the next moment, represents the reactive power output of node i load at the next moment, U i,t+1 Indicates the voltage value of node i at the next moment;

[0079] A training data group generating subunit, used for combining the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data group;

[0080] The training data group generating subunit is further used to repeat the above steps to generate multiple training data groups;

[0081] The training data set generating subunit is used to combine a plurality of the training data sets to generate a training data set.

[0082] Preferably, before the reward value calculation subunit, the training data set acquisition unit further includes a control parameter determination subunit, which is used to:

[0083] The renewable energy output constraint is expressed as follows:

[0084]

[0085] Among them, P DG,i Represents the active output of renewable distributed energy, P DG,min Represents the lower limit of the active output of renewable distributed energy, P DG,max represents the upper limit of the active output of renewable distributed energy, Q DG,i represents the reactive power output of renewable distributed energy, Q DG,min represents the lower limit of the reactive power output of renewable distributed energy, Q DG,max It represents the upper limit of the reactive power output of renewable distributed energy;

[0086] The reactive power processing constraint condition of reactive power generating compensation equipment is expressed as the following formula:

[0087]

[0088] Among them, Q SVC,i Indicates the reactive power output of the SVC reactive power compensation device, Q SVC,min Indicates the lower limit of reactive output regulation of SVC reactive compensation equipment, Q SVC,max Indicates the upper limit of reactive output regulation of SVC reactive compensation equipment, Q CB,i Indicates the reactive output of the capacitor bank, Q CB,min Indicates the lower limit of reactive output regulation of capacitor bank, Q CB,max Indicates the upper limit of reactive output regulation of capacitor bank;

[0089] If the current control parameter satisfies both the renewable energy output constraint and the reactive power processing constraint of the reactive power generating compensation equipment, the current control parameter is added to the training data set to calculate the reward value of the current control parameter in the training data set.

[0090] Preferably, before the training data set generating subunit, the training data set acquiring unit further includes a next moment state parameter determining subunit, which is used to:

[0091] The power flow constraint condition is expressed as follows:

[0092]

[0093] Among them, P G,i represents the active power of the generator at node i, P DG,i represents the active power of distributed generation at node i, P L,i represents the active power of the load at node i, Q G,i represents the reactive power of the generator at node i, Q DG,i represents the reactive power of distributed generation at node i, Q C,i represents the reactive power of the capacitor bank at node i, Q L,i represents the reactive power of the load at node i, U i represents the voltage amplitude of node i, U j represents the voltage amplitude of node j, G ij and B ij represents the line admittance value between node i and node j, θ ij represents the voltage phase difference between node i and node j, P DG,i represents the active output of renewable distributed energy, Q DG,i Represents the reactive power output of renewable distributed energy, S DG,i Represents the apparent power of renewable distributed energy;

[0094] The node voltage constraint is expressed as follows:

[0095] U imin ≤U i ≤U imax

[0096] Among them, U i represents the voltage amplitude of node i, U imin Indicates the minimum voltage at node i, U imax represents the maximum value of the voltage at node i;

[0097] The current safety constraint condition is expressed as follows:

[0098] I ij ≤I ij,max

[0099] Among them, I ij represents the current in the branch between node i and node j, I ij,max represents the maximum allowable current in the branch between node i and node j;

[0100] If the state parameter at the next moment satisfies the power flow constraint condition, the node voltage constraint condition, and the current safety constraint condition at the same time, the state parameter at the next moment is added to the training data set.

[0101] Preferably, before the reward value calculation subunit, the training data set acquisition unit further includes a noise adding subunit, which is used to:

[0102] Gaussian noise is added to the current moment control parameter to calculate the current moment control parameter reward value according to the current moment control parameter after the Gaussian noise is added.

[0103] Preferably, the state parameters include: active output and power factor of the distributed power source, reactive output of the SVC reactive compensation device, reactive output of the capacitor bank, and transformer tap position value.

[0104] According to another aspect of the present application, a storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned method for generating control parameters of a voltage regulating device based on deep reinforcement learning.

[0105] According to another aspect of the present application, a terminal is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0106] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned method for generating control parameters of a voltage regulating device based on deep reinforcement learning.

[0107] By means of the above technical solution, the technical solution provided by the embodiment of the present application has at least the following advantages:

[0108] The present application provides a method and device for generating control parameters of a voltage regulating device based on deep reinforcement learning, firstly, the state parameters of the target power grid at a preset time are obtained; secondly, a control parameter generation model based on the completed model training is generated, and the control parameters of the voltage regulating device are generated according to the state parameters, and the control parameter generation model is constructed based on a reinforcement learning network, and is obtained by model training in combination with the interaction state between the target power grid on the basis of a deep deterministic policy gradient algorithm; finally, each voltage regulating device in the target power grid is controlled according to the control parameters of the voltage regulating device. Compared with the prior art, the embodiment of the present application constructs a control parameter generation model based on a reinforcement learning network, simulates the nonlinear relationship between the voltage regulating device and the state of the target power grid, and performs model training in combination with the interaction state between the target power grid on the basis of a deep deterministic policy gradient algorithm, and then generates the control parameters of the voltage regulating device based on the trained model, and controls the voltage regulating device according to the control parameters, so as to optimize the power grid flow distribution, reduce the power grid loss, and realize the normal operation of the power grid under the premise of ensuring the stability of the power grid voltage, and at the same time avoids the problems of non-convergence and falling into the local optimal solution that may occur when the control parameters are generated by the traditional analytical method and the heuristic algorithm, and meets the needs of the new power grid when generating the control parameters of the voltage regulating device.

[0109] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0111] Figure 1 A flow chart of a method for generating control parameters of a voltage regulating device based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0112] Figure 2 A control parameter generation model training flow chart provided in an embodiment of the present application is shown;

[0113] Figure 3 A flow chart for obtaining a training data set provided in an embodiment of the present application is shown;

[0114] Figure 4 A comparison diagram of voltage control effects provided by an embodiment of the present application is shown;

[0115] Figure 5 A block diagram of a device for generating control parameters of a voltage regulating device based on deep reinforcement learning provided in an embodiment of the present application is shown;

[0116] Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0117] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0118] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0119] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0120] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0121] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0122] The embodiments of the present application can be applied to computer systems / servers, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0123] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0124] The present application embodiment provides a method for generating control parameters of a voltage regulating device based on deep reinforcement learning, such as Figure 1 As shown, the method includes:

[0125] 101. Obtain state parameters of a target power grid at a preset time.

[0126] Among them, the state parameters may include the active output and power factor of the distributed power source, the reactive output of the SVC reactive compensation device, the reactive output of the capacitor bank, the transformer tap position value, etc.; the target power grid may include new energy power generation systems, such as solar energy, wind energy, etc., traditional energy power generation systems, such as thermal power generation, etc., and voltage regulating equipment for regulating voltage, etc.; the preset time may be the current time or a certain time in the future. In the embodiment of the present application, the current execution end may be a voltage regulating equipment control module in the power grid.

[0127] 102. Generate a control parameter model based on the completed model training, and generate a voltage regulating device control parameter according to the state parameter.

[0128] Among them, the control parameter generation model is constructed based on a reinforcement learning network, and is obtained by model training in combination with the interaction state between the target power grid on the basis of a deep deterministic policy gradient algorithm. In the embodiment of the present application, a control parameter generation model is pre-constructed based on a reinforcement learning network to simulate the nonlinear relationship between the voltage regulating device and the target power grid state. Further, on the basis of a deep deterministic policy gradient algorithm, the model is trained in combination with the interaction state between the target power grid to obtain a control parameter generation model that completes the model training, so that in the process of generating the control parameters of the voltage regulating device based on the control parameter generation model, it is possible to avoid the problems of non-convergence and falling into the local optimal solution that may occur when the control parameters are generated by the traditional analytical method and the heuristic algorithm, and meet the needs of the new power grid in generating the control parameters of the voltage regulating device. Further, according to the generated control parameters of the voltage regulating device, the voltage regulating device in the power grid is controlled to act, which can optimize the power grid flow distribution under the premise of ensuring the stability of the power grid voltage, reduce the power grid loss, and realize the normal operation of the power grid.

[0129] 103. Control each voltage regulating device in the target power grid according to the voltage regulating device control parameters.

[0130] The voltage regulating device control parameters are used to control the voltage regulating device to perform actions, for example, adjusting the parameters of the reactive power compensation device to a, adjusting the gear of the on-load voltage regulating transformer tap to b, and so on.

[0131] Compared with the prior art, the embodiment of the present application constructs a control parameter generation model based on a reinforcement learning network, simulates the nonlinear relationship between the voltage regulating device and the target power grid state, and performs model training based on the interaction state with the target power grid on the basis of a deep deterministic policy gradient algorithm. The control parameters of the voltage regulating device are then generated based on the trained model, and the voltage regulating device is controlled according to the control parameters, so as to optimize the power flow distribution of the power grid, reduce the power grid loss, and achieve the normal operation of the power grid while ensuring the stability of the power grid voltage. At the same time, it avoids the problems of non-convergence and falling into local optimal solutions that may occur when using traditional analytical methods and heuristic algorithms to generate control parameters, and meets the needs of new power grids in generating control parameters for voltage regulating devices.

[0132] In an embodiment of the present application, in order to further define and illustrate, Figure 2 As shown, in step 102 of the embodiment, a model is generated based on control parameters of the completed model training, and before the voltage regulating device control parameters are generated according to the state parameters, the embodiment method further includes:

[0133] 201. Based on the reinforcement learning network, the initial control parameter generation model, the initial control parameter evaluation model, the initial target control parameter generation model and the initial target control parameter evaluation model are constructed.

[0134] In an embodiment of the present application, the reinforcement learning network may select an Actor-Critical reinforcement learning network. Further, a control parameter generation model is constructed based on the Actor network to learn the policy function to generate control parameters; a control parameter evaluation model is constructed based on the Critical network to learn the state value function to evaluate the generated control parameters. Further, the model parameters of the control parameter generation model and the control parameter evaluation model are initialized to obtain an initial control parameter generation model and an initial control parameter evaluation model. The model parameters of the initial control parameter generation model and the initial control parameter evaluation model are then copied to the target control parameter generation model and the target control parameter evaluation model to obtain an initial target control parameter generation model and an initial target control parameter evaluation model.

[0135] It should be noted that the target control parameter evaluation model and the target control parameter generation model are equivalent to delayed versions of the control parameter evaluation model and the control parameter generation model, and their parameters are regularly copied from the control parameter evaluation model to stabilize the learning process.

[0136] 202. Obtain a training data set.

[0137] The training data set includes multiple training data groups, and the training data groups include current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters.

[0138] In an embodiment of the present application, a batch of data can be randomly obtained from a preset experience playback container as a training data set. It can be understood that the training data set can include multiple training data groups, each training data group includes current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters.

[0139] It should be noted that the experience replay container can use a circular queue to store the experience data generated by the interaction between the agent and the environment (i.e., current state parameters, current control parameters, current control parameter reward values, and next state parameters).

[0140] 203. Based on the initial control parameter evaluation model, according to the current state parameters and the current control parameters, generate the action value of the current control parameter.

[0141] 204. Based on the initial target control parameter evaluation model, a target action value of the control parameter at the next moment is generated according to the state parameter at the next moment and the control parameter at the next moment.

[0142] The control parameters at the next moment are generated based on the initial target control parameter generation model and are predicted according to the state parameters at the next moment.

[0143] 205. Update the parameters of the initial control parameter evaluation model based on the mean square error loss function between the action value and the target action value to obtain an updated control parameter evaluation model.

[0144] In steps 203-205, the model is first evaluated based on the initial control parameters (the model parameters are θ μ ), according to the current state parameter s t And the current control parameter a t , generate the action value Q(s) of the control parameter at the current moment t ,a t |θ μ ); further, based on the initial target control parameter generation model (model parameter is θ Q′ ), according to the next moment state parameter s t+1 Predict the next moment control parameter μ′(s t+1 |θ Q′), and then evaluate the model based on the initial target control parameters (the model parameters are θ μ′ ), according to the next moment state parameter s t+1 And the next moment control parameter a t+1 , generate the target action value y of the control parameter at the next moment t , which can be expressed as the following formula,

[0145] y t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )

[0146] Among them, r t represents the reward value of the control parameter at the current moment, γ represents the weight parameter, and s t+1 represents the state parameter at the next moment, θ μ′ represents the model parameters of the target control parameter evaluation model, θ Q′ represents the model parameters of the target control parameter generation model, μ′(s t+1 |θ Q′ ) represents the generation model based on the target control parameters (the model parameters are θ Q′ ), according to the next moment state parameter s t+1 The control parameters at the next moment are predicted; further, the parameters of the initial control parameter evaluation model are updated by minimizing the mean square error loss function between the action value and the target action value, thereby obtaining an updated control parameter evaluation model, wherein the mean square error loss function between the action value and the target action value can be expressed as the following formula:

[0147]

[0148] Among them, L represents the mean square error loss function between the action value and the target action value, N represents the number of training samples, and y t represents the target action value of the control parameter at the next moment, Q(s t ,a t |θ μ ) represents the action value of the control parameter at the current moment, θ μ Represents the model parameters that control the parameter estimation model.

[0149] 206. Generate a model based on the initial control parameters, generate predicted control parameters according to the current state parameters, and generate predicted action values ​​of the predicted control parameters according to the current state parameters and the predicted control parameters based on the updated control parameter evaluation model.

[0150] 207. Calculate the deep deterministic policy gradient based on the gradient of the predicted action value to the predicted control parameter and the gradient of the predicted control parameter to the parameters of the initial control parameter generation model.

[0151] 208. Based on the deep deterministic policy gradient, the parameters of the initial control parameter generation model are updated to obtain an updated control parameter generation model.

[0152] In steps 206-208 of the embodiment, the parameters of the initial control parameter generation model are updated by a gradient ascent algorithm so that the action value changes in an increasing direction, wherein the deep deterministic policy gradient can be expressed as the following formula:

[0153]

[0154] in, represents the deep deterministic policy gradient, N represents the number of batches drawn from the experience replay pool, represents the gradient of the predicted action value with respect to the predicted control parameter, Represents the gradient of the predicted control parameter to the parameters of the initial control parameter generation model, wherein the predicted control parameter is pre-generated based on the initial control parameter generation model and according to the current state parameters, and the predicted action value of the predicted control parameter is generated based on the updated control parameter evaluation model and according to the current state parameters and the predicted control parameter.

[0155] 209. Copy the parameters of the updated control parameter evaluation model to the initial target control parameter evaluation model to obtain an updated target control parameter evaluation model, so as to generate a target action value based on the updated target control parameter evaluation model.

[0156] 210. Copy the parameters of the updated control parameter generation model to the initial target control parameter generation model to obtain an updated target control parameter generation model, so as to generate the prediction control parameters based on the updated target control parameter generation model.

[0157] It should be noted that the target control parameter evaluation model is equivalent to a delayed version of the control parameter evaluation model, and its parameters are regularly copied from the updated control parameter evaluation model to stabilize the learning process. Specifically, the update can be performed in a soft update manner, which can be expressed as the following formula:

[0158]

[0159] Where τ represents the soft update coefficient.

[0160] Similarly, the target control parameter generation model is also equivalent to a delayed version of the control parameter generation model.

[0161] 211. Repeat the above steps until the preset model training stop condition is met, and obtain the control parameter generation model and the control parameter evaluation model that have completed the model training, so as to generate the control parameters of the voltage regulating device based on the control parameter generation model that has completed the model training.

[0162] Among them, the preset model training stop condition can be a preset number of iterations or a preset error tolerance threshold, which is not specifically limited in the embodiments of the present application.

[0163] In an embodiment of the present application, in order to further define and illustrate, Figure 3 As shown, step 202 of the embodiment obtains a training data set, including:

[0164] 301. Generate a model based on the initial control parameter and generate the current moment control parameter according to the current moment state parameter.

[0165] Among them, the initial control parameter generation model is the current control strategy.

[0166] 302. If the current control parameter satisfies both the renewable energy output constraint and the reactive power processing constraint of the reactive power generating compensation device, the current control parameter is added to the training data set to calculate the reward value of the current control parameter in the training data set.

[0167] Among them, the renewable energy output constraint condition is expressed as the following formula:

[0168]

[0169] Among them, P DG,i Represents the active output of renewable distributed energy, P DG,min Represents the lower limit of the active output of renewable distributed energy, P DG,max represents the upper limit of the active output of renewable distributed energy, Q DG,i represents the reactive power output of renewable distributed energy, Q DG,min represents the lower limit of the reactive power output of renewable distributed energy, Q DG,max It represents the upper limit of the reactive power output of renewable distributed energy;

[0170] The reactive power processing constraint condition of reactive power generating compensation equipment is expressed as the following formula:

[0171]

[0172] Among them, Q SVC,i Indicates the reactive power output of the SVC reactive power compensation device, Q SVC,min Indicates the lower limit of reactive output regulation of SVC reactive compensation equipment, Q SVC,maxIndicates the upper limit of reactive output regulation of SVC reactive compensation equipment, Q CB,i Indicates the reactive output of the capacitor bank, Q CB,min Indicates the lower limit of reactive output regulation of capacitor bank, Q CB,max Indicates the upper limit of reactive output regulation of capacitor bank.

[0173] It should be noted that by performing constraint judgment on the generated current moment control parameters, the closeness of the interaction between the training data and the environment can be effectively improved.

[0174] Preferably, Gaussian noise may be added to the control parameter at the current moment, so as to calculate the control parameter reward value at the current moment according to the control parameter at the current moment after the Gaussian noise is added.

[0175] For example, the variance of the initial Gaussian noise can be set to 3, and the voltage penalty coefficient can be set to 10000 and 100. It should be noted that when selecting actions, by adding Gaussian noise, the agent has a larger exploration range in the early stage of training, and as the number of iterations increases, the variance of the Gaussian noise decays with the increase in the number of iterations. In the later stage of training, the algorithm focuses more on the exploration space, improving the decision-making ability of the algorithm.

[0176] 303. Calculate the control parameter reward value at the current moment.

[0177] Specifically, the current control parameter reward value can be calculated according to the following formula:

[0178]

[0179] Among them, R represents the current control parameter reward value, c loss Represents the weight factor of network loss, P loss represents the grid loss, c u represents the weight factor of node voltage deviation, ΔU represents voltage deviation, c m Represents the weight factor of reactive power margin, Q margin represents reactive power margin, n represents the number of network nodes, U i represents the voltage amplitude of node i, U ref represents the reference voltage, m1 represents the number of capacitor banks, m2 represents the number of SVC reactive compensation devices, Indicates the maximum reactive power output of the capacitor bank, Q CB，i Indicates the reactive output of the capacitor bank. Indicates the maximum reactive power output of the SVC reactive power compensation device, Q SVC，i Indicates the reactive power output of the SVC reactive power compensation device.

[0180] 304. Generate state parameters for the next moment.

[0181] Specifically, the state parameters at the next moment can be generated based on the following formula:

[0182]

[0183] Among them, s t+1 Indicates the state parameters at the next moment, represents the active power output of the photovoltaic node i at the current moment, represents the power factor of the photovoltaic node i at the current moment, represents the reactive power output of the capacitor bank at node i at the current moment, represents the reactive power output of the SVC reactive power compensation device at node i at the current moment, T i,t Indicates the transformer tap position value. represents the active output of the load at node i at the next moment, represents the reactive power output of node i load at the next moment, U i,t+1 Indicates the voltage value of node i at the next moment.

[0184] 305. If the state parameter at the next moment satisfies the power flow constraint condition, the node voltage constraint condition, and the current safety constraint condition at the same time, then the state parameter at the next moment is added to the training data set.

[0185] The power flow constraint condition is expressed as follows:

[0186]

[0187] Among them, P G,i represents the active power of the generator at node i, P DG,i represents the active power of distributed generation at node i, P L,i represents the active power of the load at node i, Q G,i represents the reactive power of the generator at node i, Q DG,i represents the reactive power of distributed generation at node i, Q C,i represents the reactive power of the capacitor bank at node i, Q L,i represents the reactive power of the load at node i, U i represents the voltage amplitude of node i, U j represents the voltage amplitude of node j, G ij and B ij represents the line admittance value between node i and node j, θ ij represents the voltage phase difference between node i and node j, P DG,i represents the active output of renewable distributed energy, Q DG,i Represents the reactive power output of renewable distributed energy, S DG,iRepresents the apparent power of renewable distributed energy;

[0188] The node voltage constraint is expressed as follows:

[0189] U imin ≤U i ≤U imax

[0190] Among them, U i represents the voltage amplitude of node i, U imin Indicates the minimum voltage at node i, U imax represents the maximum value of the voltage at node i;

[0191] The current safety constraint condition is expressed as follows:

[0192] I ij ≤I ij,max

[0193] Among them, I ij represents the current in the branch between node i and node j, I ij,max Represents the maximum allowable current in the branch between node i and node j.

[0194] It should be noted that by performing constraint judgment on the generated next-moment state parameters, the interaction between the training data and the environment can be effectively improved.

[0195] 306. Combine the current state parameter, the current control parameter, the current control parameter reward value, and the next state parameter to generate a training data set, and repeat the above steps to generate multiple training data sets, and combine the multiple training data sets to generate a training data set.

[0196] In the embodiment of the present application, each training data group includes current moment state parameters, current moment control parameters, current moment control parameter reward value, and next moment state parameters; the training data set includes multiple training data groups.

[0197] In a specific application scenario, in order to verify the rationality and effectiveness of the above method, an IEEE33 system with renewable energy generation was built. Based on NREL's real wind and solar data and typical load data, optimization analysis was performed under benchmark load, light load and heavy load conditions, where light load and heavy load are 0.8 times and 1.2 times the standard load, respectively. The reactive compensation equipment includes capacitor banks and SVCs, whose adjustable capacity is 0-300kvar, and the CB capacity of the capacitor bank is 6×25kvar; the rated capacity of wind power generation and photovoltaic systems is 200kW and 150kW, respectively; the adjustment gear of the transformer tap is ±7, a total of 15 gears, and the node voltage range is 0.95-1.05pu. The sampling time of photovoltaic and wind power data is 15 minutes, and the system data set is constructed with 2880 operation scenarios for one month, of which 2800 operation scenarios are training data sets and 80 operation scenarios are test scenarios for deep reinforcement learning training and testing.

[0198] During the model training process, the actor network of each agent is a strategy selection network, and the agent convergence is guided by the strategy reward value; during the agent training process, the learning rate of the Actor network is 0.01, the learning rate of the Critic network is 0.02, the system attenuation factor is 0.9, the number of iterations is 2000, the sampling batch size is 32, the initial Gaussian noise variance is 3, and the voltage penalty coefficients are 10000 and 100 respectively. When selecting actions, Gaussian noise is added to give the agent a larger exploration range in the early stage of training; as the number of iterations increases, the variance of the Gaussian noise decays with the increase in the number of iterations, and the algorithm focuses more on the exploration space in the later stage of training, improving the decision-making ability of the algorithm.

[0199] After the training of reactive power optimization based on deep reinforcement learning, the reactive power optimization results at a certain moment are compared. Figure 4 As shown in Figure 2, a comparative analysis is performed with the particle swarm algorithm and the non-optimization process. Figure 4 It can be seen that after optimization, the system adjusts the active and reactive power of renewable energy generation, the reactive output of reactive compensation equipment and the on-load voltage regulation tap position, so that the system voltage is strictly within the safety range, while reducing the system's active power loss and improving the system's reactive margin. Due to the radial distribution of the original system, the node voltage gradually decreases with the increase of lines, and the voltage drop is particularly obvious at the end of the line. Through the joint control of the active power and reactive power of distributed power sources through reactive optimization, the system voltage level is significantly improved, and the peak-to-valley difference of the voltage is effectively reduced.

[0200] Table 1 shows the changes in voltage deviation, system network loss and reactive margin after reactive optimization calculation in the time section operation scenario. It can be seen that the system has stronger voltage regulation capability and adjustment flexibility. In addition, it has obvious advantages when solving reactive optimization. The single reactive optimization calculation time is only 4.83s, which greatly reduces the solution process and can be applied to online calculation.

[0201] Table 1 Comparison of reactive power and voltage control effects

[0202]

[0203] Reactive power optimization calculation and analysis are performed for light load and heavy load of the system under different penetration rates of new energy. The calculation results of light load and heavy load when the access power of new energy is 1.5MW and 2MW respectively are shown in Table 2. It can be seen that with the increase of the access capacity of new energy, the network loss of the system is significantly reduced. The local consumption of new energy effectively reduces the line current and thus reduces the system loss. At this time, the control equipment action is relatively small, and the reactive power margin is significantly improved. In addition, under heavy load, the system needs more control means to improve the safety of system operation and voltage stability.

[0204] Table 2 Reactive power optimization under different operation scenarios

[0205]

[0206] The present application provides a method for generating control parameters of a voltage regulating device based on deep reinforcement learning, firstly, the state parameters of the target power grid at a preset time are obtained; secondly, a control parameter generation model based on the completed model training is generated, and the control parameters of the voltage regulating device are generated according to the state parameters, and the control parameter generation model is obtained by model training based on the deep deterministic policy gradient algorithm and the interaction state with the target power grid; finally, each voltage regulating device in the target power grid is controlled according to the control parameters of the voltage regulating device. Compared with the prior art, the embodiment of the present application constructs a control parameter generation model based on a reinforcement learning network, simulates the nonlinear relationship between the voltage regulating device and the state of the target power grid, and performs model training based on the deep deterministic policy gradient algorithm and the interaction state with the target power grid, and then generates the control parameters of the voltage regulating device based on the trained model, and controls the voltage regulating device according to the control parameters, so as to optimize the power grid flow distribution, reduce the power grid loss, and realize the normal operation of the power grid under the premise of ensuring the stability of the power grid voltage, and avoid the problems of non-convergence and falling into the local optimal solution that may occur when the control parameters are generated by the traditional analytical method and the heuristic algorithm, and meet the needs of the new power grid when generating the control parameters of the voltage regulating device.

[0207] Furthermore, as a response to the above Figure 1The implementation of the method shown in the embodiment of the present application provides a device for generating control parameters of a voltage regulating device based on deep reinforcement learning, such as Figure 5 As shown, the device comprises:

[0208] State parameter acquisition module 41, control parameter generation module 42, voltage regulation equipment control module 43;

[0209] The state parameter acquisition module 41 is used to obtain the state parameters of the target power grid at a preset time;

[0210] A control parameter generation module 42, which is used to generate a control parameter generation model based on the completed model training, and to generate a voltage regulating device control parameter according to the state parameter, wherein the control parameter generation model is obtained by model training based on a deep deterministic policy gradient algorithm and in combination with the interaction state with the target power grid;

[0211] The voltage regulating device control module 43 is used to control each voltage regulating device in the target power grid according to the voltage regulating device control parameters.

[0212] In a specific application scenario, before the control parameter generation module, the device further includes a model training module, including:

[0213] A model building unit, used to build an initial control parameter generation model, an initial control parameter evaluation model, an initial target control parameter generation model, and an initial target control parameter evaluation model based on a reinforcement learning network;

[0214] A training data set acquisition unit, used to acquire a training data set, wherein the training data set includes a plurality of training data groups, wherein the training data groups include current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters;

[0215] A control parameter evaluation model training unit, configured to generate an action value of the control parameter at the current moment based on the initial control parameter evaluation model, according to the current state parameter and the current control parameter;

[0216] The control parameter evaluation model training unit is further used to generate a target action value of the control parameter at the next moment based on the initial target control parameter evaluation model, according to the next moment state parameter and the next moment control parameter, wherein the next moment control parameter is predicted based on the initial target control parameter generation model according to the next moment state parameter;

[0217] The control parameter evaluation model training unit is further used to update the parameters of the initial control parameter evaluation model based on the mean square error loss function between the action value and the target action value to obtain an updated control parameter evaluation model;

[0218] A control parameter generation model training unit, configured to generate a predicted control parameter based on the initial control parameter generation model and the current state parameter, and to generate a predicted action value of the predicted control parameter based on the updated control parameter evaluation model and the current state parameter and the predicted control parameter;

[0219] The control parameter generation model training unit is further used to calculate the deep deterministic policy gradient according to the gradient of the predicted action value to the predicted control parameter and the gradient of the predicted control parameter to the parameter of the initial control parameter generation model;

[0220] The control parameter generation model training unit is further used to update the parameters of the initial control parameter generation model based on the deep deterministic policy gradient to obtain an updated control parameter generation model;

[0221] A target model updating unit, used for copying the parameters of the updated control parameter evaluation model to the initial target control parameter evaluation model to obtain an updated target control parameter evaluation model, so as to generate a target action value based on the updated target control parameter evaluation model;

[0222] The target model updating unit is further used to copy the parameters of the updated control parameter generation model to the initial target control parameter generation model to obtain an updated target control parameter generation model, so as to generate a prediction control parameter based on the updated target control parameter generation model;

[0223] The model training stopping unit is used to repeat the above steps until the preset model training stopping condition is met, so as to obtain the control parameter generation model and the control parameter evaluation model that have completed the model training, so as to generate the control parameters of the voltage regulating device based on the control parameter generation model that has completed the model training.

[0224] In a specific application scenario, the training data set acquisition unit includes:

[0225] A control parameter generation subunit, configured to generate a current control parameter based on the initial control parameter generation model and according to the current state parameter;

[0226] The reward value calculation subunit is used to calculate the current control parameter reward value according to the following formula:

[0227]

[0228] Among them, R represents the current control parameter reward value, c loss Represents the weight factor of network loss, P loss represents the grid loss, c u represents the weight factor of node voltage deviation, ΔU represents voltage deviation, c m Represents the weight factor of reactive power margin, Q margin represents reactive power margin, n represents the number of network nodes, U i represents the voltage amplitude of node i, U ref represents the reference voltage, m1 represents the number of capacitor banks, m2 represents the number of SVC reactive compensation devices, Indicates the maximum reactive power output of the capacitor bank, Q CB，i Indicates the reactive output of the capacitor bank. Indicates the maximum reactive power output of the SVC reactive power compensation device, Q SVC，i Indicates the reactive power output of the SVC reactive power compensation device;

[0229] The next moment state parameter generation subunit is used to generate the next moment state parameter based on the following formula:

[0230]

[0231] Among them, s t+1 Indicates the state parameters at the next moment, represents the active power output of the photovoltaic node i at the current moment, represents the power factor of the photovoltaic node i at the current moment, represents the reactive power output of the capacitor bank at node i at the current moment, represents the reactive power output of the SVC reactive power compensation device at node i at the current moment, T i,t Indicates the transformer tap position value. represents the active output of the load at node i at the next moment, represents the reactive power output of node i load at the next moment, U i,t+1 Indicates the voltage value of node i at the next moment;

[0232] A training data group generating subunit, used for combining the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data group;

[0233] The training data group generating subunit is further used to repeat the above steps to generate multiple training data groups;

[0234] The training data set generating subunit is used to combine a plurality of the training data sets to generate a training data set.

[0235] In a specific application scenario, before the reward value calculation subunit, the training data set acquisition unit further includes a control parameter determination subunit, which is used to:

[0236] The renewable energy output constraint is expressed as follows:

[0237]

[0238] Among them, P DG,i Represents the active output of renewable distributed energy, P DG,min Represents the lower limit of the active output of renewable distributed energy, P DG,max represents the upper limit of the active output of renewable distributed energy, Q DG,i represents the reactive power output of renewable distributed energy, Q DG,min represents the lower limit of the reactive power output of renewable distributed energy, Q DG,max It represents the upper limit of the reactive power output of renewable distributed energy;

[0239] The reactive power processing constraint condition of reactive power generating compensation equipment is expressed as the following formula:

[0240]

[0241] Among them, Q SVC,i Indicates the reactive power output of the SVC reactive power compensation device, Q SVC,min Indicates the lower limit of reactive output regulation of SVC reactive compensation equipment, Q SVC,max Indicates the upper limit of reactive output regulation of SVC reactive compensation equipment, Q CB,i Indicates the reactive output of the capacitor bank, Q CB,min Indicates the lower limit of reactive output regulation of capacitor bank, Q CB,max Indicates the upper limit of reactive output regulation of capacitor bank;

[0242] If the current control parameter satisfies both the renewable energy output constraint and the reactive power processing constraint of the reactive power generating compensation equipment, the current control parameter is added to the training data set to calculate the reward value of the current control parameter in the training data set.

[0243] In a specific application scenario, before the training data set generation subunit, the training data set acquisition unit further includes a next moment state parameter determination subunit, which is used to:

[0244] The power flow constraint condition is expressed as follows:

[0245]

[0246] Among them, P G,irepresents the active power of the generator at node i, P DG,i represents the active power of distributed generation at node i, P L,i represents the active power of the load at node i, Q G,i represents the reactive power of the generator at node i, Q DG,i represents the reactive power of distributed generation at node i, Q C,i represents the reactive power of the capacitor bank at node i, Q L,i represents the reactive power of the load at node i, U i represents the voltage amplitude of node i, U j represents the voltage amplitude of node j, G ij and B ij represents the line admittance value between node i and node j, θ ij represents the voltage phase difference between node i and node j, P DG,i represents the active output of renewable distributed energy, Q DG,i Represents the reactive power output of renewable distributed energy, S DG,i Represents the apparent power of renewable distributed energy;

[0247] The node voltage constraint is expressed as follows:

[0248] U imin ≤U i ≤U imax

[0249] Among them, U i represents the voltage amplitude of node i, U imin Indicates the minimum voltage at node i, U imax represents the maximum value of the voltage at node i;

[0250] The current safety constraint condition is expressed as follows:

[0251] I ij ≤I ij,max

[0252] Among them, I ij represents the current in the branch between node i and node j, I ij,max represents the maximum allowable current in the branch between node i and node j;

[0253] If the state parameter at the next moment satisfies the power flow constraint condition, the node voltage constraint condition, and the current safety constraint condition at the same time, the state parameter at the next moment is added to the training data set.

[0254] In a specific application scenario, before the reward value calculation subunit, the training data set acquisition unit further includes a noise adding subunit, which is used to:

[0255] Gaussian noise is added to the current moment control parameter to calculate the current moment control parameter reward value according to the current moment control parameter after the Gaussian noise is added.

[0256] In a specific application scenario, the state parameters include: the active output and power factor of the distributed power source, the reactive output of the SVC reactive compensation device, the reactive output of the capacitor bank, and the transformer tap position value.

[0257] The present application provides a device for generating control parameters of a voltage regulating device based on deep reinforcement learning, firstly, the state parameters of the target power grid at a preset time are obtained; secondly, a control parameter generation model based on the completed model training is generated, and the control parameters of the voltage regulating device are generated according to the state parameters, and the control parameter generation model is obtained by model training based on the deep deterministic policy gradient algorithm and the interaction state with the target power grid; finally, each voltage regulating device in the target power grid is controlled according to the control parameters of the voltage regulating device. Compared with the prior art, the embodiment of the present application constructs a control parameter generation model based on a reinforcement learning network, simulates the nonlinear relationship between the voltage regulating device and the state of the target power grid, and performs model training based on the deep deterministic policy gradient algorithm and the interaction state with the target power grid, and then generates the control parameters of the voltage regulating device based on the trained model, and controls the voltage regulating device according to the control parameters, so as to optimize the power grid flow distribution, reduce the power grid loss, and realize the normal operation of the power grid under the premise of ensuring the stability of the power grid voltage, and avoid the problems of non-convergence and falling into the local optimal solution that may occur when the control parameters are generated by the traditional analytical method and the heuristic algorithm, and meet the needs of the new power grid when generating the control parameters of the voltage regulating device.

[0258] According to one embodiment of the present application, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the method for generating control parameters of a voltage regulating device based on deep reinforcement learning in any of the above method embodiments.

[0259] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0260] Figure 6 A schematic diagram of the structure of a terminal provided according to an embodiment of the present application is shown, and the specific embodiment of the present application does not limit the specific implementation of the terminal.

[0261] like Figure 6As shown, the terminal may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0262] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0263] The communication interface 504 is used to communicate with other devices such as clients or other servers.

[0264] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above-mentioned embodiment of the method for generating control parameters of a voltage regulating device based on deep reinforcement learning.

[0265] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0266] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0267] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0268] The program 510 may be specifically configured to enable the processor 502 to perform the following operations:

[0269] Obtaining state parameters of the target power grid at a preset time;

[0270] Based on the control parameter generation model of the completed model training, the voltage regulating device control parameters are generated according to the state parameters, wherein the control parameter generation model is constructed based on a reinforcement learning network and is obtained by model training based on a deep deterministic policy gradient algorithm and combined with the interaction state between the target power grid;

[0271] Each voltage regulating device in the target power grid is controlled according to the voltage regulating device control parameter.

[0272] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device of the method for generating control parameters of the voltage regulating device based on deep reinforcement learning, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between components inside the storage medium, and communication with other hardware and software in the information processing physical device.

[0273] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0274] The method and system of the present application may be implemented in many ways. For example, the method and system of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present application may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers a recording medium storing a program for executing the method according to the present application.

[0275] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0276] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating control parameters of a voltage regulating device based on deep reinforcement learning, characterized in that: include: Obtaining state parameters of the target power grid at a preset time; Based on the control parameter generation model of the completed model training, the voltage regulating device control parameters are generated according to the state parameters, wherein the control parameter generation model is constructed based on a reinforcement learning network and is obtained by model training based on a deep deterministic policy gradient algorithm and combined with the interaction state between the target power grid; Control each voltage regulating device in the target power grid according to the voltage regulating device control parameter; Before generating a control parameter model based on the trained model and generating a voltage regulating device control parameter according to the state parameter, the method further includes: Construct the initial control parameter generation model, initial control parameter evaluation model, initial target control parameter generation model, and initial target control parameter evaluation model based on the reinforcement learning network; Acquire a training data set, wherein the training data set includes multiple training data groups, and the training data groups include current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters; Based on the initial control parameter evaluation model, generating an action value of the control parameter at the current moment according to the current state parameter and the current control parameter; Based on the initial target control parameter evaluation model, the target action value of the control parameter at the next moment is generated according to the next moment state parameter and the next moment control parameter, wherein the next moment control parameter is predicted based on the initial target control parameter generation model according to the next moment state parameter; updating the parameters of the initial control parameter evaluation model based on a mean square error loss function between the action value and the target action value to obtain an updated control parameter evaluation model; Based on the initial control parameter generation model, a predicted control parameter is generated according to the current state parameter, and based on the updated control parameter evaluation model, a predicted action value of the predicted control parameter is generated according to the current state parameter and the predicted control parameter; Calculating a deep deterministic policy gradient based on a gradient of the predicted action value to the predicted control parameter and a gradient of the predicted control parameter to a parameter of the initial control parameter generation model; Updating the parameters of the initial control parameter generation model based on the deep deterministic policy gradient to obtain an updated control parameter generation model; Copying the parameters of the updated control parameter evaluation model to the initial target control parameter evaluation model to obtain an updated target control parameter evaluation model, so as to generate a target action value based on the updated target control parameter evaluation model; Copying the parameters of the updated control parameter generation model to the initial target control parameter generation model to obtain an updated target control parameter generation model, so as to generate a prediction control parameter based on the updated target control parameter generation model; Repeat the above steps until the preset model training stop condition is met, and obtain a control parameter generation model and a control parameter evaluation model that have completed model training, so as to generate control parameters of the voltage regulating device based on the control parameter generation model that has completed model training; The step of obtaining a training data set includes: Based on the initial control parameter generation model, the current moment control parameters are generated according to the current moment state parameters; Calculate the current control parameter reward value according to the following formula: Among them, R represents the current control parameter reward value, c loss Represents the weight factor of network loss, P loss represents the grid loss, c u represents the weight factor of node voltage deviation, ΔU represents voltage deviation, c m Represents the weight factor of reactive power margin, Q margin represents reactive power margin, n represents the number of network nodes, U i represents the voltage amplitude of node i, U ref represents the reference voltage, m1 represents the number of capacitor banks, m2 represents the number of SVC reactive compensation devices, Indicates the maximum reactive power output of the capacitor bank, Q CB，i Indicates the reactive output of the capacitor bank. Indicates the maximum reactive power output of the SVC reactive power compensation device, Q SVC，i Indicates the reactive power output of the SVC reactive power compensation device; The next moment state parameters are generated based on the following formula: Among them, s t+1 Indicates the state parameters at the next moment, represents the active power output of the photovoltaic node i at the current moment, represents the power factor of the photovoltaic node i at the current moment, represents the reactive power output of the capacitor bank at node i at the current moment, represents the reactive power output of the SVC reactive power compensation device at node i at the current moment, T i,t Indicates the transformer tap position value. represents the active output of the load at node i at the next moment, represents the reactive power output of node i load at the next moment, U i,t+1 Indicates the voltage value of node i at the next moment; Combine the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data set; Repeat the above steps to generate multiple sets of training data sets; A plurality of the training data sets are combined to generate a training data set.

2. The method according to claim 1, characterized in that Before calculating the current moment control parameter reward value, the method further includes: The renewable energy output constraint is expressed as follows: Among them, P DG,i Represents the active output of renewable distributed energy, P DG,min Represents the lower limit of the active output of renewable distributed energy, P DG,max represents the upper limit of the active output of renewable distributed energy, Q DG,i represents the reactive power output of renewable distributed energy, Q DG,min represents the lower limit of the reactive power output of renewable distributed energy, Q DG,max It represents the upper limit of the reactive power output of renewable distributed energy; The reactive power output constraint condition of reactive power compensation equipment is expressed as follows: Among them, Q SVC,i Indicates the reactive power output of the SVC reactive power compensation device, Q SVC,min Indicates the lower limit of reactive output regulation of SVC reactive compensation equipment, Q SVC,max Indicates the upper limit of reactive output regulation of SVC reactive compensation equipment, Q CB,i Indicates the reactive output of the capacitor bank, Q CB,min Indicates the lower limit of reactive output regulation of capacitor bank, Q CB,max Indicates the upper limit of reactive output regulation of capacitor bank; If the current control parameter satisfies both the renewable energy output constraint and the reactive output constraint of the reactive power generating compensation device, the current control parameter is added to the training data set to calculate the reward value of the current control parameter in the training data set.

3. The method according to claim 1, characterized in that Before combining the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data set, the method further includes: The power flow constraint condition is expressed as follows: Among them, P G,i represents the active power of the generator at node i, P DG,i represents the active power of distributed generation at node i, P L,i represents the active power of the load at node i, Q G,i represents the reactive power of the generator at node i, Q DG,i represents the reactive power of distributed generation at node i, Q C,i represents the reactive power of the capacitor bank at node i, Q L,i represents the reactive power of the load at node i, U i represents the voltage amplitude of node i, U j represents the voltage amplitude of node j, G ij and B ij represents the line admittance value between node i and node j, θ ij represents the voltage phase difference between node i and node j, P DG,i represents the active output of renewable distributed energy, Q DG,i Represents the reactive power output of renewable distributed energy, S DG,i Represents the apparent power of renewable distributed energy; The node voltage constraint is expressed as follows: IN imin ≤U i ≤U imax Among them, U i represents the voltage amplitude of node i, U imin Indicates the minimum voltage at node i, U imax represents the maximum value of the voltage at node i; The current safety constraint condition is expressed as follows: I ij ≤I ij,max Among them, I ij represents the current in the branch between node i and node j, I ij,max represents the maximum allowable current in the branch between node i and node j; If the state parameter at the next moment satisfies the power flow constraint condition, the node voltage constraint condition, and the current safety constraint condition at the same time, the state parameter at the next moment is added to the training data set.

4. The method according to claim 1, characterized in that: Before calculating the current moment control parameter reward value, the method further includes: Gaussian noise is added to the current moment control parameter to calculate the current moment control parameter reward value according to the current moment control parameter after the Gaussian noise is added.

5. The method according to claim 1, characterized in that The state parameters include: the active output and power factor of the distributed power source, the reactive output of the SVC reactive compensation device, the reactive output of the capacitor bank, and the transformer tap position value.

6. A device for generating control parameters of a voltage regulating device based on deep reinforcement learning, characterized in that: include: A state parameter acquisition module is used to obtain the state parameters of the target power grid at a preset time; A control parameter generation module, which is used to generate a control parameter generation model based on a completed model training, and to generate a voltage regulating device control parameter according to the state parameter, wherein the control parameter generation model is obtained by model training based on a deep deterministic policy gradient algorithm and in combination with the interaction state with the target power grid; A voltage regulating device control module, used to control each voltage regulating device in the target power grid according to the voltage regulating device control parameters; Before the control parameter generation module, the device further includes a model training module, including: A model building unit, used to build an initial control parameter generation model, an initial control parameter evaluation model, an initial target control parameter generation model, and an initial target control parameter evaluation model based on a reinforcement learning network; A training data set acquisition unit, used to acquire a training data set, wherein the training data set includes a plurality of training data groups, wherein the training data groups include current moment state parameters, current moment control parameters, current moment control parameter reward values, and next moment state parameters; A control parameter evaluation model training unit, configured to generate an action value of the control parameter at the current moment based on the initial control parameter evaluation model, according to the current state parameter and the current control parameter; The control parameter evaluation model training unit is further used to generate a target action value of the control parameter at the next moment based on the initial target control parameter evaluation model, according to the next moment state parameter and the next moment control parameter, wherein the next moment control parameter is predicted based on the initial target control parameter generation model according to the next moment state parameter; The control parameter evaluation model training unit is further used to update the parameters of the initial control parameter evaluation model based on the mean square error loss function between the action value and the target action value to obtain an updated control parameter evaluation model; A control parameter generation model training unit, configured to generate a predicted control parameter based on the initial control parameter generation model and the current state parameter, and to generate a predicted action value of the predicted control parameter based on the updated control parameter evaluation model and the current state parameter and the predicted control parameter; The control parameter generation model training unit is further used to calculate the deep deterministic policy gradient according to the gradient of the predicted action value to the predicted control parameter and the gradient of the predicted control parameter to the parameter of the initial control parameter generation model; The control parameter generation model training unit is further used to update the parameters of the initial control parameter generation model based on the deep deterministic policy gradient to obtain an updated control parameter generation model; A target model updating unit, used for copying the parameters of the updated control parameter evaluation model to the initial target control parameter evaluation model to obtain an updated target control parameter evaluation model, so as to generate a target action value based on the updated target control parameter evaluation model; The target model updating unit is further used to copy the parameters of the updated control parameter generation model to the initial target control parameter generation model to obtain an updated target control parameter generation model, so as to generate a prediction control parameter based on the updated target control parameter generation model; A model training stopping unit is used to repeat the above steps until a preset model training stopping condition is met, and obtain a control parameter generation model and a control parameter evaluation model that have completed model training, so as to generate control parameters of the voltage regulating device based on the control parameter generation model that has completed model training; The training data set acquisition unit comprises: A control parameter generation subunit, configured to generate a current control parameter based on the initial control parameter generation model and according to the current state parameter; The reward value calculation subunit is used to calculate the current control parameter reward value according to the following formula: Among them, R represents the current control parameter reward value, c loss Represents the weight factor of network loss, P loss represents the grid loss, c u represents the weight factor of node voltage deviation, ΔU represents voltage deviation, c m Represents the weight factor of reactive power margin, Q margin represents reactive power margin, n represents the number of network nodes, U i represents the voltage amplitude of node i, U ref represents the reference voltage, m1 represents the number of capacitor banks, m2 represents the number of SVC reactive compensation devices, Indicates the maximum reactive power output of the capacitor bank, Q CB，i Indicates the reactive output of the capacitor bank. Indicates the maximum reactive power output of the SVC reactive power compensation device, Q SVC，i Indicates the reactive power output of the SVC reactive power compensation device; The next moment state parameter generation subunit is used to generate the next moment state parameter based on the following formula: Among them, s t+1 Indicates the state parameters at the next moment, represents the active power output of the photovoltaic node i at the current moment, represents the power factor of the photovoltaic node i at the current moment, represents the reactive power output of the capacitor bank at node i at the current moment, represents the reactive power output of the SVC reactive power compensation device at node i at the current moment, T i,t Indicates the transformer tap position value. represents the active output of the load at node i at the next moment, represents the reactive power output of node i load at the next moment, U i,t+1 Indicates the voltage value of node i at the next moment; A training data group generating subunit, used for combining the current moment state parameter, the current moment control parameter, the current moment control parameter reward value, and the next moment state parameter to generate a training data group; The training data group generating subunit is further used to repeat the above steps to generate multiple training data groups; The training data set generating subunit is used to combine a plurality of the training data sets to generate a training data set.

7. A storage medium, wherein at least one executable instruction is stored in the storage medium, characterized in that: The executable instructions enable the processor to perform operations corresponding to the method for generating control parameters of a voltage regulating device based on deep reinforcement learning as described in any one of claims 1-5.

8. A terminal, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, characterized in that the executable instruction enables the processor to perform operations corresponding to the method for generating control parameters of a voltage regulating device based on deep reinforcement learning as described in any one of claims 1-5.

Images