A method for coordinated voltage control in AC / DC distribution networks

Abstract

The present invention provides a method for coordinated voltage control of AC / DC distribution network partitions, including establishing an AC / DC distribution network voltage control model; partitioning the AC / DC distribution network; modeling the voltage control problem after partitioning as a state-adversarial Markov decision process; solving the state-adversarial Markov decision process; interactively training an attacking intelligent agent and a defensive intelligent agent; and deploying the trained defensive intelligent agent online. The present invention solves the problem of coordinated control of multiple devices in an AC / DC distribution network, improves the coordination of multiple controllable devices, and at the same time considers the uncertainty of the observed state, improves the robustness of the control method, and achieves a better voltage control effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of distribution network voltage control, and in particular relates to a partition coordinated voltage control method for an AC / DC distribution network. Background Art

[0002] Under the background of "carbon peak and carbon neutrality", distributed photovoltaics are increasingly connected to AC and DC distribution networks, which has changed the power flow distribution of AC and DC distribution networks and brought about problems such as voltage over-limit. At the same time, the controllable devices in the AC and DC distribution network include voltage source converters (VSCs), photovoltaic inverters, energy storage, on-load voltage regulators (OLTCs), capacitor banks (CBs), etc. When regulating the voltage, it is necessary to consider the coordination between multiple devices at the same time to achieve the optimal voltage regulation effect. VSC, photovoltaic inverters and energy storage devices are power electronic devices with fast response speeds and can be adjusted continuously. However, OLTC and CB are mechanical devices with slow response speeds and can only be adjusted discretely. The coordinated control of the two types of equipment constitutes a multi-time scale optimization problem. In addition, when voltage control is actually applied, due to the influence of factors such as sensor errors and communication delays, the observed state deviates from the actual state, which interferes with the voltage control decision and cannot achieve the optimal voltage control effect. In view of the above problems, the present invention proposes a partitioned coordinated voltage control method for AC and DC distribution networks to achieve coordinated control of multiple controllable devices when there are errors in the observed state. Summary of the invention

[0003] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a partitioned collaborative voltage control method for AC / DC distribution networks to solve the problem of collaborative control of multiple devices in AC / DC distribution networks, enhance the coordination of multiple controllable devices, and at the same time consider the uncertainty of the observed state, improve the robustness of the control method, and achieve better voltage control effect.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0005] A method for coordinated voltage control in AC / DC distribution network zones, comprising the following steps:

[0006] Step 1: Establish a voltage control model for AC and DC distribution networks;

[0007] Step 2: Partition the AC and DC distribution network;

[0008] Step 3: Model the partitioned voltage control problem as a state-adversarial Markov decision process;

[0009] Step 4: Solve the state-adversarial Markov decision process;

[0010] Step 5: The attacking agent and the defending agent conduct interactive training;

[0011] Step 6: Deploy the trained defense agent online.

[0012] Furthermore, in step 1, the voltage control model includes a branch power flow model and a voltage source converter model, and the objective function of the voltage control model is to minimize the voltage deviation:

[0013] ;

[0014] In the formula, is the objective function, where Indicates the time, express The moment time interval, represents the sum of the voltage deviations of the AC side nodes, represents the sum of the DC side node voltage deviations, , , , , , are the control variables, which are the PV reactive power, energy storage charging and discharging power, VSC active power, VSC reactive power, number of capacitor banks put into operation and on-load tap-changing transformer gear position;

[0015] The constraints of the voltage control model include system power balance constraints, voltage upper and lower limit constraints, photovoltaic inverter power constraints, energy storage power constraints, and voltage source converter power constraints.

[0016] Furthermore, in step 2, the method for partitioning the AC / DC distribution network includes:

[0017] The objective function for distribution network partitioning is constructed as follows:

[0018] ;

[0019] ;

[0020] In the formula, , , is the weight, ISCI is the aggregation index within the sub-region, ISSI is the sparseness index between sub-regions, SCI is the number of sub-regions, ISRCI is the sub-region regulation capacity index, and PCI is the physical connectivity index;

[0021] ;

[0022] is the set of internal node numbers of the sub-region to which node i belongs, is the electrical distance between two points on the AC side;

[0023] ;

[0024] , ;

[0025] In the formula, is the actual number of partitions; is the ideal number of partitions; is the penalty parameter, which is used to control the width of the function;

[0026] ;

[0027] ;

[0028] The improved particle swarm algorithm is used to solve the partitioning problem.

[0029] Furthermore, the method of using the improved particle swarm algorithm to solve the partitioning problem includes:

[0030] Define the signal function:

[0031] ,

[0032] ,

[0033] Based on the signal function, a particle velocity update strategy of discrete particle swarm method is proposed:

[0034] ;

[0035] In the formula, the subscript Representative particles; the subscript k represents the node number; is the number of iterations; v is the velocity of the particle; x is the current position of the particle; is the individual optimal value; is the optimal value for the group; is the inertia weight; and is the learning factor; and is a random number in the interval [0,1]; Represents the exclusive-or operation.

[0036] Except for the boundary nodes of the region, the adjacent nodes of the remaining nodes all belong to the same sub-region. Based on this feature, a particle position update method is proposed:

[0037] ;

[0038] In the formula, is the set of nodes connected to node k; is the number of the sub-region to which any node selected randomly belongs;

[0039] The particle swarm algorithm is used to partition the AC distribution network and the DC distribution network respectively to obtain AC sub-areas and DC sub-areas.

[0040] Furthermore, in step 3, modeling the partitioned voltage control problem as a state-adversarial Markov decision process includes: defense process modeling and attack process modeling; defense process modeling includes: modeling the voltage control problem under long time scales and modeling the voltage control problem under short time scales.

[0041] Furthermore, in step 4, solving the state-adversarial Markov decision process includes:

[0042] The BDQ algorithm is used to solve the long-time scale voltage control problem;

[0043] The MASAC algorithm is used to solve the short-time-scale voltage control problem;

[0044] The SAC algorithm is used to solve the state attack problem.

[0045] Furthermore, in step 5, by constructing a training framework of synchronous training and asynchronous learning, the defense agent and the attack agent perform alternating adversarial training. step, fix the parameters of the attacking agent and train the defending agent; after each round Step 1, fix the parameters of the defense agent, train the attack agent, and repeat this process. Second, the synchronous training and asynchronous learning of offensive and defensive agents are realized.

[0046] Furthermore, in step 5, the defense agent training includes:

[0047] At the beginning of training, the parameters of the long-time-scale agent Q network, the parameters of the short-time-scale agent Actor network, and the parameters of the Critic network are randomly initialized. No perturbation is applied in the first round of training. The long-time scale agent and the short-time scale agent are updated in rounds. In each round, a random Moment At each time t, the long-time scale agent first obtains the global state, inputs it into the Q network to generate dispatching instructions for the on-load tap-changing transformer and capacitor bank, and transmits the dispatching instructions to the short-time scale agent, which obtains the local observation state of the sub-region.

[0048] In the initial stage of training, short-time-scale agents randomly generate scheduling instructions and interact with the environment to obtain rewards. At the same time, the environment enters the next state, and each agent obtains a new state observation and stores the interaction data with the environment in the experience pool. When the stored experience data reaches the set amount, each agent begins to generate scheduling instructions based on the Actor network and randomly extracts from the experience pool at each time interval. Group data, used for updating agents on short time scales; is a pre-set hyperparameter;

[0049] When time t ends, the long-time scale agent obtains a reward value and stores the data of interaction with the environment in the experience pool. When the data stored in the experience pool reaches a certain amount, a part of the data is randomly extracted from the experience pool at each moment to update the network parameters of the long-time scale agent.

[0050] In the second and subsequent rounds of training, the perturbation of the attacking agent to the state is added, the parameters of the attacking agent are fixed, and the time interval is At this time, the attacking agent makes perturbations according to the observed state, perturbs the observed state of the defending agent, the long-time scale agent makes decisions, the short-time scale agent makes decisions, and obtains rewards after interacting with the environment. At the same time, the attacking agent obtains corresponding rewards. In this process, the experience data of the long-time scale agent, the experience data of the short-time scale agent, and the experience data of the attacking agent are stored in their respective experience pools.

[0051] Furthermore, in step 5, the attack agent training includes:

[0052] At the beginning, the parameters of the attack agent neural network are randomly initialized, the parameters of the long-time scale agent and the short-time scale agent neural network are fixed, and then the cycle rounds to train the attacking agent, The moment At each time interval, the attack agent obtains the global state. In the initial stage of training, the attack agent randomly generates attack actions and stores the interaction data with the environment in the experience pool for updating network parameters. When the stored experience data reaches the set amount, the attack agent begins to generate attack actions according to the Actor network. The long-time scale agent and the short-time scale agent make scheduling actions based on the state after the disturbance, calculate the voltage of the AC / DC distribution network through power flow calculation, and calculate the reward value. Then the system enters the next state, the attack agent obtains new state observations, and stores the experience data of the interaction with the environment in the experience pool. At the same time, the long-time scale agent and the short-time scale agent also store the experience data in their respective experience pools. At each time interval, a portion of data is randomly extracted from the experience pool for updating the attack agent.

[0053] Furthermore, in step 6, after the alternating training is completed, only the Q network of the long-time scale agent and the Actor network of the short-time scale agent are retained, and the long-time scale agent and the short-time scale agent are deployed online to schedule the controllable devices.

[0054] Compared with the prior art, the AC / DC distribution network partition coordinated voltage control method described in the present invention has the following advantages:

[0055] The present invention establishes a voltage control method framework with long-short dual time scale coordination, which performs hour-level centralized control on slow-responding mechanical equipment in the long time scale and minute-level distributed control on fast-responding power electronic equipment in the short time scale. On this basis, in order to enhance the robustness of the control method, the deviation between the actual state and the observed state is modeled as the disturbance of the observed state by the attacking intelligent agent, and the SA-MDP model of robust voltage control is established;

[0056] The present invention proposes a solution framework for synchronous training and asynchronous learning of a defense agent and an attack agent, solves the defense agent based on the BDQ and MASAC algorithms, and solves the attack agent based on the SAC algorithm; the present invention can effectively coordinate a variety of controllable devices, has good robustness, and can effectively reduce voltage deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0058] Figure 1 A schematic diagram of a flow chart of a method for coordinated voltage control in AC and DC distribution networks according to the present invention;

[0059] Figure 2It is a schematic diagram of a voltage source converter model of the present invention;

[0060] Figure 3 A schematic diagram of a state-adversarial Markov decision process for multi-time-scale robust voltage control of the present invention;

[0061] Figure 4 It is a schematic diagram of the structure of the Q network of the BDQ algorithm of the present invention;

[0062] Figure 5 A schematic diagram of the alternate training method of the defense agent and the attack agent of the present invention;

[0063] Figure 6 Schematic diagram of the training process of the defense agent of the present invention;

[0064] Figure 7 Schematic diagram of the attack agent training process of the present invention. DETAILED DESCRIPTION

[0065] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0066] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0067] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0068] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0069] like Figure 1 As shown, the present invention provides a method for coordinated voltage control of AC / DC distribution network partitions, comprising the following steps:

[0070] Step 1: Establish a voltage control model for AC and DC distribution networks;

[0071] (1) Based on the voltage requirements of the power grid, the on-load tap changer (OLTC), capacitor bank (CB), photovoltaic inverter, voltage source converter and energy storage are controlled. The objective function of the voltage control model is to minimize the voltage deviation:

[0072] (1)

[0073] In the formula, is the objective function, where Indicates the time, express The moment time interval. represents the sum of the voltage deviations of the AC side nodes, represents the sum of the DC side node voltage deviations, , , , , , are the control variables, which are the PV reactive output, energy storage charging and discharging power, VSC active power, VSC reactive power, CB input group number and OLTC gear position.

[0074] (2) The voltage control model includes the branch flow model and the voltage source converter model.

[0075] The branch power flow model is:

[0076] (2)

[0077] (3)

[0078] (4)

[0079] In the formula, and is the injected active power and reactive power at node i, is the voltage amplitude at node i, and are the conductance and susceptance of the AC branch ij, is the voltage phase difference between nodes i and j.

[0080] The voltage source converter model is as follows Figure 2 As shown, and are the voltage amplitude and phase angle at the connection between VSC and AC system, respectively; and are the active power and reactive power flowing into the VSC from the AC bus, respectively; and are the equivalent resistance and reactance reflecting the converter losses respectively; is the voltage amplitude and phase angle at the VSC AC outlet; is the active power flowing out of the VSC DC side; is the voltage amplitude on the DC side of the VSC, which can be obtained by equation (5).

[0081] (5)

[0082] In the formula, is the DC voltage utilization rate, is the modulation ratio.

[0083] The injected power at the AC side bus can be expressed as:

[0084] (6)

[0085] In the formula, for and The phase angle difference, .

[0086] The injected power at C on the AC side of the converter can be expressed as:

[0087] (7)

[0088] The active power injected into the DC side of the VSC can be expressed as:

[0089] (8)

[0090] In the formula, is the active power loss of VSC, and is related to the AC side current It conforms to the quadratic function relationship and can be calculated by formula (9):

[0091] (9)

[0092] (10)

[0093] In the formula, , , Represents the loss factor of the converter.

[0094] (3) The constraints of the voltage control model include system power balance constraints, voltage upper and lower limit constraints, photovoltaic inverter power constraints, energy storage power constraints, and voltage source converter power constraints.

[0095] The system power balance constraint is:

[0096] (11)

[0097] (12)

[0098] (13)

[0099] In the formula, , They are The moment The node injection power within a time interval; , , , They are The moment AC load, DC load, PV active output, and VSC power loss within a time interval. is the reactive power of the AC load, is the capacity of the capacitor bank.

[0100] The voltage upper and lower limits are constrained as follows:

[0101] (14)

[0102] (15)

[0103] In the formula, and are the upper and lower limits of the AC node voltage respectively, and are the upper and lower limits of the DC node voltage respectively.

[0104] The PV inverter power constraint is:

[0105] (16)

[0106] In the formula, is the maximum reactive output power of the inverter; is the inverter capacity, which is generally set to 1.0~1.1 times the active capacity; is the active power of photovoltaic.

[0107] The energy storage power constraint is:

[0108] (17)

[0109] (18)

[0110] In the formula, is the time t The energy storage charging and discharging power in a time interval, less than 0 indicates charging, and greater than 0 indicates discharging; and are the maximum charging power and maximum discharging power of energy storage respectively; is the time t The state of charge of the energy stored in each time interval; and are the maximum and minimum values ​​of the energy storage charge state, respectively.

[0111] After the energy storage is charged and discharged, the new state of charge can be calculated by the following formula.

[0112] (19)

[0113] In the formula, is the rated capacity of energy storage, is the time step.

[0114] The voltage source converter power constraint is:

[0115] (20)

[0116] (twenty one)

[0117] In the formula, is the active power of VSC, is the reactive power of VSC, is the capacity of the VSC, It is the maximum reactive power output of VSC.

[0118] Step 2: Partition the AC and DC distribution network;

[0119] In power flow calculation, based on the Jacobian matrix, the power flow calculation can be expressed as:

[0120] (twenty two)

[0121] In the formula, and are the changes of active power and reactive power injected into the node respectively; , , , are the elements of the Jacobian matrix; and are the voltage phase angle and voltage amplitude changes respectively.

[0122] From this we can deduce:

[0123] (twenty three)

[0124] In the formula, and They represent the sensitivity of active power, reactive power and node voltage phase angle respectively; and They represent the sensitivity of active power and reactive power to voltage amplitude respectively. Therefore, the change of node voltage amplitude can be expressed as:

[0125] (twenty four)

[0126] The active-voltage influence matrix and the reactive-voltage influence matrix can be calculated according to the following formula.

[0127] (25)

[0128] (26)

[0129] In the formula, It represents the active power-voltage influence capability of node j on node i; It represents the reactive power-voltage influence capability of node j on node i; and are the elements of the jth row and jth column and the elements of the ith row and jth column of the active voltage sensitivity matrix respectively; and are the j-th row and j-th column element and the i-th row and j-th column element of the reactive voltage sensitivity matrix respectively; and They are the voltage change of node j and the voltage change of node i when the active power injected into node j changes respectively; and They are the voltage change of node j and the voltage change of node i when reactive power changes are injected into node j respectively.

[0130] Define the active electrical distance between two nodes and reactive electrical distance for:

[0131] (27)

[0132] (28)

[0133] Since the converter can control the active power and reactive power and realize the decoupling of the AC distribution network and the DC distribution network, the electrical distance between the AC side and the DC side can be defined and partitioned separately.

[0134] For the AC side, it is necessary to consider the impact of both active power and reactive power on voltage. Define the electrical distance between two points on the AC side for:

[0135] (29)

[0136] For the DC side, only the effect of active power on voltage needs to be considered, and the electrical distance between two points on the DC side is defined. for:

[0137] (30)

[0138] The present invention proposes the following partition index system:

[0139] Intra-Subnetwork Cohesiveness Index (ISCI).

[0140] (31)

[0141] In the formula, is the set of internal node numbers of the sub-region to which node i belongs.

[0142] Inter-Subnetworks Sparsity Index (ISSI).

[0143] (32)

[0144] Subnetwork Count Index (SCI).

[0145] (33)

[0146] (34)

[0147] In the formula, is the actual number of partitions; is the ideal number of partitions; is the penalty parameter, which is used to control the width of the function.

[0148] Intra-Subnetwork Regulation Capacity Index (ISRCI).

[0149] (35)

[0150] Introducing the Physical Connection Index (PCI).

[0151] (36)

[0152] Based on the above five indicators, the objective function for distribution network partitioning is constructed as shown in Equation (37).

[0153] (37)

[0154] (38)

[0155] In the formula, , , is the weight.

[0156] The improved particle swarm algorithm is used to solve the partitioning problem.

[0157] First, define the signal function:

[0158] (39)

[0159] (40)

[0160] Based on the signal function, a particle velocity update strategy of discrete particle swarm method is proposed:

[0161] (41)

[0162] In the formula, the subscript Representative particles; the subscript k represents the node number; is the number of iterations; v is the velocity of the particle; x is the current position of the particle; is the individual optimal value; is the optimal value for the group; is the inertia weight; and is the learning factor; and is a random number in the interval [0,1]; Represents the exclusive-or operation.

[0163] Except for the boundary nodes of the region, the adjacent nodes of the remaining nodes all belong to the same sub-region. Based on this feature, the particle position update method is proposed as shown.

[0164] (42)

[0165] In the formula, is the set of nodes connected to node k; It is the number of the sub-region to which any node selected randomly belongs.

[0166] The particle swarm algorithm is used to partition the AC distribution network and the DC distribution network respectively to obtain AC sub-areas and DC sub-areas.

[0167] Step 3: Model the partitioned voltage control problem as a state-adversarial Markov decision process.

[0168] By partitioning the AC and DC distribution networks, the voltage control problem is decomposed into multiple sub-problems for solution. At the same time, considering the influence of measurement errors, a state-adversarial Markov decision model for multi-time-scale robust voltage control is established, as shown in Figure 3, which mainly includes two parts: defense agent and attack agent. Among them, the defense agent is responsible for the scheduling of controllable devices, including the control of mechanical devices on a long time scale and the control of power electronic devices on a short time scale. In the process of training the defense agent, a powerful attack agent is designed to find the inherent weaknesses of the defense agent and reduce the rewards obtained by the defense agent. This design enables the defense agent to adapt to different types of strong adversarial attacks and improve the robustness of the control method.

[0169] 1) Defense agent: The defense agent consists of long-time scale agents and short-time scale agents. The long-time scale agent is responsible for the control of mechanical equipment. At time t, it makes decisions based on the observation of the global state and dispatches OLTC and CB. The short-time scale agent is responsible for the control of power electronic equipment. At each time interval, it makes decisions based on the observation of the sub-region state and controls VSC, photovoltaic inverter and energy storage.

[0170] 2) Attacking agent: The attacking agent is used to simulate the interference of sensor errors and other factors on the state in the actual distribution network. The attack target of the agent is the actual state observation value of the AC / DC distribution network. By modifying the state observation value, it guides the defensive agent to make wrong decisions, making it unable to achieve the optimal voltage control effect.

[0171] Specifically, defense process modeling includes:

[0172] a) Long time scale voltage control modeling

[0173] The voltage control problem on a long time scale mainly consists of the following parts:

[0174] State Space: is the state space of the long-term intelligent agent, including the power of the load after the disturbance and the active output of the photovoltaic power plant. Therefore, the long-term intelligent agent is defined as The status of the period is The load power and photovoltaic active output of the first time interval within the time, that is, .in, and The active power and reactive power of the AC load after the disturbance, is the power of the DC load after the disturbance, Contribute to photovoltaic power.

[0175] Action Space: is the action space of the long-time scale agent. The long-time scale agent outputs actions to schedule OLTC and CB. Moments, defining the actions of agents over long time scales .

[0176] Reward function: is the reward function of the long-time scale agent. The goal of the long-time scale agent is to reduce the voltage deviation by scheduling OLTC and CB, so the long-time scale agent is defined as The reward for the moment is The sum of voltage deviations at all time intervals within a moment:

[0177] (43)

[0178] In the formula, There is a penalty for exceeding the voltage limit.

[0179] b) Short-time scale voltage control modeling

[0180] The distributed voltage control problem in the short time scale is mainly to achieve voltage control by scheduling the active power and reactive power of the converter, the reactive power of the photovoltaic inverter, and the charging and discharging of the energy storage. To achieve distributed control, the electrical distance is first constructed through the active voltage sensitivity matrix and the reactive voltage sensitivity matrix, and the partitioning method in step 2 is used to construct relevant indicators for AC and DC distribution network partitioning. Then, the voltage control problem in the short time scale is modeled, which specifically includes the following parts:

[0181] State Space: is the state space of the short-time-scale intelligent agent, including the power of the load after the disturbance, the photovoltaic output, the state of charge of the energy storage, the OLTC gear position and the number of CB input groups. time interval, the state of agent j is the local state of subregion j ,definition .in, , , , They are the The active power and reactive power of AC load, the active power of DC load and the active power output of PV after the disturbance at a certain time interval.

[0182] Action Space: is the action space of the short-time-scale agent, including the scheduling instructions for photovoltaics , dispatching instructions for energy storage charging and discharging , dispatching instructions for active and reactive power of VSC converter stations , , at time t time interval, the action of agent j ,definition The photovoltaic reactive power can be calculated according to formula (44): .

[0183] (44)

[0184] Due to the limitation of VSC capacity, its reactive power needs to be corrected as follows.

[0185] (45)

[0186] In the formula, is the capacity of the i-th VSC.

[0187] Due to the capacity limitation of energy storage, the charging and discharging power of energy storage needs to be corrected as follows according to its state of charge.

[0188] (46)

[0189] Reward function: is the reward function of the short-time scale agent. The goal of the short-time scale agent is to minimize the voltage deviation of the AC / DC distribution network, and all agents share the same reward. Moment The reward for each time interval is:

[0190] (47)

[0191] Specifically, the attack process modeling includes:

[0192] The state attack problem mainly consists of the following parts:

[0193] State Space: is the state space of the attacking agent, including the active power and reactive power of the load in the AC / DC distribution network environment, and the active output of photovoltaics. The moment time interval, defining the state of the attacking agent .

[0194] Action Space: is the action space of the attacking agent. The action of the attacking agent is a disturbance to the observation state of the defending agent. The specific physical meaning is the interference of sensor error, photovoltaic prediction error, load prediction error and other disturbance factors in the actual distribution network on the operation of the AC and DC distribution network. The moment time intervals, defining the actions of the attacking agent .

[0195] Reward function: is the reward function of the attacking agent. The goal of the attacking agent is to change the observation state of the defending agent so that it makes wrong decisions, thereby increasing the voltage deviation. Therefore, the reward of the attacking agent is defined as:

[0196] (48)

[0197] Step 4: Solve the state-adversarial Markov decision process, such as Figure 3 As shown, including:

[0198] 1) Solution to long-time voltage control problem

[0199] In the long time scale, the discrete action devices OLTC and CB are controlled at each time t. The BDQ algorithm is used to solve the optimization problem of this discrete control variable. The structure of the Q network is as follows: Figure 4 shown.

[0200] The BDQ algorithm updates the parameters of the network by minimizing the loss function, which is expressed as the mean square error:

[0201] (49)

[0202] In the formula, d is the dimension of the action, representing the dth controllable device; is the branch d of the Q network; are the parameters of the Q network; is the value of the d-th dimension action; is the temporal difference target value. For all action dimensions, As the timing difference target value:

[0203] (50)

[0204] In the formula, is the discount factor, is the branch d of the target Q network, and the parameters of the target Q network are .

[0205] After calculating the mean square error, calculate the parameters About the gradient of the loss function , the parameters of the neural network are updated using stochastic gradient descent :

[0206] (51)

[0207] In the formula, is the learning rate. By repeating the above training process, the network gradually approaches the true Q value function.

[0208] 2) Solution of short-time scale voltage control problem

[0209] In the short time scale, at time t, The reactive power of photovoltaic, energy storage charging and discharging, and VSC of continuous action equipment are dispatched at time intervals. For this optimization problem where the control variable is a continuous variable, the MASAC algorithm is used to solve it. The short-time scale agent includes the Actor network , target Actor network , Critic Network , target critic network , the parameters are , , , .

[0210] The neural network update process in the MASAC algorithm is as follows:

[0211] The goal during training is shown in formula (52).

[0212] (52)

[0213] In the formula, is the expected cumulative return of the strategy, is the action function of agent i, i.e., Actor network, for Parameters, is the temperature coefficient of entropy, is the strategy entropy, which can be specifically expressed as:

[0214] (53)

[0215] Each agent consists of an Actor network, a Critic network, a target Actor network, and a target Critic network.

[0216] During training, the Critic network is used to collect the observed states and actions of all agents and obtain global information to evaluate the actions of the agents. The target Critic network has the same structure as the Critic network, but its parameters are not updated in real time with training. Instead, they are copied from the Critic network at a fixed frequency to improve the stability of neural network training. The Critic network is updated by minimizing the loss function, which can be expressed as:

[0217] (54)

[0218] (55)

[0219] In the formula, is the target Actor network, for Parameters; For the Critic network, for Parameters; is the target Critic network, for Parameters; is the discount factor; is the global observation state; is the set of actions of all agents.

[0220] The Actor network of each agent takes actions based on the local observation state and does not need to interact with other agents. The target Actor network has the same structure as the Actor network, and its parameters are copied from the Actor network at a fixed frequency to improve the stability of neural network training. The Actor network of each agent is updated by gradient ascent, and the gradient calculation formula is:

[0221] (56)

[0222] (57)

[0223] (58)

[0224] In the formula, is the advantage function, which is used to determine whether the increase in rewards belongs to other agents; is the baseline of the multi-agent system, which represents the average value of all actions taken by the agent in the current state. It is the experience pool of agent i, which is used to store past experience and improve the efficiency and stability of learning.

[0225] 3) Solution to the state attack problem

[0226] At each time interval , the attacking agent disturbs the state of the environment, causing the defending agent to observe non-real load power and active output of photovoltaics. The SAC algorithm is used to solve this process. On the basis of maximizing the expected cumulative return, the SAC algorithm adds the entropy of the action to promote the agent to explore more actions and improve the global search ability and generalization performance of the algorithm. The training process objective is shown in formula (59).

[0227] (59)

[0228] In the formula, is the expected cumulative return of the strategy, is the action function of the agent, i.e., the Actor network, whose parameters are , is the temperature coefficient of entropy, is the strategy entropy, which can be specifically expressed as:

[0229] (60)

[0230] The agent consists of two action-value networks: the Critic network and the target Critic network. During training, the Critic network is used to evaluate the agent's actions based on the observed state of the environment and the agent's actions. The target Critic is used to improve the stability of neural network training. The Critic network is updated by minimizing the loss function, which can be expressed as:

[0231] (61)

[0232] (62)

[0233] In the formula, is the target Actor network, and the parameters are ; is the Critic network, and its parameters are ; is the target Critic network, and its parameters are ; is the discount factor; is the observation state; is the action of the attacking agent.

[0234] The attacking agent's Actor network takes actions based on the global observation state, and the target Actor network is used to improve the stability of neural network training. The Actor network is updated by maximizing the following objective function:

[0235] (63)

[0236] Step 5: The attacking agent and the defending agent conduct interactive training;

[0237] The parameter set to be optimized in the proposed method includes: the parameter set to be optimized for the long-term intelligent agent ; Parameter set to be optimized for short-time-scale agents , the set of parameters to be optimized for the attacking agent By constructing a training framework of synchronous training and asynchronous learning, the defense agent and the attack agent perform alternating adversarial training, so that the defense agent can learn the optimal control strategy while having stronger robustness. This is equivalent to making the AC and DC distribution network have lower voltage deviation during operation in the presence of external interference and measurement errors.

[0238] In this stage, the defense agent and the attack agent are trained alternately through the proposed synchronous training asynchronous learning framework. The specific process is as follows Figure 5 Before each round step, fix the parameters of the attacking agent and train the defending agent; after each round Step 1, fix the parameters of the defense agent and train the attack agent. Second, the synchronous training and asynchronous learning of offensive and defensive agents are realized.

[0239] Specifically, the defense agent training process is as follows: Figure 6 As shown, at the beginning of training, the parameters of the long-time scale agent Q network are first randomly initialized , parameters of the short-timescale Actor network And the parameters of the Critic network . Parameters of the target network , , Copy from , , In the first round of training, no perturbation is applied, and the cycle The long-time scale agent and the short-time scale agent are updated in rounds. In each round, a random Moment At each time t, the long-time scale agent first obtains the global state , input Q network to generate dispatch instructions for OLTC and CB , and transmit the scheduling instructions to the short-time scale agent. The short-time scale agent obtains the local observation state of the sub-region In the initial stage of training, in order to better explore the action space, the short-time scale agent randomly generates scheduling instructions and interacts with the environment to obtain rewards. , and the environment enters the next state, and each agent obtains a new state observation , the interaction data with the environment Store in experience pool In the process, when the stored experience data reaches a certain amount, each agent begins to generate scheduling instructions based on the Actor network. At each time interval, a random number is drawn from the experience pool Group data is used to update the short-time-scale agent. The parameters of the Critic network are updated as follows:

[0240] (64)

[0241] The Actor network parameters are updated as follows:

[0242] (65)

[0243] The target Actor network and target Critic network are updated as follows:

[0244] (66)

[0245] (67)

[0246] In the formula, is the soft update coefficient.

[0247] When time t ends, the long-time agent receives a reward value , and the data interacting with the environment Deposit into experience pool In. When When the stored data reaches a certain amount, a portion of the data is randomly extracted from the experience pool at each moment, and the network parameters of the long-time scale agent are updated according to equations (64)-(67).

[0248] In the second and subsequent rounds of training, the attack agent is added to disturb the state. The parameters of the attack agent are fixed and the time interval When the attacking agent is based on the observed state Make a disturbing move , perturbs the observed state of the defense agent, and the long-term agent follows and Q network to make decisions , short-time-scale agents are based on Making decisions with the Actor network , and get rewards after interacting with the environment and , and attack the agent to get the corresponding reward . The long-term experience data of intelligent agents in this process , empirical data of short-time-scale agents and experience data of the attacking agent Store them in their respective experience pools.

[0249] Specifically, the attack agent training process is as follows: Figure 7 As shown, at the beginning, the parameters of the attack agent neural network are randomly initialized and , fix the parameters of the long-time-scale agent and the short-time-scale agent neural network, and then loop rounds to train the attacking agent. The moment time interval, the attacking agent obtains the global state , in the initial stage of training, the attacking agent randomly generates attack actions The interaction data with the environment is stored in the experience pool for updating network parameters. When the stored experience data reaches a certain amount, the attacking agent begins to generate attack actions according to the Actor network. , the long-time scale agent and the short-time scale agent make scheduling actions based on the disturbed state and , calculate the voltage of the AC and DC distribution network through power flow calculation, and calculate the reward value , then the system enters the next state, and the attacking agent obtains a new state observation , and the empirical data of interaction with the environment Deposit into experience pool At the same time, the long-time scale agent and the short-time scale agent also store experience data in their respective experience pools. At each time interval, A portion of data is randomly extracted from the network to be used for updating the attack agent. The Critic network is updated by minimizing the loss function, which can be expressed as:

[0250] (68)

[0251] The Actor network is updated by maximizing the policy gradient, which can be expressed as:

[0252] (69)

[0253] The target Actor network and target Critic network update formulas are as follows:

[0254] (70)

[0255] (71)

[0256] Step 6: Deploy the trained defense agent online.

[0257] After the alternating training is completed, only the Q network of the long-time scale agent and the Actor network of the short-time scale agent are retained. The long-time scale agent and the short-time scale agent are deployed online to schedule the controllable devices.

[0258] First load the parameters of the long-time-scale agent Q network and parameters of the short-timescale agent actor network At each time t, the long-term agent generates scheduling instructions for discrete action devices based on global state observations , and then pass the states of OLTC and CB to the short-time scale agent. At each time interval at time t In the network, the agent generates scheduling instructions for continuous action devices based on the local observation status of the sub-area to which it belongs and the status of OLTC and CB. .Will and It is executed in an AC / DC distribution network environment. Since the short-time-scale intelligent agent only needs to observe the local state when making decisions, and does not need to transfer information between sub-regions, it does not rely on communication equipment and realizes distributed control.

[0259] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for coordinated voltage control in AC / DC distribution networks, characterized in that: The steps include: Step 1: Establish a voltage control model for AC and DC distribution networks; Step 2: Partition the AC and DC distribution network; including: The objective function for distribution network partitioning is constructed as follows: ； ； In the formula, , , is the weight, ISCI is the aggregation index within the sub-region, ISSI is the sparseness index between sub-regions, SCI is the number of sub-regions, ISRCI is the sub-region regulation capacity index, and PCI is the physical connectivity index; ； is the set of internal node numbers of the sub-region to which node i belongs, is the electrical distance between two points on the AC side; ； ， ； In the formula, is the actual number of partitions; is the ideal number of partitions; is the penalty parameter, which is used to control the width of the function; ； ； Use improved particle swarm optimization to solve partitioning problems, including: Define the signal function: ； ； Based on the signal function, a particle velocity update strategy of discrete particle swarm method is proposed: ； In the formula, the subscript Representative particles; the subscript k represents the node number; is the number of iterations; v is the velocity of the particle; x is the current position of the particle; is the individual optimal value; is the optimal value for the group; is the inertia weight; and is the learning factor; and is a random number in the interval [0,1]; Represents the exclusive OR operation; Except for the boundary nodes of the region, the adjacent nodes of the remaining nodes all belong to the same sub-region. Based on this feature, a particle position update method is proposed: ； In the formula, is the set of nodes connected to node k; is the number of the sub-region to which any node selected randomly belongs; The particle swarm algorithm is used to partition the AC distribution network and the DC distribution network respectively to obtain AC sub-areas and DC sub-areas. Step 3: Model the partitioned voltage control problem as a state-adversarial Markov decision process; Step 4: Solve the state-adversarial Markov decision process; Step 5: Interactive training of the attacking agent and the defending agent; including: By building a training framework of synchronous training and asynchronous learning, the defense agent and the attack agent perform alternating adversarial training. step, fix the parameters of the attacking agent and train the defending agent; after each round Step 1, fix the parameters of the defense agent, train the attack agent, and repeat this process. Second, realize the synchronous training and asynchronous learning of attack and defense agents; Defense agent training includes: At the beginning of training, the parameters of the long-time-scale agent Q network, the parameters of the short-time-scale agent Actor network, and the parameters of the Critic network are randomly initialized. No perturbation is applied in the first round of training. The long-time scale agent and the short-time scale agent are updated in rounds. In each round, a random Moment At each time t, the long-time scale agent first obtains the global state, inputs it into the Q network to generate dispatching instructions for the on-load tap-changing transformer and capacitor bank, and transmits the dispatching instructions to the short-time scale agent, which obtains the local observation state of the sub-region. In the initial stage of training, short-time-scale agents randomly generate scheduling instructions and interact with the environment to obtain rewards. At the same time, the environment enters the next state, and each agent obtains a new state observation and stores the interaction data with the environment in the experience pool. When the stored experience data reaches the set amount, each agent begins to generate scheduling instructions based on the Actor network and randomly extracts from the experience pool at each time interval. Group data, used for updating the agent in a short time scale; among them, is a pre-set hyperparameter; When time t ends, the long-time scale agent obtains a reward value and stores the data of interaction with the environment in the experience pool. When the data stored in the experience pool reaches the set amount, a portion of data is randomly extracted from the experience pool at each moment to update the network parameters of the long-time scale agent. In the second and subsequent rounds of training, the perturbation of the attacking agent to the state is added, the parameters of the attacking agent are fixed, and the time interval is At this time, the attacking agent makes a perturbation action according to the observed state, perturbs the observed state of the defending agent, the long-time scale agent makes a decision, the short-time scale agent makes a decision, and gets a reward after interacting with the environment. At the same time, the attacking agent gets a corresponding reward. In this process, the experience data of the long-time scale agent, the experience data of the short-time scale agent, and the experience data of the attacking agent are stored in their respective experience pools. Attacker agent training includes: At the beginning, the parameters of the attack agent neural network are randomly initialized, the parameters of the long-time scale agent and the short-time scale agent neural network are fixed, and then the cycle rounds to train the attacking agent, The moment At each time interval, the attack agent obtains the global state. In the initial stage of training, the attack agent randomly generates attack actions and stores the interaction data with the environment in the experience pool for updating network parameters. When the stored experience data reaches the set amount, the attack agent begins to generate attack actions according to the Actor network. The long-time scale agent and the short-time scale agent make scheduling actions based on the state after the disturbance, calculate the voltage of the AC / DC distribution network through power flow calculation, and calculate the reward value. Then the system enters the next state, the attack agent obtains new state observations, and stores the experience data of the interaction with the environment in the experience pool. At the same time, the long-time scale agent and the short-time scale agent also store the experience data in their respective experience pools. At each time interval, a part of the data is randomly extracted from the experience pool for updating the attack agent. Step 6: Deploy the trained defense agent online.

2. The method for coordinated voltage control in AC / DC distribution network partitions according to claim 1, characterized in that: In step 1, the voltage control model includes a branch power flow model and a voltage source converter model, and the objective function of the voltage control model is to minimize the voltage deviation: ； In the formula, is the objective function, where Indicates the time, express The moment time interval, represents the sum of the voltage deviations of the AC side nodes, represents the sum of the DC side node voltage deviations, , , , , , are the control variables, which are the PV reactive power, energy storage charging and discharging power, VSC active power, VSC reactive power, number of capacitor banks put into operation and on-load tap-changing transformer gear position; The constraints of the voltage control model include system power balance constraints, voltage upper and lower limit constraints, photovoltaic inverter power constraints, energy storage power constraints, and voltage source converter power constraints.

3. The method for coordinated voltage control in AC / DC distribution network partitions according to claim 1, characterized in that: In the step 3, modeling the partitioned voltage control problem as a state-adversarial Markov decision process includes: defense process modeling and attack process modeling; defense process modeling includes: modeling the voltage control problem under long time scales and modeling the voltage control problem under short time scales.

4. The method for coordinated voltage control in AC / DC distribution network partitions according to claim 1, characterized in that: In step 4, solving the state-adversarial Markov decision process includes: The BDQ algorithm is used to solve the long-time scale voltage control problem; The MASAC algorithm is used to solve the short-time-scale voltage control problem; The SAC algorithm is used to solve the state attack problem.

5. The method for coordinated voltage control in AC / DC distribution network partitions according to claim 1, characterized in that: In step 6, after the alternating training is completed, only the Q network of the long-time scale agent and the Actor network of the short-time scale agent are retained, and the long-time scale agent and the short-time scale agent are deployed online to schedule the controllable devices.

Images