Multi-time scale voltage regulation method and system based on deep reinforcement learning

Abstract

The invention belongs to the technical field of power distribution networks, and particularly discloses a multi-time scale voltage regulation method and system based on deep reinforcement learning. The invention provides a voltage regulation method for an active power distribution network with a time sequence characteristic and a hybrid equipment combined control capability, which comprises the following steps of: firstly, setting an intelligent agent capable of outputting discrete actions and continuous actions; and hour-level and minute-level dual-time-scale scheduling of an on-load voltage regulator, a capacitor bank and a renewable energy inverter is realized. In addition, the invention further designs a redundant multi-agent cooperative control system, and action output of all agents is coordinated through a multiplication mechanism of transverse interaction. In addition, a priority experience playback technology, a Thompson sampling technology and a security module mechanism are also integrated, and the learning efficiency and the control effect of the intelligent agent are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of distribution networks, and relates to a voltage control method for an active distribution network based on deep reinforcement learning, and specifically to a multi-time-scale voltage regulation method and system based on deep reinforcement learning. Background Art

[0002] As the penetration rate of renewable energy sources (RES) in active distribution networks (ADN) increases, the uncertainty of RES brings serious operational problems to ADN, such as voltage fluctuation and over-limit, high active power loss, etc. Therefore, in order to fully deal with the problems caused by the uncertainty of RES to ADN, it is necessary to design a real-time online voltage control method for ADN. Deep reinforcement learning (DRL) is a real-time control method that performs well in games, robotics, and power system applications.

[0003] At present, although DRL-based voltage regulation methods have been gradually applied to ADN, such voltage regulation methods still cannot well solve the needs of multi-time scale and hybrid equipment joint control in voltage regulation. Summary of the invention

[0004] In view of the above problems existing in the existing DRL voltage regulation method, the present invention proposes a multi-time-scale voltage regulation method based on deep reinforcement learning for optimizing voltage control in active distribution networks.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a multi-time scale voltage regulation method based on deep reinforcement learning, comprising the following steps: step 1. modeling an active distribution network with a large amount of renewable energy, and establishing a voltage optimization model of the active distribution network; step 2. establishing an intelligent agent with continuous action and discrete action characteristics containing time series characteristics; the intelligent agent includes two networks, one is a slow network for discrete control on the hourly time scale for on-load voltage regulators and capacitor bank scheduling, and the other is a fast network for continuous control on the minute-level time scale for inverter and electric energy storage system scheduling; the fast network and slow network of the intelligent agent have two internal networks, which are respectively defined as a fast control Q network, a fast control target Q network, a slow control Q network, and a slow control target Q network; step 3. A multi-time-scale decision-making process is constructed using a single-layer Markov to describe the interaction process between the active distribution network voltage optimization model and the agent, and the distribution network state, fast network state and slow network state are introduced; a state transition function of three state couplings is specially designed, and a time counter is used to separate the fast time scale and the slow time scale, and the output of the time counter is used as an activation signal to control the slow network; Step 4. A redundant multi-agent collaborative control system for multi-time-scale voltage regulation based on deep reinforcement learning for active distribution networks is proposed; multiple agents are used in the system to perform the same voltage regulation task, and then the action output of the agent is adjusted through a redundant coordination mechanism, and finally the coordinated action is applied to the active distribution network; Step 5. Establish the training process of the agent; the trained agent, combined with the redundant multi-agent collaborative control system, is applied to the voltage control of the active distribution network to achieve multi-time-scale voltage regulation.

[0006] In addition, based on the above-mentioned multi-time-scale voltage regulation method based on deep reinforcement learning, the present invention also proposes a corresponding multi-time-scale voltage regulation system based on deep reinforcement learning, which adopts the following technical scheme: the multi-time-scale voltage regulation system based on deep reinforcement learning, including the following modules: a model building module, used to model the active distribution network with a large amount of renewable energy, and establish an active distribution network voltage optimization model; an intelligent agent building module, used to establish an intelligent agent with continuous action and discrete action characteristics containing time series characteristics; the intelligent agent includes two networks, namely a slow network for discrete control on a time scale of hours for on-load voltage regulators and capacitor bank scheduling, and a fast network for continuous control on a time scale of minutes for inverter and electric energy storage system scheduling; the fast network and slow network of the intelligent agent have two internal networks, which are defined as a fast control Q network, a fast control target Q network, a slow control Q network, and a slow control target Q network. network; interaction process description module, which uses a single-layer Markov to construct a multi-time-scale decision-making process to describe the interaction process between the active distribution network voltage optimization model and the agent, and introduces the distribution network state, fast network state and slow network state; a state transfer function with three state couplings is specially designed, and a time counter is used to separate the fast time scale and the slow time scale, and the output of the time counter is used as an activation signal to control the slow network; a multi-agent collaborative control system construction module, which is used to propose a redundant multi-agent collaborative control system for multi-time-scale voltage regulation based on deep reinforcement learning for active distribution networks; multiple agents are used in the system to perform the same voltage regulation task, and then the action output of the agent is adjusted through a redundant coordination mechanism, and finally the coordinated action is applied to the active distribution network; and a voltage control module of the active distribution network, which is used to apply the trained agent, combined with the redundant multi-agent collaborative control system, to the multi-time-scale voltage control of the active distribution network.

[0007] The present invention has the following advantages:

[0008] As described above, the present invention relates to a multi-time-scale voltage regulation method and system based on deep reinforcement learning. The present invention makes up for the deficiency that the existing deep reinforcement learning algorithm cannot simultaneously regulate the on-load voltage regulator, capacitor bank, inverter, and electric energy storage system in the voltage control of the active distribution network. The redundant multi-agent coordination mechanism proposed in the present invention can significantly reduce the impact of individual agent erroneous decisions on the overall system, making the system more stable in the face of uncertainty. In addition, the priority experience replay and safety module introduced in the present invention can accelerate the convergence speed by using training data more effectively. In addition, the present invention also introduces Thompson sampling, which can provide a flexible exploration mechanism for the agent, so that when facing an unknown environment, the agent can balance exploration and utilization to avoid falling into a suboptimal solution too early. The present invention significantly improves the learning efficiency and control effect of the agent by integrating the priority experience replay technology, Thompson sampling technology and safety module mechanism, and well meets the needs of multi-time scale and hybrid equipment joint control in voltage regulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Flow chart of a multi-time-scale voltage regulation method based on deep reinforcement learning in an embodiment of the present invention.

[0010] Figure 2 A network structure diagram of an intelligent agent constructed in an embodiment of the present invention.

[0011] Figure 3 This is a network structure diagram of the redundant multi-agent collaborative control system built in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0013] Example 1

[0014] like Figure 1As shown, this embodiment describes a multi-time scale voltage regulation method based on deep reinforcement learning, in which a detailed voltage optimization model is constructed and advanced algorithm strategies are introduced to effectively manage the behavior of voltage and equipment. Specifically, the present invention establishes a dual-time scale parallel network structure described by a single-layer Markov in an intelligent agent. One is a slow control network for on-load voltage regulators and capacitor bank scheduling that is controlled on an hourly time scale, and the other is a fast control network for inverters and electric energy storage systems that is controlled on a minute-level time scale. For the network controlled on an hourly time scale, the present invention designs a control method suitable for on-load voltage regulators and capacitor banks. For the network controlled on a minute-level time scale, the present invention designs a control method suitable for inverters and electric energy storage systems. During the control process, the two networks use the same objective function as a guide, and then the obtained scheduling strategies are stacked and uniformly output by the intelligent agent. In addition, the present invention also proposes a redundant multi-agent collaborative decision-making architecture. For the same voltage regulation task, the present invention designs and uses multiple agents, optimizes the scheduling results of individual agents by using a collective decision-making model, and then outputs an optimal scheduling result for the voltage regulation task. This redundant multi-agent collaborative decision-making architecture effectively improves the robustness of the system. On the other hand, the present invention also uses a safety module, priority experience playback technology, and Thompson sampling technology to improve the control effect of the proposed method.

[0015] The multi-time scale voltage regulation method based on deep reinforcement learning in this embodiment includes the following steps:

[0016] Step 1. Model the active distribution network with a large amount of renewable energy and establish a voltage optimization model of the active distribution network, including the line characteristic model of the active distribution network, the on-load voltage regulator model, the capacitor bank model, the inverter model installed at the photovoltaic power generation system, the inverter model installed at the wind power generation system, and the electric energy storage system model.

[0017] The distribution network voltage optimization model is used to calculate the sum of active power losses of each line within the time range T, the number of distribution network nodes and the real-time voltage of each node. In particular, a safety module mechanism is used for the electric energy storage system model.

[0018] Specifically, in order to improve voltage quality and achieve voltage stability, the present invention implements voltage regulation by reducing active power loss, so the objective function is defined as follows:

[0019] .

[0020] The above formula represents the sum of active power losses of each node in the distribution network within the time range T.

[0021] in, is the number of distribution network nodes, Indicates the distribution network Nodes and The lines between nodes On time The current value at that time, Indicates line impedance.

[0022] The formula of the line characteristic model of the active distribution network is as follows:

[0023] For the line characteristic model of the active distribution network, it is described as active power and reactive power constraints:

[0024] .

[0025] .

[0026] in, and Respectively represent nodes exist The voltage amplitude and phase angle at the moment, and express Node in The actual amount of active power and reactive power injected at each moment. and Is a node and nodes The real and imaginary parts of the admittance elements between lines, namely conductance and susceptance. Is a node and nodes The phase angle difference between the lines is expressed in the form of: .

[0027] Actual active power injection is through Always access the node Active power output of photovoltaic power generation device , the active power output of the wind turbine , the amount of active power absorbed or released by the energy storage system and load active power demand The actual reactive power injection amount is obtained. is through Access Node The reactive power injection amount of the inverter at the photovoltaic power generation , reactive power injection amount of inverter at wind power generation , reactive power injection amount of capacitor bank Reactive power demand of load To obtain.

[0028] In summary, for the nodes connected to the control device, the amount of power injection after the device is activated is:

[0029] ; .

[0030] The voltage safety constraint is expressed as follows:

[0031] .

[0032] in, Represents a distribution network node The voltage value, It indicates the minimum voltage value of each node under normal operation of the distribution network; It indicates the maximum voltage value of each node under normal operation of the distribution network.

[0033] The formula of the on-load voltage regulator model is as follows:

[0034] ; ; .

[0035] in, The primary voltage of the on-load voltage regulator at the busbar; It is the tap position of the on-load voltage regulator; The number of operations of the on-load voltage regulator; The maximum number of operations of the on-load voltage regulator in one day; It is the maximum tap position change range of the on-load voltage regulator.

[0036] The capacitor bank model is expressed as follows:

[0037] ; ; .

[0038] in, Represents the reactive power injected into the distribution network by the capacitor bank, Indicates the reactive power of a group of capacitors; Indicates the number of capacitor banks connected to the active distribution network; is the number of operations of the capacitor bank; It is the maximum number of operations of the capacitor bank in one day; It is the maximum range of the number of action groups of the capacitor bank.

[0039] The formula of the inverter model installed at the photovoltaic power generation system is as follows:

[0040] ; .

[0041] in Represents the active power injected into the distribution network by the photovoltaic power generation system. is the maximum active power output of the photovoltaic power generation system; It is the available reactive power of the inverter of the photovoltaic power generation system; is the rated power of the inverter of the photovoltaic power generation system. The formula for installing the inverter model at the wind power generation system is as follows:

[0042] ; .

[0043] in, It represents the active power injected into the distribution network by the wind power generation system. is the maximum active power output of the wind power generation system; is the available reactive power of the inverter of the wind power generation system; is the rated power of the inverter of the wind power generation system.

[0044] The formula of the electric energy storage system model is as follows:

[0045] .

[0046] .

[0047] .

[0048] .

[0049] in, , are the minimum and maximum energy storage levels of the energy storage system; , Respectively expressed in , Time Node The power content of the energy storage system; express At the node The charging power of the energy storage system; Indicates the maximum charging power of the energy storage system; Indicates the maximum discharge power of the energy storage system; express At the moment The discharge power of the energy storage system of the node; Representation Node Equipment efficiency when charging the energy storage system; Representation Node Equipment efficiency when the energy storage system is discharging; represents the time step; Indicates the time range, is the number of distribution network nodes.

[0050] The electric energy storage system model is optimized and the safety module is used. The formula is as follows:

[0051] .

[0052] .

[0053] .

[0054] in, and Respectively expressed in , Time Node The power content of the energy storage system.

[0055] Step 2. Propose a Dueling DeepQ-Network algorithm with hybrid action space, priority experience replay, and Thompson Sampling characteristics, namely the Hybrid Preferred Adversarial Deep Q Network, referred to as HPTS-Dueling DQN.

[0056] Among them, H stands for Hybrid Action Space, P stands for Priority Experience Replay, and TS stands for Thompson Sampling.

[0057] HPTS-Dueling DQN is used to establish an agent with continuous and discrete action characteristics with time series characteristics. The agent has two neural network structures, one is a slow network for discrete control on the hourly time scale for on-load voltage regulators and capacitor bank scheduling. The other is a fast network for continuous control on the minute-level time scale for inverter and electric energy storage system scheduling.

[0058] The present invention uses the HPTS-Dueling DQN algorithm to control all devices, that is, the fast network and slow network of the intelligent body are designed based on the HPTS-Dueling DQN algorithm, and HPTS-DUELING DQN is used to complete the real-time multi-time scale voltage regulation within the day. As a deep reinforcement learning algorithm, HPTS-Dueling DQN can quickly give a control strategy and give control based on observations within the day, especially when controlling OLTC and CB devices, it is not necessary to control OLTC and CB based on the predicted results before the day, but directly give control based on the observations of the day. Other methods either control OLTC and CB before the day and control PV, WT, and ES within the day, or control OLTC and CB within the day and control PV, WT, and ES before the day, and they adopt two stages or a combination of multiple methods. The present invention uses a method to uniformly schedule OLTC, CB, PV, WT, and ES within the day.

[0059] like Figure 2 As shown, each network of the intelligent agent (i.e., the fast network and the slow network) has two internal networks, one is the target control network and the other is the control network. These networks are respectively the fast control Q network, the fast control target Q network, the slow control Q network, and the slow control target Q network.

[0060] Step 2.1. Establishment of the time scale control network.

[0061] The design is based on the Dueling Deep Q-Network (Dueling DQN) algorithm, which decomposes the Q value into two parts: state value and advantage function, so as to more effectively estimate the action value under different states.

[0062] The state value function and advantage function use a shared feed-forward neural network for feature extraction.

[0063] Feedforward neural networks are established for the fast control Q network, the fast control target Q network, the slow control Q network, and the slow control target Q network respectively. , :

[0064] .

[0065] .

[0066] in Enter for the state; , is the weight matrix, , is the bias term; 1, 2 represents the number of layers of the neural network, that is, 1 represents the first layer of the neural network, and 2 represents the second layer of the neural network; is the activation function.

[0067] The neural network parameters are used Indicates that:

[0068] .

[0069] There are fast network parameters and slow network parameters .

[0070] Using state value function Measuring the current state The overall value of represents the basic reward that can be obtained no matter which action is chosen in the current state.

[0071] Note: The fast control Q network and the fast control target Q network use the same network structure. , the slow control Q network and the slow control target Q network use the same network structure .

[0072] To avoid trouble, the present invention uses fast network and slow network to replace description when describing.

[0073] The structures and parameters of the fast control Q network and the fast control target Q network are exactly the same when they are established. The main difference lies in the parameter update method and naming. The structures and parameters of the slow control Q network and the slow control target Q network are exactly the same when they are established. The main difference lies in the parameter update method and naming.

[0074] Establish state value functions for fast networks and slow networks respectively and :

[0075] .

[0076] .

[0077] in, , , , Represents the network parameters used to calculate the state value network in the fast network. , is the weight matrix, , is the bias term; 1, 2 represents the number of layers of the neural network, that is, 1 represents the first layer of the neural network, and 2 represents the second layer of the neural network; , , , Represents the network parameters used to calculate the state value network in the slow network. , is the weight matrix, , is the bias term; 1, 2 represents the number of layers of the neural network, that is, 1 represents the first layer of the neural network and 2 represents the second layer of the neural network.

[0078] Using advantage function Indicates in status Perform a specific action Advantage over average, i.e. how much better performing an action is than other actions in that state.

[0079] Establish advantage functions for fast networks and slow networks respectively and :

[0080] .

[0081] .

[0082] in, , , , represents the network parameters used to calculate the advantage value network in the fast network, , is the weight matrix, , is the bias term; , , , represents the network parameters used to calculate the advantage value network in the slow network, and similarly , is the weight matrix, , is the bias term; subscript 1 and subscript 2 represent the number of layers of the neural network, that is, 1 represents the first layer of the neural network and 2 represents the second layer of the neural network.

[0083] The final Q value is calculated by combining the value and advantage.

[0084] Establish Q value functions for fast networks and slow networks respectively and :

[0085] .

[0086] .

[0087] in, represents the action space of the fast network; Indicates the next action of the fast network, and ; represents the action space of the slow network; Indicates the next action of the slow network, and . and It is the average value of the advantage function of all actions, which is used to normalize the advantage value and ensure that the calculation of Q value is more stable.

[0088] The output of the fast network is converted into a continuous action value by the tanh function. , mapping the actions to the range [-1,1].

[0089] .

[0090] The output of the slow network is transformed from the discrete action set into a discrete action set through the argmax operation. Choose the best action. Indicates the gear position of the on-load voltage regulator OLTC or the number of capacitors CB connected to the ADN:

[0091] .

[0092] Step 2.2. Action splicing.

[0093] In multiple time frames, the fast network and the slow network generate fast actions and slow actions alternately on different time scales, that is, on-load voltage regulators and capacitor banks will not generate new actions on a minute-level time scale, only inverters and energy storage systems will act; on an hour-level time scale, on-load voltage regulators, capacitors, inverters, and energy storage systems will all act; the details are as follows:

[0094] .

[0095] in, and are the number of continuous actions and discrete actions respectively; represents the new set of actions for the fast network, represents the old set of actions for the slow network, Indicates new action for slow network.

[0096] It should be noted that under the action of the time counter, the fast network will output a new action value at each time step, while the slow network needs to wait for the activation signal of the counter. The slow network will output a new action value only after receiving the activation signal of the counter. When it does not receive the activation signal of the counter, it will output the last updated action, that is, the original old action.

[0097] Represents the new action set content of the fast network, Represents the old action set content of the slow network, Represents the new action set content for slow networks.

[0098] The action set of the fast network and the action set of the slow network are stacked. Moment Agent Action for:

[0099] .

[0100] Step 3. Use a single-layer Markov to construct a multi-time scale decision-making process to describe the interaction process between the active distribution network voltage optimization model and the intelligent agent, and introduce the distribution network state, fast network state, and slow network state.

[0101] Design three state-coupled state transfer functions, namely:

[0102] is the transfer function of the environment, i.e. the ADN state, is the transition function of the internal state of the fast network, is the transition function of the internal state of the slow network.

[0103] Their coupling forms are:

[0104] .

[0105] A time counter is used to separate the fast time scale from the slow time scale, and the output of the time counter is used as an activation signal to control the slow network.

[0106] Markov decision process (MDP) is used to represent the interaction process between the agent and the environment. Usually the necessary elements are: .in is the state space; It is the action space; is the transition probability function; is the reward function, defined according to the specific goals of the system; is the discount factor.

[0107] To deal with the problem of multiple time scales, a counter is introduced , explicitly representing and handling the different time scales of the two networks in the decision process. The multi-time scale decision process is constructed using a single-layer Markov as follows:

[0108] a) State space:

[0109] Defining states .

[0110] in, Is the current state of the environment, i.e. ADN ; is the internal state of the fast network; is the internal state of the slow network.

[0111] b) Action Space:

[0112] Defining actions ;

[0113] in, Is fast network in The action of the moment, Is the slow network in Moment of action.

[0114] ;in Indicates the control rate of the reactive power of the photovoltaic inverter, Indicates the control rate of the active power of the inverter of the wind power generation device; Indicates the active power control rate of the energy storage device.

[0115] if The energy storage system absorbs electrical energy if The energy storage system releases electrical energy.

[0116] For photovoltaic and wind turbine inverters, first use and Get the available reactive power, and then get the reactive power injected into the ADN under the corresponding control rate , ;

[0117] , .

[0118] in, Represents the reactive power available from a photovoltaic power generation device, represents the reactive power available to the wind turbine, where are the rated capacities of photovoltaic and wind turbine inverters respectively, and Indicates the control rate.

[0119] For energy storage systems, the control rate The actual active power injected into the ADN is obtained under the influence of .

[0120] if but ;if but .

[0121] .

[0122] in Indicates the position of the tap contact of OLTC. Indicates the number of capacitor banks connected to the ADN, that is, Determine the tap of the on-load voltage regulator OLTC location, through Determine the number of groups of capacitors connected to the distribution network .

[0123] c) Transition probability:

[0124] .

[0125] Among them, the transition probability Further broken down into:

[0126] .

[0127] in, Describe the transition of the environment, i.e. the ADN state, Describe the transition of the internal state of the fast network, Describes the transition of the internal state of a slow network.

[0128] d) Counter:

[0129] For slow networks, introduce a counter to keep track of the time steps.

[0130] According to the result of the counter, when the hour time scale of the slow network action is reached, the slow network should take a new action; at other time steps, it maintains the state of the previous action.

[0131] If the time step reaches the hour unit then .

[0132] If the time step is in minutes then , then there are action choices for the slow network:

[0133] .

[0134] Step 4. A redundant multi-agent collaborative control system for multi-time-scale voltage regulation based on deep reinforcement learning for active distribution networks is proposed. The redundant multi-agent collaborative control system in this embodiment specifically refers to that after multiple agents perform the same voltage regulation task and give control actions, the action outputs of each agent are coordinated using a lateral interaction multiplication mechanism, and then the action of one agent is randomly selected and applied to voltage regulation.

[0135] In a multi-agent collaborative control system, multiple agents are used to perform the same voltage regulation task, and then the action outputs of the agents are adjusted through a redundant coordination mechanism, and finally the coordinated actions are applied to the active distribution network.

[0136] Step 4.1. Establishment of redundant coordination matrix.

[0137] Features agents, the initial redundant coordination matrix For one The identity matrix of .

[0138] ;

[0139] The above formula means that initially, each agent only refers to its own actions and has no influence on each other.

[0140] Step 4.2. Coordination of actions among multiple agents.

[0141] At some point , Agent The action is represented as a vector .

[0142] The actions of all agents are concatenated to form a dimension The redundant action matrix is ​​connected through the communication matrix Coordinate and produce coordinated actions :

[0143] .

[0144] in It is The redundant coordination matrix at time t represents the influence weight of each agent on the actions of other agents.

[0145] Finally, the coordinated actions application.

[0146] Step 4.3. Update of the redundant coordination matrix.

[0147] The redundant coordination matrix is ​​updated according to the system performance after each action is executed.

[0148] The system's performance indicators are rewarded .

[0149] Each time it is updated, the matrix is ​​added with noise , and then according to the performance Adjust the matrix:

[0150] .

[0151] Among them, noise is a random matrix following a normal distribution:

[0152] .

[0153] in, Represents the parameters of the normal distribution. In order to ensure that the weights in the communication matrix are within a preset reasonable range, usually [0,1], and keep the sum of each row to 1, the communication matrix needs to be normalized.

[0154] Assume After the matrix of the step is updated, the new matrix is , the normalization process is as follows:

[0155] .

[0156] in, Representation Matrix No. Line The elements of the column, is the normalized matrix.

[0157] Step 5. Establish the training process of the intelligent agent based on steps 1, 2, 3, and 4.

[0158] During training, the fast network and the slow network use their own target Q network , To update the Q value, the target Q value and The calculation of is as follows:

[0159] .

[0160] .

[0161] in It’s an instant reward. Represents the discount factor.

[0162] Indicates that the fast control target Q network is at the input Output the Q value of all possible actions, and select the action that maximizes the Q value from these actions This maximized Q value is then used to calculate the target Q value.

[0163] Indicates that the slow control target Q network is at the input Output the Q value of all possible actions, and select the action that maximizes the Q value from these actions This maximized Q value is then used to calculate the target Q value.

[0164] In order to optimize the network parameters of the fast network and the slow network, the loss function is established using the target Q value and Q value for the two networks respectively. and , and then use gradient descent to minimize the loss function.

[0165] .

[0166] .

[0167] .

[0168] .

[0169] in is the learning rate, represents the expected value, Indicates the fast control Q network in the network parameters Next, in the state Take action when The Q value obtained; Indicates the slow control Q network in the network parameters Next, in the state Take action when The Q value obtained is is the loss function of the fast control Q network with respect to the parameter The gradient of is the loss function of the slow control Q network with respect to the parameter gradient.

[0170] In order to stabilize the training process, the target Q network is quickly controlled Parameters and slow control target Q network Parameters Instead of updating in every training step, every 100 time steps the parameters of the current fast Q-network are updated. and slow control Q network parameters Copy it, that is:

[0171] ; .

[0172] This updating method reduces fluctuations during training and ensures more stable calculation of target values.

[0173] The priority experience replay introduced in the Dueling Deep Q-Network algorithm is as follows:

[0174] Experience replay stores experience tuples , different experience tuples are given different priorities through the experience evaluation function, so that the experience that has a greater impact on the learning process is preferentially selected for playback and training. The details are as follows:

[0175] Each time a new experience is stored in the buffer, its error is calculated and a priority is assigned to each experience based on the error; Group experience, his experience evaluation function for:

[0176] .

[0177] in, represents the target Q network input , output the Q value of all possible actions , select the action that maximizes the Q value from these actions This maximized Q value is then used to calculate the target Q value.

[0178] The value of and Find the average value.

[0179] Same reason Indicates that the Q network is in the network participation state Take action when The Q value obtained.

[0180] The value of and Get the average.

[0181] No. The probability of experience sampling corresponding to the group experience for: .

[0182] It means that the sum of the priorities of all experiences is used to normalize the priority of each experience. is the total number of experiences in the buffer. An importance sampling weight is used when updating the Q network Correct the sampled samples, the calculation process is as follows:

[0183] ; is a parameter that controls the smoothness of the weights.

[0184] The Thompson sampling introduced in the Dueling Deep Q-Network algorithm is as follows:

[0185] Thompson sampling naturally combines exploration and exploitation through random sampling. Even if the current average reward of an action is high, other actions may still be sampled due to the uncertainty of its distribution.

[0186] Therefore, the choice of action is dynamically adjusted according to its uncertainty, and the specific steps are as follows:

[0187] Set each action Rewards The distribution is Beta distribution:

[0188] .

[0189] in, Indicates action Based on the reward The distribution of The content of the group experience tuple is used index), parameters , Represents the number of successes and failures respectively, and the initial value is 1.

[0190] Whenever an action needs to be selected, sample from the posterior reward distribution of each action and choose the action with the highest reward, that is, the action with the highest reward in the given state Next, from the reward distribution of each action Sampling is performed to generate a sample value for each action Then select the action with the largest sample value ,Right now:

[0191] For each action , sample a value from its corresponding Beta distribution: .

[0192] in, is a sample value drawn from a Beta distribution, and then the action with the largest sample value is selected: .

[0193] Update distribution parameters.

[0194] Rewards based on actions To update the corresponding Beta distribution parameters.

[0195] If the reward If it is a positive number, it increases the number of successes of the action. : .

[0196] If the reward If it is zero or negative, the number of failures of the action will be increased. : .

[0197] In step 1, an active distribution network voltage optimization model is established. In step 2, an intelligent agent for multi-time scale control is established. In step 3, the multi-time scale state transition principle in the active distribution network voltage optimization problem is described, and a multi-time scale decision-making process based on a single-layer Markov construction is established. In step 4, a redundant coordination mechanism for coordination between multiple agents is established.

[0198] Step 5 of the present invention is to train the intelligent agent. The specific training is divided into a multi-time scale action process, that is, executing a multi-time scale decision process and a network parameter update process based on a single-layer Markov model, as follows:

[0199] Multi-time-scale decision-making process based on single-layer Markov:

[0200] At a certain moment, the active distribution network inputs the state of the distribution network to the intelligent agent for multi-time scale control through the active distribution network voltage optimization model. The time counter of the intelligent agent outputs the control signal according to the time scale for the activation of the fast network and the slow network. The fast network and the slow network of the intelligent agent randomly give control actions according to the state. After the two network actions are integrated, the redundant coordination machine is used to coordinate the actions, and then the action is applied to the active distribution network voltage optimization model. The distribution network completes the state update according to the state transfer principle. At this time, a reward value is obtained using the new state.

[0201] In this process, the state of the distribution network , Actions acting on the voltage optimization model of the active distribution network , Reward Value , New Status Composing experience tuples Then, the experience tuple is evaluated using the experience evaluation function to obtain an evaluation result of the experience tuple, and the experience tuple is stored in the experience replay mechanism according to the evaluation result. Specifically, the experience tuple with better evaluation is stored in the experience replay mechanism first according to the evaluation result.

[0202] This process is repeated until the upper limit of the experience replay mechanism is reached. In the process of action sampling, the reward distribution of the action is first established based on the reward function and the action, and the action is sampled according to the reward distribution.

[0203] Then the agent network parameter update process is carried out:

[0204] Extract small batches of experience tuples from the experience replay mechanism. Then, establish the reward distribution of actions based on rewards and actions, and then select actions for updating parameters based on the reward distribution of actions. Complete the update of network parameters based on the internal structure of HPTS-DUELING DQN. At the same time, according to the action and reward value Update the coordination matrix of the redundant coordination mechanism.

[0205] The specific training steps of the agent are as follows:

[0206] Step 5.1. Initialization phase.

[0207] Step 5.1.1. Initialize the neural network parameters of the fast network and the slow network.

[0208] Step 5.1.2. Initialize the experience replay buffer and create an experience replay buffer to store the experience tuples generated by the interaction between the agent and the environment.

[0209] Step 5.1.3. Initialize the experience priority and set all priorities to 1.

[0210] Step 5.1.4. Initialize the redundant coordination matrix.

[0211] Step 5.1.5. Initialize the Bayesian Q-value distribution.

[0212] Step 5.2. Interaction process.

[0213] Step 5.2.1. Select actions using Thompson sampling.

[0214] Step 5.2.2. Use redundant multi-agent cooperative control system.

[0215] Step 5.2.3. Execute the Markov process to interact the agent with the active distribution network model.

[0216] Step 5.2.4. Store the experience tuples and assign an initialized priority to each experience tuple.

[0217] Step 5.3. Replay training with new experience.

[0218] Step 5.3.1. Calculate the loss function and priority.

[0219] Step 5.3.2. Sample experience and calculate importance sampling weights.

[0220] Step 5.3.3. Update the Q network parameters of the fast network and the slow network.

[0221] Step 5.3.4. Update the target Q network parameters of the fast network and the slow network.

[0222] Step 5.4. Bayesian distribution update.

[0223] Step 5.5. After executing all the training processes, save the agent with the new neural network parameters.

[0224] The agent trained in step 5 is combined with a redundant multi-agent collaborative control system to be applied to the voltage control of the active distribution network. Specifically, after the agent gives control according to the state of the active distribution network, it acts through the redundant multi-agent collaborative control system and then applies the control to the active distribution network.

[0225] As an advanced deep reinforcement learning algorithm, the HPTS-DUELING DQN algorithm has shown significant advantages in controlling devices. The core concept of the algorithm is to quickly generate control strategies so that in intraday scheduling, the system can make decisions directly based on real-time observation data without relying on previous prediction results. This feature is particularly important in the control of key equipment such as OLTC (on-load tap changer) and CB (circuit breaker), because traditional methods usually require control planning on the day before, relying on predictions of future load and power generation. Compared with other methods, the biggest highlight of the HPTS-DUELINGDQN algorithm is that its single method framework can achieve unified scheduling of OLTC, CB, photovoltaic (PV), wind power (WT) and energy storage system (ES). This means that in intraday control, the system can respond to changes in real time without having to adopt different control strategies in different time periods. For example, traditional control methods often take the form of two stages or a combination of multiple methods: one is to control OLTC and CB on the day before, and control PV, WT and ES on the day before; the other is to control OLTC and CB on the day before, and control PV, WT and ES on the day before. Although this staged control method is effective in some cases, it often leads to a decrease in response speed and dispatch efficiency. Through the HPTS-DUELING DQN algorithm, the system can achieve higher flexibility and real-time performance in intraday dispatch, and can quickly adapt to the dynamic changes in the power market and real-time feedback of equipment status. This not only improves the operating efficiency of the power grid, but also enhances the stability of the system and reduces the risks caused by prediction errors.

[0226] The present invention proposes a voltage regulation method for an active distribution network with time series characteristics and hybrid equipment joint control capability, that is, firstly, an intelligent agent that can output discrete actions and continuous actions is set up, and through a network containing time series characteristics, the hourly and minute-level dual time scale scheduling of on-load voltage regulators, capacitor banks, and renewable energy inverters is realized. In addition, the present invention also designs a redundant multi-agent collaborative control system, which coordinates the action outputs of each intelligent agent through a multiplication mechanism of lateral interactions. In addition, the present invention also integrates priority experience playback technology, Thompson sampling technology, and safety module mechanism, thereby improving the learning efficiency and control effect of the intelligent agent.

[0227] Example 2

[0228] This embodiment 2 describes a multi-time-scale voltage regulation system based on deep reinforcement learning, which is based on the same inventive concept as the multi-time-scale voltage regulation method based on deep reinforcement learning in the above embodiment 1.

[0229] The multi-timescale voltage regulation system based on deep reinforcement learning includes the following modules:

[0230] The model building module is used to model the active distribution network with a large amount of renewable energy and establish the voltage optimization model of the active distribution network; the intelligent agent building module is used to build an intelligent agent with continuous and discrete action characteristics containing time series characteristics; the intelligent agent includes two networks, namely, a slow network for discrete control on the time scale of hours for on-load voltage regulators and capacitor banks, and a fast network for continuous control on the time scale of minutes for inverters and electric energy storage systems; the fast network and slow network of the intelligent agent have two internal networks, which are defined as fast control Q network, fast control target Q network, slow control Q network, and slow control target Q network. network; interaction process description module, which uses a single-layer Markov to construct a multi-time-scale decision-making process to describe the interaction process between the active distribution network voltage optimization model and the agent, and introduces the distribution network state, fast network state and slow network state; a state transfer function with three state couplings is specially designed, and a time counter is used to separate the fast time scale and the slow time scale, and the output of the time counter is used as an activation signal to control the slow network; a multi-agent collaborative control system construction module, which is used to propose a redundant multi-agent collaborative control system for multi-time-scale voltage regulation based on deep reinforcement learning for active distribution networks; multiple agents are used in the system to perform the same voltage regulation task, and then the action output of the agent is adjusted through a redundant coordination mechanism, and finally the coordinated action is applied to the active distribution network; and a voltage control module of the active distribution network, which is used to apply the trained agent, combined with the redundant multi-agent collaborative control system, to the multi-time-scale voltage control of the active distribution network.

[0231] It should be noted that in the multi-time-scale voltage regulation system based on deep reinforcement learning, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method in Example 1, and will not be repeated here.

[0232] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A multi-time scale voltage regulation method based on deep reinforcement learning, characterized in that: The steps include: Step 1. Model the active distribution network with a large amount of renewable energy and establish a voltage optimization model for the active distribution network; Step 2. Establish an intelligent agent with continuous action and discrete action characteristics with time series characteristics; The intelligent agent includes two networks, one is a slow network for discrete control on the hourly time scale for on-load voltage regulators and capacitor bank dispatching, and the other is a fast network for continuous control on the minute-level time scale for inverter and electric energy storage system dispatching; the fast network and slow network of the intelligent agent have two internal networks, which are defined as fast control Q network, fast control target Q network, slow control Q network, and slow control target Q network respectively; Step 3. Use a single-layer Markov to construct a multi-time scale decision process to describe the interaction process between the active distribution network voltage optimization model and the intelligent agent, and introduce the distribution network state, fast network state, and slow network state; A state transfer function with three coupled states is specially designed, and a time counter is used to separate the fast time scale from the slow time scale. The output of the time counter is used as an activation signal to control the slow network. Step 4. Propose a redundant multi-agent cooperative control system for multi-time-scale voltage regulation based on deep reinforcement learning for active distribution networks; use multiple agents in the system to perform the same voltage regulation task, then adjust the action output of the agents through a redundant coordination mechanism, and finally apply the coordinated actions to the active distribution network; Step 5. Establish the training process of the intelligent agent; apply the trained intelligent agent, combined with the redundant multi-agent collaborative control system, to the voltage control of the active distribution network to achieve multi-time scale voltage regulation.

2. The multi-time scale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: In step 1, the distribution network voltage optimization model established includes: Line characteristic model of active distribution network, on-load voltage regulator model, capacitor bank model, inverter model installed at photovoltaic power generation system, inverter model installed at wind power generation system and electric energy storage system model; The distribution network voltage optimization model is used to calculate the sum of active power losses of each line within the time range T, the number of distribution network nodes and the real-time voltage value of each node, in which a safety module mechanism is used for the electric energy storage system model.

3. The multi-time-scale voltage regulation method based on deep reinforcement learning according to claim 2 is characterized in that: In the above, the formula of the electric energy storage system model is as follows: ； ； ； ； in, , are the minimum and maximum energy storage levels of the energy storage system; , Respectively expressed in , Time Node The power content of the energy storage system; express At the node The charging power of the energy storage system; Indicates the maximum charging power of the energy storage system; Indicates the maximum discharge power of the energy storage system; express At the moment The discharge power of the energy storage system of the node; Representation Node Equipment efficiency when charging the energy storage system; Representation Node Equipment efficiency when the energy storage system is discharging; represents the time step; Indicates the time range, is the number of distribution network nodes; The electric energy storage system model is optimized using the safety module mechanism. The optimization formula is as follows: ； ； ； in, and Respectively expressed in , Time Node The power content of the energy storage system.

4. The multi-time scale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.

1. Establishment of time scale control network; Designed based on Dueling Deep Q-Network algorithm, the Q value is decomposed into two parts: state value and advantage function. The state value function and advantage function use a shared feedforward neural network for feature extraction. For fast networks and slow networks, build feedforward neural networks respectively , : ； ； in, Enter for the state; , is the weight matrix, , is the bias term; subscript 1 and subscript 2 represent the number of layers of the neural network, i.e. 1 represents the first layer of the neural network and 2 represents the second layer of the neural network; is the activation function; The neural network parameters are used Indicates that: ； There are fast network parameters and slow network parameters ; Establish state value functions for fast networks and slow networks respectively and : ； ； in, , , , represents the network parameters used to calculate the state value network in the fast network, , is the weight matrix, , is the bias term; , , , Represents the network parameters used to calculate the state value network in the slow network. , is the weight matrix, , is the bias term; Subscript 1 and subscript 2 represent the number of layers of the neural network, i.e. 1 represents the first layer of the neural network and 2 represents the second layer of the neural network; Establish advantage functions for fast networks and slow networks respectively and : ； ； in, , , , represents the network parameters used to calculate the advantage value network in the fast network, , is the weight matrix, , is the bias term; , , , represents the network parameters used to calculate the advantage value network in the slow network, and similarly , is the weight matrix, , is the bias term; subscript 1 and subscript 2 represent the number of layers of the neural network, i.e. 1 represents the first layer of the neural network and 2 represents the second layer of the neural network; Establish Q value functions for fast networks and slow networks respectively and : ； ； in, represents the action space of the fast network; Indicates the next action of the fast network, and ; represents the action space of the slow network; Indicates the next action of the slow network, and ; and is the average value of the advantage function of all actions, which is used to normalize the advantage value to ensure that the calculation of the Q value is more stable; The output of the fast network is converted into a continuous action value by the tanh function. , map the action to the range [-1,1]; ； The output of the slow network is transformed from the discrete action set into a discrete action set through the argmax operation. Choose the best action among Indicates the gear position of the on-load voltage regulator OLTC or the number of capacitors CB connected to the ADN: ； Step 2.

2. Action splicing: In multiple time frames, the fast network and the slow network generate fast actions and slow actions alternately at different time scales, respectively; the details are as follows: ； in, and are the number of continuous actions and discrete actions respectively; represents the new set of actions for the fast network, represents the old set of actions for the slow network, Indicates new actions for slow networks; Represents the new action set content of the fast network, Represents the old action set content of the slow network, Represents the new action set content of the slow network; Under the action of the time counter, the fast network will output a new action value at each time step, while the slow network needs to wait for the activation signal of the counter; The slow network will output a new action value only after receiving the activation signal of the counter. If it does not receive the activation signal of the counter, it will output the last updated action, that is, the original old action. The action set of the fast network and the action set of the slow network are stacked. Moment Agent Action for: 。 5. The multi-timescale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, a single-layer Markov model is used to construct a multi-time scale decision process as follows: a) State space: Defining states ; in, Is the current state of the environment, i.e. ADN ; is the internal state of the fast network; is the internal state of the slow network; in, Indicates the load active power demand, Indicates the reactive power demand of the load, Representation Node exist The voltage amplitude at the moment, Indicated in Time Node The energy content of the energy storage system, , Respectively expressed in Always access the node The active power output of the photovoltaic power generation device and the active power output of the wind turbine; b) Action Space: Defining actions ; in, Is fast network in The action of the moment, Is the slow network in Momentary action; ； in, Indicates the control rate of the reactive power of the photovoltaic inverter, Indicates the control rate of the active power of the inverter of the wind power generation device; Indicates the active power control rate of the energy storage device; For photovoltaic and wind turbine inverters, first use and Get the available reactive power, and then get the reactive power injected into the ADN under the corresponding control rate , ; ， ； in, Represents the reactive power available from a photovoltaic power generation device, Represents the reactive power available to the wind turbine; are the rated capacities of photovoltaic and wind turbine inverters respectively, and represents the control rate; For energy storage systems, the control rate The actual active power injected into the ADN is obtained under the influence of ; if but ; if but ; ； in Indicates the position of the tap contact of the OLTC. Indicates the number of capacitor banks connected to the ADN, that is, Determine the tap of the on-load voltage regulator OLTC location, through Determine the number of groups of capacitors connected to the distribution network ; c) Transition probability: ； Transition probability Further broken down into: ； in, The transfer function describing the environment, i.e. the ADN state, The transition function describing the internal state of the fast network, A transition function describing the internal state of the slow network; d) Counter: For slow networks, introduce a counter to keep track of time steps; According to the result of the counter, when the hour time scale of the slow network action is reached, the slow network should take a new action; at other time steps, it maintains the state of the previous action; If the time step reaches the hour unit then ; If the time step is in minutes then , then there are action choices for the slow network: 。 6. The multi-timescale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.

1. Establishment of redundant coordination matrix; Features agents, the initial redundant coordination matrix For one The identity matrix of ； The above formula means that initially, each agent only refers to its own actions and has no influence on each other; Step 4.

2. Coordination of actions between multiple agents; At some point , Agent The action is represented as a vector , ; The actions of all agents are concatenated to form a dimension The redundant action matrix is ​​connected through the communication matrix Coordinate and produce coordinated actions : ； in It is The redundant coordination matrix at the moment represents the influence weight of each agent on the actions of other agents; and are the number of continuous actions and discrete actions respectively; Step 4.

3. Update of redundant coordination matrix; The redundant coordination matrix is ​​updated according to the system performance after each action is performed; The system's performance indicators are rewarded ; Each time it is updated, the matrix is ​​added with noise , and then according to the performance Adjust the matrix: ； Among them, noise is a random matrix following a normal distribution: ； in, Represents the parameters of the normal distribution. In order to ensure that the weights in the communication matrix are within a preset reasonable range and keep the sum of each row equal to 1, the communication matrix needs to be normalized; Assume After the matrix of the step is updated, the new matrix is , the normalization process is as follows: ； in, Representation Matrix No. Line The elements of the column, is the normalized matrix.

7. The multi-timescale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: In step 5, during the training process, the fast network and the slow network use their respective target Q networks , To update the Q value, the target Q value and The calculation formula is as follows: ； ； in It’s an instant reward. represents the discount factor; Indicates that the fast control target Q network is at the input Output the Q value of all possible actions, and select the action that maximizes the Q value from these actions , and then use this maximized Q value to calculate the target Q value; Indicates that the slow control target Q network is at the input Output the Q value of all possible actions, and select the action that maximizes the Q value from these actions This maximized Q value is then used to calculate the target Q value; In order to optimize the network parameters of the fast network and the slow network, the loss function is established using the target Q value and Q value for the two networks respectively. and , and then use gradient descent to minimize the loss function; ； ； ； ； in is the learning rate, represents the expected value, Indicates the fast control Q network in the network parameters Next, in the state Take action when The Q value obtained; Indicates the slow control Q network in the network parameters Next, in the state Take action when The Q value obtained is is the loss function of the fast control Q network with respect to the parameter The gradient of is the loss function of the slow control Q network with respect to the parameter The gradient of In order to stabilize the training process, the target Q network is quickly controlled Parameters and slow control target Q network Parameters Instead of updating it in every training step, it updates the parameters of the current fast Q network every preset number of time steps. and slow control Q network parameters Copy it, that is: ； ； The priority experience replay introduced in the Dueling Deep Q-Network algorithm is as follows: Experience replay stores experience tuples , through the experience evaluation function, different priorities are given to different experience tuples, so that the experience that has a greater impact on the learning process is preferentially selected for playback and training; Each time a new experience is stored in the buffer, its error is calculated and a priority is assigned to each experience based on the error; Group experience, its experience evaluation function for; ； in, represents the target Q network input , output the Q value of all possible actions, and select the action that maximizes the Q value from these actions This maximized Q value is then used to calculate the target Q value; The value of and Get the average value; Indicates that the Q network is in the network participation state Take action when The Q value obtained; The value of and Average is obtained; No. The probability of experience sampling corresponding to the group experience for: ； The sum of the priorities of all experiences is used to normalize the priority of each experience; is the total number of experiences in the buffer; an importance sampling weight is used when updating the Q network Correct the sampled samples, the calculation process is as follows: ； is a parameter that controls the smoothness of the weights; The Thompson sampling introduced in the Dueling Deep Q-Network algorithm is as follows: Thompson sampling naturally combines exploration and exploitation through random sampling. Even if the current average reward of an action is high, other actions may still be sampled due to the uncertainty of its distribution. Therefore, the choice of action is dynamically adjusted according to its uncertainty, and the specific steps are as follows: Set each action Rewards The distribution is Beta distribution: ； in, Indicates action Based on the reward The distribution of , Represent the number of successes and failures respectively, with an initial value of 1; whenever an action needs to be selected, sample from the posterior reward distribution of each action and select the action with the highest reward, that is, in a given state Next, from the reward distribution of each action Sampling is performed to generate a sample value for each action Then select the action with the largest sample value ,Right now: For each action , sample a value from its corresponding Beta distribution: ； in, is a sample value drawn from a Beta distribution, and then the action with the largest sample value is selected: ； Rewards based on actions To update the corresponding Beta distribution parameters; If the reward If it is a positive number, it increases the number of successes of the action. : ； If the reward If it is zero or negative, the number of failures of the action will be increased. : 。 8. The multi-timescale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: In step 5, the training is divided into a multi-time scale action process, that is, a multi-time scale decision process and a network parameter update process based on a single-layer Markov model, as follows: The multi-time scale decision-making process based on single-layer Markov is as follows: At a certain moment, the active distribution network inputs the state of the distribution network to the intelligent agent for multi-time scale control through the active distribution network voltage optimization model. The time counter of the intelligent agent outputs a control signal according to the time scale for the activation of the fast network and the slow network. The fast network and the slow network of the intelligent agent randomly give control actions according to the state. After the two network actions are integrated, the redundant coordination machine is used to coordinate the actions, and then the action is applied to the active distribution network voltage optimization model. The distribution network completes the state update according to the state transfer principle; at this time, a reward value is obtained using the new state; In this process, the state of the distribution network , Actions acting on the voltage optimization model of the active distribution network , Reward Value , New Status Composing experience tuples , and then use the experience evaluation function to evaluate the experience tuple to obtain an evaluation result of the experience tuple, and store the experience tuple into the experience playback mechanism according to the evaluation result; Repeat this process until the upper limit of the experience replay mechanism is reached; Then the agent network parameter update process is carried out, as follows: Extract small batches of experience tuples from the experience replay mechanism; then establish the reward distribution of actions based on rewards and actions, and then select actions for updating parameters based on the reward distribution of actions; complete the update of network parameters based on the internal structure of the intelligent agent; and at the same time, according to the action and reward value Update the coordination matrix of the redundant coordination mechanism.

9. The multi-timescale voltage regulation method based on deep reinforcement learning according to claim 1, characterized in that: In step 5, the trained intelligent agent and the coordination matrix adjusted by the redundant coordination mechanism are obtained. The network parameters and coordination matrix of the intelligent agent remain fixed after training. The active distribution network voltage optimization model inputs the state to the trained intelligent agent, and the intelligent agent gives a control action. The coordination matrix adjusted by the redundant coordination mechanism directly acts on the active distribution network voltage optimization model, that is, the control is completed according to the multi-time scale decision-making process constructed based on the single-layer Markov.

10. A multi-timescale voltage regulation system based on deep reinforcement learning, characterized in that: Includes the following modules: A model building module is used to model the active distribution network with a large amount of renewable energy and establish a voltage optimization model for the active distribution network; The agent building module is used to build agents with continuous and discrete action characteristics with time series characteristics; The intelligent agent includes two networks, namely a slow network for discrete control on the hourly time scale for on-load voltage regulators and capacitor bank dispatching, and a fast network for continuous control on the minute-level time scale for inverter and electric energy storage system dispatching; the fast network and slow network of the intelligent agent have two internal networks, which are defined as fast control Q network, fast control target Q network, slow control Q network, and slow control target Q network respectively; The interactive process description module uses a single-layer Markov model to construct a multi-time scale decision process to describe the interactive process between the active distribution network voltage optimization model and the intelligent agent, and introduces the distribution network state, fast network state, and slow network state. A state transfer function with three coupled states is specially designed, and a time counter is used to separate the fast time scale from the slow time scale. The output of the time counter is used as an activation signal to control the slow network. Multi-agent cooperative control system building block, used to propose a redundant multi-agent cooperative control system for multi-timescale voltage regulation based on deep reinforcement learning for active distribution networks; Use multiple agents in the system to perform the same voltage regulation task, then adjust the action output of the agents through a redundant coordination mechanism, and finally apply the coordinated actions to the active distribution network; And the voltage control module of the active distribution network is used to apply the trained intelligent agent, combined with the redundant multi-agent collaborative control system, to the multi-time scale voltage control of the active distribution network to achieve multi-time scale voltage regulation.

Images