A soft coordinated control method for on-load voltage-changing transformer and photovoltaic inverter

Abstract

A soft cooperative control method for an on-load voltage-changing transformer and a photovoltaic inverter, comprising: S1: establishing a soft cooperative control framework; determining the action logic of OLTC and determining the dead zone range ΔV of LDC control db and action delay TD t ; Determine the collaborative control framework of OLTC and photovoltaic inverter; S2: Establish a memory-based Markov decision process with distribution network voltage management as the background; S3: Establish a cyclic soft actor-critic algorithm; Determine the structure of the RSAC algorithm, which includes the actor network, the critic network, the value network, and the experience pool; Determine the parameter update method of each network, update the parameters of the critic network through policy evaluation, and update the parameters of the actor network through policy improvement; S4: Train the agent according to the algorithm; Use the offline training and online execution method to put the trained agent into use. The present invention can appropriately adjust the reactive power output of the inverter to "softly" coordinate the tapping operation of the OLTC to achieve the purpose of coordinated voltage regulation and line loss minimization.

Description

Technical Field

[0001] The invention relates to a soft coordinated control method of an on-load voltage regulating transformer and a photovoltaic inverter. Background Art

[0002] The increasing penetration of photovoltaics in distribution networks has brought opportunities for reducing carbon emissions, but it has also brought challenges to the voltage regulation of distribution networks. At present, the voltage over-limit problem has become the main obstacle to increasing the penetration of photovoltaics in distribution networks. In traditional distribution network voltage management systems, the voltage regulation of distribution networks usually relies on the tap switch of the upstream on-load tap changer (OLTC), among which the most commonly used OLTC control logic is the line droop control (LDC) method. Specifically, an analog circuit is used in the OLTC to simulate the voltage drop of the distribution line. Therefore, the OLTC can sense the change of remote voltage through local measurement. If the remote voltage exceeds the allowable voltage range for a certain period of time, the OLTC will adjust the position of its tap to restore the remote voltage to the allowable range. These widely deployed OLTCs were originally designed to compensate for voltage changes caused by slow load changes. This method is fully applicable in situations where the penetration of photovoltaics is low. However, with the continuous increase in the penetration of photovoltaics, a large number of reverse flows and uneven photovoltaic output have occurred, and it is difficult to solve the overvoltage problem of the distribution network by relying solely on OLTC. In addition, the rapid fluctuation of PV output may further induce OLTC tap over-action, thereby accelerating the aging of the equipment. Compared with OLTC with slow response speed and discrete tap action, PV inverters can smoothly adjust their reactive power output in real time, providing more flexible possibilities for voltage management of distribution networks.

[0003] Currently, most algorithm-based methods require full control of newly installed PV inverters and previously deployed voltage regulators (such as OLTCs). Therefore, traditional voltage regulators that have been in operation for many years must completely overturn their existing operating rules before they can be controlled by new algorithms. At the same time, the corresponding upgrade of traditional equipment will make on-site implementation more complicated, thereby increasing costs. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the present invention proposes a learning-based soft collaborative control method, which fully considers previously deployed equipment and existing operating rules.

[0005] In order to achieve the coordinated control of OLTC and photovoltaic inverter, the present invention establishes a learning-based soft collaborative control method on the basis of traditional deep reinforcement learning, which is used to online control the photovoltaic inverter to output reactive power to achieve the purpose of OLTC and photovoltaic inverter coordination.

[0006] In order to achieve the above object, the technical solution of the present invention is:

[0007] A soft coordinated control method for an on-load voltage-changing transformer and a photovoltaic inverter, the method comprising the following steps:

[0008] S1: Establish a soft cooperative control framework. Determine the action logic of OLTC. OLTC adopts the most commonly used LDC strategy for control. Determine the resistance R and reactance X in the LDC control circuit. est =V0-I0(R+jX) calculate the remote voltage V est , where V0 and I0 represent the voltage and current at the OLTC. Determine the dead band range ΔV of LDC control db and action delay TD t . Determine the coordinated control framework of OLTC and PV inverter.

[0009] S2: Establish a memory-based Markov decision process with distribution network voltage management as the physical background. Determine the Markov decision process tuple with distribution network voltage management as the physical background, which includes the action A of the intelligent agent. t , the state S of the Markov decision process t , the agent’s reward R t , and the probability of state transition ρ. Determine the historical trajectory K according to which the decision is taken t In order to enable the agent to fully explore the action space and avoid falling into the local optimum, entropy is added to the Markov decision process.

[0010] S3: Establishment of Recurrent Soft Actor-critic (RSAC) algorithm. Determine the structure of RSAC algorithm, which includes actor network, critic network, value network, and experience pool. Determine the parameter update method of each network, update the parameters of critic network through policy evaluation, and update the parameters of actor network through policy improvement.

[0011] S4: Train the agent according to the algorithm. Use the offline training and online execution method. Put the trained agent into use.

[0012] Furthermore, in step S1, the construction of the soft collaborative control framework includes the following processes:

[0013] S1-1: The present invention proposes a new soft cooperative control framework. Within this framework, the OLTC will retain its existing tap action control logic, and the inverter can indirectly control its reactive power output to assist the OLTC tap action, with the goal of collaborative control to stabilize the voltage of the distribution network and minimize line losses. In order to achieve this soft coordination mechanism, the present invention formulates the voltage control problem as a memory-based Markov decision process, in which the inverter interacts with the distribution system (environment) equipped with the OLTC by outputting reactive power. The GRU-DNN-based agent does not use the current system state, but acts for the inverter based on the historical trajectory observed so far (i.e., a series of system states and actions). When the inverter outputs reactive power to change the state, the agent will evaluate the reward for its output action at the current moment. The changed system state and the corresponding action will be fed back to the agent as a historical trajectory, and ultimately help the agent make the action decision for the next moment at the next moment.

[0014] In step S2, the establishment of the memory-based Markov decision process includes the following process:

[0015] S2-1: Establish a memory-based Markov decision process tuple. The distribution network voltage management problem using a soft control architecture can be solved using the tuple To indicate that and Represent the set of actions, states, rewards, and state transition probabilities respectively.

[0016] In the present invention, the action is the reactive power output by the inverter. At time t, the action output by the agent is the reactive power output of the photovoltaic inverter cluster Right now The upper and lower limits of the output action are and At time t, the complete Markov state can be expressed as:

[0017]

[0018] in, and Indicates the active and reactive power of the load at time t; Indicates the photovoltaic output at time t; V t Represents the voltage of each node in the distribution network at time t; Z t Indicates the tap position of OLTC at time t; TD t Represents the delay value at time t. The soft cooperative control objectives of the present invention are mainly two: voltage does not exceed the limit and network loss is minimized. The reward function set according to the objectives is as follows:

[0019]

[0020]

[0021] When the voltage of the distribution network exceeds the limit, the reward value is calculated according to formula (2); when the voltage of the distribution network exceeds the limit, the reward value is calculated according to formula (3). Among them, M<0 represents the penalty coefficient of voltage exceeding the limit; and V are the maximum and minimum values ​​of the system voltage respectively; represents the voltage of the jth node at time t; is the set of nodes; λ>0 is the reward coefficient when the voltage does not exceed the limit and the network loss is reduced; Represents taking action A at time t t The resulting line loss represents the line loss obtained without taking any action. The state of the system changes from S t Transfer to S t+1 , accompanied by a reward R t and action A t This state transition can be expressed as follows:

[0022] (S t+1 ,R t )~ρ(·|S t ,A t ) (4)

[0023] S2-2: Establishing historical trajectories. Since the action logic of OLTC is about time coupling, the agent can make decisions by establishing historical trajectories:

[0024]

[0025] From formula (5), we can see that K t It contains a series of states and actions. t The probability distribution of the output action π φ (·|K t ) instead of directly outputting actions. Actions are randomly selected from a generated probability distribution:

[0026] A t ~π φ (·|K t ) (6)

[0027] where π φ represents the policy function, and φ is the parameter of the policy network.

[0028] S2-3: Establish a Markov decision process that takes entropy into account. Different from the traditional goal of deep reinforcement learning, which is to maximize the cumulative rewards, the present invention considers entropy in the Markov decision process to enable the agent to fully explore the action space and avoid falling into the local optimum too early. The learning goal is defined as follows:

[0029]

[0030] τ represents the φ The future trajectory that can be obtained; N represents the length of the trajectory; α>0 is the temperature coefficient that weighs the reward and entropy; γ∈(0,1) is the discount factor; represents the entropy of the probability distribution. According to the goal shown in formula (7), according to the strategy π φ The amount that can be obtained in the future can be expressed by the value function and the action value function, as shown in formula (8) and formula (9) respectively:

[0031]

[0032]

[0033] In step S3, the establishment of the RSAC algorithm includes the following contents:

[0034] S3-1: Establish the structure of RSAC algorithm. The RSAC algorithm used in the present invention is constructed based on the SAC algorithm. The difference is that RSAC adds time coupling on the basis of SAC. The RSAC algorithm includes an actor network and a critic network. The actor network is responsible for making decisions and taking actions; the critic network is responsible for evaluating the quality of the actions taken by the actor network.

[0035] The actor network is composed of a deep neural network embedded with GRU, which is a type of recurrent neural network. The mapping rule of GRU is as follows:

[0036]

[0037] in represents the mapping rule of GRU, φ r represents the network parameters, H t represents the hidden layer state of GRU at time t. After formula (10), the variable-dimensional historical trajectory K t is mapped into a fixed-dimensional H t Then H t As the input of the deep neural network, the output is the probability distribution of the action, that is, the mean and variance. The mapping rule of DNN is as follows:

[0038]

[0039] μ t and σ t denote the mean and variance of the action distribution, φ d represents the parameters of the deep neural network, Represents the mapping rule of the deep neural network. The strategy π of the actor network φ It can be expressed as:

[0040]

[0041] where φ=[φ r ,φ d ],(S t ,A t-1 ,H t-1 ) The specific action taken is ultimately randomly drawn from the distribution:

[0042]

[0043] ε t and σ t Represent the mean and variance of the standard Gaussian distribution respectively. The value range of the action is The critic gives a Q value to evaluate the performance of the given strategy π φ Next Take action A t The quality of. It can be expressed by the following formula:

[0044]

[0045] Similarly, the value network shown in formula (8) can be expressed as follows:

[0046]

[0047] In order to train the parameters of the network, the present invention adopts the method of extracting small batches of data from the experience pool and performing stochastic gradient descent to update the parameters. Indicates that there is a series of tuples

[0048] S3-2: Determine the update method of each network parameter. In the RSAC algorithm proposed in the present invention, the parameters φ of the policy network and the parameters θ of the critic network are updated iteratively through policy evaluation and policy improvement. The parameter θ for a given strategy π φ Has better action Q-value evaluation performance. Improved critic network We further update the actor network parameters φ through strategy improvement, and the updated actor network π φ′Better performance in action selection.

[0049] The value function and action value function described by equations (8) and (9) satisfy the above equations:

[0050]

[0051]

[0052]

[0053] The calculation formula for entropy is as follows:

[0054]

[0055] in Represents in strategy π φ The probability distribution of the next action is obtained by equations (16) and (17):

[0056]

[0057]

[0058] in Representing tuples From the experience pool Randomly selected from express From the experience pool Randomly drawn from . According to equations (19) and (20), the parameters θ and ψ can be updated using the stochastic gradient descent method:

[0059]

[0060]

[0061] When updating the parameter θ, the parameter Keep constant. Parameter The update method is as follows:

[0062]

[0063] where η,β∈(0,1) represents the learning rate of mini-batch stochastic gradient descent. After sufficient training, the critic network The Q value can be output for action evaluation. The larger the Q value, the better the action. In order to improve the performance of the actor network, the most intuitive way is to make the distribution of actions and the distribution of Q values ​​the same. This will make the probability of actions corresponding to large Q values ​​larger. Therefore, the goal of the policy distribution is:

[0064]

[0065] Where exp{} is an instruction fetch operation, It is guaranteed that the probability is non-negative, and will make In addition, this design can make KL divergence can be used to measure the similarity between two probabilities, so the update method of the policy network can use KL divergence:

[0066]

[0067] Where D KL represents the KL divergence between two probability densities. Yes and A t Unrelated expressions, so you can use To simplify the representation. According to (25), the loss function can be expressed as:

[0068]

[0069] Substituting formula (13) into formula (26), we can obtain:

[0070]

[0071] Finally, the update method of the policy network is as follows:

[0072]

[0073] In order to achieve coordinated voltage regulation, current methods often rely on a decision algorithm to completely take over the control of all devices, which requires the OLTC to abandon the existing tap switch logic and upgrade accordingly. To bridge this gap, the present invention proposes a soft coordination framework. Specifically, the decision command only acts on the inverter, and the OLTC retains its own operating rules, indirectly controlling the change of system voltage by appropriately adjusting the reactive power output of the inverter. The method achieves soft coordination by establishing an improved actor-critic algorithm to train the intelligent agent model of the inverter. The well-trained intelligent agent model can appropriately adjust the reactive power output of the inverter to "softly" coordinate the tap operation of the OLTC, ultimately achieving the purpose of coordinated voltage regulation and line loss minimization.

[0074] The beneficial effects of the present invention are:

[0075] 1. After sufficient training, the recurrent deep strategy network can learn the different characteristics and time coupling relationship of OLTC and photovoltaic inverter clusters in the voltage management process, so that each fully trained intelligent agent can achieve coordinated control of the upstream OLTC and downstream photovoltaic inverter clusters of the distribution network by simply adjusting the reactive output of the photovoltaic inverter during the voltage regulation process.

[0076] 2. Compared with the existing coordinated voltage regulation method for heterogeneous devices in the same network, the present invention does not need to break the voltage regulation logic of the original equipment, which is more conducive to the grid connection of photovoltaic inverters. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of the LDC control principle of the OLTC of the present invention.

[0078] Figure 2 It is a logic block diagram of LDC control of OLTC of the present invention.

[0079] Figure 3 It is a collaborative control framework diagram of the OLTC and photovoltaic inverter of the present invention.

[0080] Figure 4 This is a topological structure diagram of the IEEE33 node distribution network adopted by the present invention.

[0081] Figure 5 It is a convergence curve diagram of the algorithm proposed in the present invention.

[0082] Figure 6 This is a 24-hour photovoltaic output and prediction result diagram when the photovoltaic power generation fluctuates violently.

[0083] Figure 7 This is a comparison diagram of the OLTC tap position changes over 24 hours when the photovoltaic power generation fluctuates violently.

[0084] Figure 8 This is a comparison diagram of the voltage control effect of the present invention for 24 hours when the photovoltaic power generation fluctuates violently.

[0085] Fig. 9 It is a comparison diagram of the reactive power output of the inverter of the present invention for 24 hours when the photovoltaic power fluctuates violently.

[0086] Fig.10 It is a 24-hour photovoltaic output and prediction result diagram when the photovoltaic power slightly fluctuates according to the present invention.

[0087] Fig.11 This is a comparison diagram of the OLTC tap position changes over 24 hours when the photovoltaic power generation of the present invention is slightly fluctuating.

[0088] Fig.12 This is a comparison diagram of the voltage control effect of the present invention for 24 hours when the photovoltaic power generation fluctuates slightly.

[0089] Fig.13 This is a 24-hour inverter output reactive power comparison diagram when the photovoltaic power of the present invention has a slight fluctuation.

[0090] Fig.14 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0091] The present invention will be further described below in conjunction with the accompanying drawings.

[0092] Reference Figure 1 , Figure 2 , Figure 3 , a soft coordinated control method of an on-load tap-changing transformer and a photovoltaic inverter, the method comprising the following steps:

[0093] S1: Establish a soft cooperative control framework. Determine the action logic of OLTC. OLTC adopts the most commonly used LDC strategy for control. Determine the resistance R and reactance X in the LDC control circuit. est =V0-I0(R+jX) calculate the remote voltage V est , where V0 and I0 represent the voltage and current at the OLTC. Determine the dead band range ΔV of LDC control db and action delay TD t . Determine the coordinated control framework of OLTC and PV inverter.

[0094] S2: Establish a memory-based Markov decision process with distribution network voltage management as the physical background. Determine the Markov decision process tuple with distribution network voltage management as the physical background, which includes the action A of the intelligent agent. t , the state S of the Markov decision process t , the agent’s reward R t , and the probability of state transition ρ. Determine the historical trajectory K according to which the decision is taken t In order to enable the agent to fully explore the action space and avoid falling into the local optimum, entropy is added to the Markov decision process.

[0095] S3: Establishment of Recurrent Soft Actor-critic (RSAC) algorithm. Determine the structure of RSAC algorithm, which includes actor network, critic network, value network, and experience pool. Determine the parameter update method of each network, update the parameters of critic network through policy evaluation, and update the parameters of actor network through policy improvement.

[0096] S4: Train the agent according to the algorithm. Use the offline training and online execution method. Put the trained agent into use.

[0097] In order to enable those skilled in the art to better understand the present invention, the example analysis includes the following components:

[0098] 1. Case Description

[0099] This example uses Figure 4 The IEEE33-node distribution system shown in the figure is the research object, and two different types of photovoltaic fluctuation curves are selected as the test environment test. In each photovoltaic fluctuation case, this example uses two control groups for comparison, namely the traditional droop control (i.e., OLTC uses the LDC control method and the photovoltaic inverter uses the droop control method) and the two-stage control method (i.e., solving the OLTC tap position through the optimization algorithm on a long time scale and then eliminating the voltage limit through the photovoltaic inverter output reactive power on a short time scale).

[0100] 2. Analysis of Actual Control Behavior

[0101] The example uses a distribution network with 7 distributed photovoltaic inverters to verify the effectiveness of the proposed RSAC algorithm in coordinating OLTC and photovoltaic inverter clusters, as well as in distribution network voltage control and distribution network line loss optimization. A distribution network flow calculation model is built in OpenDSS and regarded as an environment, and the training process of the RSAC algorithm is established in the Python environment. In addition, the maximum active power and reactive power that can be output by the photovoltaic inverters distributed at each node are shown in Table 1; the discount coefficient γ and the learning rate λ of the critic network in the algorithm proposed in this invention Q , the learning rate λ of the value function network V , the learning rate λ of the policy network π , the learning rate β of the target value network is shown in Table 2.

[0102]

[0103] Table 1

[0104]

[0105] Table 2

[0106] 3. Case Comparative Analysis

[0107] This case first compares the droop control method, the two-stage method, and the RSAC algorithm established by the present invention in terms of coordinated OLTC, voltage control, reactive power output, network loss, etc. under the condition of severe photovoltaic fluctuations. We call the two-stage method the control group No. 1 and the droop control method the control group No. 2. Figure 7The comparison between the proposed algorithm and the other two control groups in terms of OLTC coordination is shown. As can be seen from the figure, the two-stage method adds the minimization of network loss to the optimization objective of the first stage, so the position of the OLTC tap will be higher. In the second stage, in order to eliminate the voltage over-limit risk caused by the high tap position, the photovoltaic inverter will output excess reactive power to compensate for the tap in the first stage (such as Fig. 9 ), and when the capacity is insufficient, a voltage over-limit occurs (as shown in Figure 8 As shown). However, the RSAC algorithm and droop control method proposed in the present invention do not have such problems. However, since the droop control method changes the reactive power output of the photovoltaic inverter with the voltage fluctuation, it does not take into account the coordination of the OLTC and the photovoltaic inverter cluster, which leads to over-action of the OLTC tap and increased line loss. The comparison results of the distribution network line loss corresponding to the three methods are shown in Table 1, and the number of actions of the OLTC tap is shown in Table 2.

[0108]

[0109] Table 1

[0110]

[0111] Table 2

[0112] After verifying the effectiveness of the algorithm proposed in the present invention under the condition of severe photovoltaic fluctuations, the effectiveness of the algorithm proposed in the present invention is further verified under the condition of slight photovoltaic fluctuations. Fig.11 The distribution of OLTC tap positions of the three methods within one day is shown. Since the photovoltaic fluctuation is moderate at this time, the two-stage method is relatively accurate in prediction, so there is no voltage over-limit situation. When the photovoltaic fluctuation is moderate, the network loss of the three methods is shown in Table 3, and the number of tap operations is shown in Table 4.

[0113]

[0114] Table 3

[0115]

[0116] Table 4

[0117] It can be seen from Table 3 and Table 4 that the RSAC algorithm proposed in the present invention is superior to other methods in terms of reducing network loss and the number of OLTC tap operations.

[0118] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also includes equivalent technical means that can be thought of by those skilled in the art based on the inventive concept.

Claims

1. A soft coordinated control method for an on-load tap-changing transformer and a photovoltaic inverter, comprising the following steps: S1: Establish a soft cooperative control framework; determine the action logic of OLTC, OLTC adopts the most commonly used LDC strategy for control; determine the resistance R and reactance X in the LDC control circuit, and use V est =V0-I0(R+jX) calculate the remote voltage V est , where V0 and I0 represent the voltage and current at the OLTC; determine the dead zone range ΔV of the LDC control db and action delay TD t ; Determine the coordinated control framework of OLTC and PV inverter; S2: Establish a memory-based Markov decision process with distribution network voltage management as the physical background; determine the Markov decision process tuple with distribution network voltage management as the physical background, which includes the action A of the intelligent agent t , the state S of the Markov decision process t , the agent’s reward R t , and the probability of state transition ρ; Determine the historical trajectory K according to which the decision is taken t ; In order to enable the agent to fully explore the action space and avoid falling into the local optimum, entropy is added to the Markov decision process; S3: Establish a Recurrent Soft Actor-critic (RSAC) algorithm; determine the structure of the RSAC algorithm, which includes the actor network, critic network, value network, and experience pool; determine the parameter update method of each network, update the parameters of the critic network through policy evaluation, and update the parameters of the actor network through policy improvement; S4: Train the agent according to the algorithm; adopt the method of offline training and online execution; put the trained agent into use; The establishment of the soft collaborative control framework described in step S1 includes the following processes: A soft cooperative control framework is proposed. In this framework, OLTC will retain its existing tap action control logic, and the inverter can indirectly control its reactive power output to assist the tap action of OLTC. The cooperative control makes the voltage of the distribution network stable and minimizes the line loss. In order to realize this soft coordination mechanism, the voltage control problem is formulated as a memory-based Markov decision process, in which the inverter interacts with the distribution system equipped with OLTC by outputting reactive power. The GRU-DNN-based agent does not use the current system state, but takes actions for the inverter based on the historical trajectory observed so far. When the inverter outputs reactive power to change the state, the agent will evaluate the reward of its output action at the current moment. The changed system state and the corresponding action will be fed back to the agent as the historical trajectory, which will eventually help the agent make the action decision at the next moment. The step S2 of establishing a memory-based Markov decision process includes the following steps: S2-1: Establish a memory-based Markov decision process tuple; the distribution network voltage management problem using a soft control architecture can be solved using the tuple To indicate that and Represent the set of actions, states, rewards, and state transition probabilities respectively; Action is the reactive power output by the inverter; at time t, the action output by the agent is the reactive power output of the photovoltaic inverter cluster Right now The upper and lower limits of the output action are and At time t, the complete Markov state can be expressed as: in, and Indicates the active and reactive power of the load at time t; Indicates the photovoltaic output at time t; V t Represents the voltage of each node in the distribution network at time t; Z t Indicates the tap position of OLTC at time t; TD t represents the delay value at time t; there are two main objectives of soft collaborative control: voltage does not exceed the limit and network loss is minimized; the reward function set according to the objectives is as follows: When the voltage of the distribution network exceeds the limit, the reward value is calculated according to formula (2); when the voltage of the distribution network exceeds the limit, the reward value is calculated according to formula (3); where M<0 represents the penalty coefficient of voltage exceeding the limit; and V are the maximum and minimum values ​​of the system voltage respectively; represents the voltage of the jth node at time t; is the set of nodes; λ>0 is the reward coefficient when the voltage does not exceed the limit and the network loss is reduced; Represents taking action A at time t t The resulting line loss represents the line loss obtained without taking any action; the state of the system changes from S t Transfer to S t+1 , accompanied by a reward R t and action A t ; This state transition can be expressed as follows: (S t+1 ,R t )~ρ(·|S t ,A t ) (4) S2-2: Establish historical trajectories. Since the action logic of OLTC is about time coupling, the agent can make decisions by establishing historical trajectories: It can be seen from formula (5) that K t Contains a series of states and actions; the agent passes K t The probability distribution of output actions π φ (·|K t ) instead of outputting actions directly; actions are randomly selected from a generated probability distribution: where π φ represents the policy function, φ is the parameter of the policy network; S2-3: Establish a Markov decision process that takes entropy into account; different from the traditional deep reinforcement learning goal of maximizing the cumulative reward; considering entropy in the Markov decision process aims to enable the agent to fully explore the action space and avoid falling into the local optimum too early; the learning goal is defined as follows: τ represents the φ The future trajectory that can be obtained; N represents the length of the trajectory; α>0 is the temperature coefficient that weighs the reward and entropy; γ∈(0,1) is the discount factor; represents the entropy of the probability distribution; according to the goal shown in formula (7), according to the strategy π φ The amount that can be obtained in the future can be expressed by the value function and the action value function, as shown in formula (8) and formula (9) respectively:

2. The soft coordinated control method of an on-load tap-changing transformer and a photovoltaic inverter according to claim 1, characterized in that: The establishment of the RSAC algorithm described in step S3 includes the following contents: S3-1: Establish the structure of RSAC algorithm; RSAC algorithm is built on the basis of SAC algorithm, the difference is that RSAC adds time coupling on the basis of SAC; RSAC algorithm includes actor network and critic network; actor network is responsible for making decisions and taking actions; critic network is responsible for evaluating the quality of the actions taken by actor network; The actor network is composed of a deep neural network embedded with GRU, which is a type of recurrent neural network; the mapping rule of GRU is as follows: in represents the mapping rule of GRU, φ r represents the GRU network parameters, H t represents the hidden layer state of GRU at time t; after equation (10), the variable-dimensional historical trajectory K t is mapped into a fixed-dimensional H t ; then H t As the input of the deep neural network, the output is the probability distribution of the action, that is, the mean and variance; the mapping rule of DNN is as follows: μ t and σ t denote the mean and variance of the action distribution, φ d represents the parameters of the deep neural network, represents the mapping rule of the deep neural network; the strategy π of the actor network φ It can be expressed as: where φ=[φ r ,φ d ],(S t ,A t-1 ,H t-1 ) The specific action taken is ultimately randomly drawn from the distribution: ε t and σ' t Represent the mean and variance of the standard Gaussian distribution respectively; the value range of the action is The critic gives a Q value to evaluate the performance of the given strategy π φ Next Take action A t The quality of; can be expressed by the following formula: Similarly, the value network shown in formula (8) can be expressed as follows: In order to train the parameters of the network, a small batch of data is extracted from the experience pool and the random gradient descent method is used to update the parameters; the experience pool is used Indicates that there is a series of tuples S3-2: Determine the update method of each network parameter; In the proposed RSAC algorithm, the parameters φ of the policy network and the parameters θ of the critic network are iteratively updated through policy evaluation and policy improvement; Through policy evaluation, the critic network The parameter θ for a given strategy π φ Better performance in action Q-value evaluation; improved critic network We further update the actor network parameters φ through strategy improvement, and the updated actor network π φ′ Better performance in action selection; The value function and action value function described by equations (8) and (9) satisfy the above equations: The calculation formula for entropy is as follows: in Represented in strategy π φ The probability distribution of the next action is obtained by equations (16) and (17): in Representing tuples From the experience pool Randomly selected from express is randomly drawn from the experience pool D; according to equations (19) and (20), the parameters θ and ψ can be updated using the stochastic gradient descent method: When updating the parameter θ, the parameter Keep constant; parameter The update method is as follows: where η,β∈(0,1) represents the learning rate of mini-batch stochastic gradient descent; after sufficient training, the critic network The Q value can be output for action evaluation. The larger the Q value, the better the action. In order to improve the performance of the actor network, the most intuitive way is to make the distribution of actions and the distribution of Q values ​​the same. This will make the probability of actions corresponding to large Q values ​​larger. Therefore, the goal of the policy distribution is: Where exp{} is an instruction fetch operation, It is guaranteed that the probability is non-negative, and will make In addition, this design can make KL divergence can be used to measure the similarity between two probabilities, so the update method of the policy network can use KL divergence: Where D KL represents the KL divergence between two probability densities; since Yes and A t Unrelated expressions, so you can use To simplify the representation; According to (25), the loss function can be expressed as: Substituting formula (13) into formula (26), we can obtain: Finally, the update method of the policy network is as follows:

Images