A static synchronous compensator control method based on composite deep reinforcement learning

Abstract

The application provides a static synchronous compensator control method based on a composite deep reinforcement learning method, the deep reinforcement learning method inputs data processed by a preposed convolutional neural network into a current network and a target network after experience pool and random sampling, updates parameters by using a loss function and a Kronecker factor approximation natural policy gradient descent, and finally outputs control of the current network. The deep reinforcement learning method can not only obtain higher reward returns, but also improve sample sampling efficiency, solves the problem of poor control effect of a distribution network static synchronous compensator, better realizes the control function of the distribution network static synchronous compensator, improves voltage stability and transient performance, optimizes the absorption or emission of reactive power of the distribution network static synchronous compensator in a power system, improves the stability of the power system, and improves power quality.

Description

Technical Field

[0001] This invention belongs to the field of power system control and relates to a deep reinforcement learning control method, which is applicable to the control of static synchronous compensators in power system distribution networks. Background Technology

[0002] The control input process of existing static synchronous compensator control methods is slow.

[0003] In addition, the control output results of existing static synchronous compensator control methods are poor.

[0004] Therefore, a static synchronous compensator control method based on composite deep reinforcement learning is proposed to solve the problems of slow input speed and poor output performance of static synchronous compensator control. Summary of the Invention

[0005] This invention proposes a control method for static synchronizing compensators (SSCs) based on composite deep reinforcement learning. It combines experience pool replay and an actor-commentator network structure with Kronecker factor confidence region for the control of SSCs in distribution networks. This method improves the control of SSCs in distribution networks, enhances voltage stability and transient performance, optimizes the absorption or generation of reactive power in the power system by SSCs, improves power system stability, and enhances power quality. The steps in its application are as follows:

[0006] Step (1): Preprocess the data to be collected;

[0007] Data collected from the distribution network The input is fed into the trained agent. For AC voltage deviation, DC voltage deviation, for Shaft current reference value for Shaft current reference value for Shaft current deviation, for Shaft current deviation; agent from input value Column vectors are obtained by collecting time series data. ; Column vector for:

[0008] (1)

[0009] in, for The column vector at time t, , , , , , , column vector The first element in; column vector The second element in; column vector The third element in; column vector The fourth element in; column vector The fifth element in; column vector The sixth element in; for AC voltage deviation at any given time; for DC voltage deviation at any given time; for Moment Shaft current reference value; for Moment Shaft current reference value; for Moment Shaft current deviation; for Moment Shaft current deviation;

[0010] Then, the collected one-dimensional time series data are arranged into a 6×150 data matrix; the 6×150 data matrix is ​​as follows:

[0011] (2)

[0012] in, ,for AC voltage deviation at any given time; ,for AC voltage deviation at any given time; ,for DC voltage deviation at any given time; ,for DC voltage deviation at any given time; ,for Moment Shaft current reference value; ,for Moment Shaft current reference value; ,for Moment Shaft current reference value; ,for Moment Shaft current reference value; ,for Moment Shaft current deviation; ,for Moment Shaft current deviation; ,for Moment Shaft current deviation; ,for Moment Shaft current deviation; for The column vector at time t, for The column vector at time points;

[0013] Secondly, rearrange the 6×150 data matrix into a 30×30 data matrix. , as the input matrix of the convolutional neural network ;matrix for:

[0014] (3)

[0015] in, for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points;

[0016] Next, the 30×30 matrix Normalization is performed to make the matrix Each element in the array is in the interval [0,1]; the process is as follows:

[0017] First, the matrix Process in blocks, starting from the first row, sequentially assembling the 6 column vectors into a new matrix. , It is a 6×6 square matrix with a total of 36 elements. Divided into 25 blocks, matrix for:

[0018] (4)

[0019] make ; From 1 to 25, For matrix In arrive ;

[0020] in, For matrix Block 1 in the middle; For matrix Block 2 in the middle; For matrix Block 3 in the middle; For matrix Block 4 in the middle;

[0021] For matrix Block 5 in the middle; For matrix Block 6 in the middle; For matrix Block 7 in the middle; For matrix Block 8 in the middle;

[0022] For matrix Block 9 in the middle; For matrix Block 10 in the middle; For matrix Block 11 in the middle; For matrix Block 12 in the middle;

[0023] For matrix Block 13 in the middle; For matrix Block 14 in the middle; For matrix Block 15; For matrix Block 16 in the middle;

[0024] For matrix Block 17 in the middle; For matrix Block 18 in the middle; For matrix Block 19 in the middle; For matrix Block 20 in the middle;

[0025] For matrix Block 21 in the middle; For matrix Block 22; For matrix Block 23; For matrix Block 24 in the middle;

[0026] For matrix Block 25;

[0027] for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points;

[0028] Then each matrix Perform the following processing:

[0029] (5)

[0030] in, For matrix middle OK Column elements, For matrix The minimum value among the 36 elements. For matrix The maximum value among the 36 elements; ; Normalized All 36 elements in the dataset are in the interval [0,1].

[0031] From this point on, the normalized matrix Linear transformation, making the matrix Each element in the array is in the interval [0, 255]; the process is as follows:

[0032] (6)

[0033] (7)

[0034] in, Indicates will Round down to the nearest integer. Natural numbers from 0 to 255 are used to facilitate processing by convolutional neural networks;

[0035] Step (2): The preprocessed matrix The data is input into a convolutional neural network (CNN) for feature classification; the data input into the CNN is a matrix. ,matrix The sequence of layers is: input layer, convolutional layer, ReLU activation layer, average pooling layer, perceptron model, convolutional layer, ReLU activation layer, pooling layer, Dropout layer, fully connected layer, fully connected layer, sigmoid activation layer, and output layer. The convolutional neural network then outputs labels for classification, categorizing them into three classes:

[0036] The first type: Deep reinforcement learning has been trained enough, that is, training is complete, and the convolutional neural network selects deep reinforcement learning methods for control;

[0037] The second type: Deep reinforcement learning has not had enough training, and the training is not completed. The convolutional neural network chooses the proportional-integral-differential method for control.

[0038] The third type: Deep reinforcement learning has a certain amount of training. Although the training is not complete, it can ensure the safety and stability of subsequent control. Convolutional neural networks still choose deep reinforcement learning methods for control.

[0039] Step (3): When the convolutional neural network selects a deep reinforcement learning method for control, the deep reinforcement learning output... shaft voltage and shaft voltage Controlling the PWM pulse signal generator: converting time series data After replaying the experience pool, it is possible to effectively break the correlation between samples, increase the diversity of parameter update directions and policy function distributions, reuse samples, increase the frequency of sample use and parameter update, and reduce training time; then, a temporary matrix is ​​obtained by randomly sampling from the experience pool. ,in yes The state at any given moment, For time series data ,Right now ; yes Actions at any moment for shaft voltage and shaft voltage ; yes The reward system evaluates the effectiveness of each action, and the reward is determined by recording the highest reward achieved during the deep reinforcement learning process. The reward design is as follows:

[0040] (8)

[0041] in, and They are respectively shaft voltage and shaft voltage The proportionality coefficient, for Shaft voltage reference value, for Shaft voltage reference value, As a reward;

[0042] yes The state at any given moment, for Then, put the temporary matrix The inputs are fed into both the target network and the current network, which include the actor network and the critic network. Whether it's the target network or the current network, the actor network will perform actions. or Input into the critic network, For the target network Actions at any moment In the current network Actions at any given moment; a network of critics judges and filters actions and states, introducing... The value is used to estimate the value of the strategy. Values ​​are represented in the state space. Next Action Set The value of using a network of critics as a transition from state and action Value mapping function, denoted as , The parameters are for the critic network; the critic network in the target network inputs the state and actions into the loss function, and updates the critic network based on backpropagation of the loss function. Commentator network parameters of the moment loss function for:

[0043] (9)

[0044] in, For the critic's network parameters, for Number of samples of the value for Commentator network parameters at any time for Values ​​are represented in the state space. Next Action Set The greatest value, for Values ​​are represented in the state space. Next Action Set The value of the network parameters; the loss function is the variance between the actual reward and the estimated reward. The smaller the loss function, the more accurate the network parameters are.

[0045] The current network updates the actor network using Kronecker factor approximation natural policy gradient descent, while the critic network updates bidirectionally with state and action and loss function. Finally, the current network outputs... and To control the PWM pulse signal generator, and in turn control the static synchronous compensator of the power distribution network;

[0046] When a convolutional neural network selects the proportional-integral-derivative (PID) method for control, the PID output... shaft voltage and shaft voltage Controlling PWM pulse signal generator: Based on proportional-integral-derivative (PID) shaft voltage and shaft voltage They are respectively:

[0047] (10)

[0048] (11)

[0049] in, for The proportionality factor of the shaft, for The proportionality factor of the shaft, for Integral coefficients of the axis, for Integral coefficients of the axis, for The differential coefficients of the axis, for The differential coefficients of the axis, for Value Shaft current deviation, for Value Shaft current deviation, for Value shaft voltage, for Value shaft voltage, For time;

[0050] Finally, the proportional-integral-differential output and This is used to control the PWM pulse signal generator, which in turn controls the static synchronous compensator of the power distribution network.

[0051] The present invention has the following advantages and effects compared with the prior art:

[0052] (1) Solve the problem of insufficient control effect of static synchronous compensator in distribution network.

[0053] (2) Improve voltage stability and transient performance.

[0054] (3) Optimize the absorption or generation of reactive power in the power system by the static synchronous compensator of the distribution network.

[0055] (4) Improve the stability of the power system and improve the quality of power. Attached Figure Description

[0056] Figure 1 This is a framework diagram of the static synchronous compensator control system for power distribution networks according to the method of the present invention.

[0057] Figure 2 This is a flowchart of the intelligent agent structure of the method of the present invention.

[0058] Figure 3 This is a flowchart of the convolutional neural network structure of the method of the present invention.

[0059] Figure 4 This is a flowchart of the deep reinforcement learning structure of the method of the present invention. Detailed Implementation

[0060] This invention proposes a static synchronous compensator control method based on composite deep reinforcement learning, which is described in detail below with reference to the accompanying drawings:

[0061] Figure 1This is a framework diagram of the static synchronous compensator control system for power distribution networks according to the method of this invention. First, [the following is a description of the system]. Input is fed into the trained agent; These are AC voltage deviation, DC voltage deviation, Shaft current reference value Shaft current reference value Shaft current deviation and Shaft current deviation. Then, the agent outputs... shaft voltage and shaft voltage The signal is then transferred to a PWM pulse signal generator. Finally, the PWM pulse signal generator controls the static synchronous compensator of the distribution network, thereby absorbing or generating reactive current that meets the requirements, achieving the purpose of dynamic reactive power compensation.

[0062] Figure 2 This is a flowchart of the intelligent agent structure of the method of the present invention. First, the intelligent agent starts from the input value... Time series data collected in China Then, the collected one-dimensional time series data were arranged into a 6×150 data matrix. Next, the 6×150 data matrix was rearranged into a 30×30 matrix. Next, consider a 30x30 matrix. Normalization is performed to make the matrix Each element in the matrix is ​​in the interval [0,1]. Therefore, the normalized matrix... Linear transformation, making the matrix Each element in the matrix is ​​in the interval [0, 255]. Finally, the 30×30 matrix processed above will be... The input is fed into a convolutional neural network, undergoes feature classification and proportional-integral-differential (PI) processing, and is then processed by deep reinforcement learning to output the final value. and .

[0063] Figure 3 This is a flowchart of the convolutional neural network structure of the method of this invention. First, the input matrix mentioned above... The data is fed into the input layer of a convolutional neural network. Then, the data sequentially passes through the input layer, convolutional layer, ReLU activation layer, average pooling layer, perceptron, convolutional layer, ReLU activation layer, pooling layer, Dropout layer, fully connected layer, fully connected layer, sigmoid activation layer, and output layer. Finally, the result is output to proportional-integral-differential (PID) and deep reinforcement learning algorithms.

[0064] Figure 4 This is a flowchart of the deep reinforcement learning structure of the method of this invention. First, [the flowchart is missing from the original text]. That is A temporary matrix is ​​obtained through experience pooling and random sampling. ,in yes The state at any given moment, yes Actions at any moment yes Momentary rewards and benefits yes The state at any given time. Then, the temporary matrix... The inputs are fed into the target network and the current network, which includes an actor network and a critic network, respectively. The critic network in the target network inputs the state and actions into the loss function. The current network updates the actor network using Kronecker factor approximation natural policy gradient descent, and the critic network in the current network updates bidirectionally with the loss function using the state and actions. Finally, the current network outputs... and .

[0065] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A static synchronous compensator control method based on composite deep reinforcement learning, characterized in that, The experience pool replay and the actor-critic network structure of the Kronecker factor confidence region are combined for the control of static synchronous compensators in distribution networks. The steps in the process are as follows: Step (1): Preprocess the data to be collected; then, arrange the collected one-dimensional time series data into a data matrix; next, rearrange the data matrix into a data matrix. , as the input data matrix of the convolutional neural network ; Next, the data matrix Perform normalization processing, and then... (The sentence is incomplete and requires more context to translate accurately.) Linear change; Step (2): The preprocessed matrix The input is fed into a convolutional neural network, which then performs feature classification. They are divided into the following three categories: The first type: Deep reinforcement learning has been trained enough, that is, training is complete, and the convolutional neural network selects deep reinforcement learning methods for control; The second type: Deep reinforcement learning has not had enough training, and the training is not completed. The convolutional neural network chooses the proportional-integral-differential method for control. The third type: Deep reinforcement learning has a certain amount of training. Although the training is not complete, it can ensure the safety and stability of subsequent control. Convolutional neural networks still choose deep reinforcement learning methods for control. Step (3): When the convolutional neural network selects a deep reinforcement learning method for control, the deep reinforcement learning output... shaft voltage and shaft voltage Controlling the PWM pulse signal generator: converting time series data After replaying the experience pool, a temporary matrix is ​​obtained by randomly sampling from the experience pool. ,in yes The state at any given moment, For time series data ; yes Actions at any given moment; yes Momentary rewards and benefits yes The state at each moment; then, the temporary matrix The inputs are fed into the target network and the current network, which includes the actor network and the critic network, respectively. The critic network in the target network inputs the state and actions into the loss function, and updates the loss function based on backpropagation. Commentator network parameters of the moment The current network updates the actor network using Kronecker factor approximation natural policy gradient descent, while the critic network updates bidirectionally using state and action with the loss function. Finally, the current network outputs... and To control the PWM pulse signal generator, and in turn control the static synchronous compensator of the power distribution network; When a convolutional neural network selects the proportional-integral-derivative (PID) method for control, the PID output... shaft voltage and shaft voltage Controlling PWM pulse signal generator: Based on proportional-integral-derivative (PID) shaft voltage and shaft voltage They are respectively: in, for The proportionality factor of the shaft, for The proportionality factor of the shaft, for Integral coefficients of the axis, for Integral coefficients of the axis, for The differential coefficients of the axis, for The differential coefficients of the axis, for Value Shaft current deviation, for Value Shaft current deviation, for Value shaft voltage, for Value shaft voltage, For time; Finally, the proportional-integral-differential output and This is used to control the PWM pulse signal generator, which in turn controls the static synchronous compensator of the power distribution network.

2. The static synchronous compensator control method based on composite deep reinforcement learning according to claim 1, characterized in that, The static synchronous compensator control method based on composite deep reinforcement learning can better control the static synchronous compensator in the distribution network, improve voltage stability and transient performance, optimize the absorption or generation of reactive power by the static synchronous compensator in the power system, improve power system stability, and improve power quality.

3. The static synchronous compensator control method based on composite deep reinforcement learning according to claim 1, characterized in that, In step (1), the data to be collected is preprocessed as follows: Data collected from the distribution network The input is fed into the trained agent. For AC voltage deviation, DC voltage deviation, for Shaft current reference value for Shaft current reference value for Shaft current deviation, for Shaft current deviation; agent from input value Column vectors are obtained by collecting time series data. ; Column vector for: in, for The column vector at time t, , , , , , , column vector The first element in; column vector The second element in; column vector The third element in; column vector The fourth element in; column vector The fifth element in; column vector The sixth element in; for AC voltage deviation at any given time; for DC voltage deviation at any given time; for Moment Shaft current reference value; for Moment Shaft current reference value; for Moment Shaft current deviation; for Moment Shaft current deviation; The collected one-dimensional time series data are arranged into a data matrix, which is as follows: in, ,for AC voltage deviation at any given time; ,for AC voltage deviation at any given time; ,for DC voltage deviation at any given time; ,for DC voltage deviation at any given time; ,for Moment Shaft current reference value; ,for Moment Shaft current reference value; ,for Moment Shaft current reference value; ,for Moment Shaft current reference value; ,for Moment Shaft current deviation; ,for Moment Shaft current deviation; ,for Moment Shaft current deviation; ,for Moment Shaft current deviation; for The column vector at time t, for The column vector at time points; Rearrange the data matrix into a data matrix , as the input data matrix of the convolutional neural network ;matrix for: in, for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; Next, the data matrix Normalization is performed on the data matrix Each element in the array is in the interval [0,1]; the process is as follows: First, the data matrix Process in blocks, starting from the first row, sequentially assembling the 6 column vectors into a new matrix. , The data matrix is ​​a 6×6 square matrix with a total of 36 elements. Divided into 25 blocks, data matrix for: make ; From 1 to 25, For data matrix In arrive ; in, For matrix Block 1 in the middle; For matrix Block 2 in the middle; For matrix Block 3 in the middle; For matrix Block 4 in the middle; For matrix Block 5 in the middle; For matrix Block 6 in the middle; For matrix Block 7 in the middle; For matrix Block 8 in the middle; For matrix Block 9 in the middle; For matrix Block 10 in the middle; For matrix Block 11 in the middle; For matrix Block 12 in the middle; For matrix Block 13 in the middle; For matrix Block 14 in the middle; For matrix Block 15; For matrix Block 16 in the middle; For matrix Block 17 in the middle; For matrix Block 18 in the middle; For matrix Block 19 in the middle; For matrix Block 20 in the middle; For matrix Block 21 in the middle; For matrix Block 22; For matrix Block 23; For matrix Block 24 in the middle; For matrix Block 25; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; for The column vector at time points; Then each matrix Perform the following processing: in, For matrix middle OK Column elements, For matrix The minimum value among the 36 elements. For matrix The maximum value among the 36 elements; ; Normalized All 36 elements in the dataset are in the interval [0,1]. From this point on, the normalized data matrix Linear transformation, making the data matrix Each element in the array is in the interval [0, 255]; the process is as follows: in, Indicates will Round down to the nearest integer. It is a natural number from 0 to 255, which is convenient for convolutional neural networks to process.

4. The static synchronous compensator control method based on composite deep reinforcement learning according to claim 1, characterized in that, The data input into the convolutional neural network in step (2) is a matrix. ,matrix The layers sequentially pass through an input layer, a convolutional layer, a ReLU activation layer, an average pooling layer, a perceptron model, another convolutional layer, a ReLU activation layer, a pooling layer, a Dropout layer, a fully connected layer, a sigmoid activation layer, and an output layer; then the convolutional neural network outputs labels for classification.

5. The static synchronous compensator control method based on composite deep reinforcement learning according to claim 1, characterized in that, The experience pool in step (3) can effectively break the correlation between samples, increase the diversity of parameter update direction and policy function distribution, and reuse samples to increase the frequency of sample use and parameter update, thereby reducing training time. Time series data for ; for shaft voltage and shaft voltage ; The reward system is designed as follows: in, and They are respectively shaft voltage and shaft voltage The proportionality coefficient, for Shaft voltage reference value, for Shaft voltage reference value, As a reward; for ; The agent evaluates the benefits of each action and the reward is obtained by recording the highest reward obtained during the deep reinforcement learning process. Whether it's the target network or the current network, the actor network will take action. or Input into the critic network, For the target network Actions at any moment In the current network Actions at any given moment; a network of critics judges and filters actions and states, introducing... The value is used to estimate the value of the strategy. Values ​​are represented in the state space. Next Action Set The value of using a network of critics as a transition from state and action The value mapping function, denoted as , For the critic's network parameters; loss function for: in, For the critic's network parameters, for Number of samples of the value for Commentator network parameters at any time for Values ​​are represented in the state space. Next Action Set The greatest value, for Values ​​are represented in the state space. Next Action Set The value of the network parameter is determined by the loss function, which is the variance between the actual reward and the estimated reward. The smaller the loss function, the more accurate the network parameters are.

Images