Photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning

Abstract

The present invention discloses a photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning. The method first obtains the measured data of the photovoltaic power station to construct an identification training data set, and establishes an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station. Secondly, a Transformer deep reinforcement learning environment based on BPA software is established. Finally, in the Transformer deep reinforcement learning environment, a Transformer-PPO intelligent agent model is built according to the equivalent mathematical model, and the identification training data set is input for parameter identification training to obtain the final identification result. According to the photovoltaic power station model, the present invention combines the deep reinforcement learning algorithm PPO with the Transformer deep learning model architecture to screen parameters and perform accurate and rapid identification.

Description

Technical Field

[0001] The present invention belongs to the field of electric power information technology and relates to a photovoltaic power station parameter identification method based on a deep reinforcement learning algorithm PPO (Proximal Policy Optimization) and a Transformer model. Background Art

[0002] With the rapid development of renewable energy, a large number of wind, photovoltaic, and other renewable energy generation systems have been integrated into the power grid, posing significant challenges to the security and stability of the power system. Photovoltaic power plant modeling involves establishing a photovoltaic array model and an inverter control model, followed by parameter identification of these models. The inverter in a photovoltaic power plant is the core of the grid-connected photovoltaic power generation system, and its control parameters are often difficult to directly obtain during actual grid-connected operation. Therefore, research on parameter identification of photovoltaic power plant models is urgently needed.

[0003] Parameter identification for photovoltaic power plants not only helps improve system modeling accuracy but also holds significant significance for system performance optimization, fault diagnosis, and control strategy design. Traditional optimization algorithms, such as least squares, maximum likelihood estimation, and Kalman filtering, typically rely on physical or simplified mathematical models of the photovoltaic power plant and analyze the volt-ampere characteristics of photovoltaic cells or other operating data for parameter identification. While these methods have demonstrated promising results in simple applications, providing a certain degree of accuracy and stability, they also have significant drawbacks. Traditional methods suffer from low identification accuracy, are sensitive to the choice of initial points, and are susceptible to measurement noise. Furthermore, they can lead to computational complexity and the risk of overfitting in high-dimensional data. Furthermore, these methods suffer from poor interpretability and perform poorly with nonlinear or noisy data. Therefore, their applicability and limitations require careful evaluation in practical applications.

[0004] In this context, deep reinforcement learning (DRL) algorithms, as an emerging intelligent optimization method, have demonstrated their advantages in dynamic environments. Deep reinforcement learning learns optimal strategies through interaction with the environment, and can continuously optimize the parameter identification process through experience without the need for an explicit model. Compared with traditional algorithms, deep reinforcement learning not only has stronger global optimization capabilities but also can adapt to complex nonlinear and dynamic changes. In particular, when faced with environmental uncertainty and variable factors, it can adjust parameters through self-learning, thereby improving the adaptability and robustness of the system. In addition, deep reinforcement learning can handle optimization problems involving large-scale data and high-dimensional spaces, and has high flexibility and scalability. Therefore, deep reinforcement learning can provide more efficient and accurate solutions for real-time parameter identification of photovoltaic power plants, providing strong support for the operation optimization of photovoltaic power plants and the stability of power systems. Summary of the Invention

[0005] This paper proposes a photovoltaic power station parameter identification method based on the Transformer-PPO deep reinforcement learning algorithm, aiming to achieve accurate and rapid identification of the main parameters of the photovoltaic power station. This method combines the powerful feature extraction capabilities of the Transformer model with the efficient strategy optimization characteristics of the PPO algorithm. It can quickly adapt to changes in the operating conditions of the photovoltaic power station in complex dynamic environments, improve the stability and convergence speed of photovoltaic power station parameter identification, accurately identify the key parameters of the photovoltaic power station, and provide data support for subsequent photovoltaic power station performance optimization and fault diagnosis. It also helps to analyze the dynamic characteristics of the photovoltaic power station system, thereby ensuring the safe and stable operation of the power grid.

[0006] The present invention proposes to use the power system simulation software BPA, then set the parameter range, use Pytorch (an open source Python machine learning library that includes a deep neural network with an automatic differentiation system) to build a PPO model, and then use the output curve of the photovoltaic power station measured data and the PPO intelligent agent to finally identify the parameters in the photovoltaic power station model, solving the problems of poor stability and difficulty in convergence that often occur in traditional algorithms, reducing the complexity of parameter identification, improving the identification accuracy of parameters that have a greater impact on the controller, and improving identification efficiency.

[0007] The method of the present invention specifically is:

[0008] A photovoltaic power station parameter identification method based on Transformer-PPO, including:

[0009] Obtain the measured data of the photovoltaic power station to construct an identification training data set, and establish an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station;

[0010] Establish a Transformer deep reinforcement learning environment based on BPA software;

[0011] In the Transformer deep reinforcement learning environment, a Transformer-PPO intelligent agent model is built based on the equivalent mathematical model, and the identification training data set is input for parameter identification training to obtain the final identification result.

[0012] Furthermore, the establishment of the Transformer deep reinforcement learning environment based on BPA software includes the following steps:

[0013] Status t Set it as the parameter to be identified in the photovoltaic power station, and take the change value of the parameter to be identified in the photovoltaic power station as action a t , change the state s t With action a t Add up to get the state s at time t+1t+1 :

[0014] s t+1 ←s t +a t (1)

[0015] According to the transient file SWI file format in BPA, the state s t+1 , that is, the values ​​of the photovoltaic power station parameters are written into the transient file SWI, and then a transient calculation is performed to obtain the new model response.

[0016] Furthermore, the result file SWX in the BPA software is obtained based on the reactive power curve in the current state. The reactive power curve calculated by BPA is compared with the measured real data curve, and the reward R is used to determine whether the error standard is met; if the error is less than the allowable value, the photovoltaic parameters are output. If the error is greater than the allowable value, the parameter identification of the Transformer-PPO algorithm is performed, and the reactive power Q in the current state is read from the result file SWX to calculate the reward R. The calculation formula of reward R is as follows:

[0017]

[0018] where Q 实测 (t) is the measured reactive data, Q 等值 (t) is the reactive data output by the equivalent model; n is the number of sampling points. The size of the reward R is used as an evaluation indicator. During each training, we observe whether the value of the reward R meets the target.

[0019] Furthermore, the Transformer-PPO agent model structure includes a Transformer module, an Actor neural network, an Actor-old neural network and a Critic neural network.

[0020] Transformer is a deep learning model architecture that introduces a self-attention mechanism, which enables it to perform well in processing sequence data. The PPO algorithm is a policy gradient algorithm based on the deep reinforcement learning Actor-Critic framework, which fits the optimal action by updating the policy function through the obtained reward. PPO improves the strategy by optimizing a specific objective function, can achieve small batch updates in multiple training steps, and can cope with complex situations such as high-dimensional and continuous action spaces. Its core idea is to limit the amplitude of policy updates to achieve stable and efficient training results. The process of the Transformer-PPO algorithm interacting and iterating with the environment: the agent obtains the current state s from the environment t , the agent takes action a tThe results are fed back to the environment, which generates a new state and reward R, and stores the continuously collected data in the RolloutBuffer, which is then sampled to train the agent. Transformer is used to extract features, and the Actor network is used to output strategies. The Actor-old network and the Actor network have the same structure, and the parameters of the two networks are also the same at the beginning of training. The Actor-old network is used to output strategies and select actions but does not perform gradient updates; the Critic network is used to calculate state values. The Transformer module includes a sequentially constructed fully connected layer, positional encoding, and multiple encoding layers. Each encoding layer includes a sequentially cascaded multi-head attention layer, a normalization layer, a feedforward network, and a GRU gating unit.

[0021] Furthermore, the input identification training data set is used to perform parameter identification training to obtain a final identification result, including the following steps:

[0022] Generate a set of small batches of data (s) from the collection Rollout Buffer t , memory, R) is input into the Transformer module. The memory matrix is ​​used in the Transformer to store and process the embedding vectors obtained by the interaction of past time steps, representing the historical memory of each time step of the model. First, a fully connected layer is used for linear embedding to embed the state data s input at the current time step. t Mapping from a low dimension to an embedding dimension D allows the Transformer to capture more complex relationships. Position encoding is performed on the linearly embedded data to provide the position information of each element in the time series.

[0023] The positional encoding is added to the embedded data of each time step to inject the sequential information of each time step. The encoded data together with the historical state encoding information stored in the memory are used as input x t (batch_size, T, N, D) is provided to the attention mechanism module. Where batch_size represents the number of training batches, T represents the number of time series steps in the round, N represents the number of Transformer layers, and D represents the dimension of the embedding. In the multi-head attention mechanism, the input sequence x is first t Perform a linear transformation:

[0024] Q=W Q ·x t (3)

[0025] K=W K ·x t (4)

[0026] V=WV ·x t (5)

[0027] Among them, W Q ,W K and W V It is a learnable weight matrix; Q represents the query matrix, which represents the query information of the model at the current time step and is used to query information in other parts of the input sequence; K represents the key matrix, which represents the key information of each position and is used to compare with the query Q to determine the relevance; V represents the value matrix, which contains the actual information of each position and determines the output value of each position after the similarity (attention weight) is calculated between Q and K. After the transformation, Q, K and V are divided into multiple heads according to the total embedding dimension. The dimension of each head is the total embedding dimension divided by the number of heads. The attention weight is calculated for each head. The calculation formula of the attention weight is as follows:

[0028]

[0029] The softmax() function maps each element to a value between 0 and 1; T represents the length of the input time series; D is the embedding dimension of the input sequence, h is the number of heads of the segmentation; mask is a mask used to mask positions that do not need to be paid attention to; Q i ,K i and V i is the query, key and value of the i-th head. Finally, we combine the outputs of all heads and pass them through another linear transformation weight matrix W o Integration information:

[0030] Multihead(Q,K,V)=Concat(head1,head2,…head h )W o (7)

[0031] The multi-head attention mechanism module concatenates the multiple vectors output by all heads along the embedding dimension to form a complete feature vector of size (T, D). This vector contains the information of all heads and can therefore capture the features of different subspaces. The concatenated vector is passed through the normalization layer to obtain y t Next, the GRU gated unit enters. The GRU gated unit introduces two gated units: a reset gate and an update gate. These gated units learn how to adjust the importance of the input information and the hidden state information from the previous time step at each time step. The GRU gated unit accepts the input of the current time step and the hidden state from the previous time step, and returns the hidden state from the current time step. This recursive application allows the hidden state from the previous time step to be obtained at each time step and used in the calculations for the next time step.

[0032] For the reset gate:

[0033] r t =σ(W r ·y t-1 +U r ·y t ) (8)

[0034] Among them, y t is the input of the current time step, i.e. the output of multi-head attention, y t-1 is the hidden state of the previous time step, σ is the Sigmoid activation function, W r is the linear transformation weight from the hidden state to the reset gate, U r is the linear transformation weight from input to reset gate.

[0035] For the update gate:

[0036] z t =σ(W z ·y t-1 +U z ·y t -b g ) (9)

[0037] Among them, σ is the Sigmoid activation function, W z is the linear transformation weight from the hidden state to the update gate, U z is the linear transformation weight from input to update gate, b g is the shared bias term.

[0038] For candidate hidden states:

[0039]

[0040] Among them, tanh is the hyperbolic tangent activation function, W g is the linear transformation weight from hidden state to candidate hidden state, U g is the linear transformation weight from input to candidate hidden state, r t is the output of the reset gate, and ⊙ represents element-wise multiplication.

[0041] For the final hidden state:

[0042]

[0043] Among them, z tIs the output of the update gate, out1 is the output of the first GRU, out1 passes through the feedforward network and the normalization layer output E, and is sent to the second GRU gate unit and the feedforward network layer for calculation to produce the final output out2 (batch_size, T, D), where T and D represent the number of time steps and embedding dimensions of each round respectively. And the memory is updated to include the information of the current time step, continuously accumulating new information, and can be used in subsequent time steps. The output out2 of the Transformer module and the reward R are input to the Critic network, and the state value function is output.

[0044] Then calculate the discounted reward δ based on the reward t and advantage function A t :

[0045] δ t =γR t+1 +γ 2 R t+2 +γ 3 R t+3 +…=∑ t′>t γ t′-t R t′ (12)

[0046]

[0047] Among them A t is the advantage function; is state s t The state value function of R t′ is the reward at time t′; t is the current time step, t′ is the future time step greater than t; γ is the discount factor; is the critic network parameter; δ t is the discounted reward, which represents the cumulative discounted reward value of future rewards starting from the current time.

[0048] Then the network parameters are updated, starting with the update of the Critic network. The Critic network loss function as follows:

[0049]

[0050] in, is state s t The state value function of R t′ is the reward at time t′; t is the current time; γ is the discount factor; the critic network loss function is the mean square value of the advantage function; T is the termination time, i.e. the total number of time steps. The parameters of the critic network are updated through backpropagation:

[0051]

[0052] in are the parameters of the Critic network, is the learning rate, is a derivative operation.

[0053] Initialize the number of loops i = 0. Then input out2 into the Actor network and the Actor-old network to obtain two normal distributions Normal1 and Normal2 and the probability distribution function log p corresponding to each action θ (a t |s t )and

[0054] Then update the Actor network. For the Actor network, the objective function obtained by PPO-clip operation is:

[0055]

[0056] L clip (θ)=-E[min(r t (θ)A t ,clip(r t (θ),1-∈,1+∈)A t )] (17)

[0057] Among them, r t (θ) is the update amplitude of the policy, also known as KL divergence, which is used to constrain the difference between the current policy and the old policy. θ (a t |s t )and It is the logarithm of the strategy distribution obtained based on the Actor network and the Actor-old network, indicating the current strategy in state s t Next take action a t The probability of the old policy in state s t Next take action a t ∈ is a hyperparameter used to control the clipping amplitude; E is the averaging operation.

[0058] The Actor network objective function consists of two parts, and the smallest one is selected as the alternative objective function: t (θ)A t Represents the current strategy update amplitude multiplied by the value of the advantage function; clip(r t (θ),1-∈,1+∈)A tIndicates that the current strategy update amplitude is clipped and then multiplied by the value of the advantage function. If r t The value of (θ) is out of the range of (1-∈, 1+∈), so it is intercepted. t If the value of (θ) is greater than 1+∈, then set it to 1+∈. If its value is less than 1-∈, then set it to 1-∈. If its value is between (1-∈, 1+∈), then keep its value. The role of the clipping function is to limit the magnitude of the policy update to ensure that the policy update magnitude is not too large. By calculating L clip The derivative of (θ) implements backpropagation to update the parameters of the Actor network:

[0059]

[0060] Where θ is the parameter of the Actor network, α θ is the learning rate, is a derivative operation. Repeat the trimming and updating operation.

[0061] After a certain step, the Actor network loop ends and the Actor-old network is updated. The parameters of the Actor network after the loop are copied to the Actor-old network, so that the parameters of the Actor-old network are updated.

[0062] Finally, the Transformer network is updated. By solving the above Actor and Critic network loss functions, the total loss of the network is obtained as:

[0063]

[0064] in is the loss function of the Critic network, L clip (θ) is the loss function of the Actor network, λ is the weight coefficient used to control the importance of KL divergence in the total loss function, r t (θ) is the KL divergence. The gradient of the loss function with respect to the network parameters is calculated by backpropagation Use the Adam optimizer to update the network parameters by gradient descent:

[0065]

[0066] Where η is the parameter of the Transformer network, α η is the learning rate, is a derivative operation.

[0067] After the update is completed, the Transformer and Actor networks interact with the environment again to obtain data and update the parameters of each network structure.

[0068] The current identification state, action and error are continuously calculated and output. When the error is less than the allowable value, the current parameter value of the photovoltaic model is output, which is the final result of the parameter identification of the photovoltaic power station using the measured data curve of the photovoltaic power station and the Transformer-PPO algorithm.

[0069] The beneficial effects of the present invention are as follows: a corresponding equivalent mathematical model is established in the BPA software based on the photovoltaic power station model, the deep reinforcement learning algorithm PPO is combined with the Transformer deep learning model architecture, and the Transformer-PPO algorithm is used to screen parameters and perform accurate and rapid identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is the framework diagram of the mathematical model of the photovoltaic system;

[0071] Figure 2 is the reactive power control model;

[0072] Figure 3 This is the Transformer-PPO network structure diagram;

[0073] Figure 4 Transformer structure diagram;

[0074] Figure 5 This is the principle diagram of the Transformer-PPO algorithm;

[0075] Figure 6 This is the flow chart of photovoltaic power station parameter identification based on Transformer-PPO;

[0076] Figure 7 It is the voltage disturbance comparison diagram;

[0077] Figure 8 This is a comparison chart of reactive power output;

[0078] Figure 9 This is a comparison chart of active power output;

[0079] Figure 10 This is a comparison chart of reactive current curves. DETAILED DESCRIPTION

[0080] The present invention is described in detail with reference to the accompanying drawings. A photovoltaic power station parameter identification method based on Transformer-PPO includes:

[0081] Obtain the measured data of the photovoltaic power station to construct an identification training data set, and establish an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station;

[0082] Establish a Transformer deep reinforcement learning environment based on BPA software;

[0083] In the Transformer deep reinforcement learning environment, a Transformer-PPO intelligent agent model is built based on the equivalent mathematical model, and the identification training data set is input for parameter identification training to obtain the final identification result.

[0084] Example:

[0085] The present invention follows the steps:

[0086] Step (1): Obtain the measured data of the RTDS of the photovoltaic power station, use the power system simulation software BPA to simulate the measured curve environment of the photovoltaic power station, and establish a mathematical model for the photovoltaic power station to be connected to the grid.

[0087] Obtain measured data for the photovoltaic power station, including time (s), active power P3 (MW), reactive power Q3 (Mvar), reactive current (A), line voltage Uab (KV) between phases AB, line voltage Ubc (KV) between phases BC, line voltage Uac (KV) between phases AC, and frequency (Hz). Use PSD-BPA (BPA), a simulation tool developed by the China Electric Power Research Institute, to establish an equivalent mathematical model. BPA is comprehensive power simulation software for power system calculation and analysis. It has been widely used by domestic power dispatching and operation organizations, power system planning-related units, and various universities. It provides functions such as power flow calculation, transient stability simulation, short-circuit calculation, and small disturbance stability calculation.

[0088] Flow calculation: The flow calculation file format is DAT. The flow data file is a DAT text file that provides data and instructions for flow calculation and is defined in BPA. The parameters can be directly edited and modified.

[0089] Transient calculation: The transient calculation file format is SWI. Similar to the power flow data file, component dynamic parameters, fault operation, calculation, and output control are all input in the form of cards. By setting the parameters of the relevant cards, stability calculations can be performed under different interference modes.

[0090] In the BPA software, a single-machine mathematical model of the photovoltaic power station is established and the parameters to be identified are selected. The photovoltaic power generation mathematical model based on the BPA software contains multiple card modules. The overall framework diagram is as follows: Figure 1 and Figure 2 The card introduction is shown in Table 1.

[0091] Table 1 Card identification of main control models of photovoltaic model

[0092] Card Name Card logo Photovoltaic power generation model MS Required Active power control model under normal operating conditions EU Required Reactive power control model under normal operating conditions EZ Usually fill in Active and reactive current limiting model ES Required Low voltage and high voltage state judgment model EV Required Active power control model under low voltage ride-through state LP Usually fill in Reactive power control model under low voltage ride-through state LQ Usually fill in Frequency control model EY Generally do not fill

[0093] The PV power generation model is represented by an MS card, which contains important parameters such as the rated capacity of a single PV array and the DC link capacitance C. The active and reactive power models under normal conditions are represented by EU and EZ cards, which include a series of control model parameters such as time constants. These parameters have a significant impact on the model, and their accuracy also reflects the accuracy of the model.

[0094] Step (2): Establish a deep reinforcement learning environment for Transformer-PPO based on BPA software.

[0095] Select the photovoltaic power station parameter voltage measurement time constant T R , proportional link coefficient K PV , integral link coefficient K VI , time constant T V and time constant T PV are parameters to be identified, and their ranges are shown in Table 2.

[0096] Table 2 Optimization range of photovoltaic power station parameters

[0097] parameter Lower limit Upper limit <![CDATA[T R ]]> 0.0001 0.01 <![CDATA[K PV ]]> 0.0001 1 <![CDATA[K VI ]]> 0.0001 0.01 <![CDATA[T V ]]> 0.0001 0.01 <![CDATA[T PV ]]> 0.0001 0.011

[0098] After determining the identification parameters and parameter range, the value of the photovoltaic power station parameter is used as the state s t , the change in parameter value is action a t , according to formula (1) we can get the next state s t+1 Then the current state s t The parameter values ​​of the photovoltaic power station are written into the SWI file of the BPA software, and then the transient calculation program of the BPA software is called to perform simulation calculations to obtain the reactive power data in the current state, that is, under the current photovoltaic parameter values.

[0099] like Figure 3 and Figure 4 As shown in Figure 2, if the action given by Transformer-PPO causes the next state to exceed the set parameter range, the Transformer-PPO agent is given a penalty, making the reward R -10. If the next state is within the range, it is trained normally, and the reward R is obtained according to formula (2).

[0100] Step (3): Establish the Transformer-PPO agent.

[0101] The network structure of Transformer-PPO deep reinforcement learning is as follows Figure 5 As shown in the figure, the deep learning framework is Pytorch. The number of hidden layers in the Actor network, Actor_old network, and Critic network is 2, and each layer has 64 neurons. The learning rate of the Actor network is α θis 0.0001, the learning rate of the Critic network is 0.001, the discount factor γ is 0.99, the cropping factor ∈ is 0.2, the number of small batch sample data MINIBATCH generated from the Rollout Buffer each time is 64, the number of multi-head attention num_heads is 6, and the embedding dimension embed_dim is 384.

[0102] a. Step (4): Start parameter identification training and obtain the final identification result.

[0103] The experiment was conducted on an NVIDIA GeForce RTX 4060 GPU. The neural network was built and trained using PyTorch, with Adam as the optimizer. When the error between the simulation results and the measured data is less than the allowable limit, the current parameter value is output, which is the identification result.

[0104] The present invention uses the Transformer-PPO algorithm to compare and identify the real data values ​​measured by the PPO algorithm, the SAC multi-agent algorithm, and the RTDS. Figure 6 The identification results are shown in the attached figure. Figure 7 Voltage disturbance output comparison chart, Figure 8 Reactive power output comparison chart, Figure 9 Active power output comparison chart, Figure 10 This is a comparison chart of reactive current output.

[0105] The identification results of each parameter are shown in Table 3:

[0106] Table 3 Photovoltaic parameter identification results

[0107] Photovoltaic parameters Transformer-PPO algorithm PPO Algorithm SAC multi-agent algorithm <![CDATA[T R ]]> 0.0057 0.0085 0.0086 <![CDATA[K PV ]]> 0.2542 0.2564 0.3115 <![CDATA[K VI ]]> 0.0010 0.0004 0.0018 <![CDATA[T V ]]> 0.0097 0.0079 0.0061 <![CDATA[T PV ]]> 0.0102 0.0108 0.0104

[0108] The root mean square error of the simulation curve and the detailed model curve is selected as the error judgment standard. The error results and identification time results are shown in Table 4:

[0109] Table 4 Errors and recognition times of three algorithms

[0110] RMSE error Identification time (min) Transformer-PPO of the present invention 0.6099 3.36 PPO 0.6177 3.28 SAC Multi-Agent 0.6556 5.34

[0111] Judging from the error results, the errors of the Transformer-PPO algorithm and the PPO algorithm are relatively close, and both are better than the identification results of the SAC multi-agent. In terms of time, the PPO algorithm is faster, followed by the Transformer-PPO algorithm, and the SAC multi-agent algorithm has the longest identification time.

Claims

1. A photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning, characterized by: The process includes the following: Obtain the measured data of the photovoltaic power station to construct an identification training data set, and establish an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station; Establish a Transformer deep reinforcement learning environment based on BPA software; In the Transformer deep reinforcement learning environment, a Transformer-PPO agent model is built based on the equivalent mathematical model, and the identification training dataset is input for parameter identification training to obtain the final identification result; The Transformer-PPO agent model is specifically implemented as follows: The Transformer-PPO agent model structure includes the Transformer module, Actor neural network, Actor-old neural network and Critic neural network; The Transformer-PPO algorithm interacts and iterates with the environment, and the agent obtains the current state s from the environment. t , the agent takes action a t The results are fed back to the environment, which generates a new state and reward R. The environment stores the continuously collected data in the Rollout Buffer, and then samples the Rollout Buffer to train the agent. During training, the Transformer module is used to extract features, and the Actor network is used to output policies. The Actor-old network and the Actor network have the same structure, and the parameters of the two networks are also the same at the beginning of training. The Actor-old network is used to output policies and select actions but does not perform gradient updates. The Critic network is used to calculate state values. The Transformer module consists of a sequentially constructed fully connected layer, positional encoding, and multiple encoding layers. Each encoding layer consists of a sequentially cascaded multi-head attention layer, a normalization layer, a feedforward network, and a GRU gating unit.

2. The photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning according to claim 1 is characterized in that: The specific implementation process of establishing a Transformer deep reinforcement learning environment based on BPA software is as follows: Status t Set it as the parameter to be identified in the photovoltaic power station, and take the change value of the parameter to be identified in the photovoltaic power station as action a t , change the state s t With action a t Add up to get the state s at time t+1 t+1 ; The result file SWX in the BPA software is obtained based on the reactive power curve in the current state. The reactive power curve calculated by BPA is compared with the measured real data curve. The reward R is used to determine whether the error standard is met. If the error is less than the allowable value, the photovoltaic parameters are output. If the error is greater than the allowable value, the parameter identification of the Transformer-PPO algorithm is performed on it. The reactive power Q in the current state is read from the result file SWX to calculate the reward R. The calculation formula of reward R is as follows: where Q 实测 (t) is the measured reactive data, Q 等值 (t) is the reactive data output by the equivalent model; n is the number of sampling points; each training is performed to observe whether the value of the reward R meets the target.

3. The photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning according to claim 2 is characterized in that: The input identification training data set is used for parameter identification training to obtain the final identification result. The specific process is as follows: Generate a set of batch data (s) from the collection Rollout Buffer t , memory, R) is input into the Transformer module, where the memory matrix is ​​used in the Transformer to store and process the embedded vectors obtained by the interaction of past time steps, representing the historical memory of each time step of the model; first, a fully connected layer is used for linear embedding to transform the state data s input at the current time step into t Mapping to an embedding dimension D, position encoding the data after linear embedding, and providing the position information of each element in the time series; The positional encoding is added to the embedded data of each time step, injecting the sequential information of each time step; the encoded data together with the historical state encoding information stored in the memory are used as input x t ; Then calculate the discounted reward δ based on the reward t and advantage function A t : Among them A t is the advantage function; is state s t The state value function of R t′ is the reward at time t′; t is the current time step, t′ is the future time step greater than t; γ is the discount factor; is the critic network parameter; δ t is the discounted reward, which represents the cumulative discounted reward value of future rewards starting from the current time; Then the network parameters are updated, starting with the update of the Critic network; the Critic network loss function as follows: in, is state s t The state value function of R t′ is the reward at time t′; t is the current time; γ is the discount factor; the Critic network loss function is the mean square value of the advantage function; T is the termination time, that is, the total number of time steps; the parameters of the Critic network are updated through backpropagation; Then the Actor network is updated. For the Actor network, the objective function obtained by PPO-clip operation is: L clip (θ)=-E[min(r t (i)A t ,clip(r t (θ),1-∈,1+∈)A t )] Among them, r t (θ) is the update amplitude of the strategy, also known as KL divergence, which is used to constrain the difference between the current strategy and the old strategy, logp θ (a t |s t )and It is the logarithm of the strategy distribution obtained based on the Actor network and the Actor-old network, indicating the current strategy in state s t Next take action a t The probability of the old policy in state s t Next take action a t ∈ is a hyperparameter used to control the clipping amplitude; E is the averaging operation; The Actor network objective function consists of two parts, and the smallest one is selected as the alternative objective function: t (θ)A t Represents the current strategy update amplitude multiplied by the value of the advantage function; clip(r t (θ),1-∈,1+∈)A t Indicates that the current strategy update amplitude is clipped and then multiplied by the value of the advantage function. If r t The value of (θ) is out of the range of (1-∈, 1+∈), so it is intercepted. t If the value of (θ) is greater than 1+∈, then let its value be 1+∈. If its value is less than 1-∈, then let its value be 1-∈. If its value is between (1-∈, 1+∈), then keep its value. By calculating L clip The derivative of (θ) implements backpropagation to update the parameters of the Actor network; The Actor network loop ends and the Actor-old network is updated. The Actor network parameters after the loop are copied to the Actor-old network and the parameters of the Actor-old network are updated. Finally, the Transformer network is updated; by solving the above Actor and Critic network loss functions, the total loss is: in is the loss function of the Critic network, L clip (θ) is the loss function of the Actor network, λ is the weight coefficient used to control the importance of KL divergence in the total loss function, r t (θ) is the KL divergence; the gradient of the loss function with respect to the network parameters is calculated by backpropagation The Adam optimizer is used to update the network parameters by gradient descent. After the update is completed, the Transformer and Actor networks interact with the environment again to obtain data and update the parameters of each network structure. The current identification state, action and error are continuously calculated and output. When the error is less than the allowable value, the current parameter value of the photovoltaic model is output, which is the final result of the parameter identification of the photovoltaic power station using the measured data curve of the photovoltaic power station and the Transformer-PPO algorithm.

Images