Photovoltaic power station parameter identification method based on Transform-PPO deep reinforcement learning

Abstract

The invention discloses a photovoltaic power station parameter identification method based on Transform-PPO deep reinforcement learning, and the method comprises the steps: firstly obtaining the actual measurement number of a photovoltaic power station, constructing an identification training data set, and building an equivalent mathematical model which is the same as the actual measurement curve operation environment of the photovoltaic power station; secondly, a Transform deep reinforcement learning environment based on BPA software is established; and finally, in a Transform deep reinforcement learning environment, according to the equivalent mathematical model, establishing a Transform-PPO intelligent agent model, and inputting an identification training data set to carry out parameter identification training so as to obtain a final identification result. According to a photovoltaic power station model, a deep reinforcement learning algorithm PPO and a Transform deep learning model architecture are combined to screen parameters and perform accurate and rapid identification.

Description

Technical Field

[0001] The present invention belongs to the field of electric power information technology and relates to a photovoltaic power station parameter identification method based on a deep reinforcement learning algorithm PPO (Proximal Policy Optimization) and a Transformer model. Background Art

[0002] With the rapid development of new energy, a large number of wind power, photovoltaic and other new energy power generation systems have been connected to the power grid, and the safety and stability of the power system have been greatly challenged. Photovoltaic power station modeling includes the establishment of photovoltaic array models and inverter control models, as well as the subsequent parameter identification of these models. The inverter of the photovoltaic power station is the core of the photovoltaic grid-connected power generation system, and its control parameters are usually difficult to obtain directly in actual grid-connected operation. Therefore, it is urgent to study the parameter identification of the photovoltaic power station model.

[0003] Parameter identification of photovoltaic power stations not only helps to improve the accuracy of system modeling, but also has important significance for system performance optimization, fault diagnosis and control strategy design. Traditional optimization algorithms such as least squares method, maximum likelihood estimation method and Kalman filtering method usually rely on the physical model or simplified mathematical model of the photovoltaic power station, and perform parameter identification by analyzing the volt-ampere characteristics or other operating data of photovoltaic cells. Although these methods have shown good results in some simple applications and can provide a certain degree of accuracy and stability, they also have significant disadvantages. Traditional methods have low identification accuracy, are sensitive to the selection of initial points, and are easily affected by measurement noise. And they may cause computational complexity and overfitting risks in high-dimensional data. In addition, the method has poor interpretability and is not effective when dealing with nonlinear or high-noise data. Therefore, its applicability and limitations need to be carefully evaluated in practical applications.

[0004] In this context, deep reinforcement learning (DRL) algorithm, as an emerging intelligent optimization method, has demonstrated its advantages in dynamic environments. Deep reinforcement learning learns the optimal strategy through interaction with the environment, and can continuously optimize the parameter identification process through experience without a clear model. Compared with traditional algorithms, deep reinforcement learning not only has strong global optimization capabilities, but also can adapt to complex nonlinear and dynamic changes. In particular, when facing environmental uncertainties and variable factors, it can adjust parameters through self-learning, thereby improving the adaptability and robustness of the system. In addition, deep reinforcement learning can handle optimization problems of large-scale data and high-dimensional space, and has high flexibility and scalability. Therefore, deep reinforcement learning can provide more efficient and accurate solutions in the real-time parameter identification of photovoltaic power stations, providing strong support for the operation optimization of photovoltaic power stations and the stability of power systems. Summary of the invention

[0005] The present invention proposes a photovoltaic power station parameter identification method based on the Transformer-PPO deep reinforcement learning algorithm, aiming to achieve accurate and rapid identification of the main parameters of the photovoltaic power station. This method combines the powerful feature extraction capability of the Transformer model and the efficient strategy optimization characteristics of the PPO algorithm. It can quickly adapt to changes in the operating conditions of the photovoltaic power station in a complex dynamic environment, improve the stability and convergence speed of photovoltaic power station parameter identification, accurately identify the key parameters of the photovoltaic power station, and provide data support for subsequent photovoltaic power station performance optimization and fault diagnosis, and help analyze the dynamic characteristics of the photovoltaic power station system, thereby ensuring the safe and stable operation of the power grid.

[0006] The present invention proposes to use the power system simulation software BPA, and then set the range of parameters, use Pytorch (an open source Python machine learning library, including a deep neural network of an automatic differentiation system) to build a PPO model, and then use the output curve of the measured data of the photovoltaic power station and the PPO agent to finally identify the parameters in the photovoltaic power station model, solve the problems of poor stability and difficulty in convergence that often occur in traditional algorithms, reduce the complexity of parameter identification, improve the identification accuracy of parameters that have a greater impact on the controller, and improve the identification efficiency.

[0007] The method of the present invention specifically comprises:

[0008] A photovoltaic power station parameter identification method based on Transformer-PPO, comprising:

[0009] Obtain the measured data of the photovoltaic power station to construct an identification training data set, and establish an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station;

[0010] Establish a Transformer deep reinforcement learning environment based on BPA software;

[0011] In the Transformer deep reinforcement learning environment, a Transformer-PPO intelligent agent model is built based on the equivalent mathematical model, and the identification training data set is input for parameter identification training to obtain the final identification result.

[0012] Furthermore, the establishment of the Transformer deep reinforcement learning environment based on the BPA software includes the following steps:

[0013] Status t Set as the parameter to be identified in the photovoltaic power station, and take the change value of the parameter to be identified in the photovoltaic power station as action a t , the state s t With action a t Add together to get the state s at time t+1t+1 :

[0014] s t+1 ←s t +a t (1)

[0015] According to the transient file SWI file format in BPA, the state s t+1 , that is, the values ​​of the PV power station parameters are written into the transient file SWI, and then a transient calculation is performed to obtain a new model response.

[0016] Furthermore, the result file SWX in the BPA software is obtained according to the reactive power curve in the current state, and the reactive power curve calculated by BPA is compared with the measured real data curve, and the reward R is used to determine whether the error standard is met; if the error is less than the allowable value, the photovoltaic parameters are output, and if the error is greater than the allowable value, the parameter identification of the Transformer-PPO algorithm is performed, and the reactive power Q in the current state is read from the result file SWX to calculate the reward R. The calculation formula of reward R is as follows:

[0017]

[0018] Where Q 实测 (t) is the measured reactive power data, Q 等值 (t) is the reactive data output by the equivalent model; n is the number of sampling points. The size of the reward R is used as an evaluation indicator, and each training is performed to observe whether the value of the reward R meets the target.

[0019] Furthermore, the Transformer-PPO agent model structure includes a Transformer module, an Actor neural network, an Actor-old neural network and a Critic neural network.

[0020] Transformer is a deep learning model architecture that introduces a self-attention mechanism, which enables it to perform well when processing sequence data. The PPO algorithm is a policy gradient algorithm based on the deep reinforcement learning Actor-Critic framework. It fits the optimal action by updating the policy function through the obtained reward. PPO improves the strategy by optimizing a specific objective function. It can achieve small batch updates in multiple training steps and can cope with complex situations such as high-dimensional and continuous action spaces. Its core idea is to limit the range of policy updates to achieve stable and efficient training results. The process of the Transformer-PPO algorithm interacting and iterating with the environment: The agent obtains the current state s from the environment t , the agent takes action a tThe results are fed back to the environment, which generates a new state and reward R, and stores the continuously collected data in the RolloutBuffer, and then samples the RolloutBuffer to train the agent. Transformer is used to extract features, and the Actor network is used to output strategies; the Actor-old network and the Actor network have the same structure, and the parameters of the two networks are also the same at the beginning of training. The Actor-old network is used to output strategies and select actions but does not perform gradient updates; the Critic network is used to calculate state values. The Transformer module includes a fully connected layer, a position encoding, and multiple encoding layers constructed in sequence. Each encoding layer includes a sequentially cascaded multi-head attention layer, a normalization layer, a feedforward network, and a GRU gating unit.

[0021] Furthermore, the input identification training data set is used to perform parameter identification training to obtain a final identification result, including the following steps:

[0022] Generate a set of small batches of data (s) from the collection Rollout Buffer t , memory, R) is input into the Transformer module. The memory matrix is ​​used in the Transformer to store and process the embedding vectors obtained by the interaction of past time steps, representing the historical memory of each time step of the model. First, a fully connected layer is used for linear embedding to embed the state data s input at the current time step. t Mapping from a low dimension to an embedding dimension D allows the Transformer to capture more complex relationships. Position encoding is performed on the linearly embedded data to provide the position information of each element in the time series.

[0023] The positional encoding is added to the embedded data at each time step to inject the sequential information of each time step. The encoded data together with the historical state encoding information stored in memory are used as input x t (batch_size,T,N,D) is provided to the attention mechanism module. Where batch_size represents the number of training batches, T represents the number of time series steps in an episode, N represents the number of Transformer layers, and D represents the dimension of the embedding. In the multi-head attention mechanism, the input sequence x is first t Perform a linear transformation:

[0024] Q=W Q ·x t (3)

[0025] K=W K ·x t (4)

[0026] V=WV ·x t (5)

[0027] Among them, W Q ,W K and W V is a learnable weight matrix; Q represents the query matrix, which represents the query information of the model at the current time step, and is used to query the information of other parts of the input sequence; K represents the key matrix, which represents the key information of each position, and is used to compare with the query Q to determine the relevance; V represents the value matrix, which contains the actual information of each position, and determines the output value of each position after Q and K calculate the similarity (attention weight). After the transformation, Q, K and V are divided into multiple heads according to the total embedding dimension, and the dimension of each head is the total embedding dimension divided by the number of heads. The attention weight is calculated for each head, and the calculation formula of the attention weight is as follows:

[0028]

[0029] The softmax() function maps each element to a value between 0 and 1; T represents the length of the input time series; D is the embedding dimension of the input sequence, h is the number of heads for segmentation; mask is a mask used to mask positions that do not need to be paid attention to; Q i ,K i and V i is the query, key and value of the ith head. Finally, we combine the outputs of all heads and pass them through another linear transformation weight matrix W o Integration information:

[0030] Multihead(Q,K,V)=Concat(head1,head2,…head h )W o (7)

[0031] The multi-head attention mechanism module concatenates the multiple vectors output by all heads in the direction of the embedding dimension to form a complete feature vector of size (T, D). This vector contains the information of all heads and can capture the features of different subspaces. The concatenated vector is passed through the normalization layer to obtain y t Then enter the GRU gating unit. The GRU gating unit introduces two gating units, the reset gate and the update gate, which can learn how to adjust the importance of the input information and the hidden state information of the previous moment at each time step. The GRU gating unit accepts the input of the current time step and the hidden state of the previous time step, and returns the hidden state of the current time step. In this way, through recursive application, the hidden state of the previous time step can be obtained at each time step and used for the calculation of the next time step.

[0032] For the reset gate:

[0033] r t =σ(W r ·y t-1 +U r ·y t ) (8)

[0034] Among them, y t is the input of the current time step, i.e., the output of multi-head attention, y t-1 is the hidden state of the previous time step, σ is the Sigmoid activation function, W r is the linear transformation weight from the hidden state to the reset gate, U r is the linear transformation weight from input to reset gate.

[0035] For the update gate:

[0036] z t =σ(W z ·y t-1 +U z ·y t -b g ) (9)

[0037] Among them, σ is the Sigmoid activation function, W z is the linear transformation weight from the hidden state to the update gate, U z is the linear transformation weight from input to update gate, b g is the shared bias term.

[0038] For candidate hidden states:

[0039]

[0040] Among them, tanh is the hyperbolic tangent activation function, W g is the linear transformation weight from hidden state to candidate hidden state, U g is the linear transformation weight from input to candidate hidden state, r t is the output of the reset gate and ⊙ represents element-wise multiplication.

[0041] For the final hidden state:

[0042]

[0043] Among them, z tis the output of the update gate, out1 is the output of the first GRU, out1 is sent to the second GRU gating unit and the feedforward network layer to generate the final output out2 (batch_size, T, D), T and D represent the number of time steps and embedding dimensions of each round respectively. And the memory is updated to contain the information of the current time step, continuously accumulate new information, and can be used in subsequent time steps. The output out2 and reward R of the Transformer module are input to the Critic network, and the state value function is output

[0044] Then calculate the discounted reward δ based on the reward t and advantage function A t :

[0045] δ t =γR t+1 +γ 2 R t+2 +γ 3 R t+3 +…=∑ t′>t γ t′-t R t′ (12)

[0046]

[0047] Among them A t is the advantage function; is the state t The state value function of R t′ is the reward at time t′; t is the current time step, t′ is the future time step greater than t; γ is the discount factor; is the critic network parameter; t is the discounted reward, which represents the cumulative discounted reward value of future rewards starting from the current time.

[0048] Then the network parameters are updated, starting with the update of the Critic network. The Critic network loss function as follows:

[0049]

[0050] in, is the state t The state value function of R t′ is the reward at time t′; t is the current time; γ is the discount factor; the Critic network loss function is the mean square value of the advantage function; T is the termination time, i.e. the total number of time steps. The parameters of the Critic network are updated through back propagation:

[0051]

[0052] in are the parameters of the Critic network, is the learning rate, It is a derivative operation.

[0053] Initialize the number of loops i = 0. Then input out2 into the Actor network and the Actor-old network to obtain two normal distributions Normal1 and Normal2 and the probability distribution function log p corresponding to each action θ (a t |s t )and

[0054] Then the Actor network is updated. For the Actor network, the objective function obtained by PPO-clip operation is:

[0055]

[0056] L clip (θ)=-E[min(r t (θ)A t ,clip(r t (θ),1-∈,1+∈)A t )] (17)

[0057] Among them, r t (θ) is the update amplitude of the policy, also known as KL divergence, which is used to constrain the difference between the current policy and the old policy. θ (a t |s t )and It is the logarithmic value of the strategy distribution obtained based on the Actor network and the Actor-old network, indicating that the current strategy is in state s t Take action a t The probability of the old policy in state s t Take action a t ∈ is a hyperparameter used to control the clipping amplitude; E is the averaging operation.

[0058] The Actor network objective function consists of two parts, and the smallest one is selected as the alternative objective function: t (θ)A t represents the current strategy update amplitude multiplied by the value of the advantage function; clip(r t (θ),1-∈,1+∈)A tIndicates that the current strategy update amplitude is clipped and then multiplied by the value of the advantage function. If r t The value of (θ) is out of the range (1-∈, 1+∈), so it is truncated. t If the value of (θ) is greater than 1+∈, set it to 1+∈; if it is less than 1-∈, set it to 1-∈; if it is between (1-∈, 1+∈), keep it. The role of the clipping function is to limit the magnitude of the policy update to ensure that the policy update is not too large. By calculating L clip The derivative of (θ) implements backpropagation to update the parameters of the Actor network:

[0059]

[0060] Where θ is the parameter of the Actor network, α θ is the learning rate, is a derivative operation. Repeat the trimming and updating operation.

[0061] After a certain step, the Actor network loop ends and the Actor-old network is updated. The Actor network parameters after the loop are copied to the Actor-old network, so that the parameters of the Actor-old network are updated.

[0062] Finally, the Transformer network is updated. By solving the above Actor and Critic network loss functions, the total loss of the network is obtained as:

[0063]

[0064] in is the loss function of the Critic network, L clip (θ) is the loss function of the Actor network, λ is the weight coefficient used to control the importance of KL divergence in the total loss function, and r t (θ) is the KL divergence. The gradient of the loss function with respect to the network parameters is calculated by back propagation Use the Adam optimizer to update the network parameters by gradient descent:

[0065]

[0066] Where η is the parameter of the Transformer network, α η is the learning rate, It is a derivative operation.

[0067] After the update is completed, the Transformer and Actor networks interact with the environment again to obtain data and update the parameters of each network structure.

[0068] The current identified state, action and error are continuously calculated and output. When the error is less than the allowable value, the current parameter value of the photovoltaic model is output, which is the final result of parameter identification of the photovoltaic power station using the measured data curve of the photovoltaic power station and the Transformer-PPO algorithm.

[0069] The beneficial effects of the present invention are as follows: a corresponding equivalent mathematical model is established in the BPA software according to the photovoltaic power station model, the deep reinforcement learning algorithm PPO is combined with the Transformer deep learning model architecture, and the Transformer-PPO algorithm is used to screen parameters and perform accurate and rapid identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is the framework diagram of the mathematical model of the photovoltaic system;

[0071] Figure 2 is the reactive power control model;

[0072] Figure 3 This is the Transformer-PPO network structure diagram;

[0073] Figure 4 It is the Transformer structure diagram;

[0074] Figure 5 This is the principle diagram of the Transformer-PPO algorithm;

[0075] Figure 6 This is the flow chart of photovoltaic power station parameter identification based on Transformer-PPO;

[0076] Figure 7 It is the voltage disturbance comparison diagram;

[0077] Figure 8 It is a comparison chart of reactive power output;

[0078] Fig. 9 This is a comparison chart of active power output;

[0079] Fig.10 This is a comparison chart of reactive current curves. DETAILED DESCRIPTION

[0080] The present invention is described in detail with reference to the accompanying drawings. A photovoltaic power station parameter identification method based on Transformer-PPO includes:

[0081] Obtain the measured data of the photovoltaic power station to construct an identification training data set, and establish an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station;

[0082] Establish a Transformer deep reinforcement learning environment based on BPA software;

[0083] In the Transformer deep reinforcement learning environment, a Transformer-PPO intelligent agent model is built based on the equivalent mathematical model, and the identification training data set is input for parameter identification training to obtain the final identification result.

[0084] Example:

[0085] The present invention follows the steps:

[0086] Step (1): Obtain the measured data of the RTDS of the photovoltaic power station, use the power system simulation software BPA to simulate the measured curve environment of the photovoltaic power station, and establish a mathematical model for the photovoltaic power station to be connected to the power grid.

[0087] The measured data of the photovoltaic power station are obtained, including time (s), active power P3 (MW), reactive power Q3 (Mvar), reactive current (A), line voltage Uab (KV) between AB phases, line voltage Ubc (KV) between BC phases, line voltage Uac (KV) between AC phases, and frequency (Hz). The equivalent mathematical model is established using the simulation tool PSD-BPA (BPA for short) written by the China Electric Power Research Institute. BPA is a comprehensive power simulation software for power system calculation and analysis. It has been widely used in domestic power dispatching and operation organizations, power system planning related units, and various universities. It has functions such as power flow calculation, transient stability simulation calculation, short-circuit calculation, and small interference stability calculation.

[0088] Power flow calculation: The power flow calculation file format is DAT. The power flow data file is a DAT text file defined in BPA that provides data and instructions for power flow calculation. The parameters can be directly edited and modified.

[0089] Transient calculation: The transient calculation file format is SWI. Similar to the power flow data file, component dynamic parameters, fault operation, calculation and output control are all input in the form of cards, and stable calculations under different interference modes can be performed by setting the parameters of the relevant cards.

[0090] In the BPA software, a single-machine mathematical model of the photovoltaic power station is established and the parameters to be identified are selected. The photovoltaic power generation mathematical model based on the BPA software contains multiple card modules. The overall framework diagram is as follows: Figure 1 and Figure 2 The card introduction is shown in Table 1.

[0091] Table 1 Main control model card identification of photovoltaic model

[0092] Card Name Card Logo Photovoltaic power generation model MS Required Active power control model under normal operating conditions EU Required Reactive power control model under normal operation EZ Usually fill in Active and reactive current limiting model ES Required Low voltage and high voltage state judgment model EV Required Active power control model under low voltage ride-through state LP Usually fill in Reactive power control model under low voltage ride-through state LQ Usually fill in Frequency Control Model EY Generally do not fill

[0093] The photovoltaic power generation model is represented by the MS card, which contains important parameters such as the rated capacity of a single photovoltaic array and the DC side capacitor C. The active and reactive models under normal conditions are represented by EU and EZ cards, including a series of control model parameters such as time constants. These parameters have a great influence on the model, and their accuracy also represents the accuracy of the model.

[0094] Step (2): Establish a deep reinforcement learning environment for Transformer-PPO based on BPA software.

[0095] Select the photovoltaic power station parameter voltage measurement time constant T R , Proportional link coefficient K PV , Integral link coefficient K VI , time constant T V and time constant T PV are parameters to be identified, and their ranges are shown in Table 2.

[0096] Table 2 Optimization range of photovoltaic power station parameters

[0097] parameter Lower limit Upper limit <![CDATA[T R ]]> 0.0001 0.01 <![CDATA[K PV ]]> 0.0001 1 <![CDATA[K VI ]]> 0.0001 0.01 <![CDATA[T V ]]> 0.0001 0.01 <![CDATA[T PV ]]> 0.0001 0.011

[0098] After determining the identification parameters and parameter range, the value of the photovoltaic power station parameter is used as the state s t , the change in parameter value is action a t , according to formula (1), we can get the next state s t+1 Then the current state s t The parameter values ​​of the photovoltaic power station are written into the SWI file of the BPA software, and then the transient calculation program of the BPA software is called to perform simulation calculations to obtain the reactive power data in the current state, that is, under the current photovoltaic parameter values.

[0099] like Figure 3 and Figure 4 As shown in the figure, if the action given by Transformer-PPO causes the next state to exceed the set parameter range, the Transformer-PPO agent is given a penalty, making the reward R -10. If the next state is within the range, it is trained normally, and the reward R is obtained according to formula (2).

[0100] Step (3): Establish the Transformer-PPO agent.

[0101] The network structure of Transformer-PPO deep reinforcement learning is as follows Figure 5 As shown in the figure. The deep learning framework is Pytorch. The number of hidden layers of the Actor network, Actor_old network and Critic network is 2, and each layer has 64 neurons. The learning rate of the Actor network is α θis 0.0001, the learning rate of the Critic network is 0.001, the discount factor γ is 0.99, the cropping factor ∈ is 0.2, the number of small batch sample data MINIBATCH generated from the Rollout Buffer each time is 64, the number of multi-head attention num_heads is 6, and the embedding dimension embed_dim is 384.

[0102] a. Step (4): Start parameter identification training and obtain the final identification result.

[0103] The experiment was conducted on an NVIDIA GeForce RTX 4060 GPU. Pytorch was used to build and train the neural network, and Adam was selected as the optimizer. When the error between the simulation result and the measured value is less than the allowable value, the current parameter value is output, which is the identification result.

[0104] The present invention uses the Transformer-PPO algorithm to compare and identify the real data values ​​measured by the PPO algorithm, the SAC multi-agent algorithm, and the RTDS. Figure 6 The identification results are shown in the attached figure. Figure 7 Voltage disturbance output comparison chart, Figure 8 Reactive power output comparison chart, Fig. 9 Active power output comparison chart, Fig.10 This is a comparison chart of reactive current output.

[0105] The identification results of each parameter are shown in Table 3:

[0106] Table 3 Photovoltaic parameter identification results

[0107] Photovoltaic parameters Transformer-PPO Algorithm PPO Algorithm SAC Multi-Agent Algorithm <![CDATA[T R ]]> 0.0057 0.0085 0.0086 <![CDATA[K PV ]]> 0.2542 0.2564 0.3115 <![CDATA[K VI ]]> 0.0010 0.0004 0.0018 <![CDATA[T V ]]> 0.0097 0.0079 0.0061 <![CDATA[T PV ]]> 0.0102 0.0108 0.0104

[0108] The root mean square error of the simulation curve and the detailed model curve is selected as the error judgment standard. The error results and identification time results are shown in Table 4:

[0109] Table 4 Errors and identification time of three algorithms

[0110] RMSE error Identification time (min) Transformer-PPO of the present invention 0.6099 3.36 PPO 0.6177 3.28 SAC Multi-Agent 0.6556 5.34

[0111] Judging from the error results, the errors of the Transformer-PPO algorithm and the PPO algorithm are relatively close, and both are better than the recognition results of the SAC multi-agent. In terms of time, the PPO algorithm is faster, followed by the Transformer-PPO algorithm, and the SAC multi-agent algorithm has the longest recognition time.

Claims

1. A photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning, characterized in that: The process includes the following: Obtain the measured data of the photovoltaic power station to construct an identification training data set, and establish an equivalent mathematical model with the same operating environment as the measured curve of the photovoltaic power station; Establish a Transformer deep reinforcement learning environment based on BPA software; In the Transformer deep reinforcement learning environment, a Transformer-PPO intelligent agent model is built based on the equivalent mathematical model, and the identification training data set is input for parameter identification training to obtain the final identification result.

2. The photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning according to claim 1 is characterized in that: The specific implementation process of establishing a Transformer deep reinforcement learning environment based on BPA software is as follows: Status t Set as the parameter to be identified in the photovoltaic power station, and take the change value of the parameter to be identified in the photovoltaic power station as action a t , the state s t With action a t Add together to get the state s at time t+1 t+1 ; According to the reactive power curve in the current state, the result file SWX in the BPA software is obtained. The reactive power curve calculated by BPA is compared with the measured real data curve, and the reward R is used to determine whether the error standard is met; if the error is less than the allowable value, the photovoltaic parameters are output; if the error is greater than the allowable value, the parameter identification of the Transformer-PPO algorithm is performed, and the reactive power Q in the current state is read from the result file SWX to calculate the reward R; the calculation formula of the reward R is as follows: Where Q 实测 (t) is the measured reactive power data, Q 等值 (t) is the reactive data output by the equivalent model; n is the number of sampling points; each training is performed to observe whether the value of the reward R meets the target.

3. The photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning according to claim 2 is characterized in that: The Transformer-PPO agent model is specifically implemented as follows: The Transformer-PPO agent model structure includes a Transformer module, an Actor neural network, an Actor-old neural network and a Critic neural network; The Transformer-PPO algorithm interacts and iterates with the environment, and the agent obtains the current state s from the environment. t , the agent takes action a t The results are fed back to the environment, which generates a new state and reward R, and stores the continuously collected data in the RolloutBuffer, which is then sampled to train the agent. During the training process, the Transformer module is used to extract features, and the Actor network is used to output strategies. The Actor-old network has the same structure as the Actor network, and the parameters of the two networks are also the same at the beginning of training. The Actor-old network is used to output strategies and select actions but does not perform gradient updates. The Critic network is used to calculate state values. The Transformer module includes a fully connected layer, a positional encoding, and multiple encoding layers constructed sequentially. Each encoding layer includes a sequentially cascaded multi-head attention layer, a normalization layer, a feedforward network, and a GRU gating unit.

4. The photovoltaic power station parameter identification method based on Transformer-PPO deep reinforcement learning according to claim 3 is characterized in that: The input identification training data set is used for parameter identification training to obtain the final identification result. The specific process is as follows: Generate a set of batch data (s) from the collection Rollout Buffer t ,menory,R) is input into the Transformer module, where the menory matrix is ​​used in the Transformer to store and process the embedded vectors obtained by the interaction of past time steps, representing the historical memory of each time step of the model; first, it is linearly embedded through a fully connected layer to transform the state data s input at the current time step t Mapped to an embedding dimension D, the linearly embedded data is positionally encoded to provide the position information of each element in the time series; The positional encoding is added to the embedded data at each time step, injecting the order information of each time step; the encoded data together with the historical state encoding information stored in memory are used as input x t ; Then calculate the discounted reward δ based on the reward t and advantage function A t : is the current time step, t′ is the future time step greater than t; γ is the discount factor; is the critic network parameter; t is the discounted reward, which represents the cumulative discounted reward value of future rewards starting from the current time; Then the network parameters are updated, starting with the update of the Critic network; the loss function of the Critic network γ is the discount factor; the loss function of the Critic network is the mean square value of the advantage function; T is the termination time, that is, the total number of time steps; the parameters of the Critic network are updated through back propagation; Then the Actor network is updated. For the Actor network, the objective function obtained by PPO-clip operation is: L clip (θ)=-E[min(r t (i)A t ,clip(r t (θ),1-∈,1+∈)A t )] Among them, r t (θ) is the update amplitude of the strategy, also known as KL divergence, which is used to constrain the difference between the current strategy and the old strategy, logp θ (a t |s t )and It is the logarithmic value of the strategy distribution obtained based on the Actor network and the Actor-old network, indicating that the current strategy is in state s t Take action a t The probability of the old policy in state s t Take action a t ∈ is a hyperparameter used to control the clipping amplitude; E is the averaging operation; The Actor network objective function consists of two parts, and the smallest one is selected as the alternative objective function: t (θ)A t represents the current strategy update amplitude multiplied by the value of the advantage function; clip(r t (θ),1-∈,1+∈)At represents the current strategy update amplitude after clipping and then multiplied by the value of the advantage function. If r t The value of (θ) is out of the range (1-∈, 1+∈), so it is truncated. t If the value of (θ) is greater than 1+∈, then set it to 1+∈; if it is less than 1-∈, then set it to 1-∈; if it is between (1-∈, 1+∈), then keep its value; by calculating L clip The derivative of (θ) implements back-propagation to update the parameters of the Actor network; The Actor network loop ends, and the Actor-old network is updated; the Actor network parameters after the loop are copied to the Actor-old network, and the parameters of the Actor-old network are updated; Finally, the Transformer network is updated; by solving the above Actor and Critic network loss functions, the total loss is: in is the loss function of the Critic network, L clip (θ) is the loss function of the Actor network, λ is the weight coefficient used to control the importance of KL divergence in the total loss function, and r t (θ) is the KL divergence; the gradient of the loss function with respect to the network parameters is calculated by backpropagation The Adam optimizer is used to update the network parameters by gradient descent. After the update is completed, the Transformer and Actor networks interact with the environment again to obtain data and update the parameters of each network structure. The current identified state, action and error are continuously calculated and output. When the error is less than the allowable value, the current parameter value of the photovoltaic model is output, which is the final result of parameter identification of the photovoltaic power station using the measured data curve of the photovoltaic power station and the Transformer-PPO algorithm.

Images