Power load model parameter identification method based on improved deep reinforcement learning

Abstract

The invention discloses a power load model parameter identification method based on improved deep reinforcement learning, and belongs to the technical field of power load identification. The method comprises the following steps: determining a to-be-identified power load parameter based on a complete comprehensive load model considering power distribution network impedance and reactive compensation; then designing a power load parameter identification model based on a bidirectional gating cycle unit depth deterministic strategy gradient algorithm of priority experience playback; circuit data are collected for model training, and the trained model is used for parameter identification. On the basis of an existing power system mechanism model and a data driving model, the load model parameters are identified by improving the deep reinforcement learning model, and verification shows that the method can accurately identify the power system load model parameters, can improve the fitting precision of the power system load model, and has a good application prospect. The operation strategy of the power grid is optimized, and the operation cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of power load identification, and relates to the technology of cross-integration of power system load model research and information science, and specifically to a method for identifying power load model parameters based on improved deep reinforcement learning. Background Art

[0003] The load change of the power system is a nonlinear, strongly random time-varying process. It is difficult to build an accurate load model and identify key parameters. Reference (Chen Qian, Tang Yong, Ju Ping, et al. Parameter Identification of a Complete Integrated Load Model Considering Distribution Network Impedance and Reactive Power Compensation [J]. Proceedings of the CSEE, 2010, 30(22): 44-50.) In the technical solution published in August 2010, the following was constructed: Figure 1 The comprehensive load model shown in the figure takes into account the impedance of the distribution network, motors, static loads and capacitor compensation. In this scheme, only key parameters are identified based on the comprehensive load model, and typical values ​​are used for other parameters. The accuracy of the parameters in the load model directly affects the correctness of the operation and planning decisions of the power system. In recent years, a series of major power outages have occurred at home and abroad, causing huge economic losses, indicating that it is still necessary to explore accurate load models with important economic and social values. Accurate identification of load model parameters is crucial for research on the planning, operation and control of power systems.

[0004] At present, the algorithms used for power system modeling and parameter identification are mainly divided into two categories: data-driven models and mechanism-driven models. The data-driven model needs to collect sufficient data to extract the characteristic indicators of load changes, and then build a model for the power load changes. The mechanism-driven model describes the load change process through dynamic equations, analyzes the transmission and transfer of various power parameters, and thus builds a power load model. Although the data-driven model can mine the inherent laws of power load in the data, as a type of black box model, it is difficult to reveal the inherent mechanism of the power load change process, and the problem of poor interpretability hinders the promotion of the data-driven model. In addition, although the regularity described by the mechanism-driven model is more intuitive, because it focuses on the complex relationship between various factors in a specific environment, the construction of the mechanism-driven model requires more parameters and is more complicated in practical applications. Summary of the invention

[0005] In view of the problem that many parameters are required when constructing a mechanism-driven model in the current power system modeling and parameter identification technology, which is not easy to apply in practice, the present invention provides a power load model parameter identification method based on improved deep reinforcement learning. On the basis of the existing power system mechanism model and data-driven model, the deep reinforcement learning model is improved, and the load model parameters are identified using IEEE9-node system simulation data. The comprehensive load model of the power system after parameter optimization and identification is tested and verified, which proves that the present invention can achieve the purpose of accurately identifying the power system load model parameters.

[0006] The present invention proposes a method for identifying power load model parameters based on improved deep reinforcement learning, comprising the following steps:

[0007] Step 1: Establish a complete integrated load model taking into account distribution network impedance and reactive power compensation;

[0008] Determine the input state vector, output data vector and parameters to be identified in the comprehensive load model; collect active power P(k), reactive power Q(k) and voltage data U(k) at time k from the measurement point of the circuit to be studied to form an input state vector x(k). Let the parameter vector to be identified at time k of the comprehensive load model be θ(k). The output data vector s(k) calculated by the comprehensive load model at time k includes: the active power P consumed by the induction motor M (k) and reactive power Q M (k), active power consumed by the static load part P ZIP (k) and reactive power Q ZIP (k), the active power consumed on the distribution network branch P D (k) and reactive power Q D (k), and the reactive power Q consumed by the compensation capacitor C (k);

[0009] The input and output of the comprehensive load model are expressed as: s(k)=SLM(x(k),θ(k)); SLM represents the comprehensive load model;

[0010] Step 2: Use the improved deep reinforcement learning model structure to establish a power load parameter identification model;

[0011] The proposed improved deep reinforcement learning model is implemented by a deep deterministic policy gradient method based on bidirectional gated recurrent units with priority experience replay. The overall model is composed of an execution network and an evaluation network structure composed of bidirectional gated recurrent units as the basic framework. The structures and parameters of the four bidirectional gated recurrent neural networks corresponding to the main execution network, the target execution network, the main evaluation network and the target evaluation network are designed. The experience replay pool is improved to a priority experience replay pool to speed up the learning and training of the agent. The priority of the four-tuple sample in the pool is defined based on the amplitude of the temporal difference error of the sample. The larger the amplitude of the temporal difference error, the greater the priority of the sample. The temporal difference error of the sample refers to: the execution network calculates the value function of the current state-policy pair, and updates the parameters of the evaluation network by minimizing the difference between the predicted value function and the Bellman optimal value through the gradient descent method. The difference is the temporal difference error.

[0012] The input of the power load parameter identification model is the state vector x(k). The execution network calculates the optimal strategy π representing the change of the parameter to be identified based on the input x(k). ν (x(k)), combined with the previous moment parameter θ(k-1), we get the updated parameter θ(k).

[0013] Step 3: training the power load parameter identification model, and using the trained model to perform parameter identification;

[0014] First, set the maximum number of iterations N max The maximum time step k in each iteration max ; Then, obtain the experience sample from the priority experience replay pool and input it into the improved deep reinforcement learning model, and

[0015] [x(k),θ(k),r(k),x(k+1)], the main execution network outputs strategy π for x(k) ν (x(k)), the main evaluation network calculates the value function Q for the state-strategy pair (x(k), θ(k)) μ (x(k),θ(k)), the target execution network outputs strategy π for x(k+1) ν (x(k+1)), the target evaluation network calculates the value function Q for the state-strategy pair (x(k+1),θ(k+1)) μ' (x(k+1),θ(k+1)); iteratively update the weight parameters of the execution network and the evaluation network to minimize the set model evaluation index value until the number of iterations reaches the maximum number, end the iteration, and obtain the currently trained power load parameter identification model; use the trained power load parameter identification model to perform parameter identification.

[0016] In the step 2, the main execution network and the target execution network in the execution network both include: 3 input neurons, corresponding to the three elements of the input state vector x(k); a There are 11 hidden layer neurons, each of which is implemented by a bidirectional gated recurrent unit; 11 output neurons, corresponding to the changes of 11 parameters to be identified; the activation function of the hidden layer uses a unipolar sigmoid function, and the activation function of the output layer uses a nonlinear tanh function;

[0017] The main evaluation network and the target evaluation network in the evaluation network both contain: 14 input neurons, corresponding to the 3 elements of the input state vector x(k) and the 11 parameters to be identified of θ(k); c hidden layer neurons, each hidden layer neuron is implemented using a bidirectional gated recurrent unit; 1 output neuron, corresponding to the value function of the current state-strategy pair; the hidden layer activation function uses a unipolar sigmoid function, and the output layer activation function uses a nonlinear tanh function.

[0018] Compared with the prior art, the advantages of the present invention are:

[0019] (1) The method of the present invention utilizes the deviation feedback between the output of the power system load model and the IEEE9 node system measurement value to optimize and reduce the deviation of the power system load model, so as to improve the fitting accuracy of the power system comprehensive load model. The power load parameter identification model constructed by the present invention can show strong robustness and generalization ability under different line faults.

[0020] (2) The present invention improves the structure of the power system load model parameter identification method based on deep reinforcement learning, cleverly integrates the mechanism model of the power load with the time series data of the simulation software, and uses the data to optimize and identify the parameters of the comprehensive load model of the power system, so that the constructed power load model parameter identification method based on improved deep reinforcement learning has the advantages of both mechanism and data-driven models.

[0021] (3) The method of the present invention is experimented, compared with the real data curve of the simulated fault, and the verification results are analyzed, which shows the effectiveness of the method of the present invention, and can provide reference and support for researchers to further study the parameter identification method of the power system load model. The use of the method of the present invention helps to improve the fitting accuracy of the power system load model, optimize the operation strategy of the power grid, reduce the operating cost, and provide decision support for achieving sustainable development of energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The structure diagram of the comprehensive load model considering the distribution network impedance, motor, static load and capacitor compensation;

[0023] Figure 2It is a flow chart of the method for identifying parameters of a power system load model based on improved deep reinforcement learning of the present invention;

[0024] Figure 3 An improved deep reinforcement learning structure diagram for time-varying parameter identification used in the method of the present invention;

[0025] Figure 4 This is a diagram of the execution network structure of the deep reinforcement learning used in the method of the present invention;

[0026] Figure 5 This is a structural diagram of the evaluation network of deep reinforcement learning used in the method of the present invention;

[0027] Figure 6 Flowchart for time-varying parameter identification for a comprehensive load model of a power system;

[0028] Figure 7 This is the IEEE9-node system structure diagram;

[0029] Figure 8 It is a comparison diagram of the comprehensive load model prediction structure based on different models when the line 6-4 fails in the embodiment of the present invention; wherein 8A, 8B, 8C and 8D are active power comparison curve, reactive power comparison curve, loss curve and reward curve respectively;

[0030] Fig. 9 It is a comparison diagram of the prediction results of the comprehensive load model based on different models when the line 6-9 fails in the embodiment of the present invention; wherein 9A, 9B, 9C and 9D are active power comparison curve, reactive power comparison curve, loss curve and reward curve respectively;

[0031] Fig.10 It is a comparison chart of the prediction results of the comprehensive load model based on different models when the line 8-9 fails in the embodiment of the present invention; wherein 10A, 10B, 10C and 10D are active power comparison curve, reactive power comparison curve, loss curve and reward curve respectively. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and examples.

[0033] The present invention provides a method for identifying the parameters of an electric load model based on improved deep reinforcement learning. Aiming at the parameter identification problem of the comprehensive load model of the power system, a deep deterministic policy gradient method of a bidirectional gated cyclic unit based on priority experience replay is used for identification. The method of the present invention first constructs a complete comprehensive load model taking into account the impedance of the distribution network and reactive compensation, and then studies the use of the execution network and the evaluation network in the improved deep reinforcement learning to perform parameter identification of the complete comprehensive load model taking into account the impedance of the distribution network and reactive compensation. Finally, based on the identified complete comprehensive load model taking into account the impedance of the distribution network and reactive compensation, the active power and reactive power of different line faults are verified, thereby providing a scientific basis for the optimized operation, reliability analysis and fault recovery of the power system. By accurately fitting the changes in active power and reactive power under different line fault conditions, power system operators can better carry out load balancing, voltage stability control and the formulation of power grid dispatching strategies.

[0034] like Figure 2 As shown, the power load model parameter identification method based on improved deep reinforcement learning of an embodiment of the present invention mainly includes the following four steps, and each step is described below.

[0035] Step 1: Construct a complete integrated load model taking into account distribution network impedance and reactive power compensation.

[0036] Build as Figure 1 The complete integrated load model taking into account the impedance of the distribution network and reactive power compensation is shown. It is assumed that in this model, the normal operating frequency is the rated frequency, and the power system parameters are all in per-unit value, with the unit being pu. is the actual load bus voltage, is the virtual load bus voltage, and the part between the actual load bus and the virtual load bus is the equivalent distribution network impedance. The power system comprehensive load model equation consists of four parts, namely the distribution network impedance part, the motor part, the static load part and the compensation capacitor part. The calculation formula of each part of the model is as follows:

[0037] (1) The calculation formula of the impedance part of the distribution network is as follows:

[0038] make Depend on Figure 1 It can be seen that:

[0039]

[0040] Among them, U S , P, Q are the voltage value, active power and reactive power on the actual load bus, R D is the distribution network resistance, X D is the reactance of the distribution network, f is the frequency, P D , Q Dare the active power and reactive power on the distribution network, P L , Q L are the active power and reactive power on the virtual load bus respectively; j represents the imaginary unit.

[0041] (2) The calculation formula for the motor part is as follows:

[0042] The induction motor adopts a third-order model that takes into account electromechanical transients and is converted into the system's common xy coordinate system. The dynamic equation is:

[0043]

[0044] Where, t represents time; E' x , E' y are the direct axis and quadrature axis transient potentials under synchronous coordinates respectively; X=X S +X m is the steady-state reactance, X m is the magnetizing reactance, X S is the stator reactance; X'=X S +X m X S / (X m +X S ) is the transient reactance; ω r is the rotor angular velocity; T d '0=(X m +X r ) / R r is the rotor winding time constant; R r is the rotor resistance; X r is the rotor reactance; I x ,I y The stator current is The real and imaginary parts of T j is the inertia time constant; T E =(E' x I x +E' y I y ) / ω r is the electromagnetic power; is the mechanical power, A, B, c are the mechanical torque coefficients, T M0 is the initial mechanical torque.

[0045] (3) The calculation formula of the static load model is as follows:

[0046]

[0047] Among them, P ZIP , Q ZIP are respectively the active power and reactive power of the static load model, PZIP0 , Q ZIP0 are respectively the initial active power and reactive power of the static load model, L DP , L DQ are the characteristic coefficients of active frequency and reactive frequency respectively, Δf is the frequency change; U L is the voltage on the virtual load bus, U L0 is the initial voltage on the virtual load bus; Z p ,I p , P p , Z q ,I q , P q They are active constant impedance coefficient, active constant current coefficient, active constant power coefficient, reactive constant impedance coefficient, reactive constant current coefficient, and reactive constant power coefficient, and they satisfy:

[0048]

[0049] (4) The calculation formula for the compensation capacitor part is as follows:

[0050] At per unit value,

[0051]

[0052] So the compensation capacitor capacity is Q C for:

[0053]

[0054] The capacitance C of the compensation capacitor can be further obtained as follows:

[0055]

[0056] Among them, X C is the capacitive reactance of the capacitor, X C0 is the capacitor impedance during normal operation, Q C0 is the initial value of the compensation capacitor.

[0057] At this point, the complete power system comprehensive load model equation is obtained. The overall model structure is denoted as SLM, which is composed of equations (1) to (9).

[0058] The parameters of the power system comprehensive load model to be identified are: stator reactance X S , distribution network reactance X D , Active constant impedance coefficient Z P , Active constant current coefficient I P , Active constant power coefficient P p , Reactive constant impedance coefficient Z q , Reactive constant current coefficient I q, Reactive constant power coefficient P q 、Capacitor reactance X C , Motor ratio P MP and motor load factor K L .

[0059] In the embodiment of the present invention, after setting different line faults in the IEEE9 system collected in the simulation software, the fault data voltage, active power, and reactive power on the load bus of node 5 are collected and set as the input x(k) of the comprehensive load model of the power system. Then, the above equation can be written as:

[0060] s(k)=SLM(x(k),θ(k)) (10)

[0061] s(k)=[s1(k),s2(k)] T =[P M (k),P ZIP (k),P D (k),Q M (k),Q ZIP (k),Q D (k),Q C (k)] T (11)

[0062] s1(k)=[s 11 (k),s 12 (k),s 13 (k)] T =[P M (k),P ZIP (k),P D (k)] T (12)

[0063] s2(k)=[s 21 (k),s 22 (k),s 23 (k),s 24 (k)] T =[Q M (k),Q ZIP (k),Q D (k),Q C (k)] T (13)

[0064] x(k)=[x1(k),x2(k),x3(k)] T =[P(k),Q(k),U(k)] T (14)

[0065]

[0066] Among them, SLM(·) represents the complete comprehensive load model of the power system. The collected active power P(k), reactive power Q(k), and voltage data U(k) at time k are used as the input state x(k) of reinforcement learning and input into the comprehensive load model of the power system. At the same time, according to the improved deep reinforcement learning model, the deep deterministic policy gradient algorithm of the bidirectional gated recurrent unit based on priority experience replay is adopted to obtain the identification value θ(k) of the parameter to be identified, where θ(k) is the set vector of the parameter to be identified. Therefore, according to the input data x(k) and the identification parameter value θ(k), the comprehensive load model of the power system can calculate the active power P consumed by the output induction motor. M (k), reactive power Q M (k) Active power consumed in the static part P ZIP (k), reactive power Q ZIP (k) Active power consumed on the distribution network branch P D (k), reactive power Q D (k), and the reactive power Q consumed by the compensation capacitor C (k), and the calculated output is denoted as s(k).

[0067] Step 2: Design an improved deep reinforcement learning model for parameter identification, that is, construct an evaluation network and an execution network in the improved deep reinforcement learning algorithm to generate a power load parameter identification model.

[0068] The improved deep reinforcement learning model of the present invention is implemented by a bidirectional gated recurrent unit (BiGRU) deep deterministic policy gradient (DDPG) algorithm based on priority experience replay, and is marked as a PER-BIGRU-DDPG (A bidirectionalgated recurrent unit depth deterministic policy gradient algorithm based on priority experience replay) model. Among them, the idea of ​​the deep deterministic policy gradient DDPG algorithm is to use the function approximation ability of deep learning to approximate the action value function and deterministic strategy in reinforcement learning. By combining the fitting ability of neural networks with the optimization method of policy gradients, the deep deterministic policy gradient can effectively learn the optimal control strategy in the continuous action space. The deep deterministic policy gradient algorithm includes an execution network module and an evaluation network module, the execution network module includes a main execution network and a target execution network, and the evaluation network module includes a main evaluation network and a target evaluation network. Here, the main execution network is used to map the nonlinear relationship with the strategy adopted by the agent in the current state, the main evaluation network is used to map the nonlinear relationship between the strategy adopted by the agent in the current state and the state strategy value that can be obtained, and the target execution network and the target evaluation network are used as network update indicators for the main execution network and the main evaluation network.

[0069] The present invention realizes the identification of power system load model parameters based on the improved deep reinforcement learning model. According to the reinforcement learning concept, the intelligent agent interacts with the environment to generate training data for updating the execution network and evaluation network parameters. The training data is called experience and stored in the priority experience playback pool. The overall structure of the power load parameter identification model of the present invention is as follows: Figure 3 As shown, the parameter vector to be identified is used as an intelligent agent to interact with the environment.

[0070] The process of generating experience and storing it in the priority experience replay pool is as follows: input state x(k), and obtain strategy π through the main execution network ν (x(k)), this strategy represents the change in the parameter to be identified, that is, Δθ(k) = π ν (x(k)), in order to improve the exploratory nature of the strategy and avoid the strategy falling into the local optimum, Gaussian noise is added to the strategy so that the model can better find the optimal strategy. That is, the strategy θ(k) taken at the kth time step is:

[0071] θ(k)=θ(k-1)+π ν (x(k))+N(ξ,σ) (16)

[0072] Among them, N(ξ,σ) represents the introduced Gaussian noise, which satisfies the Gaussian distribution with mean ξ and variance σ.

[0073] The adopted strategy θ(k) and the input state x(k) are input into the comprehensive load model of the power system. According to formula (10), the output s(k) can be obtained. The output is compared with the fixed environment set initially, where the fixed environment is the data collected by the simulation software, and the reward r(k) based on the improved deep reinforcement learning model is obtained. Therefore, the reward of the deep deterministic policy gradient algorithm of the bidirectional gated recurrent unit based on priority experience replay of the present invention is defined as:

[0074]

[0075] After executing the above steps, the agent interacts with the environment and reaches the next state x(k+1). Based on this, a four-tuple [x(k), θ(k), r(k), x(k+1)] is obtained as experience and stored in the priority experience replay pool. The agent interacts with the environment continuously, generates experience and stores it in the priority experience replay pool. When the number of experiences stored in the replay pool, batch_size, accumulates to equal the minimum training batch, mini_batch, the execution network and the evaluation network are trained.

[0076] The data of the training execution network and the evaluation network are obtained from the priority experience replay pool by sampling experience quadruple, and the sampled quadruple follows the priority probability distribution. In the priority experience replay pool, the priority of the sampled data tuple is defined as the amplitude of the time difference error. The quadruple data stored in the priority experience replay pool is used as experience. Each experience is given a priority. For the newly generated samples of the interaction between the agent and the environment, the samples with larger values ​​of the time difference error are given a greater priority. The formula for calculating the priority p(i) of each experience sample in the priority experience replay pool is:

[0077] p(i)=|δ(i)|+ε (18)

[0078] Wherein, δ(i) is the timing differential error of the i-th sample, which is calculated as shown in formula (26), and ε is a very small positive number to avoid the situation where the sample will not be sampled when the timing differential error is 0.

[0079] After determining the priority of each experience sample, their sampling probability can be calculated based on these priorities. The sampling probability P(i) is calculated as follows:

[0080]

[0081] Where p(i) represents the priority of the i-th experience sample in the experience replay pool, α is the priority parameter, which is used to control the priority distribution index, that is, to control the bias of the sampling strategy, emphasizing the importance of the priority. If α = 0, the sampling degenerates into the basic uniform sampling. M is the total number of experience samples in the experience replay pool.

[0082] Since the defined priority adjusts the sampling probability of the empirical sample, that is, it introduces a bias in the data distribution, the importance-sampling weights are used to compensate for the bias. The calculation formula of the importance sampling weight w'(i) is:

[0083]

[0084] Among them, when the weight β = 1, it can be considered that the deviation introduced by non-uniform probability sampling is fully compensated. At the same time, considering the convergence, β starts from a value between 0 and 1 and gradually increases to 1. When training a deep reinforcement learning model, the sampling probability of the experience sample in the priority experience replay pool is finally determined by the product of the importance sampling weight of the sample and the priority p(i). In addition, in order to increase the stability of training, the importance sampling weight of each experience sample is normalized. Finally, the sampling weight of the experience quadruple data is:

[0085]

[0086] In the above, the normalized weight calculation is performed by dividing the importance sampling weight w'(i) of each experience sample by the maximum value maxw'(i) of all weights, and adjusting the scale of the weights so that the largest weight becomes 1, while the other weights are scaled according to their ratio to the maximum weight. The maximum value of the weight is normalized to 1, and the other weights become smaller accordingly, but still maintain the original proportional relationship.

[0087] The input of the execution network module and the evaluation network module is the four-dimensional data group in the priority experience playback pool, and the output of the execution network module is the strategy π taken at time k ν (x(k)), according to the strategy, the estimated value of the parameters to be identified in the power system load model can be obtained. The output of the evaluation network module is the strategy π adopted by the execution network module at time k ν The evaluation score Q of (x(k)) μ (x(k),θ(k)).

[0088] The purpose of the main execution network is to output the value function Q according to the input state x(k) μ (x(k),θ(k)) The optimal strategy π ν (x(k)). The purpose of the main evaluation network is to output an accurate value Q based on the input state and strategy pair (x(k), θ(k)) at time k μ (x(k),θ(k)).

[0089] The updating process of the main execution network is as follows: a batch of experience data is sampled from the priority experience playback pool, and the input state data x(k) of each batch of experience is sent to the online execution network (i.e., the main execution network). The main execution network outputs the strategy π ν (x(k)) to change the initial setting value of the parameter to be identified, that is, to obtain the estimated parameter value θ(k) of the comprehensive load model of the power system to be identified. Then, the state and strategy pair (x(k), θ(k)) is input to the main evaluation network, and the main evaluation network outputs the value (i.e., Q value) of the state-strategy pair Q μ (x(k),θ(k)), and perform policy gradient ascent on this value to update the main execution network parameters.

[0090] The updating process of the main evaluation network is as follows: a batch of experience data is sampled from the priority experience replay pool, and the state and strategy pairs (x(k), θ(k)) in each batch of experience data are input into the main evaluation network, and the main evaluation network outputs the value function value Q of the state-strategy pair μ (x(k),θ(k)). At the same time, the next state value x(k+1) in each batch of experience data is input into the target execution network, and the target execution network outputs the strategy π for the next state value x(k+1) ν (x(k+1)), the input state-strategy pair (x(k+1),θ(k+1)) is input into the target evaluation network, and the target evaluation network outputs the corresponding value function value Q μ' (x(k+1),θ(k+1)), the value function is calculated by Bellman optimal equation to obtain the optimal expected return as the update index of the main evaluation network. The goal of the main evaluation network is to make the predicted Q μ The more accurate the (x(k),θ(k)) value is, the better. Therefore, its loss function is the difference between the output of the main evaluation network and the sum of the output value of the target evaluation network at the next moment and the discounted reward. Thus, the main evaluation network parameters are updated by minimizing the loss function.

[0091] The training evaluation index E of the power system load parameter identification model based on improved deep reinforcement learning in the present invention is defined as:

[0092]

[0093] Where N represents the number of samples in the current batch.

[0094] The purpose of the deep deterministic policy gradient algorithm is to obtain the optimal strategy that can maximize the long-term cumulative return of formula (17) by training the evaluation network and the execution network, so as to minimize the indicator E of formula (22) and obtain the parameters to be identified.

[0095] The execution network is trained to learn a deterministic policy. The execution network outputs the optimal policy under a given input state to maximize the maximum long-term return evaluated by the evaluation network. The input vector of the execution network is x(k) = [P(k), Q(k), U(k)] T , respectively represent the active power, reactive power and voltage stored in the priority experience playback pool, and the output vector is

[0096] a(k)=[a1(k),a2(k),a3(k),a4(k),a5(k),a6(k),a7(k),a8(k),a9(k),a 10 (k),a 11 (k)] T , each component represents the change of 11 parameters to be identified in the power system load model. Therefore, the execution network has 3 input neurons, h a The hidden layer neurons and 11 output neurons. The hidden layer activation function of the execution network uses a unipolar sigmoid function, and the output layer activation function uses a nonlinear tanh function. The execution network structure is as follows Figure 4 As shown in Figure 1. The execution network uses a bidirectional gated recurrent unit neural network, which includes two gated recurrent unit layers, forward and backward. The calculation process of the execution network includes both forward and backward processing, and can use both past information and future information to make decisions. Figure 4 As shown in Figure 1, each hidden layer neuron is implemented using a bidirectional gated recurrent unit. The execution network is trained using the policy gradient method to calculate the value gradient of the action output by the execution network, and the gradient ascent method is used to maximize the increase in the output function Q of the evaluation network. μ (x(k),θ(k)) to update the parameters of the execution network.

[0097]

[0098] Among them, suppose that at time k, the weight parameter of the main execution network is ν(k), and the weight parameter of the main evaluation network is μ(k). To simplify the description, the subscripts ν(k) and μ(k) are abbreviated as ν and μ. is the gradient of the policy weight parameter output by the main execution network, is the gradient of the weight parameters of the main evaluation network, is the gradient of the weight parameter of the main execution network. J(ν(k)) represents the main execution network with weight parameter ν(k).

[0099] The network parameters of the main execution network are updated using the policy gradient ascent method as follows:

[0100]

[0101] Among them, α ais the learning rate of the main execution network, ν(k) is the weight parameter at time k, and ν(k+1) is the weight parameter after one step of update.

[0102] The input of the target execution network is the input state x(k+1) of the next time step, and the output is the strategy π taken in this state ν (x(k+1)). The network parameters of the target execution network are updated as soft updates, with the aim of smoothly updating the target network during training, thereby improving learning stability.

[0103] ν'(k+1)=τν(k)+(1-τ)ν'(k) (25)

[0104] Among them, τ is a very small positive number, indicating the step size of the update, ν'(k) is the target execution network weight parameter at time k, and ν'(k+1) is the target execution network weight parameter after one step of soft update.

[0105] Train the evaluation network to learn the value function of a state policy, i.e., Q μ (x(k),θ(k)) function, which can accurately estimate the expected return that can be obtained by the strategy selected by the execution network under a given input state, provide feedback to the execution network, evaluate the performance of the current strategy, and guide the direction of strategy improvement.

[0106] The input vector of the evaluation network consists of the state characteristics of the power system load model in the empirical quadruple and the strategy output by the execution network, that is, [x(k),θ(k)] T , the output vector is the estimated value Q of the current state strategy pair predicted by the evaluation network μ (x(k),θ(k)). This value function estimates the average return that can be obtained by taking strategy π starting from state x(k). Therefore, the evaluation network has 14 input neurons, h c The hidden layer neurons and 1 output neuron. The hidden layer activation function of the evaluation network uses a unipolar sigmoid function, and the output layer activation function uses a nonlinear tanh function. The evaluation network structure is as follows: Figure 5 As shown in Figure 1. The execution network uses a bidirectional gated recurrent unit neural network, which includes two gated recurrent unit layers, forward and backward. The calculation process of the execution network includes both forward and backward processing, and can use both past information and future information to make decisions. Figure 5 As shown, each hidden layer neuron is implemented using a bidirectional gated recurrent unit. The training of the execution network uses the value estimation method to calculate the value function Q of the current state-policy pair μ (x(k),θ(k)), and minimize the predicted value function Q by gradient descent μThe difference between (x(k),θ(k)) and Bellman's optimal value y(k+1) is used to update the parameters of the evaluation network. This difference is the temporal difference error, as shown in the following two formulas:

[0107]

[0108]

[0109] Among them, at time k, γ is the discount factor, Q μ' (·) is the output of the target evaluation network, ν' is the weight parameter of the target execution network, δ(k) is the temporal difference error, Main evaluation network J Q The gradient of the weight parameter, is the gradient of the temporal difference error.

[0110] The gradient of the loss function with respect to the evaluation network parameters is calculated through the back propagation algorithm, and the parameters of the main evaluation network are updated using the gradient descent method to minimize the loss function, as follows:

[0111]

[0112] Among them, μ(k+1) is the weight parameter of the main evaluation network at the next moment of update, μ(k) is the weight parameter of the main evaluation network at moment k, and α c is the learning rate of the evaluation network. In this way, the evaluation network learns how to accurately estimate the value of a given state-action pair, providing feedback to the execution network to help it find the optimal policy.

[0113] The input of the target evaluation network is the state x(k+1) of the next time step and the strategy π output by the target execution network ν' (x(k+1)), the output is the estimated value Q of the state-policy pair μ' (x(k+1),π ν' (x(k+1)), the optimal expected return is calculated according to the Bellman optimal equation as the indicator y(k+1) for updating the main evaluation network:

[0114] y(k+1)=r(k)+γQ μ' (x(k+1),π ν' (x(k+1))) (29)

[0115] The target evaluation network weight update adopts the soft update method, and the update formula is:

[0116] μ'(k+1)=τμ(k)+(1-τ)μ'(k) (30)

[0117] Among them, μ'(k) is the weight parameter of the target evaluation network at time k, and μ'(k+1) is the weight parameter of the target evaluation network after one step of soft update.

[0118] Step 3: Parameter identification of the complete integrated load model taking into account distribution network impedance and reactive power compensation.

[0119] After designing the structure of the deep deterministic policy gradient algorithm with bidirectional gated recurrent unit based on priority experience replay, this step performs parameter identification on the complete integrated load model taking into account the distribution network impedance and reactive power compensation. Figure 6 The paper shows the process of comprehensive load model parameter identification based on the deep deterministic policy gradient method of bidirectional gated recurrent unit with priority experience replay. The focus is on the training of the execution network and the evaluation network in the deep deterministic policy gradient algorithm, which includes two training processes: the inner loop and the outer loop. The inner loop is the neural network training for each monitoring time k, and the outer loop is the neural network training for each iteration number N. Whenever the training completes all the monitoring time k=k max , the inner loop iteration training is completed; when the number of completed iterations N reaches the maximum number of iterations N max When , the trained execution network and evaluation network can be obtained.

[0120] Figure 6 The specific process is as follows: Initialize the parameters of the evaluation network and the target network, and create the target network of the two networks, whose parameters are the same as the original network. Input the data in the priority experience revisit pool into the evaluation network and the execution network, and the execution network outputs the strategy π corresponding to each time k ν (x(k)), thus obtaining the predicted parameter value θ(k) to be identified, and the evaluation network outputs the value Q of the strategy taken by the execution network under the input state μ (x(k),θ(k)), complete the adjustment of the weights of the execution network and the evaluation network until the maximum monitoring time k=k is met max ; Then check whether the maximum number of iterations N is reached max If it is reached, the trained execution network and evaluation network and the optimal identification parameter value corresponding to each moment are obtained, otherwise it returns to the inner loop and re-adjusts the weights of the execution network and evaluation network until the number of iterations n = N is met. max .

[0121] Step 4: Use the trained power load parameter identification model based on improved deep reinforcement learning for verification.

[0122] After completing the parameter identification in step 3, the power system load model based on improved deep reinforcement learning is used for verification. That is, the input vector x(k) constructed at time k is input into the trained execution network, and the execution network outputs the optimal strategy at time k. The parameter value of the power system load model to be identified is obtained by equation (16), and the output s(k) is obtained by substituting it into the comprehensive load model of the power system, thereby realizing the verification of the active and reactive power of the power system load model.

[0123] Embodiment 1:

[0124] The present invention is implemented as follows based on the improved deep reinforcement learning method for identifying power load model parameters:

[0125] Step 1: Construct a complete integrated load model taking into account distribution network impedance and reactive power compensation.

[0126] The data in the examples are from Figure 7 In the IEEE 9-node system, load points 6 and 8 use constant impedance models, and the data of load node 5 is recorded. The faults are three-phase short circuit (3 cycles) in the middle of line 6-4, three-phase short circuit (4 cycles) in the middle of line 6-9, and three-phase short circuit (5 cycles) in the middle of line 8-9. The voltage, active and reactive power of the load bus of node 5 under the three faults are recorded. Table 1 shows the typical values ​​of the parameters of the comprehensive load model of the power system. Table 2 shows the search range of the parameters to be identified in the comprehensive load model of the power system.

[0127] Table 1 Typical values ​​of non-critical parameters

[0128]

[0129] Among them, X m is the magnetizing reactance; R r is the rotor resistance; X r is the rotor reactance; T j is the inertia time constant; A and B are the mechanical torque coefficients, is the power factor, X D / R D is the ratio of the distribution network reactance to the distribution network resistance, R S is the stator resistance.

[0130] Table 2 Parameters to be identified and search range

[0131] parameter <![CDATA[P MP ]]> <![CDATA[K L ]]> <![CDATA[X S ]]> <![CDATA[X D ]]> <![CDATA[I p ]]> <![CDATA[P p ]]> <![CDATA[I q ]]> <![CDATA[P q ]]> <![CDATA[Z p ]]> <![CDATA[Z q ]]> <![CDATA[X C ]]> scope 0.1-0.8 0.3-0.8 0.08-0.2 0-0.1 0-1 0-1 0-1 0-1 0-1 0-1 0-5

[0132] According to step one, the search range and dynamic equation of the parameter values ​​to be identified of the comprehensive load model of the power system are obtained, and in step two, the improved deep reinforcement learning algorithm is used to identify the time-varying parameters.

[0133] Step 2: Design a bidirectional gated recurrent unit deep deterministic policy gradient algorithm based on prioritized experience replay for time-varying parameter identification.

[0134] The training parameter settings of the bidirectional gated recurrent unit deep deterministic policy gradient algorithm based on priority experience replay are shown in Table 3. The execution network and the evaluation network are trained according to the set parameters in Table 3.

[0135] Table 3 Training parameter settings of the deep deterministic policy gradient algorithm for bidirectional gated recurrent units based on priority experience replay

[0136] Training parameters Set value or range Training parameters Set value or range <![CDATA[Maximum number of iterations N max > 500 Execution network structure 3-124-11 Initial weights of neural network [-0.1,0.1] Evaluation network structure 14-124-1 <![CDATA[Execute network learning rate α a > 0.01 Soft update parameter τ 0.005 <![CDATA[Evaluation network learning rate α c > 0.01 Gaussian noise mean ξ 0 Minimum batch size 64 Gaussian noise variance σ 0.1 Training batches 64 Prioritize experience replay pool capacity 20000 Discount Factor 0.99 Step length h 1 Prioritized experience replay parameter α 1 Importance Sampling β [0,1]

[0137] Table 4 shows the parameters of the comprehensive load model of the power system identified by the method of the present invention after the trained execution network and evaluation network, and E in the table is the fitting error of the dynamic curve. The execution network thus obtained combined with the comprehensive load model of the power system can predict future active power and reactive power.

[0138] Table 4 Parameter identification results of power system comprehensive load model

[0139]

[0140] Step 4: Use the identified lake and reservoir cyanobacteria growth model to make predictions.

[0141] Given the state input x(k) at time k, the execution network can output the optimal control strategy π ν (x(k)) and obtain the parameters to be identified θ(k) of the power system comprehensive load model. The power system comprehensive load model calculates the active and reactive power P consumed by the output prediction induction motor according to the state x(k) and the parameter θ(k) M (k), Q M (k) Active and reactive power consumed in the static part P ZIP (k), Q ZIP (k) Active and reactive power consumed on the distribution network branch P D (k), Q D (k), and the reactive power Q consumed by the compensation capacitor C(k). Under the conditions of line 6-4 fault, line 6-9 fault and line 8-9 fault, the deep deterministic policy gradient algorithm (DDPG), deep deterministic policy gradient algorithm based on bidirectional gated recurrent unit (DDPG-BiGRU), deep deterministic policy gradient algorithm based on priority experience replay (PER-DDPG) and deep deterministic policy gradient algorithm based on bidirectional gated empty recurrent unit based on priority experience replay (PER-DDPG-BiGRU, i.e., the power load parameter identification model of the present invention) are respectively used to predict the complete comprehensive load model taking into account the distribution network and reactive power compensation, and the obtained reward is used as the model evaluation index, as shown in Table 5. Curve comparison Figure 8 to Figure 10 shown.

[0142] Table 5 Reward values ​​of different models under different line faults

[0143] Line / Model DDPG DDPG-BiGRU PER-DDPG PER-DDPG-BiGRU 6-4 -4.9737 -4.0049 -5.3479 -4.1976 6-9 -3.6524 -3.6029 -3.7964 -3.4017 8-9 -1.7674 -1.3498 -3.1952 -1.1979

[0144] In Table 5, the horizontal axis represents four comparison models, and the vertical axis represents different line faults. The values ​​in the table are the reward values ​​of different models under different line faults. The higher the reward value, the better the model effect. It can be seen from the table that under line 6-4 fault, line 6-9 fault and line 8-9 fault, the reward value of the PER-DDPG-BiGRU model is the highest, which proves that the reinforcement learning model has higher performance, can learn better strategies, and identify more accurate load model parameters.

[0145] Figure 8 , Fig. 9 and Fig.10 The following are the active power fitting curves, reactive power fitting curves, reward comparison curves and dynamic error comparison curves of the four comparison models under line 6-4 fault, line 6-9 fault and line 8-9 fault. The experimental results show that Figure 8 In the fitting results of active power and reactive power of line 6-4 fault in A and 8B, the fitting curve of PER-DDPG-BiGRU model is more accurate than that of DDPG-BiGRU model. Figure 8 The reward value of the PER-DDPG-BiGRU model in C increases the fastest, and the DDPG-BiGRU model obtains the highest reward value. Figure 8 In D, the error value of the PER-DDPG-BiGRU model decreases the fastest, and the error of the DDPG-BiGRU model is the lowest; Fig. 9 Among the fitting results of active power and reactive power of line 6-9 fault in A and 9B, the fitting curve of PER-DDPG-BiGRU model is the most accurate. Fig. 9 The reward value of the PER-DDPG-BiGRU model in C increases the fastest and has the highest stable reward value. Fig. 9The error value of the PER-DDPG-BiGRU model in D decreases the fastest and achieves the lowest error; Fig.10 Among the fitting results of active power and reactive power of line 8-9 fault in A and 10B, the fitting curve of PER-DDPG-BiGRU model is the most accurate. Fig.10 The reward value of the PER-DDPG-BiGRU model in C increases the fastest and has the highest reward value. Fig.10 The error of the PER-DDPG-BiGRU model in D decreases the fastest and has the lowest stable error value. It can be seen that the PER-DDPG-BiGRU model constructed by the method of the present invention shows strong robustness and generalization ability under different line faults, and exhibits the best performance. The comparative experimental curve results in terms of reward value and dynamic error prove that the PER-DDPG-BiGRU model combines the advanced deep learning framework with the reinforcement learning algorithm, and adopts a bidirectional gated recurrent unit structure to more effectively process time series data, capture fault characteristics and respond quickly, verifying the optimization ability of the model proposed by the present invention.

[0146] Based on this, the method of the present invention realizes the power load parameter identification model by adopting a deep reinforcement learning model improved by the deep deterministic policy gradient algorithm of the bidirectional gated recurrent unit based on priority experience replay. It can simultaneously capture the nonlinear evolution characteristics of the active power and reactive power of the power system. The model has strong adaptability and stability, thereby providing more accurate prediction results and verifying the effectiveness of the proposed model.

[0147] It should be noted that, except for the technical features described in the specification, all are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the scope of protection of the present invention.

Claims

1. A method for identifying power load model parameters based on improved deep reinforcement learning, characterized in that: include: Step 1: Establish a comprehensive load model of the power system taking into account the impedance of the distribution network and reactive power compensation, and determine the input state vector, output data vector and parameters to be identified in the comprehensive load model; The active power P(k), reactive power Q(k) and voltage data U(k) at time k are collected from the measurement point of the research circuit to form the input state vector x(k). Let the parameter vector to be identified at time k of the comprehensive load model be θ(k). The output data vector s(k) calculated by the comprehensive load model at time k includes: the active power P consumed by the induction motor M (k) and reactive power Q M (k), active power consumed by the static load part P ZIP (k) and reactive power Q ZIP (k), the active power consumed on the distribution network branch P D (k) and reactive power Q D (k), and the reactive power Q consumed by the compensation capacitor C (k); The input and output of the comprehensive load model are expressed as: s(k)=SLM(x(k),θ(k)); SLM represents the comprehensive load model; Step 2: Use the improved deep reinforcement learning model structure to establish a power load parameter identification model; The improved deep reinforcement learning model structure is implemented by a deep deterministic policy gradient algorithm based on a bidirectional gated recurrent unit with priority experience replay. When constructing the execution network and the evaluation network, a bidirectional gated recurrent unit is used to construct the main execution network, the target execution network, the main evaluation network and the target evaluation network; the experience replay pool is improved to a priority experience replay pool, and the priority of the four-tuple samples in the pool is determined based on the amplitude of the temporal difference error of the sample. The larger the amplitude of the temporal difference error, the greater the priority of the sample; the temporal difference error of the sample is: the execution network calculates the value function of the current state-strategy pair, and updates the parameters of the evaluation network by minimizing the difference between the predicted value function and the Bellman optimal value by the gradient descent method, and the difference is the temporal difference error; The input of the power load parameter identification model is the state vector x(k), and the execution network calculates the optimal strategy π representing the change amount of the parameter to be identified for the input x(k). ν (x(k)), combined with the previous moment parameter θ(k-1), to get the updated parameter θ(k); Step 3: training the power load parameter identification model, and using the trained model to perform parameter identification; First, set the maximum number of iterations N max The maximum time step k in each iteration max ; Then, the experience samples are obtained from the priority experience revisit pool and input into the improved deep reinforcement learning model. For the experience samples [x(k), θ(k), r(k), x(k+1)], the main execution network outputs the strategy π for x(k) ν (x(k)), the main evaluation network calculates the value function Q for the state-strategy pair (x(k), θ(k)) μ (x(k),θ(k)), the target execution network outputs strategy π for x(k+1) ν (x(k+1)), the target evaluation network calculates the value function Q for the state-strategy pair (x(k+1),θ(k+1)) μ' (x(k+1),θ(k+1)); iteratively update the weight parameters of the execution network and the evaluation network to minimize the set model evaluation index value until the number of iterations reaches the maximum number, end the iteration, and obtain the current trained power load parameter identification model; The trained power load parameter identification model is used for parameter identification.

2. The method according to claim 1, characterized in that In the step 1, the parameters of the power system comprehensive load model to be identified include: stator reactance X S , distribution network reactance X D , Active constant impedance coefficient Z P , Active constant current coefficient I P , Active constant power coefficient P p , Reactive constant impedance coefficient Z q , Reactive constant current coefficient I q , Reactive constant power coefficient P q 、Capacitor reactance X C , Motor ratio P MP and motor load factor K L .

3. The method according to claim 1, characterized in that In the step 2, the power load parameter identification model generates experience and stores it in the priority experience playback pool in the following manner: (1) Input x(k) into the main execution network to obtain the strategy π ν (x(k)), Gaussian noise N(ξ,σ) is introduced when updating the parameters as follows: θ(k)=θ(k-1)+π ν (x(k))+N(ξ,σ) Among them, Gaussian noise N(ξ,σ) satisfies Gaussian distribution with mean ξ and variance σ; The updated parameter θ(k) and the input state x(k) are input into the comprehensive load model of the power system, the output s(k) is calculated, and s(k) is compared with the initial fixed environment to obtain the reward r(k), as follows: r(k)=-[[x1(k)-s1(k)] 2 +[x2(k)-s2(k)] 2 ] =-[[P(k)-(P M (k)+P ZIP (k)+P D (k))] 2 +[Q(k)-(Q M (k)+Q ZIP (k)+Q D (k)+Q C (k))] 2 ] Among them, x1(k), x2(k), s1(k) and s2(k) are all intermediate parameters, s(k)=s1(k)+s2(k); The agent interacts with the environment to obtain the next state x(k+1), thereby obtaining a four-tuple [x(k), θ(k), r(k), x(k+1)], which is stored in the priority experience replay pool as experience; the agent continuously interacts with the environment, generates experience and stores it in the priority experience replay pool. When the amount of experience accumulates to the minimum training batch, the execution network and the evaluation network begin to be trained; (2) Calculate the priority of each experience sample in the priority experience playback pool, the priority of experience sample i is p(i) = |δ(i)| + ε, where δ(i) is the time difference error of experience sample i, and ε is a small positive number; Further determine the sampling probability of empirical sample i Among them, α is the priority parameter, M is the total number of experience samples in the priority experience replay pool; Since sample priority leads to non-uniform probability sampling and introduces bias, the bias is compensated by importance sampling weights. The importance sampling weight of empirical sample i is The weight β starts from a value between 0 and 1 and gradually increases to 1. When β = 1, it is considered that the deviation introduced by non-uniform probability sampling is fully compensated; the importance sampling weights of the empirical samples are normalized; During training, the sampling probability of an experience sample in the priority experience replay pool is ultimately determined by the product of the importance sampling weight of the sample and the priority.

4. The method according to claim 3, characterized in that In the step 2, the timing difference error is calculated as follows: For the empirical sample [x(k),θ(k),r(k),x(k+1)], the temporal difference error δ(k) of the sample is calculated as follows: δ(k)=y(k+1)-Q μ (x(k),π v (k)) =r(k)+γQ μ' (x(k+1),π ν' (k+1))-Q μ (x(k),π v (k)) Among them, y(k+1) represents the Bellman optimal value and γ is the discount factor.

5. The method according to claim 1, 2 or 3, characterized in that: In the step 2, the main execution network and the target execution network in the execution network both include: 3 input neurons, corresponding to the three elements of the input state vector x(k); a There are 11 hidden layer neurons, each of which is implemented by a bidirectional gated recurrent unit; 11 output neurons, corresponding to the changes of 11 parameters to be identified; the activation function of the hidden layer uses a unipolar sigmoid function, and the activation function of the output layer uses a nonlinear tanh function; The main evaluation network and the target evaluation network in the evaluation network both contain: 14 input neurons, corresponding to the 3 elements of the input state vector x(k) and the 11 parameters to be identified of θ(k); c hidden layer neurons, each hidden layer neuron is implemented using a bidirectional gated recurrent unit; 1 output neuron, corresponding to the value function of the current state-strategy pair; the hidden layer activation function uses a unipolar sigmoid function, and the output layer activation function uses a nonlinear tanh function.

6. The method according to claim 1 or 3, characterized in that: In the step 3, when the power load parameter identification model is trained, the evaluation index of the power load parameter identification model is defined as E, as follows: Where N is the number of training samples in the current batch, and the intermediate parameters are x1(k) = P(k), x2(k) = Q(k), s1(k) = P M (k)+P ZIP (k)+P D (k), s2(k)=Q M (k)+Q ZIP (k)+Q D (k)+Q C (k); By training the execution network and the evaluation network, the index E is minimized, and then the trained target execution network is used to obtain the parameters to be identified.

7. The method according to claim 5, characterized in that In step 3, the model training process includes: (1) By comparing the value Q μ (x(k),θ(k)) performs policy gradient ascent calculation to update the parameters of the main execution network as follows: Assume that the weight parameter of the main execution network at time k is ν(k), the weight parameter of the main evaluation network at time k is μ(k), N is the number of training samples in the current batch, ▽ ν J(ν(k)) is the gradient of the weight parameters of the main execution network, ▽ ν π ν (x(k)) is the gradient of the output strategy weight parameter of the main execution network, ▽ μ Q μ (x(k),π v (x(k))) is the gradient of the weight parameter of the main evaluation network; Update the next moment main execution network weight parameter ν(k+1)=ν(k)+α a ▽ ν J(ν(k));α a is the learning rate of the main execution network; Update the weight parameter ν' of the target execution network through soft update; (2) The loss function of the main evaluation network is the difference between the output of the main evaluation network and the sum of the output value of the target evaluation network at the next moment and the discounted reward; the predicted value function Q is minimized by the gradient descent method μ The difference between (x(k),θ(k)) and Bellman's optimal value is used to update the weight parameters of the main evaluation network as follows: Among them, μ J Q (μ(k)) is the main evaluation network J Q The gradient of the weight parameter; δ(k) is the time difference error of the empirical sample, ▽ μ δ(k) is the gradient of the temporal difference error; The gradient of the loss function with respect to the parameters of the main evaluation network is calculated by the back propagation algorithm, and the parameters of the main evaluation network are updated using the gradient descent method to minimize the loss function, as follows: μ(k+1)=μ(k)-α c ▽ μ J Q (μ(k)) Among them, μ(k+1) is the weight parameter of the main evaluation network at the next moment of update, α c is the learning rate of the main evaluation network; The weight parameter μ' of the target evaluation network is updated by soft updating.

Images