Photovoltaic grid-connected inverter control method based on behavior cloning

Abstract

The invention provides a photovoltaic grid-connected inverter control method based on behavioral cloning. A pre-training process of the photovoltaic grid-connected inverter control method comprises the following steps: presetting a self-adaptive updating strategy of a controller parameter; initializing M strategy neural networks with the same structure; an inverter is started, and controller parameters of a PI controller are dynamically selected according to a self-adaptive updating strategy; extracting input and output data of a plurality of groups of PI controllers in real time to form teaching data, storing the teaching data into a teaching database, and updating weight parameters of each strategy neural network through a gradient descent method; and inputting input data in each group of data in the teaching database into the updated strategy neural network to obtain calculation results, comparing each calculation result with corresponding output data in the teaching database, and if a difference value between the calculation results and the corresponding output data is smaller than a preset threshold value, putting the group of data into an online training process. According to the method, a data driving method is adopted, clone pre-training is carried out firstly, then a DDPG stage of deep reinforcement learning is carried out, and a PI controller is replaced with a learning model.

Description

Technical Field

[0001] The present invention relates to the field of data prediction, and in particular to a photovoltaic grid-connected inverter control method based on behavior cloning. Background Art

[0002] Photovoltaic grid-connected power generation has become one of the most promising new energy utilization methods in distributed power generation systems due to its advantages of being clean, environmentally friendly, rich in resources and easy to develop. Photovoltaic grid-connected inverters are the core equipment that connects photovoltaic power generation systems to the power grid. They are responsible for converting the direct current generated by the photovoltaic array into alternating current used by the power grid and ensuring the stable output of electric energy. However, photovoltaic power generation systems are susceptible to fluctuations in external factors such as light intensity and temperature; at the same time, the internal components of the inverter (such as switching tubes and filters) may age or become unbalanced in performance during long-term use, resulting in an increase in harmonic components in the inverter output current, thereby reducing the quality of the current waveform and affecting system stability.

[0003] The photovoltaic array is connected between the positive and negative poles of the grid-connected inverter as the input DC power supply. Figure 1 As shown, the electrolytic capacitor C dc As input filter capacitor, it is connected to both ends of the DC bus of the grid-connected inverter; boost inductor L c Located between the photovoltaic array and the inverter, the current flows through the switch tube T c and DC bus voltage U dc The inverter is connected to the grid-connected inverter. Each phase bridge arm of the inverter is connected to the DC bus in series with two switch tubes, and diodes are connected in anti-parallel at both ends of the switch tube to form a bridge circuit structure. The midpoint of each phase leads to the three-phase output A, B, and C, which are connected to the three-phase grid through the LCL filter composed of the filter inductor L and the filter capacitor C to ensure the smoothness of the output current and the quality of the power. Figure 2 As shown in the figure, the equivalent circuit of the LCL filter includes the inverter side filter inductor L and filter capacitor C, which can effectively filter out the high-frequency harmonics in the inverter output current and ensure the smoothness of the current waveform. i With grid voltage u g The interaction directly affects the dynamic performance of the system and is an important theoretical basis for designing control strategies.

[0004] like Figure 3As shown in the figure, the current control method of photovoltaic grid-connected inverter is mainly based on PI controller design, which can achieve certain control effects under steady-state conditions. However, under nonlinear, time-varying and high-uncertainty conditions, the dynamic response speed and robustness of the traditional PI controller are difficult to meet the requirements, resulting in obvious deficiencies in harmonic suppression, output current waveform quality and system stability. Therefore, how to suppress harmonics, optimize current waveform quality and improve system robustness while ensuring the fast response of the inverter has become an important technical challenge in the current field of photovoltaic grid-connected inverter control. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a photovoltaic grid-connected inverter control method based on behavior cloning.

[0006] The technical solution of the present invention is: a photovoltaic grid-connected inverter control method based on behavior cloning, the method includes a data pre-training process and an online training process, wherein the pre-training process includes the steps of:

[0007] S11, presetting an adaptive update strategy for controller parameters;

[0008] S12, initializing M strategy neural networks with the same structure;

[0009] S13, starting the inverter, and dynamically selecting controller parameters of the PI controller according to the adaptive update strategy;

[0010] S14, extracting input and output data of multiple groups of PI controllers in real time to form teaching data and storing them in a teaching database, using the teaching data as input and output of M strategy neural networks, and updating the weight parameters of each strategy neural network by gradient descent method;

[0011] S15. Input the input data in each group of data in the teaching database into the updated strategy neural network to obtain the calculation results, and compare each calculation result with the corresponding output data in the teaching database. If the difference between the two is less than a preset threshold, the group of data is put into the online training process.

[0012] Preferably, the teaching database includes multiple groups of teaching data, each group of teaching data is one-to-one corresponding input data and output data, wherein each group of data includes one-to-one corresponding current error values ​​and voltage output values ​​in the inner current loop, and one-to-one corresponding voltage error values ​​and current output values ​​in the outer voltage loop.

[0013] Preferably, a priority experience replay strategy is adopted in step S14.

[0014] Preferably, in step S15, the evaluation relationship between the calculation result of the strategy neural network and the corresponding output data in the teaching database is:

[0015]

[0016] Among them, a i,RL For state s i The output of the next policy neural network, U PI It is the output corresponding to the teaching database of the PI controller.

[0017] Preferably, the online training process includes the steps of:

[0018] S21. Use the DDPG algorithm to further train the M policy neural networks that have completed behavior cloning online, and combine the priority experience replay mechanism to optimize the behavior based on the priority of the Q value;

[0019] S22, connect M strategy neural networks in parallel and calculate the average output

[0020] S23, compare output average Output U corresponding to the teaching database of the PI controller PI , if the difference between the two is greater than the dynamic threshold, then U PI As the control signal used by the inverter; on the contrary, if the difference between the two is small, the average value of the output of the strategy neural network is used. Used as a control signal for the inverter.

[0021] Preferably, an additional playback buffer R is provided D , which is used to store the initial expert data. In the DDPG algorithm, each update is from R D Randomly select N D An example is taken and used as off-policy replay data to participate in the current training process.

[0022] Preferably, the actuator network is updated by a policy gradient method in the DDPG algorithm.

[0023] Preferably, the loss function is updated in the DDPG algorithm using a temporal difference error method, and the evaluator network is updated using the temporal difference method.

[0024] Preferably, in the DDPG algorithm, the target neural network adopts a soft update strategy.

[0025] Better, the reward function r in the DDPG stage t for:

[0026] r t =-(ω1·|V ref -V out |+ω2·|I d,ref -I d |+ω3·|Iq,ref -I q |+ω4·|ΔU|)

[0027] Among them, V ref and V out are the reference voltage and the actual output voltage respectively; I d,ref and I d are the reference value and actual value of the d-axis current respectively; I q,ref and I q are the reference value and actual value of the q-axis current respectively; ΔU represents the change amplitude of the control quantity to prevent instability caused by excessive control action; ω1, ω2, ω3, ω4 are weight coefficients, which control the weights of each reward respectively.

[0028] The above technical solution has the following advantages or beneficial effects: In the above method, the output current of the inverter accurately tracks the set grid-connected current reference value through the coordinated adjustment of the current inner loop and the voltage outer loop, thereby significantly improving the dynamic response performance and robustness of the system. The inner loop detects the deviation of the inverter output current from the reference value in real time, and after the learning model generated by the above control method, it is compared with the triangular carrier signal to generate the switch control signal of the inverter, ensuring that the output current can stably follow the reference value. In addition, combined with digital signal processing technology and priority experience playback mechanism, through the strategy of combining offline pre-training with online optimization, the output current harmonics of the photovoltaic grid-connected inverter are effectively suppressed, the dynamic response performance and control robustness of the system are improved, thereby significantly enhancing the overall stability of the photovoltaic grid-connected power generation system. The present invention mainly adopts a data-driven method, first behavioral cloning pre-training, and then the DDPG stage of deep reinforcement learning, so as to use the learning model to replace the PI controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0030] Figure 1 The prior art of the present invention is shown in FIG. Figure 1 ;

[0031] Figure 2 The prior art of the present invention is shown in FIG. Figure 2 ;

[0032] Figure 3 The prior art of the present invention is shown in FIG. Figure 3 ;

[0033] Figure 4 The flow chart of a photovoltaic grid-connected inverter control method based on behavior cloning of the present invention is as follows Figure 1 ;

[0034] Figure 5 The flow chart of a photovoltaic grid-connected inverter control method based on behavior cloning of the present invention is as follows Figure 2 ;

[0035] Figure 6 The figure is an overall schematic diagram of a photovoltaic grid-connected inverter control method based on behavior cloning according to the present invention. DETAILED DESCRIPTION

[0036] A photovoltaic grid-connected inverter control method based on behavior cloning of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] In the description of the present invention, it should be understood that the orientations or positional relationships indicated by the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must and has a specific orientation, is constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. At the same time, the terms "first", "second", etc. are only used to distinguish the names of the components, and there is no primary and secondary relationship, and therefore cannot be understood as a limitation on the present invention.

[0038] A photovoltaic grid-connected inverter control method based on behavior cloning includes a data pre-training process and an online training process. Figure 4 As shown, the pre-training process includes the steps of:

[0039] S11, presetting an adaptive update strategy for controller parameters;

[0040] S12, initializing M strategy neural networks with the same structure;

[0041] S13, starting the inverter, and dynamically selecting controller parameters of the PI controller according to the adaptive update strategy to ensure that the controller is in the best response state under different working conditions;

[0042] S14, extracting input and output data of multiple groups of PI controllers in real time to form teaching data and storing them in a teaching database, using the teaching data as the input and output of M strategy neural networks, and updating the weight parameters of each strategy neural network by gradient descent method after entering the strategy neural network; wherein the input and output data of the two PI controllers refer to the current error value and the corresponding voltage output, or the voltage error value and the corresponding current output obtained in real time, respectively, the current error value refers to the error between the reference current preset in the inner current loop and the actual measured current, and the voltage error value refers to the error between the reference voltage preset in the outer voltage loop and the actual measured voltage;

[0043] S15. Input the input data in each group of data in the teaching database into the updated strategy neural network to obtain the calculation result after the neural network is obtained, and compare the calculation result with the corresponding output data in the teaching database. If the difference between the calculation result and the output data of all strategy neural networks is less than a preset threshold, then the group of data enters the online training stage.

[0044] Specifically, in the data pre-training process, an adaptive update strategy for controller parameters is first preset, wherein the adaptive update strategy is that in the dual closed-loop PI control system of the photovoltaic grid-connected inverter, different controller parameters correspond to different light intensities, ambient temperatures and load conditions, and the controller parameters include proportional coefficients and integral time. Afterwards, M strategy neural networks with the same structure are initialized for subsequent pre-training processes. At this time, after starting the inverter, the controller parameters of the PI controller are dynamically selected according to the adaptive update strategy to ensure that the controller is in the best response state under different working conditions. Afterwards, the behavior cloning process is entered, and multiple groups of input and output data are extracted in real time to form teaching data and stored in the teaching database. The teaching database is used as the input and output of M strategy neural networks, and the weight parameters of each strategy neural network are updated by the gradient descent method, that is, the pre-training process of the strategy neural network is completed. The input variables of the teaching database are recalculated using the pre-trained neural network, and the calculation results are compared with the corresponding output data in the teaching database to screen and update the teaching database for the first time. In the above process, the behavior cloning process pre-trains the policy neural network through the teaching data generated by offline learning PI controller to ensure the rationality of the initial strategy.

[0045] It is worth pointing out that the teaching database includes multiple groups of teaching data, each group of teaching data is a one-to-one corresponding input data and output data, wherein each group of data includes a one-to-one corresponding current error value (input data) and voltage output value (output data) in the inner current loop, and a one-to-one corresponding voltage error value (input data) and current reference value (output data) in the outer voltage loop. The current error value here is the difference between the real-time measured current value and the reference current value, and the voltage error value is the difference between the real-time measured voltage value and the reference voltage value. The preset reference voltage refers to: usually set by the output voltage of the battery or photovoltaic panel or the grid voltage, which is set to 600V here; the preset d-axis reference current is usually calculated based on the outer loop voltage control, and the q-axis current preset value is 0, usually to maximize the active power output and minimize the reactive power output. The basic principle behind this involves power control and synchronization with the grid.

[0046] In addition, the proportional coefficient K in the controller parameters pIt is the proportional gain of the controller, which determines the proportional relationship between the controller output and the control error, that is, the output control quantity is a proportional multiple of the error, and the proportional term generates a control signal by amplifying the error. p The larger the value, the stronger the controller's response to the error, the larger the output control signal, and the faster the system's response. Assuming the control error is e(t) (i.e., the difference between the target value and the actual value), the output of the proportional controller is:

[0047] u(t)=K p e(t)

[0048] Among them, u(t) is the output signal of the controller and e(t) is the current error.

[0049] Integration time T i Determines the strength of the integral action. Specifically, it is related to the integral gain K i (Right now The integral term adjusts the system based on past errors in order to eliminate steady-state errors.

[0050] The integral term adjusts based on the accumulated amount of error, thus eliminating the steady-state error caused by insufficient proportional control. The integral term allows the controller to continue to react to the error until the error is zero. The integral term output is:

[0051]

[0052] where e(τ) is the time function of the error and u(t) is the output of the controller.

[0053] Furthermore, in step S14, the behavior cloning loss function is:

[0054]

[0055] That is, the behavior cloning loss is defined as the mean square error, which is used to maximize the strategy neural network to imitate the expert behavior. D is the number of samples selected in the teaching database, π(s i |θ π ) is the policy neural network in state s i The predicted output data under i is the output variable of the teaching database, θ π is the policy neural network parameter. It is worth noting that the state s i Refers to different controller parameters.

[0056] Among them, the policy neural network parameter θ π The update formula is:

[0057]

[0058] Among them, λ1 and λ2 control the weights of reinforcement learning loss and behavior cloning loss, α is the learning rate, and J is the objective function of reinforcement learning.

[0059] Furthermore, in step S14, a priority experience playback strategy is adopted, that is, in the experience sampling, based on the priority δ of each teaching data i The proportional priority sampling probability p i The calculation is as follows:

[0060]

[0061] p i In order to correct the deviation caused by non-uniform sampling, ∈ is the threshold and the importance sampling weight ω is introduced i :

[0062]

[0063] Where N is the total number of samples and β controls the degree of sampling bias correction.

[0064] Furthermore, the priority of the preset teaching data δ i for:

[0065] δ i =|Q(s i ,a i )-Q(s i ,π(s i ))|

[0066] Where Q is the neural network model, s i is the state, a i is the output variable of the teaching database, π(s i ) is the policy neural network in state s i The output is only when δ i >∈ (∈ is the threshold), the teaching sample is included in the loss calculation of behavior cloning. This process can also be called the Q-value filtering mechanism process.

[0067] Furthermore, in step S15, the difference between the strategy neural network output and the teaching data output variable is less than the preset threshold, that is, the judgment relationship is:

[0068]

[0069] Among them, a i,RL For state s i The output of the next policy neural network, U PI It is the output corresponding to the teaching database of the PI controller.

[0070] Further, if Figure 5 and Figure 6 As shown, the online training process includes the following steps:

[0071] S21. Use the DDPG algorithm (Deep Deterministic Policy Gradient) to further train the M policy neural networks that have completed behavior cloning online, and combine it with the priority experience replay mechanism to optimize the behavior based on the priority of the Q value.

[0072] S22, connect M strategy neural networks in parallel and calculate the average output

[0073] S23, compare output average Output U corresponding to the teaching database of the PI controller PI , if the difference between the two is greater than the dynamic threshold, then U PI As the control signal used by the inverter; on the contrary, if the difference between the two is small, the average value of the output of the strategy neural network is used. As the control signal used by the inverter. It is worth pointing out that the control signal here corresponds to the output signal in the teaching data, that is, through the above comparison process, the teaching database is updated again.

[0074] Specifically, during the online training process, the DDPG algorithm is used to further train the M policy neural networks that have completed behavior cloning online, and then the M policy neural networks are connected in parallel, that is, the average value of the output of all policy neural networks is calculated. Then, the average Output U corresponding to the teaching database of the PI controller PI The difference is compared with the dynamic threshold to select the control signal, that is, the teaching database is updated again. This data update improves the control accuracy and stability, avoids system instability caused by inconsistent control strategies; enhances the adaptability of the model, and can self-adjust according to dynamic environmental changes; reduces overfitting and improves generalization ability, making the model more suitable for different practical application scenarios; strengthens the two-way feedback mechanism between strategy optimization and teaching data to improve training efficiency; improves data utilization efficiency and reduces the impact of redundant data on the training process; enhances the robustness and performance of the strategy, so that the control system can still operate stably under complex conditions.

[0075] The Q value mentioned above usually refers to the value-action function, that is, for a given state and action, the Q value represents the expected cumulative return that can be obtained by taking the action in the state. In DDPG, the calculation of the Q value usually relies on the Critic Network, which is responsible for evaluating the action quality of the current strategy.

[0076] Furthermore, an additional playback buffer R is provided D , which is used to store the initial expert data. This buffer has the same format as the standard replay buffer R in the DDPG algorithm. In the DDPG algorithm, each time a small batch is updated, D Randomly select N D This method can improve the stability of the initial stage and accelerate convergence; reduce policy instability and balance exploration with existing knowledge; improve data efficiency, accelerate the learning process, and reduce invalid exploration; enhance the generalization ability of the strategy and improve the ability to adapt to new environments; speed up policy optimization and provide more refined policy adjustments; reduce unnecessary exploration and improve learning efficiency.

[0077] Furthermore, in step S21, the DDPG algorithm updates the actuator network through the policy gradient method, and the policy gradient formula is:

[0078]

[0079] Update the evaluator network through the temporal difference error method, that is, the loss function is updated:

[0080] y i =r i +Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ )

[0081]

[0082] Where N and N D is the sum of all sample numbers, μ(s i |θ π ) is the output of the actuator network, Q(s i ,a i |θ Q ) is the output of the evaluator network, Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ) This is the target Q network (also a neural network), θ π and θQ are the parameters of the policy network and the evaluator network, y i is the predicted value of the entire model, L is used to update the loss function parameters, s i is the state, a i is the action. J represents the objective function or expected reward function (usually used for optimization goals in reinforcement learning). In the DDPG algorithm, J usually represents the performance or reward target of the agent, and the goal is to maximize this function to optimize the strategy. a represents action. In reinforcement learning, a is the action selected by the agent in a given state, usually represented by the policy function μ(s|θ π ) is generated. In this formula, action a is selected by strategy μ(si) (the action when given state si). N represents the number of samples extracted from the standard playback buffer R, which is the experience data obtained by the agent through exploration during training. D Represents the playback buffer R from the expert data D The number of samples extracted from the training data is the original teaching data, which is the demonstration data used to guide and strengthen the learning of the intelligent agent.

[0083] In the DDPG algorithm, the target neural network adopts a soft update strategy, and its update formula is:

[0084] θ Q′ ←τθ Q +(1-τ)θ Q′

[0085] θ μ′ ←τθ μ +(1-τ)θ μ '

[0086] Among them, θ Q′ is the target evaluator network parameter, θ μ′ is the target actuator network parameter, τ is the soft update coefficient, usually 0.001. Use soft update to update the target neural network parameters so that they gradually approach the main network parameters.

[0087] And, the reward function r in the DDPG stage t The design is as follows to optimize the steady-state performance, dynamic response and stability of the photovoltaic grid-connected inverter:

[0088] r t =-(ω1·|V ref -V out |+ω2·|I d,ref -I d |+ω3·|I q,ref -I q |+ω4·|ΔU|)

[0089] Among them, V ref and V out are the reference voltage and the actual output voltage respectively; I d,ref and I d are the reference value and actual value of the d-axis current respectively; I q,ref and I q are the reference value and actual value of the q-axis current respectively; ΔU represents the change amplitude of the control quantity to prevent instability caused by excessive control action; ω1, ω2, ω3, ω4 are weight coefficients, which control the weights of each reward respectively. The setting of the reward function helps to improve the accuracy of voltage and current tracking, limit the change of DC bus voltage, and ensure the stability of the system. In addition, through the weighted penalty mechanism, the optimization priority of different objectives can be flexibly adjusted according to actual needs.

[0090] Furthermore, in step S22, the average value The calculation method is:

[0091]

[0092] Furthermore, when outputting the average Output U corresponding to the teaching database of the PI controller PI The comparison method is:

[0093]

[0094] Where δ is the dynamic threshold.

[0095] Furthermore, the update formula for the dynamic threshold for updating the difference is:

[0096]

[0097] Where d is the number of days into the pre-commissioning phase. d is a time variable, indicating the number of days into the pre-commissioning phase. That is, how many days the system has been running since a certain moment (the date of entering the pre-commissioning phase). This period of time usually marks the transition of a system from a formal experiment or development phase to a phase closer to actual use. The use of this variable can help dynamically adjust the threshold δ to automatically adjust the sensitivity and optimization process of the control strategy according to the passage of time and changes in the system state. The δ in the formula is a dynamic threshold that changes with the passage of time d. The purpose is to gradually adjust the sensitivity and optimization objectives of the control strategy in the pre-commissioning phase according to the time the system is running. In the application, as d (the number of days into the pre-commissioning phase) increases, the exponential decay term in the formula will gradually decrease. This means that over time, the threshold δ will gradually approach 0.5, but always remain at a level slightly above 0.5.

[0098] In the above method, the behavior cloning process pre-trains the policy neural network through the teaching data generated by the offline learning PI controller to ensure the rationality of the initial policy. Then the deep deterministic policy gradient (DDPG) reinforcement learning algorithm is used to interact with the environment in real time, and the policy neural network is optimized in combination with the priority experience replay mechanism to improve the dynamic response performance and stability of the system.

[0099] Furthermore, the above training process also includes the following steps: using the deep deterministic policy gradient algorithm (DDPG) to continuously train M pre-trained policy networks online, updating the evaluator network parameters by minimizing the temporal difference error (TD error) of the evaluator network, and updating the actuator network parameters by maximizing the Q value output by the evaluator network.

[0100] In the above method, the output current of the inverter accurately tracks the set grid-connected current reference value through the coordinated regulation of the current inner loop and the voltage outer loop, thereby significantly improving the dynamic response performance and robustness of the system. The inner loop detects the deviation of the inverter output current from the reference value in real time. After the learning model generated by the above control method, it is compared with the triangular carrier signal to generate the switch control signal of the inverter, ensuring that the output current can stably follow the reference value. In addition, combined with digital signal processing technology and priority experience playback mechanism, through the strategy of combining offline pre-training with online optimization, the output current harmonics of the photovoltaic grid-connected inverter are effectively suppressed, the dynamic response performance and control robustness of the system are improved, thereby significantly enhancing the overall stability of the photovoltaic grid-connected power generation system. The present invention mainly adopts a data-driven method, first behavioral cloning pre-training, and then the DDPG stage of deep reinforcement learning, so as to replace the PI controller with a learning model.

[0101] For those skilled in the art, various changes and modifications will undoubtedly be obvious after reading the above description. Therefore, the attached claims should be regarded as covering all changes and modifications of the true intent and scope of the present invention. Any and all equivalent ranges and contents within the scope of the claims should be considered to still be within the intent and scope of the present invention.

Claims

1. A photovoltaic grid-connected inverter control method based on behavior cloning, characterized in that: The method includes a data pre-training process and an online training process, wherein the pre-training process includes the steps of: S11, presetting an adaptive update strategy for controller parameters; S12, initializing M strategy neural networks with the same structure; S13, starting the inverter, and dynamically selecting controller parameters of the PI controller according to the adaptive update strategy; S14, extracting multiple groups of PI controller input and output data in real time to form teaching data and storing them in a teaching database, using the teaching data as the input and output of M strategy neural networks, and after entering the strategy neural network, updating the weight parameters of each strategy neural network by a gradient descent method; S15. Input the input data in each group of data in the teaching database into the updated strategy neural network to obtain new calculation results, and compare each calculation result with the corresponding output data in the teaching database. If the difference between the two is less than a preset threshold, the group of data is put into the online training process.

2. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 1, characterized in that: The teaching database includes multiple groups of teaching data, each group of teaching data is one-to-one corresponding input data and output data, wherein each group of data includes one-to-one corresponding current error values ​​and voltage output values ​​in the inner current loop, and one-to-one corresponding voltage error values ​​and current output values ​​in the outer voltage loop.

3. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 1 is characterized in that: In step S14, a priority experience replay strategy and a Q filtering mechanism are adopted.

4. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 3 is characterized in that: In step S15, the evaluation relationship between the calculation result of the strategy neural network and the corresponding output data in the teaching database is: Among them, a i,RL For state s i The output of the next policy neural network, U PI It is the output corresponding to the teaching database of the PI controller.

5. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 2, characterized in that: The online training process includes the following steps: S21. Use the DDPG algorithm to further train the M policy neural networks that have completed behavior cloning online, and combine the priority experience replay mechanism to optimize the behavior based on the priority of the Q value; S22, connect M strategy neural networks in parallel and calculate the average output ; S23, compare output average value Output U corresponding to the teaching database of the PI controller PI , if the difference between the two is greater than the dynamic threshold, then U PI As the control signal used by the inverter; on the contrary, if the difference between the two is small, the average value of the output of the strategy neural network is used. Used as a control signal for the inverter.

6. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 2 is characterized in that: Provides an additional playback buffer R D , which is used to store the initial expert data. In the DDPG algorithm, each update is from R D Randomly select N D An example is taken and used as off-policy replay data to participate in the current training process.

7. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 6 is characterized in that: In the DDPG algorithm, the actuator network is updated through the policy gradient method.

8. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 7 is characterized in that: In the DDPG algorithm, the evaluator network is updated by the temporal difference method.

9. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 7, characterized in that: In the DDPG algorithm, the target neural network adopts a soft update strategy.

10. The photovoltaic grid-connected inverter control method based on behavior cloning according to claim 6, characterized in that: The reward function r in the DDPG stage t for: r t =-(ω1·|V ref -V out |+ω2·|I d,ref -I d |+ω3·|I q,ref -I q |+ω4·|ΔU|) Among them, V ref and V out are the reference voltage and the actual output voltage respectively; I d,ref and I d are the reference value and actual value of the d-axis current respectively; I q,ref and I q are the reference value and actual value of q-axis current respectively; ΔU represents the change amplitude of the control amount, which is used to prevent instability caused by excessive control actions; ω1, ω2, ω3, and ω4 are weight coefficients, which control the weights of each reward respectively.

Images