Adaptive controller tuning method based on reinforcement learning

Abstract

The invention discloses an adaptive controller tuning method based on reinforcement learning. The method comprises the following steps: S1, constructing a photovoltaic control task environment; s2, inputting the state sequence into a long short-term memory network, and outputting an embedded state vector; s3, constructing a hierarchical policy network comprising a basic policy module and a fast adaptation module; s4, training the hierarchical strategy network, inputting the embedded state vector into the hierarchical strategy network to generate a controller parameter action, interacting with a photovoltaic control task environment by using a multi-layer feedforward neural network to obtain response and reward feedback, and executing target value calculation and strategy optimization; s5, obtaining a general strategy initial parameter by using a meta-learning optimization method; and S6, deploying the optimized hierarchical strategy network to a target photovoltaic control task. The method is suitable for photovoltaic control and other multi-working-condition dynamic environments, and has the advantages of being high in strategy migration capacity, high in robustness and excellent in self-adaptive performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of photovoltaic power generation control and intelligent controller optimization, and particularly relates to an adaptive controller tuning method based on reinforcement learning. BACKGROUND

[0002] With the rapid development of new energy technology, photovoltaic power generation as a green and environmentally friendly energy form is increasingly widely used in smart grids and distributed energy systems. In order to improve the energy conversion efficiency and operational stability of the photovoltaic system, researchers generally use dynamic adjustment strategies based on controllers to real-time regulate the key parameters of the system, so as to cope with changes in factors such as light intensity, environmental temperature, and load disturbance. The common photovoltaic control methods at present mostly rely on traditional PID control or empirical formula adjustment, and the parameters thereof are mostly obtained through manual setting or empirical rules, which are difficult to adapt to complex and variable actual working conditions, especially in multi-task scenarios, and have poor universality and generalization ability.

[0003] In order to overcome the shortcomings of traditional control methods in modeling accuracy and adaptability, in recent years, some researches have tried to introduce adaptive control technology based on reinforcement learning, which continuously learns control strategies through interaction with the environment to improve control performance. However, the existing reinforcement learning methods face many challenges in actual deployment. On the one hand, the control strategy trained in the multi-task scenario is difficult to generalize to new tasks, and the strategy transfer ability is limited; on the other hand, a single strategy network structure is difficult to balance universality and task adaptability, and the strategy updating process is highly sensitive to environmental noise and feedback error, which easily leads to unstable convergence or strategy degradation. In addition, the current interaction mechanism mostly uses the method of directly applying actions to the real environment, without fully considering the introduction of model-assisted mechanisms and robust feedback correction strategies, resulting in low sample efficiency and potential control risks in complex systems.

[0004] In summary, the existing technology still lacks an adaptive controller tuning method that can balance the strategy generalization ability and local rapid adaptation ability in photovoltaic control tasks, especially in multi-task environments. How to realize efficient training and stable optimization of the control strategy through hierarchical strategy modeling, meta-learning parameter aggregation, and model-assisted interaction mechanisms is a key technical problem to be solved at present. SUMMARY

[0005] One object of the present application is to provide an adaptive controller tuning method based on reinforcement learning. The present application fully combines recurrent neural network modeling, hierarchical strategy structure, and meta-learning mechanism, and describes in detail the intelligent tuning process of controller parameters in a multi-task photovoltaic control environment through state dynamic coding, strategy optimization training, and rapid adaptation fine-tuning, which has the advantages of strong generalization ability, fast adaptation speed, and high control precision.

[0006] The application discloses a reinforcement learning-based adaptive controller tuning method, which comprises the following steps: S1, constructing a photovoltaic control task environment; S2, inputting a state sequence into a long short-term memory network to output an embedded state vector; S3, constructing a hierarchical policy network comprising a basic policy module and a rapid adaptation module, wherein the basic policy module is used for extracting general policy features, and the rapid adaptation module is used for local policy fine-tuning in a current task environment; S4, training the hierarchical policy network, inputting the embedded state vector into the hierarchical policy network to generate controller parameter actions, and using a multi-layer feedforward neural network to interact with the photovoltaic control task environment to obtain responses and reward feedback, and performing target value calculation and policy optimization; S5, after the training is completed, using a meta-learning optimization method to aggregate and update parameters of the basic policy module to obtain general policy initial parameters; S6, deploying the optimized hierarchical policy network to a target photovoltaic control task, freezing the basic policy module, and only performing local fine-tuning on the rapid adaptation module.

[0007] Optionally, the task environment has different light intensities, temperature conditions and load disturbance parameters, forming a task set, and a state sequence is defined as a group of historical observation states in each task, wherein the states comprise error signals, voltages, currents, light intensities and temperature information; Optionally, the S2 specifically comprises the following steps: S21, collecting states at consecutive time steps to construct a state sequence, wherein the states comprise output errors, voltages, currents, light intensities and temperature data; S22, inputting the state sequence into a recurrent neural network model, wherein the recurrent neural network model is a multi-layer LSTM network and is used for extracting dynamic features in a time sequence; S23, calculating a hidden state vector corresponding to each time step by using the LSTM network, and selecting a hidden state at a last time step as an embedded state vector. .

[0008] Optionally, the S3 specifically comprises the following steps: S31, receiving the embedded state vector as an input of the hierarchical policy network; S32, constructing a basic policy module, wherein the basic policy module comprises a two-layer fully connected neural network structure and is used for extracting the embedded state vector ​​​​​The intermediate feature representation is outputted by extracting general strategy features ; S33, constructing a fast adaptation module, the fast adaptation module being a multi-layer fully connected neural network structure, receiving the intermediate feature representation as input, and being designed to perform a small amount of gradient more Newly implement local fine-tuning of the strategy S34, connecting the base strategy module and the fast adaptation module in a series mode to form a complete layered strategy network structure.

[0009] Optionally, the S4 specifically comprises: S41, inputting the embedded state vector to the layered strategy network, and generating the controller parameter action by the base strategy module and the fast adaptation module jointly ; S42, applying the controller parameter action to the photovoltaic control task environment, performing the controller adjustment behavior, and synchronously recording the response data to construct the interaction input pair ; S43, concatenating the current embedded state vector and the controller parameter action to form an interaction input pair, and inputting the interaction input pair into a multi-layer feedforward neural network with noise disturbance to output a predicted state vector ; simultaneously collecting an actual feedback state and a reward signal from the photovoltaic control task environment, and calculating a state residual error ; if the state residual error exceeds a threshold value , triggering a robust feedback correction mechanism, and calling a reward predictor to perform weighted correction on the reward signal ; S44, storing a four-tuple in an experience replay pool , the corresponding state representing a next state obtained by the photovoltaic control task environment in response to the execution of the controller parameter action , and being used for target Q value calculation S45, sampling a batch of interaction samples from the experience replay pool , and inputting the corresponding state into a target strategy network to obtain a target action , wherein is a Gaussian disturbance noise S46, inputting the target action corresponding state input into the target Critic network, calculate two target Q values, take the minimum value of the two target Q values to construct the target value ; S47, update the parameters of the Critic network with , the parameters with approximate the target value , minimize the loss function ; S48, update the policy network parameters every steps , use the current embedded state vector as input, and then send the action output by the policy network together with the state into the Critic network ; S49, perform a soft update operation on the parameters with , use the scale factor to perform an exponential moving average on the current network parameters and the target policy network , complete the soft update of the target policy network ; Optionally, the feedforward neural network includes two linear mapping layers and a nonlinear activation unit, wherein the first linear layer increases the dimension of the input to the hidden layer, the activation function uses the GELU function, and noise disturbance is introduced to follow the normal distribution , and the second linear layer restores the activation result to the state space dimension.

[0010] Optionally, the S5 specifically includes: S51, respectively execute the training process of the hierarchical policy network in a plurality of photovoltaic control task environments, obtain the main policy network parameters of the hierarchical policy network in the convergence state in each task , extract the parameter part of the basic policy module to form a basic policy parameter set; S52, define an aggregated optimization objective function for the basic policy module , take the generalization ability of the policy performance between tasks as the optimization objective, and update the basic policy parameters through the meta-gradient method; S53, use a model-independent meta-learning mechanism to locally update the fast adaptation module of each task, and calculate the updated task loss function about the meta-gradient of the shared basic policy module parameters , aggregate the meta-gradents of all tasks and perform parameter update; S54, update the basic policy module parameters initial parameters of a general strategy.

[0011] Optionally, the S6 specifically comprises: S61, initializing an embedded state encoding module and an aggregated updated hierarchical strategy network in a target photovoltaic control task environment to be deployed, the hierarchical strategy network comprising a frozen basic strategy module and a fast adaptation module to be fine-tuned . S62, constructing a state-action-reward interaction process, inputting an embedded state vector into the frozen basic strategy module and the fast adaptation module to jointly constitute a hierarchical strategy network, generating a controller parameter action , applying it to the photovoltaic control task environment, collecting a next state and a reward , and constituting a triple . S63, fixing the basic strategy module parameters in the fine-tuning stage, and performing local optimization on the fast adaptation module parameters based only on the triple , and adopting a policy gradient method to maximize the expected cumulative reward: S64, completing the adaptation of the hierarchical strategy network in the target photovoltaic control task environment based on the fine-tuned fast adaptation module .

[0012] Optionally, the hierarchical strategy network is a neural network structure composed of a basic strategy module and a fast adaptation module in series, wherein the basic strategy module extracts general strategy features in the embedded state vector through a two-layer fully connected neural network, the fast adaptation module receives intermediate feature representations output by the basic module and realizes local fine-tuning of the strategy in a specific photovoltaic control task; the hierarchical strategy network is used to generate a controller parameter action in the training stage, interact with the environment, and improve the control performance through target value calculation and policy optimization mechanism, and the basic strategy module is frozen in the deployment stage, and the fast adaptation module is locally updated.

[0013] The beneficial effects of the present application are: The application has made significant technical progress in control strategy modeling and parameter optimization by introducing an adaptive controller tuning method based on reinforcement learning. First, the state sequence in the photovoltaic system is dynamically encoded using a recurrent neural network, which can fully extract the time-dependent features of system operation and provide embedded state representation with time correlation for the subsequent policy network, significantly improving the accuracy and stability of state modeling. Second, a hierarchical policy network architecture including a basic policy module and a fast adaptation module is constructed, which can effectively extract a universal policy with migration ability during the multi-task training phase, and achieve rapid adaptation to specific task environments during the deployment phase, significantly reducing the policy adjustment time and improving the response flexibility of regulation and control.

[0014] During the training process, a multi-layer feedforward neural network with noise disturbance is introduced to interact with the controller action and the environment, effectively improving the robustness of the system to non-ideal feedback and disturbance noise. Combined with the robust feedback correction mechanism, the abnormal state feedback is corrected, further enhancing the stability and safety of the system. At the same time, the double-target Critic structure and delayed policy update mechanism in the TD3 algorithm are used to effectively alleviate the problems of policy overestimation and unstable policy convergence. Finally, through the parameter aggregation method based on model-independent meta-learning, the unified update and generalization ability of the basic policy module are realized, providing a solid foundation for the rapid deployment and efficient operation of control strategies in different photovoltaic scenarios.

[0015] In summary, the application not only significantly outperforms traditional methods in terms of modeling accuracy and policy optimization efficiency, but also has good scalability, adaptability and engineering application value in actual deployment, with wide application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0017] Fig. 1 The overall flowchart of an adaptive controller tuning method based on reinforcement learning proposed by the application; Fig. 2 The hierarchical policy network structure diagram of a power communication analysis method based on big data proposed by the application. DETAILED DESCRIPTION

[0018] The application will now be further described in detail in conjunction with the drawings. These drawings are simplified schematic diagrams that only schematically illustrate the basic structure of the application, and therefore only show the components related to the application.

[0019] REFERENCES Figs. 1-2An adaptive controller tuning method based on reinforcement learning includes the following steps: S1. Construct the photovoltaic control task environment; S2. Input the state sequence into the Long Short-Term Memory network and output the embedded state vector; S3. Construct a hierarchical policy network that includes a basic policy module and a rapid adaptation module. The basic policy module is used to extract general policy features, and the rapid adaptation module is used to fine-tune local policies in the current task environment. S4. Train the hierarchical policy network, input the embedded state vector into the hierarchical policy network to generate controller parameters and actions, use the multi-layer feedforward neural network to interact with the photovoltaic control task environment to obtain response and reward feedback, and perform target value calculation and policy optimization. S5. After training is completed, the parameters of the basic policy module are aggregated and updated using the meta-learning optimization method to obtain the initial parameters of the general policy. S6. Deploy the optimized hierarchical strategy network to the target photovoltaic control task, freeze the basic strategy module, and only make local fine-tuning to the rapid adaptation module.

[0020] This invention proposes an adaptive controller tuning method based on reinforcement learning. First, in a constructed photovoltaic control task environment, the state sequence is dynamically encoded using a long short-term memory network to obtain embedded state vectors. Then, a hierarchical policy network is constructed, consisting of a basic policy module and a fast adaptation module. The basic module extracts general features, while the fast adaptation module performs local policy fine-tuning. During the training phase, the hierarchical policy network generates controller parameter actions, and a multi-layer feedforward neural network interacts with the environment to obtain feedback information, performing target value calculation and policy optimization. A meta-learning method is then used to aggregate and update the parameters of the basic policy module. Finally, the optimized hierarchical policy network is deployed in a new task, the basic policy module is frozen, and only the fast adaptation module is locally fine-tuned, achieving rapid adaptation and efficient controller tuning.

[0021] In this embodiment, the task environment has different light intensities, temperature conditions, and load disturbance parameters, forming a task set. In each task, a state sequence is defined as a set of historical observation states, which include error signals, voltage, current, light intensity, and temperature information. In this embodiment, S2 specifically includes: S21, Acquisition of continuous time steps Under the given system state, construct a state sequence. ,in Indicates time The system status includes output error, voltage, current, light intensity, and temperature data; S22, Sequence of states The input is fed into a recurrent neural network model, which is a multi-layered long short-term memory network (LSTM) used to extract dynamic features from the time series. S23. Calculate each time step using an LSTM network. The corresponding hidden state vector And select the last time step The hidden state is used as the embedded state vector ,Right now ; S24. Embedded state vector As input to the subsequent hierarchical policy network, it is used to generate the controller parameter actions at the corresponding time.

[0022] In this embodiment, S3 specifically includes: S31. Receive the embedded state vector encoded by the Long Short-Term Memory Network. And use this embedded state vector as the input to the hierarchical policy network; S32. Construct a basic policy module, which includes a two-layer fully connected neural network structure for using embedded state vectors. Extract general strategy features and output intermediate feature representations. The calculation of each layer is expressed as , , , This is the weight matrix. , For bias terms, For activation functions; S33. Construct a fast adaptation module, wherein the fast adaptation module is a multi-layer fully connected neural network structure, and receives the intermediate feature representation. As input, it is designed to improve performance in the target task environment with a small number of gradients. The new implementation strategy has undergone minor adjustments. S34. Connect the basic policy module and the rapid adaptation module in series to form a complete hierarchical policy network structure, and update the corresponding parameters in subsequent training phases.

[0023] In this embodiment, S4 specifically includes: S41. Encode the embedded state vector obtained in step S2 using the Long Short-Term Memory network. The input is fed into the hierarchical policy network constructed in step S3, and the controller parameter actions are generated jointly by the basic policy module and the fast adaptation module. ,in This is the proportionality coefficient. The integral coefficient is... is a differential coefficient; S42, the controller parameter action acts on the photovoltaic control task environment, executes the controller adjustment behavior and synchronously records the system response data, constructs the interactive input-output pair ; the current embedded state vector and the controller parameter action are spliced to form a joint input feature, which is input into a multi-layer feedforward neural network with noise disturbance, the feedforward neural network includes at least two linear mapping layers and a nonlinear activation unit, wherein the first linear layer increases the dimension of the input to the hidden layer, the activation function adopts the GELU function, and the noise disturbance conforming to the normal distribution is introduced , the second linear layer restores the activation result to the state space dimension, and outputs the predicted state vector ; at the same time, the actual system feedback state and the reward signal are collected, and the state residual error is calculated ; if the residual error exceeds the threshold value, the robust feedback correction mechanism is triggered, the reward predictor is called to perform weighted correction to generate the corrected reward , wherein is the confidence weight coefficient S43, the four-tuple is stored in the experience replay pool , the corresponding state represents the next state obtained by the photovoltaic control task environment after executing the controller parameter action , which is used for target Q value calculation S44, a batch of interactive samples are sampled from the experience replay pool , the target action is generated using the target actor network, and the target policy network is used to infer the corresponding state to obtain the target action , wherein the Gaussian disturbance noise S45, the target action and the corresponding state are input into the target Critic network to calculate two target Q values and , and the minimum value is taken to construct the target value , wherein is the discount factor S46, the parameters of the current Critic network and are updated to approximate the target value Minimize the loss function ; S47, every Update network parameters every step Using the current embedded state vector As input, the Actor main network outputs an action, which is then fed into the Critic network along with the state. The optimization objective is to maximize ; S48. Perform a soft update operation on the target network parameters, using a scaling factor to perform an exponential moving average between the current network parameters and the target network parameters, thus completing the soft update of the target network. The update rules are as follows:

[0024] ; S49. Repeat S41 to S48 until the hierarchical policy network converges in all training tasks.

[0025] In the training phase, this invention inputs an embedded state vector encoded by a long short-term memory network into a hierarchical policy network to generate controller parameter actions including proportional, integral, and differential coefficients. These actions are then applied to the photovoltaic control task environment, and interactive input pairs are constructed using system response data. A multi-layer feedforward neural network with noise perturbation is used to predict the next state, and the state residual is calculated with the actual feedback state. A reward correction mechanism is triggered to generate a correction reward when the threshold is exceeded. Subsequently, the interactive quadruples are stored in an experience replay pool. A target action with perturbation is generated through a target policy network, and it is input into a bi-target Critic network along with the corresponding state to construct a target Q-value. Based on this, the Critic network parameters are updated to approximate the target value. Simultaneously, the Actor main network parameters are periodically updated to maximize the Q-value output. An exponential moving average is applied to the target network parameters using a soft update mechanism, ultimately achieving stable convergence of the hierarchical policy network within the training task set.

[0026] In this embodiment, the feedforward neural network includes two linear mapping layers and a nonlinear activation unit. The first linear layer upscales the input to the hidden layer, uses the GELU activation function, and introduces noise perturbation that follows a normal distribution. The second linear layer restores the activation results to the state space dimension.

[0027] In this embodiment, S5 specifically includes: S51. Execute the training process from S41 to S49 in multiple photovoltaic control task environments respectively, and obtain the main policy network parameters of the hierarchical policy network in the convergence state in each task. Extract the parameter portion of the basic strategy module to form the basic strategy parameter set. wherein denotes the base policy module parameter in the th task, is the number of trained task environments; S52, define the aggregated optimization objective function of the base policy module , taking the generalization ability of the policy performance between tasks as the optimization objective, update the base policy parameters through the meta-gradient method; S53, adopt the model-independent meta-learning (MAML) mechanism to locally update the fast adaptation module of each task respectively, calculate the meta-gradient of the updated task loss function with respect to the shared base policy module parameters, aggregate the meta- gradients of all tasks and perform parameter update, and the update formula is: ; wherein, is the shared parameter of the base policy module, is the meta-learning rate, denotes the loss function value of the th task after fine-tuning; S54, take the aggregated and updated base policy module parameters as the initial parameters of the general policy, and use them as the starting parameters for initializing the hierarchical policy network in the subsequent new photovoltaic control task environment, and only need to fine-tune the fast adaptation module to complete the optimization of the controller parameters.

[0028] After the training is completed, the hierarchical policy network training process constructed in the present application is first executed in multiple photovoltaic control task environments, the converged base policy module parameters in each task are extracted to form a base policy parameter set; then an aggregated optimization function taking the strategy generalization performance between tasks as the objective is constructed, and the model-independent meta-learning (MAML) method is adopted to locally update the fast adaptation module of each task, the meta-gradient of the fine-tuned loss function with respect to the shared base policy parameters is calculated, the gradient information of all tasks is aggregated, and the aggregated update of the base policy module parameters is completed; finally, the updated parameters are taken as the initialization value of the general policy, which is used for the subsequent fast adaptation of new tasks, thereby significantly improving the generalization ability and adaptation efficiency of the controller in the multi-task environment.

[0029] In the present embodiment, the S6 specifically comprises: S61, initialize the embedded state encoding module and the aggregated and updated hierarchical policy network in the target photovoltaic control task environment to be deployed, wherein the hierarchical policy network comprises a frozen base policy module and a fast adaptation module to be fine-tuned wherein the general policy initial parameters obtained in step S5 are taken as the initial parameters of the hierarchical policy network, are the trainable sub-module parameters; S62. Construct a state-action-reward interaction flow, inputting the embedded state vector into a hierarchical policy network jointly formed by the frozen basic policy module and the rapid adaptation module, to generate controller parameter actions. Applying this to the photovoltaic control task environment, it collects the next state and reward. To form a triple ; S63. During the fine-tuning phase, fix the parameters of the basic strategy module. Based solely on interaction samples, local optimization is performed on the parameters of the fast adaptation module, and the policy gradient method is used to maximize the expected cumulative reward. ; Where is the learning rate. This represents the expected total reward obtained using the current strategy; S64, Based on the finely tuned rapid adaptation module This enables the hierarchical strategy network to be adapted to the target photovoltaic control task environment, ensuring that the generated controller parameters meet the control accuracy and response performance requirements of the target system.

[0030] In the target photovoltaic control task environment, the embedded state coding module and the aggregated and updated hierarchical policy network are first initialized, and the parameters of the basic policy module are frozen, leaving only the rapid adaptation module as trainable. Then, environmental feedback data is obtained through the state-action-reward interaction process to form the triples required for training. In the fine-tuning stage, local optimization is performed only on the parameters of the rapid adaptation module, and the expected cumulative reward is maximized using the policy gradient method. Finally, through the rapid adjustment of this module, the hierarchical policy network is efficiently adapted to the target environment, ensuring that the generated controller parameters and actions meet the system's requirements for control accuracy and dynamic response performance.

[0031] In this embodiment, the hierarchical policy network is a neural network structure composed of a basic policy module and a fast adaptation module connected in series. The basic policy module extracts general policy features from the embedded state vector through a two-layer fully connected neural network. The fast adaptation module receives the intermediate feature representation output by the basic module and performs local fine-tuning of the policy in specific photovoltaic control tasks. During the training phase, the hierarchical policy network is used to generate controller parameter actions, interact with the environment, and improve control performance through target value calculation and policy optimization mechanisms. During the deployment phase, the basic policy module is frozen, and the fast adaptation module is locally updated.

[0032] Example 1 To verify the feasibility and effectiveness of the adaptive controller tuning method based on reinforcement learning proposed in the present application, we carried out a 30-day comparative test on an intelligent photovoltaic control system integration experimental platform. The test platform has a programmable load simulation module, a multi-source environmental parameter control device, and a complete grid-connected inverter and data acquisition system, and can truly reproduce the photovoltaic control requirements under various complex task conditions.

[0033] Table 1: Photovoltaic controller tuning comparative experimental data

[0034] As shown in Table 1, during the experiment, we set up 10 different task environments, each of which sets the combination of factors such as light intensity, temperature conditions, and load disturbance amplitude, covering typical morning, noon, high temperature, high disturbance, and other scenarios, forming a task set to train and verify the generalization and adaptation ability of the hierarchical policy network. The traditional method uses a fixed coefficient PID controller for control, while the method of the present application trains and deploys the optimized control strategy through the proposed reinforcement learning tuning mechanism based on LSTM state encoding and hierarchical policy network under the same conditions.

[0035] In the present application, the basic policy module extracts the general control features between tasks and performs meta-learning aggregation in multiple environments, while the fast adaptation module fine-tunes quickly to adapt to the current task in the deployment stage. In the experiment, the system collects error signals, voltage, current, and other state sequences, inputs them into the LSTM network to encode and generate embedded state vectors, and then generates controller parameter actions through the joint of the basic and fast modules, and acts on the target photovoltaic control task environment. In the interaction process, the system response and reward signals are collected, the action value is evaluated through the double Critic network, and the policy network and Critic network update are performed using the TD3 optimization strategy.

[0036] The experimental results show that in all tasks, the method of the present application achieves significantly better regulation performance than the traditional PID control. The average steady-state error of the traditional PID controller is maintained at 3.5-5.0, while the error of the controller tuned by the present application is significantly reduced to 1.0-2.5, with an average reduction of more than 60%. In addition, the convergence time of the control strategy in each task is maintained at 60-150 seconds, and the system stable time is generally less than that of the traditional method, showing higher response efficiency. The experimental data also show that the general initial strategy generated on the basis of multi-task training has good generalization ability, so that only fast fine-tuning is needed in new tasks to achieve high-precision control effect.

[0037] In summary, the reinforcement learning optimization method proposed by the application can significantly improve the dynamic response capability and control accuracy of the photovoltaic system, has good task adaptability and migration ability, and is especially suitable for actual photovoltaic application scenes with strong dynamic fluctuation of the environment and frequent load disturbance. The table "Comparison of photovoltaic controller optimization experimental data" records the detailed performance index comparison of the application under different task conditions, further verifying the effectiveness and engineering feasibility of the method.

[0038] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the application within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. A method for optimizing an adaptive controller based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct the photovoltaic control task environment; S2. Input the state sequence into the Long Short-Term Memory network and output the embedded state vector; S3. Construct a hierarchical policy network that includes a basic policy module and a rapid adaptation module. The basic policy module is used to extract general policy features, and the rapid adaptation module is used to fine-tune local policies in the current task environment. S4. Train the hierarchical policy network, input the embedded state vector into the hierarchical policy network to generate controller parameters and actions, use a multi-layer feedforward neural network with noise perturbation to interact with the photovoltaic control task environment to obtain response and reward feedback, and perform target value calculation and policy optimization. S5. After training is completed, the parameters of the basic policy module are aggregated and updated using the meta-learning optimization method to obtain the initial parameters of the general policy. S6. Deploy the optimized hierarchical strategy network to the target photovoltaic control task, freeze the basic strategy module, and only make local fine-tuning to the rapid adaptation module.

2. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, The task environment has different light intensity, temperature conditions and load disturbance parameters, forming a task set. In each task, a state sequence is defined as a set of historical observation states, which include error signals, voltage, current, light intensity and temperature information.

3. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: S21, Acquisition of continuous time steps Under the given conditions, a state sequence is constructed, including output error, voltage, current, light intensity, and temperature data; S22. Input the state sequence into the recurrent neural network model, wherein the recurrent neural network model is a multi-layer LSTM network, used to extract dynamic features in the time series; S23. Calculate each time step using an LSTM network. The corresponding hidden state vector And select the last time step The hidden state is used as the embedded state vector .

4. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31, Receive embedded state vector , as input to the hierarchical policy network; S32. Construct a basic policy module, which includes a two-layer fully connected neural network structure for using embedded state vectors. Extract general strategy features and output intermediate feature representations. ; S33. Construct a fast adaptation module, wherein the fast adaptation module is a multi-layer fully connected neural network structure, and receives the intermediate feature representation. As input, it is designed to improve performance in the target task environment with a small number of gradients. The new implementation strategy has undergone minor adjustments. S34. Connect the basic strategy module and the rapid adaptation module in series to form a complete hierarchical strategy network structure.

5. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: S41. Embedded state vector The input is fed into a hierarchical policy network, where the basic policy module and the rapid adaptation module jointly generate controller parameter actions. ; S42, Activate controller parameters Operating within a photovoltaic control environment, it executes controller adjustment behaviors and synchronously records response data, constructing an interactive input pair. ; S43. Transfer the current embedded state vector Action with controller parameters The input pairs are concatenated to form interactive input pairs, which are then fed into a multi-layer feedforward neural network with noise perturbation, and the output is a predicted state vector. Simultaneously, actual feedback status is collected from the photovoltaic control task environment. With reward signals And calculate the state residuals. If the state residual Exceeding the threshold This triggers the robust feedback correction mechanism, which calls the reward predictor. For reward signals Perform weighted adjustments to generate correction rewards. ; S44, the quadruple Store in the experience replay pool Corresponding state This indicates the execution of controller parameter actions. Then, the next state obtained from the feedback of the photovoltaic control task environment is used to calculate the target Q value; S45, From the Experience Replay Pool A batch of interaction samples were sampled, and the corresponding states were... Input to the target policy network To obtain the target action ,in This is Gaussian perturbation noise. S46, Target Action Corresponding state The input is fed into the target Critic network, two target Q-values ​​are calculated, and the minimum of the two target Q-values ​​is taken to construct the target value. ; S47. Update the parameters of the Critic network. and , make parameters and Approaching the target value Minimize the loss function ; S48, every Update the policy network parameters every step Using the current embedded state vector As input, the actions and states output by the policy network are then fed into the Critic network. ; S49, Regarding parameters and Perform a soft update operation using a scaling factor. For the current network parameters and the target policy network Perform an exponential moving average to complete the target policy network. Soft update.

6. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, The feedforward neural network comprises two linear mapping layers and a nonlinear activation unit. The first linear layer upscales the input to the hidden layer, uses the GELU activation function, and introduces noise perturbation that follows a normal distribution. The second linear layer restores the activation results to the state space dimension.

7. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, S5 specifically includes: S51. Execute the training process of the hierarchical policy network in multiple photovoltaic control task environments, and obtain the main policy network parameters of the hierarchical policy network in the convergence state in each task. Extract the parameter part of the basic strategy module to form the basic strategy parameter set; S52. Define the aggregate optimization objective function for the basic strategy module. With the optimization objective of generalizing policy performance across tasks, the basic policy parameters are updated using the meta-gradient method. S53. Employing a model-independent meta-learning mechanism, locally update the fast adaptation module for each task, and calculate the task loss function after the update. Regarding parameters of the shared basic strategy module The meta-gradients of all tasks are aggregated and parameter updates are performed. S54. Aggregate and update the parameters of the basic strategy module. Used as initial parameters for the general strategy.

8. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, S6 specifically includes: S61. In the target photovoltaic control task environment to be deployed, initialize the embedded state coding module and the aggregated and updated hierarchical policy network, wherein the hierarchical policy network includes the frozen basic policy module. With the rapid adaptation module to be fine-tuned ; S62. Construct the state-action-reward interaction flow, embedding the state vector Input to the freeze basic strategy module With rapid adaptation module In the hierarchical policy network formed by the joint system, controller parameters and actions are generated. Applying it to the photovoltaic control task environment to collect the next state With rewards To form a triple ; S63. During the fine-tuning phase, fix the parameters of the basic strategy module. Based solely on triples For rapid adaptation module parameters Perform local optimization by using the policy gradient method to maximize the expected cumulative reward: S64, Based on the finely tuned rapid adaptation module This completes the adaptation of the hierarchical strategy network to the target photovoltaic control task environment.

9. The adaptive controller tuning method based on reinforcement learning according to claim 1, characterized in that, The hierarchical policy network is a neural network structure composed of a basic policy module and a fast adaptation module connected in series. The basic policy module extracts general policy features from the embedded state vector through a two-layer fully connected neural network. The fast adaptation module receives the intermediate feature representation output by the basic module and performs local fine-tuning of the policy in specific photovoltaic control tasks. During the training phase, the hierarchical policy network is used to generate controller parameter actions, interact with the environment, and improve control performance through target value calculation and policy optimization mechanisms. During the deployment phase, the basic policy module is frozen, and the fast adaptation module is locally updated.

Images