Method, system and device for adaptive voltage regulator parameter optimization
Through adaptive reinforcement learning algorithm and multi-layer adaptive mechanism, the regulator parameters in the power system are optimized, and the problems of insufficient robustness and universality in the existing technology are solved, and the intelligent optimization of regulator parameters is realized. Dynamic adjustment to adapt to complex changes in the power system.
Patent Information
- Application Number
- CN202411154868.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing technology is difficult to adapt to the dynamic changes and complexity of power systems, especially in the case of new energy grid connection and load sudden changes, the robustness and universality of the voltage regulator parameter optimization method is insufficient, making it difficult to achieve real-time optimization and multi-objectives. balance.
Adaptive reinforcement learning algorithm is adopted to dynamically adjust the learning rate and exploration strategy of the reinforcement learning network through micro, mesoscopic and macro-adaptive mechanisms, calculate parameter sensitivity and update step size, and perform power system operating conditions recognition and model migration. Optimize regulator parameters.
It improves the robustness and versatility of the voltage regulator parameter adjustment system, realizes intelligent optimization and dynamic adjustment, can quickly adapt to system changes, and improves control effect and generalization capabilities.
Smart Images

Figure CN118657072B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning, and in particular to a method, system and device for optimizing parameters of an adaptive voltage regulator. Background Art
[0002] Voltage regulator parameter optimization has become an important topic in the operation and control of power systems, which is closely related to the stability, reliability and economy of the power grid. As a key device in the power system, the voltage regulator's main function is to maintain the voltage within a preset range to ensure the safe and stable operation of the power system. The optimization of voltage regulator parameters involves adjusting multiple control parameters, such as proportional gain, integral time constant and differential time constant, to adapt to different operating conditions and load changes. Reasonable voltage regulator parameter settings are essential for suppressing voltage fluctuations, improving system dynamic performance and improving power quality. Through parameter optimization, the voltage regulator can better adapt to different operating conditions and load characteristics, and improve the flexibility and robustness of the system. With the large-scale access of renewable energy, the power system faces more uncertainty and volatility. Optimized voltage regulator parameters can better cope with these challenges and promote the efficient use of new energy. The traditional trial and error method mainly relies on the experience and intuition of engineers to determine the appropriate parameters by repeatedly debugging and observing the system response. This method is simple and intuitive, but inefficient and difficult to cope with complex power systems. The linearization model method first simplifies the complex power system into a linear model, and then uses classical control theory (such as the root locus method, frequency response method, etc.) to design the parameters of the voltage regulator. This method is relatively simple to calculate, but its accuracy is limited for large power systems with strong nonlinearity. Intelligent optimization algorithms, such as genetic algorithms and particle swarm optimization algorithms, search for the optimal solution by simulating natural evolution or group behavior. These algorithms usually define an objective function (such as a system stability index) and then find the optimal voltage regulator parameters through iterative optimization. This type of method can search for the global optimal solution in a complex nonlinear system, but the calculation is large and may fall into a local optimum.
[0003] However, these existing technologies still have some problems in practical applications. Most methods are difficult to adapt to the dynamic changes and complexity of the power system, especially when facing situations such as the grid connection of new energy sources and sudden load changes. Although some intelligent optimization algorithms can obtain good results, they are computationally intensive and difficult to achieve real-time optimization. Many methods require accurate system models, and accurate modeling of actual power systems is often difficult. Actual power systems need to consider multiple optimization objectives at the same time, such as stability, economy, and environmental protection. Traditional methods are difficult to effectively balance these objectives. Some methods are difficult to achieve real-time optimization and cannot respond to rapid changes in system status in a timely manner. The performance of some methods may drop significantly when system parameters change. Many methods perform well under specific conditions, but are difficult to generalize to other conditions or systems, and their generalization capabilities are limited.
[0004] Therefore, there is an urgent need for a technical solution that can improve the robustness and versatility of the parameter adjustment system and enhance the control effect. Summary of the invention
[0005] In order to solve the deficiencies of the prior art, the embodiments of the present application provide a method, system and device for optimizing parameters of an adaptive voltage regulator. The present application solves the technical problems of the prior art such as poor performance and limited generalization capability.
[0006] An embodiment of the present application provides a method for adaptive voltage regulator parameter optimization, including: initializing a decision network and a value network of a reinforcement learning network using a target reinforcement learning algorithm; sampling the power system where the target voltage regulator is located to model the parameter optimization problem of the target voltage regulator as a Markov decision process; performing micro-adaptation to dynamically adjust the learning rate of the reinforcement learning network and update the exploration strategy; performing meso-adaptation to calculate the parameter sensitivity of the reinforcement learning network and adjust the update step size; performing macro-adaptation to identify the operating conditions of the power system, and the results of the identification are used to perform model migration of the reinforcement learning network; and performing one or more adaptive iterative updates on the reinforcement learning network until the network converges or reaches a preset number of iterations.
[0007] In a possible implementation, micro-adaptation is performed to dynamically adjust the learning rate of the reinforcement learning network and update the exploration strategy, including: obtaining the relative change rate of the cumulative rewards of the reinforcement learning network at the current moment and the previous moment to calculate the target attenuation factor; updating the learning rate at the current moment based on the attenuation factor; obtaining the difference between the current cumulative reward and the historical minimum reward, and the difference is used to construct the target exploration factor; updating the exploration rate at the current moment based on the exploration factor.
[0008] In one possible implementation, mesoscopic adaptation is performed to calculate the parameter sensitivity of the reinforcement learning network and adjust the update step size, including: calculating the partial derivative of the performance indicator with respect to the target parameter, and the partial derivative is used to construct the sensitivity of the target parameter; calculating the policy gradient and multiplying it with the current learning rate and sensitivity to obtain the update step size of the target parameter.
[0009] In a possible implementation, macro-adaptation is performed to identify the operating conditions of the power system, and the results of the identification are used to perform model migration of the reinforcement learning network, including: using an unsupervised learning method to cluster the operating conditions of the power system historical data, and calculating the migration distance of the current reinforcement learning network based on the clustering results; adjusting the reinforcement learning network parameters of the target operating condition by presetting the migration function and the migration distance.
[0010] In one possible implementation, the learning rate at the current moment is updated based on the decay factor, including: ,in, Represents the cumulative reward at the current moment, Indicates the cumulative reward at the previous moment, represents the relative rate of change of the cumulative reward, represents the attenuation adjustment coefficient, represents the attenuation factor, represents the initial learning rate, Represents the updated learning rate.
[0011] In a possible implementation, updating the exploration rate at the current moment based on the exploration factor includes: ,in, Represents the cumulative reward at the current moment, represents the minimum historical reward, Indicates the largest reward in history, represents the exploration adjustment coefficient, represents the exploration factor, represents the initial exploration rate, represents the updated exploration factor.
[0012] A possible implementation method also includes: calculating the predicted mean square error of the reinforcement learning network and counting the number of parameters of the reinforcement learning network; taking a weighted sum of the predicted mean square error and the number of parameters to obtain a complexity evaluation value of the reinforcement learning network; and dynamically increasing or decreasing the number of layers and / or the number of neurons of the reinforcement learning network according to the complexity evaluation value.
[0013] In a possible implementation, the reinforcement learning network parameters of the target working condition are adjusted by presetting the migration function and the migration distance, including: ,in, represents the reinforcement learning network parameters of the adjusted target condition, represents the reinforcement learning network parameters of the target working condition before adjustment, represents the migration distance, represents the first weight matrix, represents the second weight matrix, represents the reinforcement learning network parameters of the current working condition, represents the gradient of the loss function of the reinforcement learning network for the current working condition, represents random noise, Represents element-wise multiplication.
[0014] An embodiment of the present application also provides a system for adaptive voltage regulator parameter optimization, comprising: a modeling unit, a microscopic adaptive unit, a mesoscopic adaptive unit, a macroscopic adaptive unit and an iterative update unit; wherein the modeling unit is used to initialize the decision network and the value network of the target reinforcement learning algorithm; the power system where the target voltage regulator is located is sampled to model the parameter optimization problem of the target voltage regulator as a Markov decision process; the microscopic adaptive unit is used to perform microscopic adaptation to dynamically adjust the learning rate and update the exploration strategy; the mesoscopic adaptive unit is used to perform mesoscopic adaptation to calculate parameter sensitivity and adjust the update step size; the macroscopic adaptive unit is used to perform macroscopic adaptation to identify the operating conditions of the power system, and the identification results are used for model migration; the iterative update unit is used to perform one or more adaptive iterative updates on the reinforcement learning network until the network converges or reaches a preset number of iterations.
[0015] An embodiment of the present application also provides a device for adaptive regulator parameter optimization, comprising: a processor, a memory, and a system bus; wherein the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method described in the above embodiment.
[0016] In the method, system and device for adaptive voltage regulator parameter optimization provided above, the embodiments of the present application can improve the robustness and versatility of the parameter adjustment system and enhance the control effect by introducing adaptive mechanisms at the microscopic, mesoscopic and macroscopic levels. Further, in some embodiments, by utilizing a deep reinforcement learning algorithm, it is possible to autonomously learn the optimal control strategy without manual intervention, thereby realizing intelligent optimization of voltage regulator parameters. Further, in some embodiments, by utilizing the expressive power of deep neural networks, it is possible to process high-dimensional nonlinear systems and is suitable for modern complex power systems. Further, in some embodiments, by introducing mechanisms such as parameter sensitivity analysis, the interpretability of the method can be improved, which is conducive to practical engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 A flow chart of a method for optimizing parameters of an adaptive voltage regulator provided in an embodiment of the present application;
[0019] Figure 2 A schematic block diagram of a system for adaptive voltage regulator parameter optimization provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present application.
[0021] Those skilled in the art will appreciate that the terms "first", "second" and the like in the embodiments of the present application are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they represent the necessary logical order between them. It should also be understood that in the embodiments of the present application, "multiple" may refer to two or more, and "at least one" may refer to one, two or more. It should also be understood that for any component, data or structure mentioned in the embodiments of the present application, in the absence of explicit limitation or contrary revelation given in the context, it can generally be understood as one or more. In addition, the term "and / or" in the present application is only a kind of association relationship describing the associated objects, indicating that there can be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present application generally indicates that the associated objects before and after are an "or" relationship. It should also be understood that the description of each embodiment in the present application emphasizes the differences between the embodiments, and the same or similar parts can refer to each other. For the sake of brevity, they will not be repeated one by one.
[0022] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered as part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.
[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0024] Figure 1 A flow chart of a method for adaptive voltage regulator parameter optimization provided in an embodiment of the present application. It should be noted that as a key device in the power system, the parameter optimization of the voltage regulator is crucial to ensure the stable operation of the power grid. Traditional voltage regulator parameter setting methods often rely on expert experience or simplified models, which are difficult to adapt to the complex and changeable power grid environment. The present application proposes a voltage regulator parameter optimization method based on multi-layer adaptive reinforcement learning, which realizes the intelligent optimization and dynamic adjustment of voltage regulator parameters by introducing an innovative adaptive mechanism.
[0025] This application is based on the deep reinforcement learning framework and introduces three layers of adaptive mechanisms: micro-adaptation, meso-adaptation and macro-adaptation. These three layers of adaptive mechanisms act on different levels and together constitute a comprehensive adaptive optimization system. Micro-adaptation: For the reinforcement learning algorithm itself, dynamic learning rate adjustment and adaptive exploration strategies are introduced. Meso-adaptation: For the voltage regulator model, parameter sensitivity adaptation and model structure adaptation mechanisms are proposed. Macro-adaptation: For the power system environment, an adaptive mechanism combining operating condition identification and transfer learning is designed.
[0026] like Figure 1 As shown, at step S101, the decision network and value network of the reinforcement learning network are initialized using the target reinforcement learning algorithm. The Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm can be used as the basic algorithm framework. The TD3 algorithm includes an Actor network and two Critic networks, where the Actor network is used to generate actions (i.e., regulator parameters) and the Critic network is used to evaluate the value of actions.
[0027] Specifically, the Actor network update formula is: ,
[0028] Critic network update formula: , ,in, and are the parameters of the Actor and Critic networks, is the policy function, is the action-value function, is the discount factor, It's noise.
[0029] At step S102, the power system where the target voltage regulator is located is sampled to model the parameter optimization problem of the target voltage regulator as a Markov decision process. In one embodiment, the voltage regulator parameter optimization problem is modeled as a Markov decision process (MDP), which is defined as follows:
[0030] State space S: includes system state variables such as generator speed deviation, power angle deviation, terminal voltage deviation, etc.
[0031] Action space A: includes adjustable parameters such as the gain and time constant of the regulator.
[0032] Reward function R: designed based on system transient performance indicators, such as maximum overshoot, adjustment time, etc.
[0033] State transition probability P: determined by the dynamic characteristics of the power system.
[0034] At step S103, micro-adaptation is performed to dynamically adjust the learning rate of the reinforcement learning network and update the exploration strategy. On the one hand, an adaptive learning rate mechanism is introduced to dynamically adjust the learning rate according to the performance changes during the training process. Specifically, the relative change rate of the cumulative reward of the reinforcement learning network at the current moment and the previous moment is obtained to calculate the target attenuation factor; the learning rate at the current moment is updated based on the attenuation factor. This includes: ,in, Represents the cumulative reward at the current moment, Indicates the cumulative reward at the previous moment, represents the relative rate of change of the cumulative reward, represents the attenuation adjustment coefficient, represents the attenuation factor, represents the initial learning rate, Represents the updated learning rate.
[0035] On the other hand, based on the performance improvement of adaptive exploration strategy, the exploration rate Dynamically adjust as performance improves. First, obtain the difference between the current cumulative reward and the historical minimum reward, and use the difference to construct the target exploration factor; based on the exploration factor, update the exploration rate at the current moment. This includes: ,in, Represents the cumulative reward at the current moment, represents the minimum historical reward, Indicates the largest reward in history, represents the exploration adjustment coefficient, represents the exploration factor, represents the initial exploration rate, represents the updated exploration factor.
[0036] At step S104, mesoscopic adaptation is performed to calculate the parameter sensitivity of the reinforcement learning network and adjust the update step size. In one embodiment, it includes: calculating the partial derivative of the performance index with respect to the target parameter, the partial derivative is used to construct the sensitivity of the target parameter; calculating the policy gradient, and multiplying it with the current learning rate and sensitivity to obtain the update step size of the target parameter.
[0037] Specifically, parameter sensitivity analysis is introduced to dynamically adjust the optimization weights of different parameters. Define parameter sensitivity: ,in, It is The sensitivity of the parameters, is the parameter value, is the system performance indicator. Next, adjust the parameter update step size according to the sensitivity: .
[0038] In another implementation scenario, it also includes: calculating the prediction mean square error of the reinforcement learning network and counting the number of parameters of the reinforcement learning network; taking a weighted sum of the prediction mean square error and the number of parameters to obtain a complexity evaluation value of the reinforcement learning network; and dynamically increasing or decreasing the number of layers and / or the number of neurons of the reinforcement learning network according to the complexity evaluation value.
[0039] In order to dynamically adjust the model complexity according to system characteristics, the embodiment of the present application defines a model complexity evaluation function: ,in, is the model prediction error, is the number of model parameters, and is the weight coefficient. The results can be evaluated based on the complexity, for example, by using multiple threshold intervals to dynamically increase or decrease the number of model layers and neurons.
[0040] At step S105, macro-adaptation is performed to identify the power system operating conditions, and the results of the identification are used to migrate the model of the reinforcement learning network. First, the power system historical data is clustered using an unsupervised learning method, and the migration distance of the current reinforcement learning network is calculated based on the clustering results. An improved K-means algorithm can be used to introduce a dynamic cluster center update mechanism: ,in, is the kth cluster center, is the sample point, is the learning rate. The migration distance can be obtained by calculating the distance between the current network parameters and the target network parameter condition cluster center.
[0041] Then, the reinforcement learning network parameters of the target working condition are adjusted by presetting the migration function and the migration distance. In one embodiment, based on the working condition identification results, a migration learning strategy is designed to achieve knowledge migration between different working conditions. The preset functions include:
[0042] ,
[0043] in, represents the reinforcement learning network parameters of the adjusted target condition, represents the reinforcement learning network parameters of the target working condition before adjustment, represents the migration distance, represents the first weight matrix, represents the second weight matrix, represents the reinforcement learning network parameters of the current working condition, represents the gradient of the loss function of the reinforcement learning network for the current working condition, represents random noise, Represents element-wise multiplication. Through model migration, the network parameters of the current working condition can be used to synchronously update the models of other working conditions.
[0044] In step S106, the reinforcement learning network is adaptively iterated and updated once or multiple times until the network converges or reaches a preset number of iterations.
[0045] In summary, the present invention provides a method for optimizing parameters of an adaptive voltage regulator. The process of the method is as follows:
[0046] 1. Initialize the Actor and Critic networks of the TD3 algorithm.
[0047] 2. Sample the power system and collect state, action, and reward data.
[0048] 3. Perform micro-adaptation: dynamically adjust the learning rate and update the exploration strategy.
[0049] 4. Perform mesoscopic adaptation: calculate parameter sensitivity and adjust the update step size, evaluate model complexity, and adjust the model structure if necessary.
[0050] 5. Perform macro-adaptation: identify the working conditions and perform model migration based on the identification results.
[0051] 6. Update the Actor and Critic networks.
[0052] 7. Repeat steps 2-6 until convergence or the maximum number of iterations is reached.
[0053] It should be understood that in an implementation scenario, in order to verify the method described in the embodiment of this application, the IEEE39 node system is selected as the simulation test system. The system includes 10 generators, 19 load nodes and 34 transmission lines. MATLAB / Simulink is used for modeling and simulation. The following methods are selected for comparison:
[0054] 1. Traditional PSS parameter setting method
[0055] 2. Basic TD3 algorithm (without adaptive mechanism)
[0056] 3. Multi-layer adaptive TD3 algorithm proposed in this application
[0057] The following indicators are selected to evaluate the performance of the method:
[0058] 1. Maximum Overshoot (MO)
[0059] 2. Settling Time (ST)
[0060] 3. Integral of Time-weighted Absolute Error (ITAE)
[0061] The simulation results of single working condition performance comparison, under the nominal working condition, compare the control performance of the three methods. The results are shown in Table 1 below:
[0062] Table 1 Comparison of control performance of three methods
[0063]
[0064] It can be seen that the multi-layer adaptive TD3 method proposed in the embodiment of the present application is superior to the comparison method in all indicators.
[0065] In addition, to verify the adaptability of the method, the following scenario is designed:
[0066] 1. Load mutation: At t=5s, the load increases by 20%.
[0067] 2. Topology change: At t=10s, a key transmission line is disconnected.
[0068] 3. New energy access: At t=15s, a large wind farm is connected.
[0069] The performance adaptability of the three methods in the above scenarios is compared. The results show that the method of the embodiment of the present application can quickly adapt to system changes and maintain good control performance. Specifically:
[0070] 1. In the load mutation scenario, the adjustment time of the present application method is shortened by 35% compared with the traditional method and 20% compared with the basic TD3 method.
[0071] 2. In the topology change scenario, the maximum overshoot of the proposed method is reduced by 40% compared with the traditional method and by 25% compared with the basic TD3 method.
[0072] 3. In the scenario of new energy access, the ITAE index of this application method is reduced by 50% compared with the traditional method and 30% compared with the basic TD3 method.
[0073] In summary, these results fully demonstrate the effectiveness of the multi-layer adaptive mechanism proposed in the embodiment of the present application. This method comprehensively covers the three levels of algorithm, model and environment by introducing adaptive mechanisms at the micro, meso and macro levels. Intelligent optimization and dynamic adjustment of the voltage regulator parameters are realized. At the micro level, dynamic learning rate adjustment and adaptive exploration strategy are introduced. At the meso level, parameter sensitivity adaptation and model structure adaptation mechanisms are proposed. At the macro level, an adaptive mechanism combining working condition identification with transfer learning is designed. The simulation results show that the method can not only obtain a control effect that is superior to the traditional method, but also has good adaptability and can adapt to different system working conditions and disturbances.
[0074] Figure 2 A schematic block diagram of a system for adaptive regulator parameter optimization provided in an embodiment of the present application. It should be understood that the system shown in the figure is exemplary and not restrictive. This means that the system architecture involved is not limited to a specific form or design, but is presented as an example. In other words, the architecture shown in the figure can be regarded as a way of expression to clearly describe related concepts and relationships, and does not exclude other forms of architecture. Therefore, when interpreting the architecture in the picture, it should be understood that the model is flexible and diverse, and its purpose is to provide an exemplary description rather than a restrictive provision on a specific form.
[0075] Specifically, an embodiment of the present application includes: a modeling unit 201, a microscopic adaptive unit 202, a mesoscopic adaptive unit 203, a macroscopic adaptive unit 204 and an iterative update unit 205; wherein the modeling unit 201 is used to initialize the decision network and the value network of the target reinforcement learning algorithm; the power system where the target voltage regulator is located is sampled to model the parameter optimization problem of the target voltage regulator as a Markov decision process; the microscopic adaptive unit 202 is used to perform microscopic adaptation to dynamically adjust the learning rate and update the exploration strategy; the mesoscopic adaptive unit 203 is used to perform mesoscopic adaptation to calculate the parameter sensitivity and adjust the update step size; the macroscopic adaptive unit 204 is used to perform macroscopic adaptation to identify the operating conditions of the power system, and the results of the identification are used for model migration; the iterative update unit 205 is used to perform one or more adaptive iterative updates on the reinforcement learning network until the network converges or reaches a preset number of iterations.
[0076] Furthermore, an embodiment of the present application also provides a device for adaptive regulator parameter optimization, comprising: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute any of the above methods.
[0077] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the above methods.
[0078] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0079] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0080] It should also be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.
[0081] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in the embodiments of the present application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown in the embodiments of the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the embodiments of the present application.
Claims
1. A method for optimizing parameters of an adaptive voltage regulator, characterized in that: include: Initialize the decision network and value network of the reinforcement learning network using the target reinforcement learning algorithm; Sampling the power system where the target voltage regulator is located to model the parameter optimization problem of the target voltage regulator as a Markov decision process; Perform micro-adaptation to dynamically adjust the learning rate of the reinforcement learning network and update the exploration strategy; The steps include: obtaining the relative change rate of the cumulative rewards of the reinforcement learning network at the current moment and the previous moment to calculate the target attenuation factor; updating the learning rate at the current moment based on the attenuation factor; obtaining the difference between the current cumulative reward and the historical minimum reward, and using the difference to construct the target exploration factor; and updating the exploration rate at the current moment based on the exploration factor. Perform mesoscopic adaptation to calculate the parameter sensitivity of the reinforcement learning network and adjust the update step size; including: calculating the partial derivative of the performance indicator with respect to the target parameter, the partial derivative is used to construct the sensitivity of the target parameter; calculating the policy gradient and multiplying it with the current learning rate and sensitivity to obtain the update step size of the target parameter; Performing macro-adaptation to identify the working conditions of the power system, and the results of the identification are used to perform model migration of the reinforcement learning network; including: using an unsupervised learning method to cluster the working conditions of the power system historical data, and calculating the migration distance of the current reinforcement learning network based on the clustering results; adjusting the reinforcement learning network parameters of the target working condition by presetting the migration function and the migration distance; Perform one or more adaptive iterative updates on the reinforcement learning network until the network converges or reaches a preset number of iterations; Among them, the learning rate at the current moment is updated based on the attenuation factor, including: ,in, Represents the cumulative reward at the current moment, Indicates the cumulative reward at the last moment, represents the relative rate of change of the cumulative reward, represents the attenuation adjustment coefficient, represents the attenuation factor, represents the initial learning rate, Represents the updated learning rate; Among them, the exploration rate at the current moment is updated based on the exploration factor, including: ,in, Represents the cumulative reward at the current moment, represents the minimum historical reward, Indicates the largest reward in history, represents the exploration adjustment coefficient, represents the exploration factor, represents the initial exploration rate, represents the updated exploration factor.
2. The method according to claim 1, characterized in that Also includes: Calculate the prediction mean square error of the reinforcement learning network and count the number of parameters of the reinforcement learning network; The weighted sum of the prediction mean square error and the number of parameters is taken to obtain the complexity evaluation value of the reinforcement learning network; Dynamically increase or decrease the number of layers and / or neurons of the reinforcement learning network according to the complexity evaluation value.
3. The method according to claim 1, characterized in that in, By presetting the migration function and the migration distance, the reinforcement learning network parameters of the target working condition are adjusted, including: , in, represents the reinforcement learning network parameters of the adjusted target condition, represents the reinforcement learning network parameters of the target condition before adjustment, represents the migration distance, represents the first weight matrix, represents the second weight matrix, represents the reinforcement learning network parameters of the current working condition, represents the gradient of the loss function of the reinforcement learning network for the current working condition, represents random noise, Represents element-wise multiplication.
4. A system for optimizing parameters of an adaptive voltage regulator using the method according to any one of claims 1 to 3, characterized in that: include: Modeling unit, microscopic adaptive unit, mesoscopic adaptive unit, macroscopic adaptive unit and iterative updating unit; wherein, The modeling unit is used to initialize the decision network and the value network of the target reinforcement learning algorithm; sample the power system where the target voltage regulator is located to model the parameter optimization problem of the target voltage regulator as a Markov decision process; The micro-adaptive unit is used to perform micro-adaptation to dynamically adjust the learning rate and update the exploration strategy; including: obtaining the relative change rate of the cumulative rewards of the reinforcement learning network at the current moment and the previous moment to calculate the target attenuation factor; updating the learning rate at the current moment based on the attenuation factor; obtaining the difference between the current cumulative reward and the historical minimum reward, and the difference is used to construct the target exploration factor; updating the exploration rate at the current moment based on the exploration factor; The mesoscopic adaptation unit is used to perform mesoscopic adaptation to calculate parameter sensitivity and adjust the update step size; including: calculating the partial derivative of the performance index with respect to the target parameter, the partial derivative is used to construct the sensitivity of the target parameter; calculating the policy gradient and multiplying it with the current learning rate and sensitivity to obtain the update step size of the target parameter; The macro-adaptive unit is used to perform macro-adaptation to identify the working condition of the power system, and the result of the identification is used to perform model migration; including: using an unsupervised learning method to cluster the working conditions of the power system historical data, and calculating the migration distance of the current reinforcement learning network based on the clustering results; adjusting the reinforcement learning network parameters of the target working condition by presetting the migration function and the migration distance; The iterative updating unit is used to perform one or more adaptive iterative updates on the reinforcement learning network until the network converges or reaches a preset number of iterations; Among them, the learning rate at the current moment is updated based on the attenuation factor, including: ,in, Represents the cumulative reward at the current moment, Indicates the cumulative reward at the last moment, represents the relative rate of change of the cumulative reward, represents the attenuation adjustment coefficient, represents the attenuation factor, represents the initial learning rate, Represents the updated learning rate; Among them, the exploration rate at the current moment is updated based on the exploration factor, including: ,in, Represents the cumulative reward at the current moment, represents the minimum historical reward, Indicates the largest reward in history, represents the exploration adjustment coefficient, represents the exploration factor, represents the initial exploration rate, represents the updated exploration factor.
5. A device for optimizing parameters of an adaptive voltage regulator, characterized in that: include: A processor, a memory, and a system bus; wherein the processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 3.
