Transformer area micro-grid voltage control method and system based on improved deep reinforcement learning

Abstract

The invention provides a transformer area micro-grid voltage control method and system based on improved deep reinforcement learning, and the method comprises the following steps: collecting the internal source load power data of a transformer area micro-grid, and carrying out the preprocessing of the collected data, and obtaining the preprocessed internal source load power data; constructing a voltage management model of the district micro-grid in combination with line loss and potential risks of voltage out-of-limit in the electric energy transmission process; improving a Q-learning algorithm based on an epsilon-greedy decline strategy to obtain an improved Q-learning model, and training the improved Q-learning model in combination with the voltage management model; and obtaining a micro-grid voltage control result based on the trained and improved Q-learning model and the preprocessed internal source load power data. The method has remarkable advantages in the aspects of reducing voltage fluctuation, reducing network loss and the like, and is outstanding in effectiveness and practicability.

Description

Technical Field

[0001] The present invention relates to the technical field of microgrid voltage control, and more specifically, to a method and system for controlling voltage of a microgrid in an area based on improved deep reinforcement learning. Background Art

[0002] With the rapid progress of renewable energy power generation technology and power electronics technology, the modern new power system is characterized by a high proportion of new energy penetration and a high degree of power electronics. As the core component of this new power system, the substation microgrid plays a role in promoting the effective integration of a large number of new energy sources. However, with the increasing prevalence of high-proportion new energy power generation and diversified load access, substation microgrids face severe challenges in maintaining safe, efficient and high-quality operation. Specifically, the mismatch between new energy and loads and the bidirectional flow characteristics of power flow have significantly aggravated the severity of the voltage over-limit problem in the power grid system.

[0003] Traditional voltage regulation devices, such as on-load tap-changing transformers and shunt capacitor banks, are limited by their mechanical operation mechanisms and can only provide discontinuous and discrete control methods, making it difficult to achieve continuous and accurate voltage adjustment. In addition, these devices are subject to physical limitations on the number of operations, and their switching process takes a long time, and their operation strategies are mostly designed as offline preset modes. Given that renewable energy generation has significant random characteristics, voltage fluctuations are large and frequent, which places more stringent demands on the instant response capability of voltage regulation.

[0004] At present, the two mainstream strategies for the challenge of reactive power-voltage real-time regulation are classical mathematical optimization technology and deep reinforcement learning technology. Traditional mathematical optimization approaches aim to minimize voltage deviation and network losses by using techniques such as convex relaxation to solve the optimal power flow problem. However, such methods are highly dependent on detailed physical models and precise network parameters, while the perception capabilities of the power grid in reality are often insufficient and parameter information is often incomplete. In addition, with the expansion of the scale of the power grid, the complexity of the optimization problem has increased dramatically, and the time-consuming solution process has increased, limiting the feasibility of its online real-time application.

[0005] The existing public patent with application number CN118739313A discloses a bus voltage control method for a wind farm based on an on-load tap-changing transformer. The method combines OLTC with the transformer of the wind farm. When the node voltage on the higher-order side of the OLTC exceeds the limit, the voltage to be adjusted for a single wind turbine is calculated based on the consistency protocol, and the voltage control strategy is further implemented when the wind farm is connected to the grid. However, traditional voltage regulating devices, such as on-load tap-changing transformers and shunt capacitor banks, are limited by their mechanical operation mechanisms and can only provide non-continuous and discrete control means, making it difficult to achieve continuous and precise voltage adjustment. In addition, these devices are subject to physical limitations on the number of operations, and their switching process is time-consuming, and their operation strategies are mostly designed as offline preset modes. In view of the significant randomness of renewable energy power generation, the voltage fluctuations are large and frequent, which puts more stringent requirements on the instant response capability of voltage regulation.

[0006] The existing public patent with application number CN118659454A discloses an optimization scheduling method for a multi-source cooperative power system based on tight quadratic convex relaxation, which aims to minimize voltage deviation and network loss by using techniques such as convex relaxation to solve the optimal power flow problem. However, such methods are highly dependent on detailed physical models and precise network parameters, while the perception ability of the power grid in reality is often insufficient and parameter information is often incomplete. In addition, with the expansion of the scale of the power grid, the complexity of the optimization problem has increased dramatically, and the time-consuming solution process has increased, limiting the feasibility of its online real-time application.

[0007] In contrast, deep reinforcement learning, as a data-driven strategy that does not rely on precise models, demonstrates excellent adaptability and generalization capabilities. It can autonomously learn the control strategy of the intelligent agent from past interaction data, getting rid of the dependence on precise model parameters. More importantly, deep reinforcement learning allows the heavy training process to be moved to an offline environment, and in actual operation, only the fast forward calculation of the neural network needs to be performed, with a response time of milliseconds, ensuring extremely high real-time performance, which perfectly meets the rapid response requirements of reactive voltage control for online decision-making.

[0008] Although deep learning has shown remarkable results in power system applications, it faces challenges when its application scope is extended to large-scale distribution networks, and it is difficult to effectively achieve the dual goals of voltage stability and loss reduction. Especially when dealing with the complex and changeable environment of substation microgrids, traditional models are limited in processing power, resulting in unsatisfactory control effects. These traditional methods often only perform basic splicing of the state vectors of regional nodes, ignoring the deep extraction of key features, which increases the difficulty of strategy learning and fails to fully capture the timing characteristics of renewable energy generation and load changes. In addition, the existing methods have a slow convergence speed during training, which restricts the improvement of computing efficiency. Summary of the invention

[0009] In order to solve the deficiencies in the prior art, the present invention provides a method for controlling voltage of a microgrid in a substation based on improved deep reinforcement learning, which can solve the problem of inaccurate voltage control of a microgrid in a substation in the prior art.

[0010] The present invention adopts the following technical solution.

[0011] A voltage control method for a microgrid in a substation based on improved deep reinforcement learning comprises the following steps:

[0012] Collecting the internal source load power data of the substation microgrid, and preprocessing the collected data to obtain the preprocessed internal source load power data;

[0013] Considering the line loss during power transmission and the potential risk of voltage exceeding the limit, a voltage management model for the microgrid in the substation area is constructed.

[0014] An improved Q-learning model is obtained by improving the Q-learning algorithm based on the ε-greedy decreasing strategy, and the improved Q-learning model is trained in combination with the voltage management model.

[0015] Based on the trained improved Q-learning model and the preprocessed internal source load power data, the microgrid voltage control result is obtained.

[0016] Preferably, the preprocessing of the collected data specifically includes:

[0017] Identify and remove outliers in the data, check the integrity of the data, and record damaged and missing parts;

[0018] The Z-score standardization method is used to convert the internal source load power data into a unified format and dimension;

[0019] Design the RNN architecture: including setting and adjusting the number of network layers, the number of neurons in each layer, and the activation function parameters. The input layer receives the intrinsic source power data sequence; the hidden layer is used to capture the temporal dependencies in the data; the output layer predicts the intrinsic source power value of the current time step based on the output of the hidden layer.

[0020] Preferably, the voltage management model of the constructed substation microgrid specifically includes:

[0021] Establish the objective function of the voltage control model for the substation microgrid:

[0022]

[0023] Where N represents the total number of nodes in the system; C Loss P represents the unit network loss economic penalty coefficient;Loss，t It reflects the network loss of the microgrid in the substation at the tth moment; C a Indicates the economic penalty coefficient when the voltage deviates from the set value; ΔU i，t It represents the voltage deviation at the i-th node at time t.

[0024] Preferably, constraints of the voltage management model are established, including voltage amplitude constraints, new energy output constraints, photovoltaic system capacity and output power constraints, and active-reactive power flow balance constraints;

[0025] The voltage amplitude constraint is as follows:

[0026]

[0027] Where U i,t represents the voltage of the i-th node at time t, and They represent the minimum and maximum allowable values ​​of the voltage of the ith node at time t respectively;

[0028] The output constraints of new energy are as follows:

[0029]

[0030] Where P pv，t represents the active power output of the photovoltaic system at time t, and They represent the minimum active power output and maximum active power output of the photovoltaic system respectively; Q pv，t represents the active power output of the photovoltaic system at time t, and They represent the minimum active power output and maximum active power output of the PV system respectively;

[0031] The PV system capacity and output power constraints are as follows:

[0032]

[0033] In the formula, Indicates the square value of the rated capacity of the PV inverter;

[0034] The active-reactive power flow balance constraints are as follows:

[0035]

[0036] Where P G，i and Q G，i Respectively represent the active power and reactive power of the ith node in the system; P L，i and Q L，iRespectively represent the active load and reactive load of the i-th load node; U i and U j Represent the voltages of the i-th node and the j-th node respectively; n is the total number of nodes, G ij and B ij denote the conductance and susceptance between the i-th node and the j-th node respectively; δ ij Represents the voltage phase difference between the i-th and j-th nodes.

[0037] Preferably, the improved Q-learning model obtained by improving the Q-learning algorithm based on the ε-greedy decreasing strategy specifically includes:

[0038] Define the state space S t : State space S t Represents the state summary of all agents at any time point t. The state of each agent is determined by the active power demand and reactive power demand of the node connected to the PV inverter, the actual active output power of the PV inverter, and the voltage level of the corresponding node.

[0039] Construct collaborative action set A t :Collaborative action set A t It refers to the set of actions taken by all agents at a specific time t, which includes the control measures implemented by each agent by adjusting the reactive power output of its PV inverter;

[0040] Set the immediate feedback function R t :The negative value of the reactive power-voltage control objective function of the microgrid in the substation is set as the instant reward R of each decision step t ;

[0041] The ε-greedy decreasing strategy is used to guide the action selection of the agent, where ε is used as the adjustment parameter of the greedy factor: the agent first randomly selects actions from the action set according to the probability of ε, and the remaining probability of 1-ε is used to select the action with the highest expected reward; during the model training process, the ε value is dynamically adjusted, specifically, the ε value gradually decreases as the training process progresses;

[0042] Preferably, the greed factor ε is calculated as follows:

[0043]

[0044] In the formula, ε0 represents the initial value of the greed factor, k represents the current number of training times, and K tatal Indicates the maximum number of training times.

[0045] Preferably, the training of the improved Q-learning model in combination with the voltage management model specifically includes:

[0046] A training data set is constructed based on the preprocessed historical internal source power data of the substation microgrid and the corresponding dispatching strategy. The operation of the microgrid is simulated by the voltage management model to provide a training scenario for the improved Q-learning model. The improved Q-learning model is then trained to obtain the improved Q-learning model after training.

[0047] Preferably, the preprocessed internal source load power data is used as the input of the improved Q-learning model, and the microgrid voltage control result obtained by the improved Q-learning model is the optimal scheduling strategy.

[0048] The present invention also proposes a substation microgrid voltage control system based on improved deep reinforcement learning, which is used to implement the substation microgrid voltage control method based on improved deep reinforcement learning, including: a data acquisition module, a preprocessing module, a voltage management model construction module, an improved Q-learning model construction module, a training module and a control module;

[0049] The data acquisition module is used to collect the internal source load power data of the microgrid in the substation area.

[0050] The preprocessing module is used to preprocess the collected data to obtain the preprocessed internal source load power data;

[0051] The voltage management model building module is used to build a voltage management model for the microgrid in the substation area by combining the line loss in the power transmission process and the potential risk of voltage exceeding the limit;

[0052] The improved Q-learning model building module is used to improve the Q-learning algorithm based on the ε-greedy decreasing strategy to obtain an improved Q-learning model;

[0053] The training module is used to train the improved Q-learning model in combination with the voltage management model;

[0054] The control module obtains the microgrid voltage control result based on the trained improved Q-learning model and the preprocessed internal source load power data.

[0055] The present invention also provides a terminal, comprising a processor and a storage medium;

[0056] The storage medium is used to store instructions;

[0057] The processor is used to operate according to the instructions to execute the steps of the substation microgrid voltage control method based on improved deep reinforcement learning.

[0058] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the substation microgrid voltage control method based on improved deep reinforcement learning are implemented.

[0059] The beneficial effect of the present invention is that, compared with the prior art, the present invention proposes a new strategy for voltage control of the substation microgrid based on an optimized deep reinforcement learning framework. First, the recurrent neural network (RNN) model is used to accurately identify and repair anomalies and omissions in the source-load power data in the system to ensure data quality. Subsequently, the line loss risk and the possibility of voltage over-limit in the process of power transmission are comprehensively considered, and a refined voltage management model for the substation microgrid is constructed to provide a training environment and training guidance. In view of the complex nonlinear constraints contained in the voltage control model, the present invention introduces an improved Q-learning algorithm in the field of voltage control to efficiently solve various nonlinear constraints in the model, and guides the action selection of the intelligent agent through the ε-greedy decreasing strategy to overcome the limitations of the traditional method that the convergence speed is slow and easy to fall into the local optimal solution. The control strategy proposed by the present invention shows significant advantages in reducing voltage fluctuations and reducing network losses, thereby verifying its effectiveness and practicality. The present invention has at least the following technical effects:

[0060] 1. Improve voltage control accuracy: Through the training of the Q-learning algorithm, the intelligent agent can learn the strategy of selecting the optimal voltage adjustment action under different conditions, thereby improving the accuracy and stability of voltage control, while accelerating the convergence speed and improving the real-time computing efficiency of the algorithm.

[0061] 2. Enhance the adaptability of the power grid: Since the Q-learning algorithm has adaptive learning capabilities, the intelligent agent can adjust its strategy in real time according to the changes in the power grid operation status to ensure the timeliness and effectiveness of voltage control. At the same time, the ε-greedy decreasing strategy enables the intelligent agent to achieve a balance between exploration and utilization, further enhancing the adaptability of the power grid.

[0062] 3. Optimize energy utilization: Through precise voltage control, the output distribution of distributed power sources and the voltage level of loads can be optimized, thereby improving energy utilization efficiency. This helps to reduce energy waste and reduce operating costs.

[0063] 4. Improve grid safety: Accurate voltage control helps maintain the stability and safety of the grid. When a grid failure or abnormality occurs, the intelligent agent can respond quickly and adjust the voltage level to prevent the situation from deteriorating further. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of a voltage control method of a microgrid in an area based on improved deep reinforcement learning in the present invention;

[0065] Figure 2 It is a structural diagram of the substation microgrid voltage control system based on improved deep reinforcement learning in the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0067] like Figure 1 As shown, the present invention proposes a voltage control method for a microgrid in a substation based on improved deep reinforcement learning, comprising the following steps:

[0068] S1. Collecting internal source load power data of the microgrid, and preprocessing the collected data to obtain preprocessed internal source load power data;

[0069] The preprocessing uses a recurrent neural network (RNN) model to accurately identify and process the damaged and missing parts of the source-load power data in the system to ensure data quality. The preprocessing of the collected data specifically includes:

[0070] S101: Identify and remove abnormal values ​​in the data, such as power values ​​that are clearly beyond the physical range. Check the integrity of the data and record damaged and missing parts.

[0071] S102: Use the Z-score standardization method to convert the source and load power data into a unified format and dimension;

[0072] S103: Design a suitable RNN architecture (Gated Recurrent Unit GRU), including setting and adjusting the number of network layers to 5, the number of neurons in each layer, activation function and other parameters. The input layer receives the preprocessed source load power data sequence; the hidden layer contains multiple GRU units to capture the temporal dependencies in the data; the output layer predicts the source load power value of the current time step based on the output of the hidden layer.

[0073] S2. Comprehensively considering the line loss during power transmission and the potential risk of voltage exceeding the limit, a voltage management model for the microgrid in the substation area was constructed;

[0074] Specifically, the voltage management model can reflect the voltage distribution, line loss and voltage over-limit risk in the microgrid, and is the basis for the subsequent voltage control strategy design and optimization, including:

[0075] S201: Establish the objective function of the voltage control model for the substation microgrid. Within the operation framework of the substation microgrid, the real-time reactive voltage management strategy focuses on implementing more detailed fine-tuning within the pre-established daily scheduling plan framework by integrating and coordinating reactive compensation equipment.

[0076] The voltage quality is optimized through the voltage control model objective function for the substation microgrid to ensure stability and simultaneously reduce the energy loss during the operation of the power grid. The voltage control model objective function for the substation microgrid is shown in the following formula (1):

[0077]

[0078] In this expression, N represents the total number of nodes in the system; C loss P represents the unit network loss economic penalty coefficient; Loss，t It reflects the network loss of the microgrid in the substation at the tth moment; C a Indicates the economic penalty coefficient when the voltage deviates from the set value; ΔU i,t It represents the voltage deviation at the i-th node at time t, and T represents the total scheduling time.

[0079] S202: Establish model constraints. The constraints that need to be considered in this model include:

[0080] (1) Voltage amplitude constraint

[0081]

[0082] In the above formula, and They respectively represent the minimum and maximum allowable values ​​of the voltage of the i-th node at time t.

[0083] (2) Constraints on new energy output

[0084] In this paper, the new energy type considered in the substation microgrid is mainly small-capacity distributed photovoltaic. The photovoltaic output should meet the following constraints:

[0085]

[0086] In the above formula, and They represent the minimum active power output and maximum active power output of the PV system respectively; and They represent the minimum reactive power output and maximum reactive power output of the PV system respectively.

[0087] (3) PV system capacity and output power constraints

[0088]

[0089] In the above formula, Indicates the square value of the rated capacity of the PV inverter.

[0090] (4) Active-reactive power flow balance constraints

[0091]

[0092] Where P G，i and Q G，i Respectively represent the active power and reactive power of the ith node in the system; P L，i and Q L，i Respectively represent the active load and reactive load of the i-th load node; G ij and B ij denote the conductance and susceptance between the i-th node and the j-th node respectively; δ ij It shows the voltage phase difference between the i-th and j-th nodes.

[0093] S3. In view of the complex nonlinear constraints contained in the model, the Q-learning algorithm is improved based on the ε-greedy decreasing strategy to obtain an improved Q-learning model, and the improved Q-learning model is trained in combination with the voltage management model to overcome the limitations of the traditional method.

[0094] S301: Define the state space S t :S t It represents the state summary of all agents at any time point t. The state of each agent is determined by the active power demand and reactive power demand of the node connected to the PV inverter, the actual active output power of the PV inverter, and the voltage level of the corresponding node.

[0095] Specifically, the agent decides how to adjust the controllable devices in the microgrid (such as the output of distributed power sources, switching of reactive power compensation devices, etc.) based on the current system state (such as voltage level, power flow, etc.) and the predetermined control objectives (such as minimizing voltage fluctuations and reducing network losses). t Include a comprehensive or key description of the current microgrid in the substation. In the voltage control problem, the state may include but is not limited to: the voltage level of each node, the real-time power of the power source and load, the line current and power factor, the estimation of network loss, the state and charging and discharging capacity of the energy storage device, and other variables that may affect the voltage quality and network loss.

[0096] State information can be obtained through sensor measurements, data acquisition systems or prediction models and input into the RNN model for preprocessing to ensure data quality and integrity. This state information is used to guide the decision-making process of the intelligent agent.

[0097] The state space St reflects the voltage and line loss of the microgrid in the substation at the current moment, and is the basis for the decision-making of the Q-learning algorithm. In order to obtain the state space St in the present invention, the following information can be extracted from the voltage management model:

[0098] Voltage distribution: This can be obtained by monitoring the voltage values ​​of each node in the substation microgrid. Voltage management models usually consider factors such as voltage fluctuation range and voltage stability, and integrate this information into the state representation;

[0099] Line loss: Line loss is an important indicator in the process of power transmission. It reflects the efficiency and loss level of the power grid. The voltage management model will calculate the loss value of each line in the substation microgrid and use it as part of the status.

[0100] In actual operation, the voltage and line loss values ​​can be quantified, such as dividing the voltage value into different intervals and normalizing the line loss value, so as to obtain a specific state space St.

[0101] S302: Construct a collaborative action set At: At refers to the set of actions taken by all agents at a specific time t.

[0102] The actions taken by the agents are as follows: each agent implements control measures by adjusting the reactive power output of its PV inverter.

[0103] Further preferably, the voltage management model constructed by S2 in the present invention can clearly define the state of the microgrid (such as voltage distribution, line loss, etc.) and possible actions (such as adjusting transformer taps, switching capacitors, etc.), thereby providing a clear learning goal for the Q-learning model.

[0104] Collaborative Action Set A t Including the voltage control strategies that the microgrid in the substation can adopt in the current state. According to the objective function of the voltage management model, the following actions can be taken to adjust the voltage and reduce line losses:

[0105] Adjusting transformer taps: By changing the position of the transformer taps, the voltage level of the grid can be adjusted, thereby improving voltage distribution and reducing line losses.

[0106] Switching capacitors: Capacitors can compensate for reactive power in the grid, thereby improving voltage quality and reducing line losses. According to the instructions of the voltage management model, capacitors can be switched to adjust the reactive power balance of the grid.

[0107] In actual operation, these actions can be encoded, such as using numbers or strings to represent different action options, and used as a set of collaborative actions for the Q-learning model.

[0108] S303: Setting the instant feedback reward Rt: In order to guide the decision-making process, the negative value of the reactive power-voltage control objective function of the microgrid in the substation is set as the instant feedback reward Rt of each decision step.

[0109] This design aims to optimize the control effect by indirectly minimizing the original objective function by maximizing the cumulative reward.

[0110] In the Q-learning model, the reward function is used to evaluate the quality of the actions taken by the agent. The voltage management model obtained through S2 can calculate the corresponding reward value according to the operation of the microgrid, thereby guiding the Q-learning model to learn in the direction of the optimal voltage control strategy.

[0111] The instant feedback reward Rt reflects the effect of the action taken by the microgrid in the current state. According to the objective function of the voltage management model, the present invention sets the following reward mechanism to evaluate the quality of the action:

[0112] Voltage stability reward: If the action can improve the stability of the voltage, such as keeping the voltage value within the allowed range, a positive reward is given.

[0113] Line loss reduction reward: If the action can reduce line loss, such as reducing the loss value to below the target level, a positive reward is given.

[0114] Penalty mechanism: If the action causes unfavorable situations such as voltage exceeding the limit or increased line loss, a negative reward will be given as a penalty.

[0115] S304: In order to accelerate the convergence of the optimization process, a balance mechanism of exploration and utilization is incorporated. The ε-greedy decreasing strategy is adopted to guide the action selection of the agent, where ε is used as the adjustment parameter of the greedy factor. When making decisions, the agent first randomly selects actions from the action set according to the probability of ε and conducts extensive exploration; and the remaining probability of 1-ε is used to select the action with the highest expected reward. In the early stage of training, by setting a higher ε value, the agent can try new actions more frequently and explore possible better solutions in order to obtain higher cumulative rewards in the long run. As the training process progresses, the ε value gradually decreases, and the agent gradually tends to choose the currently known optimal action, effectively utilizing the accumulated experience and knowledge. The specific mathematical representation of ε has been given in detail in the formula. This strategy promotes the efficient and stable convergence of the algorithm by dynamically adjusting the ratio of exploration and utilization.

[0116]

[0117] In the formula, ε0 represents the initial value of the greed factor, k represents the current number of training times, and K tatal Indicates the maximum number of training times.

[0118] S305, based on the pre-processed historical internal source load power data of the substation microgrid and the corresponding dispatching strategy, the voltage management model obtained by S2 simulates the operation of the microgrid, provides a training scenario for the improved Q-learning model, and then trains the improved Q-learning model to obtain the improved Q-learning model after training.

[0119] S4: Based on the trained improved Q-learning model and the preprocessed internal source load power data, the microgrid voltage control result is obtained;

[0120] Specifically, the preprocessed internal source load power data is used as the input of the improved Q-learning model, and the output result obtained by the improved Q-learning model is the optimal scheduling strategy.

[0121] like Figure 2 As shown, the present invention also proposes a substation microgrid voltage control system based on improved deep reinforcement learning, which is used to implement the above-mentioned substation microgrid voltage control method based on improved deep reinforcement learning. The system includes: a data acquisition module, a preprocessing module, a voltage management model construction module, an improved Q-learning model construction module, a training module and a control module;

[0122] Among them, the data acquisition module is used to collect the internal source load power data of the microgrid in the substation area.

[0123] The preprocessing module is used to preprocess the collected data to obtain the preprocessed internal source load power data;

[0124] The voltage management model building module is used to build a voltage management model for the microgrid in the substation area by combining the line loss in the power transmission process and the potential risk of voltage exceeding the limit;

[0125] The improved Q-learning model building module is used to improve the Q-learning algorithm based on the ε-greedy decreasing strategy to obtain an improved Q-learning model;

[0126] The training module is used to train the improved Q-learning model in combination with the voltage management model;

[0127] The control module obtains the microgrid voltage control result based on the trained improved Q-learning model and the preprocessed internal source load power data.

[0128] The beneficial effect of the present invention lies in that, compared with the prior art, the present invention introduces an improved Q-learning algorithm in the field of voltage control to efficiently solve various nonlinear constraints in the model, and guides the action selection of the intelligent agent through the ε-greedy decreasing strategy to overcome the limitations of the traditional method of slow convergence speed and easy to fall into the local optimal solution, and combines the voltage management model to provide a training environment and parameter guidance for the improved Q-learning model. The control method proposed in the present invention shows significant advantages in reducing voltage fluctuations and reducing network losses, thereby verifying its effectiveness and practicality.

[0129] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0130] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive lists) of computer readable storage medium include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or convex structures in grooves on which instructions are stored, and any suitable combination thereof. Computer readable storage medium used here is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses by optical fiber cables), or electrical signals transmitted by wires.

[0131] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0132] The computer program instructions for performing the disclosed operation may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A voltage control method for a microgrid in an area based on improved deep reinforcement learning, characterized in that: The steps include: Collecting the internal source load power data of the substation microgrid, and preprocessing the collected data to obtain the preprocessed internal source load power data; Considering the line loss during power transmission and the potential risk of voltage exceeding the limit, a voltage management model for the microgrid in the substation area is constructed. An improved Q-learning model is obtained by improving the Q-learning algorithm based on the ε-greedy decreasing strategy, and the improved Q-learning model is trained in combination with the voltage management model. Based on the trained improved Q-learning model and the preprocessed internal source load power data, the microgrid voltage control result is obtained.

2. The method for controlling voltage of a microgrid in an area based on improved deep reinforcement learning according to claim 1, characterized in that: The preprocessing of the collected data specifically includes: Identify and remove outliers in the data, check the integrity of the data, and record damaged and missing parts; The Z-score standardization method is used to convert the internal source load power data into a unified format and dimension; Design the RNN architecture: including setting and adjusting the number of network layers, the number of neurons in each layer, and the activation function parameters. The input layer receives the intrinsic source power data sequence; the hidden layer is used to capture the temporal dependencies in the data; the output layer predicts the intrinsic source power value of the current time step based on the output of the hidden layer.

3. The method for controlling voltage of a microgrid in an area based on improved deep reinforcement learning according to claim 1, characterized in that: The voltage management model of the microgrid in the substation area specifically includes: Establish the objective function of the voltage control model for the substation microgrid: Where N represents the total number of nodes in the system; C Loss P represents the unit network loss economic penalty coefficient; Loss，t It reflects the network loss of the microgrid in the substation at the tth moment; C a Indicates the economic penalty coefficient when the voltage deviates from the set value; ΔU i，t It represents the voltage deviation at the i-th node at time t.

4. The method for controlling voltage of a microgrid in an area based on improved deep reinforcement learning according to claim 1, characterized in that: Establish constraints for the voltage management model, including voltage amplitude constraints, new energy output constraints, photovoltaic system capacity and output power constraints, and active-reactive power flow balance constraints; The voltage amplitude constraint is as follows: Where U i，t represents the voltage of the ith node at time t, and They represent the minimum and maximum allowable values ​​of the voltage of the ith node at time t respectively; The output constraints of new energy are as follows: Where P pv，t represents the active power output of the photovoltaic system at time t, and They represent the minimum active power output and maximum active power output of the photovoltaic system respectively; Q pv，t represents the active power output of the photovoltaic system at time t, and They represent the minimum active power output and maximum active power output of the PV system respectively; The PV system capacity and output power constraints are as follows: In the formula, Indicates the square value of the rated capacity of the PV inverter; The active-reactive power flow balance constraints are as follows: Where P G，i and Q G，i Respectively represent the active power and reactive power of the ith node in the system; P L，i and Q L，i Respectively represent the active load and reactive load of the i-th load node; U i and U j Represent the voltages of the i-th node and the j-th node respectively; n is the total number of nodes, G ij and B ij denote the conductance and susceptance between the i-th node and the j-th node respectively; δ ij Represents the voltage phase difference between the i-th and j-th nodes.

5. The method for controlling voltage of a microgrid in an area based on improved deep reinforcement learning according to claim 1, characterized in that: The improved Q-learning model based on the ε-greedy decreasing strategy to improve the Q-learning algorithm specifically includes: Define the state space S t : State space S t Represents the state summary of all agents at any time point t. The state of each agent is determined by the active power demand and reactive power demand of the node connected to the PV inverter, the actual active output power of the PV inverter, and the voltage level of the corresponding node. Construct collaborative action set A t :Collaborative action set A t It refers to the set of actions taken by all agents at a specific time t, which includes the control measures implemented by each agent by adjusting the reactive power output of its PV inverter; Set the immediate feedback function R t :The negative value of the reactive power-voltage control objective function of the microgrid in the substation is set as the instant reward R of each decision step t ; The ε-greedy decreasing strategy is used to guide the action selection of the agent, where ε is used as the adjustment parameter of the greedy factor: the agent first randomly selects actions from the action set according to the probability of ε, and the remaining probability of 1-ε is used to select the action with the highest expected reward; during the model training process, the ε value is dynamically adjusted, specifically, the ε value gradually decreases as the training process progresses.

6. The method for controlling voltage of a microgrid in an area based on improved deep reinforcement learning according to claim 1, characterized in that: The calculation formula of greed factor ε is as follows: In the formula, ε0 represents the initial value of the greed factor, k represents the current number of training times, and K tatal Indicates the maximum number of training times.

7. The method for controlling voltage of a microgrid in an area based on improved deep reinforcement learning according to claim 1, characterized in that: The training of the improved Q-learning model in combination with the voltage management model specifically includes: A training data set is constructed based on the preprocessed historical internal source power data of the substation microgrid and the corresponding dispatching strategy. The operation of the microgrid is simulated by the voltage management model to provide a training scenario for the improved Q-learning model. The improved Q-learning model is then trained to obtain the improved Q-learning model after training.

8. The method for controlling voltage of a microgrid in a substation based on improved deep reinforcement learning according to claim 1, characterized in that: The preprocessed internal source load power data is used as the input of the improved Q-learning model, and the microgrid voltage control result obtained by the improved Q-learning model is the optimal scheduling strategy.

9. A substation microgrid voltage control system based on improved deep reinforcement learning, used to implement the substation microgrid voltage control method based on improved deep reinforcement learning as described in any one of claims 1 to 8, characterized in that: include: Data acquisition module, preprocessing module, voltage management model building module, improved Q-learning model building module, training module and control module; The data acquisition module is used to collect the internal source load power data of the microgrid in the substation area. The preprocessing module is used to preprocess the collected data to obtain the preprocessed internal source load power data; The voltage management model building module is used to build a voltage management model for the microgrid in the substation area by combining the line loss in the power transmission process and the potential risk of voltage exceeding the limit; The improved Q-learning model building module is used to improve the Q-learning algorithm based on the ε-greedy decreasing strategy to obtain an improved Q-learing model; The training module is used to train the improved Q-learning model in combination with the voltage management model; The control module obtains the microgrid voltage control result based on the trained improved Q-learning model and the preprocessed internal source load power data.

10. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images