A deep reinforcement learning power distribution network voltage control method and system based on large model progressive guidance

Abstract

The application provides a deep reinforcement learning power distribution network voltage control method and system based on large model progressive guidance, which comprises the following steps: S1, establishing a voltage optimization model, including constructing a double-target optimization function and constraint conditions; S2, adopting a Markov decision process to construct a power distribution network voltage control model, wherein the Markov decision process comprises a state space, an action space, a reward function design and a state transition; S3, adopting a large model guidance module to generate and optimize the reward mechanism of the power distribution network voltage control model, and then adopting a reinforcement learning execution module based on a TD3 algorithm to train the power distribution network voltage control model; and a deployable power distribution network voltage online regulation and control model is obtained. The application further comprises a voltage optimization model module, a Markov decision process module, a large model guidance module, a reinforcement learning execution module and a training module. The application finally establishes a deployable power distribution network voltage online regulation and control model.

Description

Technical Field

[0001] This invention proposes a deep reinforcement learning-based distribution network voltage control method and system based on a large model and progressive guidance, which relates to the fields of power systems and deep reinforcement learning technology. Background Technology

[0002] With the high proportion of distributed energy resources integrated into distribution networks, system operation exhibits significant randomness and volatility, leading to increasingly prominent problems such as increased voltage deviation at distribution network nodes and rising line losses, severely impacting power quality and system operational economy. Traditional voltage / reactive power optimization methods largely rely on analytical models and empirical rules, making them difficult to adapt to complex operating conditions. Under complex coupled operating conditions, both optimization accuracy and robustness are limited. In recent years, deep reinforcement learning has gradually become an effective means to solve such problems, enabling adaptive control decisions in complex environments and providing a new technical path for distribution network voltage optimization.

[0003] Currently, deep reinforcement learning-based methods for optimizing voltage deviation and network loss in distribution networks typically achieve adaptive control of reactive power regulation equipment by constructing an interaction model between the agent and the power grid environment, achieving some success in complex operating scenarios. However, existing reward functions are usually constructed manually, which is not only complex in design but also difficult to balance multiple objectives such as voltage quality, network loss level, and equipment operating costs, resulting in slow convergence speed, poor generalization ability, and insufficient adaptability. Therefore, it is necessary to develop an intelligent voltage control method that can dynamically adjust the optimization objective based on the training state and continuously guide the evolution of the reinforcement learning strategy. Summary of the Invention

[0004] In view of this, in order to fill the gaps and deficiencies in the existing technology, this invention proposes a deep reinforcement learning-based distribution network voltage control method and system based on large model progressive guidance, which is used to solve the problems in the background technology.

[0005] This invention aims to minimize the voltage deviation at all network nodes and the line loss. It selects photovoltaic grid-connected inverters and static var compensators (SVCs) as control devices, and comprehensively considers power flow balance constraints, voltage safety constraints, and equipment operation constraints to establish a mathematical model for distribution network voltage optimization. Subsequently, the optimization model is transformed into a Markov decision process, a reinforcement learning training environment is constructed, and the distribution network operating state, equipment control actions, reward functions, and state transition mechanisms are defined. Furthermore, this invention constructs a dual-closed-loop collaborative optimization architecture consisting of a large language model guidance layer and a reinforcement learning execution layer. The reinforcement learning execution layer is responsible for interacting with the distribution network environment and learning the optimal control strategy, while the large language model guidance layer is responsible for dynamically generating and optimizing the reward function based on training feedback information. During training, the large language model receives operating indicators such as voltage qualification rate, line loss, equipment action frequency, and strategy convergence status generated by the reinforcement learning agent. Through semantic reasoning and knowledge analysis, it automatically identifies problems existing in the current training stage and generates new reward functions or adjusts existing reward function parameters in conjunction with preset optimization objectives, thereby continuously guiding the reinforcement learning strategy towards a better direction. Meanwhile, this invention introduces a progressive guidance mechanism, dynamically adjusting the optimization objectives based on the agent's training level. This allows the agent to gradually transition from voltage safety control to network loss optimization, equipment action optimization, and comprehensive operation optimization, achieving phased learning for multi-objective optimization tasks. The reinforcement learning part uses an Actor-Critic structure to construct the control model, optimizing network parameters through experience replay and target network update mechanisms to obtain the optimal control strategy suitable for complex operating scenarios. After offline training, the trained control strategy is deployed in the actual distribution network operation system, receiving real-time distribution network operating status information and outputting reactive power regulation control commands to achieve online optimization control of the distribution network voltage.

[0006] This invention proposes a deep reinforcement learning-based voltage control method and system for distribution networks, guided by a large model and progressive approach, comprising the following:

[0007] According to a first aspect of the present invention, a deep reinforcement learning-based distribution network voltage control method guided by a large model is proposed, characterized by comprising the following:

[0008] Step S1: Establish a voltage optimization model, including constructing a dual-objective optimization function and constraints;

[0009] Step S2: Construct a distribution network voltage control model using a Markov decision process, which includes state space, action space, reward function design, and state transition.

[0010] Step S3: The reward mechanism of the distribution network voltage control model is generated and optimized by the large model guidance module, and then the distribution network voltage control model is trained by the reinforcement learning execution module based on the TD3 algorithm; thus, a deployable online voltage regulation model for the distribution network is obtained.

[0011] Further, step S1 includes the following:

[0012] Step S11: Construct a dual-objective optimization function, which includes node voltage deviation and total power loss of the distribution network, and includes the following:

[0013] ;

[0014] Where J is the comprehensive optimization target value; T is the optimization period set; and I is the distribution network node set.

[0015] The actual voltage amplitude of node i at time t; This refers to the rated voltage amplitude of the node. λ is the network loss penalty coefficient, used to balance the optimization weights of voltage deviation and network loss;

[0016] Let t be the total active power loss of the distribution network lines, which includes the power loss of the photovoltaic grid-connected inverter and SVG equipment.

[0017] Furthermore, step S1 also includes the following:

[0018] Step S12: Construct constraints, which include three-phase power flow constraints, voltage constraints, and equipment operation constraints;

[0019] The three-phase power flow constraints include: for any node i, its three-phase active and reactive power satisfy:

[0020] ;

[0021] ;

[0022] in, , These are the injected active and reactive power at node i, including the output of the photovoltaic grid-connected inverter and the compensation amount of the SVC device.

[0023] in Let i be the set of neighboring nodes of node i; , These are the line conductance and susceptance between nodes i and j, respectively; Let be the voltage phase angle difference between nodes i and j;

[0024] The voltage constraints include:

[0025] The voltage of each phase at all nodes must be controlled within the acceptable range. The constraint expression is as follows:

[0026] ;

[0027] in, and Indicates the lower and upper limits of the node voltage;

[0028] The equipment operation constraints include photovoltaic grid-connected inverter constraints and SVG constraints;

[0029] The constraint expression for the photovoltaic grid-connected inverter is:

[0030] ;

[0031] in: The reactive power output of the photovoltaic grid-connected inverter; , These are the upper and lower limits of its reactive power output, calculated from the equipment's rated capacity and real-time active power output.

[0032] The constraint expression for SVG is:

[0033] ;

[0034] in: This refers to the reactive power output of the SVG. Its rated reactive capacity allows the output value to vary continuously between the upper and lower limits.

[0035] Further, step S2 includes the following:

[0036] Step S21: Construct the state space, including the following:

[0037] Define the state space S as a continuous real-valued vector of fixed dimension to reflect the current operating state of the distribution network;

[0038] The vector is composed of electrical quantities including the voltage amplitude of all nodes in the distribution network, the active and reactive power injected into the nodes, the real-time active and reactive power output of the photovoltaic inverter, and the reactive power compensation of the SVG equipment.

[0039] The expression for the state space is:

[0040] ;

[0041] For nodes that are not equipped with photovoltaic or SVG devices, Set to 0; for single-phase or two-phase nodes, the voltage and power of the corresponding phase are 0;

[0042] Step S22: Construct the action space, including the following:

[0043] Define an action space A, which includes a set of continuous control instructions output by the agent, including the reactive power output adjustment of the photovoltaic inverter and the SVG device.

[0044] The expression for the action space is:

[0045] ;

[0046] Where: m is the number of grid-connected photovoltaic inverters; l is the number of SVG devices;

[0047] Q PVm Q SVGl These are the reactive power outputs of the m-th grid-connected photovoltaic inverter and the l-th SVG device, respectively.

[0048] Furthermore, step S2 also includes the following:

[0049] Step S23: Define a reward function to quantify the immediate effect of an agent performing a specific action in a given state, providing a direct optimization direction for policy learning; the expression for the reward function is:

[0050] ;

[0051] in: These are, respectively, voltage penalty weight, action penalty weight, and network loss penalty coefficient;

[0052] The voltage penalty weight is used to adjust the priority of voltage exceeding the limit in the overall reward. The larger the value, the more the agent tends to pull the voltage back to the safe range. The action penalty weight is used to control the importance of the adjustment range of reactive power equipment. The larger the value, the more the equipment is encouraged to maintain stable output and reduce frequent adjustments. The network loss penalty coefficient is used to balance the weight between economic operation and voltage safety.

[0053] in This is a voltage over-limit penalty item, used to quantify the degree to which the voltage of each node deviates from the safe operating range; This is a penalty item for equipment operation costs, imposing a penalty on the magnitude of reactive power output adjustment;

[0054] in The network loss penalty item uses the total active power loss of the distribution network as a measure of the degree of economic operation.

[0055] Step S23: Define state transitions to express the dynamic response of the distribution network after an action is applied, including the probability distribution of transitioning from the current state to the state at the next moment.

[0056] The expression for the state transition is:

[0057] ;

[0058] Among the actions Including reactive power control commands from photovoltaic inverters and SVG devices, state transitions also represent the dynamic evolution logic of the grid state under the control actions of photovoltaic inverters and SVG devices.

[0059] Further, step S3 includes the following:

[0060] Step S31: Use the large model guidance module to generate and optimize the reward mechanism of the distribution network voltage control model. This includes the large model guidance module periodically receiving statistical data including the voltage over-limit situation of the entire network, the active power loss level of the line, and the frequency of equipment operation; analyzing the deficiencies of the current reward mechanism, generating or revising the reward mechanism including the reward function code, and re-outputting it to the execution layer of the distribution network voltage control model.

[0061] The system employs a hierarchical prompting strategy to obtain prompts, which serve as input to the large model guidance module. This strategy constrains the inference process of the large model within the professional framework of the power system, shielding it from interference from other industries. The strategy also includes static and dynamic components. The static component defines the physical rules that the large model must follow, while the dynamic component updates the training process and injects contextual information into the large model for revision.

[0062] The design incorporates a progressive guidance mechanism based on curriculum learning principles for training large-scale models. The training process includes a pre-stage reward mechanism with voltage safety as the objective. Once the large-scale model passes performance threshold verification, the weight of the economic objective in the reward mechanism is increased, enabling the large-scale model to reduce network losses while maintaining voltage safety. The output includes a comprehensive control strategy that considers multiple objectives and a reward mechanism.

[0063] Furthermore, step S3 also includes the following:

[0064] Step S32: The reinforcement learning execution module is based on the TD3 algorithm and includes the structural design of the Acto network and Critic network, as well as the complete training and execution process;

[0065] The Actor network is used to map the current operating state of the distribution network to the action commands of each reactive power control device, including the following:

[0066] The Actor network adopts a fully connected feedforward structure. The input layer receives a state vector defined by the state space, which includes the voltage amplitude of all network nodes, injected active and reactive power, and equipment state information. The intermediate layer sets up multiple fully connected hidden layers and uses a modified linear unit activation function to extract regulation-related information from the state features layer by layer. The output layer uses a hyperbolic tangent activation function to map the output value to a normalized interval, and then scales it to the actual reactive power output range by multiplying it by the maximum regulation capacity of each device, and outputs the action vector.

[0067] The Critic network is used to evaluate the value of performing a specific action in the current state, including the following:

[0068] A dual-Critic structure is adopted to suppress the Q-value overestimation problem. Each Critic network simultaneously receives a state vector and an action vector as input. The state and action are processed by independent feature mapping layers and then concatenated into a joint feature vector, which is then mapped to a scalar Q-value output through multiple fully connected layers. The two Critic networks have the same structure but different initialization parameters and output two Q-values ​​respectively. When calculating the target value, the smaller of the two values ​​is taken to suppress the overestimation bias.

[0069] Furthermore, step S3 also includes the following:

[0070] Step S33: Design the TD3 algorithm mechanism, including the following:

[0071] The TD3 algorithm mechanism is based on the gradient of a deep deterministic policy, calculates the long-term cumulative discount reward of the target model, and obtains the control policy with the optimal expected return of the target model.

[0072] Step S331: Set up a dual network, including the following:

[0073] The TD3 algorithm mechanism adds an isomorphic evaluation network to the single Critic structure of the degree-deterministic policy gradient, forming a dual Critic architecture. During the update, the two target Critic networks are used to calculate the action value of the next state, and the smaller of the two values ​​is taken as the basis for the target value y.

[0074] The optimization objective of the TD3 algorithm mechanism is to minimize the Bellman error of the Critic network, including the following:

[0075] ;

[0076] Where E is the expected value calculated from the batch samples in the experience return pool D; This is a discount factor used to weigh current rewards against future rewards; In time The reward;

[0077] in and These are dual Critic networks; Delaying the update of the target Actor policy; target network regularization

[0078] Step S332: Target network regularization, including the following:

[0079] The TD3 algorithm mechanism adds a small amount of truncated Gaussian noise to the action output by the target policy when calculating the target action, so that the target value is averaged within a small range near the target action.

[0080] Among them, target action The calculation method is as follows:

[0081] ;

[0082] The gradient descent expressions for the two Critics are updated as follows:

[0083] ;

[0084] in: The target Actor strategy for delayed updates; For a mean of 0 and a standard deviation of Sampling and truncation in a Gaussian distribution Random noise within the range;

[0085] Step S333: Use delayed update, including the following:

[0086] The TD3 algorithm mechanism delays the timing of Actor updates, including updating the parameters of the Actor network only after the Critic network has undergone multiple iterations and the value estimation has become relatively stable.

[0087] The Actor network is based on the state. Output Action And the gradient optimization strategy parameters are estimated using Critic. This includes the following:

[0088] ;

[0089] in: Regarding strategy parameters The gradient of the strategy performance function J; In strategy State distribution under The expected value of state S in the middle;

[0090] in The Q-function (value function) with respect to action The gradient is evaluated in the current state. and actions Below, we examine how subtle changes in movement affect expected returns.

[0091] in For policy function The action selected in state S;

[0092] Step S334: The TD3 algorithm mechanism achieves training stability by performing soft updates on the target network parameters.

[0093] ;

[0094] in: This is the soft update coefficient, which is usually taken as a small value to ensure that the target network parameters change smoothly and to provide a stable optimization target for the training process.

[0095] Furthermore, step S3 also includes the following:

[0096] Step S34: The TD3 algorithm mechanism adopts an offline centralized training and online real-time execution mode. It completes policy learning through the interaction between the agent and the distribution network simulation environment. After training, the network parameters are fixed for online control decision-making, resulting in a deployable distribution network voltage online control model. The training mode of the TD3 algorithm machine includes initialization configuration, initialization of training rounds (setting the number of training rounds), initialization of the distribution network simulation environment (initializing the distribution network simulation environment), action generation and exploration to enhance the Actor network, environment interaction and feedback to obtain the distribution network simulation environment to execute actions, storage of experience samples (storing complete experience samples into the ordinary experience replay pool), batch sample extraction (randomly and uniformly extracting batch samples from the ordinary experience replay pool), updating the target network parameters of the Actor and Critic using a soft update mechanism, and round end and performance evaluation feedback.

[0097] According to a second aspect of the present invention, the present invention proposes a deep reinforcement learning distribution network voltage control system based on large-model progressive guidance, for executing a deep reinforcement learning distribution network voltage control method based on large-model progressive guidance as described in any one of the present invention, characterized in that it includes the following:

[0098] The voltage optimization model module is used to construct voltage optimization models that include dual objective optimization functions and constraints.

[0099] The Markov Decision Process module uses Markov decision processes to construct a voltage control model for the distribution network. The Markov decision process includes state space, action space, reward function design, and state transition.

[0100] The large model guidance module is used to generate and optimize the reward mechanism of the distribution network voltage control model. It adopts a hierarchical prompt word strategy to obtain prompt words, which serve as the input content of the large model guidance module. It introduces the concept of course learning to design a progressive guidance mechanism for training the large model. The output content includes a comprehensive control strategy and reward mechanism that takes into account multiple objectives.

[0101] The reinforcement learning execution module is based on the TD3 algorithm and includes the structural design of the Acto network and Critic network, as well as the complete training and execution process.

[0102] The training module, based on the TD3 algorithm mechanism, adopts an offline centralized training and online real-time execution mode. It completes policy learning through the interaction between the agent and the distribution network simulation environment. After training, the network parameters are fixed for online control decisions, resulting in a deployable online voltage control model for the distribution network.

[0103] The present invention has the following advantages:

[0104] This invention proposes a deep reinforcement learning-based distribution network voltage optimization method based on a large-model progressive guidance. This method organically integrates the knowledge reasoning capabilities of a large language model with the data-driven decision-making capabilities of reinforcement learning, breaking through the dependence of traditional reinforcement learning methods on manually designed reward functions and achieving dynamic generation and adaptive optimization of the reward function. By constructing a dual-closed-loop training architecture consisting of a large-model guidance layer and a reinforcement learning execution layer, the large language model can continuously optimize the reward mechanism based on operational feedback during training, thereby effectively reducing ineffective exploration and improving the training efficiency and convergence speed of the reinforcement learning model. Simultaneously, the proposed progressive guidance mechanism can gradually adjust the optimization objective according to the agent's capability level, guiding the agent to learn multi-objective collaborative optimization strategies from a single voltage control task. This avoids training instability caused by the simultaneous introduction of complex optimization objectives, improving the stability and generalization ability of the control strategy. Compared with existing technologies, this invention can better adapt to complex operating scenarios such as distributed photovoltaic power output fluctuations and random load changes. While ensuring that the node voltage meets the operating requirements, it effectively reduces line network losses and equipment regulation frequency, improves the safety, economy and intelligence level of distribution network operation, and provides an efficient and reliable technical solution for distribution network voltage optimization control under the condition of high proportion of new energy access in new power systems. Attached Figure Description

[0105] Figure 1 This is a schematic diagram of the steps of the present invention.

[0106] Figure 2 This is a flowchart illustrating an embodiment of the present invention.

[0107] Figure 3 This is a schematic diagram of the closed-loop structure of reinforcement learning guided by the large model of the present invention.

[0108] Figure 4 This is a schematic diagram of the prompt words for this invention.

[0109] Figure 5 This is a schematic diagram of the output reward function of the present invention. Detailed Implementation

[0110] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0111] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0112] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0113] like Figures 1 to 5 As shown, this invention proposes a deep reinforcement learning-based distribution network voltage control method and system guided by a large model, characterized by the following:

[0114] According to a first aspect of the present invention, a deep reinforcement learning-based distribution network voltage control method guided by a large model is proposed, characterized by comprising the following:

[0115] Step S1: Establish a voltage optimization model, including constructing a dual-objective optimization function and constraints;

[0116] Step S2: Construct a distribution network voltage control model using a Markov decision process, which includes state space, action space, reward function design, and state transition.

[0117] Step S3: The reward mechanism of the distribution network voltage control model is generated and optimized by the large model guidance module, and then the distribution network voltage control model is trained by the reinforcement learning execution module based on the TD3 algorithm; thus, a deployable online voltage regulation model for the distribution network is obtained.

[0118] Corresponding to step S1, in one embodiment of the present invention, step S1 further includes the following:

[0119] This invention takes the overall grid voltage balance and stability and economical operation as its core objectives. Combining the reactive power regulation characteristics of photovoltaic grid-connected inverters and SVG equipment, it constructs a dual-objective optimization function that takes into account both minimizing node voltage deviation and minimizing overall grid loss. The objective function expression is as follows: ;

[0120] Where J is the comprehensive optimization target value; T is the optimization period set; and I is the distribution network node set.

[0121] The actual voltage amplitude of node i at time t; This refers to the rated voltage amplitude of the node. λ is the network loss penalty coefficient, used to balance the optimization weights of voltage deviation and network loss;

[0122] Let t be the total active power loss of the distribution network lines, which includes the power loss of the photovoltaic grid-connected inverter and SVG equipment.

[0123] Construct constraints, which include three-phase power flow constraints, voltage constraints, and equipment operation constraints;

[0124] (1) Three-phase power flow constraints: Strictly follow the three-phase power flow balance equation of the distribution network to ensure the supply and demand balance of active and reactive power, conform to the actual power transmission law of the power grid, and include the active power output of the photovoltaic grid-connected inverter and the reactive power output of the two types of equipment in the power flow calculation. For any node i, its three-phase active and reactive power satisfy:

[0125] The three-phase power flow constraints include: for any node i, its three-phase active and reactive power satisfy:

[0126] ;

[0127] ;

[0128] in, , These are the injected active and reactive power at node i, including the output of the photovoltaic grid-connected inverter and the compensation amount of the SVC device.

[0129] in Let i be the set of neighboring nodes of node i; , These are the line conductance and susceptance between nodes i and j, respectively; Let be the voltage phase angle difference between nodes i and j.

[0130] (2) Voltage Constraint: To ensure the safe operation of the power grid and the quality of electricity consumption for users, the voltage of each phase at all nodes must be controlled within the acceptable range. The constraint expression is as follows:

[0131] The voltage of each phase at all nodes must be controlled within the acceptable range. The constraint expression is as follows:

[0132] ;

[0133] in, and Indicates the lower and upper limits of the node voltage;

[0134] (3) Equipment operation constraints: Taking the photovoltaic grid-connected inverter and SVG as the core control equipment, constrain their upper and lower limits of reactive power output to ensure that the equipment operates within the rated capacity range, avoid overload operation, and adapt to the continuous action output boundary of the TD3 algorithm:

[0135] Photovoltaic grid-connected inverters: Their reactive power output is limited by the total rated capacity of the equipment and the current active power output, and is continuously adjustable. The constraint expression is:

[0136] The constraint expression for the photovoltaic grid-connected inverter is:

[0137] ;

[0138] in: The reactive power output of the photovoltaic grid-connected inverter; , These are the upper and lower limits of its reactive power output, calculated from the equipment's rated capacity and real-time active power output.

[0139] The constraint expression for SVG is:

[0140] ;

[0141] in: This refers to the reactive power output of the SVG. Its rated reactive capacity allows the output value to vary continuously between the upper and lower limits.

[0142] Corresponding to step S2, in one embodiment of the present invention, step S2 further includes the following:

[0143] Modeling the reactive power and voltage control problem in distribution networks as a Markov decision process (MDP) is a fundamental prerequisite for applying deep reinforcement learning (DLM) to this problem. MDP provides a standardized mathematical framework to describe the interaction between the agent and its environment.

[0144] This enables the learning process of control strategies to proceed in an orderly manner according to the reinforcement learning paradigm.

[0145] Use tuples express.

[0146] This invention will describe the MDP modeling method for reactive power and voltage control problems in distribution networks from four aspects: state space, action space, reward function design, and state transition.

[0147] (1) State space S

[0148] The state space S is defined as a fixed-dimensional continuous real-valued vector that comprehensively reflects the current operating state of the distribution network. This vector is composed of core electrical quantities such as the voltage amplitude of all nodes in the network, the active / reactive power injected into the nodes, the real-time active and reactive power output of the photovoltaic inverters, and the reactive power compensation of the SVG devices. This design compresses the high-dimensional operating information of the power grid into a one-dimensional continuous vector, which can be directly input into the TD3 Actor-Critic network, enabling the agent to learn voltage regulation strategies without relying on prior knowledge of the graph structure. The state space is:

[0149] Define the state space S as a continuous real-valued vector of fixed dimension to reflect the current operating state of the distribution network;

[0150] The vector is composed of electrical quantities including the voltage amplitude of all nodes in the distribution network, the active and reactive power injected into the nodes, the real-time active and reactive power output of the photovoltaic inverter, and the reactive power compensation of the SVG equipment.

[0151] The expression for the state space is:

[0152] ;

[0153] For nodes that are not equipped with photovoltaic or SVG devices, Set to 0; for single-phase or two-phase nodes, the voltage and power of the corresponding phase are 0.

[0154] (2) Action Space A

[0155] The expression for the action space is:

[0156] ;

[0157] Where: m is the number of grid-connected photovoltaic inverters; l is the number of SVG devices;

[0158] Q PVm Q SVGl These are the reactive power outputs of the m-th grid-connected photovoltaic inverter and the l-th SVG device, respectively.

[0159] (3) Reward function

[0160] The reward function is essentially a scalar evaluation signal that quantifies the immediate effect of an agent performing a specific action in a given state, providing a direct optimization direction for policy learning. This invention uses a large model to automatically generate the reward function. Based on the optimization objectives of minimizing distribution network voltage deviation and network loss, as well as safety constraints such as voltage and reactive power limits, it can dynamically balance multiple objective requirements. The specific reward function expression is output by the large model based on the task description and is used to guide the TD3 agent in learning the optimal control strategy. The reward function is expressed as:

[0161] A reward function is defined to quantify the immediate effect of an agent performing a specific action in a given state, providing a direct direction for policy learning optimization. The expression for the reward function is:

[0162] ;

[0163] in: These are, respectively, voltage penalty weight, action penalty weight, and network loss penalty coefficient;

[0164] The voltage penalty weight is used to adjust the priority of voltage exceeding the limit in the overall reward. The larger the value, the more the agent tends to pull the voltage back to the safe range. The action penalty weight is used to control the importance of the adjustment range of reactive power equipment. The larger the value, the more the equipment is encouraged to maintain stable output and reduce frequent adjustments. The network loss penalty coefficient is used to balance the weight between economic operation and voltage safety.

[0165] in This is a voltage over-limit penalty item, used to quantify the degree to which the voltage of each node deviates from the safe operating range; This is a penalty item for equipment operation costs, imposing a penalty on the magnitude of reactive power output adjustment;

[0166] in The network loss penalty item uses the total active power loss of the distribution network as a measure of the degree of economic operation.

[0167] (4) State transition

[0168] State transitions characterize the dynamic response of the distribution network after an action is applied, that is, the probability distribution of the transition from the current state to the state at the next moment. It provides reactive power control commands for photovoltaic inverters and SVG equipment. The state transition is determined based on power flow calculations and actual operating laws, reflecting the dynamic evolution logic of the power grid state under the control actions of the two types of equipment.

[0169] State transitions are defined to express the dynamic response of a distribution network after an action is performed, including the probability distribution of transitions from the current state to the state at the next moment.

[0170] The expression for the state transition is:

[0171] ;

[0172] Among the actions Including reactive power control commands from photovoltaic inverters and SVG devices, state transitions also represent the dynamic evolution logic of the grid state under the control actions of photovoltaic inverters and SVG devices.

[0173] In this invention, the radial topology of the distribution network (including node connection relationships and line impedance parameters) The physical foundation for constructing a reinforcement learning simulation environment lies in the topology parameters and device access locations. The core logic of environment construction is: topology parameters → power flow calculation → environmental feedback. Specifically, the device actions (PV / SVG reactive power output) output by the agent are superimposed with the injected power of the current node and input to the three-phase power flow calculation module (using the Newton-Raphson method) built based on the topology structure. The power flow calculation module solves the power balance equation based on the topological adjacency relationship and outputs the voltage amplitude and line power flow distribution of each node in the entire network. These calculation results directly determine the state of the agent in the next moment of the environment feedback. Instant rewards Voltage deviation and network loss values, as well as whether state transitions satisfy security constraints, are considered. Therefore, the topology implicitly embeds the "state transition kernel" of a Markov decision process into the power flow calculation equations. In the "reward and evaluation mechanism," all inputs and outputs of the reinforcement learning algorithm are deeply bound to the physical topology of the current power grid.

[0174] Corresponding to step S3, in one embodiment of the present invention, the large model guidance module in step S3 further includes the following:

[0175] This invention employs a dual-closed-loop control system based on deep reinforcement learning guided by a large language model. It dynamically constructs a reward function using the natural language understanding and prompt word engineering of the large language model, allowing the optimization objective to evolve in stages as the agent's capabilities improve. Simultaneously, it retains the adaptive decision-making advantages of deep reinforcement learning in continuous action spaces, as shown in the attached diagram. Figure 3 As shown.

[0176] In one embodiment of the present invention, the large model guidance module introduces a large language model as a guidance component for reinforcement learning training. Its core function is to use the semantic understanding and knowledge reasoning capabilities of the large model in the power system field to dynamically generate and optimize the reward function, thereby replacing the traditional method of manually designing the reward function and improving training efficiency and policy quality.

[0177] The large model guidance module is essentially a closed-loop reasoning system of "knowledge-feedback-generation". During the training startup phase, the module first receives a static environmental description of the target distribution network, including basic information such as network topology characteristics, equipment configuration parameters, and operational safety boundaries, as well as the code format specifications required for writing the reward function. During the training process, the module periodically receives dynamic performance indicators from the reinforcement learning execution layer, including statistical data such as network-wide voltage exceedances, line active power loss levels, and equipment operation frequency. These data reflect the actual performance of the agent under the current reward function. Based on this information, the large model comprehensively utilizes its understanding of the physical laws of the power system and its code generation capabilities to reason and analyze the shortcomings of the current reward mechanism, generate or revise the reward function code, and re-output it to the execution layer. The input and output of the large model are as follows:

[0178] (1) Input content:

[0179] The input content serves as prompts, employing a layered prompt strategy. The core concept of this strategy is to constrain the large model's reasoning process within the professional framework of power systems, avoiding invalid generation detached from physical reality. The static part defines the expert role the large model should play, clarifying the physical rules, safety boundaries, and output format requirements it must follow. This part remains unchanged throughout the training process, forming the "safety baseline" for the large model's reasoning. The dynamic part updates dynamically with the training progress, injecting contextual information such as the agent's latest performance, the core contradictions of the current stage, and the code generated in the previous round into the large model, enabling it to perform targeted reasoning and code revisions. The synergy between the two layers of prompts ensures that each output of the large model is built upon a complete training history context, guaranteeing the continuity and rationality of reward function adjustments. For detailed design of the prompts, please refer to the appendix. Figure 4 As shown.

[0180] (2) Output content:

[0181] The output is the reward function code. A progressive guidance mechanism is designed, incorporating the principles of curriculum learning. The core principle is that the guidance objective of the large model is not static but dynamically evolves as the agent's capabilities improve. This design stems from observations of the human learning process—beginners should first master basic skills before gradually learning more complex integrated abilities. Mapping this concept to reinforcement learning training, the guidance mechanism divides the entire training process into several stages. In the initial stage, the large model is required to generate a reward function with voltage safety as the core objective, enabling the agent to quickly master basic voltage regulation capabilities under relatively simple evaluation signals. Once the agent passes the performance threshold verification, the mechanism automatically advances to the next stage. At this point, the large model is required to increase the weight of the economic objective, guiding the agent to learn to reduce network losses while maintaining voltage safety. In the final stage, equipment operation smoothness is incorporated into the optimization scope, and the agent gradually learns a comprehensive control strategy that considers multiple objectives. See the appendix for detailed output results. Figure 5 As shown.

[0182] Corresponding to step S3, in one embodiment of the present invention, the reinforcement learning execution module and TD3 algorithm in step S3 further include the following:

[0183] The reinforcement learning execution module is based on the TD3 algorithm and includes the structural design of the Actor network and Critic network, as well as the complete training and execution process.

[0184] In one embodiment of the present invention, the Actor-Critic network design includes the following:

[0185] The Actor network is responsible for mapping the current operating state of the distribution network to action commands for each reactive power control device. The network adopts a fully connected feedforward structure. The input layer receives a state vector defined by the state space, containing the voltage amplitude of all network nodes, injected active and reactive power, and device state information. The intermediate layer consists of multiple fully connected hidden layers, employing a modified linear unit activation function to extract control-related information from the state features layer by layer. The output layer uses a hyperbolic tangent activation function to map the output values ​​to a normalized interval, and then scales them to the actual reactive power output range by multiplying by the maximum regulation capacity of each device, outputting the action vector.

[0186] The Critic network is responsible for evaluating the value of performing a specific action in the current state. This invention employs a dual-Critic structure to suppress Q-value overestimation. Each Critic network simultaneously receives a state vector and an action vector as input. The state and action are processed by independent feature mapping layers and then concatenated into a joint feature vector, which is then mapped to a scalar Q-value output through multiple fully connected layers. The two Critic networks have identical structures but different initialization parameters, each outputting two Q-values. When calculating the target value, the smaller of the two values ​​is taken to suppress overestimation bias.

[0187] In one embodiment of the present invention, the TD3 algorithm mechanism includes the following:

[0188] TD3 is developed based on Deep Deterministic Policy Gradient (DDPG) and also aims to maximize long-term cumulative discounted rewards, providing a control policy that optimizes expected returns. In practical applications, DDPG often converges to a suboptimal solution due to overestimation of the value function. TD3 introduces several key improvements to mitigate this problem, which are explained below:

[0189] Configure dual networks:

[0190] TD3 adds an isomorphic evaluation network to the single-crit structure of DDPG, forming a dual-crit architecture. During updates, two target crit networks are used to calculate the action value of the next state, and the smaller of the two values ​​is taken as the basis for the target value y. This "smaller value" mechanism directly reduces the potentially overestimated Q-value, thus effectively mitigating overestimation bias.

[0191] The core of this mechanism lies in the way the target value y is calculated. It no longer comes from a single estimate, but rather from the result after the double-network cutoff described above. The core optimization objective of TD3 is to minimize the Bellman error of the Critic network.

[0192] ;

[0193] Where E is the expected value calculated from the batch samples in the experience return pool D; This is a discount factor used to weigh current rewards against future rewards; In time The reward;

[0194] in and These are dual Critic networks; Delaying the update of the target Actor policy; target network regularization

[0195] Target network regularization:

[0196] For the TD3 algorithm, the Critic network is susceptible to function approximation bias during parameter updates, leading to inaccurate target value estimation. To mitigate this issue, TD3 adds a small amount of truncated Gaussian noise to the target policy output action during calculation, averaging the target value within a small range around the target action. This smoothing regularization technique effectively reduces the variance of the target value, avoiding overestimation caused by an excessively steep value function in local regions, and improving stability during training.

[0197] Target action The calculation method is as follows:

[0198] ;

[0199] The gradient descent expressions for the two Critics are updated as follows:

[0200] ;

[0201] in: The target Actor strategy for delayed updates; For a mean of 0 and a standard deviation of Sampling and truncation in a Gaussian distribution Random noise within the range; by superimposing this tiny random perturbation on the target action, the Critic network is forced to learn a smooth value function in a small neighborhood of the action space, avoiding overestimation of the value of a specific action due to function approximation error.

[0202] Delayed updates are used:

[0203] TD3 does not allow the Actor network and Critic network to update their parameters synchronously. Instead, it consciously postpones the Actor's update. Specifically, the Actor network's parameters are only updated after the Critic network has undergone multiple iterations and its value estimation has become relatively stable. The logic behind this arrangement is that the Actor's gradient update depends on the value signal provided by the Critic; if the Critic itself is unstable, the Actor's update direction is prone to significant oscillations. By increasing the Critic's update frequency and decreasing the Actor's update frequency, policy improvement can be performed based on a more reliable value assessment, thereby reducing the variance of policy performance and accelerating convergence to a better solution.

[0204] The Actor network is based on the state. Output Action And the gradient optimization strategy parameters are estimated using Critic. This includes the following:

[0205] ;

[0206] in: Regarding strategy parameters The gradient of the strategy performance function J; In strategy State distribution under The expected value of state S in the middle;

[0207] in The Q-function (value function) with respect to action The gradient is evaluated in the current state. and actions Below, we examine how minute changes in movement affect expected returns; among which... For policy function The action selected in state S.

[0208] Step S334: The TD3 algorithm mechanism achieves training stability by performing soft updates on the target network parameters.

[0209] ;

[0210] in: This is the soft update coefficient, which is usually taken as a small value to ensure that the target network parameters change smoothly and to provide a stable optimization target for the training process.

[0211] Corresponding to step S3, in one embodiment of the present invention, the training mode of the TD3 algorithm machine in step S3 further includes the following:

[0212] The algorithm adopts an offline centralized training and online real-time execution mode. It completes policy learning through the interaction between the agent and the power distribution network simulation environment. After training, the Actor network parameters are fixed for online control decisions. The complete process is closed-loop and efficient. The specific training steps are as follows:

[0213] Step A1: Initialize the configuration, complete the parameter initialization of the Actor network, dual Critic network and its target network; set the training hyperparameters: Actor / Critic network learning rate, discount factor. Soft update coefficient Entropy temperature coefficient Maximum number of training rounds Maximum number of moves in a single round Number of rounds after which the large model intervenes (how many rounds does it take for the large model to intervene once) Initialize the ordinary experience replay pool: Set the upper limit of the replay pool capacity to store experience samples generated by the interaction between the agent and the environment. The sampling adopts a completely random and uniform sampling method without priority distinction.

[0214] Step A2: Initialize the training rounds. Set the number of training rounds, episode = 1, and check if episode ≤ 1. If the condition is not met, proceed to step 12; if the condition is met, proceed to the next step.

[0215] Step A3: Simulation Environment Initialization. Initialize the distribution network simulation environment, load the distribution network topology, line parameters, and photovoltaic grid-connected inverter / SVG equipment parameters, obtain the initial voltage, load power, and distributed generation output of each node, and obtain all system operating information including the initial operating status of the equipment and the initial state of the power grid. .

[0216] Step A4: Initialize the time by setting the time t=0 within a single round, and check if t≤ ;

[0217] If the condition is not met, proceed to step 11; if the condition is met, proceed to the next step.

[0218] Step A5: Action Generation and Exploration Enhance the Actor Network based on the current power grid state. The vector is used for forward propagation calculation, passing through the input layer and hidden layer to the output layer, outputting the reactive power regulation actions of the photovoltaic grid-connected inverter and SVG equipment. To enhance the exploratory nature of the strategy, Adding Gaussian noise yields the actual execution action. .

[0219] Step A6: Environmental Interaction and Feedback to Obtain Actions in the Distribution Network Simulation Environment Adjust the reactive power output of the photovoltaic grid-connected inverter / SVG, calculate and feed back the grid state at the next moment. Instant rewards (The reward value is linked to voltage deviation, network loss, and equipment operation compliance.)

[0220] Step A7: Experience Sample Storage (This will store the complete experience sample...) , , , The samples are stored in the ordinary experience replay pool. The samples contain four types of core information: power grid status, equipment control actions, instant rewards, and the next status. When the replay pool is full, old samples are eliminated according to the first-in-first-out strategy.

[0221] Step A8: Batch Sample Extraction. Randomly and uniformly extract a batch of samples from the ordinary experience replay pool:

[0222] ( , , , ), used for iterative updates of Actor / Critic network parameters.

[0223] Step A9: Critic network parameter update. Based on the extracted batch samples, calculate the Bellman error of the dual Critic network, and iteratively update the dual Critic network parameters through the gradient descent algorithm to improve the estimation accuracy of the value (Q value) of the "grid status-equipment control action" pair.

[0224] Step A10: Delayed update of the Actor network. The Actor network parameters are updated based on the Q value output by the Critic network, so that the output photovoltaic / SVG control commands are more in line with the core objectives of voltage optimization and network loss reduction. The direction of agent policy optimization is determined by the objective defined by the reward function designed by the large model.

[0225] Step A11: Use a soft update mechanism Update the target network parameters of Actor and Critic to ensure smooth evolution of the target network parameters during training and guarantee training stability. After the step-size loop ends, set t = t + 1 and return to step A4 to continue execution until all time steps within a single round are completed.

[0226] Step A12: During the training round iteration, let episode = episode + 1, return to step A2, and proceed to the next training round.

[0227] Step A13: End of Round and Performance Evaluation Feedback. After completing all time steps of the current round, the round number increments, and the process returns to Step A2 to begin the next round of training. The number of rounds after each large model intervention is calculated... Key performance indicators (KPIs) for this phase are collected, including the overall network voltage qualification rate, line active power loss statistics, equipment operation frequency, and cumulative average reward. These KPIs are compiled into structured feedback information and fed into the large model guidance module. Based on this feedback, and combined with the current phase objectives and historical code context, the large model performs chain reasoning to analyze which parts of the current reward mechanism are poorly designed, and dynamically generates a new reward function for the next phase. It then automatically replaces the old version in the execution environment. The agent then continues training under the guidance of the updated reward function.

[0228] Step A14: When the training reaches the maximum number of rounds When the core performance indicators (total network voltage deviation, total active power loss, and reward value) on the validation set tend to stabilize for several consecutive rounds, the model is deemed to have converged and training is stopped. The optimal Actor network parameters are solidified (the Critic network and related training modules are removed), and a deployable online control model is generated for real-time reactive power control of photovoltaic grid-connected inverters and SVG devices.

[0229] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A deep reinforcement learning-based distribution network voltage control method guided by a large model, characterized in that, Includes the following: Step S1: Establish a voltage optimization model, including constructing a dual-objective optimization function and constraints; Step S2: Construct a distribution network voltage control model using a Markov decision process, which includes state space, action space, reward function design, and state transition. Step S3: The reward mechanism of the distribution network voltage control model is generated and optimized by the large model guidance module, and then the distribution network voltage control model is trained by the reinforcement learning execution module based on the TD3 algorithm; thus, a deployable online voltage regulation model for the distribution network is obtained.

2. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 1, characterized in that, Step S1 includes the following: Step S11: Construct a dual-objective optimization function, which includes node voltage deviation and total power loss of the distribution network, and includes the following: ； Where J is the comprehensive optimization target value; T is the optimization period set; and I is the distribution network node set. The actual voltage amplitude of node i at time t; This refers to the rated voltage amplitude of the node. λ is the network loss penalty coefficient, used to balance the optimization weights of voltage deviation and network loss; Let t be the total active power loss of the distribution network lines, which includes the power loss of the photovoltaic grid-connected inverter and SVG equipment.

3. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 2, characterized in that, Step S1 also includes the following: Step S12: Construct constraints, which include three-phase power flow constraints, voltage constraints, and equipment operation constraints; The three-phase power flow constraints include: for any node i, its three-phase active and reactive power satisfy: ； ； in, , These are the injected active and reactive power at node i, including the output of the photovoltaic grid-connected inverter and the compensation amount of the SVC device. in Let i be the set of neighboring nodes of node i; , These are the line conductance and susceptance between nodes i and j, respectively; Let be the voltage phase angle difference between nodes i and j; The voltage constraints include: The voltage of each phase at all nodes must be controlled within the acceptable range. The constraint expression is as follows: ； in, and Indicates the lower and upper limits of the node voltage; The equipment operation constraints include photovoltaic grid-connected inverter constraints and SVG constraints; The constraint expression for the photovoltaic grid-connected inverter is: ； in: The reactive power output of the photovoltaic grid-connected inverter; , These are the upper and lower limits of its reactive power output, calculated from the equipment's rated capacity and real-time active power output. The constraint expression for SVG is: ； in: This refers to the reactive power output of the SVG. Its rated reactive capacity allows the output value to vary continuously between the upper and lower limits.

4. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 3, characterized in that, Step S2 includes the following: Step S21: Construct the state space, including the following: Define the state space S as a continuous real-valued vector of fixed dimension to reflect the current operating state of the distribution network; The vector is composed of electrical quantities including the voltage amplitude of all nodes in the distribution network, the active and reactive power injected into the nodes, the real-time active and reactive power output of the photovoltaic inverter, and the reactive power compensation of the SVG equipment. The expression for the state space is: ； For nodes that are not equipped with photovoltaic or SVG devices, Set to 0; for single-phase or two-phase nodes, the voltage and power of the corresponding phase are 0; Step S22: Construct the action space, including the following: Define an action space A, which includes a set of continuous control instructions output by the agent, including the reactive power output adjustment of the photovoltaic inverter and the SVG device. The expression for the action space is: ； Where: m is the number of grid-connected photovoltaic inverters; l is the number of SVG devices; Q PVm Q SVGl These are the reactive power outputs of the m-th grid-connected photovoltaic inverter and the l-th SVG device, respectively.

5. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 4, characterized in that, Step S2 also includes the following: Step S23: Define a reward function to quantify the immediate effect of an agent performing a specific action in a given state, providing a direct optimization direction for policy learning; the expression for the reward function is: ； in: These are, respectively, voltage penalty weight, action penalty weight, and network loss penalty coefficient; The voltage penalty weight is used to adjust the priority of voltage exceeding the limit in the overall reward. The larger the value, the more the agent tends to pull the voltage back to the safe range. The action penalty weight is used to control the importance of the adjustment range of reactive power equipment. The larger the value, the more the equipment is encouraged to maintain stable output and reduce frequent adjustments. The network loss penalty coefficient is used to balance the weight between economic operation and voltage safety. in This is a voltage over-limit penalty item, used to quantify the degree to which the voltage of each node deviates from the safe operating range; This is a penalty item for equipment operation costs, imposing a penalty on the magnitude of reactive power output adjustment; in The network loss penalty item uses the total active power loss of the distribution network as a measure of the degree of economic operation. Step S23: Define state transitions to express the dynamic response of the distribution network after an action is applied, including the probability distribution of transitioning from the current state to the state at the next moment. The expression for the state transition is: ； Among the actions Including reactive power control commands from photovoltaic inverters and SVG devices, state transitions also represent the dynamic evolution logic of the grid state under the control actions of photovoltaic inverters and SVG devices.

6. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 5, characterized in that, Step S3 includes the following: Step S31: Use the large model guidance module to generate and optimize the reward mechanism of the distribution network voltage control model. This includes the large model guidance module periodically receiving statistical data including the voltage over-limit situation of the entire network, the active power loss level of the line, and the frequency of equipment operation; analyzing the deficiencies of the current reward mechanism, generating or revising the reward mechanism including the reward function code, and re-outputting it to the execution layer of the distribution network voltage control model. The design employs a hierarchical prompting strategy to obtain prompting words, which serve as the input content for the large model guidance module. The layered prompting strategy includes constraining the reasoning process of large models within the professional framework of power systems and shielding interference from other industries; The hierarchical prompting strategy includes a static part and a dynamic part; the static part is used to define the physical rules that the large model needs to follow, while the dynamic part is used to update the training process and inject context information into the large model for revision. The design incorporates a progressive guidance mechanism based on the principles of curriculum learning for training large models. The training process of the large model includes a reward mechanism generated in the pre-stage with voltage safety as the goal. After the large model passes the performance threshold verification, the weight of the economic objective in the reward mechanism is increased, so that the large model can reduce network loss while maintaining voltage safety. The output includes a comprehensive control strategy and reward mechanism that takes into account multiple objectives.

7. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 6, characterized in that, Step S3 also includes the following: Step S32: The reinforcement learning execution module is based on the TD3 algorithm and includes the structural design of the Acto network and Critic network, as well as the complete training and execution process; The Actor network is used to map the current operating state of the distribution network to the action commands of each reactive power control device, including the following: The Actor network adopts a fully connected feedforward structure. The input layer receives a state vector defined by the state space, which includes the voltage amplitude of all network nodes, injected active and reactive power, and equipment state information. The intermediate layer sets up multiple fully connected hidden layers and uses a modified linear unit activation function to extract regulation-related information from the state features layer by layer. The output layer uses a hyperbolic tangent activation function to map the output value to a normalized interval, and then scales it to the actual reactive power output range by multiplying it by the maximum regulation capacity of each device, and outputs the action vector. The Critic network is used to evaluate the value of performing a specific action in the current state, including the following: A dual-Critic structure is adopted to suppress the Q-value overestimation problem. Each Critic network simultaneously receives a state vector and an action vector as input. The state and action are processed by independent feature mapping layers and then concatenated into a joint feature vector, which is then mapped to a scalar Q-value output through multiple fully connected layers. The two Critic networks have the same structure but different initialization parameters and output two Q-values ​​respectively. When calculating the target value, the smaller of the two values ​​is taken to suppress the overestimation bias.

8. The deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 7, characterized in that, Step S3 also includes the following: Step S33: Design the TD3 algorithm mechanism, including the following: The TD3 algorithm mechanism is based on the gradient of a deep deterministic policy, calculates the long-term cumulative discount reward of the target model, and obtains the control policy with the optimal expected return of the target model. Step S331: Set up a dual network, including the following: The TD3 algorithm mechanism adds an isomorphic evaluation network to the single Critic structure of the degree-deterministic policy gradient, forming a dual Critic architecture. During the update, the two target Critic networks are used to calculate the action value of the next state, and the smaller of the two values ​​is taken as the basis for the target value y. The optimization objective of the TD3 algorithm mechanism is to minimize the Bellman error of the Critic network, including the following: ； Where E is the expected value calculated from the batch samples in the experience return pool D; This is a discount factor used to weigh current rewards against future rewards; In time The reward; in and These are dual Critic networks; To delay updating the target Actor policy; target network regularization; Step S332: Target network regularization, including the following: The TD3 algorithm mechanism adds a small amount of truncated Gaussian noise to the action output by the target policy when calculating the target action, so that the target value is averaged within a small range near the target action. Among them, target action The calculation method is as follows: ； The gradient descent expressions for the two Critics are updated as follows: ； in: The target Actor strategy for delayed updates; For a mean of 0 and a standard deviation of Sampling and truncation in a Gaussian distribution Random noise within the range; Step S333: Use delayed update, including the following: The TD3 algorithm mechanism delays the timing of Actor updates, including updating the parameters of the Actor network only after the Critic network has undergone multiple iterations and the value estimation has become relatively stable. The Actor network is based on the state. Output Action And the gradient optimization strategy parameters are estimated using Critic. This includes the following: ； in: Regarding strategy parameters The gradient of the strategy performance function J; In strategy State distribution under The expected value of state S in the middle; in The Q-function (value function) with respect to action The gradient is evaluated in the current state. and actions Below, we examine how subtle changes in movement affect expected returns. in For policy function The action selected in state S; Step S334: The TD3 algorithm mechanism achieves training stability by performing soft updates on the target network parameters. ； in: This is the soft update coefficient, which is usually taken as a small value to ensure that the target network parameters change smoothly and to provide a stable optimization target for the training process.

9. A deep reinforcement learning-based distribution network voltage control method based on a large model progressively guided by claim 8, characterized in that, Step S3 also includes the following: Step S34: The TD3 algorithm mechanism adopts an offline centralized training and online real-time execution mode. It completes policy learning through the interaction between the agent and the distribution network simulation environment. After training, the network parameters are fixed for online control decision-making, resulting in a deployable distribution network voltage online control model. The training mode of the TD3 algorithm machine includes initialization configuration, initialization of training rounds (setting the number of training rounds), initialization of the distribution network simulation environment (initializing the distribution network simulation environment), action generation and exploration to enhance the Actor network, environment interaction and feedback to obtain the distribution network simulation environment to execute actions, storage of experience samples (storing complete experience samples into the ordinary experience replay pool), batch sample extraction (randomly and uniformly extracting batch samples from the ordinary experience replay pool), updating the target network parameters of the Actor and Critic using a soft update mechanism, and round end and performance evaluation feedback.

10. A distribution network voltage control system based on large-model progressive guidance deep reinforcement learning, used to execute the distribution network voltage control method based on large-model progressive guidance as described in any one of claims 1 to 9, characterized in that, Includes the following: The voltage optimization model module is used to construct voltage optimization models that include dual objective optimization functions and constraints. The Markov Decision Process module uses Markov decision processes to construct a voltage control model for the distribution network. The Markov decision process includes state space, action space, reward function design, and state transition. The large model guidance module is used to generate and optimize the reward mechanism of the distribution network voltage control model. It adopts a hierarchical prompt word strategy to obtain prompt words, which serve as the input content of the large model guidance module. The design incorporates a progressive guidance mechanism based on curriculum learning concepts for training large models; the output includes a comprehensive control strategy and reward mechanism that takes into account multiple objectives. The reinforcement learning execution module is based on the TD3 algorithm and includes the structural design of the Acto network and Critic network, as well as the complete training and execution process. The training module, based on the TD3 algorithm mechanism, adopts an offline centralized training and online real-time execution mode. It completes policy learning through the interaction between the agent and the distribution network simulation environment. After training, the network parameters are fixed for online control decisions, resulting in a deployable online voltage control model for the distribution network.

Images