An actor-critic reinforcement learning operation optimization method, system, device and medium for an AC-DC hybrid power distribution network

Abstract

The application relates to the technical field of power system operation control, and discloses an Actor-Critic reinforcement learning operation optimization method, system, equipment and medium for an AC-DC hybrid distribution network, which comprises the following steps: firstly, a state space containing multi-source prediction and measured information, equipment state and time characteristics is constructed, a normalized continuous action space covering a converter, energy storage and a distributed power supply is defined, and a composite reward function composed of economy, safety, new energy consumption and action smoothness is designed. A Markov decision process model is established based on the above, and an Actor-Critic system composed of a strategy network and a value network is configured. Offline training is carried out through experience replay, target network soft update and Gaussian exploration noise mechanism. In the online stage, the current operation state is periodically collected, control instructions are output through the strategy network, and the actual physical quantity is converted and executed, so that the safe and economic operation optimization of the AC-DC hybrid distribution network is realized.

Description

Technical Field

[0001] This invention relates to the field of power system operation control technology, and in particular to an Actor-Critic reinforcement learning operation optimization method, system, equipment and medium for AC / DC hybrid distribution networks. Background Technology

[0002] The AC / DC hybrid distribution network combines the power supply advantages of AC and DC systems, can flexibly accommodate distributed photovoltaic, wind power and other renewable energy sources, and supports the access of energy storage, flexible loads and various types of converter equipment, and has become an important form of new power system.

[0003] However, its operational optimization faces challenges such as high-dimensional state spaces, strong uncertainties, and multi-timescale coupling. Traditional optimization methods rely on precise mathematical models and deterministic predictions, making them difficult to adapt to real-world operating environments with frequent source load fluctuations and dynamic topological changes. In recent years, deep reinforcement learning techniques have attracted attention due to their independence from precise system models and their ability to make online adaptive decisions. Among these, the Actor-Critic architecture, which combines policy gradients and value assessment, has advantages in handling continuous control problems.

[0004] However, existing methods still have shortcomings in terms of state representation completeness, physical feasibility of action space, rationality of reward function design, and training stability, making it difficult for the generated control strategies to meet the comprehensive operational requirements of AC / DC hybrid distribution networks for safety, economy, and new energy consumption. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, this invention provides an Actor-Critic reinforcement learning operation optimization method, system, equipment, and medium for AC / DC hybrid distribution networks, which can solve the problem that traditional methods are difficult to cope with frequent source-load fluctuations and dynamic changes in topology.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an Actor-Critic reinforcement learning-based operation optimization method for AC / DC hybrid distribution networks, comprising: A Markov decision process model is constructed to describe the operation of an AC / DC hybrid distribution network. The Markov decision process model includes a state space, an action space, and a reward function. Based on the Markov decision process model, an Actor-Critic reinforcement learning system consisting of a policy network and a value network is constructed. The Actor-Critic reinforcement learning system is trained offline using historical operating data or simulated operating scenarios to obtain a trained policy network; During the online operation phase, the current operating status of the AC / DC hybrid distribution network is collected according to a set cycle; The current operating state is input into the trained policy network, and normalized continuous control commands are output. The normalized continuous control command is converted into an actual physical control quantity according to the maximum allowable power limit of the corresponding device. The actual physical control quantities are sent to the controllable devices in the AC / DC hybrid distribution network.

[0008] As a preferred embodiment of the AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method described in this invention, the state space includes: The data includes: forecasted photovoltaic power generation for multiple future time periods; forecasted wind power generation for multiple future time periods; measured photovoltaic power generation for multiple recent time periods; measured wind power generation for multiple recent time periods; forecasted AC load power for multiple future time periods; forecasted DC load power for multiple future time periods; measured AC load power for multiple recent time periods; measured DC load power for multiple recent time periods; electricity purchase price for multiple future time periods; electricity sales price for multiple future time periods; current state of charge of each energy storage device; average state of charge of all energy storage devices; network topology represented by unique thermal coding; line current measurement; hourly markers for a day; day of the week markers; and seasonal markers.

[0009] As a preferred embodiment of the AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method described in this invention, the action space includes normalized continuous control instructions, which include: Normalization commands for power exchanged with the main grid, normalization commands for active power of each AC-DC converter, normalization commands for reactive power of each AC-DC converter, normalization commands for charging and discharging power of each energy storage device, and normalization commands for output of each controllable distributed power source.

[0010] As a preferred embodiment of the AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method described in this invention, the reward function is composed of a weighted combination of economic reward terms, operation safety penalty terms, new energy consumption reward terms, and control action change smoothness penalty terms. The economic incentive is calculated based on the purchased power, sold power, distributed power generation cost, and network loss cost. The operational safety penalty items are calculated based on the degree of voltage over-limit, current over-limit, and the degree of over-limit of the energy storage device's state of charge. The new energy consumption incentive is calculated based on the difference between the actual power generation of new energy equipment and the theoretical maximum power generation. The control action change smoothness penalty term is calculated based on the change range between the current control action and the control action at the previous moment.

[0011] As a preferred embodiment of the AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method described in this invention, the policy network receives the state feature vector processed by the state feature extraction network, outputs the Gaussian distribution parameters corresponding to each control dimension through multiple parallel branches, and generates normalized continuous control commands based on the Gaussian distribution parameters. The state feature extraction network is a multilayer perceptron structure.

[0012] As a preferred embodiment of the AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method described in this invention, the value network receives the state feature vector and the normalized continuous control command, and outputs the corresponding state action value estimate. The parameters of the value network are updated by minimizing the temporal difference error; The target value of the time-series difference error is calculated jointly by the target value network and the target policy network.

[0013] As a preferred embodiment of the AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method described in this invention, in the offline training phase, an experience replay buffer is constructed to store state transition samples generated during the interaction between the agent and the AC / DC hybrid distribution network. A batch of samples is randomly drawn from the experience replay buffer to update the parameters of the policy network and the value network; Gaussian exploration noise is superimposed on the control actions output by the policy network, and the standard deviation of the exploration noise gradually decreases with the number of training steps; Set up a target policy network with the same structure as the policy network, and a target value network with the same structure as the value network; The parameters of the target policy network and the parameters of the target value network are gradually synchronized from the corresponding main network parameters through a soft update method, wherein the soft update method uses a fixed-ratio weighted average for parameter migration.

[0014] Secondly, the present invention provides an Actor-Critic reinforcement learning operation optimization system for AC / DC hybrid distribution networks, comprising: The status acquisition module is used to collect the current operating status of the AC / DC hybrid distribution network according to a set period. The strategy network module is used to receive the current operating state and output normalized continuous control commands; The instruction conversion module is used to convert the normalized continuous control instruction into an actual physical control quantity according to the maximum allowable power limit of the corresponding device; The control execution module is used to send the actual physical control quantities to controllable devices in the AC / DC hybrid power distribution network. The training module is used to train the Actor-Critic reinforcement learning system, which consists of a policy network module and a value network module, offline using historical running data or simulated running scenarios. The value network module is used to receive state feature vectors and normalized continuous control commands, and output state action value estimates. The training module is also used to construct an experience replay buffer, store state transition samples, and randomly extract batch samples from the experience replay buffer to update the parameters of the policy network module and the value network module. The training module is also used to configure the target policy network and the target value network, and to gradually synchronize the parameters of the target policy network and the target value network from the parameters of the policy network module and the value network module through a soft update method.

[0015] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0016] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0017] Compared with existing technologies, the beneficial effects of this invention are that it proposes an Actor-Critic reinforcement learning-based operation optimization method for AC / DC hybrid distribution networks. First, it constructs a state space incorporating multi-source prediction and measured information, equipment status, and time characteristics. It defines a normalized continuous action space covering converters, energy storage, and distributed power sources, and designs a composite reward function composed of economic efficiency, safety, renewable energy integration, and action smoothness. Based on this, a Markov decision process model is established, and an Actor-Critic system consisting of a policy network and a value network is configured. Offline training is performed through experience replay, soft updates of the target network, and a Gaussian exploration noise mechanism. During the online phase, the current operating status is periodically collected, and control commands are output through the policy network, converted into actual physical quantities, and then executed to achieve safe and economical operation optimization of the AC / DC hybrid distribution network. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The present invention provides a flowchart of an Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks, as an embodiment of the present invention.

[0020] Figure 2 This invention provides a typical AC / DC hybrid distribution network for an Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks, as an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of an AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method provided in an embodiment of the present invention.

[0022] Figure 4 This is an internal structure diagram of an electronic device for an AC / DC hybrid power distribution network Actor-Critic reinforcement learning operation optimization method provided in one embodiment of the present invention. Detailed Implementation

[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0024] It should be noted in advance that the system mentioned in the embodiments as the subject of real-time operation refers to any system configured with this method.

[0025] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides an Actor-Critic reinforcement learning-based operation optimization method for AC / DC hybrid distribution networks, comprising: This invention provides a method that can effectively solve the problems mentioned above. The following will describe in detail how to implement the Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks using multiple embodiments. Figure 1 A flowchart illustrating an Actor-Critic reinforcement learning-based operational optimization method for AC / DC hybrid distribution networks is shown, including: S101, Construct a Markov decision process model to describe the operation of an AC / DC hybrid distribution network. The Markov decision process model includes a state space, an action space, and a reward function. In embodiments of the present invention, the state space includes: The data includes: forecasted photovoltaic power generation for multiple future time periods; forecasted wind power generation for multiple future time periods; measured photovoltaic power generation for multiple recent time periods; measured wind power generation for multiple recent time periods; forecasted AC load power for multiple future time periods; forecasted DC load power for multiple future time periods; measured AC load power for multiple recent time periods; measured DC load power for multiple recent time periods; electricity purchase price for multiple future time periods; electricity sales price for multiple future time periods; current state of charge of each energy storage device; average state of charge of all energy storage devices; network topology represented by unique thermal coding; line current measurement; hourly markers for a day; day of the week markers; and seasonal markers.

[0026] In some embodiments, the system state space is designed to contain all key environmental information affecting decision-making, represented as a vector. ∈S. At time t, the state vector It consists of new energy power generation capacity, load demand, time-of-use electricity price signal from the upper-level power grid, energy storage system status, network topology status, time information, day-of-week coding, and seasonal coding vector. The more detailed components are as follows: The new energy power generation capacity includes the following: This represents a vector of predicted photovoltaic power values ​​for the next N time periods. This represents a vector of predicted wind power values ​​for the next N time periods. This represents a vector of measured photovoltaic power values ​​over M recent time periods; This represents a vector of measured wind power values ​​over M recent time periods.

[0027] In the formula, The projected photovoltaic power generation (in kW or MW) for the (t+1)th time period (the first future time period). N represents the forecast time length (e.g., the next 24 hours) and the size of the "forecast window". This represents the expected wind power generation capacity for the (t+1)th time period. M represents the length of the historical data time window (e.g., the most recent 6 hours). The actual photovoltaic output in the t-1th time period (i.e., the time period before the current one). This represents the actual wind power output during the t-k time period.

[0028] The load demand includes the following: This represents a vector of predicted AC load values ​​for the next N time periods. This represents a vector of predicted DC load values ​​for the next N time periods. This represents a vector of measured AC load values ​​over M recent time periods; This represents a vector of measured DC load values ​​over M recent time periods.

[0029] In the formula, This represents the estimated power consumption of the AC load in the (t+1)th time period. This represents the estimated power consumption of the DC load in the (t+1)th time period. This represents the actual AC load during the (t-1)th time period. This represents the actual DC load during the (t-1)th time period.

[0030] The time-of-use pricing signal from the upper-level power grid includes the following: This represents the vector of electricity purchase prices for N future time periods; This represents the vector of electricity sales prices for the next N time periods.

[0031] The status of the energy storage system includes the following: This represents the current power status of the i-th energy storage system. ; It represents the average SOC of all energy storage systems, reflecting the overall state of charge of the system's energy storage.

[0032] The network topology status includes the following: The topology state vector is represented by one-hot encoding to indicate the current network connection state. This represents a vector of current measurements for the line.

[0033] The time information includes the following: This represents the hour code of the day. Code representing the number of days in a week; This represents the seasonal encoding vector.

[0034] In summary, the complete state space vector is: ].

[0035] In embodiments of the present invention, the action space includes normalized continuous control instructions, which include: Normalization commands for power exchanged with the main grid, normalization commands for active power of each AC-DC converter, normalization commands for reactive power of each AC-DC converter, normalization commands for charging and discharging power of each energy storage device, and normalization commands for output of each controllable distributed power source.

[0036] In some embodiments, the action space is defined as continuous, normalized control commands, which facilitates the output of the reinforcement learning agent; specifically, it can be represented as a vector. ∈A.

[0037] First, the power exchanged with the main power grid. Indicates normalized switching power command; Actual power is expressed as: in For maximum power purchase capacity, This represents the maximum power output.

[0038] Furthermore, regarding AC / DC converter control, for the j-th converter device: This indicates a command for normalized active power. This indicates a normalized reactive power command; Indicates actual active power; This represents the actual reactive power.

[0039] Furthermore, regarding energy storage system control, for the k-th energy storage unit: This indicates a normalized charge / discharge command; Actual charge / discharge power: Furthermore, for the m-th distributed generation device, controllable distributed generation equipment... This indicates a normalized output command; Actual output is expressed as: In summary, the complete action space vector is: In this embodiment of the invention, the reward function is a weighted combination of economic reward items, operational safety penalty items, new energy consumption reward items, and control action change smoothness penalty items; The economic incentive is calculated based on the purchased power output, sold power output, distributed power generation cost, and network loss cost. The operational safety penalty items are calculated based on the degree of voltage over-limit, current over-limit, and the degree of over-limit of the energy storage device's state of charge. The incentive for renewable energy consumption is calculated based on the difference between the actual power generation of renewable energy equipment and its theoretical maximum power generation. The control action change smoothness penalty is calculated based on the change range between the current control action and the control action at the previous moment.

[0040] In some embodiments, the reward function is used to guide the agent to learn the optimal policy, and mainly includes the following four parts: The first part is the economic reward, which can be specifically represented as: in, For the power purchased; This refers to the power output for electricity sales. The cost of generating electricity from power generation equipment; Cost of network loss.

[0041] Furthermore, this includes safety penalties, which are penalties imposed for exceeding system voltage, current, and energy storage SOC limits, as follows: Furthermore, this includes incentives for the consumption of new energy sources, specifically as follows: in, This represents the theoretical maximum power output of various new energy power generation equipment. This represents the actual power output of various new energy power generation equipment.

[0042] Furthermore, this includes a motion smoothness penalty, which can be specifically expressed as: In the above formula, the typical weighting coefficient is, Between 10 and 100; Between 5 and 20; Between 0.1 and 1.0; Between 0.01 and 0.1.

[0043] In summary, the overall reward function is: S102, Constructing an Actor-Critic reinforcement learning system composed of a policy network and a value network based on the Markov decision process model; In this embodiment of the invention, the policy network receives the state feature vector processed by the state feature extraction network, outputs the Gaussian distribution parameters corresponding to each control dimension through multiple parallel branches, and generates normalized continuous control commands based on the Gaussian distribution parameters. The state feature extraction network is a multilayer perceptron structure.

[0044] In an embodiment of the present invention, the value network receives a state feature vector and a normalized continuous control command, and outputs the corresponding state action value estimate. The parameters of the value network are updated by minimizing the temporal difference error; The target value of the temporal difference error is calculated jointly by the target value network and the target policy network.

[0045] S103, The Actor-Critic reinforcement learning system is trained offline using historical running data or simulated running scenarios to obtain the trained policy network; It should be noted that in actual operation, the actions of controllable equipment in AC / DC hybrid distribution networks need to achieve multi-objective collaborative optimization under the premise of satisfying physical constraints. This control task is highly dependent on the policy network's ability to accurately model complex state-action mapping relationships.

[0046] For example, in a scenario of combined disturbances such as a sudden drop in photovoltaic output and a sudden increase in load, if the strategy network has not been sufficiently trained, the output control commands may violate the converter power regulation rate limit or cause the bus voltage to exceed the limit, and may even cause local instability or malfunction of protection devices.

[0047] Therefore, it is necessary to use historical operating data or diverse operating scenarios generated by high-fidelity simulation environments to train the Actor-Critic reinforcement learning system offline, so that the policy network has robust response capabilities to various typical and boundary conditions before deployment.

[0048] S104, during the online operation phase, collects the current operating status of the AC / DC hybrid distribution network according to a set cycle; S105 inputs the current running state into the trained policy network and outputs normalized continuous control commands. It should be noted that in actual operation, the controllable equipment in the AC / DC hybrid distribution network includes flexible DC converters, static var compensators, energy storage converters and adjustable loads, etc. Their control variables are continuous and have strong coupling characteristics, and multi-variable collaborative decision-making needs to be achieved through strategy networks.

[0049] For example, when dealing with fluctuations in the output of new energy sources, the strategy network needs to coordinate the active power redistribution and reactive power support intensity of multiple converter stations simultaneously. If the output command is a discrete action or has not been normalized, it is difficult to match the control interface specifications of different devices and cannot guarantee the consistency of the action space.

[0050] For example, if the policy network output is not constrained by an activation function, it may produce values ​​that exceed the range of legal actions, causing subsequent transformation logic to fail. If the raw network output is used directly without normalization, some control dimensions may be ignored or amplified due to differences in units.

[0051] Therefore, the current operating state needs to be input into the trained policy network to output continuous control commands normalized to the range of negative one to positive one, so as to unify the action representation and adapt to the subsequent physical quantity conversion process.

[0052] S106 converts the normalized continuous control command into an actual physical control quantity according to the maximum allowable power limit of the corresponding equipment; S107 sends the actual physical control quantities to the controllable devices in the AC / DC hybrid distribution network.

[0053] In some embodiments, a dual-network architecture that separates strategy evaluation and strategy improvement is adopted. The optimal control strategy is generated through the Actor network, and the Critic network evaluates the value of state-action pairs, thereby realizing intelligent decision-making for the optimization of AC / DC hybrid distribution network operation.

[0054] First, a state feature extraction network is designed. This network employs a deep multilayer perceptron architecture, aiming to map the high-dimensional original state space into a low-dimensional feature representation. The state feature extraction network can be described as follows: Secondly, an Actor network is designed. Based on a stochastic policy framework, the Actor network is responsible for generating continuous control actions that conform to the physical constraints of the AC / DC hybrid distribution network system. The network receives state feature vectors. This network outputs action distribution parameters through a parallel branching structure and can be described as follows: Furthermore, a Critic network is designed. The Critic network is a value function used to evaluate state-action pairs, which can be described as follows: in, and These are the main network parameters for the Actor network and the Critic network, respectively.

[0055] Furthermore, a target network and a soft update mechanism are designed. To improve training stability, the system sets up a target Actor network and a target Critic network. The target network has the same structure as the main network, and its parameters are progressively synchronized through a soft update mechanism. The target Actor network is as follows: The target Critic network is, The soft update mechanism can be described as follows: in, This is the soft update coefficient. as well as These are all target network parameters.

[0056] In some embodiments, the specific steps of the Actor-Critic reinforcement learning system training algorithm include: The Critic network is designed with a parameter update strategy that updates parameters by minimizing the temporal difference error. The target value is calculated as follows: The Actor network updates its parameters using the policy gradient ascent method. Furthermore, an experience replay mechanism is designed. This invention constructs an experience replay mechanism that stores historical experiences gained by the agent during its interaction with the AC / DC power distribution network and samples these experiences during training to break the correlation between data, thereby improving the efficiency and stability of the learning process.

[0057] By randomly sampling small batches of data during training, Where B represents the batch size and D represents the entire experience playback buffer; Establish an experience replay buffer to store historical transfer samples. in, Indicates the buffer capacity.

[0058] Furthermore, to enhance strategy exploration during the training phase, this invention adds exploration noise to the Actor network's output actions. The noise was explored using a Gaussian distribution. Furthermore, to accelerate the convergence of the training process, the noise variance decreases as the training progresses, i.e., This represents the initial noise variance.

[0059] In this embodiment of the invention, during the offline training phase, an experience replay buffer is constructed to store state transition samples generated during the interaction between the agent and the AC / DC hybrid power distribution network. A batch of samples is randomly drawn from the experience replay buffer to update the parameters of the policy network and the value network; Gaussian exploration noise is superimposed on the control actions output by the policy network, and the standard deviation of the exploration noise gradually decreases with the number of training steps. Set up a target policy network with the same structure as the policy network, and a target value network with the same structure as the value network; The parameters of the target policy network and the target value network are gradually synchronized from the corresponding main network parameters through a soft update method, which uses a fixed-ratio weighted average for parameter migration.

[0060] It should be understood that when an agent directly uses continuously generated interaction sequences to update the parameters of the policy network and the value network during offline training, the parameter updates will have a strong temporal correlation, which may cause oscillations or even divergence in the training process.

[0061] If a fixed-amplitude noise disturbance is used to control the action during the strategy exploration phase, the exploration intensity may not be able to adapt to the training process, resulting in slow policy convergence or getting stuck in a local optimum in the later stages.

[0062] Therefore, this invention constructs an experience replay buffer during the offline training phase and initiates a joint training mechanism based on random sampling, dynamic noise attenuation, and soft synchronization of the target network.

[0063] In some embodiments, the specific operations of constructing an experience replay buffer and using it to support network parameter updates may include the following steps: In response to each interaction between the agent and the AC / DC hybrid power distribution network simulation environment, the state transition sample is written into the experience replay buffer in the form of a quadruple. The state transition sample includes the state feature vector at the current moment, the executed normalized continuous control command, the obtained composite reward value, and the state feature vector at the next moment. Furthermore, batch samples are drawn from the experience replay buffer in a uniformly random manner. The number of batch samples is the preset training batch size, which is used to calculate the gradient update direction of the policy network and the value network. Furthermore, exploration noise following a Gaussian distribution is superimposed on the normalized continuous control commands output by the policy network. The standard deviation of the exploration noise gradually decreases with the number of training steps according to an exponential decay function. The initial standard deviation is a preset positive value, and the decay rate is controlled by a fixed decay coefficient. Furthermore, a target policy network with the same structure as the policy network and a target value network with the same structure as the value network are configured respectively, and the target network parameters are periodically synchronized through a soft update method. The soft update method uses a fixed-ratio weighted average for parameter migration, where the weight of the main network parameters is the soft update coefficient, and the weight of the target network historical parameters is 1 minus the soft update coefficient.

[0064] The aforementioned experience replay buffer can be understood as a first-in-first-out queue structure used in this invention to store state transition samples generated during the interaction between the agent and the AC / DC hybrid distribution network, with its capacity upper limit being the preset maximum buffer capacity; the aforementioned exploration noise can be understood as a Gaussian perturbation signal used in this invention to enhance the policy exploration capability, with its standard deviation monotonically decreasing with the number of training steps to achieve a balance between sufficient exploration in the early stage and fine convergence in the later stage; the aforementioned soft update method can be understood as a gradual parameter synchronization mechanism used in this invention to stabilize the rate of change of target network parameters, avoiding value estimation distortion caused by drastic fluctuations in target values.

[0065] The training batch size, maximum buffer capacity, initial standard deviation, decay coefficient, and soft update coefficient mentioned in this section can be set according to the actual needs of relevant technical personnel, combined with the typical hyperparameter settings in the deep deterministic policy gradient algorithm or the double-delay deep deterministic policy gradient algorithm. This invention does not limit these settings.

[0066] Example 2, refer to Figures 2-3 , Figure 2 The diagram shows a typical AC / DC hybrid distribution network, which is usually composed of new energy power generation equipment such as wind power and photovoltaics, power electronic converters such as AC / DC and DC / DC, energy storage, DC loads, AC loads, and other equipment.

[0067] Optimized scheduling is a crucial element in fully leveraging the advantages of AC / DC distribution networks. Reasonable optimized scheduling methods can improve system economy, enhance supply-demand balance, promote renewable energy utilization, and improve system flexibility and adaptability. The day-ahead optimization scheduling problem for AC / DC hybrid distribution networks is a high-dimensional, nonlinear, and highly uncertain mixed-integer programming problem. While traditional methods (such as MILP and SOCP) are mature, they have the following inherent limitations: (1) High dependence on prediction accuracy: The optimization results depend heavily on the accuracy of new energy and load prediction. Prediction deviation will significantly reduce the economy and security of the scheduling scheme.

[0068] (2) Model mismatch problem: Linearization or convexification based on physical models can introduce errors and make it difficult to accurately describe the dynamic and nonlinear characteristics of all devices (such as complex power electronic converters).

[0069] (3) The contradiction between computational efficiency and scale: For large-scale, multi-level networks, traditional optimization methods take a long time to solve and are difficult to adapt to rolling optimization with higher real-time requirements.

[0070] Reinforcement learning, an artificial intelligence method, can autonomously learn optimal strategies through interaction with the environment, enabling online dynamic decision-making without requiring precise physical models. However, directly applying reinforcement learning to AC / DC hybrid distribution network dispatching faces challenges such as unstable training, difficulty in guaranteeing security, and poor interpretability. To address these issues, this invention provides a safe, efficient, and practical reinforcement learning-based optimization scheme for AC / DC hybrid distribution network operation, aiming to optimize system operating costs and overall performance.

[0071] like Figure 3 As shown, the invented AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method combines the perception and decision-making capabilities of reinforcement learning with the safety verification capabilities of physical models by constructing an "offline training-online rolling" framework, thereby achieving complementary advantages between data-driven and model-driven approaches.

[0072] Example 3, referring to Figure 4 This embodiment also provides an AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization system, including: The status acquisition module is used to collect the current operating status of the AC / DC hybrid distribution network according to a set period. The strategy network module is used to receive the current operating status and output normalized continuous control commands; The instruction conversion module is used to convert normalized continuous control instructions into actual physical control quantities based on the maximum allowable power limit of the corresponding equipment. The control execution module is used to send actual physical control quantities to controllable devices in the AC / DC hybrid distribution network; The training module is used to train the Actor-Critic reinforcement learning system, which consists of a policy network module and a value network module, offline using historical running data or simulated running scenarios. The value network module is used to receive state feature vectors and normalized continuous control commands, and output state action value estimates. The training module is also used to build an experience replay buffer, store state transition samples, and randomly draw batches of samples from the experience replay buffer to update the parameters of the policy network module and the value network module. The training module is also used to configure the target policy network and the target value network, and to gradually synchronize the parameters of the target policy network and the target value network from the parameters of the policy network module and the value network module through soft updates.

[0073] The above-mentioned unit modules can be embedded in the processor of the electronic device in hardware form or independent of it, or they can be stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0074] This embodiment also provides an electronic device, which can be a terminal, and its internal structure diagram can be as follows: Figure 4 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an AC / DC hybrid power distribution network Actor-Critic reinforcement learning optimization method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the device's casing, or an external keyboard, touchpad, or mouse.

[0075] This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it performs the following steps: A Markov decision process model is constructed to describe the operation of a hybrid AC / DC distribution network. The Markov decision process model includes a state space, an action space, and a reward function. Actor-Critic reinforcement learning system consisting of a policy network and a value network is constructed based on the Markov decision process model. The Actor-Critic reinforcement learning system is trained offline using historical operating data or simulated operating scenarios to obtain the trained policy network; During the online operation phase, the current operating status of the AC / DC hybrid distribution network is collected according to a set cycle; Input the current running state into the trained policy network and output normalized continuous control commands; The normalized continuous control commands are converted into actual physical control quantities based on the maximum allowable power limit of the corresponding equipment. The actual physical control quantities are sent to the controllable devices in the AC / DC hybrid distribution network.

[0076] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0077] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0078] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for optimizing the operation of AC / DC hybrid distribution networks using Actor-Critic reinforcement learning, characterized in that, include: A Markov decision process model is constructed to describe the operation of an AC / DC hybrid distribution network. The Markov decision process model includes a state space, an action space, and a reward function. Based on the Markov decision process model, an Actor-Critic reinforcement learning system consisting of a policy network and a value network is constructed. The Actor-Critic reinforcement learning system is trained offline using historical operating data or simulated operating scenarios to obtain a trained policy network; During the online operation phase, the current operating status of the AC / DC hybrid distribution network is collected according to a set cycle; The current operating state is input into the trained policy network, and normalized continuous control commands are output. The normalized continuous control command is converted into an actual physical control quantity according to the maximum allowable power limit of the corresponding device. The actual physical control quantities are sent to the controllable devices in the AC / DC hybrid distribution network.

2. The Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks as described in claim 1, characterized in that, The state space includes: The data includes: forecasted photovoltaic power generation for multiple future time periods; forecasted wind power generation for multiple future time periods; measured photovoltaic power generation for multiple recent time periods; measured wind power generation for multiple recent time periods; forecasted AC load power for multiple future time periods; forecasted DC load power for multiple future time periods; measured AC load power for multiple recent time periods; measured DC load power for multiple recent time periods; electricity purchase price for multiple future time periods; electricity sales price for multiple future time periods; current state of charge of each energy storage device; average state of charge of all energy storage devices; network topology represented by unique thermal coding; line current measurement; hourly markers for a day; day of the week markers; and seasonal markers.

3. The AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method as described in claim 2, characterized in that, The action space includes normalized continuous control instructions, which include: Normalization commands for power exchanged with the main grid, normalization commands for active power of each AC-DC converter, normalization commands for reactive power of each AC-DC converter, normalization commands for charging and discharging power of each energy storage device, and normalization commands for output of each controllable distributed power source.

4. The Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks as described in claim 3, characterized in that, The reward function is a weighted combination of economic reward items, operational safety penalty items, new energy consumption reward items, and control action change smoothness penalty items; The economic incentive is calculated based on the purchased power, sold power, distributed power generation cost, and network loss cost. The operational safety penalty items are calculated based on the degree of voltage over-limit, current over-limit, and the degree of over-limit of the energy storage device's state of charge. The new energy consumption incentive is calculated based on the difference between the actual power generation of new energy equipment and the theoretical maximum power generation. The control action change smoothness penalty term is calculated based on the change range between the current control action and the control action at the previous moment.

5. The AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method as described in claim 4, characterized in that, The policy network receives the state feature vector processed by the state feature extraction network, outputs the Gaussian distribution parameters corresponding to each control dimension through multiple parallel branches, and generates normalized continuous control commands based on the Gaussian distribution parameters. The state feature extraction network is a multilayer perceptron structure.

6. The AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method as described in claim 5, characterized in that, The value network receives the state feature vector and the normalized continuous control command, and outputs the corresponding state action value estimate. The parameters of the value network are updated by minimizing the temporal difference error; The target value of the time-series difference error is calculated jointly by the target value network and the target policy network.

7. The AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization method as described in claim 6, characterized in that, During the offline training phase, an experience replay buffer is constructed to store state transition samples generated during the interaction between the agent and the AC / DC hybrid power distribution network. A batch of samples is randomly drawn from the experience replay buffer to update the parameters of the policy network and the value network; Gaussian exploration noise is superimposed on the control actions output by the policy network, and the standard deviation of the exploration noise gradually decreases with the number of training steps; Set up a target policy network with the same structure as the policy network, and a target value network with the same structure as the value network; The parameters of the target policy network and the parameters of the target value network are gradually synchronized from the corresponding main network parameters through a soft update method, wherein the soft update method uses a fixed-ratio weighted average for parameter migration.

8. An AC / DC hybrid distribution network Actor-Critic reinforcement learning operation optimization system, using the method described in any one of claims 1 to 7, characterized in that, include: The status acquisition module is used to collect the current operating status of the AC / DC hybrid distribution network according to a set period. The strategy network module is used to receive the current operating state and output normalized continuous control commands; The instruction conversion module is used to convert the normalized continuous control instruction into an actual physical control quantity according to the maximum allowable power limit of the corresponding device; The control execution module is used to send the actual physical control quantities to controllable devices in the AC / DC hybrid power distribution network. The training module is used to train the Actor-Critic reinforcement learning system, which consists of a policy network module and a value network module, offline using historical running data or simulated running scenarios. The value network module is used to receive state feature vectors and normalized continuous control commands, and output state action value estimates. The training module is also used to construct an experience replay buffer, store state transition samples, and randomly extract batch samples from the experience replay buffer to update the parameters of the policy network module and the value network module. The training module is also used to configure the target policy network and the target value network, and to gradually synchronize the parameters of the target policy network and the target value network from the parameters of the policy network module and the value network module through a soft update method.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the Actor-Critic reinforcement learning operation optimization method for AC / DC hybrid distribution networks as described in any one of claims 1 to 7.

Images