Deep reinforcement learning-based power distribution network agent group collaborative regulation method and system

Abstract

The application provides a power distribution network intelligent agent group cooperative regulation method and system based on deep reinforcement learning, comprising the following steps: in an active power distribution network, taking a power grid node or a control unit as an intelligent agent; acquiring real-time state information of different intelligent agents in the power grid, at least including voltage, current and power; based on the real-time state information, a neural network is used to represent a Q value function to evaluate an expected cumulative reward of taking a load management action under a given smart grid state, wherein a single intelligent agent obtains an optimal energy distribution strategy by maximizing the Q value function; a Maldives network communication model is constructed, and information exchange is realized between multiple intelligent agents based on the communication model; and an intelligent agent group composed of the multiple intelligent agents uses a deep deterministic policy gradient algorithm to make a decision on power grid energy distribution.

Description

Technical Field

[0001] This invention belongs to the field of power distribution network control technology, and particularly relates to a method and system for collaborative control of intelligent swarms in power distribution networks based on deep reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the increasing complexity of power systems, traditional single-agent control can no longer meet the needs of efficient management and stable operation of the system. Multi-agent collaborative control technology has gradually become a key means of power system management.

[0004] Intelligent management and optimized control of power systems can be achieved through cooperation and collaboration among multiple agents. However, such systems mainly rely on manual rule formulation and experience accumulation, which has limitations when facing increasingly complex power systems and rapidly changing load demands. To avoid the limitations of traditional methods, artificial intelligence, relying on its self-learning and adaptive characteristics, is being applied to multi-agent collaborative regulation.

[0005] Common collaborative control methods include distributed control, game theory, and swarm intelligence algorithms. Multi-agent collaborative control methods based on distributed control strategies decompose the entire system into multiple subsystems or agents, assigning corresponding control tasks to each. These agents independently perform control and decision-making according to their respective tasks and roles, achieving collaborative control of the overall system through communication or cooperation mechanisms. Multi-agent collaborative control methods based on game theory use cooperative game models to describe the cooperative relationships between agents. By analyzing the cooperation strategies and payoff distribution among agents through game models, collaborative control of the multi-agent system can be achieved. In situations with high load demand and complex power systems, the effectiveness of multi-agent swarm control largely depends on the choice of algorithm. Therefore, under these conditions, multi-agent swarm collaborative control methods are needed to effectively handle distributed learning and communication among multiple agents, thereby constructing a collaborative control model.

[0006] Furthermore, most current intelligent swarm collaborative control schemes that combine artificial intelligence often only use a single artificial intelligence algorithm or model, and their control accuracy needs to be improved, and they also have limitations such as poor fault tolerance. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a collaborative control method for distribution network intelligent agents based on deep reinforcement learning. In the smart grid, deep reinforcement learning technology is used to realize the collaborative scheduling and optimization among multiple intelligent agents to cope with complex situations such as grid load fluctuations and uneven energy distribution.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] Firstly, a method for collaborative control of intelligent swarms in distribution networks based on deep reinforcement learning is disclosed, including:

[0010] In active power distribution networks, grid nodes or control units within the grid are considered as intelligent agents.

[0011] Acquire real-time status information of different intelligent agents in the power grid, including at least voltage, current, and power;

[0012] Based on real-time state information, the expected cumulative reward for taking load management actions under a given smart grid state is evaluated by using a neural network to represent the Q-value function. In this process, a single agent obtains the optimal energy allocation strategy by maximizing the Q-value function.

[0013] A network communication model for the Maldives was constructed, and information exchange was realized between multiple agents based on this communication model;

[0014] A swarm of agents, consisting of multiple agents, uses a deep deterministic policy gradient algorithm to make decisions on power grid energy allocation.

[0015] As a further technical solution, a deep Q-network reinforcement learning model is used to store real-time state information of the power grid in an empirical memory, and batch samples are randomly selected from the memory to update the model parameters.

[0016] As a further technical solution, the deep Q-network reinforcement learning model uses a separate target network to generate a target Q value that reflects the energy stability of the power grid. During the learning process, the model adopts a delayed update method, copying the real-time state value of the current power grid node to the target network every C steps to update the real-time state of the target node.

[0017] Using the node energy state c as input to the online network, the actual Q value of action a is obtained. Using the next state as input to the objective function, the maximum Q value of all actions in the target network is obtained.

[0018] As a further technical solution, for the deep Q-network reinforcement learning model, a deep convolutional neural network is used to fit the optimal action value function during the training process. This function follows the Bellman equation.

[0019] The training process adopts the Mini-batch training method. In each round of training, M experience actions are randomly selected from the experience memory containing real-time state information of different agents in the power grid.

[0020] As a further technical solution, at the start of training, a single agent randomly selects a node energy state to explore the effects of unknown actions, thereby obtaining a better strategy;

[0021] Throughout the training process, the core AI gradually increases the probability of selecting the energy allocation value with the highest Q value, which is the most stable for the power grid, and uses the stochastic gradient descent algorithm to update the network parameters to obtain the optimal Q value.

[0022] As a further technical solution, the constructed network communication model for the Maldives includes:

[0023] Using formulas, we can express how intelligent agents can cooperate to solve constrained optimization problems.

[0024] Based on the above expression, the estimated value of agent i after the change is obtained;

[0025] Based on the estimated value expression, a distributed multi-agent subgradient algorithm for power grid information exchange among multiple agents is obtained.

[0026] A probabilistic communication model is obtained based on the gradient algorithm.

[0027] As a further technical solution, the deep deterministic policy gradient algorithm includes two deep neural networks, namely the Actor network and the Critic network.

[0028] The Actor network, representing the current state of the regional smart grid, aims to learn the optimal deterministic policy.

[0029] As a power grid energy strategy function composed of multiple nodes, the Actor generates output energy output actions given the current state.

[0030] The Critic network serves as a function to evaluate actions to be taken in a given grid state, and its goal is to learn the optimal energy allocation action value function.

[0031] The Critic evaluates the energy allocation action value function of an agent swarm based on the Actor's output and current state. The time difference generated by the Critic drives the learning process in the Actor network and the Critic network, thereby evaluating the quality of the policy.

[0032] By minimizing the loss to update the Critic and Actor networks, the weights of the intelligent agent swarm can be updated and coordinated.

[0033] As a further technical solution, both the Actor network and the Critic network adopt a multilayer sensor design when making decisions on grid energy allocation;

[0034] The Actor takes the swarm state of the agents as input and outputs an action in the Actor network. This action, along with the swarm state, is fed into the Critic network.

[0035] The state-action values, especially the Q-value and the target Q-value, are generated by two sets of Critic networks with different weights, namely the critic weights and the target critic weights.

[0036] The loss is calculated based on the state-action value and used to update the Critic's weights.

[0037] Secondly, a smart swarm collaborative control system for an active power distribution network is disclosed, comprising:

[0038] The single-agent optimal energy allocation module is configured to treat grid nodes or control units as agents in an active power distribution network.

[0039] Acquire real-time status information of different intelligent agents in the power grid, including at least voltage, current, and power;

[0040] Based on real-time state information, the expected cumulative reward for taking load management actions under a given smart grid state is evaluated by using a neural network to represent the Q-value function. In this process, a single agent obtains the optimal energy allocation strategy by maximizing the Q-value function.

[0041] The collaborative control module is configured to: construct a Maldives network communication model, based on which multiple agents exchange information;

[0042] An intelligent agent swarm consisting of multiple agents uses a deep deterministic policy gradient algorithm to collaboratively regulate the power grid energy distribution.

[0043] The above one or more technical solutions have the following beneficial effects:

[0044] The technical solution of this invention uses a deep reinforcement learning algorithm to build a model. The intelligent agent swarm can perceive the power grid status in real time and make collaborative decisions through information exchange to determine the power grid energy allocation strategy.

[0045] The technical solution of this invention is based on deep reinforcement learning algorithms. It sequentially utilizes deep Q-network algorithm, distributed projection multi-agent subgradient algorithm, communication protocol, and deep deterministic policy gradient algorithm to process the state and behavior of agents. Under complex conditions such as load fluctuations and uneven energy distribution in smart grids, the group of agents can work collaboratively to solve problems by optimizing decisions and adjusting control strategies, thereby improving the stability and efficiency of the grid, reducing operating costs, and providing an efficient and feasible solution for practical application scenarios of smart grids.

[0046] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0048] Figure 1 This is a schematic diagram of the deep Q-network algorithm structure according to an embodiment of the present invention;

[0049] Figure 2 This is a structural diagram of the multi-agent network communication model according to an embodiment of the present invention;

[0050] Figure 3 This is a diagram of the intelligent swarm collaborative regulation and update structure in an embodiment of the present invention. Detailed Implementation

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0053] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0054] Example 1

[0055] This embodiment discloses a method for collaborative control of intelligent swarms in distribution networks based on deep reinforcement learning, including:

[0056] First, a deep Q-network algorithm is used to build a model. In this model, each agent represents a Q-value function through a neural network. This function evaluates the expected cumulative reward obtained by taking a specific action under a given power grid condition. Maximizing the Q-value guides the agent to find the optimal energy allocation strategy.

[0057] Secondly, based on the established model, the projective multi-agent subgradient algorithm is studied and adopted to facilitate information exchange between individual agents and other agents through randomly changing network topology. In a power grid environment, this ensures that when energy supply and demand are imbalanced in a certain region, information can be shared rapidly, enabling collaborative perception and decision-making among multiple agents.

[0058] Finally, this invention also employs a deep deterministic policy gradient algorithm to learn the policy of the intelligent agent swarm in a continuous action space. When the smart grid undergoes continuous changes, the intelligent agent swarm learns the policy function through an Actor network to maximize cumulative rewards, and evaluates the action value of the Actors through a Critic network to guide policy updates, enabling the intelligent agent swarm to adjust its control strategy in real time and maintain the stable operation of the power grid.

[0059] The Deep Q-Network algorithm is a reinforcement learning method that uses neural networks to estimate Q-values, thereby improving decision-making efficiency. The detailed model building process and processing methods are shown in formulas (1), (2), and (3), and can also be simply expressed as:

[0060] Where Q(s,a) represents the Q-value of performing action a in state s, r is the observed reward, γ is the discount factor, and α is the learning factor.

[0061] In one or more implementation examples, the specific steps to achieve regulation include:

[0062] For intelligent agents, a deep Q-network reinforcement learning model is established;

[0063] The voltage, current, and power of different nodes in the power grid are obtained based on the model;

[0064] The voltage, current, and power information of different nodes in the power grid obtained above are used for decision-making by a single agent. With the support of this data, the performance and efficiency of decision-making can be optimized. The specific effects are as follows:

[0065] Voltage: By acquiring voltage information from different nodes, a single intelligent agent can determine the voltage stability of the nodes and take corresponding measures, such as adjusting the power output or controlling the load, to keep the grid voltage within a suitable range.

[0066] Current: A single intelligent agent can use the current information of different nodes to identify the load conditions and potential congestion points in the power grid, and then take corresponding measures, such as adjusting energy distribution or optimizing line configuration, to ensure that the current load is within an acceptable range and prevent the power grid from overloaded or short-circuited.

[0067] Power: A single agent can use power information from different nodes to assess the energy usage and energy flow paths of the power grid in order to decide on the optimal energy allocation strategy. For example, under high load conditions, the agent can adjust the power output of distributed power sources or activate energy storage systems to balance the load on the power grid.

[0068] Regarding the Projective Multi-Agent Subgradient Algorithm: The algorithm's inputs include: power grid topology information, specifically including the connection relationships of each node in the power grid, line parameters, and the location and parameters of potentially controllable devices (such as generators and energy storage systems) (i.e., obtaining voltage, current, and power data for different nodes in the power grid based on the model); energy supply and demand information, including the energy supply and demand situation of each node, such as load demand and the capacity of distributed energy sources; and system constraints, such as voltage stability, line transmission limitations, and equipment capacity.

[0069] Calculation process: As shown in formulas (4)-(8).

[0070] The algorithm outputs the following: Agent Decisions: Each agent makes corresponding decisions based on the received information and the global subgradient direction calculated by the projected multi-agent subgradient algorithm, including adjusting generator output, starting or stopping the energy storage system, and adjusting the load. Communication Information: Based on their own decision results, agents generate communication information using the global subgradient direction calculated by the projected multi-agent subgradient algorithm to exchange information with other agents.

[0071] The output information obtained by this algorithm is used to share among agents in the future to achieve coordination between agents.

[0072] Regarding the Deep Deterministic Policy Gradient Algorithm: The inputs are: a state space, including grid state information exchanged between multiple agents; an action space, the range of actions an agent can perform; a reward function, defined based on the grid operation goals, used to evaluate the behavior of the agent swarm; and initial parameters for the Actor and Critic networks.

[0073] The calculation process of the algorithm is as shown in formulas (10)-(16), and the output is the updated Actor network and Critic network parameters. The updated Actor network and Critic network parameters output by the algorithm are used to adjust the collaborative control strategy of the intelligent swarm in real time to maintain the stable operation of the power grid.

[0074] Regarding decision-making by a single agent:

[0075] In the field of machine learning, deep neural networks are adept at capturing features and patterns in complex environments. Meanwhile, reinforcement learning maps the current state to corresponding actions, evaluates action values ​​based on expected rewards, and learns sequences of decision-making actions corresponding to scene features to solve complex decision-making problems. Deep reinforcement learning algorithms combine these two techniques, providing an effective solution to complex system problems involving perception and decision-making.

[0076] In an active power distribution network, each agent represents a grid node or control unit. A deep Q-network algorithm is used to build a model to perceive and acquire real-time state information such as voltage, current, and power of different nodes in the grid. The Q-value function is represented by a neural network to evaluate the expected cumulative reward of taking actions such as load management under a given smart grid state. A single agent finds the optimal energy allocation strategy by maximizing the Q-value function.

[0077] Regarding experience storage:

[0078] The deep Q-network reinforcement learning model stores real-time node energy information in the smart grid into an empirical memory, and then updates the model parameters using a mini-batch training method (randomly selecting batches of samples from the memory). During training, the model uses a deep convolutional neural network to fit the optimal action-value function, which follows the Bellman equation, defined as follows:

[0079]

[0080] In the formula, s and s′ represent the real-time energy state and the next energy state of a node in the smart grid, respectively. a and a′ represent the real-time action and the next action of the agent to regulate the energy of the grid node, respectively. The core agent's behavior strategy π=P(s|a) represents the probability that the core agent will choose action a in state s.

[0081] The training process employs a mini-batch training method, where M experience actions D are randomly selected from an experience memory containing real-time information about the smart grid in each training round. i ={e1, e2, ..., e i}

[0082] Fixed Q function

[0083] The deep Q-network reinforcement learning model uses a separate target network to generate a target Q-value reflecting the energy stability of the power grid, with the same structure as the evaluation network. During the learning process, the model employs a delayed update method, copying the real-time state value of the current power grid node to the target network every C steps to update the real-time state of the target node. The model uses the node energy state c as input to the online network to obtain the actual Q-value of action a, and uses the next state s′ as input to the objective function to obtain the maximum Q-value of all actions in the target network. This example uses the mean squared error function, defined as follows:

[0084]

[0085] In the formula, the discount factor γ∈[0,1], θ i This represents the node state parameters in the online network during the i-th iteration. This represents the node state parameters of the target network at the i-th iteration.

[0086] At the start of training, a single agent randomly selects a node's energy state to explore the effects of unknown actions, thereby obtaining a better strategy. Throughout the training process, the core agent gradually increases the probability of selecting the action with the highest Q value (i.e., the energy allocation value with the highest grid stability) to utilize known effective strategies more frequently to obtain high rewards. Therefore, the stochastic gradient descent algorithm can be used to update the network parameters θ to obtain the maximum Q value. The optimal Q value is the maximum Q value, and the gradient calculation formula is as follows:

[0087]

[0088] The values ​​can be calculated using a neural network algorithm. The network parameters are updated using a delayed update method, whereby the node energy parameter values ​​of the current online network are copied to the target network every C steps to update the target network parameters.

[0089] The schematic diagram of the deep Q-network algorithm is shown below. Figure 1 As shown. Information exchange between intelligent agents.

[0090] The single-agent deep Q-network reinforcement learning model constructed above acquires the real-time state information of each node in the smart grid, enabling the single agent to find the optimal energy allocation strategy. This section discusses exchanging information with other agents through a randomly changing network topology to update the information state among different agents. In a power grid environment, this means that when an energy shortage or surplus occurs in a certain area, relevant information can be quickly transmitted to multiple agents.

[0091] To better understand the technical solution of this embodiment, the concept of the Q-value function is explained as follows: The Q-value function defines the expected cumulative reward that can be obtained after taking an action a in a certain state s, i.e., Q(s,a). The higher the Q-value, the greater the cumulative reward that can be obtained by taking the action in the current state, that is, the more likely the action is to bring better results.

[0092] In reinforcement learning, the core agent learns the expected long-term rewards of choosing different actions in each state by continuously interacting with the environment. These expected rewards are called Q-values. Q-values ​​guide the decision-making process, and the optimal Q-value (maximum Q-value) corresponds to the strategy with the highest expected reward in the above process, which is also the optimal energy allocation strategy.

[0093] Assume a state-dependent communication model in this topology: communication is Markovian in terms of the state of a single agent, and the probability of connection availability depends on the agent's state. Based on this, a projective multi-agent subgradient algorithm for state-dependent communication is presented. This algorithm combines energy information exchange between different nodes with subgradient steps and projection error analysis to achieve information exchange and collaboration among multiple agents, thereby enabling joint perception and decision-making regarding the state of the power grid system.

[0094] Distributed multi-agent subgradient method

[0095] In the network, each agent i maintains a state of processing the optimal solution for node energy regulation and updates it based on local energy flow information between its grid nodes, exchanging information with neighboring agents. Each agent i has an initial energy estimate x. i (0)∈X i Every k time intervals, agent i changes its estimated value as follows:

[0096]

[0097] in This represents the energy constraints on agent i at node X. i The projection on the vector [a] ij (k) j∈M Let d be the weight vector of agent i, scalar α(k) > 0 be the step size at time k, and vector d be... i (k) is the target energy operation function f of agent i. i (x) at its estimated location

[0098] The subgradient.

[0099] Therefore, to generate a new estimate, each agent i combines the node state energy information received from other agents with steps along the sub-gradient of its objective function, and projects the resulting vector onto its constraint set to maintain feasibility. We call this algorithm the Projective Multi-Agent Subgradient Algorithm.

[0100] In the analysis of this algorithm, to facilitate the separation of the different operations used to generate the new estimate in the update formula (5), formula (5) is equivalently rewritten as:

[0101]

[0102] x i (k+1)=v i (k)-α(k)d i (k)+e i (k) (7)

[0103]

[0104] The above decomposition can be based on the estimation of other agents, the subgradient step, and the projection error e. i A new estimate is generated using a linear update rule. Therefore, the nonlinear effect of the projection operation is represented by the projection error vector e. i This means that the error vector can be viewed as a perturbation of the subgradient step of the algorithm.

[0105] Network communication model

[0106] Based on the distributed multi-agent subgradient algorithm for realizing power grid information exchange among multiple agents given above, the communication matrix of the network at time k is defined as A(k) = [a ij (k)] i，j∈M Based on the defined network communication matrix, this invention establishes a probabilistic communication model.

[0107] The sequence of the communication matrix A(k) is in the state variable x(k) = [x i (k)] i∈M ∈R n×m It possesses the Markov property. Let [n(k)} k∈N It is defined in probability space Let {Ω′, F′, P′} be an independent sequence of random variables. k} k∈N This forms a sequence with the same probability space. This Markov communication model allows us to capture the set of communications when the ability of agents to communicate with each other depends on their current estimates.

[0108] The Markov communication model mainly consists of two parts: the state transition of the Markov chain and the observation model. Assume the Markov chain has N states (S1, S2, S3, ... S...). N The state transition probability matrix is ​​(PP), where P ij ) indicates from state (S) i ) transition to state (S) j The probability of state transition can be expressed as:

[0109] P ij =P(X) t+1 =S j |X t =S i (9)

[0110] Among them, X t This represents the state at time t.

[0111] Assume the observation space has M possible observations (O1, O2, O3, ..., O M The observation probability matrix is ​​O, where O ij Indicates that in state S i The following observations were made. j The probability of . Then the observation model formula can be expressed as:

[0112] O ij =P(Y t =O j |X t =S i (10)

[0113] Among them, Y t This represents the observation at time t.

[0114] If the estimates of two agents differ significantly, the probability of communication between them may be very low. The concept can be modeled as follows: for all (j, i) ∈ ε, k ≥ 0 and

[0115]

[0116] In the formula, K and C are positive real constants, and 0 < δ, γ ≤ 1. The parameter δ was added to the model to... The probability of communication is determined when the value is relatively small.

[0117] This model shows that for any two nodes i and j, there is a boundary if x i (k) and x jIf the estimates of (k) are close to each other, then the probability that they communicate at time k is at least δ. Then, if the two agents are far apart, the probability that they communicate can only be expressed as a polynomial ||x|| between the distances of their estimates. i (k)-x j The reciprocal of (k)|| is the boundary.

[0118] Multi-agent network communication model structure diagram, such as Figure 2 As shown.

[0119] Intelligent swarm collaborative regulation

[0120] Based on the Markov network communication model constructed above, which belongs to the probabilistic communication model, information exchange is realized among multiple agents. In this part, the present invention proposes to use a deep deterministic policy gradient algorithm to guide the decision-making of power grid energy allocation by an agent swarm composed of multiple agents. In this way, when uneven energy distribution occurs in the power grid, the agent swarm can adjust the collaborative control strategy in real time, such as redistributing power grid energy and adjusting strategies, to maintain the stable operation of the power grid.

[0121] Deep deterministic policy gradient algorithm

[0122] The Deep Deterministic Policy Gradient Algorithm (DPRQ) is an extension of the Deterministic Policy Gradient Algorithm (DPRQ), combining the ideas of deep neural networks and experience replay buffers. It is a swarm reinforcement learning algorithm for solving continuous action space problems.

[0123] The core idea of ​​the deep deterministic policy gradient algorithm is to utilize two deep neural networks: an Actor network and a Critic network. The Actor network represents the current state of the regional smart grid, and its goal is to learn the optimal deterministic policy. The Actor network represents the grid energy policy function μ(s|θ) composed of multiple nodes. μ Given the current state, the Actor generates an output energy action; while the Critic network, as a function to evaluate actions taken given the grid state, aims to learn the optimal energy allocation action value function. The Critic evaluates the energy allocation action value function Q(s, a|θ) of a swarm of agents based on the Actor's output and the current state. Q The time difference generated by the Critic drives the learning process in the Actor network and the Critic network, thereby evaluating the quality of the strategy.

[0124] By training these two networks simultaneously, the swarm of agents can progressively improve its policy, enhancing its ability to make better decisions in a continuous action space. This means the swarm will be able to explore its environment more effectively and obtain higher rewards. The core of the algorithm is to use a stochastic method to explore good behavior in the swarm and estimate a deterministic behavioral policy. By doing so, it only requires integration in the state space, making policy learning easier, as shown in the following formula:

[0125] a t =μ(s) t |θ μ (12)

[0126] However, it also has limitations; the algorithm may not be able to explore the complete state and operation space of the swarm of agents. To overcome this limitation, we introduce a truncated disturbance N in our random exploration study, as shown in the following formula:

[0127]

[0128] Both Actor and Critic are designed using neural networks. The deterministic policy gradient theorem provides the update rules for the Actor network, while the Critic network updates from the gradient in the time difference, which can be expressed as:

[0129]

[0130] As shown in the formula above, in order to obtain the expectation of the intelligent swarm, we only need to obtain the gradient of the Actor network with respect to its parameters and the gradient of the Critic network with respect to the actions.

[0131] Coordinated Update Regulation

[0132] The algorithm described above updates the Critic and Actor networks by minimizing the loss, achieving weight updates and coordinated control of the swarm of agents. Under complex conditions of uneven energy distribution in the power grid, the formula for real-time updates of the Critic network by the swarm of agents is shown below:

[0133]

[0134] in

[0135] The algorithm uses a sampling strategy gradient to update the Actor network, as shown in the following formula:

[0136]

[0137] Based on the steps above, the target updates for the Actor network and the Critic network are as follows:

[0138] θQ′ ←τ θ Q+(1-τ)θ Q′ (17)

[0139] θ μ′ ←τθ μ +(1-τ)θ μ′ (18)

[0140] Here, τ is the update parameter, and its value is much less than 1.

[0141] The intelligent swarm notices changes in actions and the environment, and relies on the outputs of the Actor network and Critic network to update the objective function, ultimately achieving coordinated regulation.

[0142] Intelligent swarm collaborative regulation and update structure diagram:

[0143] In this invention, both the Actor network and the Critic network employ a multilayer perceptron (MLP) design. For example... Figure 3 As shown, the Actor network takes the swarm state as input and outputs an action, which, along with the swarm state, is fed into the Critic network. The state-action value, specifically the Q-value Q(s, a|θ), is represented. Q ) and the target Q value Q(s, a|θ) Q′ The Critic network is generated from two sets of Critic weights: the Critic weights and the target Critic weights. The loss is calculated based on the state-action values ​​and used to update the Critic weights.

[0144] Example 2

[0145] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0146] Example 3

[0147] The purpose of this embodiment is to provide a computer-readable storage medium.

[0148] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0149] Example 4

[0150] The purpose of this embodiment is to provide an intelligent swarm collaborative control system for active power distribution networks, including:

[0151] The single-agent optimal energy allocation module is configured to treat grid nodes or control units as agents in an active power distribution network.

[0152] Acquire real-time status information of different intelligent agents in the power grid, including at least voltage, current, and power;

[0153] Based on real-time state information, the expected cumulative reward for taking load management actions under a given smart grid state is evaluated by using a neural network to represent the Q-value function. In this process, a single agent obtains the optimal energy allocation strategy by maximizing the Q-value function.

[0154] The collaborative control module is configured to: construct a Maldives network communication model, based on which multiple agents exchange information;

[0155] An intelligent agent swarm consisting of multiple agents uses a deep deterministic policy gradient algorithm to collaboratively regulate the power grid energy distribution.

[0156] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0157] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0158] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for collaborative control of intelligent swarms in distribution networks based on deep reinforcement learning, characterized by: include: In active power distribution networks, grid nodes or control units are treated as intelligent agents; Acquire real-time status information of different intelligent agents in the power grid, including at least voltage, current, and power; Based on real-time state information, the expected cumulative reward of taking load management actions under a given smart grid state is evaluated by using a neural network to represent the Q-value function. In this process, a single agent obtains the optimal energy allocation strategy by maximizing the Q-value function. Secondly, based on the established model, the projection multi-agent subgradient algorithm is used to facilitate the exchange of information between individual agents and other agents through randomly changing network topology; A Markov network communication model was constructed, and information exchange was realized between multiple agents based on this communication model; The construction of the Markov network communication model includes: expressing that agents can cooperate to solve constrained optimization problems; and obtaining the agents based on the above expression. i The modified estimate expression; based on the estimate expression, a distributed multi-agent subgradient algorithm for power grid information exchange among multiple agents is obtained; a probabilistic communication model is obtained based on the gradient algorithm; An agent swarm consisting of multiple agents uses a deep deterministic policy gradient algorithm to make decisions on grid energy allocation; the deep deterministic policy gradient algorithm includes two deep neural networks, namely the Actor network and the Critic network; The Actor network represents the current state of the regional smart grid, and its goal is to learn the optimal deterministic policy. The Actor, as a grid energy policy function composed of multiple nodes, generates output energy output actions given the current state. The Critic network serves as a function to evaluate actions taken in a given power grid state, with the goal of learning the optimal energy allocation action value function. The Critic evaluates the energy allocation action value function of a swarm of agents based on the Actor's output and the current state. The time difference generated by the Critic drives the learning process in the Actor network and the Critic network, thereby evaluating the quality of the policy. By minimizing the loss to update the Critic and Actor networks, the weights of the intelligent agent swarm can be updated and coordinated.

2. The method for collaborative control of distribution network intelligent swarm based on deep reinforcement learning as described in claim 1, characterized in that, A deep Q-network reinforcement learning model is used to store real-time state information of the power grid in an empirical memory, and batch samples are randomly selected from the memory to update the model parameters.

3. The method for collaborative control of distribution network intelligent swarm based on deep reinforcement learning as described in claim 1, characterized in that, The deep Q-network reinforcement learning model uses a separate target network to generate a target Q value that reflects the energy stability of the power grid. During the learning process, the model adopts a delayed update method, copying the real-time state value of the current power grid node to the target network every C steps to update the real-time state of the target node. Using the node energy state c as input to the online network, the actual Q value of action a is obtained. Using the next state as input to the objective function, the maximum Q value of all actions in the target network is obtained.

4. The method for collaborative control of distribution network intelligent swarm based on deep reinforcement learning as described in claim 1, characterized in that, For deep Q-network reinforcement learning models, a deep convolutional neural network is used to fit the optimal action value function during training. This function follows the Bellman equation. The training process adopts the Mini-batch training method. In each round of training, M experience actions are randomly selected from the experience memory containing real-time state information of different agents in the power grid.

5. The method for collaborative control of intelligent swarms in distribution networks based on deep reinforcement learning as described in claim 1, characterized in that, At the start of training, a single agent randomly selects a node energy state to explore the effects of unknown actions, thereby obtaining a better strategy; Throughout the training process, the core AI gradually increases the probability of selecting the energy allocation value with the highest Q value, which is the most stable for the power grid, and uses the stochastic gradient descent algorithm to update the network parameters to obtain the optimal Q value.

6. An intelligent swarm collaborative control system for an active power distribution network, characterized in that, include: The single-agent optimal energy allocation module is configured to treat grid nodes or control units as agents in an active power distribution network. Acquire real-time status information of different intelligent agents in the power grid, including at least voltage, current, and power; Based on real-time state information, the expected cumulative reward of taking load management actions under a given smart grid state is evaluated by using a neural network to represent the Q-value function. In this process, a single agent obtains the optimal energy allocation strategy by maximizing the Q-value function. Secondly, based on the established model, the projection multi-agent subgradient algorithm is used to facilitate the exchange of information between individual agents and other agents through randomly changing network topology; The collaborative control module is configured to: construct a Markov network communication model, based on which multiple agents exchange information; The construction of the Markov network communication model includes: expressing that agents can cooperate to solve constrained optimization problems; and obtaining the agents based on the above expression. i The modified estimate expression; based on the estimate expression, a distributed multi-agent subgradient algorithm for power grid information exchange among multiple agents is obtained; a probabilistic communication model is obtained based on the gradient algorithm; An intelligent agent swarm composed of multiple agents uses a deep deterministic policy gradient algorithm to coordinate the control of power grid energy distribution; the deep deterministic policy gradient algorithm includes two deep neural networks, namely the Actor network and the Critic network; The Actor network represents the current state of the regional smart grid, and its goal is to learn the optimal deterministic policy. The Actor, as a grid energy policy function composed of multiple nodes, generates output energy output actions given the current state. The Critic network serves as a function to evaluate actions taken in a given power grid state, with the goal of learning the optimal energy allocation action value function. The Critic evaluates the energy allocation action value function of a swarm of agents based on the Actor's output and the current state. The time difference generated by the Critic drives the learning process in the Actor network and the Critic network, thereby evaluating the quality of the policy. By minimizing the loss to update the Critic and Actor networks, the weights of the intelligent agent swarm can be updated and coordinated.

7. The intelligent swarm collaborative control system for an active power distribution network as described in claim 6, characterized in that, A deep Q-network reinforcement learning model is used to store real-time state information of the power grid in an empirical memory, and batch samples are randomly selected from the memory to update the model parameters.

8. The intelligent swarm collaborative control system for an active power distribution network as described in claim 6, characterized in that, The deep Q-network reinforcement learning model uses a separate target network to generate a target Q value that reflects the energy stability of the power grid. During the learning process, the model adopts a delayed update method, copying the real-time state value of the current power grid node to the target network every C steps to update the real-time state of the target node. Using the node energy state c as input to the online network, the actual Q value of action a is obtained. Using the next state as input to the objective function, the maximum Q value of all actions in the target network is obtained.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-5.

Images