Multi-agent distributed reinforcement learning power generation control method and device based on Reformer model of value distribution, computer equipment and medium
By employing a multi-agent distributed reinforcement learning method based on a value distribution-based fusion Reformer model, the problems of low efficiency, policy fluctuation, and high computational complexity in frequency deviation control in novel power systems are addressed, achieving fast convergence and efficient frequency deviation suppression.
Patent Information
- Application Number
- CN202511639027.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-02-06
AI Technical Summary
Existing reinforcement learning frequency deviation control methods for new power systems suffer from problems such as low efficiency in experience collection, extreme trade-offs between value bias and variance, neglect of value distribution leading to policy return fluctuations, and high computational complexity.
A multi-agent distributed reinforcement learning method based on value distribution and fusion Reformer model is adopted. Action experience is collected through K parallel online actor networks. Multi-agent collaborative learning is carried out by combining distributed deep deterministic policy gradient method and distributed critic network. The Reformer model is used to efficiently capture frequency deviation features and realize multi-agent collaborative suppression of frequency deviation.
It improves the efficiency of experience collection, rapidly reduces system frequency deviation, reduces energy waste, optimizes the learning signal balance, enhances learning robustness, and reduces computational complexity.
Smart Images
Figure CN121480604A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of high proportion of renewable energy, intelligent generation control of new power system, artificial intelligence, deep learning and reinforcement learning, and relates to a generation control method of artificial intelligence, and is suitable for intelligent generation control of a new power system. BACKGROUND
[0002] The existing reinforcement learning frequency deviation control method of the new power system has the problem of low efficiency of collecting experience because the actor network collects experience in single thread series.
[0003] The existing reinforcement learning frequency deviation control method of the new power system has the problem of extreme value deviation and variance in trade-off because it only uses one-step return to estimate value.
[0004] The existing reinforcement learning frequency control method of the actor-critic architecture of the new power system has the problems of low ability to capture strategy income fluctuation and loss of key information of the value function because it only estimates the expected value function and ignores the value distribution.
[0005] The existing system frequency deviation control method using the combination of the Transformer neural network and reinforcement learning has the problems of high computational complexity in time and space of square order, and long running time and high memory resource consumption in long-term stable control. SUMMARY
[0006] Therefore, it is necessary to provide a multi-agent distributed reinforcement learning generation control method based on value distribution and fusion Reformer model, device, computer equipment, computer readable storage medium and computer program product to solve the above technical problems.
[0007] In a first aspect, the present application provides a multi-agent distributed reinforcement learning generation control method based on value distribution and fusion Reformer model. The method comprises:
[0008] Obtaining the frequency deviation Df of the new power system region m at time t in a period of time m,t , the differential value dDf m,t / dt and the integral value Df m,t dt of Df m,t , and splicing them into a frequency deviation feature input vector x m .
[0009] x mThe embedding tensor X of the frequency deviation feature sequence is [batch size, frequency deviation feature sub-sequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], and the two tensor components X1 and X2 are input into K parallel online actor networks of the same structure, and the structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output.
[0010] Each agent writes the distributed action experience from K parallel online actor networks at the same time in the form of a frequency deviation feature-generator speed action pair into a global experience replay table, and a value distribution-based distribution deep deterministic policy gradient method learner is used to perform multi-agent collaborative reinforcement learning on the suppression process of the new power system frequency deviation, the learner is randomly sampled and learned by combining the reward value calculation with the online distribution critic network, the target actor network and the target distribution critic network are stabilized by generating a distribution Bellman target, during training, the online distribution critic minimizes the distribution loss to approximate the target distribution, the action-value distribution expectation of the online distribution critic is fed back to the online actor network for deterministic policy gradient update, and the target actor network and the target distribution critic network are slowly followed by the online actor network and the online distribution critic network through soft update, so as to realize multi-agent collaborative suppression of the frequency deviation of the new power system.
[0011] The agent of the region m sends a speed reference action instruction of the governor according to the policy mapping , that is, the active power output of the generator in the control region m.
[0012] In a second aspect, the application also provides a multi-agent distributed reinforcement learning power generation control device based on a value distribution fusion Reformer model. The device comprises:
[0013] An input vector construction module is configured to obtain the frequency deviation Δf of the new power system region m at time t within a period of time m,t , the differential value dΔf m,t / dt and the integral value ∫Δf m,t dt of Δf m,t , and splice them into a frequency deviation feature input vector x m .
[0014] An online actor network construction module is configured to x mThe embedding tensor X of the frequency deviation feature sequence is [batch size, frequency deviation feature sub-sequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], and the two tensor components X1 and X2 are input into K parallel online actor networks of the same structure, and the structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output.
[0015] The agent training and updating module is configured to write the distributed action experience from the K parallel online actor networks in the form of a frequency deviation feature-generator speed control action pair into a global experience replay table at the same time for each agent, and to use a value distribution-based distribution deep deterministic policy gradient method learner to perform multi-agent collaborative reinforcement learning on the suppression process of the new power system frequency deviation, the learner performs random sampling learning by combining the reward value calculation with the online distribution critic network, the target actor network and the target distribution critic network are stabilized by generating a distribution Bellman target, during training, the online distribution critic network minimizes the distribution loss to approximate the target distribution, the action-value distribution expectation of the online distribution critic network is fed back to the online actor network for deterministic policy gradient update, and the target actor network and the target distribution critic network are slowly followed by the online actor network and the online distribution critic network through soft update, thereby realizing multi-agent collaborative suppression of the frequency deviation of the new power system.
[0016] The agent power generation control instruction output module is configured to output a speed reference action instruction of a governor of the region m according to the policy mapping , that is, to control the active power output of the generator of the region m.
[0017] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0018] Obtain the frequency deviation Δf m,t of the new power system region m at time t within a period of time m,t , the differential value dΔf m,t / dt and the integral value ∫Δf m,t dt of Δf m ;
[0019] x mThe embedding tensor X of the frequency deviation feature sequence is [batch size, frequency deviation feature sub-sequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], and the two tensor components X1 and X2 are input into K parallel online actor networks of the same structure, and the structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output.
[0020] Each agent writes the distributed action experience from K parallel online actor networks at the same time in the form of a frequency deviation feature-generator speed action pair into a global experience replay table, and a value distribution-based distribution deep deterministic policy gradient method learner is used to perform multi-agent collaborative reinforcement learning on the suppression process of the new power system frequency deviation, the learner is randomly sampled and learned by combining the reward value calculation with the online distribution critic network, the target actor network and the target distribution critic network are stabilized by generating a distribution Bellman target, during training, the online distribution critic minimizes the distribution loss to approximate the target distribution, the action-value distribution expectation of the online distribution critic is fed back to the online actor network for deterministic policy gradient update, and the target actor network and the target distribution critic network are slowly followed by the online actor network and the online distribution critic network through soft update, so as to realize multi-agent collaborative suppression of the frequency deviation of the new power system.
[0021] The agent of the region m sends a speed reference action instruction of the governor according to the policy mapping , that is, the active power output of the generator in the control region m.
[0022] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:
[0023] Obtain the frequency deviation Δf of the new power system region m at time t in a period of time m,t , the differential value dΔf m,t / dt and the integral value ∫Δf m,t dt of Δf m,t , splice them into a frequency deviation feature input vector x m ;
[0024] x mThe embedding tensor X of the frequency deviation feature sequence is [batch size, frequency deviation feature subsequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], and the two tensor components X1 and X2 are input into K parallel online actor networks of the same structure, and the structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output.
[0025] Each agent writes the distributed action experience from K parallel online actor networks at the same time in the form of a frequency deviation feature-generator speed action pair into a global experience replay table, and a value distribution-based distribution deep deterministic policy gradient method learner is used to learn the multi-agent collaborative reinforcement learning of the suppression process of the new power system frequency deviation, the learner is randomly sampled and learned by combining the reward value calculation of the online distribution critic network, the target actor network and the target distribution critic network are stabilized by generating a distribution Bellman target, during training, the online distribution critic minimizes the distribution loss to approximate the target distribution, the action-value distribution expectation of the online distribution critic is fed back to the online actor network for deterministic policy gradient update, and the target actor network and the target distribution critic network are slowly followed by the online actor network and the online distribution critic network through soft update, so as to realize the multi-agent collaborative suppression of the frequency deviation of the new power system.
[0026] The agent of the region m sends a speed reference action instruction of the governor according to the policy mapping , that is, the active power output of the generator in the control region m.
[0027] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:
[0028] Obtain the frequency deviation Δf of the new power system region m at time t in a period of time m,t , the differential value dΔf m,t / dt and the integral value ∫Δf m,t dt of Δf m,t , splice them into a frequency deviation feature input vector x m ;
[0029] x mThe embedding tensor X of the frequency deviation feature sequence is [batch size, frequency deviation feature subsequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], and the two tensor components X1 and X2 are input into K parallel online actor networks of the same structure, and the structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output.
[0030] Each agent writes the distributed action experience from K parallel online actor networks at the same time in the form of a frequency deviation feature-generator speed action pair into a global experience replay table, and a value distribution-based distribution deep deterministic policy gradient method learner is used to learn the suppression process of the new power system frequency deviation, the learner is randomly sampled and learned by an online distribution critic network combined with a reward value calculation, the target actor network and the target distribution critic network are stabilized by generating a distribution Bellman target, during training, the online distribution critic minimizes the distribution loss to approximate the target distribution, the action-value distribution expectation of the online distribution critic is fed back to the online actor network for deterministic policy gradient update, and the target actor network and the target distribution critic network are slowly followed by the online actor network and the online distribution critic network through soft update, so as to realize the multi-agent collaborative suppression of the frequency deviation of the new power system.
[0031] The agent of the region m sends a speed reference action instruction of the governor according to the policy mapping , that is, the active power output of the generator in the control region m.
[0032] The above-mentioned multi-agent distributed reinforcement learning generation control method, device, computer equipment, storage medium and computer program product based on the value distribution fusion Reformer model, by acquiring the frequency deviation Δf m,t of the new power system region m at time t within a period of time m,t , its differential value dΔf m,t / dt and integral value ∫Δf m,t dt three characteristics, splicing into a frequency deviation feature input vector x m ; x mThe embedding tensor X is [batch size, frequency deviation feature subsequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], both of which are input into K parallel online actor networks of the same structure, and the structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output; each agent writes the distributed action experience from K parallel online actor networks at the same time in the form of a frequency deviation feature-generator speed control action pair into a global experience replay table, and a value distribution-based distributed deep deterministic policy gradient method learner is used to learn the suppression process of the new power system frequency deviation through multi-agent collaborative reinforcement learning, the learner is randomly sampled and learned through the online distribution critic network combined with the reward value calculation, the target actor network and the target distribution critic network are stabilized through the generation of the distribution Bellman target, during training, the online distribution critic minimizes the distribution loss to approximate the target distribution, the action-value distribution expectation of the online distribution critic is fed back to the online actor network for deterministic policy gradient update, and the target actor network and the target distribution critic network are slowly followed by the online actor network and the online distribution critic network through soft update, realizing the multi-agent collaborative suppression of the frequency deviation of the new power system; the agent of the region m sends a speed reference action instruction of the governor according to the policy mapping , that is, controls the active power output of the generator in the region m; K online actor networks can collect action experience in parallel, improve experience collection efficiency, enable the new power system to converge quickly, quickly reduce system frequency deviation, and thus reduce energy waste. BRIEF DESCRIPTION OF DRAWINGS
[0033] Figure 1 is the power system generation control framework of the method of the present application.
[0034] Figure 2 is the Reformer model workflow diagram of the method of the present application.
[0035] Figure 3 is an internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.
[0037] In one embodiment, as Figure 1As shown, a multi-agent distributed reinforcement learning power generation control method based on a value distribution fusion Reformer model is provided. In this embodiment, the method is applied to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is realized through the interaction of the terminal and the server.
[0038] A multi-agent distributed reinforcement learning power generation control method based on a value distribution fusion Reformer model can combine a distributed reinforcement learning method considering value distribution, a multi-agent collaborative control method, and a Reformer model with an extensible actor-critic architecture, and is used for the control of power system frequency deviation. In this embodiment, the method includes the following steps:
[0039] Step (1), obtaining the frequency deviation Δf m,t of a new power system region m at time t within a period of time m,t , the differential value dΔf m,t / dt and the integral value ∫Δf m,t dt of the frequency deviation are spliced into a frequency deviation feature input vector x m .
[0040] The new power system is divided into M regions, and the dispatch center of region m is set as agent m. The frequency deviation of region m at time t is Δf m,t . ;
[0041] In region m, the frequency deviation sequence Δf m,t (T0) of the new power system within a T0 period of time is obtained. Δf m,t (T0) is divided into n-dimensional frequency deviation sub-sequences according to a step t0. The frequency deviation sub-sequence feature observations of the new power system are calculated in three forms of Δf m,t , dΔf m,t / dt, and ∫Δf m,t dt. Δf m,t is the frequency deviation of the frequency deviation sub-sequence of region m at time stamp t; dΔf m,t / dt is the differential value of the frequency deviation of the frequency deviation sub-sequence of region m at time stamp t; and ∫Δf m,t dt is the integral value of the frequency deviation of the frequency deviation sub-sequence of region m at time stamp t.
[0042] The frequency deviation features observed by the agent are spliced into a frequency deviation feature input vector x m,t =[Δf m,t ; dΔf m,t / dt; ∫Δf m,tdt], t is the timestamp of the time series feature; the frequency modulation process of the new power system is described as a Markov decision process, and the agent m observes the frequency deviation feature sequence state x of the new power system at time t m,t , the agent sends a unit regulation action a m,t , the reward value of the action is r m,t , and then enters the next state, and the frequency deviation feature sequence after the state transition is , and the corresponding agent sends a control action ;
[0043] The reward value calculation is used to measure the pros and cons of the agent's action, and is used to punish large frequency deviation and encourage the frequency deviation to converge to 0. The agent reward function r m,t of the region m at time t is:
[0044] , (1)
[0045] In the formula, r max is the maximum reward set when the frequency deviation is 0.
[0046] Step (2), the embedding tensor X of x m is [batch size, frequency deviation feature subsequence length, d], the embedding tensor X is decomposed into two tensor components X1 and X2 of [batch size, frequency deviation feature sequence length, d / 2], and both the two tensor components X1 and X2 are input into K parallel online actor networks with the same structure. The structure of the online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output.
[0047] The online actor network is used to output the interaction experience of the agent m and the new power system region m; the structure of each online actor network is a feature input linear layer, a Reformer model, a hidden layer network and an experience output layer from input to output; the feature input linear layer of the online actor network maps the frequency deviation feature input vector x m into a query vector, a key vector and a value vector; the Reformer model is a reversible residual structure including a locally sensitive hash attention layer and a feedforward layer, which can capture the association of long-distance feature data in the frequency deviation feature sequence; the hidden layer network performs deep nonlinear transformation on the features extracted by the Reformer; and the experience output layer converts the processed features into reinforcement learning experience tuples.
[0048] After the Reformer model decomposes the input vector into two input tensor components X1 and X2, the tensor X2 is injected into the locally sensitive hash attention layer to solve the locally sensitive hash attention, and then summed with X1 to obtain the tensor , and The result obtained by injecting the feedforward layer After summation with X2, an output tensor matching the input dimension is obtained .
[0049] K identical actor networks interact with the new power system to collect experience in parallel, and for each actor network, the frequency deviation feature sequence tensor component X2 is mapped into an n-dimensional query vector q=(q i )∈F n and an n-dimensional key vector k=(k i )∈F n , and the frequency deviation feature sequence tensor component X2 is mapped into an n-dimensional value vector v=(v i )∈F n by another different feature input linear layer, and the mapping process is as follows:
[0050] (2)
[0051] (3)
[0052] In the formula, is the weight of the linear layer used to map the query vector and the key vector; is the weight of the linear layer used to map the value vector; is the bias of the linear layer used to map the query vector and the key vector; is the bias of the linear layer used to map the value vector;
[0053] The Reformer model is used to capture the association of long-distance feature data in the frequency deviation feature sequence with linear-logarithmic computational complexity, which includes but is not limited to the trend of load change over time, the association between unit output and regional frequency deviation in the inter-regional interconnected large power grid, and the association between unit output and regional frequency deviation in the region; The Reformer combines a local sensitive hash attention layer and a feedforward layer through a reversible residual structure, and the output tensor of the reversible residual structure is:
[0054] (4)
[0055] (5)
[0056] In the formula, is the feedforward layer calculation; is the local sensitive hash attention calculation; is the reversible residual intermediate quantity in tensor form;
[0057] The local sensitive hash attention layer uses n0 different hash functions Multiple rounds of hashing are executed in parallel. In each round, similar queries and key vectors are grouped into the same hash bucket, and each round of hashing determines a queryable key q. i The set of key vectors of interest It is represented as:
[0058] (6)
[0059] In the formula, r is the hash round index for performing the hash; , , and These represent the hashes for rounds 1, 2, r, and n0, respectively; to ensure Set the key with timestamp j as Take the union of the sets obtained from n0 rounds of hashing to get the final query q. i The set of key vectors R of interest i The set is represented as:
[0060] (7)
[0061] To address the issues of unbalanced hash bucket sizes leading to wasted computing resources by small buckets and low batch processing efficiency due to large buckets exceeding memory limits, it is necessary to sort and divide queries / keys into blocks to provide a uniformly structured computational unit for batch processing. In batch processing, the extended candidate key set actually participating in attention calculation during the r-th round of hash operations... It is represented as:
[0062] (8)
[0063] In the formula This is the floor function; After hashing q in the r-th round i A globally sorted index that is sorted by hash bucket number and by timestamp within the bucket; After hashing q in the r-th round j A globally sorted index that is sorted by hash bucket number and by timestamp within the bucket; m is the size of the hash bucket, and n... buckets This represents the number of hash buckets. The sorted queries / keys are divided into equal-sized blocks, rounded down to the nearest integer. The process of dividing the hash buckets by size m is as follows:
[0064] (9)
[0065] In the formula, l is the length of the frequency deviation characteristic sequence;
[0066] Output of the Locality Sensitive Hash Attention Layer in a Multi-Round Hash Process The calculation is as follows:
[0067] (10)
[0068] where is the normalization term of the normalization exponential function, denotes from i to the normalization term of the normalization exponential function, denotes from i to the normalization term of the normalization exponential function; is the exponential calculation with the natural constant e as the base; is the summation from the 1st round of hash to the nth0round of hash; is the local sensitive hash attention of the rthround of hash operation, is expressed as:
[0069] (11)
[0070] where is the summation within the range of the extended candidate key set to which the timestamp j belongs; is the dynamic mask value of the attention mechanism of each query timestamp pair (i, j) in the rthround of hash operation, which ensures that the model meets the time sequence logic of the frequency deviation feature sequence modeling under the new power system, and the dynamic mask value function of the attention mechanism can mask the elements not in , and the specific masking process is:
[0071] (12)
[0072] Formula (10) is the output of the local sensitive hash attention layer, and after superimposing the frequency deviation feature sequence tensor component X1, the tensor is obtained, and is nonlinearly transformed by the feedforward layer to enhance the frequency deviation feature expression capability, and the calculation result of the feedforward layer is:
[0073] (13)
[0074] After superimposing with X2, the output tensor of the Reformer model matching the input dimension is obtained; the tensor is input into the hidden layer, and the hidden layer network is composed of a mean full connection layer and a standard deviation full connection layer in parallel structure, the mean full connection layer is responsible for calculating the mean feature vector μ of the tensor , and the standard deviation full connection layer is responsible for calculating the standard deviation feature vector σ of the tensor , and the calculation process of the hidden layer to the vector μ and σ is:
[0075] (14)
[0076] (15)
[0077] In the formula and These are the weight matrix and bias vector of the mean-connected layer, respectively. and These are the weight matrix and bias vector of the fully connected layer, respectively, representing the standard deviation.
[0078] The outputs of the mean fully connected layer and the standard deviation fully connected layer are concatenated into an action distribution parameter matrix [μ;σ] by a concatenation layer according to the frequency deviation sequence feature dimension; the empirical output layer performs the following linear transformation on the action distribution parameter matrix [μ;σ] to output the action empirical vector. for:
[0079] (16)
[0080] Step (3): Each agent writes the distributed action experience from K parallel online actor networks at the same time into a global experience replay table in the form of frequency deviation feature-unit speed regulation action pairs. A distributed deep deterministic policy gradient method learner based on value distribution is used to perform multi-agent collaborative reinforcement learning on the suppression process of frequency deviation in the new power system. The learner uses an online distributed commentator network combined with reward value calculation for random sampling learning. The target actor network and the target distributed commentator network are trained stably by generating a distributed Bellman objective. During training, the online distributed commentator minimizes the distribution loss to approximate the target distribution. The action-value distribution expectation of the online distributed commentator is fed back to the online actor network for deterministic policy gradient update. Then, soft update is used to make the target actor network and the target distributed commentator network gradually follow the online actor network and the online distributed commentator network, so as to realize multi-agent collaborative suppression of frequency deviation in the new power system.
[0081] Distributed motion experience from K parallel online actor networks with frequency deviation characteristics - unit speed regulation action pairs (x m a m The replays are written into the global experience replay table in the form of ) to prepare for subsequent online distributed critic network sampling and learning;
[0082] The actions of agent m are determined by the policy π of the online actor network. m :x m,t →a m,t Mapping control maps the frequency deviation characteristic sequence observations at the same time to a strategy π. m The corresponding action value a m,t In strategy π mThe expected return of an action under the mapping is given by the state-action value function The computation, and thus the evaluation, of the agent issuing π m The quality of the control action, is denoted as:
[0083] (17)
[0084] where, is the mathematical expectation; is the sum of all values starting from 0 for k; is the discount factor of the agent m, which reflects the degree of attention of the agent m to the immediate reward and the future reward; is the reward value calculated by the agent m at time t+k+1; denotes the frequency deviation feature observed by the agent m at time step t ; denotes the action performed by the agent m at time step t ; denotes the discounted cumulative reward, i.e., the return, from time step t+1, which is used to measure the long-term benefit of the frequency deviation control strategy;
[0085] Considering the continuity and high dimensionality of the frequency deviation control action of the new power system, a parameterized strategy , is introduced, where the online actor network parameters are updated by maximizing the parameterized strategy expected return, and the parameterized strategy expected return objective function is:
[0086] (18)
[0087] where, is the action of the agent m under the parameterized strategy, which is mapped to the frequency deviation feature input; is the state-action value function of the parameterized agent m;
[0088] The target actor network provides the parameter update direction for the online actor network using the deterministic policy gradient, so that the strategy is converted to the direction with high long-term return. The policy optimization objective function is: the gradient of the parameter :
[0089] (19)
[0090] where ρ is the frequency deviation feature observation access distribution related to the action strategy; denotes the gradient of the parameterized strategy with respect to the parameter θ; denotes the action The gradient;
[0091] The current action-value probability distribution based on the agent's policy m. Establish an online distributed critic network, which uses a global experience replay table to compare (x) m a m Experience is used to perform random sampling learning, and the uncertainty of unit disturbances and the volatility of load power in the new power system are characterized in the form of action-value probability distribution. The relationship with the expected return of the strategy satisfies:
[0092] (20)
[0093] By constructing a target actor network and a target distributed commentator network using distributed Bellman objectives, a distributed Bellman operator is introduced. To provide the updated target probability distribution for the loss of a target-distributed commentator network, the current probability distribution of policy rewards is approximated by minimizing the loss function. The updated target distribution for the target-distributed commentator network is as follows:
[0094] (twenty one)
[0095] In the formula, For agent m, the frequency deviation characteristic is The action is to The instant reward value below; For x m,t →a m,t Under the mapping relationship, Mapping strategy; control The characteristic of agent m at the current frequency deviation is: The action is to When, the next frequency deviation characteristic after the state transition occurs and the policy rewards under the corresponding policy mapping relationship;
[0096] In order for an online distributed commentator network to approximate the target distribution with the current probability distribution, it needs to minimize the loss function. for:
[0097] (twenty two)
[0098] In the formula, The parameters for a parameterized online distributed network of critics; and These are the parameters for the target actor network and the target distributed critic network, respectively. the probability distribution of the Bellman update for the target actor network and the target distributional critic network; to measure the distributional temporal difference error;
[0099] the target actor network parameters and the target distributional critic network are updated in a soft update manner to slow down the target drift, and the update process is:
[0100] (23)
[0101] (24)
[0102] wherein, is a soft update coefficient;
[0103] The distributional policy gradient update of the online actor network is completed by incorporating the action-value distribution expectation into the online actor network update process, and the online actor network is updated in the direction of the gradient The update process is:
[0104] (25)
[0105] wherein, is the action-value probability distribution of the agent m at the frequency deviation feature , and the action is the gradient calculation of the action ;
[0106] The training target of the online distributional critic network is constructed by introducing the N-step Bellman operator , the reward information of the long policy update trajectory is used to reduce the temporal difference error, and the stability of the online distributional critic network learning is improved. The update process of the N-step Bellman is:
[0107] (26)
[0108] wherein, n is the index of the number of times of Bellman update; and are the starting frequency deviation feature and the action of the agent m, and are the frequency deviation feature and the action of the agent m after the n-th Bellman update, and are the frequency deviation feature and the policy mapping action of the agent m after the n-th Bellman update; is the real-time reward of the n-th Bellman update process of the agent m; and respectively, are the discount factor of the nth and Nth Bellman update process of agent m; is the initial frequency deviation of agent m, is the initial action of agent m, is the expected return of the Nth Bellman update under the initial frequency deviation
[0109] Step (4), the agent of region m issues a governor speed reference action instruction , i.e., controls the active power output of the generator of region m.
[0110] The new power system is coupled between m regions through tie lines to achieve power sharing, and the number of generator units in region m is n G , the i-th generator unit itself has an active power disturbance , the total active power of the load on the tie line monitored by the agent is , the agent issues a governor speed reference action instruction The frequency response model of the new power system in each region in per unit value satisfies:
[0111] (27)
[0112] wherein, , , , and are the inertia time constant, actual speed, damping coefficient, mechanical power and electromagnetic power of the i-th generator unit, respectively; is the limit as the time interval tends to 0;
[0113] In per unit value, the change in unit speed and the frequency deviation are equal, so the linearized frequency response model of the new power system is:
[0114] (28)
[0115] wherein, is the frequency deviation of the i-th generator unit;
[0116] Under the coupling state of the tie line of the new power system region m, the frequency deviation of region m and the frequency deviation of each unit are related as:
[0117] (29)
[0118] wherein, is the frequency deviation weight of the i-th generator unit in region m, is calculated as:
[0119] .(30)
[0120] The present application has the following advantages and effects relative to the prior art:
[0121] (1) The existing reinforcement learning frequency deviation control method of the new power system collects experience in a single-thread serial manner, which has the problem of low efficiency of collecting experience. The K online actor networks proposed in the present application collect action experience in parallel, all frequency deviation feature-action experience is written into a global replay pool, the learner samples and updates the network from the replay pool, and the new policy update parameters are regularly synchronized to all online actor networks, which improves the efficiency of experience collection and enables the new power system to converge quickly and reduce system frequency deviation, thereby reducing energy waste.
[0122] (2) The existing reinforcement learning frequency deviation control method of the new power system only uses one-step return to estimate the value, which has the extreme problem of value deviation and variance in trade-off. The N-step return is used in the present application to reconstruct the Bellman update operator, which balances the value deviation and variance through N-step return and optimizes the learning signal.
[0123] (3) The existing reinforcement learning frequency control method of the actor-critic architecture of the new power system only estimates the expected value function and ignores the value distribution, which has the problem of low ability to capture strategy yield fluctuations and loss of key information of the value function. The distribution type critic network is used in the present application to model return uncertainty, which improves the learning robustness.
[0124] (4) The existing system frequency deviation control method using the combination of Transformer neural network and reinforcement learning has high computational complexity in time and space of square order, and when applied to long-term stable control, it is prone to have problems of too long running time and too high memory resource consumption. The Reformer model in the present application efficiently captures the key features in high-dimensional frequency deviation feature data and the global dependency relationship between each position through local sensitive hash attention, which realizes accurate update of deep reinforcement learning strategy with lower linear logarithmic order of computational complexity. Figure 1is a power system generation control block diagram of the method of the present application. The method adopts a distributed reinforcement learning closed-loop control framework of the actor-critic architecture, first, the frequency deviation Δf and its differential and integral of each region are collected, spliced into a feature vector, and a frequency modulation Markov decision process is constructed, the frequency deviation is punished according to the designed reward function and the agent is guided to issue actions to the smaller frequency deviation convergence value; then, the feature is embedded and sent to K parallel online actor networks, and a new type of power system is interacted to generate feature-action experience; subsequently, the experience from the K online actors is written into a global experience replay table, the learner is mainly combined with the target actor and the target distribution critic, the network is trained with the distribution Bellman target, the N-step return and the soft update, and the value distribution is used to guide the online actor to make a deterministic policy gradient update; finally, a multi-region unit frequency response model containing a tie line coupling is established, the online actor and the distribution critic parameters are fixed after offline training to convergence under a random load scenario, and the online actor network is frozen after I rounds of training, the latest parameters of which are used to output unit speed control instructions in the actual scene, so that the system frequency deviation is suppressed and the load tracking is realized.
[0125] Figure 2 is a Reformer model workflow diagram of the method of the present application. The Reformer model combines a local sensitive hash attention layer and a feedforward layer through a reversible residual structure, after the input vector is decomposed into two input tensor X1 and X2 components, the tensor X2 is injected into the local sensitive hash attention layer to solve the local sensitive hash attention, and then summed with X1 to obtain the tensor , the result of injecting into the feedforward layer is summed with X2 to obtain the output tensor matching the input dimension , that is, the Reformer model output signal that captures the key information of high-dimensional frequency deviation characteristics and global dependence.
[0126] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 3As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to realize a multi-agent distributed reinforcement learning power generation control method based on a value distribution fusion Reformer model. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.
[0127] Those skilled in the art can understand that, Figure 3 The skilled in the art can understand that,
[0128] In one embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps in each of the above method embodiments.
[0129] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.
[0130] In one embodiment, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.
[0131] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.
[0132] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.
[0133] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of technical features in the above embodiments are described, but as long as the combination of technical features does not exist contradictory, it should be considered as the scope of the present application.
[0134] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.
Claims
1. A multi-agent distributed reinforcement learning power generation control method based on a value distribution-integrated Reformer model, characterized in that, The method includes: Obtain the frequency deviation Δf of region m of the new power system at time t over a period of time. m,t , with Δf m,t Its differential value dΔf m,t / dt and the integral value ∫Δf m,t The three features dt are concatenated to form the frequency deviation feature input vector x. m ; x m The embedding tensor X is [batch size, frequency deviation feature subsequence length, d]. The embedding tensor X is decomposed into two tensor components X1 and X2 [batch size, frequency deviation feature sequence length, d / 2]. Both tensor components X1 and X2 are input to a total of K parallel online actor networks with the same structure. The structure of the online actor network from input to output is as follows: feature input linear layer, Reformer model, hidden layer network and empirical output layer. Each agent writes the distributed action experience from K parallel online actor networks at the same time into a global experience replay table in the form of frequency deviation feature-unit speed regulation action pairs. A distributed deep deterministic policy gradient method learner based on value distribution is used to perform multi-agent collaborative reinforcement learning on the frequency deviation suppression process of the novel power system. The learner uses an online distributed commentator network combined with reward value calculation for random sampling learning. The target actor network and the target distributed commentator network are trained stably by generating a distributed Bellman objective. During training, the online distributed commentator minimizes the distribution loss to approximate the target distribution. The action-value distribution expectation of the online distributed commentator is fed back to the online actor network for deterministic policy gradient update. Then, soft update is used to make the target actor network and the target distributed commentator network gradually follow the online actor network and the online distributed commentator network, so as to realize multi-agent collaborative suppression of frequency deviation of the novel power system. The agent in region m issues governor speed reference action commands based on the policy mapping. , which refers to the active power output of the generator in the control area m.
2. The multi-agent distributed reinforcement learning power generation control method based on the value distribution-fused Reformer model according to claim 1, characterized in that, The scheduling center of region m is set as agent m, and the agent reward function r of region m at time t. m,t for: ， In the formula, r max The maximum reward set when the frequency deviation is 0.
3. The multi-agent distributed reinforcement learning power generation control method based on the value distribution-fused Reformer model according to claim 1, characterized in that, The feature input linear layer of the online actor network inputs the frequency deviation feature vector x. m The mapping is done as a query vector, a key vector, and a value vector.
4. The multi-agent distributed reinforcement learning power generation control method based on the value distribution-fused Reformer model according to claim 1, characterized in that, The Reformer model decomposes the input vector into two input tensor components, X1 and X2. Then, it injects tensor component X2 into a Locality Sensitive Hash (LSH) attention layer to solve for LSH attention, and finally sums it with X1 to obtain the tensor. ,Will Results obtained from injecting feedforward layer Summing with X2 yields an output tensor that matches the input dimension. .
5. A multi-agent distributed reinforcement learning power generation control device based on a value distribution-integrated Reformer model, characterized in that, The device includes: The input vector construction module is used to obtain the frequency deviation Δf of the new power system region m at time t over a period of time. m,t , with Δf m,t Its differential value dΔf m,t / dt and the integral value ∫Δf m,t The three features dt are concatenated to form the frequency deviation feature input vector x. m ; Online actor network building module for x m The embedding tensor X is [batch size, frequency deviation feature subsequence length, d]. The embedding tensor X is decomposed into two tensor components X1 and X2 [batch size, frequency deviation feature sequence length, d / 2]. Both tensor components X1 and X2 are input to a total of K parallel online actor networks with the same structure. The structure of the online actor network from input to output is as follows: feature input linear layer, Reformer model, hidden layer network and empirical output layer. The agent training and update module is used to write the distributed action experience of each agent from K parallel online actor networks at the same time into a global experience replay table in the form of frequency deviation feature-unit speed regulation action pairs. A distributed deep deterministic policy gradient method learner based on value distribution is used to perform multi-agent cooperative reinforcement learning on the suppression process of frequency deviation in the new power system. The learner uses an online distributed commentator network combined with reward value calculation for random sampling learning. The target actor network and the target distributed commentator network are trained stably by generating a distributed Bellman objective. During training, the online distributed commentator minimizes the distribution loss to approximate the target distribution. The expected action-value distribution of the online distributed commentator is fed back to the online actor network for deterministic policy gradient update. Then, soft update is used to make the target actor network and the target distributed commentator network gradually follow the online actor network and the online distributed commentator network, so as to realize multi-agent cooperative suppression of frequency deviation in the new power system. The intelligent agent power generation control command output module is used by the intelligent agent in region m to issue speed reference action commands for the speed governor according to the strategy mapping. , which refers to the active power output of the generator in the control area m.
6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multi-agent distributed reinforcement learning power generation control method based on the value distribution fusion Reformer model as described in any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-agent distributed reinforcement learning power generation control method based on the value distribution fusion Reformer model as described in any one of claims 1 to 4.