A collaborative dispatching method of energy storage on both sides of source and load based on multi-agent

Abstract

The present invention relates to the technical field of electric power dispatching, and discloses a collaborative dispatching method for source-load dual-side energy storage based on multi-agents. The method comprises: constructing an objective function and constraints for a source-load dual-side energy storage system based on a multi-agent; constructing a Markov decision model according to the objective function and constraints, and introducing an attention mechanism, solving the Markov decision model through the MAAN‑TD3 algorithm, obtaining the action space corresponding to the source-load dual-side energy storage system and executing it. The method can improve the utilization efficiency of energy storage, the new energy absorption capacity of the power supply side, and the system operation performance by collaboratively dispatching the load-source dual-side energy storage. On the other hand, the overall economic benefits can be improved on the basis of reducing the amount of abandoned wind.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power dispatching, and in particular to a source-load dual-side energy storage coordinated dispatching method based on multi-agents. Background Art

[0002] With the rapid advancement of the construction of new power systems (hereinafter referred to as systems), the proportion of renewable energy power generation has increased rapidly, but its uncertainty has posed a huge challenge to the operation of the system. Energy storage supports the consumption of renewable energy with its high flexibility and enhances the regulation capacity of the power system. However, due to the uneven distribution of energy storage resources and the single calling method, some energy storage has not been fully utilized, resulting in a waste of resources. In the power system, energy storage equipment is built on both the source and load sides. Due to the limitation of energy storage capacity, the power supply side has insufficient consumption of new energy, which limits the system's operating performance.

[0003] On the other hand, user-side suppliers integrate distributed and centralized energy storage and make profits by renting and sharing them. The unrented energy storage can be used for charging and discharging arbitrage by taking advantage of the price difference. However, the charging and discharging strategy of this model is relatively simple, the benefits obtained are low, and it is impossible to maximize the efficiency of energy storage. Summary of the invention

[0004] In order to solve the above technical problems existing in the prior art, the present invention provides a source-load dual-side energy storage coordinated scheduling method based on multi-agent, and the specific technical solution includes the following steps:

[0005] Step S1: construct an objective function based on the multi-agent source-load dual-side energy storage system and set constraints;

[0006] Step S2: constructing a Markov decision model based on the objective function and constraints;

[0007] Step S3: Based on the Markov decision model, for each Markov decision cycle, the state space corresponding to the source-load dual-side energy storage system is obtained;

[0008] Step S4: Substitute the state space into the Markov decision model for Q value fitting, obtain the corresponding action space and execute it.

[0009] Preferably, step S1 specifically includes:

[0010] Step S11: Construct an objective function with the goal of minimizing wind abandonment and maximizing energy storage utilization , the formula is as follows:

[0011] ;

[0012] In the formula, is the scheduling period ordinal number, is the total number of dispatching periods per day, is the wind abandonment penalty coefficient, For the period The actual output of the new energy units inside, For the period The maximum adjustable output of the new energy units in the is the station number of energy storage, For the The actual power during the energy storage period, For the Energy storage period Maximum power within

[0013] Step S12: Set constraints, the formula is as follows:

[0014] ;

[0015] ;

[0016] ;

[0017] In the formula, For the period Internal generator injection node The active power, For the period Internal energy storage injection node The active power, For the period Internal Node The active power, , The time periods Internal Node ,node The voltage, , Node The real and imaginary parts of the admittance matrix, For the period Internal Line The voltage phase difference on is the sequence number of the generator set, For the period Neidi The output value of the active power of the generator set, , Respectively The minimum and maximum output values ​​of the active power of each generator set. For the period Intraday The reactive power output value of each generator set, , Respectively The minimum and maximum output values ​​of reactive power of each generator set, For the Energy storage period Minimum power within.

[0018] Preferably, step S2 specifically includes: constructing a Markov decision model based on the objective function and the constraints ( , , , , );in,

[0019] is the state space, and the formula is as follows:

[0020] ;

[0021] In the formula, is the power information, is the grid voltage, is the state of charge of the energy storage, For time;

[0022] is the action space, and the formula is as follows:

[0023] ;

[0024] In the formula, is the charging and discharging power of the shared energy storage, The charging and discharging power of the power plant and station energy storage;

[0025] is the state transition probability; is the reward function, specifically , the formula is as follows:

[0026] ;

[0027] In the formula, For the period The reward obtained by the interaction between the internal agent and the environment, Penalty for out-of-limit actions;

[0028] is the discount factor, 0≦ ≦1;

[0029] For each Markov decision cycle, obtain the corresponding state space ;

[0030] The state space Substitute the Markov decision model to fit the Q value, obtain the corresponding action space and execute it.

[0031] Preferably, step S4 is specifically:

[0032] In the Actor-Critic framework, the attention network is introduced into the Critic network, and the Q value is fitted through the MAAN-TD3 algorithm (Multi-Agent Attention Noisy Twin Delayed Deep Deterministic Policy Gradient).

[0033] Step S41: The global state-action pair ( s , a ) Input the multi-layer perceptron and stack the vectors in the order of the agents to get the vector ;in, s∈ , a∈ ;

[0034] Step S42: vector Input the multi-head attention network to extract features and get feature output ;

[0035] Step S43: Layer normalization feature output , and get the result x ; The result x Input residual network combined vector , then input the fully connected layer, and the output of the fully connected layer out As the Q value.

[0036] Further, before step S41, the process further includes: s Add random noise perturbation , constituting the exploration state , the formula is as follows:

[0037] .

[0038] Furthermore, a neural network is used to parameterize the Actor network, and the Actor network is updated by the gradient descent method.

[0039] Preferably, the residual network is a Bellman residual network.

[0040] Preferably, by minimizing the loss function Update the parameters of the Critic network. The formula is as follows:

[0041] ;

[0042] ;

[0043] In the formula, are network parameters, For the The loss function of the Critic network of an agent is, is the second observation data stored in the experience pool, is the corresponding reward value, is the second observation data stored after updating, is the expected value of the loss function, is the target policy of the value function, Indicates The Q value obtained by the Actor network of the agent, is the target Q value, For the The reward value obtained by the agent is is the target strategy of the updated value function, Indicates the Q value obtained by updating the Actor network. is the number of agents, For the The corresponding action of the agent, After the update The corresponding action of the agent, For the observations of an agent; where =1,2,…, .

[0044] Compared with the prior art, the present invention has the following beneficial effects: the technical solution provided by the present invention can improve the utilization efficiency of energy storage, the new energy consumption capacity of the power supply side and the system operation performance by coordinating the energy storage on both the source and load sides. On the other hand, it can improve the overall economic benefits on the basis of reducing the amount of abandoned wind. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the scheduling method of the present invention.

[0046] Figure 2 This is a schematic diagram of introducing the attention network into the Critic network in the present invention.

[0047] Figure 3 It is a working schematic diagram of the dispatching control model based on the power grid environment model and the MAAN-TD3 framework in the present invention.

[0048] Figure 4 The figure is a schematic diagram of the structure of the example topology adopted in the present invention.

[0049] Figure 5 This is a comparison chart of the training effects of the scheduling control model in the present invention.

[0050] FIG6 (a) is a schematic diagram of the energy storage scheduling results before applying the solution of the present invention - the charging and discharging power of the energy storage at the station in the non-cooperative situation.

[0051] FIG6 (b) is a schematic diagram of the energy storage scheduling results before applying the solution of the present invention - sharing the energy storage charging and discharging power in a non-cooperative situation.

[0052] FIG7 (a) is a schematic diagram of the energy storage scheduling results after applying the solution of the present invention - the charging and discharging power of the energy storage at the cooperation site.

[0053] FIG7 ( b ) is a schematic diagram of the energy storage scheduling result after applying the solution of the present invention - the cooperative situation shares the energy storage charging and discharging power.

[0054] Figure 8 For the application of the present invention before and after energy storage Schematic diagram of the change results of (State of Charge).

[0055] The meanings of the numbers in the figure are: nodes—1~63; load—64; energy storage—65; thermal generator—66; wind generator—67; transformer—68. DETAILED DESCRIPTION

[0056] Hereinafter, the technical solution provided by the present invention will be further elaborated in combination with the accompanying drawings and embodiments.

[0057] In the new power system, energy storage equipment is built on both the source and load sides. The power supply side is limited by the energy storage capacity and there is a phenomenon of insufficient consumption of new energy. The user side cannot maximize the energy storage efficiency due to the single arbitrage method. Both parties hope to have more means to increase their profits.

[0058] In order to cope with the risks of safe operation of the power system brought about by the uncertainty of new energy, various regions require new energy plants and stations to be equipped with 10% to 20% energy storage, but due to limited capacity, it is still impossible to fully absorb new energy. Power supply operators hope to seek more flexible resources to promote the absorption of new energy. Figure 1 As shown in the figure, under the framework of the cooperative alliance, power operators call upon plant and station energy storage and power generation equipment while conducting coordinated scheduling with energy storage suppliers to achieve the goal of new energy consumption.

[0059] Generally speaking, energy storage suppliers on the user side have their own centralized energy storage, and will also aggregate the distributed energy storage scattered among various users for coordinated use. After integrating the two types of energy storage, suppliers can make profits by renting them out for sharing. The unrented energy storage can use the price difference to conduct charging and discharging arbitrage. However, the charging and discharging strategy in this case is relatively simple, and the profits obtained are relatively low. Energy storage suppliers also hope to adopt more flexible means to pursue higher profits. This is complementary to the new energy consumption needs of power operators. After reaching a cooperation, energy storage suppliers provide remaining idle energy storage to participate in the cooperation on the premise of meeting the user's energy storage rental needs, fully weigh the needs of power operators and their own profits, and call energy storage with the total cost of the cooperative alliance as the target.

[0060] Example 1

[0061] In this embodiment, the goal is to minimize the amount of wind abandonment and maximize the energy storage utilization rate. The objective function The formula is as follows:

[0062] ;

[0063] In the formula, is the scheduling period ordinal number, is the total number of dispatching periods per day, is the wind abandonment penalty coefficient, For the period The actual output of the new energy units inside, For the period The maximum adjustable output of the new energy units in the is the station number of energy storage, For the The actual power during the energy storage period, For the Energy storage period The maximum power within the range; set the constraint conditions, the formula is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] In the formula, For the period Internal generator injection node The active power, For the period Internal energy storage injection node The active power, For the period Internal Node The active power, , The time periods Internal Node ,node The voltage, , Node The real and imaginary parts of the admittance matrix, For the period Internal Line The voltage phase difference on is the sequence number of the generator set, For the period Neidi The output value of the active power of the generator set, , Respectively The minimum and maximum output values ​​of the active power of each generator set. For the period Intraday The reactive power output value of each generator set, , Respectively The minimum and maximum output values ​​of reactive power of each generator set, For the Energy storage period Minimum power within.

[0068] In addition, the goal can be to minimize the total cost of the cooperative alliance. The objective function is The formula is as follows:

[0069] ;

[0070] in, Costs for power supply operators; Profits for energy storage suppliers.

[0071] Power supply operating costs It consists of two parts: the operating cost of thermal power plants and new energy power plant operating costs , the formula is as follows:

[0072] = + ;

[0073] The cost of a thermal power plant mainly includes the unit operating cost, and the formula is as follows:

[0074] ;

[0075] In the formula, is the ordinal number of the thermal power unit, is the number of thermal power units, , and For the The power generation cost coefficient of the thermal power unit is For the period Neidi Active power of thermal power units.

[0076] The cost of new energy power plants includes daily unit operation and maintenance costs , Plant and station energy storage operation and maintenance costs and penalties for abandoning new energy , the formula is as follows:

[0077] ;

[0078] in, For the period The actual output of the new energy units in the factory; For the period The maximum available output of the internal renewable energy units; For the period Internal Factory Station The power of the energy storage unit; is the wind abandonment penalty coefficient; is the cost coefficient of new energy output per unit power; It is the energy storage operation cost coefficient of the plant.

[0079] Energy storage aggregators can charge and discharge according to electricity prices and demand. Providing revenue for energy storage, For the revenue from electricity sales; is the operating cost, including the electricity purchase cost and energy storage operation and maintenance cost, the formula is as follows:

[0080] ;

[0081] in, The unit operation and maintenance cost of shared energy storage; For the Shared energy storage devices in a unit time period The power value within For the period The electricity purchase price within the country; For the period The electricity price within the country; M , Y All are 0-1 discrimination indexes. In the electricity purchasing state, M =0, Y =1; in the electricity selling state, M =1,Y =0.

[0082] In order to ensure the stable and safe operation of the system, it is necessary to ensure the power balance of the power grid. The constraint formula is as follows:

[0083] ;

[0084] ;

[0085] in, , and They are thermal power units, new energy units and energy storage units in the time period Injection Node Active power of and for Time period node Active power and reactive power; For thermal power units in the period Injection Node of reactive power.

[0086] Due to mechanical structural limitations, thermal power units are subject to unit ramp rate restrictions during adjacent dispatch intervals, resulting in upper and lower limit constraints. The formula is as follows:

[0087] ;

[0088] ;

[0089] in, is the unit climbing rate; express Thermal power units in the period contribution; , Represents thermal power unit Minimum and maximum output of active power; , Represents thermal power unit Minimum and maximum output of reactive power; For the period Output of internal new energy generator set; It is the maximum output of the new energy generator set; A small variable in time.

[0090] System voltage safety constraint, the formula is as follows:

[0091] ;

[0092] in, Indicates time period Internal Node Voltage; and Representation Node The upper and lower voltage limits.

[0093] The exchange power of the tie line between the transmission network and the distribution network has upper and lower limit constraints, and the formula is as follows:

[0094] ;

[0095] ;

[0096] in, and The time periods Active power and reactive power of the tie lines between the internal transmission grid and the distribution grid; and are the minimum and maximum values ​​of the tie line active power respectively; and are the minimum and maximum values ​​of the reactive power of the tie line respectively.

[0097] Energy storage constraint condition, the formula is as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] in, Represents energy storage in the period within level; and Represents the energy storage charging efficiency and discharging efficiency; , They respectively represent the upper and lower limits of the power that can ensure the normal service life of the energy storage equipment; Represents the energy storage charging power; Represents the energy storage discharge power; Indicates the maximum capacity of energy storage; for Time period The power of the energy storage unit; For the The minimum value of the energy storage power; For the The maximum value of the energy storage power.

[0102] Energy storage aggregators must prioritize meeting users’ energy storage needs before the remaining energy storage can be used for cooperative scheduling. The formula for their business behavior constraints is as follows:

[0103] ;

[0104] in, The energy storage demand of the user; For other energy storage needs; To allocate time; is the total dispatchable energy storage capacity.

[0105] In Example 1, for the collaborative scheduling problem studied in the present invention, if the quantity not controlled by the scheduling strategy in each decision cycle is approximated as a constant, the collaborative scheduling model can be regarded as a sequential model, and then can be converted into an MDP (Markov Decision Process) model, which can be solved using a reinforcement learning algorithm.

[0106] Reinforcement learning generally formalizes the decision problem into a state space , Action Space , state transition probability And the reward function , and the discount factor Together they form a quintuple ( , , , , ).

[0107] State Space : Define the time of sharing , Grid voltage and power information , and the energy storage charge state partially observed by different intelligent agents (in this embodiment, mainly including energy storage, generators, distribution networks, loads, transformers and instruments for power measurement) Together they form the state space of the agent, and the formula is as follows:

[0108] ;

[0109] Action Space :The action space represents the decision-making behavior of the agent, and the action is defined as the charging and discharging power instructions of the agent to different energy storage devices. The formula is as follows:

[0110] ;

[0111] in, is the charging and discharging power of the shared energy storage, It is the charging and discharging power of the energy storage in the plant.

[0112] Reward Function :When using deep reinforcement learning to solve the problem, it is equivalent to minimizing the total cost of the alliance as the reward to maximize the cumulative return, and then consider the corresponding penalties for over-limit actions To correct the agent's actions, the reward function is defined as the following formula:

[0113] ;

[0114] in, Represents the reward obtained by the agent's interaction with the environment.

[0115] State transition probability :After the agent generates a strategy and inputs it into the environment, it will get the information of the next state. In this process, due to the uncertainty of the environment, the interaction process and the transition to the next state will also produce uncertain changes. Usually, the state transition matrix is ​​used to describe the probability of transitioning to the next uncertain state after executing the strategy. In the model studied in this invention, the probability transition matrix is ​​implicitly generated by Monte Carlo sampling of the environment.

[0116] Discount Factor : The discount factor is used to weigh the importance of current rewards versus future rewards. Higher discount factor values ​​emphasize future rewards, while lower discount factor values ​​focus more on immediate rewards. The range is [0,1].

[0117] After introducing the discount factor, a complete Markov decision process obtains the return corresponding to the "trajectory" , the formula is as follows:

[0118] ;

[0119] In the formula, For the period The discount factor decreases with time.

[0120] In this example, improvements are made to the shortcomings of the MAAN-TD3 algorithm in fitting Q values. In the Actor-Critic network architecture, an attention mechanism is introduced to assist in fitting Q values.

[0121] The Actor network is also parameterized using a neural network, and the update method is the gradient descent method.

[0122] Specifically, the formula for gradient descent is as follows:

[0123] ;

[0124] In the formula, is the gradient symbol, is the serial number of the agent, For the The network parameters of the agents, In anticipation of return, and are the experience pool and the first observation data stored in the experience pool, respectively. is the expectation of the return policy gradient, For the The continuous strategy corresponding to each agent is For the The corresponding action of the agent, Indicates The observation of an agent, Indicates The state-action function fitted by the agent set, is the number of actions; =1,2,…, .

[0125] Update the parameters of the Critic network by minimizing the loss function; the formula for the corresponding loss function is as follows:

[0126] ;

[0127] ;

[0128] In the formula, are network parameters, For the The loss function of the Critic network of an agent is, is the second observation data stored in the experience pool, is the corresponding reward value, is the second observation data stored after updating, is the expected value of the loss function, is the target policy of the value function, Indicates The Q value obtained by the Actor network of the agent, is the target Q value, For the The reward value obtained by the agent is is the target strategy of the updated value function, Indicates the Q value obtained by updating the Actor network. is the number of agents, After the update The corresponding action of the agent, For the observations of an agent; where =1,2,…, .

[0129] like Figure 2 As shown, first, the global state-action pair ( s, a ) are processed by a multilayer perceptron (MLP) and vectorized in the order of agents to form the input of the attention network, which then extracts the correlation features of different agents.

[0130] Secondly, the stacked vector is input into the attention network and divided into multiple heads for feature extraction. Relying on the multi-head self-attention mechanism in the network, different types of features can be effectively identified and the hidden dependencies between agents can be captured. The formula is as follows:

[0131] ;

[0132] in, represents the attention calculation mechanism, It means concatenating vectors in the same dimension. is the number of heads of the attention network, is the key matrix No. elements, The value matrix No. elements, The query matrix No. elements, =1,2,…, The input vector passes through the multi-head attention network to obtain the vector feature .

[0133] Furthermore, inside the attention network, the input vector Multiplying matrices , , Get the query matrix , key matrix , Value Matrix , the formula is as follows:

[0134] ;

[0135] In the formula, , and The key matrix , Value Matrix and the query matrix The weight matrix of

[0136] In the calculation process, the query matrix is ​​first and key matrix The weight coefficient matrix is ​​obtained by performing a dot multiplication operation, and then the final attention score matrix is ​​obtained through the softmax layer. Finally, this matrix is ​​used as the importance basis of the value matrix. The formula is as follows:

[0137] ;

[0138] in, is the dimension of the sequence, Represents the result of the attention mechanism output, that is, the feature output.

[0139] Finally, the output of the attention mechanism is normalized by layers. At the same time, the residual network structure ResNet is introduced to compare the normalized results with the original data. After combining, the output is obtained through the fully connected layer out , and as the Q value, the formula is as follows:

[0140] ;

[0141] in, Representation layer normalization processing, represents a feed-forward neural network.

[0142] Optionally, the residual network is a Bellman residual network.

[0143] In addition, in order to improve the stability of the strategy, s Add random Gaussian noise disturbance Expand the exploration space, the formula is as follows:

[0144] = s + ;

[0145] like Figure 3 As shown, the specific execution process is as follows:

[0146] For each agent, each agent has its own policy network, represented as ,in, It is The state of an agent, are the parameters of the policy network. The corresponding target policy network Used to calculate target actions and ensure the stability of updates.

[0147] The central evaluation network consists of two independent evaluation networks, denoted as and These two networks are used to evaluate a given state s and joint actions aThe existence of two evaluation networks is intended to mitigate the bias of overestimation of Q-values.

[0148] Corresponding to each evaluation network, there are two target evaluation networks, namely and The target evaluation network is used to calculate the target value, which is used to calculate the loss function during the actual update. The parameters of the policy network and the evaluation network are updated through software updates.

[0149] On the basis of the establishment of the above scheduling model, the source-load dual-side energy storage coordinated scheduling method based on improved multi-agent provided by the present invention includes the following steps:

[0150] Step S1: After each decision cycle, obtain the time period The corresponding status information, ; including time stamp , Grid voltage And the output power information of each equipment in the power grid , and different energy storage charge states ;

[0151] Step S2: Input the state information obtained in step S1 into the trained collaborative scheduling model to obtain the corresponding scheduling strategy; wherein the collaborative scheduling model is set as a Markov decision process with the goal of minimizing the total cost of the alliance and uses the MAAN-TD3 algorithm to train the agent;

[0152] Step S3: Execute the dispatching strategy obtained in step S2 to control the charging and discharging status of the corresponding energy storage devices on both sides of the source and load, as well as the output status of other equipment in the power system under this status.

[0153] In order to further verify the effectiveness of the source-load dual-side energy storage coordinated scheduling method based on improved multi-agent provided by the present invention, a simulation experiment was conducted. Figure 4 As shown, specifically, a 63-node system improved by connecting the IEEE (Institute of Electrical and Electronics Engineers) 30-node system and the IEEE 33-node system through a transformer was selected for simulation experiments. A 100MW wind turbine was connected at the 9th node of the IEEE 30-node system, equipped with two 10MWh plant energy storages; 3.5MWh centralized energy storage and two 1.5MWh distributed energy storages were respectively equipped at the 40th and 55th nodes of the IEEE 33-node system. The topology diagram is shown in the figure below. Figure 5 The wind power generation data is from an actual system in the country and is multiplied by a coefficient to adapt to the capacity of the system. Figure 4 In the figure, the numerical sequence number is the node sequence number of the 63-node system.

[0154] This experiment performs day-ahead optimization scheduling with a scheduling cycle of 1 hour, using the Pytorch framework and Python 3.7 as the programming environment. It is implemented on a computer equipped with an Intercore i7-12700H CPU and an NVIDIA RTX 3060 graphics card.

[0155] The scheduling control model converged after 3500 rounds of training and obtained the corresponding training results. The algorithm was compared with the mainstream reinforcement learning algorithms MATD3 (Multi-Agent Twin Delayed Deep Deterministic Policy Gradient) and MADDPG (Multi-Agent Deep Deterministic Policy Gradient). In order to verify the effect of adding noise interference to the action space, it was also compared with the noise-free MAA-TD3 algorithm (Multi-Agent Attention Twin Delayed DeepDeterministic Policy Gradient).

[0156] In the initial 500 rounds of training, the two algorithms that introduced the attention mechanism had a faster reward increase speed than the latter two algorithms (MATD3 and MADDPG), and the reward value increased from around -5.65 to around -5.25 within 400 rounds; while the reward value of the latter two algorithms only increased to around -5.35 within 400 rounds. In the later stage of training, the algorithm of the present invention converged after about 2000 rounds, and the reward value stabilized at around -5.05; however, the latter two algorithms tended to converge after 2500 and 3000 rounds, respectively, and the reward values ​​stabilized at around -5.22 and -5.25, which were slightly inferior to the algorithm proposed in the present invention in terms of convergence speed and convergence value. In addition, if Figure 5 As shown in the partial figure, the algorithm in the present invention has smaller fluctuations and is more stable than the MAA-TD3 algorithm in the later stage of training. This is because the introduction of noise increases the randomness of the input of the strategy evaluation network, which can prevent overfitting of the network and improve stability.

[0157] Using the trained dispatch model to optimize the charging and discharging strategy of base station energy storage can better improve the system economy and the level of wind power consumption. The charging and discharging of the source-load dual-side energy storage before and after cooperation are shown in Figure 6 (a)-Figure 6 (b) and Figure 7 (a)-Figure 7 (b). In the case of non-cooperation, shared energy storage and plant-site energy storage make independent decisions based on their own goals, as shown in Figure 6 (a)-Figure 6 (b). The charging and discharging behavior of shared energy storage follows the changes in electricity prices, charging during low electricity price periods (0-8 o'clock) and discharging during peak electricity price periods (9-14 o'clock) for arbitrage; plant-site energy storage follows the changes in load, charging and consuming wind power during low load valleys, such as (0-8 o'clock, 18-19 o'clock and 22-23 o'clock), and discharging when the load increases to reduce thermal power output and reduce costs, such as (9-15 o'clock and 20-21 o'clock). It can be seen that in the case of non-cooperation, the charging and discharging choices of dual-side energy storage are completely independent decisions based on their own goals.

[0158] The charge and discharge curves under cooperation are shown in Figure 7 (a)-Figure 7 (b). Under the guidance of the economic goals of the cooperative alliance, the shared energy storage and the power plant energy storage have similar goals and made similar actions, achieving strategic coordination. Specifically, in the 0-8 period, both sides of the energy storage choose to charge to reduce wind curtailment. Near the highest value. In the period from 9 to 23, the energy storage on both sides showed cooperation in behavior. Among them, in the period from 9 to 17, the load was at a high level, and the plant energy storage chose to discharge instead of thermal power output; the shared energy storage also discharged to reduce the pressure of thermal power, while taking into account the impact of electricity prices, and quickly discharged arbitrage during the peak electricity price period. Reduced to the lowest level. During the 18:00-19:00 period, the load decreased and the output of new energy increased. The energy storage on both sides was charged to absorb the new energy and prepare for the upcoming evening peak. From 20:00 to 21:00, the load entered the evening peak. The shared energy storage chose to discharge for arbitrage and replaced part of the thermal power output. The plant energy storage also chose to discharge to cope with the load peak. From 22:00 to 23:00, the load decreased and the output of new energy increased. The energy storage on both sides was charged to reduce wind curtailment.

[0159] The SOC change curve of some energy storage before and after cooperation is as follows: Figure 8 shown. Figure 8In the figure, the left broken line represents the SOC of the energy storage in the corresponding period of the horizontal axis, and the right shadow graph represents the charging and discharging power through the depth of color, that is, the rate of change of the energy storage SOC in the corresponding period. Specifically, orange represents the power of the charging process, and blue represents the power of the discharging process. The darker the color, the higher the power, and the greater the change of the energy storage SOC in the corresponding period. As can be seen from the figure, under the cooperative situation, the shared energy storage undergoes two rounds of charging and discharging in one day, and the utilization efficiency doubles. It is mainly manifested in that from 18:00 to 21:00, the shared energy storage is guided by the cooperative goal to charge and discharge, charging at the maximum power at 18:00 when the electricity price is low, reducing the charging power at 19:00 when the electricity price is high, and selling all the electricity from 20:00 to 21:00 (the evening peak period). In addition, the first round of high-power discharge was transferred to the load peak from 10:00 to 11:00, which shows that the shared energy storage can track the load better and adjust the output to reduce the peak load regulation pressure of thermal power.

[0160] As shown in Table 1, the collaborative strategy can reduce the total cost (i.e., the additional benefits of the alliance) by 2,560 yuan in one day's dispatch. According to the Shapley value method, the power supply operator gets an additional benefit of 1,830 yuan, and the energy storage supplier gets an additional benefit of 730 yuan. After the cooperation, the net income of the energy storage supplier (1,529 yuan) almost doubled compared with that before the cooperation (799 yuan), achieving its goal of pursuing economic benefits. It is worth noting that although the operating cost of the energy storage supplier increased by 679 yuan after the cooperation, the cost of wind curtailment was reduced by about 4,750 yuan through strategic collaboration. The improved Shapley allocation method not only compensated for the increased costs, but also increased the net income of the supplier.

[0161] Table 1 Comparison of income

[0162] .

[0163] It can be seen that the collaborative dispatching method of source-load dual-side energy storage based on multi-agents is proposed in this embodiment. On the basis of considering the different spatiotemporal and behavioral characteristics and basic needs of power supply operators and energy storage suppliers, a collaborative dispatching framework of source-load dual-side energy storage is established, and the operating status of various equipment such as new energy, thermal power and energy storage equipment on the power generation side and energy storage on the user side is further studied, and a collaborative dispatching model of source-load dual-side energy storage based on this alliance is established; and an improved reinforcement learning algorithm MAAN-TD3 is used to solve the dispatching strategy. The algorithm introduces an attention mechanism into the evaluation network to ensure that power supply operators and energy storage suppliers can capture the interdependence between cooperation, realize collaborative behavior perception and dynamic behavior adjustment, and at the same time, add random noise to the information input into the evaluation network to expand the exploration space of the agent. Finally, the operation plan with the minimum dispatching cost is sought with the total cost of the cooperative alliance as the goal, and the improved Shapley value is used for profit distribution, so as to reduce the amount of abandoned wind and improve the economic efficiency of the alliance.

[0164] It can be seen from the above embodiments that the technical solution provided by the present invention can improve the utilization efficiency of energy storage, the new energy consumption capacity of the power supply side and the system operation performance by coordinating the energy storage on both sides of the load and the source. On the other hand, it can improve the overall economic benefits on the basis of reducing the amount of abandoned wind.

[0165] Furthermore, on the basis of achieving the above-mentioned beneficial effects, each preferred scheme has also achieved the following beneficial effects: the Markov decision model can improve the efficiency of constructing the collaborative scheduling scheme; the introduction of the attention mechanism can realize the perception of collaborative behavior and the adjustment of dynamic behavior; adding random noise to expand the exploration space of the intelligent agent can improve the rationality of the constructed collaborative scheduling scheme; using the gradient descent method to update the Actor network and updating the parameters of the Critic network by minimizing the loss function can improve the accuracy of the collaborative scheduling model.

Claims

1. A multi-agent-based source-load dual-side energy storage coordinated scheduling method, characterized in that: The following steps are involved: Step S1: construct an objective function based on the multi-agent source-load dual-side energy storage system and set constraints; Step S2: constructing a Markov decision model based on the objective function and constraints; Step S3: Based on the Markov decision model, for each Markov decision cycle, the state space corresponding to the source-load dual-side energy storage system is obtained; Step S4: Substitute the state space into the Markov decision model for Q value fitting, obtain the corresponding action space and execute it; Step S1 specifically includes: Step S11: Construct an objective function with the goal of minimizing wind abandonment and maximizing energy storage utilization , the formula is as follows: ； In the formula, is the scheduling period ordinal number, is the total number of dispatching periods per day, is the wind abandonment penalty coefficient, For the period The actual output of the new energy units inside, For the period The maximum adjustable output of the new energy units in the is the station number of energy storage, For the The actual power during the energy storage period, For the Energy storage period Maximum power within Step S12: Set constraints, the formula is as follows: ； ； ； In the formula, For the period Internal generator injection node The active power, For the period Internal energy storage injection node The active power, For the period Internal Node The active power, , The time periods Internal Node ,node The voltage, , Node The real and imaginary parts of the admittance matrix, For the period Internal Line The voltage phase difference on is the sequence number of the generator set, For the period Neidi The output value of the active power of the generator set, , Respectively The minimum and maximum output values ​​of the active power of each generator set. For the period Intraday The reactive power output value of each generator set, , Respectively The minimum and maximum output values ​​of reactive power of each generator set, For the Energy storage period Minimum power within Step S2 specifically includes: constructing a Markov decision model based on the objective function and constraints ( , , , , );in, is the state space, and the formula is as follows: ； In the formula, is the power information, is the grid voltage, is the state of charge of the energy storage, For time; is the action space, and the formula is as follows: ； In the formula, is the charging and discharging power of the shared energy storage, The charging and discharging power of the power plant and station energy storage; is the state transition probability; is the reward function, specifically , the formula is as follows: ； In the formula, For the period The reward obtained by the interaction between the internal agent and the environment, Penalty for out-of-limit actions; is the discount factor, 0≦ ≦1; For each Markov decision cycle, obtain the corresponding state space ; The state space Substitute the Markov decision model to fit the Q value, obtain the corresponding action space and execute it; Step S4 is specifically as follows: In the Actor-Critic framework, the attention network is introduced into the Critic network, and the Q value is fitted through the MAAN-TD3 algorithm; Step S41: The global state-action pair ( s , a ) Input the multi-layer perceptron and stack the vectors in the order of the agents to get the vector ;in, s∈ , a∈ ; Step S42: vector Input the multi-head attention network to extract features and get feature output ; Step S43: Layer normalization feature output , and get the result x ; The result x Input residual network combined vector , then input the fully connected layer, and the output of the fully connected layer out As the Q value.

2. A multi-agent based source-load dual-side energy storage coordinated scheduling method as claimed in claim 1, characterized in that: Before step S41, the following step is also included: s Add random noise perturbation , constituting the exploration state , the formula is as follows: 。 3. A multi-agent based source-load dual-side energy storage coordinated scheduling method as claimed in claim 1, characterized in that: The Actor network is parameterized using a neural network and updated using gradient descent.

4. The method for coordinated dispatching of energy storage on both sides of source and load based on multi-agent as claimed in claim 1, characterized in that: The residual network is a Bellman residual network.

5. The method for coordinated dispatching of energy storage on both sides of source and load based on multi-agent as claimed in claim 1, characterized in that: It also includes minimizing the loss function Update the parameters of the Critic network. The formula is as follows: ； ； In the formula, are network parameters, For the The loss function of the Critic network of an agent is, is the second observation data stored in the experience pool, is the corresponding reward value, is the second observation data stored after updating, is the expected value of the loss function, is the target policy of the value function, Indicates The Q value obtained by the Actor network of the agent, is the target Q value, For the The reward value obtained by the agent is is the target strategy of the updated value function, Indicates the Q value obtained by updating the Actor network. is the number of agents, For the The corresponding action of the agent, After the update The corresponding action of the agent, For the observations of an agent; where =1,2,…, .

Images