A scheduling method for 5G base station photovoltaic storage microgrid system based on multi-agent deep reinforcement learning
Through the deep reinforcement learning technology of multi-agents, the energy storage scheduling of the 5G base station optical storage microgrid system has been optimized, and multiple 5G base station energy storage regulation problems have been solved, thus realizing the low storage and high release strategy under time-sharing electricity prices, reducing operating costs and improve system efficiency.
Patent Information
- Application Number
- CN202211025661.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-08-25
AI Technical Summary
It is difficult for the prior art to formulate optimal energy storage regulation plans in the case of multiple 5G base stations, resulting in low efficiency and high operating costs of microgrid systems.
Adopt the 5G base station optical storage microgrid system scheduling method based on multi-agent deep reinforcement learning. By building a reinforcement learning environment, using DDPG algorithm and actor-criticist network, the agent is trained to optimize the scheduling of energy storage equipment to minimize Purchase electricity from the main grid when the electricity price is high.
It has realized a low storage and high release strategy for energy storage equipment under time-sharing electricity prices, which has reduced the operating costs of 5G base stations and improved the working efficiency of the system.
Smart Images

Figure CN115459359B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of microgrid dispatching, and in particular to a 5G base station photovoltaic storage microgrid system dispatching method based on multi-agent deep reinforcement learning. Background Art
[0002] With the rapid development of mobile communications, along with the transition from 4G to 5G with high bandwidth and high speed, the power consumption of base stations is increasing, so energy saving of base stations has become an urgent need for the development of the times. Since RIS has the characteristics of low power consumption, flexible deployment, and amplified transmission signals, combining it with 5G base stations can not only eliminate interference from neighboring cells, but more importantly, it can reduce the transmission power of base stations and reduce the power consumption of base stations by enhancing the transmission signal, thereby further saving the operating costs of base stations. As the technology of clean energy becomes more and more mature, it is also a trend of the times to apply clean energy to mobile communications, and the development of renewable energy can not only protect the environment but also save energy to a large extent. However, renewable energy generation is intermittent and uncertain. In order to ensure the stability of renewable energy generation, it needs to be adjusted in combination with energy storage equipment. For 5G base stations, they have backup batteries and are idle most of the time. If they are fully combined with renewable energy generation, the operating costs of 5G base stations can be saved to a greater extent.
[0003] However, the combination of renewable energy and energy storage equipment will further bring about problems related to energy management. For this reason, it is necessary to use microgrids for flexible regulation. Most of the existing solutions are based on the DQN algorithm to achieve optimization and management of microgrids, but this solution is only for the case of a single microgrid. It is difficult to make the optimal energy storage regulation solution when multiple 5G base stations have multiple microgrids. Therefore, the present invention proposes a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning to improve the working efficiency of the 5G base station photovoltaic storage microgrid system and reduce its operating costs. Summary of the invention
[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning, which adopts the optimal energy storage scheduling strategy to schedule the energy storage equipment. The scheduling goal is to reduce the daily operating cost of the 5G base station, so that it can utilize the time-of-use electricity price difference, adjust the energy storage charge and discharge amount, and minimize the purchase of electricity from the main power grid when the electricity price is high, thereby improving the working efficiency of the 5G base station photovoltaic storage microgrid system and reducing its operating costs.
[0005] Technical solution: To solve the above technical problems, the present invention adopts the following technical solution: a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning, the method comprising the following steps:
[0006] Step (1): Obtain historical data of household microgrids in a certain area for one year, and form a training set with these data after preprocessing.
[0007] Step (2): Equip each 5G base station connected to the main grid with photovoltaic modules, and form a microgrid system model with the backup battery model of the 5G base station itself and its own load model. Establish an operation cost minimization model, power balance conditions and constraint conditions based on the system models;
[0008] Step (3): According to the system models and conditions, a reinforcement learning environment for the 5G base station photovoltaic storage microgrid system based on multi-agent deep reinforcement learning is constructed, including states, actions, and rewards, and an actor-critic network is constructed according to the DDPG algorithm;
[0009] Step (4): In the reinforcement learning environment of the 5G base station photovoltaic storage microgrid system with multi-agent deep reinforcement learning, the training set is input into the actor-critic network for training, and the trained model is saved;
[0010] Step (5): Collect data information such as the daily power generation, daily load, daily battery power status, and time-of-use electricity price of the photovoltaic power generation components in the 5G base station photovoltaic storage microgrid system model to be scheduled, and input the data information into the trained model after processing to obtain a scheduling plan that minimizes the operating cost of the 5G base station photovoltaic storage microgrid.
[0011] Preferably, the step (1) comprises the following steps:
[0012] Obtain one year's historical data of household microgrids in a certain area, including photovoltaic power generation, load and time-of-use electricity price information, and pre-process the acquired historical data, including data cleaning and data standardization.
[0013] Preferably, the step (2) comprises the following steps:
[0014] In the 5G base station photovoltaic storage microgrid system, the operating cost C t Including the cost of electricity traded between the base station and the main grid, the cost of electricity traded between base stations, and the cost of charging and discharging loss of the base station battery, during the transaction between the 5G base station photovoltaic storage microgrid system and the main grid, assuming that the number of base stations is M, for the i-th base station, the model is as follows:
[0015]
[0016] in, The price at which the base station buys electricity from the main power grid at the current moment; is the price at which the microgrid sells electricity to the main grid at the current moment, is the electricity price of the electricity traded between base stations, E i,grid (t) is the amount of electricity traded with the main grid, and its value depends on the battery charging and discharging action and the battery charge level. i,B (t) is the transaction power between base station i and base station j, C i,b (t) is the charge and discharge loss cost of the battery of the i-th base station at the t-th time, C i,b (t) follows the following relationship:
[0017] C i,b (t) = c ω *[E i,c (t)+E i,d (t)]
[0018] Among them, C i,b (t) is the charge and discharge loss cost of battery base station i at hour t, c ω is the battery loss coefficient, E i,c (t) is the actual charge capacity of the backup battery of the ith base station at time t, E i,d (t) is the actual discharge capacity of the backup battery of the ith base station at time t;
[0019] E i,grid (t) follows the following relationship:
[0020] E i,grid (t) = E i,PV (t)+E i,B (t)+E i,b (t)-E i,load (t)
[0021] E i,load (t) is the load consumption of base station i at time t, E i,PV (t) is the photovoltaic power generation at time t, E i,b (t) is the charge and discharge amount of the energy storage battery at time t.
[0022] E i,b (t) and E i,B (t) follows the following relationship:
[0023]
[0024]
[0025] Among them, P i,ch (t) is the maximum charging power of the ith base station at time t, P i,diss (t) is the maximum discharge power of the ith base station at time t, P i,j (t) represents the charge and discharge power between base stations i and j, Δt represents the time interval, t c and td Respectively represent the charging time and the discharging time;
[0026] E i,load (t) is the load consumption of base station i at time t, including the static power, dynamic power of the base station and the static power consumed by the users connected to the base station. Assuming that each base station is equipped with m antennas and K single-antenna mobile users, the number of reflective elements of RIS is n, and the power consumption of each element is Pn, then its model is expressed as follows
[0027]
[0028] Where ω represents the inverse of the transmit power amplifier efficiency, P i,k represents the transmission power of base station i, represents the static power consumed by mobile user k connected to base station i, P i,BS represents the static power of base station i, nP n represents the power consumption of RIS, Δt represents the time interval;
[0029] If the amount of electricity that can be charged or discharged by the battery reaches the upper and lower limits of the battery, the maximum charge and discharge power is equal to the ratio of the upward charge capacity or the downward discharge capacity to the time t. The battery has upper and lower limits for charge and discharge, so its constraints are:
[0030]
[0031] Among them, S i (t) is the energy storage battery capacity of base station i at time t, S i,max , S i,min are the maximum and minimum capacity of the energy storage battery of base station i, P i,ch (t) is the charging power of the backup battery of base station i at time t; P i,diss (t) is the discharge power of the backup battery of base station i at time t, are the maximum charging and discharging powers of the backup battery of base station i at hour t, respectively, which follow the following relationship:
[0032]
[0033] in, is the maximum charging power limit of the energy storage battery, is the maximum discharge power limit of the energy storage battery, and They are the upward charging capacity and downward discharging capacity of base station i, which follow the following relationship:
[0034]
[0035]
[0036] E i is the rated capacity of the backup battery;
[0037] From the above relationship, it can be concluded that the power of the base station energy storage battery model at time t+1 can be expressed as:
[0038]
[0039] Δt is the unit time interval; η c , η d are the charging and discharging efficiencies of the backup battery of base station i, and η c ∈(0,1),η d ∈(0,1); S i (t+1), S i (t) is the power of the backup battery of base station i at hour t and hour t+1 respectively.
[0040] Preferably, the step (3) comprises the following steps:
[0041] (3.1) Define the environmental state space S: the photovoltaic power generation E of multiple microgrids composed of multiple 5G base stations at a certain time t i,PV (t), load E i,load (t), energy storage battery capacity S i (t), current time-of-use electricity price p t 、Average electricity price in the past 24 hours , used as the environment state information s observable by the agent t , where the average electricity price in the past 24 hours is used as the electricity price for electricity transactions between base stations;
[0042] (3.2) Define the agent action space A: Each agent’s action a i,t It includes charging, discharging and idling of energy storage batteries, as well as buying and selling electricity with other base stations and the main power grid. At present, the action of dispatching the power of all 5G base station photovoltaic storage microgrid systems is a t express;
[0043] (3.3) Define the reward function R: Reward function r t The operating costs within 24 hours a day include the cost of buying electricity from the main power grid, the cost of battery charging and discharging losses, and the cost of electricity trading with other base stations. Penalties for overcharging and over-discharging of energy storage batteries are set, as well as the backup function of the battery and penalties for operating under high load.
[0044] (3.4) Constructing the Actor-Critic Network: Each agent has two networks, namely the actor network πi and critic network Q i , and each network has its corresponding target network, the actor network π i Update the parameters by maximizing the cumulative expected return The critic network Qi updates its parameters by minimizing the error between the evaluation value and the target value. Update the parameters of the target network through soft update.
[0045] Preferably, step (4) comprises the following steps:
[0046] The training process is trained by a centralized method, and the execution phase is executed by a distributed method. In the reinforcement learning training environment defined in step (3), the photovoltaic power generation, load and time-of-use electricity price data in the training set, the average electricity price in the past 24 hours, and the remaining power data of the energy storage battery generated by the interaction with the environment are input into the neural network as training data. Each data represents the state at a moment, and every 24 data represents a round, that is, one day. After all data are run, it means that the training is completed. The specific process is as follows:
[0047] (4.1) Set the total number of training segments to T;
[0048] (4.2) Initialize the weight parameters of each agent’s actor network and critic network and Weight parameters of the actor network and critic network of the target network and and the state set S, action set A, and capacity N of the experience pool D of the 5G base station photovoltaic storage microgrid system;
[0049] (4.3) Set the length of each segment to P;
[0050] (4.4) Select behavior a for each agent i,t ;
[0051] (4.5) Observe the initial state space s t , and the behavior a i,t Execute in each 5G base station photovoltaic storage microgrid system, change the state of the energy storage system, and calculate the reward value r obtained by executing the current action according to the reward function t ;
[0052] (4.6) Observe the new system state s t+1 , will (s t , a t , r t ,s t+1 ) is stored in the experience pool D;
[0053] (4.7) Randomly select a set of training data from the experience pool D for learning and update the critic network weight parameters by minimizing the loss function;
[0054] (4.8) Update the actor network using the sampled policy gradient method, and update the weight of the target network of each agent using the following method (where τ is a soft update parameter):
[0055]
[0056]
[0057] (4.9) Repeat steps (3.7) to (3.8) until all neural network parameters are updated;
[0058] (4.10) Repeat steps (3.4) to (3.8) until the training of each segment length is completed;
[0059] (4.11) Repeat steps (3.1) to (3.8) until the total number of fragments is trained and save the neural network parameters after training.
[0060] (4.12) According to the parameters of the neural network after training, at each time slot t, the agent makes a decision based on the learning strategy and executes action a i,t .
[0061] (4.13) Observe the state s at time slot t+1 i,t+1 ;
[0062] (4.14) Repeat steps (3.12) to (3.13) until the optimization decision is terminated.
[0063] Preferably, the step (5) comprises the following steps:
[0064] In each time period, the system collects data on photovoltaic power generation, load, current electricity price, and energy storage battery power in each 5G base station photovoltaic storage microgrid system and inputs them into the trained neural network to obtain the scheduling plan for each moment.
[0065] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:
[0066] The present invention describes the energy scheduling problem of the 5G base station photovoltaic storage microgrid system as a Markov decision process under the reinforcement learning framework. The intelligent agent does not need to know all the information about the environment. It only needs to be trained on the actual electricity consumption data of a certain area in the past year, so that the intelligent agent and the environment can continuously interact to finally obtain the optimal strategy. The optimal strategy is used to control the energy storage system, which can reduce the operating cost of the 5G base station and has strong practicality and portability. The present invention involves a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning. The 5G base station combined with RIS is combined with energy storage equipment, renewable energy, etc. to form a microgrid. Under the time-of-use electricity price, the energy storage equipment is low-storage and high-released to reduce the total operating cost of the 5G base station. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 It is a 5G base station photovoltaic storage microgrid system model;
[0068] Figure 2 Implementation steps for training the proposed method;
[0069] Figure 3 It is the MADDPG algorithm structure;
[0070] Figure 4 Energy storage dispatch for 5G base stations in commercial areas;
[0071] Figure 5 Energy storage dispatch for 5G base stations in residential areas;
[0072] Figure 6 Energy storage dispatch for 5G base stations in the work area;
[0073] Figure 7 Comparison of one-day costs of 5G base stations in three regions. DETAILED DESCRIPTION
[0074] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the accompanying drawings. The following embodiments are only preferred embodiments of the present invention, not all, and are only used to more clearly illustrate the technical solutions of the present invention, and cannot be used to limit the protection scope of the present invention. Based on the embodiments in the implementation mode, those skilled in the art can also make several improvements and deformations without departing from the technical principles of the present invention, and these improvements and deformations should also be regarded as the protection scope of the present invention.
[0075] The present invention mainly provides a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning, and adopts the optimal energy storage scheduling strategy to schedule the energy storage equipment, thereby achieving the purpose of improving the working efficiency of the 5G base station photovoltaic storage microgrid system and reducing operating costs.
[0076] like Figure 1As shown, the system model of the present invention includes: the microgrid includes photovoltaic power generation, a backup battery of a 5G base station, a load of the 5G base station, and RIS, etc. Photovoltaic power generation can convert solar energy into electrical energy to supply the AC load or convert the electrical energy into DC power through an inverter to supply the 5G base station load or the backup battery. When the photovoltaic power generation and the backup battery power cannot meet or exceed the base station load, power balance can be achieved by charging and discharging the battery or buying and selling electricity from the main power grid or other base stations. RIS can enhance the transmission signal of the base station and thus reduce the transmission power of the base station, which can reduce the base station load to a certain extent.
[0077] Combined with Figure 2 The specific steps of a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning disclosed in the present invention are described as follows:
[0078] Obtain one year of historical data of household microgrids in a certain area, and form a training set with these data after preprocessing:
[0079] Obtain the historical data of household microgrids in a certain area for one year, taking each hour as a time slot. These historical data include photovoltaic power generation, load and time-of-use electricity price information at each moment. Clean these data, remove data that does not conform to the format, and standardize the data before grouping them.
[0080] Establish a system model and establish an operating cost minimization model, power balance conditions and constraints based on the system component models:
[0081] I) During the transaction between the 5G base station photovoltaic storage microgrid system and the main grid, for simplicity, it is assumed that the number of 5G base stations in different areas is M, which are divided into 3 groups and divided into 3 microgrids. The photovoltaic power generation, battery power and load of each group are also summarized. Following the law of energy conservation, the following relationship is obtained:
[0082] E i,grid (t) = E i,PV (t)+E i,B (t)+E i,b (t)-E i,load (t)
[0083] Among them, E i,grid (t) is the transaction power of the main power grid, and its value depends on the battery charging and discharging action and the battery charge level, E i,grid (t)>0 is the amount of electricity purchased by the microgrid from the main grid in the current hour; E i,grid (t) < 0 is the amount of electricity sold by the microgrid to the main grid in the current hour. i,PV (t) is the photovoltaic power generation at time t, E i,b (t) is the charge and discharge amount of the energy storage battery at time t, Ei,B (t) is the transaction power between base station i and base station j, E i,load (t) is the load at time t, E i,b (t), E i,B (t) and E i,load (t) follows the following relationship:
[0084]
[0085]
[0086]
[0087] Where ω represents the inverse of the transmit power amplifier efficiency, P i,k represents the transmission power of base station i, represents the static power consumed by mobile user k connected to base station i, P i,BS represents the static power of base station i, nP n represents the power consumption of RIS, Δt represents the time interval, and the household load is taken as the value of the static power consumption of the base station;
[0088] II) The scheduling goal of the 5G base station photovoltaic storage microgrid system is to reduce the daily operating cost of the 5G base station C t , so that it can take advantage of the time-of-use electricity price difference, adjust the energy storage charging and discharging amount, and minimize the purchase of electricity from the main grid when the electricity price is high. The optimization objectives are as follows:
[0089]
[0090] in, The price of electricity purchased by the base station from the main power grid at the current moment depends on the time-of-use electricity price list at each moment; is the price at which the microgrid sells electricity to the main grid at the current moment, and its value is is the electricity price of the electricity traded between base stations, and its value is the average electricity price in the past 24 hours, C i,b (t) is the charge and discharge loss cost of the battery of the i-th base station at the t-th time, C i,b (t) follows the following relationship:
[0091] C i,b (t) = c ω *[E i,c (t)+E i,d (t)]
[0092] Among them, C i,b (t) is the charge and discharge loss cost of battery base station i at hour t, c ω The battery loss coefficient is taken as 0.007, E i,c(t) is the actual charge capacity of the backup battery of the ith base station at time t, E i,d (t) is the actual discharge capacity of the backup battery of the i-th base station at time t.
[0093] The constraints are:
[0094]
[0095] Among them, S i (t) is the backup battery capacity of base station i at time t. The upper and lower limits of the backup battery capacity of base station i are set to 0.8 and 0.1 of the maximum capacity respectively. i,ch (t) is the charging power of the backup battery of base station i at time t; P i,diss (t) is the discharge power of the backup battery of base station i at time t. It follows the following relationship:
[0096]
[0097] in The maximum charging power limit of the energy storage battery is 1KW / h. The maximum discharge power limit of the energy storage battery is 1KW / h. and They are the upward charging capacity and downward discharging capacity of base station i, which follow the following relationship:
[0098]
[0099]
[0100] Then the power status of the base station energy storage battery at time t+1 is:
[0101]
[0102] Δt is the unit time interval; η c , η d The charging and discharging efficiencies of the backup battery of base station i are both set to 0.95; S i (t+1), S i (t) are the state of charge (SOC) of the energy storage battery of base station i at hour t and hour t+1 respectively. Here E i The rated capacity of the backup battery is 50KW.h;
[0103] According to the system models and conditions, a reinforcement learning environment for the 5G base station photovoltaic storage microgrid system based on multi-agent deep reinforcement learning is constructed, including states, actions, and rewards. And an actor-critic network is constructed based on the DDPG algorithm:
[0104] Define the state space: In each time period, the system obtains the photovoltaic power generation E of each 5G base station photovoltaic storage microgrid system i,PV (t), load E i,load (t), current electricity price p t , Energy storage battery capacity S i (t) and average electricity price Status information i,t Constitute the state space, that is:
[0105] S: 1,t ,s 2,t , ..., s M,t}
[0106] Define action space: The action space A is defined as {a 1,t , a 2,t , ..., a M,t}, where a i,t represents the action of the i-th base station at time t, which includes the battery charging and discharging action and the action of trading electricity with other base stations, that is, in is {-1, 0, 1}, where -1 means the energy storage battery is discharging, 0 means the energy storage battery is idle, and 1 means the energy storage battery is charging. To trade electricity with the main power grid and other base stations;
[0107] Define the reward function: For the reward function r t , maximizing the long-term reward is to minimize the negative operating cost, and the penalty for overcharge and overdischarge, penalty 1, and the penalty for operating the battery at high load, penalty 2, are both 100:
[0108] r t =C t -penalty1-penalty2
[0109] Constructing an Actor-Critic network: Each agent has two networks, the actor network π i and critic network Q i , and each network has its corresponding target network. Actor network π i Update the parameters by maximizing the cumulative expected return Critics Network Q i Update the parameters by minimizing the error between the evaluation value and the target value Update the parameters of the target network through soft update.
[0110] In the reinforcement learning environment of the 5G base station photovoltaic storage microgrid system with multi-agent deep reinforcement learning, the photovoltaic power generation, load and time-of-use electricity price data in the grouped training set are input into the neural network for training together with the average electricity price in the past 24 hours and the remaining power data of the energy storage battery generated by interaction with the environment:
[0111] Step 1: Set the total number of training segments to 20,000;
[0112] Step 2: Initialize the weight parameters of all actor networks and critic networks, as well as the weight parameters of the actor network and critic network of the target network, the state set S of the 5G base station photovoltaic storage microgrid system, the action set A, and the capacity N of the experience pool D to 200000, the soft update parameter τ = 0.0001, the initial exploration noise to 6.0, set the learning rates of the actor network and the critic network to 0.0003 and 0.0001 respectively, and set it in 6000 training segments, the exploration noise is linearly reduced to 0.01, and they all contain an input layer, two hidden layers and one output layer, 280 neurons are set in each hidden layer of the actor network, and 200 neurons are set in each hidden layer of the critic network;
[0113] Step 3: Set the length of each segment to 24;
[0114] Step 4: Select behavior a for each agent i,t ;
[0115] Step 5: Observe the initial state space s t , and the behavior a i,t Execute in each 5G base station photovoltaic storage microgrid system, change the state of the energy storage system, and calculate the reward value r obtained by executing the current action according to the reward function t ;
[0116] Step 6: Observe the new system status t+1 , will (s t , a t , r t ,s t+1 ) is stored in the experience pool D;
[0117] Step7: Randomly select a set of data from the experience pool D for learning and update the critic network weight parameters by minimizing the loss function to minimize the loss function.
[0118] Step 8: Update the actor network using the sampled policy gradient method, and update the weights of the target network of each agent using the following method:
[0119]
[0120]
[0121] Step 9: Repeat Step 7 to Step 8 until all neural network parameters are updated.
[0122] Step 10: Repeat steps 4 to 8 until the training of each segment length is completed.
[0123] Step 11: Repeat steps 1 to 8 until the total number of fragments is trained and save the neural network parameters after training.
[0124] Step 12: Based on the trained neural network parameters, at each time slot t, the agent makes a decision based on the learning strategy and executes action a i,t .
[0125] Step 13: Observe the state s of time slot t+1 i,t+1 .
[0126] Step 14: Repeat steps 12 to 13 until the optimization decision is terminated.
[0127] Save the trained model parameters.
[0128] The daily power generation, daily load, daily battery power status, time-of-use electricity price and other data information of the photovoltaic power generation components in the 5G base station photovoltaic storage microgrid system model to be scheduled are collected, and the data information is processed and input into the trained model to obtain a scheduling plan that minimizes the operating cost of the 5G base station photovoltaic storage microgrid.
[0129] Through existing technologies, the photovoltaic power generation of 5G base stations in three areas in each period of the day can be obtained. According to the time-of-use electricity price, the electricity price information of each period and the average electricity price of the past 24 hours can be obtained. According to the power consumption of the 5G base station in each period, its load can be obtained. By monitoring the backup battery power of the 5G base station, the battery power can be obtained. These status information are input into the trained neural network model, and the changes in the load, electricity price and photovoltaic power generation of the 5G base stations in the three areas are analyzed. Figure 4 , Figure 5 and Figure 6 The specific action execution strategies are given respectively. It can be seen from the figure that it meets the characteristics of low charging and high discharging, and when the load is too high, the battery remains idle to ensure the backup battery backup. Figure 7As shown, there are three situations, including no photovoltaic components and no backup batteries, 50% photovoltaic power generation with backup batteries, and 100% photovoltaic power generation with backup batteries. It can be seen that the present invention can not only make full use of backup batteries to improve work efficiency, but more importantly, it can reduce the operating costs of 5G base stations through RIS and regulation of energy storage systems.
Claims
1. A 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning, characterized in that: The following steps are involved: Step (1) obtaining historical data of household microgrids in a certain area for one year, and forming a training set with these data after preprocessing; Step (2) equip each 5G base station connected to the main grid with photovoltaic modules, and form a microgrid system model with the backup battery model of the 5G base station itself and its own load model, and establish an operation cost minimization model, power balance conditions and constraint conditions according to each system model; Step (3) According to the system models and conditions, a reinforcement learning environment for the 5G base station photovoltaic storage microgrid system based on multi-agent deep reinforcement learning is constructed, including states, actions and rewards, and an actor-critic network is constructed according to the DDPG algorithm; Step (4) in the reinforcement learning environment of the 5G base station photovoltaic storage microgrid system with multi-agent deep reinforcement learning, the training set is converted into state information and input into the actor-critic network for training, and the trained model is saved; Step (5) collects the daily power generation, daily load, daily battery power status and time-of-use electricity price data information of the photovoltaic power generation components in the 5G base station photovoltaic storage microgrid system model to be scheduled, and inputs the data information into the trained model after processing to obtain a scheduling plan that minimizes the operating cost of the 5G base station photovoltaic storage microgrid.
2. According to a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning according to claim 1, it is characterized in that: The step (1) comprises the following steps: (1.1) Obtain one year of historical data of household microgrids in a certain area, including photovoltaic power generation, load, and time-of-use electricity price information; (1.2) Preprocess the acquired historical data, including data cleaning and data standardization.
3. According to a 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning according to claim 1, it is characterized in that: The step (2) comprises the following steps: In the 5G base station photovoltaic storage microgrid system, the operating cost C t Including the cost of electricity traded between the base station and the main grid, the cost of electricity traded between base stations, and the cost of charging and discharging loss of the base station battery, during the transaction between the 5G base station photovoltaic storage microgrid system and the main grid, assuming that the number of base stations is M, for the i-th base station, the model is as follows: in, The price at which the base station buys electricity from the main power grid at the current moment; is the price at which the microgrid sells electricity to the main grid at the current moment, is the electricity price of the electricity traded between base stations, E i,grid (t) is the amount of electricity traded with the main grid, and its value depends on the battery charging and discharging action and the battery charge level. i,B (t) is the transaction power between base station i and base station j, C i,b (t) is the charge and discharge loss cost of the battery of the i-th base station at the t-th time, C i,b (t) follows the following relationship: C i,b (t)=c ω *[E i,c (t)+E i,d (t)] Among them, C i,b (t) is the charging and discharging loss cost of the i-th battery base station at time t, c ω is the battery loss coefficient, E i,c (t) is the actual charge capacity of the backup battery of the ith base station at time t, E i,d (t) is the actual discharge capacity of the backup battery of the ith base station at time t; E i,grid (t) follows the following relationship: E i,grid (t)=E i,PV (t)+E i,B (t)+E i,b (t)-E i,load (t) E i,load (t) is the load consumption of base station i at time t, E i,PV (t) is the photovoltaic power generation at time t, E i,b (t) is the charge and discharge capacity of the energy storage battery at time t; E i,b (t) and E i,B (t) follows the following relationship: Among them, P i,ch (t) is the charging power of the backup battery of base station i at time t, F i,diss (t) is the discharge power of the backup battery of base station i at time t, P i,j (t) represents the charge and discharge power between base stations i and j, Δt represents the time interval, t c and t d Respectively represent the charging time and the discharging time; E i,load (t) is the load consumption of base station i at time t, including the static power, dynamic power of the base station and the static power consumed by the users connected to the base station. Assume that each base station is equipped with m antennas and K single-antenna mobile users, the number of reflective elements of RIS is n, and the power consumption of each element is P n , then its model is expressed as follows: Where ω represents the inverse of the transmit power amplifier efficiency, P i,k represents the transmission power of base station i, represents the static power consumed by mobile user k connected to base station i, P i,BS represents the static power of base station i, nP n represents the power consumption of RIS, Δt represents the time interval; If the amount of electricity that can be charged or discharged by the battery reaches the upper and lower limits of the battery, the maximum charge and discharge power is equal to the ratio of the upward charge capacity or the downward discharge capacity to the time t. The battery has upper and lower limits for charge and discharge, so its constraints are: Among them, S i (t) is the energy storage battery capacity of base station i at time t, S i,max , S i,min are the maximum and minimum capacity of the energy storage battery of base station i, P i,ch (t) is the charging power of the backup battery of base station i at time t, P i,diss (t) is the discharge power of the backup battery of base station i at time t, are the maximum charging and discharging powers of the backup battery of base station i at hour t, respectively, which follow the following relationship: in, is the maximum charging power limit of the energy storage battery, is the maximum discharge power limit of the energy storage battery, and They are the upward charging capacity and downward discharging capacity of base station i, which follow the following relationship: E i is the rated capacity of the backup battery; From the above relationship, the power of the base station energy storage battery model at time t+1 can be expressed as: Among them, η c , η d are the charging and discharging efficiencies of the backup battery of base station i, and η c ∈(0,1),η d ∈(0,1); S i (t+1), S i (t) is the power of the backup battery of base station i at hour t and hour t+1 respectively.
4. The 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: The step (3) comprises the following steps: (3.1) Define the environmental state space S: the photovoltaic power generation E of multiple microgrids composed of multiple 5G base stations at a certain time t i,PV (t), load E i,load (t), energy storage battery capacity S i (t), current time-of-use electricity price p t 、Average electricity price in the past 24 hours Used as the environment state information s that can be observed by the agent t , where the average electricity price in the past 24 hours is used as the electricity price for electricity transactions between base stations; (3.2) Define the agent action space A: Each agent’s action a i,t It includes charging, discharging and idling of energy storage batteries, as well as buying and selling electricity between other base stations and the main power grid. At this moment, the action of dispatching the power of all 5G base station photovoltaic storage microgrid systems is used for a t express; (3.3) Define the reward function R: Reward function r t The operating costs within 24 hours a day include the cost of buying electricity from the main power grid, the cost of battery charging and discharging losses, and the cost of electricity trading with other base stations. It also sets penalties for overcharging and over-discharging of energy storage batteries, as well as the backup function of the battery, and sets penalties for operating when the load exceeds the preset threshold. (3.4) Constructing the Actor-Critic Network: Each agent has two networks, namely the actor network π i and critic network Q i , and each network has its corresponding target network, the actor network π i Update the parameters by maximizing the cumulative expected return Critics Network Q i Update the parameters by minimizing the error between the evaluation value and the target value Update the parameters of the target network through soft update.
5. The 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning according to claim 4 is characterized in that: The step (4) comprises the following sub-steps: The training process is trained by a centralized method, and the execution phase is executed by a distributed method. In the reinforcement learning training environment defined in step (3), the photovoltaic power generation, load and time-of-use electricity price data in the training set, the average electricity price in the past 24 hours, and the remaining power data of the energy storage battery generated by the interaction with the environment are input into the neural network as training data. Each data represents the state at a moment, and every 24 data represents a round, that is, one day. After all the data are run, it means that the training is completed. The specific process is as follows: (4.1) Set the total number of training segments to T; (4.2) Initialize the weight parameters of each agent’s actor network and critic network and Weight parameters of the actor network and critic network of the target network and and the state set S, action set A, and capacity N of the experience pool D of the 5G base station photovoltaic storage microgrid system; (4.3) Set the length of each segment to P; (4.4) Select behavior a for each agent i,t ; (4.5) Observe the initial state space s t , and the behavior a i,t Execute in each 5G base station photovoltaic storage microgrid system, change the state of the energy storage system, and calculate the reward value r obtained by executing the current action according to the reward function t ; (4.6) Observe the new system state s t+1 , will (s t , a t , r t ,s t+1 ) is stored in the experience pool D; (4.7) Randomly select a set of data from the experience pool D for learning, and update the critic network weight parameters by minimizing the loss function; (4.8) Update the actor network using the sampled policy gradient method, and update the weight of the target network of each agent using the following method, where τ is the soft update parameter; (4.9) Repeat steps (3.7) to (3.8) until all neural network parameters are updated; (4.10) Repeat steps (3.4) to (3.8) until the training of each segment length is completed; (4.11) Repeat steps (3.1) to (3.8) until the total number of fragments is trained and save the neural network parameters after training; (4.12) According to the parameters of the neural network after training, at each time slot t, the agent makes a decision based on the learning strategy and executes action a i,t ; (4.13) Observe the state s at time slot t+1 i,t+1 ; (4.14) Repeat steps (3.12) to (3.13) until the optimization decision is terminated.
6. The 5G base station photovoltaic storage microgrid system scheduling method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The step (5) includes the following sub-steps: in each time period, the system collects the photovoltaic power generation, load, current electricity price, and energy storage battery power data in each 5G base station photovoltaic storage microgrid system and inputs them into the trained neural network to obtain the scheduling plan for each moment.
Citation Information
Patent Citations
Micro-power-grid energy storage scheduling method and device based on deep Q-value network (DQN) reinforcement learning
CN109347149A
Microgrid energy scheduling method based on double-Q-value network deep reinforcement learning
CN112117760A
Charging and discharging scheduling method for electric vehicles in microgrid under time-of-use price
US20170337646A1
Method and apparatus for constructing dispatching model of integrated energy system, medium, and electronic device
WO2022160705A1
