Base station energy storage scheduling method and device based on time and graph embedded reinforcement learning

Abstract

The present invention discloses a base station energy storage scheduling method and device based on time and graph embedded reinforcement learning, which belongs to the field of distribution network scheduling. According to the power grid system containing 5G base station energy storage, a graph topology is obtained; B power grid training data are randomly selected to train the reinforcement learning agent decision network model to obtain a trained reinforcement learning agent decision network model; according to the trained reinforcement learning agent decision network model, the state observation information of the power grid system at the current moment is analyzed and decided to obtain the original action vector at the current moment and perform safety constraints to obtain a safe action vector; finally, the power grid system is adjusted through the safe action vector to complete the scheduling of the power grid system. The present invention can perform scheduling actions according to the real-time status information of the distribution network, and fully utilize the dispatchable capacity of the base station energy storage on the premise of ensuring the safe standby capacity of the base station energy storage, promote the consumption of new energy, and reduce the operating costs of the power grid.

Description

Technical Field

[0001] The present invention belongs to the field of distribution network scheduling, and in particular relates to a base station energy storage scheduling method and device based on time and graph embedded reinforcement learning. Background Art

[0002] In recent years, 5G base stations have been built and promoted on a large scale. By December 2024, China had built and commissioned over 4.1 million 5G base stations. During construction, 5G base stations were equipped with energy storage batteries as uninterruptible power supplies. This ensures power supply reliability and communication service quality while providing a degree of power flexibility. This allows for the potential to participate in grid scheduling, leverage peak load shaving, and accommodate renewable energy generation. However, in the current high-reliability power grid environment, the energy storage batteries in 5G base stations remain idle for extended periods, remaining in a floating charge state. This results in significant power wastage and failure to fully utilize their energy storage advantages. It is urgent to prioritize the characteristics of 5G base stations and control their participation in grid scheduling to maximize their energy storage scheduling potential. Furthermore, because the power load of 5G base stations is closely linked to the communication load, the large number of 5G base stations introduces additional uncertainty into the scheduling process, placing higher demands on the algorithm's environmental adaptability and ability to withstand the effects of uncertainty.

[0003] At present, domestic and foreign scholars have conducted extensive research on the control strategy of 5G base station energy storage participating in distribution networks containing new energy. Traditional 5G base station energy storage scheduling algorithms mainly use mathematical optimization methods. The advantage is that the algorithm results have a relatively complete theoretical explanation. However, they rely on accurate prediction information and are more susceptible to environmental interference, which may lead to deviations from the optimal solution. In addition, the model solution process takes a long time and has a large amount of calculation, making it difficult to use this algorithm in real-time adjustment processes.

[0004] With the rise of artificial intelligence algorithms, reinforcement learning algorithms have achieved remarkable results in solving sequential decision-making problems. Reinforcement learning's ability to learn scheduling rules based on environmental data rivals expert experience, enabling it to provide effective adjustment plans in real time and offering strong resilience to uncertainty. However, existing reinforcement learning scheduling algorithms are relatively straightforward in extracting features from the real-time status information of 5G base station energy storage. They lack a targeted structural design tailored to the characteristics of 5G base station distribution, underutilize the structural features of the base station distribution graph, and fail to fully explore these relationships. This fails to fully leverage the advantages of reinforcement learning, and leaves much to be desired in terms of decision-making capabilities. Therefore, there is an urgent need for an algorithm that can fully exploit the features of the base station graph, deliver better decision-making results, and provide more effective real-time scheduling strategies for the safe and economical operation of distribution networks containing 5G base station energy storage resources. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the existing technology and provide a base station energy storage scheduling method and device based on time and graph embedded reinforcement learning.

[0006] The object of the present invention is achieved through the following technical solutions: a base station energy storage scheduling method based on time and graph embedded reinforcement learning, comprising the following steps:

[0007] (1) Obtaining the graph topology based on the power grid system containing 5G base station energy storage ;

[0008] (2) Obtaining grid training data of the grid system at multiple times and putting it into the experience replay pool;

[0009] (3) Randomly select B power grid training data from the experience replay pool to train the reinforcement learning agent decision network model to obtain a trained reinforcement learning agent decision network model;

[0010] (4) According to the trained reinforcement learning agent decision network model, the state observation information of the power grid system at the current moment is analyzed and decided, the original action vector at the current moment is obtained and safety constraints are performed to obtain the safe action vector; finally, the power grid system is adjusted through the safe action vector to complete the scheduling of the power grid system.

[0011] Furthermore, the step (1) is specifically as follows:

[0012] According to the power grid system containing 5G base station energy storage, determine the connection relationship between node branches, and obtain the access location and number of nodes of the upper power grid , the number of new energy units , access location and capacity parameters and the number of 5G base stations and access locations as a graph topology of the power grid system .

[0013] Furthermore, the power grid training data includes time State observation information , original motion vector , reward value and time State observation information ;

[0014] The moment State observation information Including time ,time The grid node status information collection , Active power aggregate of new energy units and the active power aggregate of the upper grid The time The grid node status information collection Including time The active load, reactive load, voltage amplitude, phase angle and charge state of 5G base station energy storage of all nodes; Active power aggregate of new energy units Including time The active power injected by the new energy units of all nodes; The active power aggregate of the upper grid Including time Active power injected by the upper network of all nodes;

[0015] The moment Reward value By combining the operating costs of the entire power grid system including 5G base station energy storage It is calculated after normalization and subtraction operation.

[0016] Furthermore, the moment The original motion vector Obtained through the following sub-steps:

[0017] (a.1) First, use the Time2Vec time encoding embedding model to embed the time Processing to obtain the time coding vector ;

[0018] (a.2) Then use the GCN graph convolution model to collect the grid node status information Processing is performed to obtain the node feature matrix ;

[0019] (a.3) Time encoding vector , node feature matrix ,time The output power of all new energy units and time The interactive power of the upper grid After splicing and splicing into column vectors and inputting them into the fully connected neural network model, the original action vector is obtained. ; The original motion vector Including time of The predicted output power of each new energy unit, the predicted output power of the upper network, and The predicted output power of a 5G base station.

[0020] Furthermore, the moment State observation information Obtained through the following sub-steps:

[0021] (b.1) The original action vector Input to the power grid system including 5G base station energy storage, adjust the upper power grid, New energy units and The output power of each 5G base station; after adjustment, the time is calculated by the active power flow equation and reactive power flow equation in the AC power flow. Active load, reactive load, voltage amplitude, phase angle, active power injected by energy units and active power injected by upper network of all nodes;

[0022] (b.2) The time is calculated by the energy transfer relationship of the battery power The state of charge of the 5G base station energy storage of all nodes;

[0023] (b.3) Change the time Active load, reactive load, voltage amplitude, phase angle, active power injected by new energy units and active power injected by the upper network of all nodes, as well as time The state of charge of the 5G base station energy storage of all nodes is combined to obtain the time State observation information .

[0024] Furthermore, the reinforcement learning agent decision network model consists of an actor network and a critic network; the actor network includes a main policy network and a target policy network; the critic network includes a main action value network and a target action value network; the neural network parameters of the main policy network are , the neural network parameters of the target policy network are , the neural network parameters of the main action value network are , the neural network parameters of the target action value network are .

[0025] Furthermore, the step (3) is specifically as follows:

[0026] (3.1) Randomly select B power grid training data from the experience replay pool. Any power grid training data is represented by a 4-tuple ,in, Indicates time The state observation information, Indicates time The original motion vector, Indicates time The reward value, Indicates time Status observation information;

[0027] (3.2) The loss function is constructed by B power grid training data and the policy gradient objective function ;

[0028] The loss function The calculation formula is: ,in, represents the evaluation value of the main action value network, represents the target value of the target action value network;

[0029] The target value of the target action value network for ,in, Represents the discount factor in the reinforcement learning algorithm; Represents the target strategy network's response to state observation information Predicted output action;

[0030] The policy gradient objective function The calculation formula is: ;

[0031] The policy gradient objective function is then calculated Gradient : ,in, represents the gradient of the main action value network to the main policy network, Represents the main strategy network's own neural network parameters gradient;

[0032] (3.3) By minimizing the loss function And the gradient Maximize the policy gradient objective function through gradient ascent , optimize the neural network parameters of the main action value network and the neural network parameters of the main strategy network;

[0033] (3.4) Use Soft update the neural network parameters of the target policy network using Soft update the neural network parameters of the target action value network, where represents the neural network parameters after the soft update of the target policy network, Represents the neural network parameters after soft update of the target action value network;

[0034] (3.5) Repeat steps (3.1) to (3.4) until the training process of the reinforcement learning agent decision network model converges to obtain a trained reinforcement learning agent decision network model.

[0035] Furthermore, the original action vector at the current moment is subjected to safety constraints to obtain a safe action vector, specifically:

[0036] By minimizing the original action vector at the current moment The predicted output power of each 5G base station and the current time The squared Euclidean distance between the constrained predicted output powers of the 5G base stations and the current moment The corresponding content in the original action vector at the current moment is replaced by the constrained predicted output power of each 5G base station to obtain the corresponding safe action vector;

[0037] The current moment Constrained predicted output power of a 5G base station The following constraints need to be met:

[0038] c.1) At the current moment No. Energy storage capacity of a 5G base station Greater than the next moment No. The upper limit of energy storage capacity of a 5G base station When , the constraint is satisfied: and ,in, Indicates the The discharge power limit of each 5G base station;

[0039] c.2) At the current moment No. Energy storage capacity of a 5G base station Less than the next moment No. The lower limit of energy storage capacity for a 5G base station When , the constraint is satisfied: and ,in, Indicates the The charging power limit of each 5G base station;

[0040] c.3) Otherwise, the constraint is satisfied: and .

[0041] The present invention also includes a base station energy storage scheduling device based on time and graph embedded reinforcement learning, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used for the above-mentioned base station energy storage scheduling method based on time and graph embedded reinforcement learning.

[0042] The present invention also includes a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the above-mentioned base station energy storage scheduling method based on time and graph embedded reinforcement learning is implemented.

[0043] The beneficial effects of the present invention are:

[0044] 1) In response to the temporal characteristics and graph-like structural characteristics of 5G base stations participating in distribution network scheduling, this paper designs a deep deterministic policy gradient algorithm (T2V-GCN-DDPG) that integrates temporal embedding coding and graph convolutional neural networks. This algorithm improves the ability of reinforcement learning agents to extract real-time state information from the environment, thereby improving the performance of real-time decision-making.

[0045] 2) To ensure the safety of decision-making actions, this paper proposes a safety constraint layer based on quadratic programming. This fine-tunes the original action output by the model. Through minimal adjustments, the action is brought into the safety domain, ensuring the safety of the backup power of the base station energy storage.

[0046] 3) The structural improvement proposed in this invention plays a key role in improving the decision-making ability of the intelligent agent. The final decision-making results show that the proposed algorithm can reasonably mobilize the energy storage resources of 5G base stations, reduce the operating costs of the power grid, and improve the overall operational economy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a topological diagram of the power grid system including 5G base station energy storage;

[0048] Figure 2 This is a flow chart of a base station energy storage scheduling method based on time and graph embedded reinforcement learning;

[0049] Figure 3 A comparison of the cumulative rewards obtained after training before and after adding temporal encoding and graph convolutional embedding;

[0050] Figure 4 A comparison of the cumulative rewards obtained before and after training with safety constraints;

[0051] Figure 5 This is a structural diagram of a base station energy storage scheduling device based on time and graph embedded reinforcement learning in Example 2. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention, rather than to represent all embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention, rather than to represent all embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0054] Example 1: The power grid system targeted in this example is a power grid system including 5G base station energy storage. The topology diagram of the power grid system including 5G base station energy storage is as follows: Figure 1 As shown. The power grid system with 5G base station energy storage is divided into different zones based on functional attributes, including residential areas, campus areas, commercial areas, and industrial areas. The user power load and 5G base station communication load ratio vary significantly between zones. Therefore, the present invention sets different power load curves and base station communication load fluctuation curves for the four different zones (Zones 1, 2, 3, and 4). This power grid system with 5G base station energy storage is equipped with a 1.5MW photovoltaic generator (PV) at node 6 and a 3.5MW wind turbine (WT) at node 12. The 5G base station energy storage converts the branch impedance between different nodes into an approximate distance. Based on the 5G base station communication coverage radius principle, the number of 5G base stations required for each node is determined. The power grid system has a scheduling period of 24 hours, with a scheduling step interval of 1 hour. To simulate the prediction deviation in real-world scenarios, the predicted information for the power grid environment is the actual value superimposed on the prediction deviation. The prediction deviation follows a Gaussian distribution with a mean of 0 and a standard deviation of 10% of the actual value.

[0055] In this example, the power grid environment runs on MATLAB 2022b, using MATPOWER 7.1 to calculate real-time power flow information. The intelligent agent algorithm model runs on Python 3.8 and is built using PyTorch 2.0.1. Communication between the power grid environment and the intelligent agent is established using the MATLAB engine for Python 9.13.

[0056] like Figure 2 As shown, the present invention provides a base station energy storage scheduling method based on time and graph embedded reinforcement learning, comprising the following steps:

[0057] (1) Obtain the graph topology based on the power grid system containing 5G base station energy storage.

[0058] The step (1) is specifically as follows: according to the power grid system containing 5G base station energy storage, determine the connection relationship between node branches, obtain the access location of the upper power grid and the number of nodes , the number of new energy units , access location and capacity parameters and the number of 5G base stations and access locations as a graph topology of the power grid system .

[0059] (2) Obtain grid training data of the grid system at multiple times and put it into the experience replay pool.

[0060] Experience replay pool moments The grid training data includes the time State observation information , original motion vector , reward value and time State observation information .

[0061] The moment State observation information Obtained from the power grid system including 5G base station energy storage. State observation information Including time ,time The grid node status information collection , Active power aggregate of new energy units and the active power aggregate of the upper grid ,Right now The moment The grid node status information collection Including time The active load, reactive load, voltage amplitude, phase angle and charge state of 5G base station energy storage of all nodes. The grid node status information collection for ,in, , Indicates time No. The characteristic information vector of each node, , Indicates time No. The active load of each node, Indicates time No. The reactive load of each node, Indicates time No. The voltage amplitude of each node, Indicates time No. The phase angle of each node, Indicates time No. The charge status of the 5G base station energy storage of each node; the grid node status information collection The size is The moment Active power aggregate of new energy units Including time The active power injected by the energy units of all nodes at the moment Active power aggregate of new energy units for ,in, Indicates time No. The active power injected by the new energy unit at each node. The active power aggregate of the upper grid Including time The active power injected by the upper network of all nodes at the moment The active power aggregate of the upper grid for .

[0062] The moment The original motion vector Obtained through the following sub-steps:

[0063] In this embodiment, an improved T2V-GCN-DDPG model is proposed. The T2V-GCN-DDPG model includes a Time2Vec temporal coding embedding model, a GCN graph convolution model, and a fully connected neural network model DDPG.

[0064] (a.1) First, use the Time2Vec time encoding embedding model to embed the time Processing to obtain the time coding vector , specifically:

[0065] The moment Input into the Time2Vec time encoding embedding model, first the time Perform linear transformation using Calculate the first element of the time code vector , and serves as the non-periodic feature of the time-encoding vector; and Indicates that Time2Vec calculates the Elements The model weight parameter is then changed periodically. Calculate the other elements of the time coding vector as the periodic characteristics of the time coding vector; finally, embed the Time2Vec time coding into the k-dimensional time coding vector calculated by the model Output. The purpose of Time2Vec encoding is to convert time Encoded into a k-dimensional time encoding vector that can represent other implicit information : , which enables subsequent models to simultaneously mine the non-periodic and periodic characteristics of time information, facilitating algorithm decision-making.

[0066] (a.2) Then use the GCN graph convolution model to collect the grid node status information Processing is performed to obtain the node feature matrix , specifically including the following sub-steps:

[0067] (a.2.1) The GCN graph convolution model consists of an L-layer graph convolutional network; according to the graph topology Get the size of The adjacency matrix , the adjacency matrix Used to represent the power grid system containing 5G base station energy storage The interconnection relationship between nodes.

[0068] (a.2.2) When the grid node status information is collected After inputting into the GCN graph convolution model, the grid node status information set is directly used Extracted node feature matrix as the first layer of graph convolutional network ;use Extract the implicit features in the graph information and obtain the first Extracted node feature matrix of layer graph convolutional network ,in, , Indicates size The identity matrix of for , Representation matrix Middle Row, No. Elements of the column, , Representation matrix Middle Row, No. Elements of the column, , . Indicates the The weight matrix of the layer graph convolutional network; Indicates the The extracted node feature matrix of the layer graph convolutional network; Denotes a nonlinear activation function. In this example, the ReLU nonlinear activation function is used. Through layer-by-layer convolution operations, the GCN graph convolution model can aggregate information from each node's neighboring nodes, which means that as the number of network layers increases, nodes can capture more global information through their embeddings.

[0069] (a.3) Time encoding vector , node feature matrix ,time The output power of all new energy units and time The interactive power of the upper grid After splicing, the column vector is input into the fully connected neural network model DDPG to obtain the original action vector ; The original motion vector Including time of The predicted output power of each new energy unit, the predicted output power of the upper network, and The predicted output power of a 5G base station is:

[0070] First, the time encoding vector , node feature matrix ,time The output power of all new energy units and time The interactive power of the upper grid Concatenate all elements in into a single-dimensional column vector ; Then the column vector Input to the fully connected neural network model, after forward propagation, output the original action vector ; The original motion vector for ,in, Indicates time No. The predicted output power of each new energy unit, Indicates time The predicted output power of the upper network, Indicates time No. The predicted output power of a 5G base station, In order to adapt to the characteristics of loads in different areas, 5G base station energy storage adopts a partition scheduling method, and 5G base stations in the same group share the same scheduling strategy. The output of the original action vector The output power of each controllable unit in the entire power grid system including 5G base station energy storage can be used as a coordinated action instruction for the distribution network to control the joint action of each unit and improve the overall operation effect of the power grid.

[0071] The moment State observation information Obtained through the following sub-steps:

[0072] (b.1) The original action vector Input to the power grid system including 5G base station energy storage, adjust the upper power grid, New energy units and The output power of each 5G base station; after adjustment, the time is calculated by the active power flow equation and reactive power flow equation in the AC power flow. The active load, reactive load, voltage amplitude, phase angle, active power injected by energy units and active power injected by the upper network of all nodes.

[0073] Specifically: the original motion vector Input to the power grid system including 5G base station energy storage, adjust the upper power grid, New energy units and The output power of each 5G base station; after adjustment, the active power flow equation in the AC power flow is used to calculate the power flow. Relationship with the reactive power flow equation Calculate the time The active load, reactive load, voltage amplitude, phase angle, active power injected by new energy units and active power injected by the upper network of all nodes, among which, Indicates time Time The active power injected by the upper grid at each node, Indicates time Time The active power injected by the new energy units at each node, Indicates time Time The active load at each node, Indicates time Time The active power injected by the 5G base station at each node, Indicates time Time The reactive power injected by the upper power grid at each node, Indicates time Time The reactive power injected by the new energy units at each node, Indicates time Time Reactive load at each node; Indicates time Time The voltage amplitude at each node, Indicates time Time The voltage amplitude at each node, ; Indicates the node admittance matrix of the power grid system. Row, No. the real parts of the elements of the column, Indicates the node admittance matrix of the power grid system. Row, No. imaginary part of the elements of the column; Indicates time Time Node and The phase angle difference between the nodes, , Indicates time Time The phase angle of each node, Indicates time Time The phase angle of each node.

[0074] (b.2) The time is calculated by the energy transfer relationship of the battery power The charge status of the 5G base station energy storage of all nodes.

[0075] Specifically: Energy transfer relationship based on battery power Calculate the state of charge of the 5G base station energy storage at each node ,in, Indicates the charging efficiency, represents the discharge efficiency, Indicates time Time The charging power of a 5G base station, time Time The discharge power of a 5G base station, Indicates the The rated battery capacity of a 5G base station.

[0076] (b.3) Change the time ,time Active load, reactive load, voltage amplitude, phase angle, active power injected by new energy units and active power injected by the upper network of all nodes, as well as time The state of charge of the 5G base station energy storage of all nodes is combined to obtain the time State observation information .

[0077] The moment State observation information Including time ,time The grid node status information collection , Active power aggregate of new energy units and the active power aggregate of the upper grid ,Right now The moment The grid node status information collection Including time The active load, reactive load, voltage amplitude, phase angle and charge state of 5G base station energy storage of all nodes. The grid node status information collection for ,in, Indicates time No. The characteristic information vector of each node, , Indicates time No. The active load of each node, Indicates time No. The reactive load of each node, Indicates time No. The voltage amplitude of each node, Indicates time No. The phase angle of each node, Indicates time No. The charge status of the 5G base station energy storage of each node; the grid node status information collection The size is The moment Active power aggregate of new energy units Including time The active power injected by the energy units of all nodes at the moment Active power aggregate of new energy units for ,in, Indicates time No. The active power injected by the new energy unit at each node. The active power aggregate of the upper grid Including time The active power injected by the upper network of all nodes at the moment The active power aggregate of the upper grid for .

[0078] The moment Reward value By combining the operating costs of the entire power grid system including 5G base station energy storage After normalization, the subtraction operation is performed and the calculation is obtained, specifically:

[0079] To reward environmental feedback for distribution networks containing 5G base station energy storage It can guide the training of reinforcement learning agents. As a reward function, Indicates the overall operating cost of the power grid, which is determined by the power purchase cost of the upper power grid. 2. Cost of curtailing wind and solar power generation And the maintenance cost of 5G base stations participating in scheduling Composition; According to the active power of the upper power grid and electricity purchase cost per unit time , calculate the power purchase cost of the upper power grid ; Based on the cost of curtailing wind and solar power per unit time and new energy units At the moment Maximum power With actual power The difference between the two is used to calculate the cost of abandoning wind and solar power of new energy units. ; According to 5G base station energy storage Cost coefficient for participating in scheduling 、 Time 5G base station Charging power , discharge power , calculate the maintenance cost of all 5G base stations participating in the scheduling ; In addition, the dimensionless Through the normalization function Convert it into a normalized value between [0,1], and then negate the normalized value to transform the original cost minimization problem into a cumulative reward maximization problem, which facilitates algorithm training and convergence.

[0080] (3) Randomly select B power grid training data from the experience replay pool to train the reinforcement learning agent decision network model to obtain a trained reinforcement learning agent decision network model.

[0081] The reinforcement learning agent decision network model consists of an actor network and a critic network; the actor network includes a main policy network and a target policy network; the critic network includes a main action value network and a target action value network; the neural network parameters of the main policy network are , the neural network parameters of the target policy network are , the neural network parameters of the main action value network are , the neural network parameters of the target action value network are .

[0082] The step (3) specifically includes the following sub-steps:

[0083] (3.1) Randomly select B power grid training data from the experience replay pool. Any power grid training data is represented by a 4-tuple ,in, Indicates time The state observation information, Indicates time The original motion vector, Indicates time The reward value, Indicates time state observation information.

[0084] (3.2) Through B power grid training data Construct the loss function and the policy gradient objective function .

[0085] The loss function The calculation formula is: ,in, represents the evaluation value of the main action value network, Represents the target value of the target action-value network.

[0086] The target value of the target action value network for ,in, Represents the discount factor in the reinforcement learning algorithm, which is a hyperparameter of the algorithm; Represents the target strategy network's response to state observation information The predicted output action.

[0087] The policy gradient objective function The calculation formula is: .

[0088] The policy gradient objective function is then calculated Gradient : ,in, represents the gradient of the main action value network to the main policy network, Represents the main strategy network's own neural network parameters The gradient of , and then use the chain derivation rule to get the main action value network right gradient.

[0089] (3.3) By minimizing the loss function And the gradient Maximize the policy gradient objective function through gradient ascent , optimize the neural network parameters of the main action value network and the neural network parameters of the main strategy network.

[0090] (3.4) Use Soft update the neural network parameters of the target policy network using Soft update the neural network parameters of the target action value network, where represents the neural network parameters after the soft update of the target policy network, Represents the neural network parameters after soft update of the target action value network.

[0091] (3.5) Repeat steps (3.1) to (3.4) until the training process of the reinforcement learning agent decision network model converges to obtain a trained reinforcement learning agent decision network model.

[0092] (4) Analyze and make decisions on the current state observation information of the power grid system based on the trained reinforcement learning agent decision network model to obtain the original action vector at the current moment And perform safety constraints to obtain the safety action vector ; Finally, through the safe action vector Adjust the power grid system and complete the dispatch of the power grid system.

[0093] The original action vector at the current moment is subjected to safety constraints to obtain a safe action vector, specifically:

[0094] The current moment The original motion vector for ,in, Indicates the current time No. The predicted output power of each new energy unit, Indicates the current time The predicted output power of the upper network, Indicates the current time No. The predicted output power of a 5G base station.

[0095] By minimizing the original action vector at the current moment middle The predicted output power of each 5G base station and the current time The squared Euclidean distance between the constrained predicted output powers of 5G base stations , and use the current moment The corresponding content in the original action vector at the current moment is replaced by the predicted output power of the 5G base station after constraint to obtain the safe action vector corresponding to the current moment; wherein, Indicates the current moment The predicted output power of a 5G base station, Indicates the current moment Constrained predicted output power of a 5G base station.

[0096] The safety action vector corresponding to the current moment for .

[0097] The current moment Constrained predicted output power of a 5G base station The following constraints need to be met:

[0098] c.1) At the current moment No. Energy storage capacity of a 5G base station Greater than the next moment No. The upper limit of energy storage capacity of a 5G base station When, it indicates the current time No. A 5G base station can only discharge, corresponding to the current moment No. Predicted output power of 5G base stations Must be less than 0, that is, the output power needs to reduce the energy storage capacity to a level lower than the next moment The upper limit of energy storage capacity. Constrained predicted output power of a 5G base station The following constraints need to be met: and ,in, Indicates the The discharge power limit of a 5G base station, Indicates discharge efficiency.

[0099] c.2) At the current moment No. Energy storage capacity of a 5G base station Less than the next moment No. The lower limit of energy storage capacity for a 5G base station When, it indicates the current time No. 5G base stations can only charge, corresponding to the current time No. Predicted output power of 5G base stations Greater than 0, that is, the output power needs to make the energy storage capacity rise to a higher level than the next moment The lower limit of energy storage capacity. Constrained predicted output power of a 5G base station The following constraints need to be met: and ,in, Indicates the The charging power limit of each 5G base station, Indicates charging efficiency.

[0100] c.3) On the contrary, when the current moment No. Energy storage capacity of a 5G base station No less than the next moment No. The lower limit of energy storage capacity for a 5G base station and no more than the next moment No. The upper limit of energy storage capacity of a 5G base station When, it indicates the current time No. 5G base stations can be charged and discharged, and the discharge does not exceed the next moment The lower limit of energy storage capacity, charging does not exceed the next moment The upper limit of energy storage capacity. Constrained predicted output power of a 5G base station The following constraints need to be met: and .

[0101] The charging and discharging power of each 5G base station needs to meet the energy storage capacity constraint of the 5G base station, that is, for the current moment No. Energy storage capacity of a 5G base station Needs to be satisfied , where the next moment No. The upper limit of energy storage capacity of a 5G base station Depends on The maximum battery capacity parameters of a 5G base station; the next moment No. The lower limit of energy storage capacity for a 5G base station Depends on the next moment No. Communication load power of a 5G base station At minimum backup time The integral value on ,Right now ; Current time No. Energy storage capacity of a 5G base station By It is obtained by converting the physical parameters of the battery of a 5G base station.

[0102] Next moment No. The lower limit of energy storage capacity for a 5G base station By The physical parameters of the battery of a 5G base station and the next moment It is calculated from the communication load rate of the base station.

[0103] Since the original action vector in the output The predicted output power of a 5G base station cannot fully guarantee the safe operation of the 5G base station, so through safety constraints, The predicted output power of each 5G base station is safely constrained to ensure the safe operation of the 5G base station.

[0104] The principle of implementing safety constraints is to use the predicted output power as the decision variable, and then No. The upper and lower limits of energy storage capacity for a 5G base station , reverse the current moment No. Energy storage capacity of a 5G base station In the case of , the power boundary of charging and discharging can be taken as the safe feasible region of decision variables. of The predicted output power of each 5G base station and the current time of The squared Euclidean distance between the constrained predicted output powers of 5G base stations The predicted output power that is not within the safe feasible region can be moved to the nearest feasible region boundary using the minimum moving distance, thereby obtaining the constrained predicted output power that meets the safety constraints. This method preserves the original action information as much as possible while ensuring the safety constraint boundary of the energy storage.

[0105] To verify the effectiveness of the base station energy storage scheduling method based on time and graph embedded reinforcement learning proposed in this invention. Figure 3 Shows the cumulative rewards obtained from training before and after adding temporal encoding and graph convolutional embedding The comparison of Figure 3 From the comparison, it can be seen that the convergence speed and stability of the improved T2V-GCN-DDPG model proposed in this patent are significantly improved compared with the unimproved fully connected neural network model DDPG, indicating that the use of the Time2Vec time coding embedding model and the GCN graph convolution model can improve the feature extraction ability of the intelligent agent and significantly accelerate the convergence process of the model.

[0106] Figure 4 The comparison of the cumulative rewards obtained before and after training with safety constraints is shown. Figure 4 It can be seen that under the condition of safety constraints, the convergence speed of algorithm training will be significantly accelerated, the cumulative reward of the final convergence result will be higher, and the decision-making level will be stronger.

[0107] Example 2: This embodiment relates to a base station energy storage scheduling device based on time and graph embedded reinforcement learning, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used for a base station energy storage scheduling method based on time and graph embedded reinforcement learning in the above-mentioned Example 1; the device embodiment can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer.

[0108] like Figure 5 At the hardware level, the device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0109] Improvements to a technology can be clearly categorized as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can integrate a digital system onto a PLD through their own programming, eliminating the need for a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using "logic compiler" software. This is similar to the software compiler used during program development. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0110] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0111] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0112] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0113] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0114] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0115] Example 3: An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the base station energy storage scheduling method based on time and graph embedded reinforcement learning of the above-mentioned Example 1 is implemented.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A base station energy storage scheduling method based on time and graph embedded reinforcement learning, characterized in that: The following steps are involved: (1) Obtaining the graph topology based on the power grid system containing 5G base station energy storage ; The step (1) is specifically as follows: According to the power grid system containing 5G base station energy storage, determine the connection relationship between node branches, and obtain the access location and number of nodes of the upper power grid , the number of new energy units , access location and capacity parameters and the number of 5G base stations and access locations as a graph topology of the power grid system ; (2) Obtaining grid training data of the grid system at multiple times and putting it into the experience replay pool; The power grid training data includes time State observation information , original motion vector , reward value and time State observation information ; The moment State observation information Including time ,time The grid node status information collection , Active power aggregate of new energy units and the active power aggregate of the upper grid The time The grid node status information collection Including time The active load, reactive load, voltage amplitude, phase angle and charge state of 5G base station energy storage of all nodes; Active power aggregate of new energy units Including time The active power injected by the new energy units of all nodes; The active power aggregate of the upper grid Including time Active power injected by the upper network of all nodes; The moment Reward value By combining the operating costs of the entire power grid system including 5G base station energy storage It is calculated after normalization and then subtraction operation; The moment The original motion vector Obtained through the following sub-steps: (a.1) First, use the Time2Vec time encoding embedding model to embed the time Processing to obtain the time coding vector ; (a.2) Then use the GCN graph convolution model to collect the grid node status information Processing is performed to obtain the node feature matrix ; (a.3) Time encoding vector , node feature matrix ,time The output power of all new energy units and time The interactive power of the upper grid After splicing and splicing into column vectors and inputting them into the fully connected neural network model, the original action vector is obtained. ; The original motion vector Including time of The predicted output power of each new energy unit, the predicted output power of the upper network, and The predicted output power of each 5G base station; (3) Randomly select B power grid training data from the experience replay pool to train the reinforcement learning agent decision network model to obtain a trained reinforcement learning agent decision network model; (4) According to the trained reinforcement learning agent decision network model, the state observation information of the power grid system at the current moment is analyzed and decided, the original action vector at the current moment is obtained and safety constraints are performed to obtain the safe action vector; finally, the power grid system is adjusted through the safe action vector to complete the scheduling of the power grid system.

2. A base station energy storage scheduling method based on time and graph embedded reinforcement learning according to claim 1, characterized in that: The moment State observation information Obtained through the following sub-steps: (b.1) The original action vector Input to the power grid system including 5G base station energy storage, adjust the upper power grid, New energy units and The output power of each 5G base station; after adjustment, the time is calculated by the active power flow equation and reactive power flow equation in the AC power flow. Active load, reactive load, voltage amplitude, phase angle, active power injected by energy units and active power injected by upper network of all nodes; (b.2) The time is calculated by the energy transfer relationship of the battery power The state of charge of the 5G base station energy storage of all nodes; (b.3) Change the time Active load, reactive load, voltage amplitude, phase angle, active power injected by new energy units and active power injected by the upper network of all nodes, as well as time The state of charge of the 5G base station energy storage of all nodes is combined to obtain the time State observation information .

3. The base station energy storage scheduling method based on time and graph embedded reinforcement learning according to claim 1 is characterized in that: The reinforcement learning agent decision network model consists of an actor network and a critic network; the actor network includes a main policy network and a target policy network; the critic network includes a main action value network and a target action value network; the neural network parameters of the main policy network are , the neural network parameters of the target policy network are , the neural network parameters of the main action value network are , the neural network parameters of the target action value network are .

4. The base station energy storage scheduling method based on time and graph embedded reinforcement learning according to claim 3 is characterized in that: The step (3) is specifically as follows: (3.1) Randomly select B power grid training data from the experience replay pool. Any power grid training data is represented by a 4-tuple ,in, Indicates time The state observation information, Indicates time The original motion vector, Indicates time The reward value, Indicates time Status observation information; (3.2) The loss function is constructed by B power grid training data and the policy gradient objective function ; The loss function The calculation formula is: ,in, represents the evaluation value of the main action value network, represents the target value of the target action value network; The target value of the target action value network for ,in, Represents the discount factor in the reinforcement learning algorithm; Represents the target strategy network's response to state observation information Predicted output action; The policy gradient objective function The calculation formula is: ; The policy gradient objective function is then calculated Gradient : ,in, represents the gradient of the main action value network to the main policy network, Represents the main strategy network's own neural network parameters gradient; (3.3) By minimizing the loss function And the gradient Maximize the policy gradient objective function through gradient ascent , optimize the neural network parameters of the main action value network and the neural network parameters of the main strategy network; (3.4) Use Soft update the neural network parameters of the target policy network using Soft update the neural network parameters of the target action value network, where represents the neural network parameters after the soft update of the target policy network, Represents the neural network parameters after soft update of the target action value network; (3.5) Repeat steps (3.1) to (3.4) until the training process of the reinforcement learning agent decision network model converges to obtain a trained reinforcement learning agent decision network model.

5. The base station energy storage scheduling method based on time and graph embedded reinforcement learning according to claim 4 is characterized in that: The original action vector at the current moment is subjected to safety constraints to obtain a safe action vector, specifically: By minimizing the original action vector at the current moment The predicted output power of each 5G base station and the current time The squared Euclidean distance between the constrained predicted output powers of the 5G base stations and the current moment The corresponding content in the original action vector at the current moment is replaced by the constrained predicted output power of each 5G base station to obtain the corresponding safe action vector; The current moment Constrained predicted output power of a 5G base station The following constraints need to be met: c.1) At the current moment No. Energy storage capacity of a 5G base station Greater than the next moment No. The upper limit of energy storage capacity of a 5G base station When , the constraint is satisfied: and ,in, Indicates the The discharge power limit of each 5G base station; c.2) At the current moment No. Energy storage capacity of a 5G base station Less than the next moment No. The lower limit of energy storage capacity for a 5G base station When , the constraint is satisfied: and ,in, Indicates the The charging power limit of each 5G base station; c.3) Otherwise, the constraint is satisfied: and .

6. A base station energy storage scheduling device based on time and graph embedded reinforcement learning, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a base station energy storage scheduling method based on time and graph embedded reinforcement learning as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, the base station energy storage scheduling method based on time and graph embedded reinforcement learning as described in any one of claims 1 to 5 is implemented.

Images