An energy management intelligent control method and system based on a reinforcement learning algorithm

Abstract

The application discloses an energy management intelligent control method and system based on a reinforcement learning algorithm, and comprises the following steps: S1, collecting data of a generator set, an energy storage device and a power grid, and defining a state space and an action space of a reinforcement learning system; S2, extracting features of the generator set, the energy storage device and the power grid, and calculating a reinforcement learning observation state vector; S3, designing a basic reward according to the state space and the action space of the reinforcement learning system; S4, designing an early warning reward and an energy balance factor, and combining the basic reward to calculate a final reward; S5, taking a Double DQN network as an optimization network of the reinforcement learning system to update parameters; and S6, calculating control instructions of the generator set and the energy storage device according to the Double DQN network after the parameter update and sending the control instructions. The application can solve the problems of rigid control strategy and insufficient coordination when a traditional method deals with dynamic changes of the power grid.

Description

Technical Field

[0001] This invention relates to the field of energy management technology, and in particular to an intelligent control method and system for energy management based on reinforcement learning algorithms. Background Technology

[0002] With the large-scale grid connection of renewable energy and the increasing volatility of power load, the real-time monitoring and intelligent control of energy management systems face severe challenges. In particular, the increasing penetration rate of intermittent energy sources such as wind and solar power, coupled with their output uncertainty, has led to increasingly prominent problems such as grid frequency deviation and voltage flicker, further amplifying the adaptability deficiencies of traditional management methods. Traditional energy management methods mainly rely on rule-based control strategies and static threshold early warning mechanisms. For example, they trigger generator or energy storage device start-up and shutdown commands through preset load thresholds, or use simple time-series prediction models for passive response. These methods are often designed based on historical steady-state data and have significant limitations: First, traditional rule-based and static model-based control strategies lack adaptability to complex and variable grid environments, making it difficult to achieve accurate and rapid responses to sudden changes in power generation or severe load fluctuations. Second, when faced with sudden changes in power generation or a sharp increase in load, traditional methods struggle to coordinate and optimize the dynamic balance between energy storage device charging and discharging and generator output, leading not only to secondary problems such as exceeding energy storage SOC limits and frequent unit start-ups and shutdowns, but also to over- or under-adjustment issues. Recent research has attempted to introduce model predictive control or fuzzy logic algorithms, but these remain limited by model mismatch risks and reliance on expert experience. Therefore, there is an urgent need for a technology that can integrate real-time monitoring and intelligent decision-making, using a collaborative framework of deep learning and reinforcement learning to autonomously mine and learn the dynamic characteristics of the power grid, achieving a coordinated output of proactive early warning and optimal control strategies. Summary of the Invention

[0003] In view of this, the present invention aims to provide an intelligent control method and system for energy management based on reinforcement learning algorithms, so as to solve the problems of rigid control strategies and insufficient coordination in traditional methods when dealing with dynamic changes in the power grid.

[0004] An intelligent control method for energy management based on reinforcement learning algorithms includes:

[0005] S1: Collect data from generator sets, energy storage devices, and the power grid, and define the state space and action space of the reinforcement learning system;

[0006] S2: Based on the data from the generator set, energy storage device, and power grid, extract the features of the generator set, energy storage device, and power grid, and calculate the reinforcement learning observation state vector. The reinforcement learning observation state vector is calculated based on the generator set features, energy storage device features, and power grid features, combined with an improved attention mechanism. The power grid features are generated as follows: based on the power grid data, extract the power grid frequency deviation features, regional load power features, and bus voltage features through one-dimensional convolution. Combine this with the Sigmoid function to generate a frequency-dominant gating vector, and weight the power grid frequency deviation features, regional load power features, and bus voltage features to obtain the power grid features.

[0007] S3: Design basic rewards based on the state space and action space of the reinforcement learning system;

[0008] S4: Based on the state space and action space of the reinforcement learning system, design early warning rewards and energy balance factors, and calculate the final reward in combination with the basic rewards; the energy balance factor is generated in the following way: generate dynamic weights of the remaining capacity of the energy storage device based on the remaining capacity of the energy storage device, calculate the difference between the power generation of the generator set and the power generation of the energy storage device, and generate the energy balance factor by combining exponential function calculation;

[0009] S5: Using the Double DQN network as the optimization network of the reinforcement learning system, the parameters are updated to obtain the Double DQN network with updated parameters;

[0010] S6: Calculate and send control commands for generator sets and energy storage devices based on the updated parameters of the Double DQN network.

[0011] It should be further explained that Double-DQN (Double Deep Q-Network), proposed by Hasselt et al. in 2015, is an improved deep reinforcement learning algorithm used to address the overestimation problem that may exist in traditional DQN (Deep Q-Network) when estimating Q-values.

[0012] Furthermore, step S1 also includes:

[0013] S11: Collect generator set data, including generator set equipment number, real-time output value, and maximum output value; collect energy storage device data, including energy storage device equipment number, real-time output value, and remaining capacity; collect grid data, including grid frequency deviation, regional load power, and bus voltage.

[0014] S12: Calculate the difference between the real-time output value and the maximum output value of the generator set to obtain the generator set regulation margin;

[0015] S13: Define the state space of the reinforcement learning system, including: generator set equipment number, generator set real-time output value, generator set regulation margin, energy storage device equipment number, energy storage device real-time output value, energy storage device remaining capacity, as well as grid frequency deviation, regional load power value, bus voltage, and early warning accuracy; the early warning accuracy is based on the output label and the true label of the reinforcement learning system, and is obtained by statistically analyzing the proportion of correct samples in the output label through a fixed-length sliding window;

[0016] S14: Define the action space of the reinforcement learning system, including: generator power output and energy storage device power output.

[0017] Furthermore, in step S2, the improved attention mechanism calculation method includes:

[0018] The generator set features and energy storage device features are concatenated to generate a query vector using a multilayer perceptron. The grid features are used as both the key vector and the value vector. The product of the query vector and the transpose of the key vector is calculated, then divided by the square root of the feature dimension. A query vector correction term, processed by linear transformation of the attention weight matrix and ReLU activation, is then added. Finally, an attention vector is generated using the Softmax function. The attention vector is multiplied by the value vector and then added to the query vector after layer normalization to generate a reinforcement learning observation state vector.

[0019] Furthermore, step S2 also includes:

[0020] S21: Based on the generator set equipment number, real-time output value, and regulation margin, and the energy storage device equipment number, real-time output value, and remaining capacity, combined with the unique thermal code, the generator set characteristics and energy storage device characteristics are generated. The calculation method is as follows:

[0021]

[0022]

[0023] in, Characteristics of generator sets, For ReLU function, This is the characteristic weight matrix of the generator set. It is a one-hot encoding. Assign a generator set equipment number. This represents the real-time output value of the generator set. For the generator set's regulation margin, As a characteristic of energy storage devices, This is the characteristic weight matrix of the energy storage device. For energy storage device equipment number, This represents the real-time output value of the energy storage device. This represents the remaining capacity of the energy storage device;

[0024] S22: Based on the aforementioned power grid frequency deviation, regional load power, and bus voltage, power grid characteristics are generated using a one-dimensional convolution and gating mechanism. The calculation method is as follows:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in, This is a characteristic of power grid frequency deviation. For power grid frequency deviation, For one-dimensional convolution, For regional load power characteristics, For regional load power, This refers to the characteristics of the bus voltage. Bus voltage For frequency-dominant gating vectors, For the Sigmoid function, For splicing operations, As a characteristic of the power grid, For Hadamah accumulation;

[0031] S23: Based on the characteristics of the generator set, energy storage device, and power grid, a reinforcement learning observation state vector is generated using an improved attention mechanism. The calculation method is as follows:

[0032]

[0033]

[0034]

[0035]

[0036] in, For query vector, It is a multilayer perceptron. For key vectors, For value vectors, For attention vectors, Here is the attention weight matrix. For the Softmax function, For transpose operation, For feature dimension, To reinforce the learning of the observed state vector, For layer normalization.

[0037] It should be further explained that this invention generates a reinforcement learning observation state vector by fusing generator set features, energy storage device features, and grid features, and uses this vector for basic reward calculation in step S3, achieving intelligent compression of the state space and extraction of key features. Specifically: First, in terms of mathematical principles, an improved attention mechanism dynamically allocates the association weights of generator set features and grid features, enabling low-frequency, high-value features (such as transient information of grid frequency deviation) to obtain higher activation intensity, while suppressing high-frequency noise components, thus solving the reward feedback delay problem caused by the explosion of state space dimensions in traditional methods. Second, compared with the benchmark algorithm that directly uses physical quantities as state inputs, this invention encodes the scattered device parameters and grid parameters into a unified low-dimensional vector expression, enabling the reward function to accurately capture the implicit coupling relationship in multi-device collaboration, significantly improving the convergence speed of the control strategy. Third, for the power oscillation problem in the grid scenario, the reinforcement learning observation state vector constructed in this invention achieves dynamic weighting of grid frequency deviation features through a gating mechanism, enabling the reward function to automatically enhance its sensitivity in key events such as frequency limit risk. This adaptive reinforcement capability based on feature importance is something that traditional fixed-weight reward functions cannot achieve.

[0038] Furthermore, step S3 also includes:

[0039] S31: Calculate the generator set reward based on the real-time output value of the generator set in the state space, the generator set regulation margin, and the generator set power generation in the action space. The calculation method is as follows:

[0040]

[0041]

[0042] in, This refers to the generator set regulation efficiency factor. To take the absolute value, The power output of the generator set in the action space. To obtain the maximum value, It is a non-zero positive minimum value. As a reward for generator sets, This is the penalty coefficient for the generator set's regulation efficiency.

[0043] S32: Calculate the energy storage device reward based on the remaining capacity of the energy storage device in the state space, the real-time output value of the energy storage device, and the power generation of the energy storage device in the action space. The calculation method is as follows:

[0044]

[0045]

[0046] in, For the health deviation of energy storage devices, Incentives for energy storage devices, This is the penalty coefficient for energy storage devices. To generate power for energy storage devices in the action space. Rated power of the energy storage device;

[0047] S33: Calculate the grid stability bonus based on the grid frequency deviation, regional load power, and bus voltage in the state space. The calculation method is as follows:

[0048]

[0049] in, As a reward for grid stability, This is the power grid stability weight matrix;

[0050] S34: Calculate the basic reward based on generator set rewards, energy storage device rewards, grid stability rewards, and reinforcement learning observation state vectors. The calculation method is as follows:

[0051]

[0052] in, As a basic reward.

[0053] Furthermore, step S4 also includes:

[0054] The basic reward, early warning reward, and energy balance factor are concatenated, and a reward gating vector is generated using the Sigmoid function. The linear combination result of the basic reward and early warning reward is weighted using the reward gating vector, and the linear transformation result of the energy balance factor is weighted to generate the final reward.

[0055] Furthermore, step S4 also includes:

[0056] S41: Calculate the warning reward based on the warning accuracy in the state space. The calculation method is as follows:

[0057]

[0058]

[0059] in, Let be the exponential moving average of the warning accuracy at time t in the state space, where t is the time index. For smoothing coefficients, Let be the early warning accuracy at time t-1 in the state space. Let be the exponential moving average of the warning accuracy at time t-1 in the state space. As a reward for early warning, As the warning penalty coefficient, It is the hyperbolic tangent function. Let be the early warning accuracy at time t in the state space. To take the exponent of the natural constant;

[0060] S42: Calculate the energy balance factor based on the generator power output and energy storage power output in the action space, and the remaining capacity of the energy storage device in the state space. The calculation method is as follows:

[0061]

[0062]

[0063] in, The remaining capacity of the energy storage device is dynamically weighted. This is the penalty factor for the remaining capacity of the energy storage device. Energy balance factor;

[0064] S43: Calculate the final reward based on the basic reward, early warning reward, and energy balance factor. The calculation method is as follows:

[0065]

[0066]

[0067] in, To reward the gating vector, To reward the gating weight matrix, As the final reward, Based on the basic reward weight matrix, For the early warning reward weight matrix, This is the energy balance weight matrix.

[0068] It should be further explained that this invention achieves intelligent dynamic balance between the output of the generator set and the energy storage device through an innovative reward function design, while embedding early warning performance into the decision-making process. Mathematically, the energy balance factor is based on the difference between the generator set's power output and the energy storage device's power output, combined with the remaining capacity of the energy storage device. A dynamic weight for the remaining capacity of the energy storage device is generated using a Sigmoid function, and then further combined with an exponential function and a ReLU function to generate the energy balance factor. The larger the output difference between the generator set and the energy storage device, the lower the energy balance factor, indicating a poorer output matching degree, resulting in a stronger penalty in the final reward. When the remaining energy storage capacity deviates from the health center by 0.5, the energy balance factor... A decrease in the balance factor indicates that the energy storage is in a dangerous state of overcharging / over-discharging, which also triggers system penalties. The early warning reward adopts a nonlinear fusion of the exponential moving average of the early warning accuracy and the current accuracy, and compresses the dimensional differences through the hyperbolic tangent function to ensure that the reward value decays significantly in high false alarm scenarios. Compared with the traditional fixed proportion balance strategy and independent early warning mechanism, this method uses the energy balance factor as the carrier and then dynamically allocates the weights of the basic reward, early warning reward and balance factor through the reward gating vector. During transient processes such as voltage drops, the contribution ratio of the balance factor is automatically increased, which solves the control lag problem caused by the disconnect between the balance strategy and the early warning in the traditional method, and simultaneously optimizes the equipment life and the transient stability of the power grid.

[0069] Furthermore, step S5 also includes:

[0070] S51: Use the network as the optimization network of the reinforcement learning system, and initialize the main network parameters and target network parameters of the Double DQN network; the Double DQN network includes discrete action branches and continuous action branches;

[0071] S52: Based on the main network parameters and target network parameters of the Double DQN network, select the optimal discrete action through the discrete action branch, calculate the continuous action limit value through the continuous action branch, and then calculate the target Q value and the action at the next moment.

[0072] S53: Based on the target Q value and the action at the next time step, calculate the network gradient using the mean squared error loss function, and use the gradient descent algorithm to update the main network parameters and target network parameters of the Double DQN network to obtain the Double DQN network with updated parameters.

[0073] Furthermore, step S6 also includes:

[0074] S61: Based on the Double DQN network with updated parameters, the generator set operation mode command and energy storage device operation mode command are decoded and parsed through discrete actions. The real-time output value command of the generator set and the real-time output value command of the energy storage device are calculated through continuous action conversion to obtain the set of equipment control commands.

[0075] S62: Perform power limiting processing on the set of equipment control commands to obtain a set of equipment execution commands after power limiting processing;

[0076] S63: Based on the set of equipment execution instructions processed by power limiting, the system sends operation mode instructions and real-time output value instructions to generator sets and energy storage devices through the power monitoring system, and obtains equipment execution status feedback and real-time grid operation parameters.

[0077] This invention also discloses an intelligent control system for energy management based on a reinforcement learning algorithm, comprising:

[0078] Data acquisition module: Collects data from generator sets, energy storage devices, and the power grid, and defines the state space and action space of the reinforcement learning system;

[0079] Reinforcement learning observation state vector extraction module: Based on the data of generator sets, energy storage devices, and power grids, extract the features of generator sets, energy storage devices, and power grids, and calculate the reinforcement learning observation state vector;

[0080] Basic Reward Module: Design basic rewards based on the state space and action space of the reinforcement learning system;

[0081] Final reward module: Based on the state space and action space of the reinforcement learning system, design early warning rewards and energy balance factors, and calculate the final reward by combining the basic rewards;

[0082] Optimize network parameter update module: Use the Double DQN network as the optimization network of the reinforcement learning system, update the parameters, and obtain the Double DQN network with updated parameters;

[0083] Control command sending module: Calculates and sends control commands for generator sets and energy storage devices based on the updated parameters of the Double DQN network.

[0084] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0085] (1) In view of the problem that traditional energy management methods rely on fixed rules and static thresholds, which are difficult to cope with the rigidity and lack of coordination of control strategies caused by dynamic changes in the power grid, this invention uses a reinforcement learning framework to integrate generator set features, energy storage device features and power grid features to generate reinforcement learning observation state vectors, thereby realizing intelligent compression of the state space and key feature reinforcement extraction. Secondly, in addition to basic rewards, early warning accuracy and output balance of generator sets and energy storage devices are also included in the unified optimization objective. Finally, network optimization is achieved through a Double DQN network, which solves the core problems of poor adaptability of rule control to complex operating conditions and the disconnect between early warning and regulation.

[0086] (2) The present invention designs an improved attention mechanism to dynamically allocate the association weights of power generation equipment features and grid features, so that low-frequency high-value features (such as transient information of grid frequency deviation) can obtain higher activation intensity; the reinforcement learning observation state vector generated by the improved attention mechanism encodes the scattered equipment parameters and grid parameters into a unified low-dimensional vector expression, so that the reward function can accurately capture the implicit coupling relationship in multi-device collaboration, automatically enhance sensitivity when there is a risk of frequency exceeding the limit, and significantly improve the convergence speed of the control strategy, overcoming the control lag defect caused by fixed weights in traditional methods.

[0087] (3) This invention achieves dynamic balance between the output of generator sets and energy storage devices through the joint design of energy balance factor and early warning reward. When the difference between the power generation of generator sets and the power generation of energy storage devices increases or the energy storage capacity deviates from the healthy range, the energy balance factor is automatically reduced to strengthen the penalty. At the same time, the weights of basic reward, early warning reward and balance factor are dynamically allocated through reward gating vector. The contribution ratio of balance factor is automatically increased during transient processes such as voltage drop. This solves the control lag problem caused by the disconnect between balance strategy and early warning in traditional methods, and simultaneously optimizes equipment life and grid transient stability. Attached Figure Description

[0088] Figure 1 A flowchart illustrating an intelligent control method for energy management based on reinforcement learning algorithm provided by this invention;

[0089] Figure 2 A schematic diagram of the algorithm flow for extracting the observation state vector in reinforcement learning provided by this invention;

[0090] Figure 3 This is a diagram illustrating the algorithm structure of the multilayer perceptron in step S42 of the present invention.

[0091] Figure 4 This is a diagram illustrating the convergence process of reinforcement learning training provided by the present invention. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0093] Example 1: An intelligent control method for energy management based on reinforcement learning algorithms, such as... Figure 1 As shown, it includes the following steps:

[0094] S1: Collect data from generator sets, energy storage devices, and the power grid, and define the state space and action space of the reinforcement learning system;

[0095] S11: Collect generator set data, including generator set equipment number, real-time output value, and maximum output value; collect energy storage device data, including energy storage device equipment number, real-time output value, and remaining capacity; collect grid data, including grid frequency deviation, regional load power, and bus voltage.

[0096] S12: Calculate the difference between the real-time output value and the maximum output value of the generator set to obtain the generator set regulation margin;

[0097] S13: Define the state space of the reinforcement learning system, including: generator set equipment number, generator set real-time output value, generator set regulation margin, energy storage device equipment number, energy storage device real-time output value, energy storage device remaining capacity, as well as grid frequency deviation, regional load power value, bus voltage, and early warning accuracy; the early warning accuracy is based on the output label and the true label of the reinforcement learning system, and is obtained by statistically analyzing the proportion of correct samples in the output label through a fixed-length sliding window;

[0098] S14: Define the action space of the reinforcement learning system, including: generator power output and energy storage device power output.

[0099] S2: Based on the data from the generator set, energy storage device, and power grid, extract the features of the generator set, energy storage device, and power grid, and calculate the reinforcement learning observation state vector. The reinforcement learning observation state vector is calculated based on the generator set features, energy storage device features, and power grid features, combined with an improved attention mechanism. The power grid features are generated as follows: based on the power grid data, extract the power grid frequency deviation features, regional load power features, and bus voltage features through one-dimensional convolution. Combine this with the Sigmoid function to generate a frequency-dominant gating vector, and weight the power grid frequency deviation features, regional load power features, and bus voltage features to obtain the power grid features.

[0100] S21: Based on the generator set equipment number, real-time output value, and regulation margin, and the energy storage device equipment number, real-time output value, and remaining capacity, combined with the unique thermal code, the generator set characteristics and energy storage device characteristics are generated. The calculation method is as follows:

[0101]

[0102]

[0103] in, Characteristics of generator sets, For ReLU function, This is the characteristic weight matrix of the generator set. It is a one-hot encoding. Assign a generator set equipment number. This represents the real-time output value of the generator set. For the generator set's regulation margin, As a characteristic of energy storage devices, This is the characteristic weight matrix of the energy storage device. For energy storage device equipment number, This represents the real-time output value of the energy storage device. This represents the remaining capacity of the energy storage device;

[0104] S22: Based on the aforementioned power grid frequency deviation, regional load power, and bus voltage, power grid characteristics are generated using a one-dimensional convolution and gating mechanism. The calculation method is as follows:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] in, This is a characteristic of power grid frequency deviation. For power grid frequency deviation, For one-dimensional convolution, For regional load power characteristics, For regional load power, The characteristics of the bus voltage. Bus voltage For frequency-dominant gating vectors, For the Sigmoid function, For splicing operations, As a characteristic of the power grid, For Hadamah accumulation;

[0111] S23: Based on the characteristics of the generator set, energy storage device, and power grid, and combined with an improved attention mechanism, a reinforcement learning observation state vector is generated, such as... Figure 2 As shown, the calculation method is as follows:

[0112]

[0113]

[0114]

[0115]

[0116] in, For query vector, It is a multilayer perceptron. For key vectors, For value vectors, For attention vectors, Here is the attention weight matrix. For the Softmax function, For transpose operation, For feature dimension, To reinforce the learning of the observed state vector, For layer normalization.

[0117] S3: Design basic rewards based on the state space and action space of the reinforcement learning system;

[0118] S31: Calculate the generator set reward based on the real-time output value of the generator set in the state space, the generator set regulation margin, and the generator set power generation in the action space. The calculation method is as follows:

[0119]

[0120]

[0121] in, This refers to the generator set regulation efficiency factor. To take the absolute value, The power output of the generator set in the action space. To obtain the maximum value, It is a non-zero positive minimum value. As a reward for generator sets, This is the penalty coefficient for the generator set's regulation efficiency.

[0122] S32: Calculate the energy storage device reward based on the remaining capacity of the energy storage device in the state space, the real-time output value of the energy storage device, and the power generation of the energy storage device in the action space. The calculation method is as follows:

[0123]

[0124]

[0125] in, For the health deviation of energy storage devices, Incentives for energy storage devices, This is the penalty coefficient for energy storage devices. To generate power for energy storage devices in the action space. Rated power of the energy storage device;

[0126] S33: Calculate the grid stability bonus based on the grid frequency deviation, regional load power, and bus voltage in the state space. The calculation method is as follows:

[0127]

[0128] in, As a reward for grid stability, This is the power grid stability weight matrix;

[0129] S34: Calculate the basic reward based on generator set rewards, energy storage device rewards, grid stability rewards, and reinforcement learning observation state vectors. The calculation method is as follows:

[0130]

[0131] in, As a basic reward.

[0132] S4: Based on the state space and action space of the reinforcement learning system, design early warning rewards and energy balance factors, and calculate the final reward in combination with the basic rewards; the energy balance factor is generated in the following way: generate dynamic weights of the remaining capacity of the energy storage device based on the remaining capacity of the energy storage device, calculate the difference between the power generation of the generator set and the power generation of the energy storage device, and generate the energy balance factor by combining exponential function calculation;

[0133] S41: Calculate the warning reward based on the warning accuracy in the state space. The calculation method is as follows:

[0134]

[0135]

[0136] in, Let be the exponential moving average of the warning accuracy at time t in the state space, where t is the time index. For smoothing coefficients, Let be the early warning accuracy at time t-1 in the state space. Let be the exponential moving average of the warning accuracy at time t-1 in the state space. As a reward for early warning, As the warning penalty coefficient, It is the hyperbolic tangent function. Let be the early warning accuracy at time t in the state space. To take the exponent of the natural constant;

[0137] S42: Calculate the energy balance factor based on the generator power output and energy storage power output in the action space, and the remaining capacity of the energy storage device in the state space. The calculation method is as follows:

[0138]

[0139]

[0140] in, The remaining capacity of the energy storage device is dynamically weighted. This is the penalty factor for the remaining capacity of the energy storage device. Energy balance factor;

[0141] S43: Calculate the final reward based on the basic reward, early warning reward, and energy balance factor. The calculation method is as follows:

[0142]

[0143]

[0144] in, To reward the gating vector, To reward the gating weight matrix, As the final reward, Based on the basic reward weight matrix, For the early warning reward weight matrix, This is the energy balance weight matrix.

[0145] The multilayer perceptron structure used in this invention for calculating the energy balance factor is as follows: Figure 3As shown, this multilayer perceptron adopts a typical three-layer feedforward neural network architecture, consisting of an input layer, a hidden layer, and an output layer connected sequentially. The input layer consists of 6 red neurons, which receive the dynamic weights of the remaining capacity of the energy storage device. The hidden layer consists of 10 cyan neurons, which realize the nonlinear transformation of the feature space. The ReLU activation function is used to process the input features in the hidden layer. The layers are fully connected and configured with L2 regularization constraints (λ=0.001). The output layer consists of 5 yellow neurons, which map the features output by the hidden layer into scalar reward components.

[0146] For example, when the generator set is running stably, The accuracy rate of early warnings has reached the standard. The energy storage condition is good. ; Calculate the reward gating vector This ensures that positive incentives dominate the reward calculation, resulting in a final reward value. This represents ideal working conditions;

[0147] If the energy storage device discharges deeply, it will lead to ,lead to Even the basic reward at this time ; Calculate the reward gating vector The final reward plummeted to This generates a strong suppression signal, punishing the energy storage device for excessive discharge.

[0148] After multiple iterations, the reinforcement learning system gradually converges, and the convergence process is as follows: Figure 4 As shown: The light blue original reward curve represents the training data containing exploration noise and periodic fluctuations. It rises rapidly in the first 300 rounds (exploration phase), the slope slows down from 300 to 600 rounds (optimization phase), and enters a stable zone after 600 rounds (convergence phase). The green dashed line represents the theoretical convergence benchmark. The changes in reward values ​​at the four red key points (rounds 100, 300, 600, and 900) intuitively reflect the training progress. Overall, it presents the complete dynamic process of reinforcement learning from exploration to stable convergence.

[0149] S5: Using the Double DQN network as the optimization network of the reinforcement learning system, the parameters are updated to obtain the Double DQN network with updated parameters;

[0150] S51: Use the network as the optimization network of the reinforcement learning system, and initialize the main network parameters and target network parameters of the Double DQN network; the Double DQN network includes discrete action branches and continuous action branches;

[0151] S52: Based on the main network parameters and target network parameters of the Double DQN network, select the optimal discrete action through the discrete action branch, calculate the continuous action limit value through the continuous action branch, and then calculate the target Q value and the action at the next moment.

[0152] S53: Based on the target Q value and the action at the next time step, calculate the network gradient using the mean squared error loss function, and use the gradient descent algorithm to update the main network parameters and target network parameters of the Double DQN network to obtain the Double DQN network with updated parameters.

[0153] S6: Calculate and send control commands for generator sets and energy storage devices based on the updated parameters of the Double DQN network;

[0154] S61: Based on the Double DQN network with updated parameters, the generator set operation mode command and energy storage device operation mode command are decoded and parsed through discrete actions. The real-time output value command of the generator set and the real-time output value command of the energy storage device are calculated through continuous action conversion to obtain the set of equipment control commands.

[0155] S62: Perform power limiting processing on the set of equipment control commands to obtain a set of equipment execution commands after power limiting processing;

[0156] S63: Based on the set of equipment execution instructions processed by power limiting, the system sends operation mode instructions and real-time output value instructions to generator sets and energy storage devices through the power monitoring system, and obtains equipment execution status feedback and real-time grid operation parameters.

[0157] Example 2: This invention also discloses an intelligent control system for energy management based on a reinforcement learning algorithm, comprising:

[0158] Data acquisition module: Collects data from generator sets, energy storage devices, and the power grid, and defines the state space and action space of the reinforcement learning system;

[0159] Reinforcement learning observation state vector extraction module: Based on the data of generator sets, energy storage devices, and power grids, extract the features of generator sets, energy storage devices, and power grids, and calculate the reinforcement learning observation state vector;

[0160] Basic Reward Module: Design basic rewards based on the state space and action space of the reinforcement learning system;

[0161] Final reward module: Based on the state space and action space of the reinforcement learning system, design early warning rewards and energy balance factors, and calculate the final reward by combining the basic rewards;

[0162] Optimize network parameter update module: Use the Double DQN network as the optimization network of the reinforcement learning system, update the parameters, and obtain the Double DQN network with updated parameters;

[0163] Control command sending module: Calculates and sends control commands for generator sets and energy storage devices based on the updated parameters of the Double DQN network.

[0164] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0166] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An intelligent control method for energy management based on reinforcement learning algorithms, characterized in that, Includes the following steps: S1: Collect data from generator sets, energy storage devices, and the power grid, and define the state space and action space of the reinforcement learning system; S2: Based on the data from the generator set, energy storage device, and power grid, extract the features of the generator set, energy storage device, and power grid, and calculate the reinforcement learning observation state vector. The reinforcement learning observation state vector is calculated based on the generator set features, energy storage device features, and power grid features, combined with an improved attention mechanism. The power grid features are generated as follows: based on the power grid data, extract the power grid frequency deviation features, regional load power features, and bus voltage features through one-dimensional convolution. Combine this with the Sigmoid function to generate a frequency-dominant gating vector, and weight the power grid frequency deviation features, regional load power features, and bus voltage features to obtain the power grid features. S3: Design basic rewards based on the state space and action space of the reinforcement learning system; S4: Based on the state space and action space of the reinforcement learning system, design early warning rewards and energy balance factors, and calculate the final reward in combination with the basic rewards; the energy balance factor is generated in the following way: generate dynamic weights of the remaining capacity of the energy storage device based on the remaining capacity of the energy storage device, calculate the difference between the power generation of the generator set and the power generation of the energy storage device, and generate the energy balance factor by combining exponential function calculation; S5: Using the Double DQN network as the optimization network of the reinforcement learning system, the parameters are updated to obtain the Double DQN network with updated parameters; S6: Calculate and send control commands for generator sets and energy storage devices based on the updated parameters of the Double DQN network.

2. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 1, characterized in that, Step S1 includes: S11: Collect generator set data, including generator set equipment number, real-time output value, and maximum output value; collect energy storage device data, including energy storage device equipment number, real-time output value, and remaining capacity; collect grid data, including grid frequency deviation, regional load power, and bus voltage. S12: Calculate the difference between the real-time output value and the maximum output value of the generator set to obtain the generator set regulation margin; S13: Define the state space of the reinforcement learning system, including: generator set equipment number, generator set real-time output value, generator set regulation margin, energy storage device equipment number, energy storage device real-time output value, energy storage device remaining capacity, as well as grid frequency deviation, regional load power value, bus voltage, and early warning accuracy; the early warning accuracy is based on the output label and the true label of the reinforcement learning system, and is obtained by statistically analyzing the proportion of correct samples in the output label through a fixed-length sliding window; S14: Define the action space of the reinforcement learning system, including: generator power output and energy storage device power output.

3. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 1, characterized in that, In step S2, the improved attention mechanism calculation method includes: The generator set features and energy storage device features are concatenated to generate a query vector using a multilayer perceptron. The grid features are used as both the key vector and the value vector. The product of the query vector and the transpose of the key vector is calculated, then divided by the square root of the feature dimension. A query vector correction term, processed by linear transformation of the attention weight matrix and ReLU activation, is then added. Finally, an attention vector is generated using the Softmax function. The attention vector is multiplied by the value vector and then added to the query vector after layer normalization to generate a reinforcement learning observation state vector.

4. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 3, characterized in that, Step S2 includes: S21: Based on the generator set equipment number, real-time output value, and regulation margin, and the energy storage device equipment number, real-time output value, and remaining capacity, combined with the unique thermal code, the generator set characteristics and energy storage device characteristics are generated. The calculation method is as follows: in, Characteristics of generator sets, For ReLU function, This is the characteristic weight matrix of the generator set. It is a one-hot encoding. Assign a generator set equipment number. This represents the real-time output value of the generator set. For the generator set's regulation margin, Characteristics of energy storage devices, This is the characteristic weight matrix of the energy storage device. For energy storage device equipment number, This represents the real-time output value of the energy storage device. This represents the remaining capacity of the energy storage device; S22: Based on the aforementioned power grid frequency deviation, regional load power, and bus voltage, power grid characteristics are generated using a one-dimensional convolution and gating mechanism. The calculation method is as follows: in, This is a characteristic of power grid frequency deviation. For power grid frequency deviation, For one-dimensional convolution, For regional load power characteristics, For regional load power, The characteristics of the bus voltage. Bus voltage For frequency-dominant gating vectors, For the Sigmoid function, For splicing operations, As a characteristic of the power grid, For Hadamah accumulation; S23: Based on the characteristics of the generator set, energy storage device, and power grid, a reinforcement learning observation state vector is generated using an improved attention mechanism. The calculation method is as follows: in, For query vector, It is a multilayer perceptron. For key vectors, For value vectors, For attention vectors, Here is the attention weight matrix. For the Softmax function, For transpose operation, For feature dimension, To reinforce the learning of the observed state vector, For layer normalization.

5. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 2, characterized in that, Step S3 includes: S31: Calculate the generator set reward based on the real-time output value of the generator set in the state space, the generator set regulation margin, and the generator set power generation in the action space. The calculation method is as follows: in, This refers to the generator set regulation efficiency factor. To take the absolute value, The power output of the generator set in the action space. To obtain the maximum value, It is a non-zero positive minimum value. As a reward for generator sets, This is the penalty coefficient for the generator set's regulation efficiency. S32: Calculate the energy storage device reward based on the remaining capacity of the energy storage device in the state space, the real-time output value of the energy storage device, and the power generation of the energy storage device in the action space. The calculation method is as follows: in, For the health deviation of energy storage devices, Incentives for energy storage devices, This is the penalty coefficient for energy storage devices. To generate power for energy storage devices in the action space. Rated power of the energy storage device; S33: Calculate the grid stability bonus based on the grid frequency deviation, regional load power, and bus voltage in the state space. The calculation method is as follows: in, As a reward for grid stability, This is the power grid stability weight matrix; S34: Calculate the basic reward based on generator set rewards, energy storage device rewards, grid stability rewards, and reinforcement learning observation state vectors. The calculation method is as follows: in, As a basic reward.

6. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 5, characterized in that, Step S4 includes: The basic reward, early warning reward, and energy balance factor are concatenated, and a reward gating vector is generated using the Sigmoid function. The linear combination result of the basic reward and early warning reward is weighted using the reward gating vector, and the linear transformation result of the energy balance factor is weighted to generate the final reward.

7. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 6, characterized in that, Step S4 includes: S41: Calculate the warning reward based on the warning accuracy in the state space. The calculation method is as follows: in, Let be the exponential moving average of the warning accuracy at time t in the state space, where t is the time index. For smoothing coefficients, Let be the early warning accuracy at time t-1 in the state space. Let be the exponential moving average of the warning accuracy at time t-1 in the state space. As a reward for early warning, As the warning penalty coefficient, It is the hyperbolic tangent function. Let be the early warning accuracy at time t in the state space. To take the exponent of the natural constant; S42: Calculate the energy balance factor based on the generator power output and energy storage power output in the action space, and the remaining capacity of the energy storage device in the state space. The calculation method is as follows: in, The remaining capacity of the energy storage device is dynamically weighted. This is the penalty factor for the remaining capacity of the energy storage device. Energy balance factor; S43: Calculate the final reward based on the basic reward, early warning reward, and energy balance factor. The calculation method is as follows: in, To reward the gating vector, To reward the gating weight matrix, As the final reward, Based on the basic reward weight matrix, For the early warning reward weight matrix, This is the energy balance weight matrix.

8. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 7, characterized in that, Step S5 includes: S51: Use the Double DQN network as the optimization network of the reinforcement learning system, and initialize the main network parameters and target network parameters of the Double DQN network; the Double DQN network includes discrete action branches and continuous action branches; S52: Based on the main network parameters and target network parameters of the Double DQN network, select the optimal discrete action through the discrete action branch, calculate the continuous action limit value through the continuous action branch, and then calculate the target Q value and the action at the next moment. S53: Based on the target Q value and the action at the next time step, calculate the network gradient using the mean squared error loss function, and use the gradient descent algorithm to update the main network parameters and target network parameters of the Double DQN network to obtain the Double DQN network with updated parameters.

9. The intelligent control method for energy management based on reinforcement learning algorithm according to claim 8, characterized in that, Step S6 includes: S61: Based on the Double DQN network with updated parameters, the generator set operation mode command and energy storage device operation mode command are decoded and parsed through discrete actions. The real-time output value command of the generator set and the real-time output value command of the energy storage device are calculated through continuous action conversion to obtain the set of equipment control commands. S62: Perform power limiting processing on the set of equipment control commands to obtain a set of equipment execution commands after power limiting processing; S63: Based on the set of equipment execution instructions processed by power limiting, the system sends operation mode instructions and real-time output value instructions to generator sets and energy storage devices through the power monitoring system, and obtains equipment execution status feedback and real-time grid operation parameters.

10. An intelligent control system for energy management based on reinforcement learning algorithms, characterized in that, include: Data acquisition module: Collects data from generator sets, energy storage devices, and the power grid, and defines the state space and action space of the reinforcement learning system; Reinforcement learning observation state vector extraction module: Based on the data of generator sets, energy storage devices, and power grids, extract the features of generator sets, energy storage devices, and power grids, and calculate the reinforcement learning observation state vector; Basic Reward Module: Design basic rewards based on the state space and action space of the reinforcement learning system; Final reward module: Based on the state space and action space of the reinforcement learning system, design early warning rewards and energy balance factors, and calculate the final reward by combining the basic rewards; Optimize network parameter update module: Use the Double DQN network as the optimization network of the reinforcement learning system, update the parameters, and obtain the Double DQN network with updated parameters; Control command sending module: Calculates and sends control commands to generator sets and energy storage devices based on the updated parameters of the Double DQN network; To achieve the intelligent control method for energy management based on reinforcement learning algorithm as described in any one of claims 1-9.

Images