A building multi-source system energy distribution method and system based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's building multi-source system energy distribution method and system, the method includes: obtaining building load and photovoltaic power generation prediction data;Building building multi-source system energy scheduling target function and physical constraint condition;State space, action space and reward function in Markov decision process model are constructed;Energy collaborative distribution model comprising Actor network structure and Critic network structure is constructed;Energy collaborative distribution model is trained, and the trained energy collaborative distribution model is obtained, and based on the trained energy collaborative distribution model output building multi-source system energy distribution strategy corresponding to the period to be scheduled.The application can improve the energy distribution prediction accuracy of building load and photovoltaic power generation by pre-sensing the dynamic evolution of environment.The application is a kind of based on reinforcement learning's building multi-source system energy distribution method and system, and can be widely applied in energy distribution technical field.

Description

Technical Field

[0001] This invention relates to the field of energy distribution technology, and in particular to an energy distribution method and system for multi-source building systems based on reinforcement learning. Background Technology

[0002] With the deepening application of artificial intelligence technology in the field of smart grids, the importance of energy collaborative allocation for multi-source building systems is becoming increasingly prominent. This task aims to coordinate the energy flow of building loads, distributed photovoltaics, energy storage devices, and electric vehicles (EVs) in real time from complex energy supply and demand networks, and is widely used in scenarios such as zero-carbon building construction, virtual power plant dispatching, and demand-side response of new power systems. Current mainstream methods typically adopt rule-based maximization strategies for photovoltaic energy management, or mathematical programming to model the interaction relationships of multi-source heterogeneous energy sources, achieving a balance between economic and environmental benefits by constructing a unified optimization objective. However, due to the high volatility of distributed energy sources and the strong randomness of the spatiotemporal distribution of mobile energy storage, effectively modeling energy coupling across multiple time scales and strict physical constraints under dynamic electricity pricing environments has always been a technical challenge.

[0003] To address the aforementioned issues, various improved solutions based on reinforcement learning have been proposed. Methods such as Deep Deterministic Policy Gradient (DDPG) parameterize policy functions through neural networks, supporting millisecond-level adaptive online decision-making. This method demonstrates strong model-free learning capabilities in general microgrid scheduling tasks, but its design has two major limitations: 1) It does not consider the strong physical repulsion constraints in multi-source cooperative systems, and direct application often relies on underlying hard logic truncation, leading to gradient breakage and policy oscillations during backpropagation of the neural network; 2) The state-aware space suffers from a "short-sighted" defect, failing to introduce a future temporal context awareness mechanism, making it difficult to adaptively capture long-term arbitrage opportunities in time-of-use pricing. Furthermore, while some reinforcement learning methods can ensure operational safety, their rigid grid compensation rules for EV off-grid power lead to severe dependence on the agent, resulting in poor performance in proactively planning off-peak charging tasks.

[0004] In contrast, deep reinforcement learning techniques that integrate continuous action spaces and dynamic constraints have shown potential in handling multi-scale energy dispatch. Techniques such as Twin Delayed Deep Deterministic Policy Gradient (TD3) generate smooth action sequences through an Actor-Critic architecture, mitigating overestimation and enhancing policy robustness. However, existing methods have not yet addressed key issues in heterogeneous building microgrid scenarios: the lack of a conservation mapping mechanism for net power redistribution in action outputs leads to a mismatch between energy flow logic and the algorithm's exploration space; the absence of sliding window encoding in state observations limits accurate responses to future price trends and capacity; and the lack of dynamic penalty guidance mechanisms for rigid constraints such as EV off-grid SOC. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention aims to provide a method and system for energy allocation in a multi-source building system based on reinforcement learning, which can improve the accuracy of energy allocation prediction for building load and photovoltaic power generation by pre-sensing the dynamic evolution of the environment.

[0006] The first technical solution adopted in this invention is: an energy allocation method for a multi-source building system based on reinforcement learning, comprising the following steps: Obtain building load and photovoltaic capacity forecast data output from the distributed source-load joint forecasting model of the new power load management system; Construct an objective function and physical constraints for energy scheduling of a building's multi-source system, wherein the objective function is used to indicate maximizing the system's overall economic and environmental benefits; Based on building load and photovoltaic capacity forecast data, objective function and multiple physical constraints, a state space, action space and reward function are constructed in the Markov decision process model. The state space includes the load forecast sequence, photovoltaic forecast sequence, time-of-use electricity price sequence and energy storage device power status for multiple future time steps. The action space is used to indicate the photovoltaic allocation ratio and the charging and discharging intensity of each energy storage device. The reward function is used to indicate the comprehensive benefit of the current scheduling action. Construct an energy collaborative allocation model that includes Actor network structure and Critic network structure; An energy collaborative allocation model is trained using building load and photovoltaic capacity forecast data, state space, action space, and reward function. The trained energy collaborative allocation model is then used to output the energy allocation strategy for the building multi-source system corresponding to the scheduling period.

[0007] Furthermore, the step of obtaining the building load and photovoltaic capacity forecast data output by the distributed source-load joint forecasting model of the new power load management system specifically includes: The input historical meteorological parameters, photovoltaic historical data and building load historical data are subjected to outlier removal, missing value filling, feature engineering and normalization to obtain preprocessed historical meteorological parameters, photovoltaic historical data and building load historical data. A source-load joint prediction model was built, which includes a multi-scale feature extraction module, an adaptive spatial attention mechanism and a Transformer encoder. The model was trained by K-fold cross-validation and the optimal model parameters were selected to obtain the trained source-load joint prediction model. A sliding window autoregressive rolling update mechanism is adopted to predict preprocessed historical meteorological parameters, photovoltaic historical data and building load historical data based on a trained source-load joint prediction model, and output building load and photovoltaic capacity prediction data.

[0008] Furthermore, the physical constraints include energy conservation constraints for multi-source systems, upper and lower limits of energy storage system capacity, charging and discharging power constraints for energy storage devices, minimum energy guarantee constraints for mobile trolleys leaving the grid, and rigid satisfaction constraints for building loads. The charging and discharging power constraints are used to dynamically adjust the charging and discharging rate based on the current energy storage capacity of the energy storage devices, and the minimum energy guarantee constraints for mobile trolleys leaving the grid are used to forcibly replenish the minimum energy requirement before the trolley leaves the grid.

[0009] Furthermore, the step of training the energy collaborative allocation model specifically includes: Define the current state vector based on building load and photovoltaic capacity forecast data; The current state vector is input into the Actor network structure to obtain the continuous action vector output by the Actor network structure; The continuous action vector is mapped to the actual charging and discharging power of each device and the photovoltaic power allocated. After the scheduling action is executed, an immediate reward is obtained and the state is updated for the next moment. Input the current state vector and continuous action vector into the Critic network structure to obtain the state-action value output by the Critic network structure; Input the next time step state vector into the Actor network structure to obtain the next time step continuous action vector. Then input the next time step state vector and the next time step continuous action vector into the Critic network structure to obtain the next time step state-action value. Calculate the time difference error based on the immediate reward, the value of the action at the current moment, and the value of the state action at the next moment; The network parameters of the Actor network structure and the Critic network structure are updated alternately according to the time difference error until the model converges or reaches the preset number of iterations, thus obtaining the trained energy collaborative allocation model.

[0010] Furthermore, the Actor network structure includes an input preprocessing layer, multiple feature extraction layers, and an action output layer. The step of inputting the current state vector into the Actor network structure to obtain the continuous action vector output by the Actor network structure specifically includes: The current state vector is standardized by input preprocessing layer, which maps the features of each dimension to the same distribution space to obtain the standardized state vector. The standardized state vector is subjected to feature extraction through a multi-layer feature extraction layer to obtain a standardized state feature vector. The multi-layer feature extraction layer includes a multi-layer fully connected layer, a layer normalization module, and a ReLU activation function. Action prediction is performed on the standardized state feature vector through the action output layer, and a 6-dimensional continuous action vector is output. When using the TD3 architecture, a deterministic action vector is output and clipped Gaussian noise is added to enhance the exploratory nature. When using the SAC architecture, the Gaussian distribution parameters of the action are output. The continuous action vector is obtained by sampling through reparameterization techniques.

[0011] Furthermore, the Critic network structure adopts a dual-Q network architecture, containing two parameter-independent value evaluation networks. The step of inputting the current state vector and continuous action vector into the Critic network structure to obtain the state-action value output by the Critic network structure specifically includes: The state vector and the continuous action vector are concatenated along the feature dimension to obtain the concatenated feature vector; The concatenated feature vectors are input into two value evaluation networks respectively to obtain two independent state-action evaluation values; The evaluation value that is less than a preset threshold among the two state-action evaluation values ​​is selected as the state-action value output by the Critic network structure.

[0012] Furthermore, the energy collaborative allocation model is trained using an experience replay mechanism, which specifically includes: Set a preset capacity experience replay pool. During the warm-up phase, fill the replay pool with random actions. After the warm-up is over, randomly select a small batch of samples from the replay pool for training each time the parameters are updated. Both the Actor network and the Critic network use the Adam optimizer for gradient updates, with the Critic network introducing L2 regularization to prevent overfitting. Configure core hyperparameters such as reward discount factor, target network soft update coefficient, policy update frequency, and entropy temperature coefficient to balance short-term and long-term returns.

[0013] Furthermore, the step of alternately updating the network parameters of the Actor network structure and the Critic network structure based on the time difference error specifically includes: The loss function of the Critic network is constructed based on the mean square error to minimize the deviation between the current evaluation value and the time difference target value. The parameters of the Critic network are updated by gradient descent. In the TD3 architecture, the time difference target introduces the target policy to smooth noise, while the time difference target in the SAC architecture introduces an entropy reward term. The Actor network loss function is constructed based on policy gradient to maximize the state-action value output by the Critic network. The Actor network parameters are updated through gradient ascent. In the SAC architecture, the Actor network loss includes two optimization objectives: expected reward and action distribution entropy.

[0014] Furthermore, the step of outputting the building multi-source system energy allocation strategy corresponding to the scheduling period based on the trained energy collaborative allocation model specifically includes: Based on the trained energy collaborative allocation model, the trained Actor network weights are loaded and set to deterministic inference mode; The current power status of the energy storage device is collected according to the preset sampling period, and the source load prediction sequence and electricity price sequence for multiple future time steps are retrieved simultaneously to construct a real-time state vector; The real-time state vector is input into the Actor network, which outputs a continuous action vector and maps it to actual device scheduling instructions. The dispatching command undergoes a second physical constraint review. If the rigid safety constraint is not met, the power grid compensation mechanism is triggered to correct the command before it is issued for execution. Finally, the energy allocation strategy for the building multi-source system corresponding to the dispatching period is output.

[0015] The second technical solution adopted in this invention is: an energy distribution system for a multi-source building system based on reinforcement learning, comprising: The first module is used to obtain building load and photovoltaic capacity prediction data output by the distributed source-load joint prediction model of the new power load management system. The second module is used to construct the objective function and physical constraints for energy scheduling of a building's multi-source system, wherein the objective function is used to indicate maximizing the system's comprehensive economic and environmental benefits; The third module is used to construct the state space, action space, and reward function in the Markov decision process model based on building load and photovoltaic capacity forecast data, objective function, and multiple physical constraints. The state space includes the load forecast sequence, photovoltaic forecast sequence, time-of-use electricity price sequence, and energy storage device power status for multiple future time steps. The action space is used to indicate the photovoltaic allocation ratio and the charging and discharging intensity of each energy storage device. The reward function is used to indicate the overall benefit of the current scheduling action. The fourth module is used to construct an energy collaborative allocation model that includes both Actor network and Critic network structures; The fifth module is used to train the energy collaborative allocation model using building load and photovoltaic capacity prediction data, state space, action space and reward function, to obtain the trained energy collaborative allocation model, and to output the building multi-source system energy allocation strategy corresponding to the scheduling period based on the trained energy collaborative allocation model.

[0016] The beneficial effects of the method and system of this invention are as follows: This invention obtains building load and photovoltaic capacity prediction data output by the distributed source-load joint prediction model of a novel power load management system; then, it constructs the objective function and physical constraints for energy scheduling of multi-source building systems, and designs a smooth and continuous action space and mapping transformation rules based on net power settlement. This mechanism ensures that the execution instructions output by the agent are always strictly within safe physical boundaries, which not only greatly reduces the invalid exploration space of the agent, but also effectively protects the continuity of gradients during backpropagation of the neural network, significantly improving the convergence speed and policy stability of the Actor-Critic algorithm in complex multi-source systems; based on the building load and photovoltaic capacity prediction data, the objective function, and multiple physical constraints, it constructs the state space, action space, and reward function in the Markov decision process model. By constructing a high-dimensional composite reinforcement learning reward function, it can adaptively seek the global optimum under multiple objectives within an extremely short online inference time of milliseconds. The optimal solution is then proposed. Further, an energy collaborative allocation model incorporating Actor and Critic network structures is constructed. Finally, the energy collaborative allocation model is trained using building load and photovoltaic capacity prediction data, state space, action space, and reward function. This results in a well-trained energy collaborative allocation model, which outputs the building's multi-source system energy allocation strategy for the scheduling period. This completes the closed loop from multimodal meteorological / historical data processing and high-precision source-load joint prediction to constructing a Markov decision process environment, algorithm training, and finally integration into the building energy management system for real-time inference and rigid secondary verification. This system not only responds to changes in energy supply and demand within the system in real time but also achieves self-iteration and evolution of strategies through continuous environmental interaction. It is widely applicable to complex and ever-changing distributed energy collaborative management scenarios such as smart buildings, zero-carbon parks, and virtual power plants. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of an energy allocation method for a multi-source building system based on reinforcement learning, as described in this invention. Figure 2 This is a structural block diagram of a building multi-source system energy distribution system based on reinforcement learning according to the present invention; Figure 3 This is a flowchart illustrating the reinforcement learning model provided in a specific embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0019] First, it should be noted that existing deep reinforcement learning-based building microgrid and multi-source system energy dispatch technologies face the following major technical bottlenecks: Current mainstream reinforcement learning action space design often neglects the strong physical exclusion constraints of energy storage devices. When executing mutually exclusive actions, it heavily relies on the hard logic truncation of the underlying environment, severely disrupting the gradient continuity of the neural network's backpropagation, making the model difficult to converge effectively and prone to policy oscillations. Simultaneously, although existing models can respond to the current source-load state in real time, their predictive ability is insufficient when modeling long-term time-varying environments, making it difficult to accurately capture future price and capacity evolution trends across multiple time dimensions, thus limiting intelligent... The system lacks forward-looking decision-making accuracy in cross-timescale arbitrage and context-dependent relationships such as "storing energy during off-peak hours and discharging during peak hours." In addition, to meet the rigid constraint of minimum power consumption when mobile energy storage (such as electric vehicles) is randomly disconnected from the grid, existing systems usually introduce a protection mechanism that directly purchases electricity from the grid. This mechanism is often accompanied by a rigid grid compensation trap for reinforcement learning agents. Especially in high-frequency vehicle access and departure scenarios, the agent is prone to evolving a passive suboptimal strategy of refusing to actively charge. On the other hand, simply pursuing static penalties for absolute safety may sacrifice the model's ability to explore low-cost charging opportunities. Existing methods are difficult to provide sufficient active planning capabilities while ensuring the bottom line of physical constraints.

[0020] Based on this, this invention proposes a reinforcement learning-based method for collaborative energy allocation in building multi-source systems. By integrating a smooth, continuous action space based on net power calculation, introducing a time-series awareness mechanism that integrates future prediction time windows, and coupling a dynamic penalty guidance strategy for off-grid power gaps, this method aims to significantly improve the continuity of reinforcement learning action gradients, the accuracy of cross-cycle time-series decision-making, and the model's forward-looking planning capability under rigid constraints. This results in higher economic efficiency and operational robustness in the collaborative scheduling of building photovoltaic, energy storage, and charging multi-source systems.

[0021] This invention employs a novel distributed source-load joint prediction model for a power load management system to obtain 7-day building load and photovoltaic capacity, dividing them into specific time intervals. The prediction data is then divided into training, validation, and test sets to construct a reinforcement learning-based method and system for coordinated energy allocation in multi-source building systems. The training dataset is input, and a reinforcement learning agent is trained using the constructed algorithm and environment. The model performance is then evaluated. Finally, the trained agent, designed for a specific task, is used to predict energy allocation for the new 7-day building load and photovoltaic capacity.

[0022] Reference Figure 1 This invention provides a reinforcement learning-based method for energy allocation in a multi-source building system, comprising the following steps: S100. Obtain building load and photovoltaic capacity prediction data output by the distributed source-load joint prediction model of the new power load management system. S110. Perform outlier removal, missing value filling, feature engineering, and normalization on the input historical meteorological parameters, photovoltaic historical data, and building load historical data to obtain the preprocessed historical meteorological parameters, photovoltaic historical data, and building load historical data. In this embodiment, the preprocessing process specifically includes: 1) Input data acquisition: Integrate meteorological parameters (temperature, humidity, wind speed, solar radiation) from 8760 hours of typical meteorological year and historical photovoltaic and load data simulated by EnergyPlus software; 2) Data cleaning: Outlier removal and missing value imputation are performed on the original time series data; 3) Feature engineering: making implicit raw data explicit and mining the correlation between meteorological parameters and photovoltaic and load patterns; 4) Data normalization: Standardization is used to make meteorological characteristics conform to a normal distribution, and normalization is used to limit the target value to the range of 0 to 1; 5) Dataset partitioning: The processed data is divided into training and test sets in a 9:1 ratio for evaluating the model's generalization ability.

[0023] S120. Build a source-load joint prediction model that includes a multi-scale feature extraction module, an adaptive spatial attention mechanism and a Transformer encoder. Train the model using K-fold cross-validation and select the optimal model parameters to obtain the trained source-load joint prediction model. In this embodiment, a serial fusion strategy is used to build the prediction model, and the core modules include: 1) Multi-scale feature extraction: By gradually expanding the historical time observation window, short-term sudden fluctuations, medium-term regular disturbances and long-term periodic evolutions are captured respectively, and the evolution law of core source load is purified; 2) Adaptive spatial attention mechanism: Assign adaptive weights to each meteorological parameter to automatically strengthen the dominant parameter and suppress interference from irrelevant variables; 3) Transformer encoder: Establishes the nonlinear coupling relationship between meteorological parameters and load and photovoltaics, and learns the differences in the contribution of each parameter to the prediction target under different scenarios; 4) K-fold cross-validation: Use 90% of the data for multiple rounds of training and validation to select the optimal model parameters, and complete the final performance evaluation on the remaining 10% of the test set.

[0024] S130. A sliding window autoregressive rolling update mechanism is adopted to predict the preprocessed historical meteorological parameters, photovoltaic historical data and building load historical data based on the trained source-load joint prediction model, and output the predicted data of building load and photovoltaic capacity.

[0025] In this embodiment, a sliding window autoregressive rolling update mechanism is adopted, and the lookback window is set to the last 120 hours out of 8760 hours, which specifically includes: 1) Single-step forecast: Based on the current review window data, forecast meteorological parameters for the next hour, output the single-step normalized forecast value, and then reverse normalize it to restore the physical value; 2) Data backfilling: The current predicted physical values ​​are used as new "historical true values" to backfill the review window and update it; 3) Cyclic forecasting: Repeat the single-step forecasting and data backfilling process 168 times to generate complete forecast data for meteorological parameters, building load and photovoltaic capacity for the next week; 4) Data set splitting: The three sets of generated prediction data are divided into training set, validation set and test set for subsequent microgrid optimization scheduling.

[0026] S200. Construct the objective function and physical constraints for energy scheduling of a building's multi-source system, wherein the objective function is used to indicate maximizing the system's comprehensive economic and environmental benefits; The physical constraints include energy conservation constraints for multi-source systems, upper and lower limits of energy storage system capacity, charging and discharging power constraints for energy storage devices, minimum energy guarantee constraints for mobile trolleys leaving the grid, and rigid satisfaction constraints for building loads. The charging and discharging power constraints are used to dynamically adjust the charging and discharging rate based on the current energy capacity of the energy storage devices, and the minimum energy guarantee constraints for mobile trolleys leaving the grid are used to forcibly replenish the minimum energy requirement before the trolley leaves the grid.

[0027] S300. Based on the building load and photovoltaic capacity forecast data, objective function and multiple physical constraints, construct the state space, action space and reward function in the Markov decision process model. The state space includes the load forecast sequence, photovoltaic forecast sequence, time-of-use electricity price sequence and energy storage device power status for multiple future time steps. The action space is used to indicate the photovoltaic allocation ratio and the charging and discharging intensity of each energy storage device. The reward function is used to indicate the comprehensive benefit of the current scheduling action. In this embodiment, based on the distributed source-load joint prediction model of the new power load management system, this step aims to map the physical scheduling process of building multi-source systems (covering fixed energy storage, photovoltaics, building loads and electric vehicle groups) into a mathematical model that can be optimized by artificial intelligence. It also introduces a reinforcement learning environment module that conforms to the energy collaborative allocation of real building multi-source systems, and uses multiple reinforcement learning model training modules to train and deploy the prediction data multiple times.

[0028] More specifically, a Markov decision process environment model for reinforcement learning is constructed, and the system scheduling process is defined as a Markov decision process, which specifically includes the state space, action space, reward function, and physical constraints.

[0029] 1) Construction and feature reconstruction of the state space, state space Including the environment Comprehensive, real-time information perception. Specifically, this includes: The Future of Normalization Load forecast sequence at time steps , dimension , ; The Future of Normalization Photovoltaic prediction sequence at time steps , dimension , ; The Future of Normalization Time-of-use electricity price sequence at each time step , dimension , ; Normalized relative capacity of stationary energy storage batteries Dimension 1 ; Normalized relative electric power of four groups of electric vehicles The dimension is 4. .

[0030] in It is the building load at the current moment. This represents the total photovoltaic production capacity at the current moment. The electricity price at the current moment. , , These represent the current fixed battery absolute charge, the trolley absolute charge, and the battery maximum charge, respectively.

[0031] 2) The mapping between the motion space setting and the actual execution of actions; motion space This refers to the action decisions that an agent can make at time t, which are set as 6-dimensional continuous variables, specifically including the percentage of photovoltaic output to the building. The dimension is 1, the action value range is 0.0 to 1.0, and the charging and discharging intensity of the fixed battery is... The dimension is 1, the action value range is -1.0 to 1.0, and the charging and discharging intensity of the mobile trolley unit. The dimension is 4, and the action value range is from -1.0 to 1.0; The first dimension represents the proportion of photovoltaic output to photovoltaic output, the second dimension controls the charging and discharging intention of the stationary battery, and the third to sixth dimensions control the charging and discharging intentions of the four EV fleets, respectively. All charging and discharging intentions are set to positive values ​​for charging and negative values ​​for discharging.

[0032] Mapping agent actions to specific execution actions. Transforming abstract execution decisions into concrete energy allocation decisions, specifically including: Photovoltaic power allocated to building load: Photovoltaic power to charge stationary batteries: Solar power charging power for mobile trolley units: The discharge power of a fixed battery to a building: The discharge power of the mobile tram unit to the building: Power allocated by the power grid to building loads: The charging power of the power grid to the stationary battery: The charging power allocated by the power grid to the mobile tram group: in, and To fix the maximum charging and discharging power of the battery, and Let represent the maximum charging power and discharging power of the i-th group of trams.

[0033] 3) Composite design of reward functions, reward function Defined as maximizing the overall economic and environmental benefits of the system, the formula is: in To normalize electricity costs in real time, Incentives for photovoltaic utilization rate To meet building load factor incentives for photovoltaic systems, This is a penalty item for violations.

[0034] The specific implementation is as follows: Real-time normalized electricity cost penalty: for The total net power of the time-of-use system interacting with the public power grid, for Real-time time-of-use electricity price at any given moment This represents the average building load over the system's prediction period. This represents the highest peak electricity price in the electricity price series.

[0035] Solar PV utilization rate incentives: Solar photovoltaic systems meet building load factor incentives: Violations include penalties for excessive discharge of mobile trams when leaving buildings, and penalties for excessive discharge of fixed batteries and mobile trams.

[0036] 4) Physical constraints: To meet the complex power supply logic of a building's multi-source system, there are rigid constraints on the charging and discharging power of fixed batteries and mobile trams, as well as the minimum charge level of mobile energy storage when it is randomly disconnected from the grid. Physical constraints are required for the system. These include: The conservation relationship of energy flow between multi-source systems.

[0037] The relationship between the electrical quantity changes in the energy storage system.

[0038] Maximum and minimum power limits for energy storage systems.

[0039] The energy storage system has charging and discharging power limitations. When the energy level is less than 0.8 but greater than 0.3, it should be charged and discharged quickly; otherwise, it should be charged and discharged slowly.

[0040] The system ensures a minimum charge level when the mobile trolley leaves. If the charge level is less than 0.21 when the trolley leaves at the next moment, the trolley's discharge action will be canceled, and the power grid will charge it to the minimum charge level required.

[0041] Tram loss: When a trolley leaves a building and returns to the building, the trolley's power loss will be calculated, with a value of 0.06.

[0042] The system needs to meet the building load every hour. When the photovoltaic and energy storage systems cannot meet the building load, the grid needs to purchase electricity to meet the building load.

[0043] S400, Construct an energy collaborative allocation model that includes Actor network structure and Critic network structure; The Actor network structure includes an input preprocessing layer, multiple feature extraction layers, and an action output layer. The input preprocessing layer standardizes the current state vector, mapping features across all dimensions to the same distribution space to obtain a standardized state vector. The multiple feature extraction layers extract features from the standardized state vector, resulting in a standardized state feature vector. These multiple feature extraction layers include multiple fully connected layers, a layer normalization module, and a ReLU activation function. The action output layer predicts actions from the standardized state feature vector, outputting a 6-dimensional continuous action vector. When using the TD3 architecture, a deterministic action vector is output with cropped Gaussian noise to enhance exploratory behavior. When using the SAC architecture, the Gaussian distribution parameters of the action are output, and a continuous action vector is obtained through reparameterization techniques.

[0044] The Critic network structure adopts a dual-Q network architecture, which includes two parameter-independent value evaluation networks. By concatenating the state vector and the continuous action vector along the feature dimension, a concatenated feature vector is obtained. The concatenated feature vector is then input into the two value evaluation networks to obtain two independent state-action evaluation values. The evaluation value that is less than a preset threshold is selected as the state-action value output by the Critic network structure.

[0045] In this embodiment, a multi-algorithm integrated network architecture based on Actor-Critic is constructed. The core algorithm adopts the Actor-Critic (AC) architecture. To adapt to the complexity of the scheduling of multi-source building systems, this system designs and supports two advanced continuous control reinforcement learning algorithm kernels in parallel: dual-delay deep deterministic policy gradient (TD3) and flexible actor-critic (SAC).

[0046] For Actor (actor) web design: It consists of a multi-layer sensing mechanism, with hidden layers introducing layer normalization. Its receiving state... As input. Depending on the selected algorithm kernel, the specific modules include: 1) Input preprocessing layer: The receiving dimension is heterogeneous feature vectors The original state vector is standardized to map the features of each dimension to the same distribution space, eliminating the influence of different units such as load, photovoltaic power, and absolute electricity on the network learning and accelerating the gradient descent process.

[0047] 2) Feature Extraction Layer: This layer consists of three fully connected hidden layers, each with 256 neurons. Each hidden layer is followed by a layer normalization function and a ReLU activation function. The layer normalization function normalizes the outputs of all neurons for each sample, effectively mitigating the internal covariate shift problem during deep network training and ensuring stability in the later stages of training.

[0048] 3) Action Output Layer: If the TD3 architecture (deterministic strategy) is adopted: The action output layer consists of a fully connected layer with 6 neurons. The terminal activation function uses Tanh to strictly limit the output value between -1 and 1, directly outputting a deterministic 6-dimensional continuous action vector. During the training phase, clipped Gaussian noise following a normal distribution is manually added to the action output to enhance the agent's exploration ability; the noise variance decreases linearly with the number of training steps, gradually decreasing from the initial value of 0.5 to 0.01, achieving a smooth transition from exploration to utilization.

[0049] If the SAC architecture (stochastic strategy) is adopted: the action output layer is split into two parallel linear network branches, which predict the mean vector of a Gaussian distribution respectively. Sum of logarithmic standard deviation vector To prevent gradient explosion or vanishing, the logarithmic standard deviation is truncated and limited to a preset safety range of -20 to 2. During training, the system samples actions from this dynamic distribution using a reparameterization technique. And ultimately through Mapped to the actual physical region.

[0050] 4) Probability Correction Module: For the SAC algorithm, due to the nonlinear compression effect of the Tanh function on the infinitely wide action space, the system mathematically corrects the logarithmic probability of the sampled action by calculating the Jacobian determinant, i.e.: For the Critic network design, the person responsible for evaluating the state... Next action The expected cumulative return (Q-value). This system uniformly adopts a dual-Q network structure for both TD3 and SAC, that is, two evaluation networks with identical structures but independent parameters (different network weights) are initialized in parallel. and .

[0051] Network splicing mechanism: The Critic network splices the state vector Action vectors output by the Actor The features are concatenated along the feature dimension and then fed into a multi-layer feature extraction network such as Actor. Finally, a single scalar is output through a linear layer as the Q value to evaluate the quality of the action.

[0052] S500 uses building load and photovoltaic capacity prediction data, state space, action space and reward function to train the energy collaborative allocation model, obtain the trained energy collaborative allocation model, and output the building multi-source system energy allocation strategy corresponding to the scheduling period based on the trained energy collaborative allocation model.

[0053] Specifically, such as Figure 3As shown, the current state vector is defined based on building load and photovoltaic capacity prediction data. The current state vector is input into the Actor network structure to obtain the continuous action vector output by the Actor network structure. This continuous action vector is mapped to the actual charging / discharging power of each device and the photovoltaic allocation power. After executing scheduling actions, an immediate reward is obtained, and the state at the next moment is updated. The current state vector and the continuous action vector are input into the Critic network structure to obtain the state-action value output by the Critic network structure. The state vector at the next moment is input into the Actor network structure to obtain the continuous action vector at the next moment. The state vector at the next moment and the continuous action vector at the next moment are then input into the Critic network structure to obtain the state-action value at the next moment. The time difference error is calculated based on the immediate reward, the current action value, and the state-action value at the next moment. The network parameters of the Actor network structure and the Critic network structure are alternately updated based on the time difference error until the model converges or reaches the preset number of iterations, resulting in a trained energy collaborative allocation model.

[0054] In this embodiment, model training and optimization are based on the constructed multi-source environment and experience replay mechanism. Training hyperparameters are configured and loss functions are designed. The Actor-Critic network is alternately optimized through gradient descent, and finally the optimal energy collaborative allocation model is output.

[0055] First, regarding the training parameter configuration, to ensure the stable convergence and efficient exploration of the deep reinforcement learning network in the continuous action space of the multi-source building system, the following core training parameters are configured for this system: 1) Network Optimizer Configuration: Both the Actor and Critic networks use the Adam optimizer for gradient updates. An L2 regularization mechanism with weight decay is introduced for the Critic network to prevent overfitting in finite states. Regarding the learning rate, the Actor learning rate in the TD3 architecture is... The Critic learning rate is Under the SAC architecture, both are uniformly set as .

[0056] 2) Experience Replay and Batch Processing Parameters: Set the maximum capacity of the experience replay pool to 50,000 state transition tuples (s, a, r, s', d). Set the warm-up steps to 15,000 steps. During the warm-up period, the agent takes purely random actions to fill the replay pool. After the warm-up period, at each network update, a small batch of samples is uniformly and randomly drawn from the replay pool for training.

[0057] 3) Core hyperparameters of reinforcement learning: setting reward discount factors To balance short-term local gains with long-term global gains; a target network soft update coefficient is set. For the TD3 algorithm, the policy delay update frequency is set to 2; for the SAC algorithm, the initial entropy temperature coefficient is set. .

[0058] 4) Validation set and early stop parameters: Set the evaluation frequency (perform validation once every 50 rounds) and set the early stop patience value (50 rounds).

[0059] For loss function design, the system's loss function consists of two parts: the Critic value network loss function and the Actor policy network loss function. The design of the loss function varies significantly depending on the selected algorithm kernel, specifically including: 1) Critic value network loss function: Regardless of whether TD3 or SAC architecture is used, the Critic network employs a dual-Q network. and The structure is designed to minimize the mean square error between the current output Q value and the time difference objective.

[0060] The total loss function of the Critic network is defined as: in, For batch size, The objective is time difference.

[0061] If the TD3 architecture is adopted, in order to suppress Q-value overestimation, the calculation of y introduces target policy smoothing noise: in, , This is the clipped Gaussian noise. and The target is the Critic network.

[0062] If the SAC architecture is adopted, since its goal is to maximize the combined objective of cumulative reward and action distribution entropy, the calculation of y needs to subtract the log probability of the target action (i.e., introduce entropy reward): 2) Actor policy network loss function: The optimization objective of the Actor network is to find the action sequence that maximizes the value predicted by the Critic network. Since the gradient descent optimizer performs a minimization operation by default, the Actor loss function is essentially a negative of the expected target value.

[0063] If the TD3 architecture (deterministic policy optimization) is adopted: the Actor only needs to maximize the first Critic network. Evaluate the value of the current deterministic action. Its loss function is defined as: If a SAC (Stochastic Policy Optimization) architecture is adopted: the Actor not only requires its output action to have a high expected reward, but also requires its action distribution to have high randomness. Actions generated in conjunction with a reparameterization sampling mechanism... and the corresponding log probability Its loss function is defined as: Finally, model evaluation and system deployment: design multi-dimensional evaluation indicators to verify the scheduling strategy offline, and deploy the optimal model to the microgrid central controller to perform real-time inference.

[0064] The design evaluation indicators include: 1) Economic benefit indicator: This is measured by calculating the total cost of the system purchasing electricity from the public grid during the test period. Specifically, it is calculated by cumulatively calculating the product of the net power of the grid and the real-time electricity price for each time period.

[0065] 2) Photovoltaic self-consumption rate: Defined as the ratio of the total photovoltaic power generated in the system that is directly consumed by local loads, stationary energy storage, and electric vehicles to the total photovoltaic power generated. This indicator is used to evaluate the model's on-site utilization efficiency of green energy.

[0066] The system performs real-time inference and deployment, extracting and integrating the best-performing model from the validation set obtained during the training phase into the decision-making logic of the building energy management system. Specific deployment steps include: 1) Model loading and initialization: Deploy the Actor network inference engine in the microgrid central controller, load the trained network weights, and set it to deterministic inference mode, that is, for the SAC algorithm, only output the mean of the Gaussian distribution, and for the TD3 algorithm, cancel all exploration noise.

[0067] 2) Real-time status observation and acquisition: Every preset sampling period, the system collects the current battery SOC and the number of electric vehicles online through the underlying sensors, and simultaneously retrieves the future data generated by the prediction module. Hourly source load prediction sequence, construct the state vector at the current time. .

[0068] 3) Deterministic reasoning decision-making: The system inputs an Actor network to generate a 6-dimensional continuous action vector in real time. Based on a preset physical mapping relationship, the system transforms the abstract action values ​​into actual charging and discharging power commands and photovoltaic distribution commands for each device.

[0069] 4) Rigid Constraint Secondary Verification: Before the action is sent to the hardware actuator, the underlying logic performs a final physical safety verification of the action. If the charging action generated by the agent still cannot meet the vehicle's off-grid requirements under extreme conditions, the system will activate the rigid grid compensation mechanism to unconditionally supplement the power supply to ensure energy reliability.

[0070] In summary, the embodiments of the present invention construct a multi-source collaborative model that includes building load, distributed photovoltaic power generation, energy storage units, and mobile electric vehicle groups. Combined with environmental state perception and continuous decision-making mechanisms, it achieves adaptive and optimized allocation of energy flow direction at multiple time scales. This improves the utilization rate of renewable energy, reduces energy costs, and enhances the self-regulation capability of building energy systems while meeting system operation constraints. It is widely applicable to smart buildings, integrated energy management in industrial parks, and new power systems.

[0071] Compared with the prior art, the embodiments of the present invention have the following advantages: 1) Breaking through the bottlenecks of traditional methods, achieving a dual leap in economic benefits and renewable energy absorption rate. Compared with energy management systems that rely on static fixed rules or traditional mathematical programming, this invention constructs a high-dimensional composite reinforcement learning reward function, which can adaptively seek the global optimal solution under multiple objectives within an extremely short online inference time of milliseconds. Under the same source load conditions, the intelligent scheduling strategy of this invention can more efficiently coordinate the energy interaction between photovoltaic, grid and multiple energy storage units, which not only significantly reduces the overall economic cost of microgrid systems purchasing electricity from the public grid, but also greatly improves the local absorption rate of distributed photovoltaic, fundamentally reducing the waste of photovoltaic resources.

[0072] 2) Reconstructing the continuous action mapping mechanism to eliminate invalid exploration and gradient breakage caused by hard truncation. Addressing the problem of numerous invalid actions resulting from excessive reliance on "hard logic truncation" of the underlying environment when traditional reinforcement learning deals with strong physical repulsion constraints in energy storage devices, this invention innovatively designs a smooth continuous action space and mapping transformation rules based on net power calculation. This mechanism ensures that the execution instructions output by the agent are always strictly within safe physical boundaries, not only greatly reducing the agent's invalid exploration space but also effectively protecting the continuity of gradients during backpropagation of the neural network, significantly improving the convergence speed and policy stability of the Actor-Critic algorithm in complex multi-source systems.

[0073] 3) Integrating future time-series prediction features, this invention endows the agent with forward-looking planning capabilities across cycles. The state observation space of this invention proactively introduces a sliding prediction time window sequence that includes building load, photovoltaic capacity, and time-of-use electricity prices at multiple future moments. This time-series context-aware mechanism breaks through the "short-sighted" decision-making limitations of traditional reinforcement learning, which relies solely on the current single-step state. By pre-perceiving the dynamic evolution of the environment, it endows the agent with superior macro-level foresight, enabling it to accurately capture long-term arbitrage opportunities in electricity price fluctuations and make forward-looking cross-cycle energy scheduling plans when facing dynamic scenarios where electric vehicles are connected to or disconnected from the grid at any time.

[0074] 4) A highly generalizable closed-loop adaptive microgrid management system was constructed. The overall system integrates a complete closed loop, from multimodal meteorological / historical data processing and high-precision source-load joint forecasting, to constructing a Markov decision process environment and algorithm training, and finally integrating into the building energy management system for real-time inference and rigid secondary verification. This system can not only respond to changes in energy supply and demand within the system in real time, but also achieve self-iteration and evolution of strategies through continuous environmental interaction. It is widely applicable to complex and ever-changing distributed energy collaborative management scenarios such as smart buildings, zero-carbon parks, and virtual power plants.

[0075] Reference Figure 2 A reinforcement learning-based energy distribution system for multi-source building systems includes: The first module 201 is used to obtain the building load and photovoltaic capacity prediction data output by the distributed source-load joint prediction model of the new power load management system. The second module 202 is used to construct the objective function and physical constraints for energy scheduling of a building multi-source system, wherein the objective function is used to indicate maximizing the comprehensive economic and environmental benefits of the system; The third module 203 is used to construct the state space, action space, and reward function in the Markov decision process model based on the building load and photovoltaic capacity forecast data, objective function, and multiple physical constraints. The state space includes the load forecast sequence, photovoltaic forecast sequence, time-of-use electricity price sequence, and energy storage device power status for multiple future time steps. The action space is used to indicate the photovoltaic allocation ratio and the charging and discharging intensity of each energy storage device. The reward function is used to indicate the comprehensive benefit of the current scheduling action. Module 4, 204, is used to construct an energy collaborative allocation model that includes both Actor and Critic network structures. The fifth module 205 is used to train the energy collaborative allocation model using building load and photovoltaic capacity prediction data, state space, action space and reward function, to obtain the trained energy collaborative allocation model, and to output the building multi-source system energy allocation strategy corresponding to the scheduling period based on the trained energy collaborative allocation model.

[0076] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0077] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this is not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for energy allocation in a multi-source building system based on reinforcement learning, characterized in that, Includes the following steps: Obtain building load and photovoltaic capacity forecast data output from the distributed source-load joint forecasting model of the new power load management system; Construct an objective function and physical constraints for energy scheduling of a building's multi-source system, wherein the objective function is used to indicate maximizing the system's overall economic and environmental benefits; Based on building load and photovoltaic capacity forecast data, objective function and multiple physical constraints, a state space, action space and reward function are constructed in the Markov decision process model. The state space includes the load forecast sequence, photovoltaic forecast sequence, time-of-use electricity price sequence and energy storage device power status for multiple future time steps. The action space is used to indicate the photovoltaic allocation ratio and the charging and discharging intensity of each energy storage device. The reward function is used to indicate the comprehensive benefit of the current scheduling action. Construct an energy collaborative allocation model that includes Actor network structure and Critic network structure; An energy collaborative allocation model is trained using building load and photovoltaic capacity forecast data, state space, action space, and reward function. The trained energy collaborative allocation model is then used to output the energy allocation strategy for the building multi-source system corresponding to the scheduling period.

2. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 1, characterized in that, The step of obtaining the building load and photovoltaic capacity forecast data output by the distributed source-load joint forecasting model of the new power load management system specifically includes: The input historical meteorological parameters, photovoltaic historical data and building load historical data are subjected to outlier removal, missing value filling, feature engineering and normalization to obtain preprocessed historical meteorological parameters, photovoltaic historical data and building load historical data. A source-load joint prediction model was built, which includes a multi-scale feature extraction module, an adaptive spatial attention mechanism and a Transformer encoder. The model was trained by K-fold cross-validation and the optimal model parameters were selected to obtain the trained source-load joint prediction model. A sliding window autoregressive rolling update mechanism is adopted to predict preprocessed historical meteorological parameters, photovoltaic historical data and building load historical data based on a trained source-load joint prediction model, and output building load and photovoltaic capacity prediction data.

3. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 2, characterized in that, The physical constraints include energy conservation constraints for multi-source systems, upper and lower limits of energy storage system capacity, charging and discharging power constraints for energy storage devices, minimum energy guarantee constraints for mobile trolleys leaving the grid, and rigid satisfaction constraints for building loads. The charging and discharging power constraints are used to dynamically adjust the charging and discharging rate based on the current energy capacity of the energy storage devices, and the minimum energy guarantee constraints for mobile trolleys leaving the grid are used to forcibly replenish the minimum energy requirement before the trolley leaves the grid.

4. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 3, characterized in that, The step of training the energy collaborative allocation model specifically includes: Define the current state vector based on building load and photovoltaic capacity forecast data; The current state vector is input into the Actor network structure to obtain the continuous action vector output by the Actor network structure; The continuous action vector is mapped to the actual charging and discharging power of each device and the photovoltaic power allocated. After the scheduling action is executed, an immediate reward is obtained and the state is updated for the next moment. Input the current state vector and continuous action vector into the Critic network structure to obtain the state-action value output by the Critic network structure; Input the next time step state vector into the Actor network structure to obtain the next time step continuous action vector. Then input the next time step state vector and the next time step continuous action vector into the Critic network structure to obtain the next time step state-action value. Calculate the time difference error based on the immediate reward, the value of the action at the current moment, and the value of the state action at the next moment; The network parameters of the Actor network structure and the Critic network structure are updated alternately according to the time difference error until the model converges or reaches the preset number of iterations, thus obtaining the trained energy collaborative allocation model.

5. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 4, characterized in that, The Actor network structure includes an input preprocessing layer, multiple feature extraction layers, and an action output layer. The step of inputting the current state vector into the Actor network structure to obtain the continuous action vector output by the Actor network structure specifically includes: The current state vector is standardized by input preprocessing layer, which maps the features of each dimension to the same distribution space to obtain the standardized state vector. The standardized state vector is obtained by extracting features from the standardized state vector through a multi-layer feature extraction layer. The multi-layer feature extraction layer includes a multi-layer fully connected layer, a layer normalization module, and a ReLU activation function. Action prediction is performed on the standardized state feature vector through the action output layer, and a 6-dimensional continuous action vector is output. When using the TD3 architecture, a deterministic action vector is output and clipped Gaussian noise is added to enhance the exploratory nature. When using the SAC architecture, the Gaussian distribution parameters of the action are output. The continuous action vector is obtained by sampling through reparameterization techniques.

6. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 5, characterized in that, The Critic network structure adopts a dual-Q network architecture, containing two parameter-independent value evaluation networks. The step of inputting the current state vector and continuous action vector into the Critic network structure to obtain the state-action value output by the Critic network structure specifically includes: The state vector and the continuous action vector are concatenated along the feature dimension to obtain the concatenated feature vector; The concatenated feature vectors are input into two value evaluation networks respectively to obtain two independent state-action evaluation values; The evaluation value that is less than a preset threshold among the two state-action evaluation values ​​is selected as the state-action value output by the Critic network structure.

7. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 6, characterized in that, The energy collaborative allocation model is trained using an experience replay mechanism, which specifically includes: Set a preset capacity experience replay pool. During the warm-up phase, fill the replay pool with random actions. After the warm-up is over, randomly select a small batch of samples from the replay pool for training each time the parameters are updated. Both the Actor network and the Critic network use the Adam optimizer for gradient updates, with the Critic network introducing L2 regularization to prevent overfitting. Configure core hyperparameters such as reward discount factor, target network soft update coefficient, policy update frequency, and entropy temperature coefficient to balance short-term and long-term returns.

8. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 7, characterized in that, The step of alternately updating the network parameters of the Actor network structure and the Critic network structure based on the time difference error specifically includes: The loss function of the Critic network is constructed based on the mean square error to minimize the deviation between the current evaluation value and the time difference target value. The parameters of the Critic network are updated by gradient descent. In the TD3 architecture, the time difference target introduces the target policy to smooth noise, while the time difference target in the SAC architecture introduces an entropy reward term. The Actor network loss function is constructed based on policy gradient to maximize the state-action value output by the Critic network. The Actor network parameters are updated through gradient ascent. In the SAC architecture, the Actor network loss includes two optimization objectives: expected reward and action distribution entropy.

9. The energy allocation method for a multi-source building system based on reinforcement learning according to claim 8, characterized in that, The step of outputting the building multi-source system energy allocation strategy corresponding to the scheduling period based on the trained energy collaborative allocation model specifically includes: Based on the trained energy collaborative allocation model, the trained Actor network weights are loaded and set to deterministic inference mode; The current power status of the energy storage device is collected according to the preset sampling period, and the source load prediction sequence and electricity price sequence for multiple future time steps are retrieved simultaneously to construct a real-time state vector; The real-time state vector is input into the Actor network, which outputs a continuous action vector and maps it to actual device scheduling instructions. The dispatching command undergoes a second physical constraint review. If the rigid safety constraint is not met, the power grid compensation mechanism is triggered to correct the command before it is issued for execution. Finally, the energy allocation strategy for the building multi-source system corresponding to the dispatching period is output.

10. A building multi-source system energy distribution system based on reinforcement learning, characterized in that, Includes the following modules: The first module is used to obtain building load and photovoltaic capacity prediction data output by the distributed source-load joint prediction model of the new power load management system. The second module is used to construct the objective function and physical constraints for energy scheduling of a building's multi-source system, wherein the objective function is used to indicate maximizing the system's comprehensive economic and environmental benefits; The third module is used to construct the state space, action space, and reward function in the Markov decision process model based on building load and photovoltaic capacity forecast data, objective function, and multiple physical constraints. The state space includes the load forecast sequence, photovoltaic forecast sequence, time-of-use electricity price sequence, and energy storage device power status for multiple future time steps. The action space is used to indicate the photovoltaic allocation ratio and the charging and discharging intensity of each energy storage device. The reward function is used to indicate the overall benefit of the current scheduling action. The fourth module is used to construct an energy collaborative allocation model that includes both Actor network and Critic network structures; The fifth module is used to train the energy collaborative allocation model using building load and photovoltaic capacity prediction data, state space, action space and reward function, to obtain the trained energy collaborative allocation model, and to output the building multi-source system energy allocation strategy corresponding to the scheduling period based on the trained energy collaborative allocation model.

Images