Building photovoltaic-energy storage cooperative control method, system and equipment based on reinforcement learning, and medium

Abstract

The invention relates to the technical field of building photovoltaic-energy storage cooperative control, in particular to a building photovoltaic-energy storage cooperative control method, system, equipment and medium based on reinforcement learning, and the method comprises the steps: collecting the operation data of a building energy system in real time, and constructing a state vector of reinforcement learning; the state vector is input into a reinforcement learning strategy model, the reinforcement learning strategy model outputs a cooperative control action vector, and the cooperative control action vector at least comprises a charging and discharging power instruction of the energy storage system and an adjusting instruction of the building adjustable load; and performing cooperative control on the energy storage system and the building adjustable load according to the cooperative control action vector. The method has the beneficial effects that the global collaborative optimization of photovoltaic-energy storage-load is realized, the adaptability of the system to illumination fluctuation and sudden load change is enhanced, and the dual improvement of energy efficiency and low-carbon operation is achieved.

Description

Technical Field

[0002] This invention relates to the field of building photovoltaic-energy storage collaborative control technology, and in particular to a building photovoltaic-energy storage collaborative control method, system, equipment and medium based on reinforcement learning. Background Technology

[0003] With global energy shortages and increasingly stringent carbon emission constraints, building energy consumption has become a crucial aspect of urban energy management and carbon reduction. In recent years, the application of distributed photovoltaic (PV) power generation and energy storage systems in buildings has become increasingly widespread, providing new ideas and methods for building energy management. PV systems can utilize renewable energy to power buildings, while energy storage systems achieve efficient energy utilization by regulating peak and off-peak electricity demand. Simultaneously, the application of artificial intelligence (AI) technology in energy management is rapidly developing, especially the advantages of reinforcement learning (RL) in dynamic decision-making and energy optimization, which are gaining increasing attention.

[0004] However, in the practical application of reinforcement learning to building photovoltaic-storage systems, existing control strategies still have significant shortcomings: First, at the state awareness level, most schemes fail to capture sufficient environmental characteristics, making it difficult for the learning strategies to accurately reflect the complex system operating environment. Second, at the optimization objective level, the design of reward functions is often too simplistic, focusing only on single objectives such as reducing electricity costs or increasing photovoltaic self-consumption rate, lacking comprehensive trade-offs and dynamic balancing of multiple objectives. Furthermore, existing strategies are mostly limited to the independent control of energy storage or load, lacking a complete scheme that treats photovoltaics, energy storage, and building adjustable loads as a whole for coordinated optimization, resulting in low overall system energy efficiency and poor adaptability and robustness when dealing with sudden events such as drastic fluctuations in photovoltaic output or sudden load changes. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a building photovoltaic-energy storage collaborative control method based on reinforcement learning, including real-time acquisition of building energy system operation data and constructing it into a reinforcement learning state vector, wherein the state vector includes at least photovoltaic power generation data, energy storage system state data, building load data and external environment data; The state vector is input into the reinforcement learning policy model, and the reinforcement learning policy model outputs the cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward. Based on the coordinated control action vector, coordinated control is performed on the energy storage system and the adjustable load of the building.

[0006] As a preferred embodiment of the reinforcement learning-based building photovoltaic-energy storage coordinated control method of the present invention, the method further includes: After executing control based on the collaborative control action vector, a new state vector and the corresponding immediate reward are collected. Based on the new state vector and the corresponding immediate reward, the parameters of the reinforcement learning policy model are updated in a rolling manner to achieve continuous learning and adaptive optimization.

[0007] As a preferred embodiment of the reinforcement learning-based building photovoltaic-energy storage coordinated control method of the present invention, wherein: the reward function used in the training process of the reinforcement learning strategy model is a dynamic multi-objective weighted reward function; Among them, the dynamic multi-objective weighted reward function simultaneously optimizes economic indicators, comfort indicators, and carbon emission indicators.

[0008] As a preferred embodiment of the reinforcement learning-based building photovoltaic-energy storage coordinated control method of the present invention, wherein the reinforcement learning policy model is trained using a proximal policy optimization algorithm; Among them, the near-end policy optimization algorithm maintains the stability of the training process by limiting the policy update magnitude and uses the pruning mechanism to prevent policy performance degradation, thus ensuring efficient convergence in complex environments.

[0009] As a preferred embodiment of the reinforcement learning-based building photovoltaic-energy storage collaborative control method of the present invention, wherein: the adjustment command of the building adjustable load includes power regulation or start-stop control of at least one of the heating, ventilation and air conditioning system, lighting system and hot water system; Among them, the adjustment instructions are dynamically allocated based on the grid electricity price signal and load priority.

[0010] As a preferred embodiment of the reinforcement learning-based building photovoltaic-energy storage collaborative control method of the present invention, the state vector further includes indoor environmental data and operating status data of the energy conversion equipment; Indoor environmental data is used to assess and ensure the comfort of the building's interior environment in real time during collaborative control processes; Operating status data of energy conversion equipment is used to comprehensively reflect the dynamic characteristics of multiple energy flows in building energy systems.

[0011] As a preferred embodiment of the reinforcement learning-based building photovoltaic-energy storage coordinated control method of the present invention, wherein: the operating status data of the energy conversion equipment is used to comprehensively reflect the dynamic characteristics of multiple energy flows in the building energy system; The energy conversion equipment includes at least one of a combined heat and power system, an electric heat pump, and a gas boiler, and the operating status data includes the equipment power output, operating efficiency, and start-up / shutdown status.

[0012] Secondly, the present invention provides a building photovoltaic-energy storage collaborative control system based on reinforcement learning, comprising: a construction module for real-time acquisition of the operation data of the building energy system and constructing it into a state vector for reinforcement learning, wherein the state vector includes at least photovoltaic power generation data, energy storage system state data, building load data and external environment data; The output module is used to input the state vector into the reinforcement learning policy model, and the reinforcement learning policy model outputs the cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward. The execution module is used to coordinate the control of the energy storage system and the adjustable load of the building based on the coordinated control action vector.

[0013] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: by introducing a dynamic multi-objective weighted reward function and an online rolling update mechanism, the reinforcement learning strategy can dynamically balance the photovoltaic self-consumption rate, energy storage economy and carbon emission indicators while ensuring building comfort. This not only achieves global collaborative optimization of photovoltaic-energy storage-load, but also enhances the system's adaptability to light fluctuations and load changes, achieving a dual improvement in energy efficiency and low-carbon operation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a reinforcement learning-based building photovoltaic-energy storage collaborative control method.

[0018] Figure 2 This is a simulation comparison chart. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0020] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a building photovoltaic-energy storage collaborative control method based on reinforcement learning, including: S100: Real-time acquisition of building energy system operation data and construction of state vectors for reinforcement learning, wherein the state vectors include at least photovoltaic power generation data, energy storage system state data, building load data and external environment data; S200: The state vector is input into the reinforcement learning policy model, and the reinforcement learning policy model outputs the cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward. S300: Based on the coordinated control action vector, it performs coordinated control of the energy storage system and the adjustable load of the building.

[0021] It should be noted that in the field of building energy management, photovoltaic power generation is significantly affected by weather conditions, exhibiting significant volatility and intermittency. Building loads also change dynamically with human activity and equipment start-up and shutdown. Furthermore, the charging and discharging strategies of energy storage systems directly affect system economics and equipment lifespan. However, existing control methods mostly adopt optimization strategies based on fixed rules or single objectives, making it difficult to simultaneously perceive the multi-dimensional dynamic characteristics of photovoltaics, energy storage, load, and the external environment. This leads to problems such as low photovoltaic self-consumption rate, poor energy storage dispatch economics, and inability to guarantee building comfort. At the same time, due to the lack of comprehensive consideration of multi-objective collaborative optimization, traditional strategies lack the ability to adapt to complex operating conditions such as sudden changes in sunlight and sharp increases in load, thus restricting the improvement of the overall energy efficiency and low-carbon operation level of building energy systems.

[0022] Therefore, to address the aforementioned problems of incomplete state perception, singular optimization objectives, and insufficient coordination capabilities, this invention, through steps S100-S300, constructs a reinforcement learning state vector that integrates multi-dimensional operational data to comprehensively perceive the dynamic characteristics of the system; through a reinforcement learning strategy model trained with the goal of maximizing long-term cumulative rewards, it achieves coordinated control command output for energy storage charging and discharging and adjustable building loads; and finally, based on the commands, it realizes integrated scheduling of the energy storage system and building loads, thereby effectively improving photovoltaic utilization, optimizing the economic operation level of energy storage, and enhancing the system's adaptability to complex environments and overall energy efficiency while ensuring building comfort.

[0023] Example 2, refer to Figure 1 As an embodiment of the present invention, based on the above embodiment, a building photovoltaic-energy storage collaborative control method based on reinforcement learning is provided.

[0024] In this embodiment of the application, step S100 involves real-time acquisition of operational data from the building energy system and constructing it into a state vector for reinforcement learning. The state vector includes at least photovoltaic power generation data, energy storage system state data, building load data, and external environmental data. The operational data can be acquired in real-time through a building energy management system (BEMS), including the following steps A1-A3: It should be noted that building energy systems (integrated building energy systems) operate through the coordinated processes of energy supply, consumption, storage, and conversion. Various energy components work together to meet the building's electrical load (EL) and heat load (HL). Common energy components include: heating, ventilation, and air conditioning (HVAC) systems for indoor comfort control; photovoltaic (PV) solar power generation devices; electrical energy storage systems (EES) and thermal energy storage systems (TES); and energy conversion equipment such as combined heat and power (CHP), electrical heat pumps (EHP), and gas boilers (GB). The energy mix varies depending on the type of building. For example, PV is typically installed on residential rooftops, HVAC is mainly used in residential and commercial buildings, while CHP is often used in industrial buildings with higher energy loads. In addition, the building energy system is connected to the electricity grid (EG) and the gas grid (GG) to enable external energy interaction.

[0025] Specifically, the mathematical model for an electrical energy storage (EES) device is as follows: ; In the formula: This represents the charging power of the energy storage system at time t. Indicates the maximum permissible charging / discharging power of the energy storage system; This represents the discharge power of the energy storage system at time t. This represents the energy stored in the electrical energy storage system at time t. This indicates the maximum storage capacity of the energy storage system; This indicates the time step, which is the time interval between two scheduling operations; This indicates the overall efficiency of the energy storage system.

[0026] That is, the charging power cannot be negative and cannot exceed the maximum charging power; the discharging power cannot be positive and cannot exceed the maximum discharging power (absolute value); the remaining battery capacity must be between zero and full capacity; furthermore, the calculation of the battery at the next moment includes: when charging, the amount of electricity charged is multiplied by the efficiency (because there is energy loss); when discharging, the amount of electricity discharged is divided by the efficiency (because more energy will actually be taken out of the battery to meet the output demand).

[0027] Understandably, in building energy systems, modeling solar photovoltaic (PV) power generation devices is the foundation of energy storage-PV-load coordinated scheduling. The accuracy of this modeling directly impacts the effectiveness of strategy optimization. PV modeling generally includes two parts: a PV power generation prediction model (which can be trained using machine learning methods (such as neural networks, support vector regression, etc.) based on historical PV output data and weather forecast data, used to output a PV power generation prediction sequence for the next few hours, and input as part of the state vector into the reinforcement learning strategy model) and a PV power output model. This is combined with weather data, PV module characteristics, and system operating status. Specifically, the photovoltaic power output model is expressed as the product of the rated power of the photovoltaic module and environmental factors, and its form is as follows: ; In the formula: This represents the actual power generation of the photovoltaic system at time t; Indicates the overall efficiency of the photovoltaic system; This indicates the rated power of the photovoltaic module under standard test conditions; This represents the actual solar irradiance on the surface of the photovoltaic panel at time t; This indicates the irradiance under standard test conditions (typically 1000 W / m²). Indicates the power temperature coefficient of a photovoltaic module; This represents the actual operating temperature of the photovoltaic module at time t. This indicates the temperature under standard test conditions (usually 25°C).

[0028] That is, the actual power generation of photovoltaics is directly proportional to the system efficiency, rated power, and actual irradiance, and is related to the operating temperature (the higher the temperature, the lower the output is usually).

[0029] Furthermore, the complete photovoltaic-energy storage dispatch environment formed by combining the photovoltaic power output model and the energy storage model is as follows: ; In the formula: This indicates the battery level at the next moment. It should be noted that "bat" and "EES" have the same meaning. Indicates the battery's charging efficiency; This represents the charging power at time t; This represents the power exchanged between the building and the power grid at time t, where a positive value represents purchasing electricity from the grid and a negative value represents selling electricity to the grid. This represents the building's total electrical load at time t.

[0030] It should be further noted that photovoltaic power generation data includes current output power, historical output power, and short-term forecast power values, used to reflect the fluctuation characteristics of renewable energy; energy storage system status data includes current SOC and charge / discharge capacity data for electrical and thermal energy storage; building load data includes real-time electrical load and real-time thermal load, as well as the adjustable range and priority data of adjustable loads; external environmental data includes meteorological data (such as ambient temperature and solar irradiance), grid electricity prices, and natural gas prices, used to reflect the supply and demand conditions of the external energy market.

[0031] A1: The state vector also includes indoor environmental data and the operating status data of energy conversion equipment; A2: Indoor environmental data, used to assess and ensure the comfort of the building's interior environment in real time during collaborative control processes; Understandably, indoor environmental data includes temperature, humidity, and comfort index data, which are used to ensure building comfort.

[0032] A3: Operating status data of energy conversion equipment, used to comprehensively reflect the dynamic characteristics of multiple energy flows in building energy systems; The energy conversion equipment includes at least one of a combined heat and power system, an electric heat pump, and a gas boiler, and the operating status data includes the equipment power output, operating efficiency, and start-up / shutdown status.

[0033] Preferably, this step, through real-time collection and fusion of this multi-dimensional information (data), not only enables reinforcement learning strategies to fully perceive the dynamic changes in the system's operating environment, but also enables accurate capture of complex features such as photovoltaic power output fluctuations, building load fluctuations, and energy storage system constraints, providing a precise and high-dimensional information foundation for the coordinated scheduling of photovoltaic, energy storage, and load.

[0034] In one optional implementation, the construction of the building energy system's operational data into a reinforcement learning state vector in step S100 can be achieved through data dimensionality reduction and feature extraction. This involves real-time acquisition of multi-dimensional operational data from the building energy system, dimensionality reduction of the acquired data, and extraction of key features: for photovoltaic power generation data, power fluctuation features (such as power change rate and fluctuation amplitude) are extracted; for energy storage system data, SOC change trends and charging / discharging efficiency are extracted; for building load data, peak load, valley load, and adjustable range are extracted; and for environmental data, key meteorological parameters such as temperature and irradiance are extracted. Finally, the dimensionality-reduced key features are fused to construct the state vector. .

[0035] In another optional implementation, the construction of the building energy system's operational data into a reinforcement learning state vector in step S100 can also be done using a deep learning-based approach. This involves annotating the collected operational data, including: high-efficiency and low-efficiency periods of photovoltaic power generation; charge / discharge cycles and health status changes of the energy storage system; peak, off-peak, and adjustable periods of building load; and comfort indicators of the indoor and outdoor environment (such as whether temperature and humidity meet set ranges). The collected data is then trained using a Convolutional Neural Network (CNN) or a Long Short-Term Memory (LSTM) network to extract latent features: CNN is used to extract the spatiotemporal features of photovoltaic power generation and energy storage status, while LSTM is used to capture the temporal variation patterns of building load and environmental parameters. Finally, the feature vector output by the trained deep learning model is used as the reinforcement learning state vector. .

[0036] In this embodiment of the application, in step S200, the state vector is input into the reinforcement learning policy model, and the reinforcement learning policy model outputs a cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward, and includes the following steps B1-B2: Understandably, reinforcement learning tasks are typically described using Markov Decision Processes (MDPs), which consist of corresponding quadruples. E=<S,A,P,R>The structure consists of an agent situated in an environment E, with its perception of the environment described as state s. The state space is S, where each state s... S; the actions that an agent can take constitute the action space A; if a certain action a If A acts on the current state s, the environment will transition from s to another state according to the state transition function P. ; in transition to another state Simultaneously, the environment provides a reward to the agent based on the reward function R. The agent needs to continuously interact with the environment to develop a policy, and the quality of the policy depends on the cumulative reward obtained after long-term execution of this policy. Therefore, in RL tasks, the goal of learning is to find a policy that maximizes the long-term cumulative reward.

[0037] (1) State space: The state space S is the set of all possible states of the system, denoted as: ; Each state s S can be composed of key variables of the system, such as photovoltaic output, energy storage SOC, building load, indoor temperature and humidity, etc. in a building energy system. ; (2) Action space: Action space A is the set of actions that the agent can choose in each state: ; In a building integrated energy system, one action It can represent the energy storage charging and discharging power, the adjustable load adjustment range, and the adjustable equipment start-stop status, that is: ; (3) State transition: State transition function Describes the transition of the environment from the current state s to a new state after action a is performed. The probability of: ; At the same time, the environment is based on the reward function. Return to Instant Rewards : ; The intelligent agent continuously selects actions And observe the status and rewards Update strategy To maximize long-term cumulative rewards: ; In the formula: This is a discount factor used to measure the importance of future rewards.

[0038] B1: During the training of the reinforcement learning policy model, the reward function used is a dynamic multi-objective weighted reward function; Among them, the dynamic multi-objective weighted reward function simultaneously optimizes economic indicators, comfort indicators, and carbon emission indicators.

[0039] It should be noted that carbon emission costs (indicators) are quantified through carbon market prices to reflect the environmental impact of building operations.

[0040] Specifically, the main reward function (which covers economic indicators and carbon emission indicators) is expressed as follows: ; In the formula: Indicates electricity price; This indicates the amount of electricity purchased from the power grid; Indicates the unit price of natural gas; This represents the total amount of natural gas purchased by the building at time t; This indicates the carbon price in the carbon market. This represents the carbon emissions of a building at time t.

[0041] That is, the intelligent agent will lose points whenever it buys electricity or gas from the outside, thereby incentivizing it to make the most of the free photovoltaic and energy storage resources inside.

[0042] Natural gas consumption Composed of the total amount of natural gas consumed by combined heat and power plants and gas-fired boilers: ; In the formula: This represents the amount of natural gas consumed by the combined heat and power system at time t. This represents the amount of natural gas consumed by the gas-fired boiler at time t.

[0043] Building carbon emissions Determined by both electrical carbon factor and natural gas carbon emissions: ; In the formula: The electricity consumption of the building at time t; For regional electrocarbon factor; This represents the average carbon density of natural gas.

[0044] Furthermore, this method incorporates photovoltaic self-consumption rate, energy storage economics, and building comfort into the reward function, and uses dynamic weights... Real-time adjustments are made to each objective, and the total reward function is expressed as follows: ; In the formula: This represents the total reward value at time t; This represents the dynamic weighting coefficient at time t; This represents the economic reward at time t, which is related to building energy costs (such as grid electricity purchase costs and gas costs) and is used to optimize the economic operation of the system. This represents the comfort bonus at time t, which is related to indoor environmental parameters (such as temperature and humidity) used to ensure the comfort of the building. This represents the carbon emission reward at time t, which is related to the system's carbon emissions and is used to achieve low-carbon operation. This represents the photovoltaic utilization rate bonus at time t, which is related to the self-consumption rate of photovoltaic power generation and is used to improve the utilization rate of the photovoltaic system. It represents the energy storage efficiency bonus at time t, which is related to the charging and discharging efficiency and economy of the energy storage system, and is used to optimize the operating state of the energy storage system.

[0045] Among them, dynamic weights It can be dynamically adjusted according to fluctuations in photovoltaic output, energy storage SOC, and load demand, so no restrictions are imposed here.

[0046] Preferably, this step incorporates multiple objectives, such as economy, comfort, carbon emissions, photovoltaic utilization, and energy storage efficiency, into the optimization process of the reinforcement learning strategy simultaneously, and adjusts the importance of each objective in real time through dynamic weights. This fundamentally breaks through the limitation of traditional single reward functions that only focus on a single indicator. Furthermore, the agent can simultaneously consider various constraints and objectives during training, achieving multi-objective balanced optimization, thereby significantly improving the comprehensiveness of the strategy and the robustness of the system, and effectively solving the problem of a single reward function design.

[0047] B2: The reinforcement learning policy model is trained using the proximal policy optimization algorithm; Among them, the near-end policy optimization algorithm maintains the stability of the training process by limiting the policy update magnitude and uses the pruning mechanism to prevent policy performance degradation, thus ensuring efficient convergence in complex environments.

[0048] Understandably, this method employs the Proximal Policy Optimization (PPO) algorithm to update the policy network during training. By limiting the KL divergence between the old and new policies, it ensures a moderate policy update magnitude, avoiding policy collapse or convergence instability caused by excessive bias in importance sampling. During iteration, the policy network continuously absorbs feedback information from the environment, learning the optimal combination of energy storage system charging and discharging timing, flexible load adjustment strategies, and photovoltaic power generation utilization. This achieves the coordinated scheduling objectives of maximizing photovoltaic power generation self-consumption, minimizing grid power purchases, and minimizing carbon emissions. Ultimately, the agent forms a dynamic scheduling strategy for the building energy system, coordinating energy flow between photovoltaic, energy storage, and building loads under different weather conditions, load fluctuations, and uncertainties in photovoltaic power generation, achieving an optimal balance between overall system operating efficiency and environmental friendliness.

[0049] It should be noted that the core principle of the algorithm is through the policy network π θ The iterative update of (a|s) enables efficient and stable learning in a continuous action space. Specifically, the PPO algorithm defines a pruning objective function as follows: ; In the formula: The importance sampling ratio, For the dominant function, This is the clipping threshold.

[0050] This pruning objective function limits the probability ratio of the new and old strategies to [[]. Within a certain range, the PPO algorithm effectively suppresses excessively rapid or large policy updates, thereby avoiding instability or performance degradation during training. Furthermore, the PPO algorithm retains its focus on the value network during the update process. Loss constraints are imposed to ensure the accuracy of state value estimation, and an entropy regularization term is used to encourage policy exploration.

[0051] It should be further explained that the relationship between the PPO algorithm and the building energy system is reflected in the optimization of photovoltaic, energy storage, and load regulation actions by the policy network. The agent observes the environmental state s in each scheduling cycle. t Output action At and receive immediate reward r t The reward function takes into account environmental costs such as electricity costs, energy storage losses, and carbon emissions.

[0052] Preferably, this step, through iterative training of the PPO algorithm, enables the policy network to continuously optimize energy storage charging and discharging strategies, load response, and photovoltaic energy utilization schemes while ensuring training stability. This achieves the optimal strategy for coordinated scheduling of multiple energy devices, thereby improving the operating efficiency of the building's integrated energy system and reducing energy costs and carbon emissions.

[0053] In this embodiment of the application, step S300 involves coordinated control of the energy storage system and the building's adjustable load based on the coordinated control action vector, including the following steps C1-C2: C1: Adjustment instructions for adjustable building loads include power regulation or start / stop control of at least one of the HVAC, lighting, and hot water systems. Understandably, HVAC, lighting, and hot water systems are the most energy-intensive and adjustable loads in a building. By regulating their power (e.g., reducing air conditioning power) or controlling their start / stop (e.g., temporarily shutting down some hot water circulation pumps), the building's total energy consumption curve can be significantly altered without substantially affecting its core functions, thus providing flexibility for energy dispatch.

[0054] C2: Adjustment commands are dynamically allocated based on grid electricity price signals and load priorities.

[0055] It should be noted that dynamic allocation is used to smooth out peak-valley differences in building load and reduce overall energy costs. Specifically: during peak electricity price periods (high electricity prices), there is a tendency to more aggressively reduce adjustable loads (such as appropriately increasing air conditioning temperature settings and dimming public area lighting) to reduce the cost of purchasing electricity from the grid. During off-peak electricity price periods (low electricity prices), restrictions can be relaxed, and equipment can even be turned on earlier to pre-cool / preheat the building, utilizing lower-priced electricity to improve comfort. It is important to emphasize that not all adjustable loads are equal. The system needs to make decisions based on preset priorities. For example: low-priority loads, such as some landscape lighting and power outlets in non-critical areas, can be cut off or reduced first when needed; medium-priority loads, such as general lighting in office areas, can be adjusted while ensuring basic illuminance; high-priority loads, such as power supplies to data centers and critical production equipment, are usually not adjustable; loads strongly related to comfort, such as air conditioning in residential buildings, will have their adjustment range and speed strictly limited to ensure comfort, and will be given a higher priority. In other words, by prioritizing the system, the system can maximize the flexibility of the load side while meeting core requirements, thus avoiding the negative impact of a one-size-fits-all approach.

[0056] It should be further noted that in the photovoltaic-energy storage-load coordinated scheduling framework of this method, strategy training and online deployment adopt a closed-loop iterative mechanism. That is, in the online deployment phase, the trained strategy network is directly applied to the real-time scheduling of the building's integrated energy system. The agent outputs control actions based on real-time observations of photovoltaic output, energy storage capacity, and load demand, achieving dynamic optimization of energy storage charging and discharging, load adjustment, and photovoltaic energy allocation. The system can automatically respond to fluctuations in photovoltaic output, load changes, and electricity price fluctuations, achieving coordinated energy scheduling across all weather conditions and multiple time scales. It is important to emphasize that this method is applicable to commercial complexes, industrial parks, and high-energy-consuming buildings. Taking a commercial complex as an example, the system can maximize photovoltaic self-consumption on sunny days while simultaneously scheduling the energy storage system for peak shaving and valley filling; on cloudy or rainy days, it releases electricity through energy storage and appropriately adjusts controllable loads to achieve load smoothing and minimize electricity purchase costs.

[0057] To further verify the beneficial effects of this method, it is compared with traditional rule-based load scheduling methods, energy storage scheduling methods with PID control, and scheduling methods based on reinforcement learning but without PPO pruning mechanisms. The results are as follows: Figure 2 As shown in the figure. Simulation results show that the proposed method outperforms the aforementioned comparative methods in terms of carbon emission reduction, photovoltaic utilization improvement, and energy storage cycle optimization: under typical operating conditions, the photovoltaic self-consumption rate is increased by 12%~18%, the energy storage peak-valley difference regulation is more stable, and carbon emissions are reduced by approximately 10%~15%. Furthermore, due to the stability of the PPO algorithm's strategy update, the proposed method converges faster and exhibits less performance fluctuation over long-term operation, adapting to the uncertainties of the actual environment and fully demonstrating the technical advantages of synergistic optimization of energy storage and photovoltaics.

[0058] In this embodiment of the application, the method further includes step S400, online iterative optimization, specifically including the following contents D1-D2: D1: After executing control based on the collaborative control action vector, collect the new state vector and the corresponding immediate reward.

[0059] Understandably, acquiring new state vectors is to reflect the impact of actions on the environment, and the further immediate reward is calculated by a dynamic multi-objective weighted reward function based on the new state vector and the previously executed actions. t The immediate reward is used to evaluate the effectiveness of the control decision in the current environment.

[0060] D2: Based on the new state vector and the corresponding immediate reward, the parameters of the reinforcement learning policy model are updated in a rolling manner to achieve continuous learning and adaptive optimization.

[0061] It should be noted that rolling updates refer to the system initiating an update of the strategy model parameters at fixed time intervals after completing the control action and collecting feedback data. It's important to note that this is a minor adjustment, a gradual, online learning approach, rather than batch processing.

[0062] Specifically, the system divides a day into T scheduling cycles, each with a length of Δt (5~15 minutes). In the t-th cycle, the reinforcement learning policy is based on the current state. Select Action : ; In the formula: As the current strategy, by The parameters indicate that the actions include energy storage charging and discharging power. Adjustable load regulation of buildings and adjustable equipment start / stop status After the action is performed, the system state transitions to the next state. : ; And receive instant rewards : After each cycle is completed, the strategy parameters Iterative updates are performed based on the current state, action, and reward feedback, using a gradient descent update method commonly found in deep reinforcement learning. ; in, For learning rate, As a discount factor, This is a state-value function. Through this rolling optimization and real-time iteration, the strategy can quickly respond to fluctuations in photovoltaic power generation, sudden load changes, and changes in energy storage SOC, while balancing multi-objective reward functions to achieve overall optimization of photovoltaic utilization, energy storage economics, building comfort, and carbon emissions.

[0063] It is important to emphasize that the core of rolling optimization lies in not only executing the policy in each cycle, but also continuously adjusting the policy parameters through feedback, so that the reinforcement learning policy maintains high robustness in dynamic environments and gradually approaches the optimal policy that maximizes long-term cumulative rewards. ; Ideally, since building characteristics (such as occupancy rate, equipment efficiency), photovoltaic system performance, and even grid electricity pricing policies will change slowly over time, the linear iteration in this step enables the model to not only track these long-term trends and maintain optimal performance, but also to quickly learn from real-world feedback and adjust its strategies to adapt to new environments when encountering extreme weather, sudden load changes, or other situations not seen in offline training, thus avoiding strategy failure.

[0064] In summary, this method, by introducing a dynamic multi-objective weighted reward function and an online rolling update mechanism, enables the reinforcement learning strategy to dynamically balance photovoltaic self-consumption rate, energy storage economics, and carbon emission indicators while ensuring building comfort. This not only achieves global synergistic optimization of photovoltaic-energy storage-load, but also enhances the system's adaptability to light fluctuations and load abrupt changes, achieving a dual improvement in energy efficiency and low-carbon operation.

[0065] Example 3 illustrates a schematic scheme of a reinforcement learning-based building photovoltaic-energy storage collaborative control method. It should be noted that the technical solution of this reinforcement learning-based building photovoltaic-energy storage collaborative control system belongs to the same concept as the technical solution of the reinforcement learning-based building photovoltaic-energy storage collaborative control method described above. Details not described in detail in this embodiment can be found in the description of the reinforcement learning-based building photovoltaic-energy storage collaborative control method described above.

[0066] This embodiment also provides a building photovoltaic-energy storage collaborative control system based on reinforcement learning, including: The module is used to collect real-time operational data of building energy systems and construct them into state vectors for reinforcement learning. The state vectors include at least photovoltaic power generation data, energy storage system state data, building load data, and external environment data. The output module is used to input the state vector into the reinforcement learning policy model, and the reinforcement learning policy model outputs the cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward. The execution module is used to coordinate the control of the energy storage system and the adjustable load of the building based on the coordinated control action vector.

[0067] This embodiment also provides an electronic device suitable for reinforcement learning-based building photovoltaic-energy storage collaborative control, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the reinforcement learning-based building photovoltaic-energy storage collaborative control method proposed in the above embodiment.

[0068] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the reinforcement learning-based building photovoltaic-energy storage collaborative control method proposed in the above embodiments.

[0069] The storage medium proposed in this embodiment and the building photovoltaic-energy storage collaborative control method based on reinforcement learning proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0070] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0071] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A building photovoltaic-energy storage collaborative control method based on reinforcement learning, characterized in that: include, Real-time acquisition of building energy system operation data and construction of state vectors for reinforcement learning, wherein the state vectors include at least photovoltaic power generation data, energy storage system state data, building load data and external environment data; The state vector is input into the reinforcement learning policy model, and the reinforcement learning policy model outputs a cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward. Based on the coordinated control action vector, the energy storage system and the building's adjustable load are coordinated for control.

2. The building photovoltaic-energy storage collaborative control method based on reinforcement learning as described in claim 1, characterized in that: The method also includes, After executing control according to the collaborative control action vector, a new state vector and the corresponding immediate reward are collected; Based on the new state vector and the corresponding immediate reward, the parameters of the reinforcement learning policy model are updated continuously to achieve continuous learning and adaptive optimization.

3. The building photovoltaic-energy storage collaborative control method based on reinforcement learning as described in claim 2, characterized in that: During the training of the reinforcement learning policy model, the reward function used is a dynamic multi-objective weighted reward function; The dynamic multi-objective weighted reward function simultaneously optimizes economic indicators, comfort indicators, and carbon emission indicators.

4. The building photovoltaic-energy storage collaborative control method based on reinforcement learning as described in claim 3, characterized in that: The reinforcement learning policy model is trained using a proximal policy optimization algorithm; The proximal policy optimization algorithm maintains the stability of the training process by limiting the policy update magnitude and uses a pruning mechanism to prevent policy performance degradation, thus ensuring efficient convergence in complex environments.

5. The building photovoltaic-energy storage collaborative control method based on reinforcement learning as described in claim 1, characterized in that: The adjustment instructions for the adjustable load of the building include power regulation or start / stop control of at least one of the HVAC system, lighting system and hot water system. The adjustment instructions are dynamically allocated based on the grid electricity price signal and load priority.

6. A building photovoltaic-energy storage collaborative control method based on reinforcement learning as described in any one of claims 1-5, characterized in that: The state vector also includes indoor environmental data and operating status data of energy conversion equipment; The indoor environmental data is used to assess and ensure the comfort of the building's interior environment in real time during the collaborative control process; The operating status data of the energy conversion equipment is used to comprehensively reflect the dynamic characteristics of multiple energy flows in the building energy system.

7. The building photovoltaic-energy storage collaborative control method based on reinforcement learning as described in claim 6, characterized in that: The operating status data of the energy conversion equipment is used to comprehensively reflect the dynamic characteristics of multiple energy flows in the building energy system; The energy conversion equipment includes at least one of a combined heat and power system, an electric heat pump, and a gas boiler, and the operating status data includes equipment power output, operating efficiency, and start / stop status.

8. A building photovoltaic-energy storage collaborative control system based on reinforcement learning, using the method described in any one of claims 1-7, characterized in that, include: A construction module is used to collect real-time operational data of building energy systems and construct them into state vectors for reinforcement learning. The state vectors include at least photovoltaic power generation data, energy storage system state data, building load data, and external environment data. The output module is used to input the state vector into the reinforcement learning policy model, and the reinforcement learning policy model outputs a cooperative control action vector. The cooperative control action vector includes at least the charging and discharging power command of the energy storage system and the adjustment command of the building's adjustable load. The reinforcement learning policy model is obtained by training with the goal of maximizing long-term cumulative reward. An execution module is used to perform coordinated control of the energy storage system and the building's adjustable load according to the coordinated control action vector.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images