An edge collaborative energy efficiency improvement method and system for a building with multiple loads, and a storage medium

Abstract

The application discloses a kind of edge collaborative energy efficiency improvement methods, systems and storage medium for multi-load-containing building.The process is: constructing energy efficiency improvement intelligent agent model based on reinforcement learning;Collect the historical power consumption data of building users;Intelligent agent model is trained using historical power consumption data;The new model trained is input to the edge computing unit inside the building, and a local energy efficiency optimization system is formed by the random load of user, adjustable load, control terminal and edge computing unit, and the energy efficiency is improved by participating in the energy scheduling of building through the local energy efficiency system.The edge technology used by the application constructs the energy scheduling strategy of building, reduces the calculation requirement of edge computing unit, and improves energy efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of power demand response, and specifically relates to a method, system and storage medium for improving edge-end collaborative energy efficiency in buildings with multiple loads. Background Art

[0002] In recent years, energy shortages have become increasingly severe, and improving energy efficiency has garnered widespread attention across all areas of electricity consumption. Statistics show that buildings account for nearly 40% of global energy consumption, and this proportion continues to grow. Therefore, improving building energy efficiency and saving energy costs are crucial to alleviating energy shortages. With advances in distributed renewable energy technologies, buildings that incorporate multiple loads, such as rooftop photovoltaics, energy storage equipment, and user loads, are rapidly developing. This has further complicated the issue of increasing user energy consumption in the absence of existing capacity.

[0003] Currently, building energy efficiency control is primarily achieved through building energy management systems (BEMs). These systems, comprised of energy scheduling strategies and load control methods, dispatch load resources within a building to meet user needs, reducing peak grid power consumption and, consequently, saving users' energy needs. Common approaches include pre-emptive scheduling, which models user electricity usage behavior, formulates the energy efficiency improvement problem as an optimization problem, and solves it to derive operational strategies for each load over several future time periods. However, due to the uncertainty of user electricity usage, pre-emptive strategies are difficult to achieve effective results, resulting in significant limitations for these approaches. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the above-mentioned background technology and to provide a method, system and storage medium for improving edge collaborative energy efficiency in buildings with multiple loads. While ensuring the user's electricity demand, the user's energy efficiency can be improved through the collaboration between the edge computing unit and the control terminal.

[0005] The technical solution adopted by the present invention is: a method for improving the energy efficiency of edge-to-edge collaboration in buildings with multiple loads.

[0006] Build an energy efficiency improvement agent model based on reinforcement learning;

[0007] Collect historical electricity consumption data of building users;

[0008] Use historical electricity consumption data to train the intelligent agent model;

[0009] The trained new model is input into the edge computing unit inside the building to form a local energy efficiency optimization system consisting of user random loads, adjustable loads, control terminals and edge computing units. Energy efficiency is improved by participating in the energy scheduling of the building through the local energy efficiency system.

[0010] Furthermore, the historical electricity consumption data includes the total electricity consumption of the building in each time period, the power generation of renewable energy, and the working status of key loads.

[0011] Furthermore, a neural network is used to build the main structure of the intelligent agent, the state quantities in the building that can be observed by the intelligent agent are set and used as the input of the main structure, the control actions of the controllable loads in the building are used as the output of the main structure, and the goal of minimizing electricity costs is used as the optimization direction of the main structure to construct an energy efficiency improvement intelligent agent model based on reinforcement learning.

[0012] Furthermore, the electricity cost minimization objective is transformed into a discrete time form, and within a discrete time period, the electricity cost of the period is formed into a reward function and input into the main structure.

[0013] Furthermore, the reward function is:

[0014]

[0015] Among them, r t is the reward value at time t; is the user's comfort cost at time t; α is the discount coefficient of the comfort cost; p t is the electricity price at time t; is the total energy consumption of the building at time t; t is the discrete time sequence number.

[0016] Furthermore, the process of training the agent model is as follows:

[0017] 1) Initialize the policy network π in the agent model θ and value network v ω ;

[0018] 2) Constructing the building simulation model E(a t ) and initialize;

[0019] 3) Read the state quantity s at time t from the data set in chronological order t ;

[0020] 4) Input the state quantity into the policy network and then output the control action;

[0021] 5) Input the control action into the building simulation model E(a t ), and calculate the state quantity s at time t+1 t+1 ;

[0022] 6) According to the state quantity s at time t+1 t+1 Calculate the reward value r at time t t ;

[0023] 7) The state quantity s at time t t , control action a t , reward value r t and the state quantity s at time t+1 t+1 Store in the cache;

[0024] 8) Repeat steps 3) to 7) until the cache is full;

[0025] 9) Randomly sample the cache area and calculate the strategy network π based on the sampled data θ and value network v ω The loss function L π (θ) and L v (ω);

[0026] 10) According to the loss function L π (θ) and L v (ω) Update the policy network π θ and value network v ω Parameters;

[0027] 11) Clear the buffer and repeat steps 3) to 10) until the reward value converges, completing the training.

[0028] Furthermore, the loss functions of the strategy network and the value network are respectively determined by the following formulas:

[0029]

[0030]

[0031] Among them, L π (θ) is the loss function of the policy network; L v (ω) is the loss function of the value network; E[] represents the expectation within the brackets; L f (θ) is the objective function of the policy network; c1 is the hyperparameter of the entropy of the policy; H is the entropy of the policy; v ω (s t ) is the output of the value network; γ is the discount factor; l is the time series label; r t+l is the reward value at the t+1th moment; T is the length of the discrete time sequence; t is the discrete time sequence number.

[0032] Furthermore, the building simulation model includes a simulation model of an energy storage device:

[0033] SoC t+1 =SoC t +(δ ch ·P t ch -1 / δ dis ·Pt dis )ΔT / E max

[0034] Among them, SoC t+1 is the state of charge of the energy storage device at time t+1; SoC t is the state of charge of the energy storage device at time t; δ ch and P t ch are the charging efficiency and charging power of energy storage equipment respectively; δ dis and P t dis Respectively represent the discharge efficiency and discharge power of the energy storage device; E max is the capacity of the energy storage device; ΔT is the time interval.

[0035] Furthermore, the building simulation model includes a simulation model of air conditioning:

[0036] T t+1 =εT t +(1-ε)(T t out +η / A·P t AC ·ΔT)

[0037] Among them, T t+1 is the indoor temperature at time t+1; T t is the indoor temperature at time t; ε is the conversion factor; T t out is the outdoor temperature at time t; η is the heat conversion coefficient; A is the thermal conductivity of the entire building; P t AC is the working power of the HVAC at time t; ΔT is the time interval.

[0038] Furthermore, the process of the local energy efficiency system participating in the energy scheduling of buildings is as follows:

[0039] The control terminal obtains the status of the building and uploads it to the edge computing unit, wherein the status includes the user random load and the operating status of the adjustable equipment;

[0040] The edge computing unit outputs a control action of the adjustable load according to the state, and outputs the control action to the control terminal;

[0041] The control terminal executes the control action to achieve energy efficiency improvement.

[0042] A system for realizing the above-mentioned edge-end collaborative energy efficiency improvement method for buildings with multiple loads, comprising

[0043] Data collection module, used to collect historical electricity consumption data of building users;

[0044] Model building module, used to build energy efficiency improvement agent model based on reinforcement learning;

[0045] The training module uses historical electricity consumption data to train the intelligent agent model;

[0046] The energy scheduling module inputs the trained new model into the edge computing unit inside the building to form a local energy efficiency optimization system composed of user random loads, adjustable loads, control terminals and edge computing units. The local energy efficiency system participates in the energy scheduling of the building to achieve energy efficiency improvement.

[0047] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0048] The beneficial effects of the present invention are:

[0049] The present invention utilizes "edge" technology to construct an energy scheduling strategy for buildings, reducing the computing requirements of edge computing units and improving energy efficiency.

[0050] The present invention collects data through the control terminal and adopts a learning method to avoid modeling uncertain factors such as users' electricity usage habits, thereby reducing the difficulty in actual deployment; at the same time, through the "historical data training, online operation" technical means, the present invention can achieve near-real-time energy scheduling strategy output and execution, greatly reducing the amount of calculation within each scheduling time period, thereby meeting the computing power limitations of the edge computing unit.

[0051] The present invention uses historical data collected by the control terminal to train the intelligent agent, converts the historical data into the form of state, action, and reward function described in the intelligent agent model, inputs the data into the intelligent agent and implements the training on the cloud server side; after the training is completed, the intelligent agent is deployed to the edge computing unit to participate in the real-time decision-making of the building energy scheduling, improve the electricity efficiency, and save electricity costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of the energy efficiency improvement method of the present invention.

[0053] Figure 2 This is the workflow diagram for participating in energy scheduling in the fourth stage of the present invention.

[0054] Figure 3 Schematic diagram of an embodiment of the present invention.

[0055] Figure 4This is an energy scheduling effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following is a further description of specific embodiments of the present invention in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is intended to facilitate understanding of the present invention and does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0057] like Figure 1 As shown, the present invention provides a method for improving edge-end collaborative energy efficiency for a building with multiple loads, comprising the following steps:

[0058] Step 1: The load control terminal collects historical electricity usage data from building users, including total electricity consumption by time period, renewable energy generation, and the operating status of key loads. This data is then uploaded to the cloud server for use during training. Renewable energy sources include photovoltaic and wind power generation, while key loads include air conditioners and energy storage devices. Operating status includes indoor temperature, set temperature, and energy storage capacity, among other information accessible to the control terminal.

[0059] Step 2. Construct an energy efficiency improvement intelligent agent model based on reinforcement learning: Use a neural network to build the main structure of the intelligent agent, set the state quantity in the building that can be observed by the intelligent agent and use it as the input of the main structure, use the control action of the controllable load in the building as the output of the main structure, and use the goal of minimizing electricity costs as the optimization direction of the main structure to construct an energy efficiency improvement intelligent agent model based on reinforcement learning.

[0060] Step 3: Use historical electricity consumption data to train the intelligent agent model: convert the historical data into the form of state, action, and reward function described in the intelligent agent model, input it into the intelligent agent and implement training on the cloud server. The specific process is described in the embodiment.

[0061] Step 4: Input the trained new model into the edge computing unit inside the building to form a local energy efficiency optimization system consisting of user random loads, adjustable loads, control terminals and edge computing units. The local energy efficiency system participates in the energy scheduling of the building to achieve energy efficiency improvement.

[0062] In the above scheme, with the goal of minimizing electricity costs, an energy efficiency improvement agent model based on reinforcement learning is constructed. The electricity cost minimization objective is converted into a discrete time form. Specifically, within a discrete time period, the electricity cost of that period is used as the agent's reward function, indicating the accurate optimization direction for the agent's subsequent training. Based on the scale and dimension of the data collected by the control terminal, the state s in the building that can be observed by the agent is set.t (Power and temperature of controllable electrical appliances, data measurable in the environment, ambient temperature). This means selecting the data uploaded by the control terminal within any time period as the building's state at the end of that time period. The controllable loads in the building are identified, and the control actions for these loads are selected as actions executable by the agent. Based on the data uploaded by the terminal, the agent can calculate the action for the controllable load at the beginning of the next time period.

[0063] In the above scheme, the energy cost is considered to be the actual cost of purchasing electricity from the grid for the building within a day. The specific calculation formula is:

[0064]

[0065] Among them, C tota l is the total electricity purchase cost of a building in one day; T is the length of the discrete time series; p t is the electricity price at time t. Optionally, this algorithm is adapted to the time-of-use electricity price billing model; is the electricity purchase amount of the building at time t. Satisfy the constraints: in, is the distributed renewable energy power generation power at time t; is the total energy consumption of controllable load at time t; is the total energy consumption of the uncontrollable load at time t.

[0066] In the above scheme, the reward function is:

[0067]

[0068] Among them, r t is the reward value at time t; is the user's comfort cost at time t; α is the discount coefficient of the comfort cost; p t is the electricity price at time t; is the electricity purchase amount of the building at time t; t is the discrete time sequence number.

[0069] In the above scheme, taking the key load air conditioning of a building as an example, its comfort cost can be considered as:

[0070]

[0071] Among them, T t in represents the indoor temperature at time t; T max Indicates the maximum value of the indoor temperature setting; T min Indicates the minimum value of the indoor temperature setting; | | + Indicates taking only the positive part.

[0072] In the above scheme, the control actions of the intelligent agent are set as the controllable loads of the building. Taking air conditioning and energy storage equipment as an example, the control action set is:

[0073] a t ={P t ac ,P t ess}

[0074] Among them, a t To control the action; P t ac and P t es are the power of the air conditioner and energy storage device at time t, and both satisfy the constraints:

[0075]

[0076]

[0077] in, is the maximum operating power of the air conditioner; is the maximum operating power of energy storage.

[0078] The main structure of the intelligent agent is built using a neural network, including a policy network and an evaluation network. The policy network calculates and outputs the control action a by inputting the state uploaded by the control terminal. t The evaluation network inputs the state and strategy (control action) and outputs the value of the current state, which is used to judge the quality of the strategy network output and guide the parameter update direction of the strategy network.

[0079] In the above scheme, the policy network and the value network can be constructed as a multi-layer perceptron (MLP), using π θ and v ω Indicates that the loss functions of the two can be set as:

[0080] Policy Network:

[0081] Evaluation network:

[0082] in, is the objective function of the policy network; θ is the network parameter of the policy network; is the policy network before iteration; c1 is the hyperparameter of the entropy of the policy; H is the entropy of the policy; s t is the state quantity at time t; a t is the control action at time t; ω is the network parameter of the value network; β is the hyperparameter of KL divergence; KL represents the KL divergence of the two objects in the brackets; is the advantage function; is the information entropy of the strategy; γ is the discount factor; r t+l is the reward value at the t+1th moment.

[0083] In order to realize the above-mentioned edge-end collaborative energy efficiency improvement method, the present invention also provides an edge-end collaborative energy efficiency improvement system for buildings with multiple loads, including

[0084] Data collection module, used to collect historical electricity consumption data of building users;

[0085] Model building module, used to build energy efficiency improvement agent model based on reinforcement learning;

[0086] The training module uses historical electricity consumption data to train the intelligent agent model;

[0087] The energy scheduling module deploys the trained new model to the edge computing unit inside the building to form a local energy efficiency optimization system composed of user random loads, adjustable loads, control terminals and edge computing units. The local energy efficiency system participates in the energy scheduling of the building to achieve energy efficiency improvement.

[0088] Example

[0089] Consider a typical smart building, whose loads include: distributed renewable energy power generation equipment, energy storage equipment, central air conditioning and other user random loads. According to the present invention, all types of loads are connected to the load control terminal, and the control terminal can obtain information such as the power consumption, switch status, and working status of each type of load. The present invention does not exclude the direct control of various loads by users, and the priority of user control instructions is higher than the instructions calculated by the method described in the present invention. If the power grid adopts a time-of-use electricity price billing mode, the control terminal will also be connected to the user-level distribution network to receive time-of-use electricity price data from the power grid. The control terminal can be connected to the edge computing unit through communication technology, such as wireless communication 5G / 4G or wired network. There is a communication link between the edge computing unit and the cloud server for intelligent body training, which can forward historical data from the terminal to the cloud server and receive trained model parameters from the cloud server. See Appendix Figure 2 The energy scheduling task in this example will be performed in discrete time, i.e., t = {1, ..., T}, and the time interval between any time periods is ΔT. According to the present invention, the work of each stage is described as follows.

[0090] Phase 1: The load control terminal uploads the collected historical data (power, status, etc.) to the cloud server where an untrained energy efficiency improvement intelligent agent model is deployed.

[0091] Phase 2: Build an energy efficiency improvement agent model. In this example, the reward function and control actions are defined as the optional forms described in this invention. The controllable loads are central air conditioning and energy storage devices. At any given moment, the state, or information acquired by the control terminal, is selected to include: the total power of the user's random loads, the power of the central air conditioning, the indoor temperature, the power of the energy storage device, the amount of energy storage device electricity, and the output power of distributed renewable energy sources.

[0092] Phase 3: The cloud server performs model training based on historical data. It should be noted that this invention uses a reinforcement learning approach, and the collected data only includes state information. The reward value required for model training is not included, and needs to be calculated from the state information through a reward function.

[0093] Considering that the agent needs an interactive simulation environment during training t+1 ~E(a t ), that is, inputting a control action and returning the state at the next moment and the reward value of the current action. In this case, dispatchable loads (energy storage equipment and air conditioning) are modeled.

[0094] The simulation model of the energy storage device can be in the following form:

[0095] SoC t+1 =SoC t +(δ ch ·P t ch -1 / δ dis ·P t dis )ΔT / E max

[0096] Among them, SoC t+1 is the state of charge of the energy storage device at time t+1; SoC t is the state of charge of the energy storage device at time t; δ ch and P t ch are the charging efficiency and charging power of energy storage equipment respectively; δ dis and P t dis Respectively represent the discharge efficiency and discharge power of the energy storage device; E max is the capacity of the energy storage device; ΔT is the time interval.

[0097] The simulation model of the air conditioner can take the following form:

[0098] T t+1 =εT t +(1-ε)(T t out +η / A·P t AC·ΔT)

[0099] Among them, T t+1 is the indoor temperature at time t+1; T t is the indoor temperature at time t; ε is the conversion factor; T t out is the outdoor temperature at time t; η is the heat conversion coefficient; A is the thermal conductivity of the entire building; P t AC is the working power of the HVAC at time t; ΔT is the time interval.

[0100] It should be noted that the above-mentioned model of energy storage equipment and air conditioning is only used for simulation analysis during training. It is not relied upon in actual applications. Therefore, it can be replaced with any physical model that conforms to the equipment described without affecting the specific deployment of the present invention.

[0101] Reinforcement learning-based agent training follows the basic rules and methods of this field. This invention only uses this agent to model the energy efficiency improvement method for buildings. The related agent training methods do not affect the implementation of this invention. According to the optional forms of the policy network and value network of this invention, the agent training can follow the following steps:

[0102] 1) Initialize the policy network π in the agent model θ and value network v ω ;

[0103] 2) Constructing the building simulation model E(a t ) and initialize;

[0104] 3) Read the state quantity s at time t from the data set in chronological order t ;

[0105] 4) Input the state quantity into the policy network and then output the control action:

[0106]

[0107] μ,σ~π θ (s t ), where a t is the control action at time t; N(μ, σ 2 ) represents the normal distribution, μ represents the mean of the normal distribution, σ 2 represents the variance of the normal distribution;

[0108] 5) Input the control action into the building simulation model E(a t ), and calculate the state quantity s at time t+1 t+1 ;

[0109] 6) According to the state quantity s at time t+1 t+1 Calculate the reward value r at time t t ;

[0110] 7) The state quantity s at time t t , control action a t , reward value r t and the state quantity s at time t+1 t+1 Store in the cache;

[0111] 8) Repeat steps 3) to 7) until the cache is full;

[0112] 9) Randomly sample the cache area and calculate the strategy network π based on the sampled data θ and value network v ω The loss function L π (θ) and L v (ω);

[0113] 10) According to the loss function L π (θ) and L v (ω) Update the policy network π θ and value network v ω Parameters;

[0114] 11) Clear the buffer and repeat steps 3) to 10) until the reward value converges, completing the training.

[0115] Phase 4: After the cloud server completes the training of the building energy efficiency improvement agent, the model parameters related to the agent will be sent to the edge computing unit of the building. At this point, a local energy efficiency optimization system consisting of user random load, adjustable load, control terminal and edge computing unit has been formed inside the building. In the operation phase, the control terminal will upload the acquired building status (including user random load and the operating status of adjustable equipment) to the edge computing unit, such as Figure 3 As shown, the edge computing unit inputs the state information into the energy efficiency improvement agent and returns the control action of the adjustable load to the control terminal for execution. The control terminal executes the control action to achieve energy efficiency improvement. The specific implementation process of the control terminal does not belong to the scope of this invention and will not be described here. According to this example, the energy scheduling effect of this stage is shown in the attached figure. Figure 4 Results show that the energy storage system charges between 11:00 and 16:00, storing excess photovoltaic power, and then continuously discharges it between 19:00 and 21:00, when electricity prices are higher, to power appliances such as air conditioners. This fully utilizes photovoltaic power generation and reduces energy costs.

[0116] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned energy efficiency improvement method and embodiments. Alternatively, the computer program, when executed by a processor, implements the functions of each module / unit in the aforementioned energy efficiency improvement system.

[0117] Those skilled in the art will also appreciate that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of the two. To clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0118] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor. Alternatively, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration. The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention. Contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for improving edge-to-edge collaborative energy efficiency in a building with multiple loads, characterized by: Build an energy efficiency improvement agent model based on reinforcement learning; Collect historical electricity consumption data of building users; Use historical electricity consumption data to train the intelligent agent model; The trained new model is input into the edge computing unit inside the building to form a local energy efficiency optimization system consisting of user random loads, adjustable loads, control terminals, and edge computing units. This local energy efficiency system participates in the energy scheduling of the building. The main structure of the intelligent agent is constructed using a neural network. The state variables in the building that the intelligent agent can observe are set as the input of the main structure. The control actions of the controllable loads in the building are used as the output of the main structure. The goal of minimizing electricity costs is used as the optimization direction of the main structure to construct an energy efficiency improvement intelligent agent model based on reinforcement learning. ； Satisfy the constraints: ; in, The total electricity cost of the building in one day; is the length of the discrete time series; is the electricity price at time t; For building buildings The amount of electricity purchased at any given moment; is the distributed renewable energy power generation power at time t; is the total energy consumption of controllable load at time t; is the total energy consumption of the uncontrollable load at time t.

2. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 1 is characterized by: The historical electricity consumption data includes the total electricity consumption of the building in each time period, the power generation of renewable energy, and the working status of key loads.

3. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 1 is characterized by: The electricity cost minimization objective is transformed into a discrete time form. Within a discrete time period, the electricity cost of the period is formed into a reward function and input into the main structure.

4. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 3 is characterized by: The reward function is: ； in, is the reward value at time t; is the user's comfort cost at time t; is the discount factor for comfort cost; is the electricity price at time t; For building buildings The amount of electricity purchased at the moment; t is the discrete time sequence number.

5. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 1 is characterized by: The process of training the agent model is: 1) Initialize the policy network in the agent model and value network ; 2) Constructing a building simulation model And initialize; 3) Read the state quantity at time t from the data set in chronological order ; 4) Input the state quantity into the policy network and then output the control action; 5) Input control actions into the building simulation model , and calculate the state quantity at time t+1 ; 6) According to the state quantity at time t+1 Calculate the reward value at time t ; 7) The state quantity at time t , control action , reward value and the state quantity at time t+1 Store in the cache; 8) Repeat steps 3) to 7) until the cache is full; 9) Randomly sample the cache area and calculate the strategy network based on the sampled data and value network The loss function and ; 10) According to the loss function and Update policy network and value network Parameters; 11) Clear the buffer and repeat steps 3) to 10) until the reward value converges, completing the training.

6. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 5 is characterized by: The loss functions of the policy network and the value network are determined by the following formulas: ， ， in, is the loss function of the policy network; is the loss function of the value network; E[] represents the expectation within the brackets; is the objective function of the policy network; c1 is the hyperparameter of the entropy of the policy; H is the entropy of the policy; is the output of the value network; is the discount factor; Mark for time series; is the reward value at the t+1th moment; T is the length of the discrete time sequence; t is the discrete time sequence number.

7. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 5 is characterized by: The building simulation model includes a simulation model of an energy storage device: ； in, is the state of charge of the energy storage device at time t+1; is the state of charge of the energy storage device at time t; and are the charging efficiency and charging power of energy storage equipment respectively; and They represent the discharge efficiency and discharge power of the energy storage device respectively; is the capacity of the energy storage device; is the time interval.

8. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 5 is characterized by: The building simulation model includes a simulation model of air conditioning: ； in, is the indoor temperature at time t+1; is the indoor temperature at time t; is the conversion factor; is the outdoor temperature at time t; is the thermal conversion coefficient; is the thermal conductivity of the building as a whole; is the working power of the HVAC at time t; is the time interval.

9. The edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads according to claim 1 is characterized by: The process of the local energy efficiency system participating in the energy scheduling of buildings is as follows: The control terminal obtains the status of the building and uploads it to the edge computing unit, wherein the status includes the user random load and the operating status of the adjustable equipment; The edge computing unit outputs a control action of the adjustable load according to the state, and outputs the control action to the control terminal; The control terminal executes the control action.

10. A system for implementing the edge-to-edge collaborative energy efficiency improvement method for a building with multiple loads as described in any one of claims 1 to 9, characterized in that: include Data collection module, used to collect historical electricity consumption data of building users; Model building module, used to build energy efficiency improvement agent model based on reinforcement learning; A training module is used to train the intelligent agent model using historical electricity consumption data; The energy scheduling module is used to input the trained new model into the edge computing unit inside the building to form a local energy efficiency optimization system composed of user random loads, adjustable loads, control terminals and edge computing units, and participate in the energy scheduling of the building through the local energy efficiency system.

11. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images