Building energy consumption optimization method based on multiple agents

Abstract

The invention discloses a building energy consumption optimization method based on multiple agents. The method comprises the following steps: establishing a plurality of independent agents which at least comprise a first agent and a second agent; the first intelligent agent is an energy consumption dynamic operation model taking an electric vehicle charging and discharging system as a controlled object, and the second intelligent agent is an energy consumption dynamic operation model taking an ice storage air conditioning system as a controlled object; establishing a Markov game model for the plurality of independent agents; and solving the Markov game model, and outputting an optimization strategy. According to the technical scheme provided by the embodiment of the invention, the intelligent agents are respectively established for the air conditioner and the electric automobile in the main energy consumption of the building, and more accurate energy consumption control and optimization are realized through cooperation among the multiple intelligent agents.

Description

Technical Field

[0001] The present invention relates to the technical field of energy optimization, and in particular to a building energy consumption optimization method based on multi-agents. Background Art

[0002] With global warming, the average temperature in my country is gradually rising, and the use of air-conditioning equipment is becoming more and more common, especially in summer. The air-conditioning load in large buildings such as shopping malls, office buildings, and hospitals has increased sharply. At the same time, the market share of electric vehicles continues to increase. The emergence of a large number of electric vehicle charging demands has caused the load on the power grid to increase sharply, which can easily lead to peak loads on the power grid and endanger the safe operation of the power grid.

[0003] In the existing technology, it is often necessary to use a control terminal to collect historical data, and then train the energy consumption model on the cloud server. After the training is completed, the energy consumption model is deployed to the edge computing unit to participate in real-time decision-making on energy scheduling of buildings.

[0004] However, this approach relies on historical data to train building energy consumption models, and usually cannot effectively consider the complex interactions of multiple users, multiple systems, and multiple environmental factors, resulting in poor optimization results. In addition, existing algorithms rely on historical data to train models and cannot adapt to changes in building operating conditions in real time, resulting in energy waste and unnecessary energy consumption. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a building energy consumption optimization method based on multi-agent to solve the problem that the existing technology cannot adapt to the changes in the building operation status in real time and cannot perform accurate energy consumption control and optimization in a dynamic environment.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a building energy consumption optimization method based on multi-agent, the method comprising:

[0007] Establishing a plurality of intelligent agents, including at least a first intelligent agent and a second intelligent agent;

[0008] The first intelligent agent is an energy consumption dynamic operation model with an electric vehicle charging and discharging system as a controlled object, and the second intelligent agent is an energy consumption dynamic operation model with an ice storage air conditioning system as a controlled object;

[0009] Establishing a Markov game model for the multiple agents;

[0010] The MADDPG algorithm is used to solve the Markov game model and output an optimization strategy.

[0011] Further, the Markov game model is represented by a five-element array (N, S, A, R, P);

[0012] Among them, N represents the set of intelligent agents; S represents the set of Markov states of intelligent agents; A represents the set of Markov actions of intelligent agents; R represents the set of Markov rewards of intelligent agents; and P represents the transition probability of the set of Markov states of intelligent agents.

[0013] Furthermore, the Markov state set of the agent includes the state space of all agents; and the Markov action set of the agent includes the action space of all agents.

[0014] Further, in the first intelligent agent, the grid demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery charge state are used as the first state space, and the electric vehicle charging and the electric vehicle discharging are used as the first action space;

[0015] In the second intelligent agent, the grid demand response, air-conditioning power, and indoor temperature are used as the second state space, and the ice-storage air-conditioning in air-conditioning mode, ice-storage air-conditioning in ice storage mode, and ice-storage air-conditioning in ice melting mode are used as the second action space.

[0016] Furthermore, in the Markov game model, a strategy network is constructed for each agent;

[0017] For any intelligent agent, the input of the policy network is the state space of the intelligent agent, and the output is the action space of the intelligent agent.

[0018] Furthermore, in the Markov game model, a reward network is jointly constructed for all agents;

[0019] The input of the reward network is the state space and action space of all agents, and the output is the global information of all agents.

[0020] Furthermore, in the Markov game model, a centralized value network is jointly constructed for all agents;

[0021] For any intelligent agent, the input of the centralized value network is the global information of all intelligent agents, and the expected value of any intelligent agent is calculated.

[0022] Furthermore, the construction process of the Markov game model is specifically as follows:

[0023] Get the state space of each agent, input the policy network corresponding to each agent, and get the action space corresponding to each agent;

[0024] Input the state space and action space of each agent into the reward network to obtain the global information of all agents;

[0025] The state space and action space of each agent and the global information of all agents are input into the centralized value network, and the centralized value network updates the policy network of each agent.

[0026] Furthermore, after the centralized value network updates the policy network of each agent, the Markov state set of the agent is updated according to the updated policy network and the transition probability of the Markov state set of the agent.

[0027] Furthermore, the solution process of the Markov game model is specifically as follows:

[0028] After the centralized value network obtains the state space and action space of each agent and the global information of all agents, the expected value of any agent is calculated;

[0029] The time difference error between the expected value of any of the agents and the target expected value is calculated. If the sum of the time difference errors meets the preset threshold, the policy network corresponding to each current agent is used as the optimization strategy; if the sum of the time difference errors meets the preset threshold, the centralized value network updates the policy network of each agent until the sum of the time difference errors meets the preset threshold.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. Create intelligent entities for air conditioners and electric vehicles, two of the main energy consumption components of a building, and improve the coordination and efficiency of energy consumption management through collaboration among multiple intelligent entities;

[0032] 2. Ability to dynamically adjust the policy network in real time to achieve more precise energy consumption control and optimization;

[0033] 3. It can comprehensively consider various environmental factors and user needs, improve the system's adaptability in dynamic environments, and realize grid demand response.

[0034] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1is a structural schematic diagram of a building energy consumption model provided by an embodiment of the present invention;

[0037] Figure 2 is a structural diagram of a Markov game model provided by an embodiment of the present invention;

[0038] Figure 3 is a structural schematic diagram of an ice storage air conditioning system provided by an embodiment of the present invention;

[0039] Figure 4 It is a structural diagram of another Markov game model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] Embodiment 1

[0043] Before introducing the specific embodiments of the present invention, an exemplary description of the building energy consumption model is first given. Figure 1 A schematic diagram of a building energy consumption model provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown in the figure, in the building energy consumption model, residences, enterprises, commercial places, etc. can receive electric energy transmitted by the power grid and distribute it to the ice storage air-conditioning system and other fixed loads in the building, and also to the electric vehicle system. The electric vehicle V2G technology provides an innovative energy management mode, allowing electric vehicles to communicate with the power grid in both directions. Electric vehicles can store electric energy in batteries during low-power consumption periods, and can feed the stored electric energy back to the power grid during peak demand periods.

[0044] This embodiment proposes a multi-agent-based building energy consumption optimization method, which can be applied to dynamically adjust the building energy consumption. The method can be executed by a multi-agent building energy consumption optimization system, and the multi-agent building energy consumption optimization system can be configured in a power grid system. The method includes:

[0045] Establishing a plurality of intelligent agents, including at least a first intelligent agent and a second intelligent agent;

[0046] The first agent is an energy consumption dynamic operation model with the electric vehicle charging and discharging system as the controlled object, and the second agent is an energy consumption dynamic operation model with the ice storage air conditioning system as the controlled object;

[0047] Establish Markov game models for multiple agents;

[0048] The MADDPG algorithm is used to solve the Markov game model and output the optimization strategy.

[0049] It should be noted that the main technical problem solved by the present invention is the optimization of building energy consumption, which involves two types of electricity loads that account for a relatively large proportion, including ice storage air conditioners and electric vehicle charging and discharging. While ensuring user comfort and satisfaction, it reduces building operating costs, achieves peak shaving and valley filling, and maintains safe operation of the power grid.

[0050] Among them, the intelligent agent refers to "intelligent agent reinforcement learning", which is a system that is in the environment and is part of this environment. It can sense the environment and perform corresponding actions at any time, and gradually establish its own activity plan to cope with environmental changes that may be sensed in the future.

[0051] The building energy consumption optimization problem based on multi-agent is a continuous decision-making problem under uncertain conditions. Therefore, a Markov game model can be constructed, and the ice storage air conditioner and electric vehicle charging and discharging process are established as two independent agents. The MADDPG algorithm is used to solve the problem.

[0052] The MADDPG algorithm, or Multi-Agent Deep Deterministic Policy Gradient, is a reinforcement learning algorithm used in a multi-agent environment. In a multi-agent environment, the behavior of each agent depends not only on the state of the environment, but also on the strategies of other agents. MADDPG improves learning effects and stability by using an independent input-output architecture for each agent and considering the strategy information of other agents during training. That is, in addition to its own information, each agent's input also includes additional information, such as the state and actions of other agents.

[0053] Figure 2 A structural diagram of a Markov game model is provided for the first embodiment of the present invention, combined with Figure 2 ,Markov game model interacts with the external environment, exemplary, Figure 2 Only the Markov game model under the cooperation of the first agent and the second agent is shown. It should be understood that the Markov game model also includes multiple agents in addition to the ice storage air conditioning system and the electric vehicle charging and discharging system.

[0054] The Markov game model is represented by a five-element array (N, S, A, R, P);

[0055] Among them, N represents the set of intelligent agents; S represents the set of Markov states of intelligent agents; A represents the set of Markov actions of intelligent agents; R represents the set of Markov rewards of intelligent agents; and P represents the transition probability of the set of Markov states of intelligent agents.

[0056] Combination Figure 2 , Figure 2 The set of agents includes the first agent and the second agent, the state set includes the state space o1 of the first agent in the environment and the state space o2 of the second agent in the environment, the action set includes the action space a1 fed back by the electric vehicle strategy network and the action space a2 fed back by the ice storage air conditioning strategy network, the reward set refers to the state space and action space of each agent collected uniformly by the reward network, and the transition probability refers to the state update of the current state set under the updated strategy.

[0057] Specifically, the Markov state set of the agent includes the state space of all agents; the Markov action set of the agent includes the action space of all agents.

[0058] It can be understood that the Markov states of the agent include: ice storage air conditioning power, indoor temperature, electric vehicle charging power, electric vehicle discharging power, electric vehicle battery capacity and electric vehicle battery charge state, grid demand response, etc.;

[0059] The Markov actions of the agent include: electric vehicle charging / discharging, ice storage air conditioner in air conditioning mode, ice storage air conditioner in ice storage mode, ice storage air conditioner in ice melting mode, etc. The state set S can be expressed as (o1,…,o N ), the action set A can be expressed as (a1,…,a N ).

[0060] Further, in the first intelligent agent, the grid demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery charge state are used as the first state space, and the electric vehicle charging and the electric vehicle discharging are used as the first action space;

[0061] In the second intelligent agent, the grid demand response, air-conditioning power, and indoor temperature are used as the second state space, and the ice-storage air-conditioning in air-conditioning mode, ice-storage air-conditioning in ice storage mode, and ice-storage air-conditioning in ice melting mode are used as the second action space.

[0062] The charging and discharging system of electric vehicles can store and supply electrical energy to ensure the normal operation of the vehicle. The basic structure of electric vehicle charging and discharging mainly includes two parts: the battery for energy storage and the battery management system.

[0063] The attenuation of electric vehicle batteries includes cycle attenuation and calendar attenuation. Cycle attenuation is usually related to the number of charge and discharge cycles. Frequent charge and discharge will accelerate the attenuation of the battery, resulting in a decrease in battery capacity and an increase in internal resistance. Calendar attenuation is usually related to the storage time of the battery. Batteries that are not used for a long time may experience significant performance degradation.

[0064] Accordingly, in the first agent, in addition to the charging and discharging power, the battery capacity and the battery charge state are also introduced as the state space of the first agent.

[0065] Figure 3 A schematic diagram of the structure of an ice storage air conditioning system provided by an embodiment of the present invention, combined with Figure 3 , the main structure of a large central ice storage air conditioning system includes a fan, an ice water pump, a chiller, an ice storage tank, an ice storage pump, a cooling water pump and a cooling fan. In order to meet the cooling load and provide a stable supply for large buildings, multiple refrigeration units are operated in parallel and connected to the public supply system, and the indoor temperature heat load is transferred to the outdoors by using a cold and heat exchange system. This architecture can effectively make the chiller and ice storage operations more flexible and reliable, as well as provide backup capacity, while requiring lower unit maintenance costs. Cold energy is stored in the form of sensible heat, and ice melts into water to release cold energy to provide the required cooling load.

[0066] Accordingly, in the second agent, in addition to the air-conditioning power, the indoor temperature is also introduced as the state space of the second agent.

[0067] Furthermore, in the Markov game model, a strategy network is constructed for each agent;

[0068] For any agent, the input of the policy network is the state space of the agent, and the output is the action space of the agent.

[0069] Combination Figure 2 , the policy network of each agent is a parameterized function, usually represented by a deep neural network, mapping a deterministic policy from the state space to the action space. For agent i, the input of the policy network is the state space o of agent i i The output is the action space a of the agent i i .

[0070] The function formula of the policy network is:

[0071]

[0072] Among them, a i refers to the action space of agent i, θ i refers to the policy network parameters, Refers to the policy network representation method, o i Refers to the state space of agent i.

[0073] The design of the policy network focuses on generating actions that are suitable for the current environment based on the independent observations of each agent. In the execution phase, each agent only needs to rely on its own policy network and observation information, ensuring the distributed execution characteristics of the algorithm.

[0074] For agent i, after receiving the state space in the environment, the policy network designs the corresponding action space for itself through the function formula of the policy network. For example, when the battery charge state SOC = 0 of the electric vehicle indicates that the battery is fully discharged, the electric vehicle is charged; when SOC = 1, it indicates that the battery is fully charged, the electric vehicle is discharged. The policy network is also used to receive the policy gradient representation of the centralized value network feedback and to adjust the policy network parameters θ i to update.

[0075] Furthermore, in the Markov game model, a reward network is jointly constructed for all agents;

[0076] The input of the reward network is the state space and action space of all agents, and the output is the global information of all agents.

[0077] Combination Figure 2 ,After each strategy network outputs the action space, the reward network collects the state space and action space of each agent to form the global information R(S, A), and sends it to the centralized value network.

[0078] Furthermore, in the Markov game model, a centralized value network is jointly constructed for all agents;

[0079] For any agent i, the input of the centralized value network is the global information of all agents, and the expected value of any agent i is calculated.

[0080] Among them, global information refers to the collection of state space and action space of all agents, combined with Figure 2In MADDPG, the value network is centralized and uses global information R(S, A) to evaluate the strategy of an agent. For agent i, the input of the value network is the global information R(S, A), and the output is the corresponding expected value Q i .

[0081] The function formula of the value network is:

[0082]

[0083] Among them, Q i refers to the expected value of agent i under the current strategy, φ i refers to the value network parameter, Refers to the value network representation method.

[0084] By using global information, the value network can more accurately assess the impact of an agent's actions on the entire system, thereby providing more effective feedback for policy optimization.

[0085] The value network of each agent uses not only its own state and action during training, but also the global state and action information of all agents. This centralized value network design allows agents to consider the behavior of other agents during training, thereby improving their collaborative capabilities. The policy network of each agent is mainly composed of an input layer, a hidden layer, and an output layer. The input layer receives local observations of the agent, the hidden layer extracts features through a multi-layer perceptron (MLP), and the output layer generates continuous actions for the agent.

[0086] Furthermore, the construction process of the Markov game model is as follows:

[0087] Get the state space of each agent, input the policy network corresponding to each agent, and get the action space corresponding to each agent;

[0088] Input the state space and action space of each agent into the reward network to obtain the global information of all agents;

[0089] The state space and action space of each agent and the global information of all agents are input into the centralized value network, and the centralized value network updates the policy network of each agent.

[0090] Combination Figure 2 The centralized value network first calculates the expected value Q of each agent. When the Q of each agent does not meet the preset conditions, the centralized value network sends the policy gradient optimization representation method to the policy network.

[0091] The function formula of gradient optimization is:

[0092]

[0093] in, refers to the policy gradient optimization representation method, θ i refers to the policy network parameters, E refers to the mathematical expectation, o i refers to the state space of agent i, a i refers to the action space of agent i, refers to the policy network representation method, Refers to the value network representation method.

[0094] The preset condition refers to: calculating the time difference error between the expected value of any agent i and the target expected value, and the sum of the time difference errors meets the preset threshold.

[0095] The function formula of timing difference error is:

[0096]

[0097] Among them, L(φ i ) is the error function, E refers to the mathematical expectation, r refers to the agent reward, γ refers to the discount factor, and y refers to the target value network value. refers to the target value network representation method, Refers to the target policy network representation method.

[0098] Furthermore, after the centralized value network updates the policy network of each agent, the Markov state set of the agent is updated according to the updated policy network and the transition probability of the Markov state set of the agent.

[0099] It should be noted that for each independent agent, when the agent selects an action space A to act on the environment, the state space S of the environment changes, and according to the transition probability P of the Markov state set, it enters the next state S', and generates a reward or penalty signal to feed back to the agent. The agent will select a new action space based on the obtained signal and the state space of the current environment until the iteration ends. Multi-agent reinforcement learning uses the idea of ​​centralized training decentralized execution (CTDE): According to the CTDE idea, the global state information of all agents can be used during model training to achieve better training results, and each agent executes independently in the decision-making stage and outputs actions only according to its own strategy.

[0100] Furthermore, the solution process of the Markov game model is specifically as follows:

[0101] After the centralized value network obtains the state space and action space of each agent and the global information of all agents, the expected value of any agent i is calculated;

[0102] The time difference error between the expected value of any agent i and the target expected value is calculated. If the sum of the time difference errors meets the preset threshold, the policy network corresponding to each current agent is used as the optimization strategy; if the sum of the time difference errors meets the preset threshold, the centralized value network updates the policy network of each agent until the sum of the time difference errors meets the preset threshold.

[0103] Specifically, when the sum of the temporal difference errors is less than a preset threshold, the expected value of each agent is close to the target expected value, satisfying the optimization strategy, and the strategy network at this time is used as the optimization strategy.

[0104] The entire Markov game model is in dynamic adjustment. The initial state space S passes through the policy network to obtain the corresponding action space A. The value network calculates the expected value Q of each intelligent agent through the current state space S and the action space A. If the sum of the time difference errors between the expected value Q of each intelligent agent and the target expected value is less than the preset threshold, the current policy network will be used as the optimization strategy; if the sum of the time difference errors between the expected value Q of each intelligent agent and the target expected value is greater than the preset threshold, the value network will provide gradient optimization for the policy network to update the policy network. At this time, the state space S will enter the next state space S' according to the updated policy network and the transition probability P, and perform the expected value calculation again until the sum of the time difference errors is less than the preset threshold.

[0105] It can be understood that: the strategy network of each agent is independent of each other. The strategy network uses the state space as the network input, outputs the action space according to the strategy of each agent, and improves the strategy according to the state-action value function Q fed back by the value network, that is, the time difference error is minimized by updating the network parameters. All agents share a value network. The value network Q uses the state and action of each agent as the network input, and outputs the state-action value function Q to evaluate the output of the strategy network. Through the cooperation of the strategy network and the value network, each agent can better specify the strategy and make corresponding decisions to achieve a mixed decision Nash equilibrium between each agent.

[0106] The technical solution of the embodiment of the present invention optimizes building energy consumption based on multi-agent reinforcement learning. Based on multi-agent reinforcement learning, intelligent agents are constructed for electric vehicles and ice storage air conditioners respectively, and both electric vehicles and ice storage air conditioners are regarded as flexible loads. It does not rely on historical data and has strong adaptability to different types of building environments. Through centralized training, distributed execution strategies are used to train multiple agents, and user satisfaction and comfort are considered while achieving grid demand response.

[0107] Embodiment 2

[0108] Figure 4This is a structural diagram of another Markov game model provided by the second embodiment of the present invention. This embodiment is further optimized based on the first embodiment. This embodiment proposes a building energy consumption optimization method based on multi-agents, which can be applied to dynamically adjust the building energy consumption. The method can be executed by a multi-agent building energy consumption optimization system, and the multi-agent building energy consumption optimization system can be configured in the power grid system. The method includes:

[0109] Establishing a plurality of intelligent agents, including at least a first intelligent agent and a second intelligent agent;

[0110] The first agent is an energy consumption dynamic operation model with the electric vehicle charging and discharging system as the controlled object, and the second agent is an energy consumption dynamic operation model with the ice storage air conditioning system as the controlled object;

[0111] Establish Markov game models for multiple agents;

[0112] The MAPPO algorithm is used to solve the Markov game model and output the optimization strategy.

[0113] Furthermore, the Markov game model is represented by a five-element array (N, S, A, R, P);

[0114] Among them, N represents the set of intelligent agents; S represents the set of Markov states of intelligent agents; A represents the set of Markov actions of intelligent agents; R represents the set of Markov rewards of intelligent agents; and P represents the transition probability of the set of Markov states of intelligent agents.

[0115] Furthermore, the Markov state set of the agent includes the state space of all agents; the Markov action set of the agent includes the action space of all agents.

[0116] Further, in the first intelligent agent, the grid demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery charge state are used as the first state space, and the electric vehicle charging and the electric vehicle discharging are used as the first action space;

[0117] In the second intelligent agent, the grid demand response, air-conditioning power, and indoor temperature are used as the second state space, and the ice-storage air-conditioning in air-conditioning mode, ice-storage air-conditioning in ice storage mode, and ice-storage air-conditioning in ice melting mode are used as the second action space.

[0118] Optionally, in a Markov game model, a policy network is constructed for each agent separately.

[0119] Optionally, in a Markov game model, construct a target network and a centralized value network.

[0120] Combination Figure 4 ,The Markov game model includes multiple agents, the corresponding ,strategy networks of the agents, the target network and the centralized value network.

[0121] Furthermore, for agent i, the input of the policy network is the state space o of the agent i The output is the action space a of the agent i The formula is:

[0122]

[0123] Among them, a i refers to the action space of agent i, θ i refers to the policy network parameters, Refers to the policy network representation method, o i Refers to the state space of agent i.

[0124] Furthermore, in a centralized value network, the function formula of the value network is:

[0125]

[0126] Among them, V refers to the expected value of agent i under the current strategy, φ i refers to the value network parameter, Refers to the value network representation method.

[0127] Furthermore, in the target network, for any agent i, the objective function is calculated:

[0128]

[0129]

[0130] in, refers to the target strategy of agent i, refers to the current strategy of agent i, refers to the old strategy of agent i, r t (θ i ) refers to the strategy update ratio of agent i, o i refers to the current state space of agent i, a i refers to the action space selected by the policy network of agent i, Refers to the estimated value of the advantage function, ∈ refers to the clipping threshold, which is used to control the amplitude of the policy update, and clip refers to the clipping function, which is used to replace all numbers less than min with min, all numbers greater than max with max, and the numbers between [min, max] remain unchanged.

[0131] Furthermore, the formula of the advantage function is:

[0132]

[0133] δ t =r t +γV(S t+1 )-V(S t )

[0134] in, refers to the estimated value of the advantage function, δ t refers to the time difference error, γ refers to the discount factor, λ refers to the weight parameter, S t Refers to the current state set, S t+1 refers to the state set at the next moment, r t Refers to the policy update ratio of agent i.

[0135] Furthermore, in the target network, for any agent i, the loss function is calculated:

[0136]

[0137] R t =r t +γV i (S t+1 ; φ i ′)

[0138] Among them, L(φ i ) is the error function, E is the mathematical expectation, φ i is the value network parameter, φ i ` is the target value network parameter, R t is the cumulative return from the current time t to the future.

[0139] Combination Figure 4 The embodiment of the present invention also provides an algorithm solution process of a Markov game model, which is specifically:

[0140] S1. Initialization: Initialize the policy network for each agent and value network And initialize the corresponding target network.

[0141] S2, Interaction and experience collection: Each agent interacts with the environment according to the current strategy and stores information such as state, action, reward, next state, value, etc. at each time step.

[0142] S3. Calculate rewards and advantages: By calculating the reward R of each agent t and advantage function

[0143] S4. Update the value network: Update the value network parameters according to the value network loss function to minimize the mean square error.

[0144] S5. Update policy network: According to the pruned PPO objective function, the policy network parameters of each agent are updated using the policy gradient method.

[0145] S6. Soft update target network: Use the soft update mechanism to gradually update the parameters of the target network.

[0146] S7. Repeat: Repeat the above steps until the agent strategy converges.

[0147] The technical solution of the embodiment of the present invention optimizes building energy consumption based on multi-agent reinforcement learning. Based on multi-agent reinforcement learning, intelligent agents are constructed for electric vehicles and ice storage air conditioners respectively, and both electric vehicles and ice storage air conditioners are regarded as flexible loads. It does not rely on historical data and has strong adaptability to different types of building environments. Through centralized training, distributed execution strategies are used to train multiple agents, and user satisfaction and comfort are considered while achieving grid demand response.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A building energy consumption optimization method based on multi-agent, characterized in that: include: Establishing a plurality of intelligent agents, including at least a first intelligent agent and a second intelligent agent; The first intelligent agent is an energy consumption dynamic operation model with an electric vehicle charging and discharging system as a controlled object, and the second intelligent agent is an energy consumption dynamic operation model with an ice storage air conditioning system as a controlled object; Establishing a Markov game model for the multiple agents; The MADDPG algorithm is used to solve the Markov game model and output an optimization strategy.

2. The method according to claim 1, characterized in that The Markov game model is represented by a five-element array (N, S, A, R, P); Among them, N represents the set of intelligent agents; S represents the set of Markov states of intelligent agents; A represents the set of Markov actions of intelligent agents; R represents the set of Markov rewards of intelligent agents; and P represents the transition probability of the set of Markov states of intelligent agents.

3. The method according to claim 2, characterized in that The Markov state set of the agent includes the state space of all agents; the Markov action set of the agent includes the action space of all agents.

4. The method according to claim 3, characterized in that In the first intelligent agent, the grid demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery charge state are used as the first state space, and the electric vehicle charging and the electric vehicle discharging are used as the first action space; In the second intelligent agent, the grid demand response, air-conditioning power, and indoor temperature are used as the second state space, and the ice-storage air-conditioning in air-conditioning mode, ice-storage air-conditioning in ice storage mode, and ice-storage air-conditioning in ice melting mode are used as the second action space.

5. The method according to claim 3, characterized in that: In the Markov game model, a strategy network is constructed for each agent; For any intelligent agent, the input of the policy network is the state space of the intelligent agent, and the output is the action space of the intelligent agent.

6. The method according to claim 3, characterized in that In the Markov game model, a reward network is jointly constructed for all agents; The input of the reward network is the state space and action space of all agents, and the output is the global information of all agents.

7. The method according to claim 6, characterized in that In the Markov game model, a centralized value network is jointly constructed for all agents; For any intelligent agent, the input of the centralized value network is the global information of all intelligent agents, and the expected value of any intelligent agent is calculated.

8. The method according to claim 5 or 7, characterized in that: The construction process of the Markov game model is specifically as follows: Get the state space of each agent, input the policy network corresponding to each agent, and get the action space corresponding to each agent; Input the state space and action space of each agent into the reward network to obtain the global information of all agents; The state space and action space of each agent and the global information of all agents are input into the centralized value network, and the centralized value network updates the policy network of each agent.

9. The method according to claim 8, characterized in that After the centralized value network updates the policy network of each agent, the Markov state set of the agent is updated according to the updated policy network and the transition probability of the Markov state set of the agent.

10. The method according to claim 8 or 9, characterized in that: The solution process of the Markov game model is specifically as follows: After the centralized value network obtains the state space and action space of each agent and the global information of all agents, the expected value of any agent is calculated; Calculate the time difference error between the expected value of any of the intelligent agents and the target expected value, and if the sum of the time difference errors meets a preset threshold, use the strategy network corresponding to each current intelligent agent as the optimization strategy; If the sum of the temporal difference errors meets a preset threshold, the centralized value network updates the strategy network of each agent until the sum of the temporal difference errors meets the preset threshold.

Images