A multi-agent based building energy consumption optimization method

Abstract

The application discloses a kind of building energy consumption optimization methods based on multi-agent.The method comprises: establishing multiple independent agents, at least including first agent and second agent;The first agent is the energy consumption dynamic operation model of electric vehicle charging and discharging system as controlled object, and the second agent is the energy consumption dynamic operation model of ice storage air conditioning system as controlled object;A Markov game model is established for the multiple independent agents;The Markov game model is solved, and the optimization strategy is output.The technical scheme in the embodiment of the application establishes agent for air conditioner and electric vehicle in main energy consumption of building respectively, and realizes more accurate energy consumption control and optimization through the cooperation between multi-agent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of energy optimization, and particularly relates to a building energy consumption optimization method based on multiple agents. BACKGROUND

[0002] With global warming, the average temperature in China is gradually rising, and the application of air conditioning equipment is gradually popular, especially in summer. In large buildings such as shopping malls, office buildings, and hospitals, the air conditioning load increases sharply. At the same time, the market share of electric vehicles is continuously increasing, and the demand for charging of a large number of electric vehicles has caused the power grid load to increase sharply, which easily leads to sharp peak load of the power grid and endangers the safe operation of the power grid.

[0003] In the prior art, historical data is often collected by using a control terminal, and then the energy consumption model is trained on the cloud server side. After the training is completed, the energy consumption model is deployed to the edge computing unit to participate in the real-time decision of building energy scheduling.

[0004] However, this method relies on historical data to train the building energy consumption model, and usually cannot effectively consider the complex interaction of multiple users, multiple systems, and multiple environmental factors, resulting in poor optimization effect. Moreover, the existing algorithm relies on historical data to train the model, and cannot adapt to the changes of the building operation state in real time, causing energy waste and unnecessary energy consumption. SUMMARY

[0005] In view of this, the present application aims to provide a building energy consumption optimization method based on multiple agents to solve the problem that the prior art cannot adapt to the changes of the building operation state in real time and cannot accurately control and optimize energy consumption in a dynamic environment.

[0006] To achieve the above-mentioned application purposes, the present application provides a building energy consumption optimization method based on multiple agents, which comprises:

[0007] establishing multiple agents, at least including a first agent and a second agent;

[0008] The first agent is an energy consumption dynamic operation model taking the electric vehicle charging and discharging system as a controlled object, and the second agent is an energy consumption dynamic operation model taking the ice storage air conditioning system as a controlled object;

[0009] establishing a Markov game model for the multiple agents;

[0010] solving the Markov game model by using the MADDPG algorithm to output an optimization strategy.

[0011] Further, the Markov game model is represented by a five-element array (N, S, A, R, P);

[0012] wherein, N represents a set of agents; S represents a Markov state set of agents; A represents a Markov action set of agents; R represents a Markov reward set of agents; P represents a transition probability of the Markov state set of agents.

[0013] Further, the Markov state set of agents includes a state space of all agents; the Markov action set of agents includes an action space of all agents.

[0014] Further, in the first agent, the grid demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery state of charge are taken as the first state space, and the electric vehicle charging and the electric vehicle discharging are taken as the first action space.

[0015] In the second agent, the grid demand response, the air conditioner power, and the indoor temperature are taken as the second state space, and the ice storage air conditioner in the air conditioning mode, the ice storage air conditioner in the ice storage mode, and the ice storage air conditioner in the ice melting mode are taken as the second action space.

[0016] Further, in the Markov game model, a strategy network is constructed for each agent respectively;

[0017] For any agent, the input of the strategy network is the state space of the any agent, and the output is the action space of the any agent.

[0018] Further, in the Markov game model, a reward network is constructed for all agents;

[0019] The input of the reward network is the state space and the action space of all agents, and the output is the global information of all agents.

[0020] Further, in the Markov game model, a centralized value network is constructed for all agents;

[0021] For any agent, the input of the centralized value network is the global information of all agents, and the expected value of the any agent is calculated.

[0022] Further, the construction process of the Markov game model is specifically as follows:

[0023] The state space of each agent is obtained, and the corresponding strategy network of each agent is input respectively to obtain the corresponding action space of each agent;

[0024] The state space and the action space of each agent are input into the reward network to obtain the global information of all agents;

[0025] Input the state space and action space of each agent and the global information of all agents into the centralized value network, and the centralized value network updates the policy network of each agent.

[0026] Further, after the centralized value network updates the policy network of each agent, the Markov state set of the agent is updated according to the updated policy network and the transition probability of the Markov state set of the agent.

[0027] Further, the solving process of the Markov game model is specifically:

[0028] After the centralized value network obtains the state space and action space of each agent and the global information of all agents, the expected value of any agent is calculated.

[0029] The time sequence difference error of the expected value of any agent and the target expected value is calculated, if the sum of the time sequence difference error meets the preset threshold, the current policy network corresponding to each agent is taken as the optimized policy, if the sum of the time sequence difference error meets the preset threshold, the centralized value network updates the policy network of each agent until the sum of the time sequence difference error meets the preset threshold.

[0030] Compared with the prior art, the beneficial effects of the present application are:

[0031] 1. The air conditioner and electric vehicle in the main energy consumption of the building are respectively established as agents, and the coordination and efficiency of energy consumption management are improved through the cooperation between multiple agents.

[0032] 2. The policy network can be dynamically adjusted in real time to realize more accurate energy consumption control and optimization.

[0033] 3. Multiple environmental factors and user demands can be comprehensively considered to improve the adaptability of the system in a dynamic environment and realize power grid demand response.

[0034] It should be understood that the contents described in this part are not intended to identify the key or important features of the embodiments of the present application, nor are they used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only preferred embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0036] Figure 1is a structural schematic diagram of a building energy consumption model provided by an embodiment of the present application;

[0037] Figure 2 is a structural diagram of a Markov game model provided by an embodiment of the present application;

[0038] Figure 3 is a structural schematic diagram of an ice storage air conditioning system provided by an embodiment of the present application;

[0039] Figure 4 is a structural diagram of another Markov game model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiment of the present application will be described clearly and completely below by combining the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, not all. Based on the embodiment in the present application, all other embodiments obtained by the ordinary skilled in the art without creative labor should belong to the scope of the present application.

[0041] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0042] Embodiment one

[0043] Before introducing the specific embodiments of the present application, the building energy consumption model is exemplarily described. Figure 1 A structural schematic diagram of a building energy consumption model provided by an embodiment of the present application is shown in Figure 1 As shown, in the building energy consumption model, residences, enterprises, commercial sites and the like can receive the electric energy transmitted by the power grid and distribute to the ice storage air conditioning system and other fixed loads in the building, and also distribute to the electric vehicle system, while the electric vehicle V2G technology provides an innovative energy management mode, allowing the electric vehicle to communicate with the power grid in both directions. The electric vehicle can store electric energy in the battery during the low electricity consumption period, and can feed the stored electric energy back to the power grid during the demand peak period.

[0044] The embodiment provides a building energy consumption optimization method based on multiple agents, which is applicable to dynamically adjusting building energy consumption, and can be executed by a building energy consumption optimization system based on multiple agents, which can be configured in a power grid system. The method comprises the following steps:

[0045] A plurality of agents are established, including at least a first agent and a second agent;

[0046] The first agent is an energy consumption dynamic operation model taking an electric vehicle charging and discharging system as a controlled object, and the second agent is an energy consumption dynamic operation model taking an ice storage air conditioning system as a controlled object;

[0047] A Markov game model is established for the plurality of agents;

[0048] An MADDPG algorithm is used to solve the Markov game model, and an optimization strategy is output.

[0049] It should be noted that the technical problem solved by the present application is to solve the building energy consumption optimization problem in which two types of power consumption loads, including an ice storage air conditioning system and an electric vehicle charging and discharging system, account for a large proportion, to reduce building operation cost while ensuring user comfort and satisfaction, to achieve peak load shifting, and to maintain safe operation of the power grid.

[0050] The agent refers to "agent reinforcement learning", which is a system in an environment and a part of the environment, can sense the environment at any time and perform corresponding actions, and gradually establishes its own activity plan to cope with possible changes in the environment in the future.

[0051] The building energy consumption optimization problem based on multiple agents is a continuous decision problem under uncertain conditions, and therefore a Markov game model can be constructed, the ice storage air conditioning and the electric vehicle charging and discharging process are established as two independent agents, and an MADDPG algorithm is used to solve the problem.

[0052] The MADDPG algorithm, i.e., the multi-agent deep deterministic policy gradient, is a reinforcement learning algorithm used in a multi-agent environment. In the multi-agent environment, the behavior of each agent depends not only on the state of the environment, but also on the strategies of other agents. The MADDPG improves the learning effect and stability by using an independent input-output architecture for each agent and considering the strategy information of other agents during the training process. That is, the input of each agent includes not only its own information, but also additional information such as the state and action of other agents.

[0053] Figure 2 A structural diagram of the Markov game model is provided for the first embodiment of the present application, which is combined withFigure 2 The Markov game model interacts with the external environment, and an example is shown in FIG. 1. Figure 2 Only the Markov game model under the cooperation of the first agent and the second agent is shown, and it should be understood that the Markov game model also includes multiple agents in addition to the ice storage air conditioning system and the electric vehicle charging and discharging system.

[0054] The Markov game model is represented by a five-tuple (N, S, A, R, P).

[0055] N represents the set of agents; S represents the Markov state set of the agent; A represents the Markov action set of the agent; R represents the Markov reward set of the agent; and P represents the transition probability of the Markov state set of the agent.

[0056] In combination with Figure 2 , Figure 2 The set of agents in the environment includes the first agent and the second agent, the state set includes the state space o1 of the first agent in the environment and the state space o2 of the second agent in the environment, the action set includes the action space a1 fed back by the electric vehicle strategy network and the action space a2 fed back by the ice storage air conditioning strategy network, the reward set refers to the collection of the state space and the action space of each agent by the reward network, and the transition probability refers to the state update of the current state set under the updated strategy.

[0057] Specifically, the Markov state set of the agent includes the state space of all agents; and the Markov action set of the agent includes the action space of all agents.

[0058] It can be understood that the Markov state of the agent includes: ice storage air conditioning power, indoor temperature, electric vehicle charging power, electric vehicle discharging power, electric vehicle battery capacity and electric vehicle battery state of charge, and power demand response, etc.

[0059] The Markov action of the agent includes: electric vehicle charging / discharging, ice storage air conditioning in air conditioning mode, ice storage air conditioning in ice storage mode, ice storage air conditioning in ice melting mode, etc. The state set S can be represented as (o1, …, o N ), and the action set A can be represented as (a1, …, a N ).

[0060] Further, in the first agent, the power demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery state of charge are taken as the first state space, and the electric vehicle charging and the electric vehicle discharging are taken as the first action space.

[0061] In the second agent, the grid demand response, air conditioner power and indoor temperature are taken as the second state space, and the ice storage air conditioner in the air conditioning mode, the ice storage air conditioner in the ice storage mode and the ice storage air conditioner in the ice melting mode are taken as the second action space.

[0062] The charging and discharging system of the electric vehicle can store and supply electric energy, and ensure the normal operation of the vehicle.

[0063] The attenuation of the electric vehicle battery includes cycle attenuation and calendar attenuation, the cycle attenuation is usually related to the number of charging and discharging, and frequent charging and discharging can accelerate the attenuation of the battery, resulting in the decrease of the battery capacity and the increase of the internal resistance, and the calendar attenuation is usually related to the storage time of the battery, and the battery that is not used for a long time can experience significant performance degradation.

[0064] Therefore, in the first agent, in addition to the charging and discharging power, the battery capacity and the battery state of charge are introduced as the state space of the first agent.

[0065] Figure 3 A structural schematic diagram of the ice storage air conditioning system provided by the embodiment of the application is provided in combination with Figure 3 The main structure of the large central ice storage air conditioning system includes a fan, an ice water pump, a cold water unit, an ice storage tank, an ice storage pump, a cooling water pump and a cooling fan. In order to meet the cooling load and provide stable supply for large buildings, multiple refrigeration units are connected to the public supply system in parallel operation, and the indoor temperature heat load is transferred to the outdoor by using a cold heat exchange system. This architecture can effectively make the cooler and ice storage operation more flexible and reliable, and provide backup capacity, while requiring lower unit maintenance costs. The cold energy is stored in the form of sensible heat, and the ice melts into water to release cold energy to provide the required cooling load.

[0066] Therefore, in the second agent, in addition to the air conditioner power, the indoor temperature is introduced as the state space of the second agent.

[0067] Further, in the Markov game model, a strategy network is constructed for each agent respectively.

[0068] For any agent, the input of the strategy network is the state space of the agent, and the output is the action space of the agent.

[0069] In combination with Figure 2 The strategy network of each agent is a parameterized function, usually represented by a deep neural network, which maps a deterministic strategy from the state space to the action space. For agent i, the input of the strategy network is the state space of the agent i i The output is the action space of the agent i i .

[0070] The function formula of the policy network is:

[0071]

[0072] Wherein, a i is the action space of the agent i, θ i is the policy network parameter, is the policy network representation method, o i is the state space of the agent i.

[0073] The design focus of the policy network is to generate actions suitable for the current environment according to the independent observation of each agent. In the execution phase, each agent only needs to rely on its own policy network and observation information, which ensures the distributed execution characteristics of the algorithm.

[0074] For the agent i, the policy network receives the state space in the environment, and designs the corresponding action space for itself through the function formula of the policy network, such as: when the state of charge SOC of the electric vehicle battery is 0, it means that the battery is completely discharged, and the electric vehicle is charged; when SOC = 1, it means that the battery is fully charged, and the electric vehicle is discharged. The policy network is also used to receive the policy gradient representation method of the centralized value network feedback, and update the policy network parameter θ i .

[0075] Further, in the Markov game model, a reward network is constructed for all agents;

[0076] The input of the reward network is the state space and action space of all agents, and the output is the global information of all agents.

[0077] In combination Figure 2 , after the output of the action space of each policy network, the reward network collects the state space and action space of each agent to form the global information R(S, A), and sends it to the centralized value network.

[0078] Further, in the Markov game model, a centralized value network is constructed for all agents;

[0079] For any agent i, the input of the centralized value network is the global information of all agents, and the expected value of any agent i is calculated.

[0080] Wherein, the global information refers to the set of state space and action space of all agents, in combination Figure 2In MADDPG, the value network is centralized, using global information R(S, A) to evaluate the policy of an agent. For agent i, the input of the value network is global information R(S, A), and the output is the corresponding expected value Q i .

[0081] The function formula of the value network is:

[0082]

[0083] Where Q i is the expected value of agent i under the current policy, φ i is the value network parameter, is the value network representation method.

[0084] By using global information, the value network can more accurately evaluate the impact of an agent's action on the entire system, thus providing more effective feedback for policy optimization.

[0085] Each agent's value network uses not only its own state and action, but also the global state and action information of all agents during training. This centralized value network design allows agents to consider the behavior of other agents during training, thus improving collaboration. Each agent's policy network is mainly composed of an input layer, a hidden layer, and an output layer. The input layer receives the local observation of the agent, the hidden layer extracts features through a multi-layer perceptron (MLP), and the output layer generates the continuous action of the agent.

[0086] Further, the construction process of the Markov game model is as follows:

[0087] Obtain the state space of each agent, input the corresponding policy network of each agent respectively, and obtain the action space corresponding to each agent;

[0088] Input the state space and action space of each agent into the reward network to obtain the global information of all agents;

[0089] Input the state space and action space of each agent and the global information of all agents into the centralized value network, and update the policy network of each agent.

[0090] In combination with Figure 2 , the centralized value network first calculates the expected value Q of each agent. When the Q of each agent does not meet the preset condition, the centralized value network sends the policy gradient optimization representation method to the policy network

[0091] The function formula of gradient optimization is:

[0092]

[0093] wherein, denotes a policy gradient optimization representation method, and i denotes a policy network parameter, and E denotes a mathematical expectation, and i denotes a state space of the agent i, and i denotes an action space of the agent i, denotes a policy network representation method, denotes a value network representation method.

[0094] wherein, the preset condition refers to: calculating a time difference error of an expected value of any agent i and a target expected value, and a sum of the time difference errors satisfies a preset threshold.

[0095] A function formula of the time difference error is:

[0096]

[0097] wherein, L(φ i ) is an error function, E denotes a mathematical expectation, r denotes an agent reward, and y denotes a target value network value, denotes a target value network representation method, denotes a target policy network representation method.

[0098] Further, after the centralized value network updates the policy network of each agent, the Markov state set of the agent is updated according to the updated policy network and the transition probability of the Markov state set of the agent.

[0099] It should be noted that, for each independent agent, when the agent selects an action space A to act on the environment, the state space S of the environment changes, and according to the transition probability P of the Markov state set, it enters the next time state S', and at the same time, a reward or punishment signal is fed back to the agent, and the agent selects a new action space according to the obtained signal and the state space of the current environment until the iteration ends. Multi-agent reinforcement learning uses the idea paradigm of centralized training and decentralized execution (CTDE): according to the CTDE idea, the global state information of all agents can be used during model training to achieve better training effect, and each agent is independent during the decision-making stage and only outputs actions according to its own strategy.

[0100] Further, the solving process of the Markov game model is specifically:

[0101] After the centralized value network obtains the state space and the action space of each agent and the global information of all agents, the expected value of any agent i is calculated.

[0102] The time difference error of the expected value of any agent i and the target expected value is calculated, and if the sum of the time difference errors meets a preset threshold, the current strategy network corresponding to each agent is taken as an optimized strategy; if the sum of the time difference errors meets the preset threshold, the centralized value network updates the strategy network of each agent until the sum of the time difference errors meets the preset threshold.

[0103] Specifically, when the sum of the time difference errors is less than the preset threshold, the expected value of each agent is close to the target expected value, which meets the optimized strategy, and the strategy network at this time is taken as the optimized strategy.

[0104] The entire Markov game model is in dynamic adjustment, the initial state space S passes through the strategy network to obtain the corresponding action space A, the value network calculates the expected value Q of each agent through the current state space S and the action space A, if the sum of the time difference errors of the expected value Q of each agent and the target expected value is less than the preset threshold, the current strategy network is taken as the optimized strategy; if the sum of the time difference errors of the expected value Q of each agent and the target expected value is greater than the preset threshold, the value network will provide gradient optimization for the strategy network to update the strategy network, at this time, the state space S will enter the next time state space S' according to the updated strategy network and the transition probability P, and the expected value calculation is performed again until the sum of the time difference errors is less than the preset threshold.

[0105] It can be understood that: the strategy network of each agent is independent, the strategy network takes the state space as the network input, outputs the action space according to the strategy of each agent, and improves the strategy according to the state-action value function Q fed back by the value network, that is, the network parameters are updated to minimize the time difference error. All agents share a value network, the value network Q takes the state and action of each agent as the network input, outputs the state-action value function Q to evaluate the output of the strategy network, and through the cooperation of the strategy network and the value network, each agent can better specify the strategy and make corresponding decisions to achieve the mixed decision Nash equilibrium among the agents.

[0106] The technical scheme of the embodiment of the application performs building energy consumption optimization based on multi-agent reinforcement learning, constructs agents for electric vehicles and ice storage air conditioners based on multi-agent reinforcement learning, regards the electric vehicles and the ice storage air conditioners as flexible loads, and is not dependent on historical data, and is highly adaptable to different types of building environments. Through centralized training and distributed execution strategy, the multi-agent is trained, the user satisfaction and comfort are considered while the power grid demand response is realized.

[0107] Embodiment two

[0108] Figure 4As shown in the structural diagram of another Markov game model provided in Embodiment Two of the present application, the present embodiment is further optimized on the basis of Embodiment One. The present embodiment provides a building energy consumption optimization method based on multiple agents, which is applicable to dynamically adjusting the building energy consumption. The method can be executed by a building energy consumption optimization system of multiple agents, and the building energy consumption optimization system of multiple agents can be configured in a power grid system. The method comprises the following steps:

[0109] establishing multiple agents, including at least a first agent and a second agent;

[0110] The first agent is an energy consumption dynamic operation model taking an electric vehicle charging and discharging system as a controlled object, and the second agent is an energy consumption dynamic operation model taking an ice-storage air conditioning system as a controlled object.

[0111] establishing a Markov game model for the multiple agents;

[0112] solving the Markov game model by using a MAPPO algorithm, and outputting an optimization strategy.

[0113] Further, the Markov game model is represented by a five-element array (N, S, A, R, P).

[0114] N represents a set of agents; S represents a Markov state set of agents; A represents a Markov action set of agents; R represents a Markov reward set of agents; and P represents a transition probability of the Markov state set of agents.

[0115] Further, the Markov state set of agents includes a state space of all agents, and the Markov action set of agents includes an action space of all agents.

[0116] Further, in the first agent, the power grid demand response, the electric vehicle charging power, the electric vehicle discharging power, the electric vehicle battery capacity, and the electric vehicle battery charge state are taken as a first state space, and the electric vehicle charging and the electric vehicle discharging are taken as a first action space.

[0117] In the second agent, the power grid demand response, the air conditioning power, and the indoor temperature are taken as a second state space, and the ice-storage air conditioning in the air conditioning mode, the ice-storage air conditioning in the ice storage mode, and the ice-storage air conditioning in the ice melting mode are taken as a second action space.

[0118] Optionally, in the Markov game model, a strategy network is constructed for each agent.

[0119] Optionally, in the Markov game model, a target network and a centralized value network are constructed.

[0120] In combination withFigure 4 The Markov game model includes a plurality of agents, a policy network corresponding to the agents, a target network, and a centralized value network.

[0121] Further, for an agent i, an input of the policy network is a state space o i An output is an action space a i of the agent i. The formula is:

[0122]

[0123] Wherein, a i denotes an action space of the agent i, θ i denotes a policy network parameter, denotes a policy network representation method, o i denotes a state space of the agent i.

[0124] Further, in the centralized value network, a function formula of the value network is:

[0125]

[0126] Wherein, V denotes an expected value of the agent i under a current policy, φ i denotes a value network parameter, denotes a value network representation method.

[0127] Further, in the target network, for any agent i, a target function is calculated:

[0128]

[0129]

[0130] Wherein, denotes a target policy of the agent i, denotes a current policy of the agent i, denotes an old policy of the agent i, r t (θ i ) denotes a policy update rate of the agent i, o i denotes a current state space of the agent i, a i denotes an action space selected by the policy network of the agent i, denotes an estimated value of the advantage function, ∈ denotes a clipping threshold, used to control the amplitude of policy update, clip denotes a clipping function, used to replace all numbers less than min to min, all numbers greater than max to max, and numbers between [min, max] unchanged.

[0131] Further, a formula of the advantage function is:

[0132]

[0133] delta t = r t + gamma * V t+1 (S t t+1 )

[0134] where, delta t is the estimate of the advantage function, gamma is the time difference error, lambda is the weight parameter, S t is the current state set, S t+1 is the next time state set, r t is the policy update rate of the agent i.

[0135] Further, in the target network, for any agent i, the loss function is calculated:

[0136]

[0137] R t = r t + gamma * V i (S t+1 t+1 ; phi i ')

[0138] where, L(phi i ) is the error function, E is the mathematical expectation, phi i is the value network parameter, phi i ` is the target value network parameter, R t is the cumulative return from the current time t to the future.

[0139] In combination Figure 4 , the embodiment of the application also provides an algorithm solving process of a Markov game model, specifically:

[0140] S1, initialization: initialize the policy network and the value network for each agent, and initialize the corresponding target network.

[0141] S2, interaction and experience collection: each agent interacts with the environment according to the current policy, and stores the state, action, reward, next state, value and other information of each time step.

[0142] S3, calculate the return and advantage: calculate the return R t and the advantage function

[0143] S4, update the value network: update the value network parameter according to the value network loss function, and minimize the mean square error.

[0144] S5, updating the policy network: according to the clipped PPO objective function, the policy network parameters of each agent are updated using the policy gradient method.

[0145] S6, soft updating the target network: the parameters of the target network are gradually updated using a soft updating mechanism.

[0146] S7, repeat: the above steps are cycled until the agent policy converges.

[0147] The technical scheme of the embodiment of the present application optimizes building energy consumption based on multi-agent reinforcement learning. Based on multi-agent reinforcement learning, agents are constructed for electric vehicles and ice storage air conditioners. Electric vehicles and ice storage air conditioners are both regarded as flexible loads, and are not dependent on historical data, and are adaptable to different types of building environments. Through centralized training and distributed execution of the policy, the multi-agent is trained, and user satisfaction and comfort are considered while achieving grid demand response.

[0148] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A building energy consumption optimization method based on multi-agent systems, characterized in that, include: Establish multiple intelligent agents, including at least a first intelligent agent and a second intelligent agent; The first intelligent agent is an energy consumption dynamic operation model with the electric vehicle charging and discharging system as the controlled object, and the second intelligent agent is an energy consumption dynamic operation model with the ice storage air conditioning system as the controlled object. Establish a Markov game model for the multiple agents; The MADDPG algorithm is used to solve the Markov game model and output the optimization strategy. The Markov game model is represented by a quintuple (N, S, A, R, P); Where N represents the set of agents; S represents the set of Markov states of agents; A represents the set of Markov actions of agents; R represents the set of Markov rewards of agents; and P represents the transition probability of the set of Markov states of agents. The Markov state set of the agent includes the state space of all agents; the Markov action set of the agent includes the action space of all agents. In the Markov game model, a policy network is constructed for each agent. For any agent, the input to the policy network is the state space of that agent, and the output is the action space of that agent. In the Markov game model, a reward network is jointly constructed for all agents; The input to the reward network is the state space and action space of all agents, and the output is the global information of all agents; In the Markov game model, a centralized value network is jointly constructed for all agents. For any agent, the input to the centralized value network is the global information of all agents, and the expected value of the agent is calculated. Alternatively, in the Markov game model, a policy network can be constructed for each agent. In the Markov game model, a target network and a centralized value network are constructed. For any agent, the input to the policy network is the state space of that agent, and the output is the action space of that agent. In the centralized value network described above, the functional formula for the value network is: ； Where V refers to the expected value of agent i under the current policy. Refers to value network parameters. Refers to the value network representation method; In the target network, for any agent i, the objective function is calculated: ； ； in, The target policy of agent i. Refers to the current policy of agent i. Refers to agent i's old policy. The policy update rate of agent i. The current state space of agent i. The action space selected by the policy network of agent i. The estimated value of the dominance function. The clipping threshold controls the magnitude of policy updates. The clipping function replaces all numbers less than min with min, all numbers greater than max with max, and numbers between [min, max] remain unchanged.

2. The method according to claim 1, characterized in that, In the first intelligent agent, the first state space is defined by grid demand response, electric vehicle charging power, electric vehicle discharging power, electric vehicle battery capacity, and electric vehicle battery charge state, and the first action space is defined by electric vehicle charging and electric vehicle discharging. In the second intelligent agent, the grid demand response, air conditioning power, and indoor temperature are used as the second state space, and the ice storage air conditioner is in air conditioning mode, ice storage air conditioner is in ice storage mode, and ice storage air conditioner is in ice melting mode as the second action space.

3. The method according to claim 1, characterized in that, The construction process of the Markov game model is as follows: Obtain the state space of each agent, input it into the policy network corresponding to each agent, and obtain the action space of each agent. Input the state space and action space of each agent into the reward network to obtain global information of all agents; The state space and action space of each agent, as well as the global information of all agents, are input into a centralized value network, which then updates the policy network of each agent.

4. The method according to claim 3, characterized in that, After the centralized value network updates the policy network of each agent, the agent's Markov state set is updated according to the updated policy network and the transition probabilities of the agent's Markov state set.

5. The method according to claim 3 or 4, characterized in that, The solution process for the Markov game model is as follows: After obtaining the state space and action space of each agent and the global information of all agents in the centralized value network, the expected value of any agent is calculated. Calculate the temporal difference error between the expected value and the target expected value of any agent. If the sum of the temporal difference errors meets a preset threshold, then the policy network corresponding to each agent is used as the optimization policy. If the sum of the temporal difference errors does not meet the preset threshold, the centralized value network updates the policy network of each agent until the sum of the temporal difference errors meets the preset threshold.

Images