Building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning

Abstract

The invention discloses a building multi-dimensional comfort level and energy management method based on multi-agent reinforcement learning, and the method comprises the steps: building a user multi-dimensional comfort level evaluation index, building a multi-agent reinforcement learning frame through building sensing information, continuously employing the historical data to adaptively adjust a user weight factor, and achieving the self-adaptive adjustment of the user weight factor. And the building comfort level and the energy consumption adjusting strategy are coordinated. According to the invention, user comfort evaluation standards are unified in a building comfort and energy management method, a multi-agent reinforcement learning framework is established according to building sensing information, and a user weight factor adaptive adjustment method is provided. A building multi-dimensional comfort and energy management method for processing building intelligent terminal equipment and sensor data in real time is established, and the contradiction between comfort and energy consumption cost is coordinated.

Description

Technical Field

[0001] The present invention relates to the field of building comfort and energy management, and in particular to a building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning. Background Art

[0002] It is of great significance to adopt effective control and optimization measures in the construction field to improve the energy utilization efficiency of the building side and achieve energy conservation and carbon reduction. At the same time, as one of the main scenarios for users to use energy, fully meeting the comfort needs of users is the basic goal of the building energy strategy.

[0003] At present, building energy management generally takes economic indicators as target constraints, and takes thermal comfort as a constraint or converts it into economic indicators, and coordinates the contradiction between comfort and energy consumption by solving optimization problems. However, this approach ignores the diverse comfort needs of users, and ignores the impact of various energy indicators such as vision and air quality on energy management when constructing comfort indicators. In addition, users have different preferences for building energy consumption and different comfort needs. This differentiated model learning requires both reasonable model fitting and a large amount of learning data for weight identification. When traditional optimization methods are applied in different building energy management systems, they lack the adaptability of weight parameter settings, and user preferences need to be determined through expert experience or manual parameter fitting. In addition, with the development of Internet of Things technology on the building side and the widespread access of smart terminal devices to buildings. Summary of the invention

[0004] Purpose of the invention: The present invention aims to provide a building multi-dimensional comfort and energy management method based on a multi-agent learning framework to establish real-time processing of building intelligent terminal equipment and sensor data.

[0005] Technical solution: The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning described in the present invention comprises the following steps: Establish user multi-dimensional comfort evaluation index, which includes temperature comfort index, visual comfort index and air quality comfort index; Obtain energy consumption information and sensor information of intelligent bodies in the building, calculate the fitting coefficient of the user's multi-dimensional comfort evaluation index, and determine the user's comprehensive comfort index; A building comfort and energy consumption regulation model based on multi-agent reinforcement learning is constructed. The comprehensive comfort index of building users and the maximization of energy consumption benefits are taken as the optimization strategy goals. Reinforcement learning training is carried out to adjust the comfort weights and optimal action values ​​to solve the optimal building comfort and energy consumption regulation strategy.

[0006] Furthermore, the user's comprehensive comfort index for ; in, is the weight of the temperature comfort index, is the weight of the visual comfort index, is the weight of the air quality comfort index; is the temperature comfort index, is the visual comfort index, It is an index of air quality comfort.

[0007] Furthermore, the building comfort and energy consumption regulation model based on multi-agent reinforcement learning, the multi-agent reinforcement learning strategy is In each study period , the agent observes the running status as follows: ; in, For intelligent agents Actual data for the period and forecast Data information of the time period; Energy data for various equipment; According to the running status Choose to execute actions for different agents , after taking action, a new operating state is observed in the next period , and select Execute New Action , and repeat this learning process; for each action Action quality value for ; in, is the action value, which is used to indicate the effect of the action; Indicates the running status Next action expectations; is the discount rate, which is used to express the relative importance of the expected action reward after k periods of time to the current one. ; is the utility at time t+1, which includes the user's comprehensive comfort target and energy consumption regulation target.

[0008] Furthermore, the optimization strategy targets the comprehensive comfort index of building users and the maximization of energy benefits. : ; in, It represents the economic value generated by users in the building participating in electricity sales or production; represents the energy cost of the building in the period [t, t+1]; is the user utility preference weight.

[0009] Furthermore, in the building comfort and energy consumption regulation model based on multi-agent reinforcement learning, the comfort constraint of the building agent is ; ; in, is the actual temperature comfort index value obtained by the intelligent agent through sensor measurement, is the actual visual comfort index value obtained by the agent through sensor measurement, The actual air quality comfort index obtained by the intelligent agent through sensor measurement; Set preferences for the user's corresponding comfort level, .

[0010] Further, optimal building comfort and energy consumption regulation strategy The corresponding action quality function for ; in, Reward for action; is the expected maximum action value.

[0011] Furthermore, The following conditions are met: ; ; in, Represents the learning rate, which is used to indicate the update efficiency of the iteration. ; Indicates the maximum allowable deviation between two action values; k is the number of iterations.

[0012] Furthermore, the target comfort weight under the optimal action value for ; in, is the target comfort weight for the k+1th time; is the k+1th online learning weight, For the k The target comfort weight of the second; It is an update parameter used to control the speed of comfort weight update.

[0013] Furthermore, the temperature comfort index for ; in, M is the human metabolic rate, in W / m2; is the heat load consumed by personnel, in W / m2; Visual comfort index for ; in, is the visual comfort fitting coefficient, ; is the illuminance, the unit is Lux; To unify the glare value; ; in, For lighting equipment j The adjustment state is in the range of ; For lighting equipment j Rated illuminance, in Lux; is the natural light illumination, the unit is Lux; Air quality comfort index for ; in, is the air quality comfort fitting coefficient, ; is the comprehensive impact index of air quality; ; in, is the indoor relative humidity, is the indoor carbon dioxide concentration, is the carbon monoxide concentration, is the formaldehyde concentration, is the concentration of inhalable particulate matter.

[0014] Furthermore, the fitting coefficient of the user's multidimensional comfort evaluation index is calculated as follows: The formula for solving the visual comfort fitting coefficient is as follows: ; ; in, is the mean square error of visual comfort; For the i Lighting power in group device information; For the i Unified glare value in group device information; For users i Lighting comfort evaluation of group data; The formula for solving the air quality comfort fitting coefficient is as follows: ; ; in, is the mean square error of air quality comfort; For the i Air quality impact index in group equipment information; For users i Lighting comfort evaluation of group data.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. The present invention establishes a multi-agent reinforcement learning framework through building sensor information, and provides a method for adaptively adjusting user weight factors; 2. The present invention unifies the user comfort evaluation standard in the building comfort and energy management method, and can be used for comfort evaluation of various types of buildings; 3. The present invention realizes the decoupling of the weight setting between multi-dimensional comfort and energy cost, flexibly sets the weight of each influencing factor under the unified "energy welfare" standard, and better coordinates the contradiction between comfort and energy cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the present invention; Figure 2 Adaptive tuning framework diagram for multi-agent reinforcement learning; Figure 3 Schematic diagram of multi-dimensional comfort weight adjustment of buildings. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings.

[0018] The multi-dimensional building comfort and energy management method based on multi-agent reinforcement learning of the present invention comprises the following steps: (1) Establish a user multi-dimensional comfort evaluation index, which includes temperature comfort index, visual comfort index and air quality comfort index.

[0019] The temperature comfort index is ; in, is the temperature comfort index, M is the metabolic rate of the human body, in W / m2, which is set manually; It is the heat load consumed by personnel, in W / m2, which is set manually and adjusted through building data.

[0020] The visual comfort index is ; in, It is an indicator of visual comfort; a i is the visual comfort fitting coefficient, , obtained by fitting the building data; is the illuminance, the unit is Lux; To unify the glare value.

[0021] Light intensity Can be described as ; in, For lighting equipment j The adjustment state of is in the range [0,1]; For lighting equipment j Rated illuminance, Lux; is the natural light illumination, Lux.

[0022] The air quality comfort index is ; In the formula, is the coefficient of the air quality comfort index polynomial model, It is the comprehensive impact index of air quality.

[0023] Air Quality Comprehensive Impact Index Can be described as ; In the formula, is the indoor relative humidity, is the indoor carbon dioxide concentration, is the carbon monoxide concentration, is the formaldehyde concentration, is the concentration of inhalable particulate matter.

[0024] (2) Obtain the energy consumption information and sensor information of the intelligent body in the building, calculate the fitting coefficient of the user's multidimensional comfort evaluation index, and determine the user's comprehensive comfort index.

[0025] The sensing devices correspond to different local agents according to the temperature, visual and air quality comfort. The agents control different types of smart devices in a decentralized manner, monitor the device information and receive the control commands issued by the agent to solve the correlation coefficient and calculate the initial weight factor. The initial value of the user comfort preference factor is selected by the user, and the formula for solving the visual comfort fitting coefficient is as follows: ; ; in, is the mean square error of visual comfort; For the i Lighting power in group device information; For the i Unified glare value in group device information; For users i Lighting comfort evaluation of group data.

[0026] The formula for solving the air quality comfort fitting coefficient is as follows: ; ; in, is the mean square error of air quality comfort; For the i Air quality impact index in group equipment information; For users i Lighting comfort evaluation of group data.

[0027] The relationship between the initial weight factor and the comfort index is: ; In the formula, Provides comprehensive comfort index for users; are the weights of each comfort index, and the initial weight values ​​are set by the user's preference.

[0028] (3) Construct a building comfort and energy consumption regulation model based on multi-agent reinforcement learning. Take the comprehensive comfort index of building users and the maximization of energy consumption benefits as the optimization strategy goals, conduct reinforcement learning training, adjust the comfort weight and optimal action value, and solve the optimal building comfort and energy consumption regulation strategy.

[0029] Building comfort and energy consumption regulation model based on multi-agent reinforcement learning. The multi-agent reinforcement learning strategy is: At each learning period t, the agent observes the running state as follows: ; in, is the actual data of the agent during period t, and predicts Data information of the time period; Energy data for various types of equipment.

[0030] According to the running status Choose to execute actions for different agents , after taking action, a new operating state is observed in the next period , and select Execute New Action , and repeat this learning process; for each action Action quality value for ; in, is the action value, which is used to indicate the effect of the action; Indicates the running status Next action expectations; is the discount rate, which is used to express the relative importance of the expected action reward after k periods of time to the current one. ; is the utility at time t+1, which includes the user's comprehensive comfort target and energy consumption regulation target.

[0031] Furthermore, the optimization strategy targets the comprehensive comfort index of building users and the maximization of energy benefits. : ; in, It represents the economic value generated by users in the building participating in electricity sales or production; represents the energy cost of the building in the period [t, t+1]; is the user utility preference weight.

[0032] Furthermore, in the building comfort and energy consumption regulation model based on multi-agent reinforcement learning, the comfort constraint of the building agent is ; The method for adjusting the comfort weight of the local agent is to satisfy the following inequality conditions and solve the optimal strategy through reinforcement learning: : ; in, is the actual temperature comfort index value obtained by the intelligent agent through sensor measurement, is the actual visual comfort index value obtained by the agent through sensor measurement, The actual air quality comfort index obtained by the intelligent agent through sensor measurement; Set preferences for the user's corresponding comfort level, .

[0033] Optimal building comfort and energy consumption regulation strategy The corresponding action quality function for ; in, Reward for action; is the expected maximum action value, which can be updated iteratively by the following formula to obtain the value of time period t: : ; ; in, Represents the learning rate, which is used to indicate the update efficiency of the iteration. ; Indicates the maximum allowable deviation between two action values; k is the number of iterations. When the inequality requirement is met, the iteration stops.

[0034] Target comfort weight under optimal action value for ; in, is the target comfort weight for the k+1th time; is the k+1th online learning weight, For the k The target comfort weight of the second; It is an update parameter used to control the speed of comfort weight update.

Claims

1. A building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Establish user multi-dimensional comfort evaluation index, which includes temperature comfort index, visual comfort index and air quality comfort index; Obtain energy consumption information and sensor information of intelligent bodies in the building, calculate the fitting coefficient of the user's multi-dimensional comfort evaluation index, and determine the user's comprehensive comfort index; A building comfort and energy consumption regulation model based on multi-agent reinforcement learning is constructed. The comprehensive comfort index of building users and the maximization of energy consumption benefits are taken as the optimization strategy goals. Reinforcement learning training is carried out to adjust the comfort weights and optimal action values ​​to solve the optimal building comfort and energy consumption regulation strategy.

2. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 1 is characterized in that: User comprehensive comfort index for ； in, is the weight of the temperature comfort index, is the weight of the visual comfort index, is the weight of the air quality comfort index; is the temperature comfort index, is the visual comfort index, It is an index of air quality comfort.

3. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 2 is characterized in that: Building comfort and energy consumption regulation model based on multi-agent reinforcement learning. The multi-agent reinforcement learning strategy is: At each learning period t, the agent observes the running state as follows: ； in, is the actual data of the agent during period t, and predicts Data information of the time period; Energy data for various equipment; According to the running status Choose to execute actions for different agents , after taking action, a new operating state is observed in the next period , and select Execute new action , and repeat this learning process; for each action Action quality value for ； in, is the action value, which is used to indicate the effect of the action; Indicates the running status Next action expectations; is the discount rate, which is used to express the relative importance of the expected action reward after k periods of time to the current one. ; is the utility at time t+1, which includes the user's comprehensive comfort target and energy consumption regulation target.

4. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 3 is characterized in that: The optimization strategy targets the comprehensive comfort index of building users and the maximization of energy benefits. : ； in, It represents the economic value generated by users in the building participating in electricity sales or production; represents the energy cost of the building in the period [t, t+1]; is the user utility preference weight.

5. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 4 is characterized in that: In the building comfort and energy consumption regulation model based on multi-agent reinforcement learning, the comfort constraint of the building agent is ； ； in, is the actual temperature comfort index value obtained by the intelligent agent through sensor measurement, is the actual visual comfort index value obtained by the agent through sensor measurement, The actual air quality comfort index obtained by the intelligent agent through sensor measurement; Set preferences for the user's corresponding comfort level, .

6. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 5 is characterized in that: Optimal building comfort and energy consumption regulation strategy The corresponding action quality function for ； in, Reward for action; is the expected maximum action value.

7. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 6 is characterized in that: The following conditions are met: ； ； in, Represents the learning rate, which is used to indicate the update efficiency of the iteration. ; Indicates the maximum allowable deviation between two action values; k is the number of iterations.

8. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 7 is characterized in that: Target comfort weight under optimal action value for ； in, is the target comfort weight for the k+1th time; is the k+1th online learning weight, For the k The target comfort weight of the second; It is an update parameter used to control the speed of comfort weight update.

9. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 1 is characterized in that: Temperature comfort index for ； in, M is the human metabolic rate, in W / m2; is the heat load consumed by personnel, in W / m2; Visual comfort index for ； in, is the visual comfort fitting coefficient, ; is the illuminance, the unit is Lux; To unify the glare value; ； in, For lighting equipment j The adjustment state range is ; For lighting equipment j Rated illuminance, in Lux; is the natural light illumination, the unit is Lux; Air quality comfort index for ； in, is the air quality comfort fitting coefficient, ; is the comprehensive impact index of air quality; ； in, is the indoor relative humidity, is the indoor carbon dioxide concentration, is the carbon monoxide concentration, is the formaldehyde concentration, is the concentration of inhalable particulate matter.

10. The building multi-dimensional comfort and energy management method based on multi-agent reinforcement learning according to claim 9 is characterized in that: The fitting coefficient of the user multidimensional comfort evaluation index is calculated as follows: The formula for solving the visual comfort fitting coefficient is as follows: ； ； in, is the mean square error of visual comfort; For the i Lighting power in group device information; For the i Unified glare value in group device information; For users i Lighting comfort evaluation of group data; The formula for solving the air quality comfort fitting coefficient is as follows: ； ； in, is the mean square error of air quality comfort; For the i Air quality impact index in group equipment information; For users i Lighting comfort evaluation of group data.

Images