A dormitory building indoor environment optimization method and system based on deep reinforcement learning combination

Abstract

The application relates to a dormitory building indoor environment optimization method and system based on deep reinforcement learning combination, wherein the method uses a pre-trained deep reinforcement learning model to optimize the indoor environment of the dormitory building; the building design parameters and the layout matrix of the indoor environment of the dormitory building are set as the state of the deep reinforcement learning model, the adjustment of the building design parameters is set as the action of the deep reinforcement learning model, and the reward of the deep reinforcement learning model includes an indoor average wind speed reward and an average effective daylight illuminance reward; and the system is used for realizing the above method. Compared with the prior art, the application has the advantages of good optimization effect and high optimization efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of indoor environment optimization, and in particular to a dormitory building indoor environment optimization method and system based on deep reinforcement learning combination. BACKGROUND

[0002] The existing building performance optimization research mainly adopts classical optimization algorithms to optimize the building environment, such as Bayesian optimization, genetic algorithm and annealing algorithm, etc. Evolutionary algorithms such as genetic algorithm are based on population update or iterative search, and are usually used to find the Pareto front solution set of multi-objective optimization problems, but such algorithms still exhibit two significant limitations, including: first, finding a set of Pareto front solutions necessarily involves a time-consuming process related to population update or iterative search, but for tasks with many control parameters, it can result in relatively low running efficiency; second, when the optimization problem needs to make some subtle adjustments to the building design, or even add some new optimization targets, the existing algorithms cannot flexibly adapt to changing conditions, and exhibit poor generalization performance. Or use machine learning, such as Chinese patent application CN109740243A, which provides a furniture layout method based on piecewise reinforcement learning technology to realize indoor environment optimization, and constructs an environment score neural network model for each type of furniture, and then combines reinforcement learning to optimize indoor furniture placement. Although furniture layout optimization is achieved, it only focuses on the static rationality of furniture layout, such as placement logic or functionality, and ignores the impact of furniture placement on the physical environment, and cannot ensure the comfort of the optimized scheme.

[0003] Therefore, it is a technical problem to provide a method that can maximize the comfort of the indoor environment. SUMMARY

[0004] The purpose of the present application is to overcome the defects of the prior art and provide a dormitory indoor environment optimization method using deep reinforcement learning, which uses deep reinforcement learning to balance exploration and efficient search for optimal solutions to obtain the highest comfort indoor environment optimization scheme.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] According to the first aspect of the present application, a dormitory building indoor environment optimization method based on deep reinforcement learning combination is provided, which uses a pre-trained deep reinforcement learning model to optimize the indoor environment of the dormitory building; the building design parameters and the layout matrix of the indoor environment of the dormitory building are set as the state of the deep reinforcement learning model, the adjustment of the building design parameters is set as the action of the deep reinforcement learning model, and the rewards of the deep reinforcement learning model include indoor average wind speed reward and average effective daylight illuminance reward.

[0007] As a preferred technical solution, the architectural design parameters include room width, room depth, shading depth, window width, toilet depth, toilet width, entrance door position, toilet window width, and toilet window height.

[0008] As a preferred technical solution, the layout matrix is an integer matrix of N×N, and each element in the matrix represents a type of building component, the building component including interior and exterior space, balcony rail plate, window, door, and wall.

[0009] As a preferred technical solution, the method for obtaining the layout matrix includes encoding the building component types, wherein the interior and exterior space is encoded as 0, the balcony rail plate is encoded as 1, the window is encoded as 2, the door is encoded as 3, and the wall is encoded as 4.

[0010] As a preferred technical solution, the expression for calculating the reward is:

[0011] ,

[0012] wherein, represents the average wind speed reward of the natural ventilation environment; represents the average effective daylight illuminance reward of the sunlight environment; represents the wind speed standard deviation; represents the effective daylight illuminance standard deviation; and respectively represent the average wind speed weight and the wind speed standard deviation weight; represents the effective daylight illuminance reward coefficient.

[0013] As a preferred technical solution, the method for calculating the average wind speed reward of the natural ventilation environment is:

[0014] ,

[0015] wherein, represents the average wind speed of the living area; represents the average wind speed of the balcony area; represents the average wind speed of the toilet area; , and respectively represent the wind speed weight values of the living area, the balcony area, and the toilet area, and .

[0016] As a preferred technical solution, the method for calculating the average effective daylight illuminance reward of the sunlight environment is:

[0017] ,

[0018] in, The average effective solar illuminance of the residential area; Indicates the effective sunlight illuminance in the balcony area; and These represent the effective solar illuminance weights for the residential area and the balcony area, respectively. .

[0019] As a preferred technical solution, the pre-training includes:

[0020] Based on the dormitory floor plan, parameterize the building design parameters within the dormitory as morphological control indicators. Construct a 3D model based on these morphological control indicators, obtain the initial state based on the 3D model, and repeat the following steps until the maximum number of iterations is reached:

[0021] The action of the agent in the deep reinforcement learning model is generated based on the state, and the new state after the action is executed is obtained; where the state at the first iteration is the initial state, and the state at the Nth iteration is the new state after the previous iteration;

[0022] Calculate the corresponding reward under the new state;

[0023] Based on the new state and reward, the parameters of the deep reinforcement learning model are updated using the deep deterministic gradient algorithm.

[0024] According to a second aspect of the present invention, a dormitory building indoor environment optimization system based on deep reinforcement learning is provided. The system is used to implement the above-mentioned method and includes: a visualization platform for parametric dormitory modeling based on the dormitory floor plan and updating the dormitory modeling based on actions generated by a deep reinforcement learning model; and a deep reinforcement learning algorithm for generating the action with the maximum reward value in the current state based on the state feedback from the visualization platform.

[0025] As a preferred technical solution, the visualization platform and the deep reinforcement learning algorithm use a socket method for data transmission.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] 1) This invention calculates rewards based on environmental parameters such as indoor wind speed and effective solar illuminance, and sets the building design parameters of the dormitory as the state, and the adjustment of the building design parameters as the action. It uses a deep reinforcement learning model to explore the coupling relationship between dynamic environmental factors, including wind speed and effective solar illuminance, and building design parameters, and eliminates unreasonable schemes generated solely based on the rationality of building space (such as maximizing the utilization rate of indoor space), so as to generate indoor environment optimization schemes with higher comfort and more suitable for human habitation, such as considering the effective solar illuminance of the main living areas when optimizing the scheme.

[0028] 2) This invention uses a pre-trained deep reinforcement learning model for indoor environment optimization design. Compared with traditional optimization algorithms, the method provided by this invention generates optimization schemes more efficiently, and the method provided by this invention has good generalization ability, and the optimization effect is still good in non-training scenarios. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention;

[0030] Figure 2 This is a schematic diagram of the parametric modeling results for dormitory type 1 of the present invention;

[0031] Figure 3 This is a schematic diagram of the parametric modeling results for dormitory type 2 of the present invention;

[0032] Figure 4 This is a schematic diagram of the parametric modeling results for dormitory type 3 of the present invention;

[0033] Figure 5 This is a graph showing the training results of the deep reinforcement learning model of the present invention.

[0034] Figure 6 This is an environmental performance optimization diagram of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0037] Example 1

[0038] Deep reinforcement learning (DRL), combining reinforcement learning and deep neural networks, excels in automatically creating novel AI agents that learn from iterative interactions with their environment and adjust their strategies through trial and error. Compared to genetic algorithms, reinforcement learning approximates complex high-dimensional environments using policies or value functions, enabling efficient exploration of high-dimensional spaces through extensive empirical learning. Furthermore, pre-trained reinforcement learning models can be transferred to similar tasks without requiring optimization from scratch. In addition, many traditional optimization algorithms lack inherent exploration mechanisms, easily getting trapped in local optima or exhibiting low efficiency in exploring global optima. DRL achieves efficient search by balancing exploration and utilization, especially in unknown or changing environments. Currently, in the construction field, while numerous studies have applied DRL to building energy and environmental control phases—such as reinforcement learning research in the design phase primarily focusing on urban building design and outdoor environment optimization, including indoor space layout and furniture layout design—research on its application in the early design phase of buildings is relatively limited. To address these issues, this application proposes a dormitory building indoor environment optimization method based on deep reinforcement learning. The process of this method is as follows: Figure 1 As shown, it includes:

[0039] S1. Parametric modeling.

[0040] We obtained real-world design examples of school dormitories in a certain region, selecting three typical dormitory floor plans. All three types consist of living areas, bathrooms, and balconies, but their layouts differ. Based on these three floor plans, we performed parametric modeling using Grasshopper shape syntax, as follows: Figure 2 , Figure 3 and Figure 4 As shown in Table 1, the architectural design parameters for the parametric dormitory model were collected in batches. These parameters include room width, room depth, shading depth, window width, bathroom depth, bathroom width, entrance door location, bathroom window width, and bathroom window height.

[0041] Table 1 Architectural Design Parameters

[0042]

[0043] S2, Deep reinforcement learning model construction.

[0044] Based on the dormitory model built in step S1 and the obtained architectural design parameters, a deep reinforcement learning model is built, including setting states, actions and rewards.

[0045] 1) State: The state represents the current environment in which the agent operates. As part of the interface between the agent's perception and environmental conditions, the state provides the agent with the set of information it needs to make decisions. In this embodiment, the state includes building design parameters and a layout matrix, where the building design parameters are shown in Table 1. The layout matrix encodes the dormitory model into a 100×100 integer matrix, where each value represents a specific building component type. The building component types include interior and exterior spaces, balcony railings, windows, doors, and walls. Specifically, the encoding method is as follows: interior and exterior spaces are encoded as 0, balcony railings as 1, windows as 2, doors as 3, and walls as 4.

[0046] 2) Actions: An agent performs actions to change the current state to the next state. Different actions will inevitably produce different effects. The goal is to move from the action space. The goal is to select the optimal action to maximize reward within a given scenario. In this embodiment, the adjustment of architectural design parameters is used as the action of the deep reinforcement learning model. As shown in Table 1, the dormitory model has nine variable design parameters, and the action space for each parameter is its variable range, as shown in the "Value Range" column of Table 1. The value of each parameter can be freely adjusted within a continuous range, and the numerical precision is retained to one decimal place. Therefore, the entire action space consists of nine continuous variables, and a complete action combination... The formula is as follows: ,in, and Let represent the minimum and maximum values ​​of the i-th parameter, respectively.

[0047] 3) Reward: The agent's specific goal is to maximize the reward, ultimately returning the optimal policy consisting of a series of operations. The reward function is highly correlated with two environmental performance values ​​and their distribution uniformity. It requires increasing indoor wind speed and effective solar illuminance (UDI) while reducing the standard deviation of wind speed and UDI. This includes rewards for average indoor wind speed and average effective solar illuminance. The detailed calculation formula is as follows: , This represents the average wind speed bonus in a naturally ventilated environment. This represents the average effective solar illuminance bonus in a solar environment. Indicates the standard deviation of wind speed; Indicates the standard deviation of effective solar illuminance; and These represent the weighting of average wind speed and the weighting of wind speed standard deviation, respectively. This represents the effective solar illuminance bonus coefficient.

[0048] The calculation method for the average wind speed bonus in naturally ventilated environments is as follows: , This indicates the average wind speed in the residential area; This indicates the average wind speed in the balcony area; This indicates the average wind speed in the restroom area; , and These represent the wind speed weights for the residential area, balcony area, and bathroom area, respectively. .

[0049] The calculation method for the average effective solar illuminance bonus in a solar environment is as follows:

[0050] ,

[0051] The average effective solar illuminance of the residential area; Indicates the effective sunlight illuminance in the balcony area; and These represent the effective solar illuminance weights for the residential area and the balcony area, respectively. .

[0052] In this embodiment, the average wind speed weighting Wind speed standard deviation weight Effective solar illuminance bonus coefficient Residential area wind speed weight Balcony area wind speed weighting Wind speed weighting in the living area Weight of effective solar illuminance in residential areas Effective solar illuminance weighting in balcony area .

[0053] S3, Deep reinforcement learning model pre-training.

[0054] Based on the dormitory floor plan, the architectural design parameters in the dormitory are used as morphological control indicators. A three-dimensional model is constructed based on the morphological control indicators. The initial state is obtained based on the three-dimensional model. The following steps are repeated until the number of iterations reaches 500.

[0055] S31. Generate the agent's actions in the deep reinforcement learning model based on the state, and obtain the new state after executing the action; wherein, the state at the first iteration is the initial state, and the state at the Nth iteration is the new state after the previous iteration.

[0056] S32. Calculate the corresponding reward under the new state.

[0057] S33. Based on the new state and reward, update the parameters of the deep reinforcement learning model using the deep deterministic gradient algorithm.

[0058] This invention also provides a dormitory building indoor environment optimization system based on deep reinforcement learning, used to implement the above method, including: a visualization platform for parametric dormitory modeling based on the dormitory floor plan, and for updating the dormitory modeling based on actions generated by the deep reinforcement learning model; and a deep reinforcement learning algorithm for generating the action with the maximum reward value in the current state based on the state feedback from the visualization platform.

[0059] In detail, during system operation, the deep reinforcement learning algorithm runs on Python. The state of the dormitory's indoor environment and the corresponding architectural design parameters are sent from the dormitory model built using Grasshopper on the visualization platform to the deep reinforcement learning algorithm. Based on the received state information, the deep reinforcement learning algorithm instructs the agent to select the next action according to the learned policy and sends these actions back to the visualization platform for execution. Upon receiving the action data, the visualization platform updates the dormitory model based on the actions, simulates the indoor wind and light environment performance, calculates the corresponding reward value, and selects the solution with the highest reward value as the final optimization solution. The visualization platform and the deep reinforcement learning algorithm use sockets for data transmission.

[0060] Example 2

[0061] To test the generalization and feasibility of the DRL model, reinforcement learning models were independently trained in each of three different indoor environment layouts (Type 1, Type 2, and Type 3) to obtain their respective pre-trained models (Model 1, Model 2, and Model 3). Each pre-trained model was then applied to all three scenarios, undergoing nine tests, and its optimal solution in each scenario was recorded. The generalization ability of the model in cross-scenario applications was evaluated by comparing the performance of the pre-trained models in the training and non-training scenarios.

[0062] The training process of the deep reinforcement learning models for the three types of scenarios is as follows: Figure 5 As shown, the reward values ​​of the training models in the three scenarios gradually stabilized after about 100 generations, indicating that the models had found a relatively optimal strategy. Figure 6 The optimal solution generated after optimization by a deep reinforcement learning model for the corresponding scenario shows that the optimization can achieve a balanced distribution of wind speed and effective sunlight illuminance in the dormitory indoor environment, and the optimization effect is good.

[0063] To verify the feasibility and superiority of the method provided in the above embodiments, this embodiment compares the method provided by the present invention with traditional optimization algorithms (taking the genetic GA algorithm as an example) and tests its optimization effect and efficiency, as shown in Table 2.

[0064] Table 2. Optimization test results of DRL and GA for three dormitory types.

[0065]

[0066] It is evident that the method (DRL) provided by this invention outperforms GA in all three scenario types, particularly in wind environment performance. In type 1, DRL's overall performance surpasses GA by 1.55%, with a 9.33% advantage in wind environment performance; in type 2, the overall advantage is 0.44%, with a 2.13% advantage in wind environment performance; and in type 3, the overall advantage is 1.97%, with a 2.35% advantage in wind environment performance and a 2.26% advantage in light environment performance. Experiments show that optimizing a dormitory-type scenario using a genetic algorithm takes an average of 9.5 hours, while training a DRL model for a scenario takes only about 6 hours. Optimizing a scenario using the method of this invention takes an average of only 15 minutes. Furthermore, the method of this invention is reusable, requiring no retraining for secondary optimizations, demonstrating a significant time advantage over GA.

[0067] To verify the generalization ability of the pre-trained models, the three pre-trained models were tested in three different scenarios. The optimal results are shown in Table 3. In scenario type 1, the maximum reward values ​​of the three pre-trained models were similar, with a relative difference of no more than 2.41%. In scenario type 2, the relative difference in the maximum reward values ​​of the three pre-trained models was less than 0.7%. In scenario type 3, the relative difference in the maximum reward values ​​of the three pre-trained models was less than 2.17%. This indicates that the three pre-trained models all have a certain degree of generalization ability in non-training scenarios.

[0068] Table 3. Optimization test results of the three pre-trained models in three scenarios.

[0069]

[0070] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optimizing the indoor environment of dormitory buildings based on deep reinforcement learning, characterized in that, The method described uses a pre-trained deep reinforcement learning model to optimize the indoor environment of dormitory buildings. The architectural design parameters and layout matrix of the dormitory building's interior environment are set as the state of the deep reinforcement learning model. The adjustment of the architectural design parameters is set as the action of the deep reinforcement learning model. The reward of the deep reinforcement learning model includes the average wind speed reward and the average effective sunlight illuminance reward of the natural ventilation environment. The layout matrix is ​​an N×N integer matrix, and each element in the matrix represents a type of building component, including interior and exterior spaces of rooms, balcony railings, windows, doors and walls. The pre-training includes: Based on the dormitory floor plan, parameterize the building design parameters within the dormitory as morphological control indicators. Construct a 3D model based on these morphological control indicators, obtain the initial state based on the 3D model, and repeat the following steps until the maximum number of iterations is reached: The action of the agent in the deep reinforcement learning model is generated based on the state, and the new state after the action is executed is obtained; where the state at the first iteration is the initial state, and the state at the Nth iteration is the new state after the previous iteration; Calculate the corresponding reward under the new state; Based on the new state and reward, the parameters of the deep reinforcement learning model are updated using the deep deterministic gradient algorithm.

2. The method for optimizing the indoor environment of dormitory buildings based on deep reinforcement learning according to claim 1, characterized in that, The architectural design parameters include room width, room depth, shading depth, window width, bathroom depth, bathroom width, entrance door location, bathroom window width, and bathroom window height.

3. The method for optimizing the indoor environment of dormitory buildings based on deep reinforcement learning according to claim 1, characterized in that, The method for obtaining the layout matrix includes: encoding the building component types, wherein the internal and external spaces of a room are encoded as 0, the balcony railing is encoded as 1, the window is encoded as 2, the door is encoded as 3, and the wall is encoded as 4.

4. The method for optimizing the indoor environment of dormitory buildings based on deep reinforcement learning according to claim 1, characterized in that, The expression for calculating the reward is: ， in, This represents the average wind speed bonus in a naturally ventilated environment. This represents the average effective solar illuminance bonus in a solar environment. Indicates the standard deviation of wind speed; Indicates the standard deviation of effective solar illuminance; and These represent the weighting of average wind speed and the weighting of wind speed standard deviation, respectively. This represents the effective solar illuminance bonus coefficient.

5. The method for optimizing the indoor environment of dormitory buildings based on deep reinforcement learning according to claim 4, characterized in that, The calculation method for the average wind speed bonus in the natural ventilation environment is as follows: ， in, This indicates the average wind speed in the residential area; This indicates the average wind speed in the balcony area; This indicates the average wind speed in the restroom area; , and These represent the wind speed weights for the residential area, balcony area, and bathroom area, respectively. .

6. The method for optimizing the indoor environment of dormitory buildings based on deep reinforcement learning according to claim 4, characterized in that, The calculation method for the average effective solar illuminance bonus in the solar environment is as follows: ， in, The average effective solar illuminance of the residential area; Indicates the effective sunlight illuminance in the balcony area; and These represent the effective solar illuminance weights for the residential area and the balcony area, respectively. .

7. A dormitory building indoor environment optimization system based on deep reinforcement learning, characterized in that, The system is used to implement the method as described in any one of claims 1 to 6, comprising: A visualization platform for parametric dormitory modeling based on dormitory floor plans, and for updating dormitory modeling based on actions generated by deep reinforcement learning models; On the deep reinforcement learning algorithm side, it is used to generate the action with the maximum reward value in the current state based on the state feedback from the visualization platform.

8. The dormitory building indoor environment optimization system based on deep reinforcement learning according to claim 7, characterized in that, The visualization platform and the deep reinforcement learning algorithm use a socket method for data transmission.

Images