A method and system for controlling indoor space temperature and humidity

Abstract

This invention provides a method and system for regulating indoor temperature and humidity. It uses temperature and humidity parameters as the state space and the airflow speed setting of a constant temperature and humidity machine as the action space. A source domain deep reinforcement learning model is obtained through pre-training in the source space. In the target space, each actuator and its surrounding temperature and humidity sensors within a set range constitute a local control component. Each local control component is controlled by an independent target domain agent. Each target agent performs parameter fine-tuning and control by transferring and learning the source domain deep reinforcement learning model. The target space observation reward value is calculated by combining the temperature and humidity values ​​collected by all temperature and humidity sensors in the target space. The source domain deep reinforcement learning model of each target domain agent is trained separately with the goal of maximizing the future multi-step target space observation reward value, enabling rapid adaptation to achieve precise temperature and humidity control in the target space environment.

Description

Technical Field

[0001] This invention relates to the field of electronic device control technology, and in particular to a method and system for regulating indoor space temperature and humidity. Background Technology

[0002] Indoor temperature and humidity have a significant impact on the comfort of occupants. In certain environments, such as museums and high-end CNC machine tools, the requirements for indoor temperature and humidity are even more stringent. Precise and uniform temperature and humidity distribution is more conducive to the preservation of cultural relics and the processing and production of precision parts. Therefore, research on indoor temperature and humidity control is of great significance. Traditional control methods have limited monitoring range and are prone to uneven distribution of indoor temperature and humidity. Currently, most indoor temperature and humidity methods do not consider both accuracy and uniformity, focusing primarily on the overall comfort of the room while ignoring the local distribution within the environment. AI-based indoor temperature and humidity technologies are also being researched, such as predictive control and deep reinforcement learning-based control. However, when the application scenario changes, the trained control model may not be suitable for the new application scenario and may even result in worse control performance. Training a new control model for a new scenario requires a significant amount of time and computational resources.

[0003] With technological advancements, various industries are increasingly demanding higher standards for temperature and humidity control, prompting researchers to propose a series of methods. These include using gas-liquid separation dehumidifier systems and solution-based dehumidifiers to maintain positive air pressure in building spaces. However, traditional temperature and humidity control relies heavily on manual labor, resulting in low efficiency and precision. Machine learning-based automatic indoor temperature and humidity control systems offer high intelligence and practicality, enabling unattended automatic control. However, existing methods are highly dependent on the specific room environment, requiring individual training and control for each room. The algorithms also place high demands on room sensors and air vents, hindering rapid and widespread deployment in practical applications. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an indoor space temperature and humidity control method and system to eliminate or improve one or more defects in the prior art and solve the problem that artificial intelligence models are affected by the environment and cannot be quickly and efficiently deployed in new environments for indoor temperature and humidity control.

[0005] In one aspect, the present invention provides a method for regulating indoor space temperature and humidity. The method operates on a central controller of a target space, the central controller being connected to a cloud server. The central controller connects to multiple temperature and humidity sensors and multiple actuators within the target space via the Internet of Things (IoT). Each actuator is the air outlet of a constant temperature and humidity unit. Each actuator, along with temperature and humidity sensors within a defined range around it, constitutes a local control component. Multiple target domain intelligent agents operate on the central controller, each target domain intelligent agent controlling a local control component within the target space. The method includes the following steps:

[0006] The source domain deep reinforcement learning model, pre-trained in the source space, is obtained from the cloud server. This model uses temperature and humidity values ​​collected by multiple temperature and humidity sensors at specified intervals within the source space as state parameters to form the source space state space. The wind speed settings of each actuator operating at the set humidity and temperature within the source space are used as action parameters to form the source space action space. The source space observation reward value is calculated based on the state parameters. The source domain deep reinforcement learning model is trained and its parameters are updated to maximize the sum of future multi-step source space observation reward values ​​as the optimization objective until convergence.

[0007] The source domain deep reinforcement learning model is transferred and distributed to each target domain agent within the target space. During the deep reinforcement learning process, each target domain agent uses the temperature and humidity values ​​collected by multiple temperature and humidity sensors in its controlled local control components at specified intervals as state parameters to form a local state space of the target domain, and uses the wind speed settings of each actuator operating at the target space set humidity and target space set temperature as action parameters to form a local action space of the target domain. The first humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the target space and the target space set humidity, as well as the first humidity uniformity deviation between each actual humidity value, are calculated. The first temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors in the target space and the target space set temperature, as well as the first temperature uniformity deviation between each actual temperature value, are calculated. The target space observation reward value is calculated based on the first humidity accuracy deviation, the first humidity uniformity deviation, the first temperature accuracy deviation, and the first temperature uniformity deviation corresponding to each time step. Each target domain agent optimizes the target by maximizing the sum of the target space observation reward values ​​in future multiple steps to train and update the parameters of the source domain deep reinforcement learning model until convergence.

[0008] In some embodiments, the training steps of the source domain deep reinforcement learning model include:

[0009] The temperature and humidity values ​​collected by multiple temperature and humidity sensors in the source space at specified intervals are used as state parameters, and the source space is used as a whole to form the overall state space.

[0010] The wind speed settings of each actuator operating within the source space with the set humidity and set temperature of the source space are obtained as action parameters to construct the overall action space.

[0011] Calculate the second humidity accuracy deviation between the actual humidity values ​​of all humidity sensors in the source space and the set humidity of the source space, as well as the second humidity uniformity deviation between each actual humidity value; calculate the second temperature accuracy deviation between the actual temperature values ​​of all temperature sensors in the source space and the set temperature of the source space, as well as the second temperature uniformity deviation between each actual temperature value.

[0012] The source space observation reward value is calculated based on the second humidity accuracy deviation, the second humidity uniformity deviation, the second temperature accuracy deviation, and the second temperature uniformity deviation corresponding to each time step;

[0013] The initial reinforcement learning model is trained and its parameters are updated until convergence is achieved by maximizing the sum of future multi-step source space observation rewards. This yields the source domain deep reinforcement learning model.

[0014] In some embodiments, the training steps of the source domain deep reinforcement learning model include:

[0015] The source domain deep reinforcement learning model is sent to multiple source domain agents. Each source domain agent controls an actuator and a set range of temperature and humidity sensors around it. Each source domain agent uses the temperature and humidity values ​​collected by the multiple temperature and humidity sensors it controls at specified intervals as state parameters to form a local state space of the source domain.

[0016] Each source domain agent uses the wind speed setting of the actuator it controls as the action parameter to form a local action space in the source domain.

[0017] Calculate the third humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the source space and the set humidity of the source space, as well as the third humidity uniformity deviation between each actual humidity value; calculate the third temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors in the source space and the set temperature of the source space, as well as the third temperature uniformity deviation between each actual temperature value.

[0018] The source space observation reward value is calculated based on the third humidity accuracy deviation, the third humidity uniformity deviation, the third temperature accuracy deviation, and the third temperature uniformity deviation corresponding to each time step;

[0019] Each source domain agent acquires the source space observation reward value of the entire source space end-to-end. The initial reinforcement learning model of each source domain agent is trained and its parameters are updated until convergence is achieved with the optimization objective of maximizing the sum of the source space observation reward values ​​in the future multi-step process. The updated model parameters of each source domain agent are then aggregated to obtain the source domain deep reinforcement learning model.

[0020] In some embodiments, the updated model parameters of each source domain agent are aggregated, including: fusing the updated parameters of each source domain agent according to a set weight to obtain the parameters of the source domain deep reinforcement learning model.

[0021] In some embodiments, a first humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the target space and the set humidity of the target space, and a first humidity uniformity deviation between the actual humidity values ​​are calculated, wherein the formula for calculating the first humidity accuracy deviation is:

[0022]

[0023] H sc This indicates the first humidity accuracy deviation. H represents the humidity detected by the i-th temperature and humidity sensor at time t. set β represents the target space humidity setting, and β represents the number of temperature and humidity sensors.

[0024] The formula for calculating the first humidity uniformity deviation is:

[0025]

[0026]

[0027] H unif This indicates the first humidity uniformity deviation. This represents the humidity detected by the i-th temperature and humidity sensor at time t. This represents the average humidity value collected by all temperature and humidity sensors, where β represents the number of temperature and humidity sensors.

[0028] Calculate the first temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors within the target space and the set temperature of the target space, and the first temperature uniformity deviation between the actual temperature values. The formula for calculating the first temperature accuracy deviation is:

[0029]

[0030] T sc T represents the first temperature accuracy deviation. t i T represents the temperature detected by the i-th temperature and humidity sensor at time t.set β represents the target space set temperature, and β represents the number of temperature and humidity sensors.

[0031] The formula for calculating the first temperature uniformity deviation is:

[0032]

[0033]

[0034] T unif T represents the first temperature uniformity deviation. t i This represents the temperature detected by the i-th temperature and humidity sensor at time t. This represents the average temperature collected by all temperature and humidity sensors, and β represents the number of temperature and humidity sensors.

[0035] In some embodiments, the target space observation reward value is calculated based on the first humidity accuracy deviation, the first humidity uniformity deviation, the first temperature accuracy deviation, and the first temperature uniformity deviation corresponding to each time step, and the calculation formula is:

[0036] R t =α1(T sc +H sc )+α2(T unif +H unif );

[0037] Among them, R t H represents the observed reward value of the state at time t, where α1 and α2 are weighting coefficients. sc H represents the first humidity accuracy deviation. unif T represents the first humidity uniformity deviation. sc T represents the first temperature accuracy deviation. unif This indicates the deviation in the first temperature uniformity.

[0038] In some embodiments, each target domain agent trains and updates the parameters of the source domain deep reinforcement learning model to optimize the objective by maximizing the sum of reward values ​​from future multi-step target space observations until convergence, including:

[0039] Construct the optimization objective function, with the expression:

[0040]

[0041] Where γ represents the discount factor and γ < 1, R t+t′ (S t+t′ A t+t′ ) represents the target space observation reward value at time t+t′.

[0042] In some embodiments, the source domain deep reinforcement learning model constructs a neural network consisting of multiple fully connected layers to predict the Q value corresponding to each action. The neural network has two branches: the first branch is used to predict the state value, and the second branch is used to predict the advantage of each action. The Q value corresponding to each action is calculated by combining the predicted state value and the advantage of each action.

[0043] On the other hand, the present invention also provides an indoor space temperature and humidity control system, the system comprising:

[0044] Multiple temperature and humidity sensors, including humidity sensors and temperature sensors, are installed within the target space;

[0045] The constant temperature and humidity machine is equipped with multiple air outlets, each air outlet has multiple wind speed settings, and the wind speed settings of each air outlet are set separately. Each air outlet is located within the target space. Each air outlet and the temperature and humidity sensors within a set range around it constitute a local control component.

[0046] A central controller is connected to a cloud server and connects to various temperature and humidity sensors in the target space and various air outlets of the constant temperature and humidity machine via the Internet of Things. Multiple target domain intelligent agents run on the central controller, and each target domain intelligent agent is used to control a local control component in the target space.

[0047] The central controller executes the above-described indoor space temperature and humidity control method to control the temperature and humidity within the target space.

[0048] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0049] The beneficial effects of the present invention are at least as follows:

[0050] The indoor space temperature and humidity control method and system of this invention uses temperature and humidity state parameters as the state space and the air speed setting of the constant temperature and humidity machine outlet as the action space. A source domain deep reinforcement learning model is obtained through pre-training in the source space. In the target space, each actuator and the temperature and humidity sensors within a set range around it constitute a local control component. Each local control component is controlled by an independent target domain agent. Each target agent performs parameter fine-tuning and control by transferring the source domain deep reinforcement learning model. The target space observation reward value is calculated by combining the temperature and humidity values ​​collected by all temperature and humidity sensors in the target space. The source domain deep reinforcement learning model of each target domain agent is trained separately with the goal of maximizing the target space observation reward value in multiple future steps, so as to quickly adapt to the precise control of temperature and humidity in the target space environment.

[0051] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0052] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0053] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0054] Figure 1 This is a logical schematic diagram of an embodiment of the indoor space temperature and humidity control method of the present invention.

[0055] Figure 2 This is a schematic diagram of the training logic of the source domain deep reinforcement learning model in the indoor space temperature and humidity control method according to an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the training logic of the source domain deep reinforcement learning model in the indoor space temperature and humidity control method according to another embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of room A, which is equipped with a constant temperature and humidity air conditioning system.

[0058] Figure 5 This is a schematic diagram of a model of room B equipped with a constant temperature and humidity air conditioning system.

[0059] Figure 6This is a schematic diagram of room C, which is equipped with a constant temperature and humidity air conditioning system.

[0060] Figure 7 This is a schematic diagram of room D, which is equipped with a constant temperature and humidity air conditioning system.

[0061] Figure 8 This is a schematic diagram of Room E, which is equipped with a constant temperature and humidity air conditioning system.

[0062] Figure 9 This is a schematic diagram of room F, which is equipped with a constant temperature and humidity air conditioning system.

[0063] Figure 10 This is a comparison chart showing the reward values ​​after migrating from C as the source domain to D in various forms and achieving convergence.

[0064] Figure 11 This is a comparison chart showing the reward values ​​after migrating from E as the source domain to D in various forms and achieving convergence.

[0065] Figure 12 This is a comparison of reward values ​​after convergence for two methods using front-end transfer: individual training, overall transfer, and random selection of one of the multi-agent methods.

[0066] Figure 13 A comparison of reward values ​​after convergence for different source domain models using front-end transfer and random selection of one of the multiple agents.

[0067] Figure 14 This is a comparison of the accuracy and uniformity of temperature and humidity under interference conditions for migration enhancement and direct training.

[0068] Figure 15 The graph compares the accuracy and uniformity of temperature and humidity under interference conditions when the C model strategy is transferred to D and when the D model is directly trained.

[0069] Figure 16 This is a performance comparison chart of policy transfer learning under different numbers of agents.

[0070] Figure 17 This is a comparison chart of the accuracy and uniformity of policy transfer learning under different numbers of agents under interference conditions, based on temperature and humidity.

[0071] Figure 18 The graph compares the accuracy and uniformity of temperature and humidity under the conditions of policy transfer learning from model E to model F under disturbance conditions and direct training of model F.

[0072] Figure 19 The graph shows a comparison of the performance of transfer learning and direct training when each agent in the F model monitors only two temperature and humidity sensors.

[0073] Figure 20This is a comparison of the accuracy and uniformity of transfer learning and direct training under interference conditions when each agent in the F model monitors only two temperature and humidity sensors.

[0074] Figure 21 The graph shows a comparison of the performance of transfer learning and direct training when the number of temperature and humidity sensors monitored by each agent in the F model varies.

[0075] Figure 22 The graph shows a comparison of the accuracy and uniformity of transfer learning and direct training under interference conditions when the number of temperature and humidity sensors monitored by each agent in the F model varies. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0077] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0078] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0079] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0080] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0081] Reinforcement learning methods can learn optimal strategies through interaction with the environment, and model-free methods can ignore the complexity and difficulty of modeling and computing different environments. Therefore, using reinforcement learning methods to regulate indoor temperature and humidity meets the needs of objective conditions, enabling high-precision and high-uniformity control of indoor temperature and humidity, minimizing damage to special items stored indoors, preventing physical damage caused by excessively high or low temperatures, and reducing the risk of permanent damage due to drastic fluctuations in temperature and humidity.

[0082] However, due to the significant differences in different spatial environments, each control model needs to be retrained and adapted in a specific space. The training computational requirements for some complex space temperature and humidity control models are high, and due to the lack of sufficient computational resources in different application environments, it is difficult to widely promote indoor temperature and humidity control through reinforcement learning.

[0083] Reinforcement learning transfer algorithms enable the rapid deployment of temperature and humidity control reinforcement models for a large number of indoor environments with similar characteristics. For example, for different warehouses storing special items, a high-efficiency, high-performance reinforcement learning algorithm can be adapted to each warehouse while being mass-produced, which is significant for reducing the computational resources and time required by equipment. However, in this process, it is difficult to achieve uniformity when a single agent controls multiple air vents, and the algorithm's stability is poor. Therefore, a method is needed that can meet the requirements of precise and uniform indoor temperature and humidity, has higher anti-interference capabilities, and can be quickly transferred to new indoor environments for application.

[0084] Specifically, this application provides a method for controlling indoor space temperature and humidity. The method is used to run on a central controller of the target space. The central controller is connected to a cloud server and connects to multiple temperature and humidity sensors and multiple actuators in the target space via the Internet of Things. The actuators are the air outlets of a constant temperature and humidity machine. Each actuator and the temperature and humidity sensors within a set range around it constitute a local control component. Multiple target domain intelligent agents run on the central controller. Each target domain intelligent agent is used to control a local control component in the target space.

[0085] In this application, to better utilize the trained model and avoid the problem of different input and output dimensions of agents due to varying spatial layouts and sensor factors in different scenarios, a multi-agent transfer reinforcement learning approach is introduced. Multiple agents cooperate to jointly control indoor temperature and humidity. The temperature and humidity sensors and actuators within the target space are divided into multiple local control components. A target domain agent is configured for each local control component to regulate the temperature and humidity of the corresponding local space. This reduces the environmental and control parameters within each target domain agent, enabling the reinforcement learning model to converge quickly. Simultaneously, the pre-trained reinforcement learning model can be transferred to the target domain agents of each local control component in the target space through transfer learning, quickly adapting to the temperature and humidity regulation requirements of the target space environment.

[0086] Specifically, such as Figure 1 As shown, the method includes the following steps S101 to S102:

[0087] Step S101: Obtain the source domain deep reinforcement learning model pre-trained based on the source space from the cloud server. The source domain deep reinforcement learning model uses the temperature and humidity values ​​collected by multiple temperature and humidity sensors in the source space at specified intervals as state parameters to form the source space state space. The wind speed settings of each actuator in the source space that operates with the set humidity and set temperature in the source space are used as action parameters to form the source space action space. Calculate the source space observation reward value based on the state parameters. The source domain deep reinforcement learning model trains and updates the parameters of the initial reinforcement learning model with the optimization objective of maximizing the sum of the source space observation reward values ​​in multiple future steps until convergence.

[0088] Step S102: The source domain deep reinforcement learning model is transferred and distributed to each target domain agent in the target space. During the deep reinforcement learning process, each target domain agent uses the temperature and humidity values ​​collected by multiple temperature and humidity sensors in its controlled local control components at specified intervals as state parameters to form the local state space of the target domain, and uses the wind speed settings of each actuator operating at the target space set humidity and target space set temperature as action parameters to form the local action space of the target domain. The first humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the target space and the target space set humidity, as well as the first humidity uniformity deviation between each actual humidity value, are calculated. The first temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors in the target space and the target space set temperature, as well as the first temperature uniformity deviation between each actual temperature value, are calculated. The target space observation reward value is calculated based on the first humidity accuracy deviation, first humidity uniformity deviation, first temperature accuracy deviation, and first temperature uniformity deviation corresponding to each time step. Each target domain agent optimizes the target by maximizing the sum of the target space observation reward values ​​in the future multi-step process to train and update the parameters of the source domain deep reinforcement learning model until convergence.

[0089] In step S101, the central controller directly obtains the pre-trained source domain deep reinforcement learning model from the cloud server. This model, capable of adjusting indoor temperature and humidity by controlling the fan speed at the outlet of the constant temperature and humidity unit, is trained based on the environmental state parameters and control parameters of the source space. The pre-training of the source domain can take two forms: first, pre-training using the overall state parameters and control parameters of the source space; second, dividing the source space into multiple local control regions, pre-training a reinforcement learning neural network separately in each region, and then merging them. The reinforcement learning process of the reinforcement learning neural networks in each local control region is optimized using the overall observation reward value of the source space.

[0090] For the pre-training format of the first type of source domain deep reinforcement learning model, such as Figure 2 As shown, the training steps of the source domain deep reinforcement learning model include steps S201 to S205:

[0091] Step S201: Obtain the temperature and humidity values ​​collected by multiple temperature and humidity sensors in the source space at specified intervals as state parameters, and construct the overall state space with the source space as a whole.

[0092] Step S202: Obtain the wind speed settings of each actuator operating within the source space with the set humidity and set temperature of the source space as action parameters, and construct the overall action space.

[0093] Step S203: Calculate the second humidity accuracy deviation between the actual humidity values ​​of all humidity sensors in the source space and the set humidity of the source space, as well as the second humidity uniformity deviation between each actual humidity value; calculate the second temperature accuracy deviation between the actual temperature values ​​of all temperature sensors in the source space and the set temperature of the source space, as well as the second temperature uniformity deviation between each actual temperature value.

[0094] Step S204: Calculate the source space observation reward value based on the second humidity accuracy deviation, second humidity uniformity deviation, second temperature accuracy deviation, and second temperature uniformity deviation corresponding to each time step.

[0095] Step S205: Train and update the parameters of the initial reinforcement learning model with the optimization objective of maximizing the sum of the reward values ​​of future multi-step source space observations until convergence, and obtain the source domain deep reinforcement learning model.

[0096] In steps S201 to S205, a state space and an action space are constructed based on the states of all temperature and humidity sensors and air outlets in the source space. Temperature and humidity are controlled through reinforcement learning, enabling a single model to regulate the overall global spatial range.

[0097] In some embodiments, to maintain the same process of forming the source domain deep reinforcement learning model as the process of running the target space reinforcement learning model, and to improve the model's generalization ability, such as... Figure 3 As shown, the training steps for the source domain deep reinforcement learning model can be set up to include steps S301 to S305:

[0098] Step S301: Send the source domain deep reinforcement learning model to multiple source domain agents respectively. Each source domain agent is used to control an actuator and the temperature and humidity sensors within a set range around it. Each source domain agent uses the temperature and humidity values ​​collected by the multiple temperature and humidity sensors it controls at specified intervals as state parameters to form the local state space of the source domain.

[0099] Step S302: Each source domain agent uses the wind speed setting of the actuator it controls as the action parameter to form a local action space in the source domain.

[0100] Step S303: Calculate the third humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the source space and the set humidity of the source space, as well as the third humidity uniformity deviation between each actual humidity value; calculate the third temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors in the source space and the set temperature of the source space, as well as the third temperature uniformity deviation between each actual temperature value.

[0101] Step S304: Calculate the source space observation reward value based on the third humidity accuracy deviation, third humidity uniformity deviation, third temperature accuracy deviation, and third temperature uniformity deviation corresponding to each time step.

[0102] Step S305: Each source domain agent acquires the source space observation reward value of the entire source space end-to-end. The initial reinforcement learning model of each source domain agent is trained and its parameters are updated until convergence is achieved with the optimization objective of maximizing the sum of the source space observation reward values ​​in the future multiple steps. The updated model parameters of each source domain agent are then aggregated to obtain the source domain deep reinforcement learning model.

[0103] In steps S301 to S305, the pre-training form of the source domain deep reinforcement learning model in the source space is the same as that in the target space of this application. Both involve dividing the local space, configuring an agent in each local space, implementing reinforcement learning training in the local space, optimizing and tuning parameters using the observation reward value of the entire source space environment, and finally aggregating the model parameters trained by each agent to obtain the final source domain deep reinforcement learning model. This form is more in line with the final operating environment.

[0104] In some embodiments, in step S102, a first humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the target space and the set humidity of the target space, and a first humidity uniformity deviation between each actual humidity value are calculated, wherein the formula for calculating the first humidity accuracy deviation is:

[0105]

[0106] H sc This indicates the first humidity accuracy deviation. H represents the humidity detected by the i-th temperature and humidity sensor at time t. set β represents the target space humidity setting, and β represents the number of temperature and humidity sensors.

[0107] The formula for calculating the first humidity uniformity deviation is:

[0108]

[0109]

[0110] H unifThis indicates the first humidity uniformity deviation. This represents the humidity detected by the i-th temperature and humidity sensor at time t. This represents the average humidity value collected by all temperature and humidity sensors, and β represents the number of temperature and humidity sensors.

[0111] Calculate the first temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors within the target space and the set temperature of the target space, as well as the first temperature uniformity deviation among the actual temperature values. The formula for calculating the first temperature accuracy deviation is:

[0112]

[0113] T sc Table 1 shows the temperature accuracy deviation, T. t i T represents the temperature detected by the i-th temperature and humidity sensor at time t. set β represents the target space set temperature, and β represents the number of temperature and humidity sensors.

[0114] The formula for calculating the first temperature uniformity deviation is:

[0115]

[0116]

[0117] T unif Indicates the first temperature uniformity deviation, T t i This represents the temperature detected by the i-th temperature and humidity sensor at time t. This represents the average temperature collected by all temperature and humidity sensors, and β represents the number of temperature and humidity sensors.

[0118] Similarly, the calculation formulas for the second humidity accuracy deviation, the second humidity uniformity deviation, the second temperature accuracy deviation, the second temperature uniformity deviation, the third humidity accuracy deviation, the third humidity uniformity deviation, the third temperature accuracy deviation, and the third temperature uniformity deviation in this application can all refer to calculation formulas 1 to 6.

[0119] In some embodiments, the target space observation reward value is calculated based on the first humidity accuracy deviation, the first humidity uniformity deviation, the first temperature accuracy deviation, and the first temperature uniformity deviation corresponding to each time step, and the calculation formula is:

[0120] R t =α1(T sc +H sc )+α2(T unif +H unif (7)

[0121] Among them, R t H represents the observed reward value of the state at time t, where α1 and α2 are weighting coefficients. sc H represents the first humidity accuracy deviation. unif T represents the first humidity uniformity deviation. sc Indicates the first temperature accuracy deviation, T unif This indicates the first temperature uniformity deviation.

[0122] In some embodiments, each target domain agent trains and updates the parameters of the source domain deep reinforcement learning model to optimize the objective by maximizing the sum of reward values ​​from future multi-step target space observations until convergence, including:

[0123] Construct the optimization objective function, with the expression:

[0124]

[0125] Where γ represents the discount factor and γ < 1, R t+t′ (S t+t′ A t+t′ ) represents the target space observation reward value at time t+t′.

[0126] Similarly, in steps S201 to S205 and S301 to S305, the training and parameter update constraints of the source domain deep reinforcement learning model can also refer to Equations 7 and 8 above.

[0127] In some embodiments, the source domain deep reinforcement learning model constructs a neural network consisting of multiple fully connected layers to predict the Q value corresponding to each action. The neural network has two branches: the first branch is used to predict the state value, and the second branch is used to predict the advantage of each action. The Q value corresponding to each action is calculated by combining the predicted state value and the advantage of each action.

[0128] The source domain deep reinforcement learning model used in this invention is labeled as the nMAD3QN-PER (n-step Multi-Agent Double Dueling DQN with Prioritized Experience Replay) algorithm.

[0129] The nMAD3QN-PER network incorporates both Double DQN and Dueling DQN structures. Compared to traditional QLearning networks, DQN utilizes deep convolutional networks to approximate the value function and independently sets up a target network to handle the Time Difference (TD) bias in the time-of-flight scoring algorithm. It also uses experience replay to train the reinforcement learning process. In addition to the value function V, the Dueling DQN network adds an advantage function A, enabling Dueling DQN to more accurately estimate the Q-value and select the most suitable action after collecting data for only one discrete action. The Double DQN network selects the action with the largest output value in the Q-network and then chooses the corresponding target Q-network, thus avoiding the problem of Q-value overestimation. The nMAD3QN-PER network used in this application combines the advantages of both Dueling DQN and Double DQN networks.

[0130] Furthermore, nMAD3QN-PER introduces Prioritized ExperienceReplay, which samples more high-expectation-value experiences. Using the absolute value of TD error as a metric, it can effectively reduce the amount of experience required for learning and improve learning efficiency.

[0131] Furthermore, nMAD3QN-PER introduces the n-step method, which observes the rewards of multiple future steps and updates accordingly, further improving learning efficiency.

[0132] Furthermore, nMAD3QN-PER introduces multi-agent integration, including: First, reward value sharing: the reward value for all agents is the same, which is the reward value of the environmental state monitored by all agents. Each agent controls the vent's setting based on the temperature and humidity status monitored by its surrounding sensors, and global reward value sharing is achieved through methods including but not limited to a central controller or end-to-end communication between agents. Because the reward objective is consistent, the agents cooperate fully, avoiding algorithmic redundancy caused by competition. Second, independent strategy and control: since the nMAD3QN-PER network monitors the current state value and the behavioral advantages in that state, it can quickly learn efficient action selection for the agents. Although the agents migrate to the same algorithm network, after training Fine-Tune in the environment, each agent can adopt the optimal strategy and control action based on the state information it has collected, making regulation more efficient and precise. Third, it facilitates transfer learning: In multi-agent strategies, each agent is responsible for selecting only one behavior and employs an independent algorithm strategy, without needing to refer to environmental state information collected by other agents or their control behaviors. Except for communicating the global reward value externally, each agent can independently monitor the environmental state, select and execute behaviors, allowing for rapid independent application in transfer scenarios with varying numbers of agents after transfer. Multiple agents participate in training and control simultaneously, and each agent can perform low-latency, localized, distributed, and refined training directly on edge nodes, saving the high-frequency communication overhead of direct global control by a central controller and avoiding channel congestion caused by simultaneous multi-channel transmission.

[0133] Finally, an indoor temperature and humidity control method based on nMAD3QN-PER (n-step Multi-Agent Dueling Double DQN with Prioritized Experience Replay) was obtained.

[0134] On the other hand, the present invention also provides an indoor space temperature and humidity control system, the system comprising:

[0135] Multiple temperature and humidity sensors, including humidity sensors and temperature sensors, are installed within the target space;

[0136] The constant temperature and humidity machine is equipped with multiple air outlets, each air outlet has multiple wind speed settings, and the wind speed settings of each air outlet are set separately. Each air outlet is located within the target space. Each air outlet and the temperature and humidity sensors within a set range around it constitute a local control component.

[0137] A central controller is connected to a cloud server and connects to various temperature and humidity sensors in the target space and various air outlets of the constant temperature and humidity machine via the Internet of Things. Multiple target domain intelligent agents run on the central controller, and each target domain intelligent agent is used to control a local control component in the target space.

[0138] The central controller executes the above-described indoor space temperature and humidity control method to control the temperature and humidity within the target space.

[0139] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0140] The present invention will now be described with reference to a specific embodiment:

[0141] This embodiment provides a multi-agent transfer reinforcement learning method for precise and uniform indoor temperature and humidity control. Utilizing a multi-agent transfer reinforcement learning algorithm, indoor temperature and humidity are controlled to achieve a precise and uniform state, including the following:

[0142] 1) Divide the indoor space into multiple local areas, design a multi-agent reinforcement learning algorithm, and have multiple agents operate on each local area to cooperate in controlling the indoor constant temperature and humidity machine.

[0143] 2) Design the control system elements according to the simulation scenario. Since the control algorithm based on reinforcement learning is used, the three elements of the reinforcement learning algorithm need to be designed according to the current scenario: state, control action and reward.

[0144] 3) Simulate the actual warehouse scene in the CFD simulation software, initialize the simulation scene, and transmit the current environmental state (temperature and humidity) to the reinforcement learning algorithm control agent.

[0145] 4) The agent calculates the reward value of the current environment state based on the input environment state, selects the optimal action in the current state through reinforcement learning neural network training, and communicates the action to the actuator, i.e. the air outlet of the constant temperature and humidity machine.

[0146] 5) Once the temperature and humidity control unit completes its operation, it sends the next state to the agent for further learning. This cycle repeats, forming an interactive process between the agent and the environment.

[0147] 6) After training and learning, the intelligent body can select the optimal action according to the current state, so as to achieve a precise and uniform indoor temperature and humidity, and obtain a reinforcement learning model of the source domain space.

[0148] 7) When a new warehouse needs to achieve precise and uniform control of indoor temperature and humidity, the pre-trained agent strategy will be transferred to the agent of the new warehouse.

[0149] 8) The agent's algorithm network is fine-tuned and trained to generate the optimal decision-making strategy for the current room. In the event of external interference, the required response time is shortened, and the temperature and humidity are maintained at the optimal state.

[0150] Specifically, configure the system environment and structure:

[0151] Environment: The warehouse is equipped with a constant temperature and humidity system, which uses multiple sensors at multiple air inlets to regulate the indoor temperature and humidity. Because different materials require specific temperature and humidity environments for storage, strict control of the air temperature and humidity within the warehouse is essential.

[0152] Sensors: Temperature and humidity sensors periodically monitor environmental conditions and upload the collected data to a central controller via the Internet of Things (IoT). Large rooms often suffer from uneven indoor temperature and humidity. Therefore, distributed temperature and humidity sensors are deployed to detect humidity data at different locations within the indoor environment.

[0153] Central Controller: The central controller employs a transfer learning-based algorithm based on Multi-Agent Deep Reinforcement Learning (MADRL). The goal of the central controller is to maintain temperature and humidity within a desired and uniform range. The central controller consists of multiple agents. Environmental information and reward feedback from multiple sensors are uploaded to each agent, and each agent adjusts the fan speed at its corresponding vent, collaboratively making control decisions.

[0154] Actuator: The constant temperature and humidity air conditioning system will adjust the air speed of the air outlet according to the decision of the central controller. The air speed of different air outlets can be adjusted to different levels.

[0155] Policy transfer: The policy part of the source agent in the source room is transferred to the target agent in the target room, and the fine-tuning method is used to train the agent according to the environment of the target room.

[0156] During the control process, each air outlet is controlled by an agent to adjust the fan speed. When a room contains multiple air outlets, the same number of agents control each outlet. Each agent controls one air outlet and multiple sensors. Each sensor detects the environmental conditions of temperature and humidity at various points in the room and uploads the data to its corresponding agent. The sensor-uploaded information is used as input to train a deep neural network (nMAD3QN-PER) to obtain corresponding decisions, which are then output to the actuators. In this embodiment, the actuator refers to the air outlet of the Constant Temperature and Humidity (CTHA) system. After a certain time slot, the sensors upload newly collected data back to the agents. All agents in the room calculate the reward value for the actions performed in the previous time slot based on the current temperature and humidity conditions. It is important to emphasize that all agent rewards are the global reward value for the target room, ensuring consistency in control objectives across all agents. Based on this reward value, multi-agent reinforcement learning training is performed.

[0157] Therefore, as long as the thermodynamic environment of the target room is similar to that of the pre-trained source room, transfer learning can be used to transfer the front-end subnetwork of the agents in the source room to each agent in the target room. Each agent, based on the temperature and humidity data monitored by multiple sensors, controls the fan speed of one vent, with the overall temperature and humidity accuracy and uniformity of the room serving as the reward. Multiple agents within the room collaborate to fine-tune the V+A layer of the Dueling network to adapt to the current room's thermodynamic environment, thus obtaining the optimal temperature and humidity accuracy and uniformity control strategy.

[0158] The system model is then established and the problem is defined:

[0159] In an indoor environment equipped with a constant temperature and humidity control system, the control objective is to achieve a precise and uniform temperature and humidity. The target temperature and humidity values ​​are defined as T... set H set The temperature and humidity control unit operates by supplying air through multiple vents, thereby regulating the indoor temperature and humidity. Each vent is defined as {I1, I2, I3…I…} n Each air vent of the constant temperature and humidity unit can be adjusted to different air supply levels, defined as {f1, f2, f3, f4}. To monitor the indoor temperature and humidity, distributed sensors are deployed, namely {P1, P2…P…P…}. m The data detected by the sensor is uploaded to the controller, which makes a decision based on the current temperature and humidity status, thereby controlling the air outlet of the constant temperature and humidity machine.

[0160] For each air outlet of a constant temperature and humidity machine I i This system utilizes a single controller for individual control, primarily monitoring the temperature and humidity of its localized surrounding area. Several nearby sensors transmit their data to this controller. Therefore, indoor temperature and humidity are controlled by multiple controllers monitoring and controlling specific areas within the room, working together to achieve precise and uniform indoor temperature and humidity control.

[0161] Since the current indoor temperature and humidity depend only on the previous indoor state and the control actions of the temperature and humidity control system, and are independent of previous indoor states, this embodiment defines the deep reinforcement learning control process as a Markov decision process. Because the indoor air temperature and humidity in the next time slot are determined by the current indoor state and the actions of the CTHA system, and are independent of previous states, control optimization can be defined as a reinforcement learning problem.

[0162] 1) Indoor Temperature and Humidity Status: The MADRL-based controller makes decisions based on the current temperature and humidity status at multiple points indoors. Therefore, the status is a crucial factor. This embodiment deploys distributed sensors to detect indoor environmental information, including temperature and humidity at each point. In this embodiment, the temperature and humidity detected by the sensors at time t are...

[0163] 2) Temperature and humidity setpoints: In this embodiment, the target temperature and humidity are defined as T. set H set This value is determined by the optimal temperature and humidity for artifact preservation. Since it is nearly impossible to maintain the target temperature and humidity precisely, the expected deviations from the desired conditions are ±0.5℃ and ±1%, respectively. The goal of the algorithm in this embodiment is to minimize the deviation from the desired state.

[0164] 3) Wind Speed: The constant temperature and humidity system operates at a constant temperature and humidity. The MADRL-based intelligent agent mainly maintains the indoor temperature and humidity at a uniform and precise state by controlling the wind speed setting of the CTHA. In this embodiment, the wind speed setting is defined as F, F = {off, low, medium, high}. That is, there are four settings: off, low, medium, and high.

[0165] 4) For multi-agent agents Agent = {Ag 1 Ag 2 …Ag m}, where m is the number of air vents in the room, and each vent is controlled by an intelligent agent. For each Ag i Its state-action space is defined as follows:

[0166] System State: Control decisions (i.e., the setting value for each air vent) are based on observations of the current indoor temperature and humidity. The system state for each time slot is determined by Ag. i The system consists of the current temperature and humidity detected by the controlled β sensors. This embodiment defines the system state as follows:

[0167]

[0168]

[0169] in, This represents the system state monitored by the i-th agent at time t, and the system state changes over time. It includes the temperature and humidity states detected by β sensors within the region, where β represents the number of temperature and humidity sensors. Each reinforcement learning agent will make decisions based on the current input state.

[0170] Control Action: In this embodiment, the controllable variables are considered to be the wind speed settings of multiple air outlets in the CTHA system, and the single intelligent agent Ag. i Control is applied to a single air vent, with each vent offering four different speed settings, denoted as F. The control agent is defined as follows:

[0171]

[0172]

[0173] This indicates the four selectable modes for each air vent. Here, 'm' represents the number of air inlets. The current air vent can be adjusted to four different modes, and the states of each vent may differ at any given time. Therefore, the entire action state space is

[0174] 5) For the entire temperature and humidity control system, its reward is defined as follows: for each agent Ag... i The reward value is uniform and is calculated based on the environmental state monitored by all sensors in the room. When the agent performs an action in the current state, the environment will enter a new state and receive a reward. The reward function is based on equations 1 to 7 above. Since this embodiment mainly considers the accuracy and uniformity of indoor temperature and humidity, the reward value consists of two parts, and the calculation formula is based on equation 7:

[0175] R t =α1(T sc +H sc )+α2(T unif +H unif(7)

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182] The first term in Equation 7 calculates the temperature and humidity deviations from the target state between each point. This embodiment defines two variables: temperature accuracy T. sc Humidity accuracy H sc To measure the accuracy of indoor temperature and humidity. Among them... These represent the temperature and humidity detected by each sensor, respectively. This embodiment aims to ensure that the temperature and humidity at each point are as close as possible to the desired conditions.

[0183] The second term in Equation 7 primarily focuses on the uniformity of indoor temperature and humidity. The intelligent agent aims to reduce uneven temperature and humidity distribution within a room. Therefore, this embodiment defines the temperature uniformity deviation T. unif Humidity uniformity deviation H unif In equation 7, α i (i = 1, 2) represents the weights, indicating the relative importance of the two parts. If the accuracy of indoor temperature and humidity is more important, the parameter α1 should be set to a larger value. Otherwise, it should be adjusted to a smaller value to obtain higher uniformity. Since the control objective is to adjust the indoor temperature and humidity to a precise and uniform state, while also ensuring that the overall indoor environment reaches the target state, the global temperature and humidity state at the current time t will be used as the reward to evaluate the value of the action currently performed.

[0184] Optimization Objective: The agent judges the quality of its actions based on the results produced by the environment. Its goal is to learn a sequence of actions that enables it to achieve its objective. The objective function is expressed as follows:

[0185]

[0186] γ represents the discount factor and γ < 1. The goal of deep reinforcement learning is to maximize the sum of discounted rewards. For a single agent, the optimization objective is to maximize the sum of its rewards.

[0187] Furthermore, indoor temperature and humidity control is performed using reinforcement learning based on the aforementioned nMAD3QN-PER network, specifically through transfer learning.

[0188] Because deep reinforcement learning algorithms are designed for specific Markov decision processes, the optimal temperature and humidity control strategy model formulated using the nD3QN-PER algorithm in the current room may not be applicable to other rooms and may even lead to worse control performance. This is because while model-free deep reinforcement learning omits the complexity of modeling and learns through continuous interaction with the environment, it ultimately learns the dynamic thermodynamic model characteristics of a specific room, and further decision-making and control are based on these specific characteristics. In new application scenarios, the environment may differ significantly from the previous one; therefore, directly using existing models is not feasible. In such cases, retraining (learning from scratch) for the new thermodynamic environment is necessary, which consumes a large amount of computational resources and time, making it highly inefficient.

[0189] Since the temperature and humidity control in the source room is jointly controlled by multiple agents, each agent mainly monitors and controls the temperature and humidity of a local area and then obtains a reward by calculating the global state. In order to achieve the same goal in the target room, this embodiment proposes to transfer the strategy of a single agent in the source room to multiple agents in the target room, and then train them to jointly complete the cooperative control in the target room.

[0190] The reinforcement learning strategy of the agent is transferred, that is, the agent policy model of the source room is used as a pre-trained model, and then fine-tuned based on the model. Compared with directly training the agent of the target room, by transferring learning and model fine-tuning, it is possible to train to the local optimum faster, save a lot of computing resources and computing time, improve convergence efficiency, and improve the accuracy and uniformity of room temperature and humidity. This embodiment proposes to compare four different transfer methods: transferring the front-end self network, transferring the back-end sub-network, transferring the global network, and the mean fusion transfer network.

[0191] Transfer reinforcement learning is a learning algorithm that can adapt to different new tasks and obtain the corresponding optimal policy. The basic assumption of transfer reinforcement learning is that the source task and the target task used for transfer learning follow the same task distribution. Therefore, different tasks share some common characteristics. In deep reinforcement learning, the goal is to learn a function f with parameter θ. θ This function can minimize a specific task. loss function In the context of transfer reinforcement learning, the goal becomes learning a learning process. This learning process can utilize a very small dataset. Quickly adapt to new tasks The transfer reinforcement learning problem can be summarized as optimizing parameters θ and ψ during the learning process:

[0192]

[0193] in, They represent the sampling from Training and testing tasks, This represents the test loss function of the policy on the task after undergoing the learning algorithm.

[0194] The nD3QN-PER network consists of multiple fully connected layers with two branches: V(a) predicts the state value (a scalar) and A(s,a; θ,α) predicts the action advantage value (a vector with a length equal to the action space size). θ refers to the parameters of the network's convolutional layers, and α and β are the parameters of the fully connected layers in the two branches, respectively. Finally, the results from these two branches are combined to output Q(s,a; θ,α,β). The value function V reflects the goodness or badness of the current state s, the action value function Q describes the goodness or badness of choosing the action in the current state, and the advantage function A describes the importance of each action. In DQN, estimating multiple Q(s,a) for each state leads to algorithm instability, while calculating the value function V and the advantage function A separately enhances the algorithm's robustness.

[0195] Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α); (14)

[0196] Migrating the Front-End Network: The front-end subnetwork of the agents in the source room is migrated to each agent in the target room. Each agent, based on the temperature and humidity data monitored by β sensors, controls the fan speed of one vent, with the overall room temperature and humidity accuracy and uniformity as the reward. Multiple agents in the room collaborate to fine-tune the V+A layer of the Dueling network to adapt to the current room's thermodynamic environment, thereby obtaining the optimal temperature and humidity accuracy and uniformity control strategy.

[0197] Migrate the backend network: Migrate the two branches of the backend subnet and the V+A layer to each agent in the target room. Each agent, based on the temperature and humidity data from β sensors it monitors, controls the fan speed of one vent, with the overall room temperature and humidity accuracy and uniformity as the reward. Multiple agents in the room collaborate to fine-tune the network frontend to train it to adapt to the current room's thermodynamic environment, obtaining the optimal temperature and humidity accuracy and uniformity control strategy.

[0198] Global network migration: The algorithm network of the agent policies in the source room is directly applied to the agents in the target room. Each agent, based on the temperature and humidity data from β sensors it monitors, controls the fan speed of one vent, with the overall room temperature and humidity accuracy and uniformity as the reward value. Multiple agents in the room collaborate to obtain the optimal temperature and humidity accuracy and uniformity control strategy.

[0199] Federate transfer network: A multi-layer fully connected network of multiple agents in the source room is weighted and fused with equal weights. The fused global network is then transferred and applied to agents in the target room. Each agent, based on the temperature and humidity data from β sensors it monitors, controls the fan speed of one vent. The reward is the accuracy and uniformity of temperature and humidity in the entire room. Multiple agents in the room collaborate to obtain the optimal temperature and humidity accuracy and uniformity control strategy.

[0200] Specifically, the temperature and humidity control simulation was completed in CFD simulation software, and the algorithm part was completed using the open-source deep learning framework TensorFlow 2.0.

[0201] In the experiment, six different room models were designed to simulate the room, such as... Figures 4 to 9 The diagrams show six rooms, A, B, C, D, E, and F, each with a different area and layout. Each room contains multiple air inlets for temperature and humidity control units, and the air conditioning system regulates indoor temperature and humidity through these inlets. The diagrams also include multiple distributed sensors. Figure 4 In Model A, two air inlets are deployed, and each inlet is equipped with three sensors located close to it. The agent monitors the data from these three sensors to make adjustments. Figures 5 to 9 In the BCDE model, there are 3 air inlets, while in the F model, there are 4 air inlets.

[0202] In the experiment of this embodiment, T set =25℃, H set =50%. First, each room model is trained separately. For example, in room A, there are 3 agents. For each agent, the system state in the current time slot is defined as... Action is defined as A t ={f t 1 ,f t 2 ,f t 3},f∈F, f∈F, where f t 1 ,f t 2 ,f t 3 These represent the fan speed settings for the three vents. Multiple agents cooperate for training through global state feedback. The same principle applies to other models. Assume the room has good insulation and no heat exchange with the outside. To improve the control system's anti-interference capability, in each round, external disturbances cause significant changes in temperature and humidity in localized areas of the room. Multiple agents are trained collaboratively to cope with these disturbances and maintain a precise and uniform indoor temperature and humidity.

[0203] The MADRL neural network has two hidden layers, each with 512 neurons. ReLU is used as the activation function, and the Adam optimizer is employed with a learning rate of 0.000005. The discount factor is set to 0.9, and the mini-batch size is 32. This embodiment uses an ε-greedy strategy for development and exploration. ε is initially 1, and after 200 rounds, it decreases to 0.001. In multi-agent transfer experiments, different models are selected for transfer testing, and the transfer performance of different methods is compared in scenarios with the same and different numbers of agents.

[0204] A comparison of the transfer performance of different algorithms was conducted, evaluating the convergence of four different transfer algorithms and the direct training algorithm. The results are shown in [the table below]. Figure 10 and Figure 11 .

[0205] Training the D model is compared with different transfer methods using the C and E models. In the D, C, and E models, the number of agents is the same, and a corresponding transfer method is used. That is, for agents at corresponding positions, such as those near entrances / exits, the corresponding agents from the source model are selected for transfer. The figure shows the transfer performance for different transfer methods. It can be seen that when transferring the entire network (transferring all C as the source model to D, denoted as D-TC-All; transferring all E as the source model to D, denoted as D-TE-All), the convergence speed is faster than training the source model alone (D), but the convergence value is comparable to training alone. Using front-end transfer (transferring C as the source model front-end to D, denoted as D-TC-Front; transferring E as the source model front-end to D, denoted as D-TE-Front) or back-end transfer (transferring C as the source model back-end to D, denoted as D-TC-Back; transferring E as the source model back-end to D, denoted as D-TE-Back) results in faster convergence and higher convergence values ​​compared to training alone. Furthermore, when using front-end network transfer, the convergence value is higher and more stable. D-TC-FEDERARE indicates that C is used as the source model, and a mean-weighted transfer method is used for global network model transfer, specifically referring to the average weighting of the parameters of multiple agents in the source model.

[0206] From the above Figure 10 and Figure 11 It is evident that when the source and target models differ significantly or are similar, transferring the front-end network can reduce training time and improve performance. This is because, for MADRL network models, the front-end network processes input information and captures specific environmental state data, while the back-end network, through training, learns corresponding control strategies based on the front-end input information. Therefore, during transfer learning, for different models, the back-end network, primarily used to generate control strategies, is fine-tuned—that is, its control strategy, specifically suited to the target scenario, is retrained for each different scenario. Furthermore, due to previously accumulated control experience, it has a higher probability of quickly exploring better control strategies during retraining in new scenarios, resulting in a higher convergence value compared to the original training method.

[0207] Comparison of agent selection algorithms:

[0208] In practical deployment, for greater convenience, it is desirable to perform migration deployment without distinguishing the location of each agent. Therefore, this embodiment verifies the effectiveness of selecting any agent in the source model to migrate all agents in the target model. Similarly, this embodiment selects model D as the target model and model E as the source model, comparing the effects of the two agent selection methods. Figure 12As shown, similarly, the performance of different transfer methods was compared. D represents training model D alone; D-TE-All represents transferring all agents from E as the source model to D; D-TE-Front represents transferring the front-end of E as the source model to D; and D-TE-Front-RANDOM represents transferring a random front-end from multiple agents in the source model E to D. It is evident that for different transfer methods, less training time is needed to achieve the convergence value of training the target model alone. Comparing the two figures above, when transferring the front-end network, whether it's transferring each agent individually or arbitrarily selecting one agent from the source model to transfer all agents in the target model, the convergence effects of the two methods are comparable, with no significant difference in convergence values. Therefore, for better practical deployment, selecting one agent from the source model is more efficient.

[0209] Comparison of the number of different intelligent agents in the source space:

[0210] In addition, the migration effects were compared between source rooms with different numbers of agents. Similarly, this embodiment selected model D as the target model and models A, B, and C as the source models. That is, an agent was arbitrarily selected from models A, B, and C, and migrated to model D using a migration front-end network. Figure 13 As shown, D-TA-Front-RANDOM represents transferring a random front end from multiple agents in source model A to D; D-TB-Front-RANDOM represents transferring a random front end from multiple agents in source model B to D; and D-TC-Front-RANDOM represents transferring a random front end from multiple agents in source model C to D. D represents independent training. In model A, there are only two agents. It can be seen that regardless of the layout of the source model or the number of agents being equal or less, transfer learning can still converge to a higher value at a relatively fast convergence speed.

[0211] Comparison of accuracy and uniformity under interference:

[0212] To evaluate the accuracy and uniformity under disturbances, this embodiment tested for 60 rounds. Model D was used as the target model, and ABCE as the source model. In each round, this embodiment assumed that random temperature and humidity disturbances (e.g., temperature 27°C, humidity 45%) entered at random times, causing the temperature and humidity in certain locations indoors to deviate from the target values ​​(temperature: 25°C, humidity: 50%). The comparative effect is shown in the bar chart. Figure 14As shown, D represents training alone, D-TA represents transferring model A to D, D-TC represents transferring model C to D, D-TB represents transferring model B to D, and D-TE represents transferring model E to D. It is evident that the transfer method provides better model control, outperforming the source model's training alone in terms of accuracy and uniformity in temperature and humidity.

[0213] Figure 15 The study compares the control performance over 50 rounds of training using Model D alone and transfer training using Model C as the source model. The line with circular markers represents the transfer training result, while the line with triangular markers represents the independent training result. Significant improvements were observed in the accuracy of temperature and humidity measurement, with increases of 14.42% and 15.05% respectively. Improvements were also made in the uniformity of temperature and humidity measurement, with increases of 6.23% and 3.38% respectively.

[0214] Comparison of transfer algorithms for rooms with different numbers of agents:

[0215] In the above experiments, the policy transfer methods for source and target agents with 2-3 or 3-3 pairs were compared. To further verify the effectiveness of the proposed method, policy transfer comparisons for 2-4 and 3-4 pairs were also conducted. The F model was selected as the target model, and ABCE as the source model. Within the target model, any agent to be transferred was arbitrarily chosen, and its front-end network was transferred to the target agent, while its back-end network was trained. For example... Figure 16 As shown, F-TA-Front means transferring A as the source model front-end to F, F-TB-Front means transferring B as the source model front-end to F, F-TC-Front means transferring C as the source model front-end to F, and F-TE-Front means transferring E as the source model front-end to F. F represents independent training. When using the ACE model as the source model, its learning speed is faster than that of the F model, and the learning effect is better. The B model has a poor training effect because its layout is relatively simple and it has not accumulated efficient control experience for more complex environments.

[0216] columnar Figure 17 The study compares the control performance of different models across 50 different rounds. F represents training alone, F-TA represents transferring model A to F, F-TC represents transferring model C to F, F-TB represents transferring model B to F, and F-TE represents transferring model E to F. Overall, the performance is consistent with the training results. During transfer learning, the ACE model demonstrates significantly better control over temperature and humidity accuracy and uniformity when dealing with different disturbances compared to the original training method of the target model. This shows that good control performance can be achieved when the target model has 4 agents and the source model has 2 or 3 agents.

[0217] Similarly, as Figure 18 As shown, the performance was compared when E was used as the transfer model and when F was trained alone. The broken line with circular markers represents the transfer training result, and the broken line with triangular markers represents the independent training result. Similarly, the changes in temperature and humidity accuracy and uniformity were tested over 50 rounds. It is evident that under different disturbance conditions, the transfer strategy significantly outperforms the original strategy, validating the effectiveness of the proposed method. In terms of temperature accuracy and uniformity, the proposed transfer method improves by 16.53% and 17.10% respectively compared to the original training, and by 16.20% and 24.68% respectively in humidity.

[0218] Comparison of transfer algorithms when the number of sensors monitored by the intelligent agent varies

[0219] Considering that the number of sensors equipped for each agent in the target model may change during actual deployment, such as the target model only equipping an agent with two sensors, meaning the agent makes action decisions by monitoring data from two sensors, and since the source model learns data from three sensors, the change in the number of sensors during transfer learning leads to a change in the input dimension. Therefore, this embodiment proposes a mapping method, mapping the four-dimensional data of temperature and humidity from the two sensors in the target scene to information with the same input dimension as in the source scene. The F model is also selected as the target scene, and the training performance is compared when the ACE model is used as the source model. Figure 19 As shown, F represents training alone, F-TA represents transferring model A to F, F-TC represents transferring model C to F, and F-TE represents transferring model E to F. It is evident that, compared to training the original model F, transfer learning achieves superior learning results with a faster learning speed.

[0220] In the ACE model, the number of agents was 2, 3, and 3, respectively, and each agent was equipped with 3 sensors. The effectiveness of the proposed method was verified when the number of agents increased to 4 and the number of sensors decreased to 2. The transfer learning performance was compared under different perturbations with a reduced number of sensors. Figure 20 The bar chart results show that F represents training alone, F-TA represents transferring model A to F, F-TC represents transferring model C to F, and F-TE represents transferring model E to F. Despite the reduction in the number of sensors, the proposed transfer methods can still efficiently control indoor temperature and humidity, achieving better accuracy and uniformity. Among them, when using model E as the source model, the transfer effect is the best, improving the accuracy of temperature and humidity by 10.43% and 13.13%, and the uniformity by 13.69% and 14.45%, respectively.

[0221] Comparison of transfer algorithms for agents with different numbers of sensors

[0222] In the real world, to further reduce costs, the number of available sensors is limited in certain scenarios. Therefore, this embodiment will verify the effectiveness of the proposed method when the number of sensors monitored by some agents is reduced to 1 or 2. Figure 21 shows the learning curves of transfer learning for different source models when, in model F, two agents monitor 1 sensor each, and the remaining agents monitor 2 sensors each. F represents training alone, F-TA represents transferring model A to F, F-TC represents transferring model C to F, and F-TE represents transferring model E to F. Similarly, a mapping method is used to expand the input data dimension to match the source model, and the front-end network is transferred. The learning results are as follows: Figure 22 As shown, the mapping migration method can still function when the number of sensors changes significantly, and the accuracy and uniformity of temperature and humidity are improved.

[0223] In summary, the indoor space temperature and humidity control method and system of the present invention uses temperature and humidity state parameters as the state space and the airflow speed setting of the constant temperature and humidity machine outlet as the action space. A source domain deep reinforcement learning model is obtained through pre-training in the source space. In the target space, each actuator and the temperature and humidity sensors within a set range around it constitute a local control component. Each local control component is controlled by an independent target domain agent. Each target agent performs parameter fine-tuning and control by transferring and learning the source domain deep reinforcement learning model. The target space observation reward value is calculated by combining the temperature and humidity values ​​collected by all temperature and humidity sensors in the target space. The source domain deep reinforcement learning model of each target domain agent is trained separately with the goal of maximizing the future multi-step target space observation reward value, so as to quickly adapt to the precise control of temperature and humidity in the target space environment.

[0224] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0225] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0226] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0227] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0228] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for controlling indoor space temperature and humidity, characterized in that, The method is used to run on a central controller in a target space. The central controller is connected to a cloud server and connects to multiple temperature and humidity sensors and multiple actuators within the target space via the Internet of Things. The actuators are the air outlets of a constant temperature and humidity machine. Each actuator and the temperature and humidity sensors within a set range around it constitute a local control component. Multiple target domain intelligent agents run on the central controller, and each target domain intelligent agent is used to control a local control component within the target space. The method includes the following steps: A source domain deep reinforcement learning model, pre-trained in the source space, is obtained from the cloud server. This model uses temperature and humidity values ​​collected by multiple temperature and humidity sensors at specified intervals within the source space as state parameters to form the source space state space. The wind speed settings of each actuator operating at the set humidity and temperature within the source space are used as action parameters to form the source space action space. The source space observation reward value is calculated based on the state parameters. The source domain deep reinforcement learning model is trained and its parameters are updated to maximize the sum of future multi-step source space observation reward values ​​as the optimization objective until convergence. The source domain deep reinforcement learning model is transferred and distributed to each target domain agent within the target space. During the deep reinforcement learning process, each target domain agent uses the temperature and humidity values ​​collected by multiple temperature and humidity sensors in its controlled local control components at specified intervals as state parameters to form a local state space of the target domain, and uses the wind speed settings of each actuator operating at the set humidity and temperature of the target space as action parameters to form a local action space of the target domain. The first humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the target space and the set humidity of the target space, as well as the first humidity uniformity deviation between each actual humidity value, are calculated. The first temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors in the target space and the set temperature of the target space, as well as the first temperature uniformity deviation between each actual temperature value, are calculated. The target space observation reward value is calculated based on the first humidity accuracy deviation, the first humidity uniformity deviation, the first temperature accuracy deviation, and the first temperature uniformity deviation corresponding to each time step. Each target domain agent optimizes the objective by maximizing the sum of the target space observation reward values ​​in future multiple steps to train and update the parameters of the source domain deep reinforcement learning model until convergence. The training steps of the source domain deep reinforcement learning model include: The initial reinforcement learning model is sent to multiple source domain agents. Each source domain agent controls an actuator and a set range of temperature and humidity sensors around it. Each source domain agent uses the temperature and humidity values ​​collected by the multiple temperature and humidity sensors it controls at specified intervals as state parameters to form a local state space of the source domain. Each source domain agent uses the wind speed setting of the actuator it controls as the action parameter to form a local action space in the source domain. Calculate the third humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors in the source space and the set humidity of the source space, as well as the third humidity uniformity deviation between each actual humidity value; calculate the third temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors in the source space and the set temperature of the source space, as well as the third temperature uniformity deviation between each actual temperature value. The source space observation reward value is calculated based on the third humidity accuracy deviation, the third humidity uniformity deviation, the third temperature accuracy deviation, and the third temperature uniformity deviation corresponding to each time step; Each source domain agent acquires the source space observation reward value of the entire source space end-to-end. The initial reinforcement learning model of each source domain agent is trained and its parameters are updated until convergence is achieved with the optimization objective of maximizing the sum of the source space observation reward values ​​in the future multi-step process. The updated model parameters of each source domain agent are then aggregated to obtain the source domain deep reinforcement learning model.

2. The method for controlling indoor space temperature and humidity according to claim 1, characterized in that, The training steps of the source domain deep reinforcement learning model include: The temperature and humidity values ​​collected by multiple temperature and humidity sensors in the source space at specified intervals are used as state parameters, and the source space is used as a whole to form the overall state space. The wind speed settings of each actuator operating within the source space with the set humidity and set temperature of the source space are obtained as action parameters to construct the overall action space. Calculate the second humidity accuracy deviation between the actual humidity values ​​of all humidity sensors in the source space and the set humidity of the source space, as well as the second humidity uniformity deviation between each actual humidity value; calculate the second temperature accuracy deviation between the actual temperature values ​​of all temperature sensors in the source space and the set temperature of the source space, as well as the second temperature uniformity deviation between each actual temperature value. The source space observation reward value is calculated based on the second humidity accuracy deviation, the second humidity uniformity deviation, the second temperature accuracy deviation, and the second temperature uniformity deviation corresponding to each time step; The initial reinforcement learning model is trained and its parameters are updated until convergence is achieved by maximizing the sum of future multi-step source space observation rewards. This yields the source domain deep reinforcement learning model.

3. The method for controlling indoor space temperature and humidity according to claim 1, characterized in that, The updated model parameters of each source domain agent are aggregated, including: The updated parameters of each source domain agent are fused according to the set weights to obtain the parameters of the source domain deep reinforcement learning model.

4. The method for controlling indoor space temperature and humidity according to claim 1, characterized in that, Calculate the first humidity accuracy deviation between the actual humidity values ​​of all temperature and humidity sensors within the target space and the set humidity of the target space, and the first humidity uniformity deviation between each actual humidity value. The formula for calculating the first humidity accuracy deviation is: H sc This indicates the first humidity accuracy deviation. H represents the humidity detected by the i-th temperature and humidity sensor at time t. set β represents the target space humidity setting, and β represents the number of temperature and humidity sensors. The formula for calculating the first humidity uniformity deviation is: H unif This indicates the first humidity uniformity deviation. This represents the humidity detected by the i-th temperature and humidity sensor at time t. This represents the average humidity value collected by all temperature and humidity sensors, where β represents the number of temperature and humidity sensors. Calculate the first temperature accuracy deviation between the actual temperature values ​​of all temperature and humidity sensors within the target space and the set temperature of the target space, and the first temperature uniformity deviation between the actual temperature values. The formula for calculating the first temperature accuracy deviation is: T sc T represents the first temperature accuracy deviation. t i T represents the temperature detected by the i-th temperature and humidity sensor at time t. set β represents the target space set temperature, and β represents the number of temperature and humidity sensors. The formula for calculating the first temperature uniformity deviation is: T unif T represents the first temperature uniformity deviation. t i This represents the temperature detected by the i-th temperature and humidity sensor at time t. This represents the average temperature collected by all temperature and humidity sensors, and β represents the number of temperature and humidity sensors.

5. The method for controlling indoor space temperature and humidity according to claim 4, characterized in that, The target space observation reward value is calculated based on the first humidity accuracy deviation, the first humidity uniformity deviation, the first temperature accuracy deviation, and the first temperature uniformity deviation corresponding to each time step. The calculation formula is as follows: R t =α1(T sc +H sc )+α2(T unif +H unif ); Among them, R t H represents the observed reward value of the state at time t, where α1 and α2 are weighting coefficients. sc H represents the first humidity accuracy deviation. unif T represents the first humidity uniformity deviation. sc T represents the first temperature accuracy deviation. unif This indicates the deviation in the first temperature uniformity.

6. The method for controlling indoor space temperature and humidity according to claim 5, characterized in that, Each target domain agent optimizes the objective by maximizing the sum of reward values ​​from future multi-step target space observations, training and updating the parameters of the source domain deep reinforcement learning model until convergence, including: Construct the optimization objective function, with the expression: Where γ represents the discount factor and γ < 1, R t+t′ (S t+t′ A t+t′ ) represents the target space observation reward value at time t+t′.

7. The method for controlling indoor space temperature and humidity according to claim 1, characterized in that, The source domain deep reinforcement learning model constructs a neural network composed of multiple fully connected layers to predict the Q value corresponding to each action. The neural network has two branches: the first branch is used to predict the state value, and the second branch is used to predict the advantage of each action. The Q value corresponding to each action is calculated by combining the predicted state value and the advantage of each action.

8. An indoor space temperature and humidity control system, characterized in that, The system includes: Multiple temperature and humidity sensors, including humidity sensors and temperature sensors, are installed within the target space; The constant temperature and humidity machine is equipped with multiple air outlets, each air outlet has multiple wind speed settings, and the wind speed settings of each air outlet are set separately. Each air outlet is located within the target space. Each air outlet and the temperature and humidity sensors within a set range around it constitute a local control component. A central controller is connected to a cloud server and connects to various temperature and humidity sensors in the target space and various air outlets of the constant temperature and humidity machine via the Internet of Things. Multiple target domain intelligent agents run on the central controller, and each target domain intelligent agent is used to control a local control component in the target space. The central controller executes the indoor space temperature and humidity control method according to any one of claims 1 to 7 to control the temperature and humidity in the target space.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images