Intelligent building air conditioner adaptive control method based on deep reinforcement learning

Abstract

The application discloses an intelligent building air conditioner self-adaptive control method based on deep reinforcement learning, and belongs to the technical field of intelligent building control, which specifically comprises the following steps: collecting multiple information of an intelligent building, combining a three-dimensional model with an air conditioner position to construct a three-dimensional space node network; defining an air conditioner control action range according to the three-dimensional space node network, forming a discrete action space, and constructing a multi-objective reward function integrated with regional weights; adopting a Dueling DQN algorithm, combining an experience replay and a target network mechanism to train a deep reinforcement learning model, and deploying the deep reinforcement learning model to a building control system; during operation, acquiring state data through the node network, selecting an optimal action from the action space to control the air conditioner, collecting new state data after the air conditioner executes the action, feeding back a reward value to the model according to the reward function, and dynamically adjusting an optimal action selection strategy; the method can accurately adapt to a complex environment of the building, balance comfort and energy consumption, and improve the intelligent level and energy utilization efficiency of intelligent building air conditioner control.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent building control, and particularly relates to an intelligent building air conditioner adaptive control method based on deep reinforcement learning. BACKGROUND

[0002] With the acceleration of urbanization, the number of intelligent buildings is increasing, and the air conditioning system is one of the largest energy-consuming devices in intelligent buildings. Its efficient control is of great significance for reducing building energy consumption and achieving the goal of energy saving and emission reduction. Traditional building air conditioning control methods mostly use preset control strategies, such as PID control based on temperature set value. However, the building environment has high complexity and uncertainty. Different seasons, different time periods, different regional personnel activities and external climate conditions will have a significant impact on air conditioning load. The preset control strategy cannot adapt to these changes in real time, resulting in serious energy waste and difficulty in ensuring indoor comfort.

[0003] In recent years, artificial intelligence technology has developed rapidly, providing a new way for building air conditioning control. Some machine learning-based methods have been tried for air conditioning control, such as predicting air conditioning load by learning historical data and then adjusting control parameters. However, these methods mostly rely on a large amount of labeled data, and the model has limited generalization ability. When facing complex and variable building environments, it is still difficult to achieve efficient adaptive control. Deep reinforcement learning, as a technology combining the powerful perception ability of deep learning and the autonomous decision-making ability of reinforcement learning, can enable agents to continuously learn optimal strategies in interaction with the environment, and is expected to bring innovative solutions to intelligent building air conditioning control. However, there are still many problems to be solved in the effective application of deep reinforcement learning to intelligent building air conditioning adaptive control, such as how to design a reasonable state space, action space and reward function to achieve efficient learning and precise control. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides an intelligent building air conditioner adaptive control method based on deep reinforcement learning, collects multi-element information of the intelligent building, constructs a three-dimensional space node network in combination with a three-dimensional model and air conditioner positions; according to this, the range of air conditioner control actions is defined, a discrete action space is formed, and a multi-objective reward function incorporating regional weights is constructed; the Dueling DQN algorithm is used to train a deep reinforcement learning model in combination with an experience replay and target network mechanism, and the model is deployed to a building control system; in operation, state data is acquired through the node network, an optimal action is selected from the action space to control the air conditioner, new state data is collected after the air conditioner executes the action, a reward value is calculated according to the reward function and fed back to the model, and the optimal action selection strategy is dynamically adjusted; the method can accurately adapt to the complex environment of the building, balance comfort and energy consumption, and improve the intelligent level and energy utilization efficiency of intelligent building air conditioner control.

[0005] To achieve the above object, the present application provides the following technical scheme:

[0006] The intelligent building air conditioner adaptive control method based on deep reinforcement learning comprises:

[0007] S1: collecting multi-element information of an intelligent building area, constructing a three-dimensional space node network in combination with a three-dimensional model of the intelligent building and air conditioner positions;

[0008] S2: defining the range of control actions of the air conditioner system according to the three-dimensional space node network, forming a discrete action space;

[0009]

[0009] S3: constructing a multi-objective reward function, setting regional weights in combination with the state differences of different regions in the three-dimensional space node network;

[0010] S4: training a pre-constructed deep reinforcement learning model according to the three-dimensional space node network, the discrete action space and the multi-objective reward function, obtaining a trained deep reinforcement learning model by introducing an experience replay mechanism and a target network mechanism, and deploying the model to a control system of the intelligent building;

[0011] S5: real-time monitoring in the control system, acquiring current state representation data through the three-dimensional space node network, selecting an optimal action from the discrete action space according to the current state representation data and outputting the optimal action to the air conditioner system;

[0012] S6: after the air conditioner system executes the optimal action, collecting new state representation data again through the three-dimensional space node network, calculating a reward value after the execution of the action according to the multi-objective reward function and feeding back the reward value to the deep reinforcement learning model, and dynamically adjusting a control strategy of the deep reinforcement learning model according to the feedback result; the control strategy is an optimal action selection process;

[0013] The specific steps of S4 comprise:

[0014] S4.1: Using the node information in the three-dimensional space node network G, the node state vector of each node is aggregated by the area weight obtained in S3 to obtain the global state representation s of the intelligent building;

[0015] S4.2: Obtain the discrete action space in S2; the discrete action space contains all feasible combinations of air conditioner control actions;

[0016] S4.3: Calculate the reward value according to the global state representation s of the intelligent building and the discrete action combination in the discrete action space using the constructed multi-objective reward function;

[0017] S4.4: Load the pre-constructed Dueling DQN model architecture; the network architecture of the Dueling DQN includes a feature extraction layer, an advantage function flow and a state value function flow;

[0018] S4.5: Randomly initialize the main network parameters and target network parameters of the pre-constructed Dueling DQN model, and initialize the experience replay buffer at the same time; the experience replay buffer is used to store the state-action-reward-next state four-tuple;

[0019] S4.6: Train the pre-constructed Dueling DQN model, when the training reaches the preset number of iterations or meets the convergence condition, the training is ended, and the trained Dueling DQN model is deployed to the control system of the intelligent building;

[0020] The reward value in S4.3 needs to be calculated in steps to calculate the reward values of the temperature deviation, energy consumption function and air quality function in the multi-objective reward function, and then weighted sum is realized. The formula for calculating the reward values of the temperature deviation, energy consumption function and air quality function in the multi-objective reward function is:

[0021] ;

[0022] Wherein, a represents the executed action, represents the temperature comfort reward, represents the energy efficiency reward, represents the air quality reward, represents the weight of the node of the sub-area type, represents the temperature deviation penalty function, and , represents the temperature of the node , represents the set temperature, represents the temperature change amount, represents the basic power consumption of the air conditioner, P k a represents the power consumption of the kth air conditioner under action a, and satisfies: , P k a represents the control action of the kth air conditioner, T k represents the current temperature of the region where the kth air conditioner is located, which is the node temperature extracted from the global state representation s, T o represents the outdoor temperature, and scale represents the power scaling coefficient, t represents the current time, C k represents the CO2 concentration of the node , C k represents the CO2 concentration, C k represents the CO2 deviation penalty function, and satisfies: , C k represents the CO2 concentration change amount, max represents the maximum function.

[0023] Specifically, the specific steps of S1 include:

[0024] S1.1: Collect multi-element information of the intelligent building area, import a three-dimensional model of the intelligent building, and obtain the air conditioner position, and at the same time, perform preprocessing;

[0025] S1.2: Divide the three-dimensional space of the intelligent building into N irregular sub-regions according to the building function type, define each sub-region as a node, form a node set , and assign the preprocessed multi-element information to the corresponding node to form a node state vector; the building function type includes but is not limited to office, conference room, corridor, wherein, represents the i th sub-region or the i th node;

[0026] S1.3: According to the node and its position coordinates in the three-dimensional space, define the spatial connection relationship between the nodes; the spatial connection relationship includes:

[0027] If the sub-regions of two nodes share an entity edge in the three-dimensional space, an adjacency relationship is established, and the weight is adjusted according to the physical properties of the shared edge to obtain an adjacency matrix A and a weight matrix W;

[0028] S1.4: Determine the node range affected by each air conditioning equipment by CFD simulation , wherein k represents the air conditioning equipment index;

[0029] If the node center coordinates are located within the node range affected by the air conditioning equipment , , a connection edge is established, and the operating parameters of the air conditioner are taken as the edge attributes of the connection edge to form a connection edge set ;

[0030] S1.5: integrate the node set V, the adjacency matrix A, and the connection edge set , and output a three-dimensional spatial node network , wherein represents a union set.

[0031] Specifically, the specific steps of S2 include:

[0032] S2.1: Obtain the three-dimensional spatial node network, and extract the attributes of each air conditioning device from the connection edge set based on the three-dimensional spatial node network; the attributes of the air conditioning device include type, energy efficiency ratio, maximum power, and coordinates;

[0033] S2.2: Associate the air conditioning device with the node state to obtain an air conditioning device attribute table associated with the state; the association content includes:

[0034] For each , bind the current temperature and the personnel density of the node ;

[0035] S2.3: Based on the basic function of the air conditioning system, obtain a basic control action set, determine the feasible range of the basic control action in combination with the temperature state vector and the environmental state component of each node in the three-dimensional spatial node network, and simultaneously determine an extended control action set in combination with the personnel quantity and activity intensity information of each region in the node network and the running state parameters of the air conditioning system itself;

[0036] S2.4: According to the control precision of the air conditioning device, discretely divide the basic control action set and the extended control action set, and combine the discrete divisions of the basic control action and the extended control action to form a discrete action combination candidate set;

[0037] S2.5: Determine the boundary condition of the action space and set the constraint condition between actions according to the physical characteristics and safety specifications of the air conditioning system;

[0038] The boundary condition of the action space includes total power limitation and compressor protection.

[0039] The constraint condition between actions is that the temperature difference between two adjacent time points is less than or equal to 1.

[0040] S2.6: Combine all the discrete actions that have been discretized and meet the boundary condition and the constraint condition, and assign a unique ID to each action to form a discrete action space.

[0041] ​Specifically, the specific process of training the pre-built Dueling DQN model in S4.6 includes:

[0042] S4.61: initializing the environment state of the intelligent building , selecting an action from the discrete action space according to the current global state representation ;

[0043] S4.62: the air conditioning system performs the action , and calculates the final reward value according to the multi-objective reward function, and obtains the new state through the three-dimensional space node network , and stores the experience sample into the experience replay buffer;

[0044] S4.63: if the buffer is full, discard old samples according to the FIFO principle;

[0045] S4.64: randomly sample M experience samples from the experience replay buffer, and calculate the target value using the target network;

[0046] S4.65: calculate the mean square error loss value between the output of the current network and the target value;

[0047] S4.66: according to the mean square error loss value, use the optimizer to perform one-step gradient descent to update the current network parameters;

[0048] S4.67: copy the current network parameters to the target network parameters every C steps, and set the next state as the current state.

[0049] Specifically, the specific steps of S4.61 include:

[0050] Determine the intelligent building environment state factors, and set initial values for each intelligent building environment state factor;

[0051] Set a probability u, and satisfy ;

[0052] Generate a random number g, and satisfy ;

[0053] If , randomly select an action from the discrete action space;

[0054] If , calculate the reward value of each action using the pre-built Dueling DQN model according to the current state representation, and select the action with the highest reward value.

[0055] Specifically, the specific steps of S4.62 include:

[0056] action index outputted by the Dueling DQN model decoded into an air conditioner control instruction and issued to a building automation system through a Modbus RTU protocol:

[0057] The building automation system receives and executes the air conditioner control instruction, and simultaneously collects actual parameters after execution of the air conditioner;

[0058] The new state data is obtained through the nodes of the three-dimensional space node network, and the actual parameters after execution of the air conditioner are combined to dynamically adjust the sub-region weights;

[0059] Based on the new state data and the adjusted sub-region weights, a new global state is generated using a weighted average method;

[0060] Based on the new state data, the actual parameters after execution of the air conditioner, and the decoded air conditioner control instruction, a final reward value is calculated using a multi-objective reward;

[0061] The new state data, action index, final reward value, and new global state are combined into a four-tuple to obtain a transition sample, and the transition sample is stored in an experience replay buffer.

[0062] Specifically, the specific steps of S5 include:

[0063] S5.1: Synchronize the collection of real-time data of each node from the sensor network of the three-dimensional space node network, and perform preprocessing to generate a real-time node state vector set;

[0064] S5.2: Weighted aggregation of real-time node state vectors according to regional weights to generate a real-time global state representation;

[0065] S5.3: Convert the real-time global state representation into a tensor format, and at the same time, add a batch dimension to form an input format that can be processed by the Dueling DQN model; the dimension is the same as the input layer dimension of the Dueling DQN model;

[0066] S5.4: Calculate the Q values of all possible actions through the trained Dueling DQN model, and select the action index with the maximum Q value;

[0067] S5.5: Map the action index with the maximum Q value to actual control parameters; the actual control parameters include temperature set value and wind speed gear; the mapping execution process mainly refers to a predefined decoding table: wherein, indicates the action index with the maximum Q value, FS indicates the wind speed gear, indicates the Dueling DQN model, indicates a decoding function, and ControlParams indicates actual control parameters;

[0068] S5.6: According to the air conditioning system communication protocol, the actual control parameters are converted into an instruction frame by CF = ProtocolEncode (ControlParams), wherein CF represents the instruction frame, ProtocolEncode represents a protocol encoding function;

[0069] S5.7: The instruction frame is sent to the corresponding air conditioning device through the network interface, and an execution log is recorded.

[0070] Specifically, the multi-element information includes a temperature state vector of the intelligent building area, temperature gradient information between areas, environmental state components, the number of personnel and activity intensity information of each area, and operation state parameters of the air conditioning system itself; and the control action includes adjustment of a refrigeration or heating mode, multi-gear setting of a fan speed, and adjustment of a fresh air valve opening degree.

[0071] Compared with the prior art, the beneficial effects of the present application are:

[0072] The present application proposes an intelligent building air conditioner adaptive control method based on deep reinforcement learning, which realizes efficient and intelligent air conditioner management through multi-step collaborative design, has significant beneficial effects, first, by constructing a three-dimensional space node network, a discrete action space and a multi-objective reward function, combining with deep reinforcement learning model training, a complete intelligent control system is established, which can comprehensively integrate the multi-element information of the intelligent building, including environmental parameters, personnel activities, air conditioning equipment properties, etc., accurately define the air conditioning control action range, form a scientific reward mechanism, so that the deep reinforcement learning model can learn the optimal control strategy considering energy saving and comfort.

[0073] The present application proposes an intelligent building air conditioner adaptive control method based on deep reinforcement learning, in actual operation, the system continuously optimizes the control strategy through real-time monitoring and dynamic feedback mechanism, the control system selects the optimal action to control the air conditioner operation according to the real-time state data obtained by the three-dimensional space node network, and after the action is executed, new data can be collected to calculate the reward value, which is fed back to the model to further optimize the parameters, this closed-loop control mode not only improves the response speed and control accuracy of the air conditioning system, but also adapts to the dynamic changes of the building environment, while ensuring the indoor environment comfort, effectively reduces the energy consumption. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a schematic diagram of the present application based on deep reinforcement learning intelligent building air conditioner adaptive control method;

[0075] Figure 2 is a principle flow chart of the present application based on deep reinforcement learning intelligent building air conditioner adaptive control method;

[0076] Figure 3 The figure is a flow chart of the three-dimensional space node network construction process of the intelligent building air conditioner adaptive control method based on deep reinforcement learning. DETAILED DESCRIPTION

[0077] Embodiment 1

[0078] Please refer to Figure 1 and Figure 2 An embodiment provided by the present application is an intelligent building air conditioner adaptive control method based on deep reinforcement learning, which comprises the following steps:

[0079] S1: Collecting multi-element information of the intelligent building area, combining the three-dimensional model of the intelligent building and the air conditioner position, and constructing a three-dimensional space node network;

[0080] The multi-element information includes temperature state vectors of the intelligent building area, temperature gradient information between areas, environmental state components, personnel quantity and activity intensity information of each area, and running state parameters of the air conditioning system itself.

[0081] It should be noted that the environment and demand of different areas in the intelligent building are very different, and it is difficult to achieve precise control relying on single data alone. By combining multi-element information, a three-dimensional model and an air conditioner position to construct a node network, the spatial relationship and state information between each area and the air conditioner can be clearly reflected.

[0082] For example, through the node network, it can be known which areas are close to a certain air conditioner and which areas are greatly affected by a certain air conditioner, so that the air conditioner can be controlled more targetedly.

[0083] S2: Defining the control action range of the air conditioning system according to the three-dimensional space node network, and forming a discrete action space;

[0084] The control actions include adjustment of cooling or heating mode, multi-gear setting of fan speed and adjustment of fresh air valve opening degree.

[0085] It should be noted that the discrete action space provides a set of selectable control actions for the model. In the decision-making process, the model can select appropriate actions from this set to adjust the running state of the air conditioner to meet the needs of different areas. For example, the action space can include different cooling or heating modes, fan speed gears and fresh air valve opening degrees.

[0086] S3: Constructing a multi-objective reward function, setting area weights in combination with the state differences of different areas in the three-dimensional space node network;

[0087] It should be noted that the reward function is the guide for the deep reinforcement learning model to learn, and the goal of the model is to maximize the reward value. By reasonably setting the reward function and the regional weight, the model can learn to take different control actions in different regions to balance multiple goals such as comfort, energy consumption, and air quality. For example, a higher weight can be set for a densely populated area to ensure the comfort of the area.

[0088] Further, the specific steps of S3 include:

[0089] (1) Obtain indoor temperature data from each node in the three-dimensional spatial node network, determine the set comfortable temperature range, at the same time, collect real-time energy consumption information of the air conditioning system, and collect indoor air quality data; The comfortable temperature range is usually a temperature interval determined according to human comfort research and actual use requirements; The real-time energy consumption information of the air conditioning system is obtained through the energy consumption monitoring device installed on the air conditioning equipment or power supply line; The indoor air quality data is obtained through the air quality sensor;

[0090] (2) For each node, the deviation of its indoor temperature from the set comfortable temperature range is calculated by difference;

[0091] (3) Construct an energy consumption function according to the real-time energy consumption information of the air conditioning system; The energy consumption function adopts a linear function, which is obtained by multiplying the real-time energy consumption information of the air conditioning system by a normal number;

[0092] (4) Construct an air quality function according to the indoor air quality index, wherein the construction of the air quality function needs to be determined according to the relationship between the air quality index and the comfort, for example: when the air quality index b is in the ideal range , when the air quality index b satisfies the condition: or , wherein is a normal number, used to adjust the weight of air quality in the reward function, is the air quality function, and respectively represent the minimum value and the maximum value of the ideal value of the air quality index;

[0093] (5) Combine the temperature deviation, the energy consumption function and the air quality function by weighted summation to construct a multi-objective reward function;

[0094] (6) Analyze the state differences of different areas in the three-dimensional space node network, which may include differences in personnel density, use function, temperature change trend, etc. For example, areas with high personnel density may have higher requirements for temperature and air quality, or the temperature in some areas may fluctuate more easily due to external environmental influences;

[0095] (7) According to the state differences of different areas, set corresponding weights for each area, wherein the selection method of the weights is: by establishing a comprehensive evaluation model, quantifying and weighting the sum of factors such as personnel density, use function, temperature change trend, to obtain the comprehensive score of each area, and then normalize to obtain the area weight.

[0096] S4: According to the three-dimensional space node network, the discrete action space and the multi-objective reward function, the pre-constructed deep reinforcement learning model is trained, and through the introduction of experience replay mechanism and target network mechanism, the trained deep reinforcement learning model is obtained and deployed to the control system of the intelligent building;

[0097] The optimal action can be selected from the discrete action space according to the state information provided by the three-dimensional space node network to maximize the multi-objective reward function. The experience replay mechanism and the target network mechanism can improve the learning efficiency and stability of the model, so that the model can better learn the optimal control strategy.

[0098] S5: Real-time monitoring in the control system, obtaining the current state representation data through the three-dimensional space node network, selecting the optimal action from the discrete action space according to the current state representation data and outputting it to the air conditioning system;

[0099] S6: After the air conditioning system executes the optimal action, new state representation data is collected again through the three-dimensional space node network, the reward value after the execution of this action is calculated according to the multi-objective reward function and fed back to the deep reinforcement learning model, and the deep reinforcement learning model dynamically adjusts the control strategy according to the feedback result; the control strategy is the optimal action selection process.

[0100] Further, the specific steps of S6 include:

[0101] (1) The sensors in the three-dimensional space node network collect environmental data of each node at a certain sampling frequency and accuracy. After preprocessing such as noise removal and data normalization, these data are combined into a new state representation vector;

[0102] (2) input the new state representation vector into the multi-objective reward function, the function calculates a comprehensive reward value according to the weights of different objectives and the difference between the state variable and the target value, for example, if the objective is to keep the indoor temperature within the comfortable range while reducing energy consumption, the reward function will calculate the reward according to the closeness of the actual temperature to the comfortable temperature and the change of energy consumption caused by the action;

[0103] (3) the reward value is transmitted to the Dueling DQN model as a feedback signal, the Dueling DQN model stores the current state, action, reward and next state as an experience sample into the experience replay buffer, then the Dueling DQN model randomly selects a batch of experience samples from the experience replay buffer according to a certain learning strategy, and updates the parameters of the model by optimizing the objective function, such as minimizing the loss function, so as to adjust the control strategy.

[0104] Embodiment 2

[0105] Please refer to Figure 3 The specific steps of S1 in the embodiment include:

[0106] S1.1: collect multi-element information of the intelligent building area, import the three-dimensional model of the intelligent building, and obtain the air conditioner position, and at the same time, perform preprocessing;

[0107] S1.2: divide the three-dimensional space of the intelligent building into N irregular sub-regions according to the building function type, define each sub-region as a node, and form a node set , and assign the preprocessed multi-element information to the corresponding node to form a node state vector; the building function type includes but is not limited to office, conference room, corridor, wherein, represents the i-th sub-region or the i-th node;

[0108] Further, the process of building function sub-region division and node definition includes:

[0109] According to the design drawings and actual use functions of the intelligent building, different functional areas such as office, conference room, corridor, elevator hall and machine room are identified;

[0110] Each functional area is further divided into irregular sub-regions, and the spatial continuity, functional consistency and air conditioning system coverage range are considered during the division, for example, a large office can be divided into multiple sub-regions according to internal partitions, personnel distribution density and other factors;

[0111] Each irregular sub-region is defined as a node, and each node has a unique identifier and spatial position information.

[0112] S1.3: defining the spatial connection relationship between nodes according to the nodes and the coordinates of the nodes in the three-dimensional space; the spatial connection relationship includes:

[0113] If the sub-regions to which the two nodes belong share a physical boundary in the three-dimensional space, an adjacency relationship is established , and the weight is adjusted according to the physical properties of the shared physical boundary to obtain an adjacency matrix A and a weight matrix W, wherein, represents the adjacency relationship between the i-th node and the j-th node, 1 represents connection, and if , it represents that the i-th node and the j-th node are not adjacent;

[0114] wherein the weight of the shared physical boundary between the i-th node and the j-th node satisfies: , represents a decay coefficient, and , represents the thermal resistance of the partition wall, and is obtained through a three-dimensional model library of an intelligent building.

[0115] For example, in the three-dimensional space, if two nodes share a common boundary, such as a common wall between adjacent offices, a connection is established between them to construct a preliminary spatial topology between nodes.

[0116] S1.4: determining the range of nodes affected by each air conditioning device through CFD simulation , wherein k represents the index of the air conditioning device;

[0117] If the node center coordinates are located within the range of nodes affected by the air conditioning device , a connection edge is established , and the operating parameters of the air conditioning device are taken as the edge attributes of the connection edge to form a connection edge set .

[0118] It should be noted that CFD (Computational Fluid Dynamics, CFD) simulation is a full name of computational fluid dynamics simulation, which mainly simulates the actual fluid flow situation by virtually doing experiments on a computer. The basic principle is to numerically solve the differential equations governing fluid flow to obtain the discrete distribution of the flow field of the fluid flow in the continuous region, thereby approximating the fluid flow situation. The specific simulation process of CFD is prior art content in the art and is not the inventive scheme of the present application, and will not be described here.

[0119] S1.5: integrating the node set V, the adjacency matrix A, and the connection edge set to output a three-dimensional spatial node network wherein, denotes the union.

[0120] It is emphasized that in the present invention, G denotes the constructed three-dimensional spatial node network, which is a weighted directed graph or undirected graph, V denotes the node set, corresponding to the sub-regions of the building, such as offices, conference rooms, corridors, etc., A denotes the set of physical space adjacency relationships between nodes, such as the adjacency relationship of nodes sharing edges, denotes the set of connection edges between the air conditioning equipment and the affected nodes, W denotes the strength of the connection between the nodes, denotes the edge set after merging the connection edge set and the physical space adjacency relationship between nodes, and the merged set contains all types of edges in the graph, wherein the graph refers to the three-dimensional spatial node network.

[0121] The specific steps of S2 include:

[0122] S2.1: Obtain the three-dimensional spatial node network, and extract the attributes of each air conditioning equipment from the connection edge set based on the three-dimensional spatial node network The attributes of the air conditioning equipment include type, energy efficiency ratio, maximum power, and coordinates.

[0123] Further, the specific steps of B1 include:

[0124] (1) Extract the connection edge set from the constructed three-dimensional spatial node network, and each edge is associated with an air conditioning equipment;

[0125] (2) Analyze each connection edge and extract the attributes of the corresponding air conditioning equipment to obtain an air conditioning equipment attribute table;

[0126] S2.2: Associate the air conditioning equipment with the node state to obtain an air conditioning equipment attribute table with state association; the association content includes:

[0127] For each , bind the current temperature and the personnel density of the node ;

[0128] Further, the specific steps of B2 include:

[0129] (1) Traverse each air conditioning equipment and associate it to the set of nodes affected by it in the three-dimensional spatial node network based on its coordinate position;

[0130] (2) For each associated node, extract the state parameters and bind them to the corresponding air conditioning equipment record to obtain an air conditioning equipment attribute table with state association; the state parameters include but are not limited to the current temperature and the personnel density.

[0131] ​S2.3: Based on the basic functions of the air conditioning system, obtain the basic control action set, combine the temperature state vector of each node in the three-dimensional space node network and the environmental state component, determine the feasible range of the basic control action, at the same time, combine the number of personnel and activity intensity information in each area in the node network and the running state parameters of the air conditioning system itself to determine the extended control action set;

[0132] Further, the specific steps of B3 include:

[0133] (1) Study the basic functions of the air conditioning system, including core function modules such as refrigeration, heating, ventilation, air speed adjustment, temperature setting, etc., and determine the control mode corresponding to each function;

[0134] (2) For the basic functions of the air conditioning system, define the basic control actions, for example, the refrigeration function corresponds to the actions of starting the refrigeration mode, stopping the refrigeration mode, adjusting the refrigeration temperature up, adjusting the refrigeration temperature down; the air speed adjustment function corresponds to the actions of increasing the fan speed and reducing the fan speed, forming the basic control action set;

[0135] (3) Extract the temperature state vector of each node from the three-dimensional space node network, compare with the set comfortable temperature range, and calculate the deviation of the current temperature from the comfortable temperature;

[0136] If the temperature of a node is higher than the upper limit of the comfortable temperature and the deviation is greater than the preset error value, the feasible range of the action of adjusting the refrigeration temperature down can be determined as reducing the temperature;

[0137] If the temperature is close to the comfortable temperature, the feasible range of the action is small;

[0138] (4) Combine the environmental state components such as humidity and air pressure, when the humidity is high, the feasible range of the dehumidification action can be increased in the refrigeration mode, when the air pressure is low, the strength and frequency range of the ventilation action can be adjusted;

[0139] (5) Synthesize the analysis results of the temperature state vector and the environmental state component to determine the specific feasible parameter range for each action in the basic control action set;

[0140] (6) According to the number of personnel and activity intensity information in each area in the node network;

[0141] If the number of personnel is large and the activity intensity is high, consider adding extended control actions such as strengthening ventilation, increasing fan speed, etc.;

[0142] If the number of personnel is small, energy-saving actions such as reducing air conditioning power and reducing fresh air volume can be added;

[0143] (7) Analyze the running state parameters of the air conditioning system itself, such as the running time of the compressor and the energy consumption data;

[0144] If the compressor runs continuously for too long, extended control actions such as pausing the compressor operation and switching to a backup air conditioner can be added;

[0145] If the energy consumption exceeds the set threshold, energy-saving actions such as reducing the intensity of air conditioning cooling / heating can be added;

[0146] (8) Based on the number of people and activity intensity information, the air conditioning system's own operating state parameters, and the analysis of the actions, an extended control action set is formed.

[0147] S2.4: According to the control accuracy of the air conditioning equipment, the basic control action set and the extended control action set are divided into discrete gear positions, and the discrete gear positions of the basic control action and the extended control action are combined to form a discrete action combination candidate set;

[0148] Further, the specific steps of B4 include:

[0149] (1) Extract all continuous control parameters such as temperature setpoint, fan speed, and valve opening from the basic control action set and the extended control action set, and classify the parameters by type, such as temperature, wind speed, and power;

[0150] (2) According to the temperature control accuracy of the air conditioning equipment, which is usually 0.5℃ or 1℃, the temperature regulation range is divided into discrete gear positions, for example, the cooling temperature range 16-30℃ can be divided into 15 gear positions (16.0, 16.5,..., 30.0);

[0151] (3) Discretize the fan speed by percentage or fixed gear positions such as low, medium, and high, for example, the fan speed range 0%~100% can be divided into 5 gear positions such as 20%, 40%, 60%, 80%, and 100%;

[0152] (4) Map the discrete gear positions of the parameters back to the corresponding control actions, for example, the action of lowering the cooling temperature corresponds to the discrete gear positions of the temperature parameter;

[0153] (5) Establish an action-parameter mapping table to clearly define the range of parameter gear positions that can be used for each action;

[0154] (6) Arrange and combine the basic control actions and the extended control actions to generate action sequences, for example, combinations of adjusting temperature and wind speed simultaneously;

[0155] (7) For each action combination, determine its parameter configuration, for example, cooling mode + temperature 24℃ + wind speed 60% constitutes a complete action combination;

[0156] (8) Based on physical constraints and control logic, filter out invalid action combinations, for example, the combination of simultaneously starting cooling and heating modes is invalid;

[0157] (9) Apply device capability constraints to exclude combinations that exceed the device performance range, e.g., combinations with power exceeding the device rated power, resulting in a discrete set of action combinations.

[0158] S2.5: Determine the boundary conditions of the action space according to the physical characteristics and safety specifications of the air conditioning system, and set the constraint conditions between actions;

[0159] The boundary conditions of the action space include total power limit and compressor protection;

[0160] The constraint condition between actions is that the temperature difference between adjacent two time points is less than or equal to 1;

[0161] S2.6: All discrete action combinations that have undergone discretization processing and meet the boundary conditions and constraint conditions are combined, and each action is assigned a unique ID to form a discrete action space.

[0162] Further, the specific steps of B6 include:

[0163] (1) From all discrete action combinations obtained by discretization processing, filter according to boundary conditions and constraint conditions;

[0164] (2) For each discrete action combination, check whether each parameter is within the range specified by the boundary condition and whether it meets the constraint condition. If not, exclude the action combination;

[0165] (3) Assign a unique ID to each discrete action combination remaining after screening. In this invention, the order number is used, starting from 1 and increasing sequentially, to ensure that each action combination has a unique identifier;

[0166] (4) Establish a mapping relationship between action combinations and IDs to facilitate subsequent ID-based quick lookup of corresponding action combinations;

[0167] (5) Integrate discrete action combinations with unique IDs into a data structure to form a discrete action space. This data structure can be a list, dictionary, or database table.

[0168] The specific steps of S4 include:

[0169] S4.1: Using the node information in the three-dimensional space node network G, aggregate the node state vectors of each node according to the region weights obtained in S3 to obtain the global state representation s of the intelligent building;

[0170] Further, the specific steps of S4.1 include:

[0171] (1) Obtain the regional weight in S3 and the three-dimensional node network G;

[0172] (2) Establish a mapping function , which maps each node to the corresponding sub-region type index, where m represents the number of sub-region types, f represents the mapping function, , and the mapping relationship is represented by

[0173] (3) Obtain the global state representation of the intelligent building by calculating the ratio of the first type of numerator variable and the second type of denominator variable; the first type of numerator variable is the weight of the sub-region type to which the node belongs and the accumulated summation result of the node state vector of the node ; the second type of denominator variable is the accumulated summation result of the weight of the sub-region type to which the node belongs.

[0174] S4.2: Obtain the discrete action space in S2; the discrete action space contains all feasible combinations of air conditioner control actions;

[0175] S4.3: Calculate the reward value according to the global state representation s of the intelligent building and the discrete action combination in the discrete action space using the constructed multi-objective reward function;

[0176] Further, the specific steps of S4.3 include:

[0177] (1) Obtain the constructed multi-objective reward function; the multi-objective reward function includes temperature deviation, energy consumption function and air quality function;

[0178] (2) Calculate the temperature comfort reward , energy efficiency reward and air quality reward according to the multi-objective reward function, respectively, and satisfy the formula:

[0179]

[0180] where a represents the executed action, represents the temperature deviation penalty function, and , represents the temperature of the node , represents the set temperature, represents the temperature change amount, represents the basic power consumption of the air conditioner, which is the minimum energy consumption of the air conditioner in the starting state, and is also the power consumption even if it does not cool, represents the power consumption of the kth air conditioner under action a, and satisfies:​ , represents the control action of the kth air conditioner, represents the current temperature of the area where the kth air conditioner is located, which is the node temperature extracted from the global state representation s, represents the outdoor temperature, and scale represents the power scaling coefficient, represents the time-of-use electricity price factor, for example, the peak period , the valley period , t represents the current time, represents the CO2 concentration of the node , represents the CO2 concentration, represents the CO2 deviation penalty function, and satisfies: , represents the CO2 concentration change, represents the maximum function;

[0181] (3) Integrate the temperature comfort reward , energy efficiency reward , and air quality reward to obtain the reward value .

[0182] S4.4: Load the pre-built Dueling DQN model architecture; the network architecture of the Dueling DQN includes one feature extraction layer, one advantage function flow, and one state value function flow;

[0183] S4.5: Randomly initialize the main network parameters and target network parameters of the pre-built Dueling DQN model, and initialize the experience replay buffer at the same time; the experience replay buffer is used to store state-action-reward-next state four-tuples;

[0184] Further, the random initialization of the main network parameters uses the following method:

[0185] (1) Weight parameters: initialized using Gaussian distribution or uniform distribution;

[0186] (2) Bias parameters: initialized to 0 or a small constant.

[0187] wherein the target network parameters are initialized synchronously with the main network parameters, satisfying: target network parameters main network parameters.

[0188] S4.6: Train the pre-built Dueling DQN model, and when the training reaches the preset number of iterations or meets the convergence condition, the training is ended, and the trained Dueling DQN model is deployed to the control system of the intelligent building.

[0189] The specific process of training the pre-constructed Dueling DQN model in S4.6 includes:

[0190] S4.61: Initialize the environment state of the intelligent building , select an action from the discrete action space according to the current global state representation ;

[0191] For example, an example code for selecting an action from the discrete action space using the greedy strategy is as follows:

[0192] if rand() < ε then

[0193] a1 ← random action

[0194] else

[0195] a1 ← argmax_a Q(s1,a;θ) # the optimal action predicted by the current network Q

[0196] end if

[0197] S4.62: The air conditioning system executes the action , and calculates the final reward value according to the multi-objective reward function, at the same time, obtains the new state through the three-dimensional space node network , and stores the experience sample into the experience replay buffer;

[0198] S4.63: If the buffer is full, discard the old samples according to the FIFO principle;

[0199] Further, the process of discarding old samples according to the FIFO principle includes:

[0200] (1) Initialize the experience replay buffer and set the maximum capacity capacity of the experience replay buffer;

[0201] (2) When a new experience sample is obtained, check whether the buffer length is equal to the capacity;

[0202] If the buffer length is equal to the maximum capacity capacity of the experience replay buffer, remove the first element in the list, i.e. the earliest sample, and then add the new experience sample;

[0203] If the buffer length is less than the maximum capacity capacity of the experience replay buffer, directly add the new experience sample.

[0204] For example, the implementation process includes: ​

[0205] if len(replay_buffer) == capacity:

[0206] del replay_buffer[0]

[0207] replay_buffer.append(new_experience)

[0208] S4.64: Randomly sample M experience samples from the experience replay buffer, and calculate target values using the target network;

[0209] An example code for calculating the target values is as follows:

[0210] if s' is a terminal state then

[0211] y ← r

[0212] else

[0213] y ← r + γ * max_a' Q'(s',a';θ') # γ is the discount factor

[0214] end if

[0215] S4.65: Calculate the mean square error loss value between the output of the current network and the target value, wherein the mean square error calculation formula is a prior art in the art and is not the inventive scheme of the present application, and thus is not described herein;

[0216] S4.66: According to the mean square error loss value, perform one-step gradient descent using the optimizer to update the current network parameters, wherein the gradient descent calculation formula is a prior art in the art and is not the inventive scheme of the present application, and thus is not described herein;

[0217] S4.67: Copy the current network parameters to the target network parameters every C steps, and set the next state as the current state. In the present application, C is set to 1000.

[0218] The specific steps of S4.61 include:

[0219] determining the intelligent building environment state factors, and setting initial values for each intelligent building environment state factor;

[0220] setting a probability u, and satisfying ;

[0221] generating a random number g, and satisfying ;

[0222] if if action_space is discrete, then randomly select an action from the discrete action space;

[0223] if action_space is discrete, then randomly select an action from the discrete action space;

[0224] The specific steps of S4.62 include:

[0225] A1: decode the action index output by the Dueling DQN model into an air conditioner control instruction, and issue it to the building automation system through the ModbusRTU protocol:

[0226] Further, the specific steps of A1 include:

[0227] (1) Predefine a mapping table of action index and air conditioner control parameter, for example:

[0228] action_mapping = {

[0229] 0: {'temperature': 24, 'fan_speed':'medium','mode': 'cooling'},

[0230] 1: {'temperature': 25, 'fan_speed':'medium','mode': 'cooling'},

[0231] }

[0232] (2) Convert the action index output by the Dueling DQN model into a control parameter dictionary, and the conversion method is: control_params = action_mapping[B], where B represents the action index output by the Dueling DQN model, control_params represents the control parameter in the control parameter dictionary, and action_mapping represents the mapping relationship, which is a control parameter dictionary in this application;

[0233] (3) Verify whether the control parameter is within the allowable range of the device, for example:

[0234] if not (16 <= control_params['temperature'] <= 30):

[0235] # Correct the temperature value or trigger an exception

[0236] ​​control_params['temperature'] = min(max(control_params['temperature'], 16), 30)

[0237] (4) Establish the correspondence between the control parameters and the Modbus register address;

[0238] (5) Convert the control parameters into Modbus register values, and construct a Modbus RTU frame;

[0239] (6) Set the baud rate, data bits, stop bits, and check bits, and send the Modbus RTU frame to complete the sending of the air conditioner control instruction.

[0240] A2: The building automation system receives and executes the air conditioner control instruction, and simultaneously collects the actual parameters after the execution of the air conditioner;

[0241] A3: Obtain new state data through the nodes of the three-dimensional space node network, and dynamically adjust the sub-region weights in combination with the actual parameters after the execution of the air conditioner;

[0242] A4: Based on the new state data and the adjusted sub-region weights, a new global state is generated using a weighted average method, wherein the weighted average method is a prior art content in the field and is not the inventive scheme of the present application, and will not be described here;

[0243] A5: Based on the new state data, the actual parameters after the execution of the air conditioner, and the decoded air conditioner control instruction, a final reward value is calculated using a multi-objective reward function;

[0244] A6: The new state data, action index, final reward value, and new global state are combined into a four-tuple to obtain a transition sample, and the transition sample is stored in an experience replay buffer.

[0245] The specific steps of S5 include:

[0246] S5.1: Synchronize the collection of real-time data of each node from the sensor network of the three-dimensional space node network, and perform preprocessing to generate a real-time node state vector set;

[0247] S5.2: Perform weighted aggregation on the real-time node state vector according to the regional weights to generate a real-time global state representation;

[0248] S5.3: Convert the real-time global state representation into a tensor format, and simultaneously add a batch dimension to form an input format that can be processed by the DuelingDQN model; the dimension is the same as the input layer dimension of the Dueling DQN model;

[0249] It should be noted that the batch dimension is a data dimension used to represent processing multiple samples at a time in deep learning. In deep learning, usually more than one data sample is processed at a time, and multiple samples are combined into a batch for processing. The batch dimension is used to represent the number of samples in the batch. By combining multiple samples into a batch for calculation, the parallel computing capability of hardware can be fully utilized, the calculation time and memory access times can be reduced, and the efficiency of training and inference can be improved. At the same time, compared with calculating the gradient of a single sample, the batch gradient can better represent the overall distribution of the data and reduce the noise and fluctuations of the gradient.

[0250] S5.4: Calculate the Q values of all possible actions through the trained Dueling DQN model, and select the action index with the maximum Q value;

[0251] S5.5: Map the action index with the maximum Q value to the actual control parameter; the actual control parameter includes the temperature set value and the wind speed gear; the mapping process mainly refers to a predefined decoding table: wherein, the action index with the maximum Q value, FS represents the wind speed gear, represents the Dueling DQN model, represents the decoding function, and ControlParams represents the actual control parameter;

[0252] S5.6: According to the air conditioning system communication protocol, convert the actual control parameter into an instruction frame through CF=ProtocolEncode(ControlParams), wherein CF represents the instruction frame, represents the protocol encoding function;

[0253] S5.7: Send the instruction frame to the corresponding air conditioning equipment through the network interface, and record the execution log.

[0254] The embodiments of the application are described above in combination with the drawings, but the application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative and not limiting. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the purpose and scope of the application, and these are all within the protection scope of the application.

Claims

1. An intelligent building air conditioner adaptive control method based on deep reinforcement learning, characterized in that, The application relates to a control method of an air conditioning system in a smart building, which comprises the following steps: S1: collecting multi-element information of a smart building area, combining a three-dimensional model of the smart building and an air conditioning position, and constructing a three-dimensional space node network; S2: defining a control action range of the air conditioning system according to the three-dimensional space node network, and forming a discrete action space; S3: constructing a multi-target reward function, combining the state differences of different areas in the three-dimensional space node network, and setting area weights; S4: training a pre-constructed deep reinforcement learning model according to the three-dimensional space node network, the discrete action space and the multi-target reward function, introducing an experience replay mechanism and a target network mechanism, obtaining a trained deep reinforcement learning model and deploying the model to a control system of the smart building; S5: real-time monitoring in the control system, acquiring current state representation data through the three-dimensional space node network, selecting an optimal action from the discrete action space according to the current state representation data and outputting the optimal action to the air conditioning system; S6: after the air conditioning system executes the optimal action, collecting new state representation data through the three-dimensional space node network again, calculating a reward value of this action execution according to the multi-target reward function and feeding back the reward value to the deep reinforcement learning model, and dynamically adjusting a control strategy of the deep reinforcement learning model according to the feedback result; The control strategy is an optimal action selection process. The specific steps of S4 comprise: S4.1: Using the node information in the three-dimensional space node network G, the node state vector of each node is aggregated by weighting the area weight obtained in S3 to obtain a global state representation of the intelligent building s ; S4.2: acquiring the discrete action space in S2; the discrete action space contains all feasible air conditioning control action combinations; S4.3: using the constructed multi-objective reward function to calculate a reward value according to the global state representation of the smart building s and the combination of discrete actions in the executed discrete action space. S4.4: loading a pre-constructed Dueling DQN model architecture; the network architecture of the Dueling DQN comprises a feature extraction layer, an advantage function flow and a state value function flow; S4.5: randomly initializing main network parameters and target network parameters of the pre-constructed Dueling DQN model, and simultaneously initializing an experience replay buffer; the experience replay buffer is used for storing state-action-reward-next state four-tuple; S4.6: training the pre-constructed Dueling DQN model, and when the training reaches a preset iteration number or meets a convergence condition, the training is ended, and the trained Dueling DQN model is deployed to the control system of the smart building; When calculating the reward value in S4.3, the reward values of temperature deviation, energy consumption function and air quality function in the multi-target reward function need to be calculated step by step, and then weighted summation is realized, and the formula for calculating the reward values of temperature deviation, energy consumption function and air quality function in the multi-target reward function is as follows: ； wherein, a represents an action performed, represents a temperature comfort reward, represents an energy efficiency reward, represents an air quality reward, represents a node of the type of the sub-area, represents a temperature deviation penalty function, and , represents a node of the type of the sub-area, represents a set temperature, represents a temperature change amount, represents a base power consumption of an air conditioner, represents a power consumption of the k th air conditioner under the action a , and satisfies: , represents a control action on the k th air conditioner, represents a current temperature of the area where the k th air conditioner is located, is a node temperature extracted from the global state representation s , represents an outdoor temperature, and scale represents a power scaling coefficient, represents a time-of-use electricity price factor, t represents a current time, represents a CO2 concentration of a node , represents a CO2 concentration, represents a CO2 deviation penalty function, and satisfies: , represents a CO2 concentration change amount, represents a maximum function. 2.The intelligent building air conditioner adaptive control method based on deep reinforcement learning of claim 1, wherein, The specific steps of S1 comprise: S1.1: collecting multi-element information of a smart building area, importing a three-dimensional model of the smart building, acquiring an air conditioning position, and simultaneously performing pretreatment; S1.2: Divide the three-dimensional space of the intelligent building into N irregular sub-regions according to the building's functional type. Each sub-region is defined as a node, forming a node set. The preprocessed multi-dimensional information is then assigned to the corresponding nodes to form node state vectors; the building function types include, but are not limited to, offices, meeting rooms, and corridors. Indicates the first i Sub-region or the i One node; S1.3: defining a space connection relationship between nodes according to the node and the position coordinates of the node in a three-dimensional space; the space connection relationship comprises: If two nodes belong to subareas sharing an entity edge in a three-dimensional space, an adjacency relationship is established, and a weight matrix W and an adjacency matrix A are obtained by adjusting the weight according to the physical property of the shared edge; S1.4: Determine the range of nodes affected by each air conditioning unit through CFD simulation the range of nodes affected wherein, k denotes the air conditioning unit index; If the node The central coordinates are located in the air conditioning equipment The node range affected Then establish a connection edge And the operating parameters of the air conditioner are taken as the edge attributes of the connection edge to form a connection edge set ; S1.5: integrate the set of nodes V, the adjacency matrix A, the set of connecting edges , output the three-dimensional spatial node network wherein, denotes the union set. 3.The intelligent building air conditioner adaptive control method based on deep reinforcement learning of claim 2, wherein, The specific steps of S2 comprise: S2.1: Obtain a three-dimensional space node network, extract the attributes of each air conditioning equipment from the connection edge set based on the three-dimensional space node network including type, energy efficiency ratio, maximum power, and coordinates​​ S2.2: associate the air conditioning equipment with the node state, obtain the air conditioning equipment attribute table associated with the state; the contents of the association include: For each , the current temperature and the density of people of the binding node ; S2.3: based on the basic function of the air conditioning system, obtain the basic control action set, determine the feasible range of the basic control action in combination with the temperature state vector of each node in the three-dimensional space node network and the environmental state component, at the same time, determine the extended control action set in combination with the number of personnel and activity intensity information in each region in the node network and the running state parameters of the air conditioning system itself; S2.4: according to the control accuracy of the air conditioning equipment, discretize the gear of the basic control action set and the extended control action set, and combine the discrete gears after the division of the basic control action and the extended control action to form a discrete action combination candidate set; S2.5: determine the boundary conditions of the action space and set the constraint conditions between actions according to the physical characteristics and safety specifications of the air conditioning system; The boundary conditions of the action space include total power limitation and compressor protection; The constraint condition between actions is that the temperature difference between adjacent two time instants is less than or equal to 1; S2.6: combine all the discrete actions that have been discretized and meet the boundary conditions and constraint conditions, and assign a unique ID to each action to form a discrete action space. 4.The intelligent building air conditioner adaptive control method based on deep reinforcement learning of claim 3, wherein, The specific process of training the pre-constructed Dueling DQN model in S4.6 includes: S4.61 : initializing the environmental state of the smart building , selecting an action in the discrete action space according to the current global state representation ; S4.62: The air conditioning system performs an action and calculates the final reward value according to the multi-objective reward function, and obtains a new state through a three-dimensional space node network and stores the experience sample into an experience replay buffer. S4.63: if the buffer is full, discard old samples according to the FIFO principle; S4.64: randomly sample M experience samples from the experience replay buffer, and calculate the target value using the target network; S4.65: calculate the mean square error loss value between the output of the current network and the target value; S4.66: according to the mean square error loss value, use the optimizer to perform one-step gradient descent to update the current network parameters; S4.67: copy the current network parameters to the target network parameters every C steps, and set the next state as the current state. 5.The intelligent building air conditioner adaptive control method based on deep reinforcement learning according to claim 4, wherein, The specific steps of S4.61 include: Determine the intelligent building environment state factors, and set the initial value for each intelligent building environment state factor; Set a probability u , and satisfies ; generating a random number g , and satisfies ; If then select a random action from the discrete action space; If , the reward value of each action is calculated by using the pre-constructed Dueling DQN model according to the current state representation, and the action with the highest reward value is selected. 6.The intelligent building air conditioner adaptive control method based on deep reinforcement learning according to claim 5, wherein, The specific steps of S4.62 include: The action index output by the Dueling DQN model Decoded into air conditioning control instructions and issued to the building automation system through the Modbus RTU protocol: The building automation system receives and executes the air conditioning control instruction, at the same time, collects the actual parameters after the execution of the air conditioner; Through the nodes of the three-dimensional space node network, obtain new state data, and dynamically adjust the sub-region weight in combination with the actual parameters after the execution of the air conditioner; Based on the new state data and the adjusted sub-region weight, generate a new global state using a weighted average method; Based on the new state data, the actual parameters after the execution of the air conditioner, and the decoded air conditioning control instruction, calculate the final reward value using a multi-objective reward; Combine the new state data, action index, final reward value, and new global state to form a four-tuple to obtain a transition sample, and store the transition sample in the experience replay buffer. 7.The intelligent building air conditioner adaptive control method based on deep reinforcement learning of claim 6, wherein, The specific steps of S5 include: S5.1: synchronously collect real-time data of each node from the sensor network of the three-dimensional space node network, and perform preprocessing to generate a real-time node state vector set; S5.2: weight-aggregate the real-time node state vector according to the regional weight to generate a real-time global state representation; S5.3: converting the real-time global state representation into a tensor format, while adding a batch dimension to form an input format that can be processed by the Dueling DQN model; the dimension is the same as the input layer dimension of the Dueling DQN model; S5.4: Calculate the Q values of all possible actions by the trained Dueling DQN model, and select the action index with the largest Q value. Q Q value.​ S5.5: mapping the action index with the largest value to an actual control parameter; the actual control parameter includes a temperature set value, a wind speed gear; the mapping execution process mainly refers to a predefined decoding table: Q wherein, wherein, represents Q the action index with the largest value, FS represents the wind speed gear, represents a Dueling DQN model, represents a decoding function, and ControlParams represents an actual control parameter; S5.6: Convert the actual control parameters into a command frame by CF = ProtocolEncode(ControlParams) according to the air conditioning system communication protocol, wherein CF represents the command frame, denotes the protocol encoding function; S5.7: sending the instruction frame to the corresponding air conditioning equipment through the network interface, and recording the execution log. 8.The intelligent building air conditioner adaptive control method based on deep reinforcement learning of claim 7, wherein, The multi-element information includes a temperature state vector of the intelligent building area, temperature gradient information between areas, an environment state component, personnel quantity and activity intensity information of each area, and an operation state parameter of the air conditioning system itself; and the control action includes adjustment of a refrigeration or heating mode, multi-gear setting of a fan rotating speed, and adjustment of a fresh air valve opening degree.

Images