A heating, ventilation and air conditioning control method based on a random probability weighted composite sampling strategy

Abstract

The application belongs to the field of heating ventilation air conditioning, and proposes a heating ventilation air conditioning control method based on a random probability weighted composite sampling strategy, and the working process is as follows: step 101, an intelligent agent is constructed and initialized, and the initial exploration rate is set to 1; step 102, the state of the current heating ventilation air conditioning system is acquired, and whether the current exploration rate is less than the set value of starting to use probability weighted sampling is judged; step 103, a random probability weighted composite sampling strategy is used to select an action; step 104, a control instruction corresponding to the action is executed, and a Q table is updated; step 105, the exploration rate is reduced with a training setting period; step 106, whether the intelligent agent training reaches an end standard is judged, if not, the training is continued, otherwise, state parameters are input into the trained intelligent agent. The method can effectively improve the performance of the heating ventilation air conditioning system, reduce the operation cost, and enhance the stability of the system, and has certain application prospect.

Description

TECHNICAL FIELD

[0001] The application relates to heating, gas supply, ventilation and air conditioning engineering, and is building equipment control technology, in particular to a Q-learning algorithm energy-saving control method for heating, ventilation and air conditioning based on a random probability weighted composite sampling strategy. BACKGROUND

[0002] Prior art introduction: when the existing heating, ventilation and air conditioning system adopts Q-Learning optimization control, the action selection usually adopts a Monte Carlo sampling strategy, which is a random exploration and utilization balancing method. The sampling method explores by randomly selecting from different actions of the environment state, and in the initial training stage, the agent randomly selects actions with a certain probability; as the training proceeds, the exploration rate gradually decreases, and the agent is more inclined to select actions according to the training experience. This action sampling can extensively explore in the initial training stage to discover more state-action pairs, and gradually reduce the exploration rate so as to focus more on utilizing the learned knowledge in the later training stage.

[0003] Problems of the prior art: the traditional heating, ventilation and air conditioning system control method usually adopts the Monte Carlo action sampling to explore in Q-learning by randomly selecting actions, which cannot adapt to the dynamic changes and complexity of the system. Randomly selecting actions may lead to unstable control strategies that are not suitable for the current system state, and cannot fully utilize the experience information of actions in the historical training data, which may cause repeated selection of bad actions or neglect of known good actions in the training process; in the later training stage, the Monte Carlo sampling still randomly explores with a certain probability, which may cause waste of known optimal actions, reduce the training efficiency and system performance.

[0004] In summary, the method of the application introduces a random-probability weighted composite sampling strategy action decision strategy, aiming to improve the instability, insufficient utilization of experience and lack of targeted action exploration of the traditional method, so as to more effectively optimize the control performance of the heating, ventilation and air conditioning system. SUMMARY

[0005] The method aims at improving the performance of a heating ventilation air conditioning system, and proposes a random probability weighted composite sampling strategy of Monte Carlo sampling and probability weighted sampling in the optimization control of the heating ventilation air conditioning system using a Q-learning algorithm, instead of a random value strategy of the Monte Carlo sampling for the action space. In the initial training stage, the traditional Monte Carlo sampling strategy is used for action selection to balance exploration and utilization; when the exploration rate continuously reduces to a certain value, the probability distribution of the action is obtained by calculating the probability of the action in the historical training data, and the action selection is performed according to the distribution in the subsequent training and control process, so that the exploration is performed specifically, thereby improving the training effect and the performance of the heating ventilation air conditioning system.

[0006] The technical scheme of the present application is as follows:

[0007] A heating ventilation air conditioning control method based on a random probability weighted composite sampling strategy, characterized by the following working process:

[0008] Step 101: constructing and initializing an intelligent agent, determining the state space, action space and reward function of the heating ventilation air conditioning system, and setting the initial exploration rate to 1.

[0009] Step 102: obtaining the current state of the heating ventilation air conditioning system, and determining whether the current exploration rate is less than the set value for starting to use the probability weighted sampling.

[0010] Step 103: selecting an action using the random probability weighted composite sampling strategy; if the current exploration rate is not less than the set value k for starting to use the probability weighted sampling, determining whether the generated random number is greater than the current exploration rate; when the random number in 0-1 is greater than the exploration rate, selecting the action with the maximum Q value in the state-action Q table under the current state; when the random number is not greater than the exploration rate, selecting a random action in the action space; if the current exploration rate is less than the set value k for starting to use the probability weighted sampling, determining whether the generated random number is greater than the current exploration rate; when the random number is greater than the exploration rate, selecting the action with the maximum Q value in the state-action Q table under the current state; when the random number is not greater than the exploration rate, calculating the probability of each action in the historical training data to obtain the probability distribution of each action in the action space, and selecting an action in the action space according to the probability distribution.

[0011] Step 104: executing the control instruction corresponding to the action in the current state, obtaining the state parameters and reward feedback of the next state of the heating ventilation air conditioning system, and updating the Q table.

[0012] Step 105: reducing the exploration rate with the training set period.

[0013] In step 106, it is judged whether the intelligent agent training reaches the end standard, if not, it returns to step 102 to continue training, if the standard is reached, the model is derived, the state parameters are input into the trained intelligent agent, and the control instruction to be issued under the state is obtained.

[0014] Beneficial effects: The method takes improving the performance of the heating and air conditioning system as the goal, and proposes a random probability weighted composite sampling strategy of Monte Carlo sampling and probability weighted sampling in the optimization control of the heating and air conditioning system using the Q-learning algorithm, instead of the random value strategy of the action space of the simple Monte Carlo sampling. In the early training, the traditional Monte Carlo sampling strategy is used for action selection to balance exploration and utilization; when the exploration rate continuously reduces to a certain value, the probability distribution of the action is obtained by calculating the probability of the action in the historical training data, and the action selection is carried out according to the distribution in the subsequent training and control process, and the exploration is carried out in a targeted manner to improve the training effect and the performance of the heating and air conditioning system. In summary, the Q-learning algorithm heating and air conditioning energy-saving control method based on the random probability weighted composite sampling strategy considers the instability of the traditional method, the inability to fully utilize experience and the lack of targeted action exploration, and through the introduction of the random probability weighted composite sampling strategy of Monte Carlo sampling and probability weighted sampling, the performance of the heating and air conditioning system can be effectively improved, the operation cost can be reduced, and the stability of the system can be enhanced, and the method has certain application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 Method workflow diagram of the present application

[0016] Figure 2 Construction and training diagram of the intelligent agent of the present application

[0017] Figure 3 Comparison diagram of the method of the present application and the epsilon-greedy strategy method DETAILED DESCRIPTION

[0018] The technical solutions provided by the present application will be further described below with reference to specific embodiments and drawings. The advantages and features of the present application will be more apparent in combination with the following description.

[0019] It should be noted that the embodiments of the present application have better implementation, and are not any form of limitation of the present application. The technical features or combinations of technical features described in the embodiments of the present application should not be considered as isolated, and they can be combined with each other to achieve better technical effects. The scope of the preferred embodiments of the present application can also include other implementations, and this should be understood by those skilled in the art to which the embodiments of the present application belong.

[0020] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art(s) can not be discussed in any detail in order to avoid obscuring the present description. Any of the examples shown and discussed herein are to be interpreted as merely examples, and not as limiting. As such, other examples of the exemplary embodiments can have different values.

[0021] The drawings of the present application are very simplified and use non-precise proportions, only to facilitate, clearly assist in the purpose of illustrating the embodiments of the present application, and are not the limiting conditions that can be implemented by the present application. Any modification of structure, change of proportional relationship or adjustment of size, without affecting the effects that can be produced by the present application and the purposes that can be achieved, should fall within the scope of the technical content disclosed by the present application. The same reference numerals appearing in the drawings of the present application represent the same features or components, which can be applied to different embodiments.

[0022] The present method is suitable for heating, ventilation and air conditioning systems, including but not limited to water systems or split heating, ventilation and air conditioning systems. To improve the performance of the heating, ventilation and air conditioning system, an agent based on the method of the present application is constructed. The agent obtains the state data of the heating, ventilation and air conditioning system, determines the control action of the heating, ventilation and air conditioning system, and uses the Q-Learning algorithm for reinforcement learning to optimize the control action.

[0023] In the optimization control of the heating, ventilation and air conditioning system using the Q-learning algorithm, the agent uses a random probability weighted composite sampling strategy of Monte Carlo sampling and probability weighted sampling to replace the traditional random value strategy of Monte Carlo sampling in the action space, which can achieve better training effect and performance of the heating, ventilation and air conditioning system.

[0024] A heating, ventilation and air conditioning control method based on a random probability weighted composite sampling strategy, characterized by a working process as shown in Figure 1

[0025] Step 101, constructing and initializing the agent, determining the state space, action space, reward function of the heating, ventilation and air conditioning system, state-action Q table, setting the initial exploration rate to 1;

[0026] Step 102, obtaining the current state of the heating, ventilation and air conditioning system, and determining whether the current exploration rate is less than the set value of starting to use probability weighted sampling;

[0027] ​Step 103, selecting an action using a random probability weighted composite sampling strategy; if the current exploration rate is not less than the set value k at which probability weighted sampling is started to be used, it is judged whether the generated random number is greater than the current exploration rate, when the random number within 0-1 is greater than the exploration rate, the action with the maximum Q value of the current state is selected from the state-action Q table, when the random number is not greater than the exploration rate, a random action is selected in the action space; if the current exploration rate is less than the set value k at which probability weighted sampling is started to be used, it is judged whether the generated random number is greater than the current exploration rate, when the random number is greater than the exploration rate, the action with the maximum Q value of the current state is selected from the state-action Q table; when the random number is not greater than the exploration rate, the probability of each action appearing in the historical training data is calculated to obtain the probability distribution of each action in the action space, and an action is selected in the action space according to the probability distribution;

[0028] Step 104, executing the control instruction corresponding to the action in the current state to obtain the state parameters and reward feedback of the next state of the heating and air conditioning system, and updating the Q table;

[0029] Step 105, reducing the exploration rate with the training set period;

[0030] Step 106, judging whether the training of the intelligent agent reaches the end standard, if not, returning to step 102 to continue training, if the standard is reached, exporting the model, inputting the state parameters into the trained intelligent agent, and obtaining the control instruction to be issued in the state.

[0031] Further, the intelligent agent is an entity that can perceive the environment, make decisions and interact with the environment. The construction of the Q-learning intelligent agent uses a function approximation method, which needs to define the state space, action space, reward function and learning algorithm, etc.

[0032] State space: The state space includes multiple state parameters. In the heating and air conditioning system, the state parameters are usually a vector or a feature set, which help the intelligent agent to make decision to select an action, such as cooling load and wet bulb temperature, which affect the energy consumption of the heating and air conditioning system, and the sampling period thereof is preferably set to 10 to 15 minutes;

[0033] Action space: The action space refers to the set of all possible actions that the intelligent agent can select, including but not limited to the frequency and number of cooling water pumps and other control command contents currently required to be issued in the heating and air conditioning system.

[0034] The actions and states in the state-action Q table need to be discretized. For the state parameters, the nearest neighbor multiple rounding processing can be performed, and the formula is,

[0035] S = round(S' / b) x b (1)

[0036] Wherein, S represents the processed data, S' represents the original state parameter set, round() represents the rounding function, b represents the value of the nearest neighbor multiple, which can be freely defined according to the value range of different types of states, for example, the nearest neighbor multiple of the cooling load is 20 times, and the nearest neighbor multiple of the wet bulb temperature is 1 times.

[0037] For action parameters, the action range and discrete interval should be set according to the control accuracy and the safety limit of the equipment. The common frequency conversion range of the cooling water pump is 40-50Hz, and the action interval can be set to 2Hz, and the number of action intervals is accurate to 1.

[0038] The state row name in the state-action Q table is usually a state tuple parameter formed by arranging combinations of various state values, for example, the state quantity of the cooling load of 1245kW and the wet bulb temperature of 267℃ is (1240, 27); The action column name in the state-action Q table is usually in the form of a list composed of various action spaces or a single action space value, for example, the action column name corresponding to the water pump quantity of 1 and the water pump frequency of 47Hz can be recorded as [1, 47], or it can be divided into two independent Q tables containing only frequency or quantity.

[0039] Reward function: The reward function is the optimization target of the agent, which can be used to evaluate the good or bad of the agent's action at each step in the environment. In the energy-saving control of heating, ventilation and air conditioning, the reward function variables include but are not limited to two categories of heating system performance parameters such as energy consumption and comfort, for example, total power consumption, room temperature, etc. The reward function formula is:

[0040] R = u E ×E + u C ×C(2)

[0041] Wherein, R represents the reward function, u E , u C represent the weights of each type of variable participating in the reward calculation, E and C represent the scores of energy consumption and comfort variables participating in the reward calculation, and u value can be set as a constant or a variable related function according to the optimization emphasis. The lower the system energy consumption and the lower the indoor comfort deviation from the standard, the larger the reward function value. The optimization control target is the comprehensive reward after weighted processing.

[0042] Energy consumption variables include but are not limited to: target device or system power, COP, total energy consumption, etc. Comfort variables include but are not limited to: indoor temperature, indoor relative humidity, etc.

[0043] Learning algorithm: Q-learning is a reinforcement learning method based on time series difference Markov decision to iteratively update the state-action Q table. The state quantity and action quantity need to be discretized, and the Q table is shown in Table 1.

[0044] Table 1. Q table

[0045]

[0046] wherein A is the action space, a represents an action, m is the total number of actions contained in the action space, S is the state space, s represents a state, n is the total number of actions contained in the state space, and each Q value is represented as Q(s, a), wherein Q(s, a) represents the action-state Q value obtained by taking action a when the environment state is s. The Q table can be directly built according to the known state and action space, or all action rows can be created according to the action space of the system, and a corresponding state column is created after each state is obtained and discretized.

[0047] In the initialization stage before the start of training, the Q table and the Q table updating strategy need to be initialized.

[0048] Initialization of the Q table: all newly created elements of the Q table are initialized to 0 or a small random value, representing the initial estimated value of the agent for all state-action pairs.

[0049] Q table updating strategy: the Q table updating formula of the function approximation learning algorithm based on the meaning of the Q table is

[0050] Q(s, a)←Q(s, a)+α[r+γmaxQ(s', a')-Q(s, a)] (3)

[0051] wherein the arrow indicates that the left variable is updated with the calculated value on the right, Q(s, a) represents the action-state Q value of taking action a under the environment state s, s' represents the next time environment state s', maxQ(s', a') represents the known maximum Q value under the state s', r is the reward brought by the last round of action-state combination, a is the learning rate, γ represents the discount factor, i.e., the degree of influence of future rewards on current action decision, and maxQ(s', a') represents the known maximum Q value under the state s'.

[0052] The action a that updates the Q table is the action decision selected according to the current state. In the Q-learning algorithm, the exploration rate ε is the probability of the agent exploring new actions in the decision-making process. The initial exploration rate ε is set to 1, indicating that the agent will completely randomly select actions for exploration in the initial stage, so as to explore the state-action space as much as possible.

[0053] The greater the exploration rate is, the stronger the exploration of the agent is, the smaller the exploration rate is, the higher the utilization rate of the intelligent experience is, and the exploration rate determines the probability of directly selecting the action with the highest Q value in the Q table of the current state and the probability of using the random probability weighted composite sampling strategy when the agent selects an action, for example, when the exploration rate is 0.6, there is a 60% probability of using the random probability weighted composite sampling strategy and a 40% probability of directly selecting the action with the highest Q value in the Q table of the current state each time the action is selected.

[0054] Generally, the agent appropriately reduces the exploration rate after each optimization period, and in actual application, a certain exploration rate is usually reserved to continuously update the Q table, and 0.1 is often taken.

[0055] In the actual application of the agent, the optimization model can be obtained by training the historical data, and then the current optimization control action can be obtained by inputting the estimated state into the trained model, and the Q table can be updated.

[0056] The random probability weighted composite sampling strategy adopts the random probability weighted composite sampling strategy of Monte Carlo sampling and probability weighted sampling to make action decisions. If the current exploration rate is not less than the set value at which the probability weighted sampling is started to be used, it is judged whether the generated random number is greater than the current exploration rate epsilon, when the random number in 0-1 is greater than the exploration rate, the action with the maximum Q value in the state-action Q table is selected, when the random number is not greater than the exploration rate, a random action is selected in the action space; if the current exploration rate is less than the set value at which the probability weighted sampling is started to be used, it is judged whether the generated random number is greater than the current exploration rate, when the random number is greater than the exploration rate, the action with the maximum Q value in the state-action Q table is selected, when the random number is not greater than the exploration rate, the probability distribution of each action in the action space is obtained by calculating the probability of each action in the historical training data, and the action is selected in the action space according to the probability distribution.

[0057] Monte Carlo sampling in Q-learning action selection is summarized as follows: with a probability of epsilon, an action in the action space is randomly selected, and with a probability of 1-epsilon, a random action is selected, which is commonly used in epsilon-greedy decision-making.

[0058] The probability weighted sampling proposed in the application is summarized in Q-learning action selection as follows: the probability of each action in the historical training data record is calculated to obtain the action probability distribution; in the action space, an action is selected according to the action probability distribution with a probability of epsilon, and an action with the maximum Q value is selected with a probability of 1-epsilon.

[0059] The random probability weighted composite sampling strategy formula applies Monte Carlo sampling and probability weighted sampling to the whole training process in the early and late stages of training respectively, and represents the probability of selecting action a in state s as pi(a|s), and the random probability weighted composite sampling strategy formula is expressed as:

[0060]

[0061] Where ε is a predefined value representing the current exploration rate, and probability-weighted sampling is started when ε is less than k. It is recommended that k be between 0.3 and 0.5. P(X=x i Y = y i ,...) a This indicates that when action variables X, Y, etc., take the value x i y i The probability of action a occurring in historical data is given by π(a|s), where π(a|s) is the probability that action a is selected in state s.

[0062] Furthermore, the method for calculating the action probability distribution is as follows:

[0063] For example, to determine the amount of data *t* needed to generate the action-probability table, read the last *t* sets of action parameters from the historical control data as the action list, calculate the probability of each action appearing in the action list, and obtain the action probability distribution, as shown in the formula:

[0064] P(X=x i Y = y i ,...) a =f(x) i y i ,...) a / t(4)

[0065] Here, variables X, Y, etc., represent different types of actions in the action list, x i y i Equals represent the action values ​​of different types of actions in the i-th term; P(X=x i Y = y i ,...) a This indicates that when action variables X, Y, etc., take the value x i y i The probability of the action of waiting occurring in historical data, f(x) i y i ,...) a This indicates that the values ​​of X, Y, etc. in the sample are x. i , y i The frequency of t represents the total number of historical data. Based on this, a probability table corresponding to different action combinations can be generated. The action-probability table is shown in Table 2.

[0066] Table 2. Example of an action-probability table

[0067] A P(X, Y) [[x1, y1,...]] [P(X=x1, Y=y1,... a ]] [[x2, y2,...]] [P(X=x2, Y=y2,... a ]] ...... ...... [[x m , y m ,...]]]> [P(X=x m -Y=y m ,...) a ]]>

[0068] Where A is the action space, P(X, Y) is the probability of occurrence of historical data corresponding to different actions, m is the total number of actions contained in the action space, and variables X, Y, etc. represent different types of actions in the action list, x i y i Equals represent the action values ​​of different types of actions in the i-th term; P(X=x i Y = y i ,...) a This indicates that when action variables X, Y, etc., take the value x i y i The probability of the action X = x occurring in historical data, ∑P(X=x i Y = y i ,...) a =1.

[0069] Choose an action according to the action probability distribution, i.e., P(X=x) i Y = y i ,...) a The probability of choosing to perform an action [x] i y i ,...].

[0070] Taking the chiller system of a subway station in Guangzhou as a case study, a simulated environment was built for training and testing using historical data. Wet-bulb temperature and cooling load were used as input state parameters, and the frequency and number of cooling water pumps and cooling towers were used as actions for output control. A reward function was constructed with the optimization objective of reducing system energy consumption to build and train the intelligent agent. The results show that, compared with the ε-greedy strategy commonly used in Q-learning algorithms, the action selection method using a random-probability weighted composite sampling strategy in this case can improve the system's energy saving rate by approximately 13.5% (the meaning of k is given in Equation 5).

[0071] Under different strategies, the agent performs multiple rounds of Q-learning training based on the same set of state parameter data (the exploration rate is continuously reduced until it becomes constant at 0.1 after 20 training iterations). The total energy saving rate of the system increases with the number of training iterations until it oscillates around a constant value. Among them, the final energy saving rate of the optimization training using this method is generally higher than the energy saving rate of the commonly used ε-greedy strategy.

[0072] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A heating, ventilation, and air conditioning (HVAC) control method based on a random probability weighted composite sampling strategy, characterized in that, Workflow: Step 101: Construct and initialize the intelligent agent, determine the state space, action space, reward function, and state-action Q-table of the HVAC system, and set the initial exploration rate to 1; Step 102: Obtain the current status of the HVAC system and determine whether the current exploration rate is less than the set value for starting probability-weighted sampling; Step 103: Select an action using a random probability weighted composite sampling strategy; if the current exploration rate is not less than the set value k at which probability weighted sampling is first used, determine whether the generated random number is greater than the current exploration rate. If the random number between 0 and 1 is greater than the exploration rate, select the action with the largest Q value in the current state from the state-action Q table. If the random number is not greater than the exploration rate, select a random action in the action space; if the current exploration rate is less than the set value k at which probability weighted sampling is first used, determine whether the generated random number is greater than the current exploration rate. If the random number is greater than the exploration rate, select the action with the largest Q value in the current state from the state-action Q table. When the random number is not greater than the exploration rate, calculate the probability of each action in the historical training data to obtain the probability distribution of each action in the action space, and select actions in the action space according to the probability distribution. Step 104: Execute the control command corresponding to the action in the current state, obtain the state parameters and reward feedback of the next state of the HVAC system, and update the Q table; Step 105: Decrease the exploration rate as the training cycle is set; Step 106: Determine whether the agent training has reached the end standard. If not, return to step 102 to continue training. If the standard is reached, export the model, input the state parameters into the trained agent, and obtain the control commands that should be issued in that state.

2. The HVAC control method based on a random probability weighted composite sampling strategy as described in claim 1, characterized in that, In step S101, the method for constructing the agent is as follows: An intelligent agent is an entity that can perceive the environment, make decisions, and interact with the environment; the construction of Q-learning agents uses the function approximation method, which requires defining the state space, action space, reward function, and learning algorithm; State space: The state space includes multiple state parameters. In HVAC systems, state parameters are usually a vector or a set of features that help the agent make decisions and choose actions. Its sampling period should be set to 10 to 15 minutes. Action space: Action space refers to the set of all possible actions that an intelligent agent can choose, including the control commands that need to be issued in the HVAC system. Both actions and states in the state-action Q-table need to be discretized. For state parameters, nearest neighbor rounding can be used, as shown in the formula. S=round(S′ / b)×b (1) Where S represents the processed data, S' represents a certain type of state parameter in the original state parameter set, round() represents the rounding function, and b represents the value of the nearest neighbor multiple, which can be freely defined according to the value range of different types of states. For motion parameters, the motion range and discrete spacing are set according to the control accuracy and equipment safety limits; The state row names in the state-action Q table are state tuple parameters formed by the permutation and combination of various state values; the action column names in the state-action Q table are either in list form or individual action space values ​​formed by the permutation of each action space. Reward Function: The reward function is the optimization objective of the agent, used to evaluate the quality of each action taken by the agent in the environment. In HVAC energy-saving control, the reward function variables include two main categories of HVAC system performance parameters: energy consumption and comfort. The reward function formula is: R=u E ×E+u C ×C (2) Where R represents the reward function, u E u C The value of u represents the weight of each type of variable involved in the reward calculation. E and C represent the scores of energy consumption and comfort variables involved in the reward calculation, respectively. The u value can be set as a constant or a variable correlation function according to the optimization focus. The lower the system energy consumption and the lower the deviation of indoor comfort from the standard, the larger the reward function value. The objective of the optimization control is to achieve the comprehensive reward after weighted processing. Energy consumption variables include: target equipment or system power, COP, and total energy consumption, which represent parameters of system energy consumption. Comfort variables include: indoor temperature and indoor relative humidity, which represent parameters of environmental comfort. Learning algorithm: Q-learning is a reinforcement learning method based on temporal difference Markov decision to iteratively update the state-action Q table. Its state and action quantities need to be discretized. The Q table is shown in Table 1. Table 1. Q Table Where A is the action space, a represents an action, m is the total number of actions in the action space, S is the state space, s represents a state, n is the total number of actions in the state space, and each Q value is represented as Q(s,a), which represents the action-state Q value obtained by taking action a when the environment state is s. The Q table can be built directly based on the known state and action space, or all action rows can be created first based on the system's action space, and the corresponding state column can be created after each state is obtained and discretized.

3. The HVAC control method based on a random probability weighted composite sampling strategy as described in claim 1, characterized in that, In step S101, the agent initialization method is as follows: During the initialization phase before training begins, the Q-table and Q-table update strategy are initialized. Initialize Q-table: Initialize all newly created elements of Q-table to 0 or a small random value, representing the agent's initial estimate of all state-action pairs; Q-table update strategy: The Q-table update formula for the function approximation learning algorithm based on the meaning of the Q-table is as follows: Q(s,a)←Q(s,a)+α[r+γmaxQ(s′,a′)-Q(s,a)] (3) Wherein, the arrow indicates that the left variable is updated with the calculated value on the right, Q(s,a) represents the action-state Q value of taking action a in environmental state s, s' represents the environmental state s' at the next moment, maxQ(s',a') represents the known maximum Q value in state s', r is the reward brought by the action-state combination in the previous round, α is the learning rate, γ represents the discount factor, that is, the degree of influence of future rewards on the current action decision, and maxQ(s',a') represents the known maximum Q value in state s'; The action 'a' that updates the Q-table is the action decision selected based on the current state. In the Q-learning algorithm, the exploration rate ε refers to the probability that the agent will explore new actions during the decision-making process. The initial exploration rate ε is set to 1, which means that in the initial stage, the agent will select actions completely randomly to explore in order to explore the state-action space as much as possible. After each optimization cycle, the agent reduces the exploration rate, and eventually retains a certain exploration rate to continuously update the Q table.

4. The HVAC control method based on a random probability weighted composite sampling strategy as described in claim 1, characterized in that, In step S103, the random probability weighted composite sampling strategy is as follows: The random probability weighted composite sampling strategy employs a combination of Monte Carlo sampling and probability weighted sampling for action decision-making: If the current exploration rate is not less than the set value at which probability weighted sampling is first applied, it checks whether the generated random number is greater than the current exploration rate ε. If the random number between 0 and 1 is greater than the exploration rate, the action with the largest Q value in the current state is selected from the state-action Q table. If the random number is not greater than the exploration rate, a random action is selected from the action space. If the current exploration rate is less than the set value at which probability weighted sampling is first applied, it checks whether the generated random number is greater than the current exploration rate. If the random number is greater than the exploration rate, the action with the largest Q value in the current state is selected from the state-action Q table. If the random number is not greater than the exploration rate, the probability of each action appearing in the historical training data is calculated to obtain the probability distribution of each action in the action space. Action selection is then performed in the action space based on the probability distribution. The formula for the random probability weighted composite sampling strategy combines Monte Carlo sampling and probability weighted sampling in the early and late stages of training, respectively, and integrates them throughout the entire training process. Let π(a|s) represent the probability of action a being selected in state s. The formula for the random probability weighted composite sampling strategy is then expressed as: Where ε is a predefined value representing the current exploration rate, and probability-weighted sampling is started when ε is less than k. It is recommended that k be between 0.3 and 0.

5. P(X=x i Y = y i ,...) a This indicates that when action variables X, Y, etc., take the value x i y i The probability of action a occurring in historical data is given by π(a|s), where π(a|s) is the probability that action a is selected in state s.

5. The HVAC control method based on a random probability weighted composite sampling strategy as described in claim 4, characterized in that, The probability distribution of each action is calculated as follows: Determine the amount of data t required to generate the action-probability table. Read the last t sets of action parameters from the historical control data as the action list. Calculate the probability of each action appearing in the action list to obtain the action probability distribution, as shown in the formula: P(X=x i ,Y=y i ,…) a =f(x i ,y i ,...) a / t (4) Here, variables X, Y, etc. represent different types of actions in the action list, x i y i Equals represent the action values ​​of different types of actions in the i-th term; P(X=x i Y = y i ,...) a This indicates that when action variables X, Y, etc., take the value x i y i The probability of the action of waiting occurring in historical data, f(x) i ,y i ,...) a This indicates that the values ​​of X, Y, etc. in the sample are x. i ,、y i The frequency of t represents the total number of historical data, which can be used to generate a probability table corresponding to different action combinations.

Images