Energy-saving control method and intelligent agent for general air-conditioning system based on reinforcement transfer learning

Abstract

The present invention discloses a general air-conditioning system energy-saving control method and intelligent agent based on reinforcement transfer learning, including: obtaining the air-conditioning system operation data and the energy consumption data of each air-conditioning device at the corresponding timestamp; inputting the operation data and the energy consumption data into the intelligent agent for learning, and setting the basic parameters of the intelligent agent; according to the implementation parameters of the target air-conditioning system project 1 to be controlled, the intelligent agent determines whether the Q table of the project 2 for migration can be found from the existing training library, if so, the state space and the action space are normalized, and the normalized Q table is processed; the threshold and step parameters of the action set of the intelligent agent that completes the migration are set, and the control instructions are issued to the target air-conditioning system; if not, the intelligent agent is directly set according to the reinforcement learning requirements. The present invention does not rely on a large number of sensor data, load forecasting, and does not need to be modeled to achieve a high level of energy-saving control, and can be used for different air-conditioning systems in various building scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a universal air-conditioning system energy-saving control method and an intelligent agent based on reinforcement transfer learning. Background Art

[0002] In the energy consumption of public building operation, the energy consumption of air conditioning system usually accounts for more than 50% of the total energy consumption. At present, although there have been many research results in the design and construction of air conditioning room, there are still many problems in the control and operation and maintenance of air conditioning system.

[0003] For example, many buildings still use manually controlled air conditioning systems. Due to the low control accuracy of this method, the air conditioning system is often used redundantly in order to achieve the indoor temperature control target, resulting in a waste of electricity.

[0004] For another example, although some buildings use a PID (proportional-integral-differential)-based building automatic monitoring system (BAS) to control the air-conditioning system, its energy efficiency improvement mainly depends on built-in expert experience. However, the energy-saving control effect based on expert experience will decrease as the system is used, because expert experience is based on the operating parameter settings of fixed working conditions, and the actual application environment of the air-conditioning system is non-static. Expert experience can only provide better system operation effects under fixed working conditions and cannot continuously adjust to changing working conditions. For complex working conditions with multiple chillers operating in coordination, the control method based on expert experience may not achieve the optimal effect.

[0005] Energy-saving control methods that rely on model optimization require high-precision equipment models to support them. Although this method can achieve certain energy-saving optimization effects, the performance of the equipment will degrade after a period of use, and the model needs to be updated to ensure the optimization effect. In addition, this method relies on long-term operation and maintenance data and can only be modeled for specific types of building scenarios, such as data centers and shopping malls, and has poor versatility.

[0006] Therefore, the current building energy-saving market urgently needs an air-conditioning system energy-saving control method that can be applied to most building scenarios, which has a high scope of application while ensuring the energy-saving rate and can be better applied to engineering practice. Summary of the invention

[0007] One of the purposes of the present invention is to provide a universal air-conditioning system energy-saving control method based on reinforcement transfer learning to solve the technical problem that the existing technology cannot be applied to automatic energy-saving control of air-conditioning systems in most building scenarios. While ensuring the energy-saving rate, it has a higher scope of application and can be better applied to engineering practice.

[0008] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides a general air-conditioning system energy-saving control method based on reinforcement transfer learning, the method comprising:

[0009] The agent obtains the operating data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp;

[0010] The operation data and the energy consumption data are input as input into the intelligent body for learning, and the basic parameters of the intelligent body are set, wherein the basic parameters include a state set, an action set, a reward function of the environment, and an ε-greed strategy setting value;

[0011] According to the implementation parameters of the target air-conditioning system project 1 to be controlled, the agent determines whether the Q table of project 2 for migration can be found from the existing training library. If so, the state space and the action space are normalized to establish a correlation between the Q tables of different state and action spaces, and the normalized Q table is processed to complete the migration. The implementation parameters include but are not limited to the air-conditioning system load, equipment type, equipment quantity and connection method. If not, the agent is directly set according to the reinforcement learning requirements;

[0012] The threshold and step size parameters of the action set of the agent are restricted, and the agent issues a control instruction to the target air-conditioning system.

[0013] Furthermore, the method further comprises:

[0014] After the intelligent agent sends a control instruction to the target air-conditioning system, it synchronously obtains the operating parameters of the current timestamp and calculates the current system energy efficiency index. The system energy efficiency index is used as a reward value and is regarded by the intelligent agent as a basis for determining whether to store it in the Q table. The system energy efficiency index includes but is not limited to one or more of COP, EER or PUE.

[0015] Preferably, the basic parameters of the agent are as follows:

[0016] ；

[0017] Wherein, S represents the state set of the agent, which includes the instantaneous cooling load of the system and the outdoor meteorological parameters; A represents the action set of the agent, which includes the operating state of the air-conditioning equipment and its current action setting parameters; r represents the reward function of the environment, which is the energy efficiency index of the air-conditioning system, and the system energy efficiency index includes but is not limited to one or more of COP, EER or PUE;

[0018] p represents the state transition probability of the environment, and γ represents the discount factor;

[0019] is the maximum Q value that can be obtained in the next state.

[0020] Preferably, when performing transfer learning, the state space and the action space are normalized so that a correlation is established between Q tables between different state and action spaces, and the normalized Q table is processed specifically including:

[0021] The normalization formula is as follows:

[0022] ;

[0023] Wherein, x is the object data to be normalized, min and max are the maximum and minimum values ​​in the object data, respectively. To serve as the benchmark value for normalization, divide the value of each state (s) and action (a) by the corresponding benchmark value and , and scaled to the interval [0,1], as shown below:

[0024] , ;

[0025] , ;

[0026] Among them, Q1 is the action of the air conditioning system at each state point in project 1, and Q2 is the action of the air conditioning system at each state point in project 2;

[0027] After normalization, a Q value in the new Q table It needs to be obtained by weighting the Q value close to its index value in the original Q table. The specific formula is as follows:

[0028] ;

[0029] in, is the value of the normalized row index of the original Q table, that is, the state space is mapped to the row index; is the value of the normalized column index of the original Q table, that is, the action space is mapped to the column index;

[0030] and is a parameter, and its value ∈(0,1) is used as the base to construct a function with the absolute value of the distance between the indices as the exponent. When the distance is close to or equal to 0, the function value is 1. When the distance increases, the function value decreases rapidly, and the speed of decrease is related to the value of the parameter.

[0031] It constitutes the contribution weight of the Q value of the point to the new Q value required; is the normalization coefficient, which is used to scale the sum of all weights to 1. The calculation formula of the normalization coefficient is as follows:

[0032] ;

[0033] When migrating systems with different energy efficiency indicators, the Q value of the entire Q table needs to be scaled according to the size of the reward value.

[0034] Preferably, the limiting of the threshold and step size parameters of the action set of the agent that completes the migration specifically includes:

[0035] The control time interval shall not be less than 10 minutes per time;

[0036] The control step size, i.e. the adjustment frequency, shall not be higher than 2Hz / time;

[0037] The temperature adjustment interval shall not exceed 1℃ / time;

[0038] The safety thresholds of chiller outlet water temperature, water pump frequency, and cooling tower fan frequency refer to the design parameters of the entire air-conditioning system and the design parameters of the corresponding equipment.

[0039] Furthermore, the method also includes: deploying hyperparameters of reinforcement learning, including learning rate, discount coefficient, initial random coefficient epsilon, random coefficient attenuation, and debugging the reward value setting of the intelligent agent according to the actual needs of the target air-conditioning system project 1 to be controlled.

[0040] Preferably, the step of debugging the reward value setting of the agent according to the actual requirements of the target air-conditioning system project 1 to be controlled specifically includes:

[0041] The system COP is used as the reward value and the system COP is standardized. The system energy efficiency ratio COP is obtained by the agent when issuing control instructions and is calculated based on the current operating parameters.

[0042] Multiply the standardized system COP by the temperature difference correction factor. The specific formula is as follows:

[0043] ;

[0044] Among them, COP is the system energy efficiency ratio, Tcws is the cooling water main supply water temperature, and Tcwr is the cooling water main return water temperature.

[0045] In a second aspect, an embodiment of the present invention further provides an intelligent agent for energy-saving control of a general air-conditioning system based on reinforcement transfer learning, the intelligent agent comprising:

[0046] The data acquisition module is used to obtain the operation data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp;

[0047] A reinforcement learning module, used to input the operation data and the energy consumption data as input into the intelligent body for learning, and set basic parameters of the intelligent body, wherein the basic parameters include a state set, an action set, a reward function of the environment, and an ε-greed strategy setting value;

[0048] A transfer learning module is used to determine whether a Q table for project 2 to be transferred can be found from an existing training library according to the implementation parameters of the target air-conditioning system project 1 to be controlled. If so, the state space and the action space are normalized to establish a correlation between the Q tables of different state and action spaces, and the normalized Q table is processed to complete the transfer, wherein the implementation parameters include but are not limited to the air-conditioning system load, equipment type, equipment quantity and connection mode;

[0049] The instruction issuing module is used to set the threshold and step parameters of the action set of the intelligent agent that has completed the migration, and issue control instructions to the target air-conditioning system.

[0050] In a third aspect, an embodiment of the present invention further provides an intelligent device, including a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method as described above.

[0051] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described above.

[0052] Compared with the prior art, the energy-saving control method for a general air-conditioning system based on reinforcement transfer learning provided by the embodiment of the present invention has at least the following beneficial effects:

[0053] The embodiment of the present invention eliminates the cumbersome equipment modeling process of existing energy-saving optimization means, directly learns the key operating parameters and energy consumption parameters of the existing air-conditioning system, and can achieve a higher level of energy-saving control without relying on a large number of sensor data and load forecasts. There is no need to perform a large amount of parameter adjustment and model optimization work like modeling, and there is no need to spend equipment costs such as high-performance GPUs or TPUs, thereby reducing the time cost of model design, data processing, and algorithm implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The preferred implementation modes will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present invention.

[0055] Figure 1 This is a flow chart of an air-conditioning system energy-saving control method based on reinforcement transfer learning according to an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the structure of an air-conditioning system in an air-conditioning system energy-saving control method based on reinforcement transfer learning according to an embodiment of the invention;

[0057] Figure 3 A schematic diagram of energy-saving control transfer learning for an air-conditioning system based on reinforcement transfer learning according to an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of normalization of a migration Q table for energy-saving control of an air-conditioning system based on reinforcement migration learning according to an embodiment of the present invention;

[0059] Figure 5 This is a performance curve of a refrigeration machine for energy-saving control of an air-conditioning system based on reinforcement transfer learning in an embodiment of the present invention;

[0060] Figure 6 This is a state distribution diagram of the fan frequency and cooling pump frequency of the cooling tower according to an embodiment of the present invention;

[0061] Figure 7 This is a comparison chart of the cooling water pump frequency operation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0063] In order to simplify the drawings, only the parts related to the invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0064] The following mainly takes some specific embodiments as examples to describe in detail the implementation of the technical solution of the present invention.

[0065] After years of research and development experience, the inventors found that in order to improve the flexibility of automatic energy-saving control, it is necessary to get rid of the limitations of the device model on the control method.

[0066] Reinforcement learning is a machine learning method that learns by interacting with the environment to obtain reward values. Compared with model optimization, reinforcement learning has the advantages of low hardware requirements, fast adaptation to environmental changes, and fast response speed. In the field of air-conditioning system control, equipment performance will change with use, and the original control strategy based on the initial parameters of the equipment may become invalid. If reinforcement learning is used for optimized control, the negative impact of changes in equipment performance on the control effect can be reduced.

[0067] However, a good agent in reinforcement learning needs to interact with the environment for a long time to reach the ideal state. In many industrial application scenarios, untrained agents without any optimization effect are not allowed to learn at will in actual scenarios.

[0068] After research, the inventor found that on the cooling side of the air-conditioning system, the equipment in various control systems is mostly similar, usually including chillers, cooling water pumps and cooling towers. The main differences are the connection methods of the equipment and the power and performance curves. On the freezing side, the project can be classified according to the host type and the type of chilled water pump.

[0069] In actual operation, the instructions given to the control system are also similar, including the number of devices to be turned on, the frequency conversion of cooling tower fans and water pumps, the outlet water temperature of the main unit, etc. Therefore, the reinforcement learning agent that has been learned in a certain air-conditioning system can potentially be transferred to other air-conditioning system controls, that is, through transfer learning, the knowledge or patterns learned in a certain field or task can be applied to different but related fields or problems, which solves the time cost problem of reinforcement learning in the process of training agents. For example, if the parameters of the load part of the reinforcement learning agent trained in an air-conditioning system of a certain structure are reduced by a certain proportion, it can be applied to a system with the same structure but a load capacity reduced by the same proportion.

[0070] However, in actual applications, the differences between systems may be more complex, and the agent cannot adapt to the new environment immediately after migration, but the learning time for the agent to reach the usable standard will be reduced. In this way, the disadvantage of reinforcement learning that it takes time to train can be compensated, so that reinforcement learning can be more widely used.

[0071] In order to solve the problems that model optimization is difficult, costly and time-consuming to implement in actual engineering projects due to the high training cost of the existing energy-saving optimization control model of the air-conditioning system in buildings, the difficulty in modeling special systems, and the high requirements on the accuracy and number of actual system sensors for model establishment, the embodiment of the present invention uses the reinforcement learning method to train an initial intelligent agent with historical operation data of air-conditioning systems in a large number of different building scenarios, and through transfer learning, the agent can quickly achieve energy-saving optimization in new projects.

[0072] The overall optimization logic of the intelligent agent is to optimize the global energy consumption of the entire air conditioning system. For example, when the wet-bulb temperature is high, appropriately increasing the outlet water temperature of the chiller can reduce the excessive workload of the cooling tower, thereby reducing energy consumption. That is, when the cooling tower efficiency is low, increasing the outlet water temperature can reduce the dependence on the cooling tower, thereby reducing the load of the cooling tower and avoiding the increase in system energy consumption due to the decrease in cooling tower efficiency; similarly, when the wet-bulb temperature is low, while reducing the outlet water temperature of the chiller, improving the cooling effect can also help save energy. The above two energy-saving paths are for reference only. The specific energy-saving strategy needs to be determined by the agent according to the actual situation of the project.

[0073] like Figure 1 As shown, a general air conditioning system energy-saving control method based on reinforcement transfer learning, the method comprising:

[0074] S1, the agent obtains the operating data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp;

[0075] S2, inputting the operation data and the energy consumption data as input into the intelligent body for learning, and setting the basic parameters of the intelligent body, wherein the basic parameters include an environment state set, an action set, an environment reward function, and an ε-greed strategy setting value;

[0076] S3. According to the implementation parameters of the target air-conditioning system project 1 to be controlled, the agent determines whether the Q table of project 2 for migration can be found from the existing training library. If so, the state space and the action space are normalized to establish a correlation between the Q tables of different state and action spaces, and the normalized Q table is processed to complete the migration. The implementation parameters include but are not limited to the air-conditioning system load, equipment type, equipment quantity and connection method. If not, the agent is directly set according to the reinforcement learning requirements.

[0077] S4. Setting the threshold and step size parameters of the action set of the agent that has completed the migration, and the agent sends a control instruction to the target air-conditioning system.

[0078] In the embodiment of the present invention, it is necessary to rely on the intelligent sensors in the air-conditioning system of the existing building to collect data to obtain the operation data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp:

[0079] The operating data of the air-conditioning system includes pipeline operating parameters and equipment operating parameters. The pipeline operating parameters include the supply and return water temperature of the chilled (cooling) water main, the supply and return water temperature of the chilled (cooling) water main, etc.; the equipment operating parameters include the cooling tower inlet and outlet tower temperature, the main unit chilled (cooling) water inlet and outlet water temperature, the chiller current percentage (load rate), the chilled (cooling) water pump frequency, the cooling tower fan frequency, etc.

[0080] The key operation control parameter data collected by the sensor include at least: the supply and return water temperature of the chilled water main, the flow rate of the chilled water main, the inlet and outlet temperatures of the cooling tower and the corresponding timestamps, outdoor meteorological parameters (air temperature, humidity), and the corresponding ambient temperature of the air conditioning terminal.

[0081] In addition, it is also necessary to rely on the intelligent data collection of the air conditioning system in the existing buildings. According to different air conditioning equipment, in addition to collecting the operating status of each air conditioning system, it is also necessary to collect key operating data, including: the evaporator inlet and outlet water temperature of the chiller, the condenser inlet and outlet water temperature, load rate, the fan frequency of the cooling tower, the cooling water flow, the frequency, flow, head of the water pump, etc. At the same time, the nameplate parameters of each air conditioning equipment need to be provided.

[0082] In this embodiment of the invention, the following real-time collected reading data of the air-conditioning system needs to be input to the Agent to complete the task of energy-saving optimization control, including:

[0083] Outside temperature (℃), outside humidity;

[0084] System load (kW), operating status of each air conditioning equipment;

[0085] Key parameters of each air-conditioning equipment (such as the inlet and outlet water temperature and load rate of the evaporator and condenser inside the chiller; the fan frequency of the cooling tower; the frequency of the water pump, etc.);

[0086] Energy consumption parameters of each device (active power and accumulated power).

[0087] In the embodiment of the present invention, it is also necessary to rely on the corresponding electricity sub-item metering data of the air-conditioning equipment in the existing building, such as the collected active power and cumulative electricity data of each air-conditioning equipment. In addition to the main unit, at least one electricity meter is required for each type of equipment, and each main unit must be equipped with an electricity meter.

[0088] In the embodiment of the present invention, the data collection protocol can support general protocols such as BACnet, Modbus-TCP, OPC-UA, etc., and also support collection via a platform or cloud based on an http API.

[0089] For example, the air conditioning system of a certain project is as shown in the attached manual. Figure 2 As shown, it consists of 2 centrifugal chillers of the same size, 3 chilled water pumps of the same size, 3 cooling water pumps of the same size and 2 cooling towers of the same size, and the cooling tower fans and water pumps are all frequency-converted, and the system is a full parallel structure. The current system operation strategy is that the outlet water temperature is fixed at 8°C, the cooling tower fans and water pumps are operated at a fixed frequency of 45Hz, and the chiller addition and subtraction is determined according to the load.

[0090] The chiller parameters are: cooling water supply and return water temperature is 37 / 30℃, rated cooling water flow rate is 820m 3 / h, chilled water supply and return water temperature is 7 / 12℃, rated chilled water flow rate is 980m 3 / h, rated COP is 5.5, rated cooling capacity is 5651kW, and rated power is 1027.5 kW.

[0091] The cooling tower parameters are: rated inlet and outlet temperature is 37 / 30℃, rated cooling water flow rate is 520m 3 / h, rated air volume 400000 m 3 / h, fan rated frequency 50Hz, fan rated power 20kW.

[0092] The parameters of the cooling water pump are: rated water flow 800m 3 / h, rated frequency 50 Hz, rated power 35 kW, rated head 20m.

[0093] Among them, the parameters of the chilled water pump are: rated water flow 800m 3 / h, rated frequency 50 Hz, rated power 35 kW, rated head 20m.

[0094] In step S1, the embodiment of the present invention assembles an edge server loaded with an algorithm, connects to the existing control system of the project through the BACnet protocol, and can obtain the operation data of the air-conditioning system and the energy consumption data of each air-conditioning equipment.

[0095] In step S2, the parameters related to reinforcement learning are set, and the environmental state can be composed of two dimensions: the load borne by the system and the wet-bulb temperature.

[0096] For complex air conditioning systems with multiple devices, the embodiment of the present invention allocates an agent for each action. For example, for a chiller group, modifying or setting the outlet water temperature and adding or subtracting the number of machines requires the establishment of two agents for learning respectively, and their state spaces are both two-dimensional spaces composed of loads and external meteorological parameters.

[0097] The maximum wet-bulb temperature is set to 32°C, the maximum load rate is set to 1, and two sets of random parameters are generated in each time step through a random process, falling in the intervals of [0.75, 1] ​​and [0.3, 1] respectively. ), the parameter group is multiplied by the corresponding maximum parameter value, and the result is used as the state parameter value of the time step.

[0098] Set the Q value update rule as shown below:

[0099] ;

[0100] The reward value R is the COP of the air-conditioning system, and may also be an energy efficiency index parameter such as EER or PUE. The reward value may be one parameter value or a combination of multiple parameter values.

[0101] In reinforcement learning, the agent organizes all Q values ​​into a Q table to store the Q value Q(s, a) of each state-action pair.

[0102] Depending on the building functions or scenarios where the air conditioning system is located, and the types or models of the air conditioning system host, the reward value can be set as a judgment indicator such as EER or PUE in addition to COP. The calculation formulas for each indicator are as follows:

[0103] ;

[0104] ;

[0105] ;

[0106] In the above formula, COP and EER are the performance parameters of the air conditioning system. is the cooling capacity of the main engine shaft work, The EER is the cooling capacity of the input power of the main engine motor, and W is the total energy consumption of the air conditioning system equipment. For the main engine, the input power of the motor is not completely converted into shaft work, and part of it is lost through friction. The EER index is often used to measure the cooling capacity of the unit motor input power of the fully enclosed compressor. The specific choice of which index needs to take into account the type and corresponding model of the main engine. PUE is an indicator that specifically reflects the energy consumption of the data center, among which is the energy consumption of the entire data center, is the total energy consumption of IT equipment in the data center.

[0107] At the beginning of learning, all Q values ​​in the Q table are 0.

[0108] If the agent's strategy is set to choose the action with the highest Q value in the current state, once the Q value starts to update, the Q value of the first action taken will become positive, so that this action will become the action with the highest Q value, and the agent will immediately start to pursue this action and will not try other actions.

[0109] At this time, we can use the ε-greed strategy, that is, when the agent is in state s, there is a probability of 1-ε to choose the action that maximizes Q(s,a), and there is a probability of ε to randomly choose an action. Here, ε is set to 0.1.

[0110] like Figure 3 As shown in the figure, the agent mainly uses the MDP (Markov Decision Process) method to evaluate the impact of state-action on reward value. The calculation formula of the MDP method is:

[0111] ; (1)

[0112] In formula (1), S represents the set of environmental states collected by the agent, A represents the set of actions of the agent, R represents the reward function of the environment, P represents the state transition probability of the environment, and γ represents the discount factor. represents the immediate reward obtained by taking action a in state s, represents the probability of taking action a in state s and transferring to the next state s, represents the probability of selecting action a in state s.

[0113] In formula (1), Represents state-action pairs In Strategy The cumulative expected reward under the action value function is usually used to evaluate the strategy The advantages and disadvantages.

[0114] In this embodiment of the present invention, the optimal strategy for maximizing the cumulative reward is obtained. , that is, the strategy of making the overall energy efficiency ratio COP of the air-conditioning system the highest is the ultimate goal of reinforcement learning. The above formula (1) can be rewritten as:

[0115] ; (2)

[0116] In formula (2), Represents reinforcement learning discovery in the current The strategy to make the current air conditioning system operate most effectively The expected reward here is , that is, the immediate reward function, is set to the instantaneous system energy efficiency index of the air-conditioning system when action a is selected at the operating state point s of the air-conditioning system; the operating state point s of the air-conditioning system is composed of outdoor meteorological parameters and the current load demand of the system; action a is the strategy issued by the Agent Instructions executed by the rear air conditioning system.

[0117] The reward function is an energy efficiency index of the air conditioning system, wherein the energy efficiency index of the system includes but is not limited to one or more of COP, EER or PUE;

[0118] Taking into account the influence of learning rate and discount factor on the overall learning effect, the update rule of Q value is shown in formula (3):

[0119] ; (3)

[0120] In formula (3), α is the learning rate, which represents how much the Q value will be learned from the new Q value when the Q value is updated. When the learning rate is 0, it means that the Q value is not updated at all. When it is 1, it means that the old Q value will be completely replaced by the new Q value.

[0121] R is the reward value, which is the reward obtained by the action agent from the environment. In the embodiment of the present invention, the reward value is set to the COP of the air conditioning system. Generally speaking, the purpose of training is to maximize the reward obtained, so the reward value function is often set in a form that is strongly associated with the goal to be achieved.

[0122] is the discount factor, The maximum Q value that can be obtained in the next state after this action. The product of the two represents the importance of future rewards to learning. When the discount factor is 0, it means that long-term rewards have no effect, and when it is 1, it means that long-term rewards are more important. In reinforcement learning, the agent will organize all Q values ​​into a Q table to store the Q value Q(s, a) of each state-action pair.

[0123] In the process of Agent's exploration of Q table, in order to balance the needs of exploration and utilization, it is necessary to introduce ε-greed strategy. In ε-greed strategy, the agent randomly selects an action with probability ε (exploration), or selects the action currently considered the best with probability 1-ε (utilization).

[0124] For complex systems such as HVAC, in order to maintain system stability and avoid phenomena such as cold machine surge that are harmful to the equipment, we should not be too aggressive in the exploration process. Generally, ε is set in the range of (0.05, 0.35).

[0125] Furthermore, the method further comprises:

[0126] Set the hyperparameters of reinforcement learning, including learning rate, discount factor, initial random coefficient epsilon, random coefficient attenuation, and debug the reward value setting of the agent according to the actual needs of the target air-conditioning system project 1 to be controlled.

[0127] In order to enable the reinforcement learning Q-learning agent to improve the energy-saving control optimization effect more quickly, before the technical solution of the invention is formally applied to the building air-conditioning system, in order to help the reinforcement learning agent to improve the energy-saving control optimization effect more quickly, in the specific implementation of the invention, it is necessary to set the hyperparameters in advance according to the characteristics of the HVAC system. The hyperparameters that need to be set here include the learning rate learning_rate, the discount coefficient (i.e., ), initial random coefficient epsilon, random coefficient decay decay.

[0128] In the embodiment of the present invention, the agent has been able to control the air conditioning system normally through the previous training. In order to ensure that the learning of the agent does not affect the comfort and stability of the internal environment temperature of the building, the learning rate should not be set too high. The learning rate can be set to the interval of (0.1, 0.2); the discount coefficient It represents the value of future rewards to the current state. Considering the non-steady-state nature of the building environment, the current state has little impact on future rewards, so the discount factor can be set to the interval of (0.05, 0.1). The initial random coefficient epsilon represents the initial exploration rate of the agent. When learning on the source system, in order to encourage the agent's exploration, the initial random coefficient is 1.

[0129] When learning on the transfer system, in order to better utilize the transferred knowledge, the initial random coefficient is taken in the interval of (0.4, 0.6) to meet the needs of different degrees of transfer; the random coefficient decay represents the speed at which the random coefficient decays as the learning process progresses. When the random coefficient decays to 0, the agent will no longer explore and the learning process will also stop. In order to prolong the learning process, the decay value decay is taken as little as possible to be no higher than 0.01.

[0130] In step S3, it is assumed that in the Agent historical data training library, there is a group of systems whose equipment models are consistent with those in the air conditioning system scenario to be planned and implemented in the embodiment of the present invention, but the system contains four chillers, seven cooling towers, four cooling water pumps, and four chilled water pumps, and the connection method is that the pumps are connected in series first and then in parallel. After setting the reinforcement learning parameters, the Agent reads the operating data and finds that under partial load conditions, the number of activated equipment in the system in the training library is consistent with that in the embodiment of the present invention, and transfer learning can be performed at this time.

[0131] In the transfer of reinforcement learning, the core idea is to apply the learning results of the agent in project 1 to project 2. When processing transfer learning, in order to establish a connection between the Q tables between different state and action spaces, it is first necessary to normalize the state and action intervals of both and compress them into the interval [0,1]. The reference value taken during normalization is specified according to the characteristics of the specific case. The normalization here uses the linear normalization method, and the specific formula is shown in formula (4):

[0132] ; (4)

[0133] In formula (4), x is the object data to be normalized, min and max are the maximum and minimum values ​​in this set of data, respectively. It is used as the reference value for normalization.

[0134] like Figure 4 As shown, in this embodiment of the present invention, the value of each state (s) and action (a) is divided by the corresponding reference value ( and ), thereby scaling them to the interval [0,1], as shown in formulas (5) and (6):

[0135] , ; (5)

[0136] , ; (6)

[0137] In equations (5) and (6), Q1 is the action of the air conditioning system at each state point in project 1, and Q2 is the action of the air conditioning system at each state point in project 2.

[0138] Figure 4As shown in the figure, before normalization, the data in the two tables Q1 and Q2 cannot be mapped to each other; since the rows and columns of the Q table are discrete, after normalization, the rows and columns still cannot correspond to each other, so a Q value in the new Q table It needs to be obtained by weighting the Q value close to its index value in the original Q table. The specific formula is as follows:

[0139] ; (7)

[0140] In formula (7), is the value of the normalized row index of the original Q table, where the state space is mapped to the row index; is the value of the normalized column index of the original Q table, where the action space is mapped to the column index; and Is a parameter, whose value ∈ (0,1), and is used as the base to construct a function with the absolute value of the distance between indices as the exponent. When the distance is close to or equal to 0, the function value is 1. When the distance increases, the function value decreases rapidly, and the speed of decrease is related to the value of the parameter.

[0141] It constitutes the contribution weight of the Q value of the point to the new Q value required; is the normalization coefficient, which is used to scale the sum of all weights to 1. The calculation formula of the normalization coefficient is as follows:

[0142] ; (8)

[0143] After the calculation is completed, the difference in the size of the reward value between the original task and the new task needs to be considered. If the reward value of the original task is large, the Q value in the Q table after training is also large. After migrating it to the new task, the smaller reward value of the new task is compared with the migrated Q value, which will cause deviations in the selection of the intelligent agent. Therefore, when migrating systems with different energy efficiency indicators, the Q value of the entire Q table needs to be scaled according to the size of the reward value.

[0144] After the migration is complete, set up the reinforcement learning agent in the project as shown in the following table:

[0145]

[0146] In the above table, , , , are the upper and lower limits of load and wet-bulb temperature respectively.

[0147] After the migration is completed, the hyperparameters of the reinforcement learning deployed in the project are set. The learning rate is set to 0.1, the discount factor is set to 0.1, the initial random coefficient is set to 0.4, and the random coefficient attenuation is set to 0.0001. The method described in the present invention is formally applied to the project.

[0148] Furthermore, setting the threshold and step size parameters of the action set of the agent that completes the migration specifically includes:

[0149] The control time interval shall not be less than 10 minutes per time;

[0150] The control step size, i.e. the adjustment frequency, shall not be higher than 2Hz / time;

[0151] The temperature adjustment interval shall not exceed 1℃ / time;

[0152] The safety thresholds of chiller outlet water temperature, water pump frequency, and cooling tower fan frequency refer to the design parameters of the entire air-conditioning system and the design parameters of the corresponding equipment.

[0153] In addition, performance degradation must be considered for equipment that has been used for more than 5 years.

[0154] In order to cope with the operating characteristics of the air-conditioning system and meet the energy-saving optimization control of complex systems under non-steady-state conditions, the hyperparameters of the intelligent agent are set. Reinforcement learning Q-learning requires that both the action space and the state space are discrete. Therefore, when setting the state space and action space of the intelligent agent, the originally continuous action space needs to be discretized. The above-mentioned input adjustment parameters, such as timestamp, external temperature and humidity, and key parameters of each device (such as temperature, frequency, etc.) are all continuous values. In the embodiment of the present invention, considering the stability of the air-conditioning system, the control time interval shall not be less than 10min / time, and the control step size shall not be higher than 2Hz / time, and the temperature adjustment interval shall not be higher than 1℃ / time.

[0155] Furthermore, before the invention design is formally applied to the building air conditioning system, under specific implementation conditions, the embodiment of the invention needs to debug the reward value setting of the intelligent agent according to the existing air conditioning operation conditions. The learning goal of Q-learning is to maximize the system COP (system energy efficiency ratio), so the system COP should be used as the reward subject. In order to make the reward smoother, the system COP should be standardized.

[0156] In addition, since the model calculation assumes that the heat exchange is sufficient, in order to prevent the intelligent agent from unilaterally reducing the cooling water flow rate, which will cause the cooling water supply and return water temperature difference to be too high, the final reward needs to be multiplied by the temperature difference correction coefficient. The specific formula is as follows:

[0157] ； (9)

[0158] In formula (9), COP is the system energy efficiency ratio, Tcws is the cooling water main supply water temperature, and Tcwr is the cooling water main return water temperature.

[0159] When the intelligent agent sends a control instruction to the air-conditioning system, it will synchronously obtain the operating parameters at the current timestamp to calculate the energy efficiency ratio of the current system.

[0160] Furthermore, the method also includes that after the intelligent agent sends a control instruction to the target air-conditioning system, it synchronously obtains the operating parameters of the current timestamp and calculates the current system energy efficiency ratio. The system energy efficiency ratio COP is used as a reward value and is regarded by the intelligent agent as a basis for determining whether to store it in the Q table.

[0161] The system energy efficiency ratio is used as a reward value and is considered by the Agent as a basis for determining whether to store it in the Q table. When the Q table exceeds a certain limit, the Agent will optimize the entire Q table based on the reward value. If a Q value with a high reward value appears at this time, and the data in the Q table has reached the limit, the Q value with the lowest reward value in the Q table will be removed from the Q table by the Agent, and then a higher Q value will be added for the Agent to learn.

[0162] After a large amount of practical data, the embodiment of the present invention also has requirements for the performance of the server as the carrier of the algorithm operation. The algorithm running on it must meet the following requirements: the equipment static point configuration capacity ≥ 4000, the data retention period ≥ 900 days, the indicator correlation analysis capability ≥ 100 items / second, the collection frequency of temperature sensing and air-conditioning system data ≥ 1 time / minute, and the collection frequency of electric meter data ≥ 1 time / minute.

[0163] In the existing system, the chiller reaches the highest COP at 60% partial load at rated water temperature. Calculate as follows:

[0164] ;

[0165] in, The number of cold machines turned on; is the total load borne by the system, kW; is the rated cooling load of the chiller; int represents the rounding down algorithm. Since the COP-plr curve of this chiller is steeper on the left side of the extreme point, as shown in the attached figure Figure 5 As shown, therefore, using rounding down instead of rounding up to calculate the number of chillers can make the operating points fall to the right of the extreme point, making the average COP larger. This formula is set according to the actual needs of the project, and other algorithms for calculating the starting number of chillers may be used in other projects.

[0166] At this time, as the system load increases, when the load has not yet increased to the point where a new chiller needs to be started, the optimal cooling tower fan frequency increases with the increase in load; when a new chiller is started, the optimal cooling tower fan frequency suddenly decreases, and then continues the previous pattern. The effect of wet-bulb temperature on the optimal cooling tower fan frequency is not obvious. The optimal cooling water pump frequency decreases as the system load increases, but when the wet-bulb temperature is high or low, there are areas where high and low frequencies alternate, such as the following. Figure 6 shown.

[0167] It can be seen that the optimal cooling tower fan frequency increases with the increase of the load on each chiller, but when the load on each chiller is high, high-frequency and low-frequency areas appear at the same time. This is because this area includes both the operating point where the system load continues to increase after the number of chillers reaches the upper limit and the operating point where the number of chillers suddenly changes. The optimal cooling pump frequency decreases with the increase of wet-bulb temperature and the load on each chiller, as shown in the attached figure. Figure 6 shown.

[0168] The cooling water pump and cooling tower are no longer in fixed frequency operation at different state points, but are in different working states. Here we take the frequency operation of the cooling water pump for demonstration, as shown in the attached figure. Figure 7 As shown, July 2 is the operating state of the cooling water pump when the present invention is not deployed, and it runs at a fixed frequency of 40Hz, wherein CwPump_#1 and CwPump_#2 are cooling water pump No. 1 and cooling water pump No. 2 respectively, Freq_FB is the frequency feedback value, 0702 is July 2, and July 3 is the operating state of the cooling water pump after the present invention is deployed. The Agent considers more from the perspective of overall energy consumption and adjusts the frequency of the cooling water pump to achieve the lowest overall energy consumption.

[0169] After testing, one month after the deployment of the present invention, the COP of the air conditioning system increased from 4.73 last year to 5.31 this year, and the difference in the average outdoor wet-bulb temperature between this year and last year was only 0.18°C, which is negligible. The energy efficiency improvement rate of the system reached 12.13%, and the present invention successfully helped the building achieve energy-saving and optimized operation of the air conditioning system.

[0170] The embodiment of the present invention further provides an intelligent agent for energy-saving control of a general air-conditioning system based on reinforcement transfer learning, comprising:

[0171] The data acquisition module is used to obtain the operation data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp;

[0172] A reinforcement learning module, used to input the operation data and the energy consumption data as input into the intelligent body for learning, and set basic parameters of the intelligent body, wherein the basic parameters include a state set, an action set, a reward function of the environment, and an ε-greed strategy setting value;

[0173] A transfer learning module is used to determine whether a Q table for project 2 to be transferred can be found from an existing training library according to the implementation parameters of the target air-conditioning system project 1 to be controlled. If so, the state space and the action space are normalized to establish a correlation between the Q tables of different state and action spaces, and the normalized Q table is processed to complete the transfer, wherein the implementation parameters include but are not limited to the air-conditioning system load, equipment type, equipment quantity and connection mode;

[0174] The instruction issuing module is used to limit the threshold and step parameters of the action set of the intelligent agent that has completed the migration, and issue control instructions to the target air-conditioning system.

[0175] An embodiment of the present invention further provides an intelligent device, including a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the aforementioned method.

[0176] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described above.

[0177] Compared with the existing air conditioning system optimization control method or the existing air conditioning system energy-saving optimization operation and maintenance method, the embodiment of the present invention has the following technical features:

[0178] The embodiment of the present invention eliminates the cumbersome equipment modeling process of existing energy-saving optimization means, directly learns the key operating parameters and energy consumption parameters of the existing air-conditioning system, and can achieve a higher energy-saving level without relying on a large number of sensor data and load forecasts; there is no need to perform a large amount of parameter adjustment and model optimization work like modeling, and there is no need to spend equipment costs such as high-performance GPUs or TPUs, thereby reducing the time cost of model design, data processing, and algorithm implementation.

[0179] The embodiments of the present invention have higher feasibility and flexibility than the existing energy-saving control scheme. In actual engineering, there are still a large number of characteristics about intuition, experience, cognition, etc. that cannot be expressed by formal methods, which means that it is difficult to truly express the deep laws of complex systems by relying solely on traditional modeling and simulation methods based on the principle of similarity and formalized knowledge. In the HVAC discipline, although its mechanism formula is scientific and experimentally verified based on the development of basic disciplines such as heat transfer, engineering thermodynamics, and fluid mechanics, in actual engineering, there are still a large number of characteristics about intuition, experience, cognition, etc. that cannot be expressed by formal methods, which means that it is difficult to truly express the deep laws of complex systems by relying solely on traditional modeling and simulation methods based on the principle of similarity and formalized knowledge.

[0180] The embodiment of the present invention uses reinforcement learning technology to conduct an in-depth analysis of the actual operating data to identify and quantify nonlinear dynamic characteristics and operating modes that are difficult to capture with traditional methods. At the same time, through the training of a large amount of project data in the early stage, the Agent will spontaneously set up a prefabrication based on the HVAC mechanism for the learning state, which is similar to a simple mechanism model built in. This model-free optimization strategy not only ensures the accuracy and interpretability of the control effect, but also enhances the adaptability and predictive ability of the model through a data-driven approach. At the same time, reinforcement learning has a high degree of adaptability and can optimize decision-making strategies according to environmental changes and feedback signals. The built-in intelligent agent of reinforcement learning can calculate the optimal system operating state through evaluation function calculation based on the current environmental parameters and system load, thereby performing real-time and optimal parameter adjustments.

[0181] In addition, the embodiments of the present invention have a higher energy-saving upper limit and operating health level than the existing PID optimization control method. Most of the PID-based optimization control methods that are widely used now rely on a pre-built rule base and fuzzy reasoning mechanism to calculate the fuzzy value of the corresponding output variable based on the fuzzy value of the input variable and convert it into a specific control command. This optimization does not improve the logical basis of PID control, but only builds in a large amount of expert experience. It divides the system load into different sectors. The control system will make fixed changes to the system's set value based on the load interval when the system is running, relying on expert experience, and will not calculate based on real-time load data to change the set value. This method requires manual writing of the rule base in the design stage. For complex and nonlinear control problems, fuzzy rules are difficult to exhaust, and the HVAC system parameter settings can only be adjusted based on pre-written expert experience.

[0182] The embodiment of the present invention can handle various complex system dynamics by training the system to run continuously, because it can learn the best strategy from a large amount of data running in real time. When encountering extreme weather or system equipment failure, the Agent can make changes according to the operating conditions to avoid serious damage to the equipment and not affect normal production activities if the system allows. At the same time, the performance fluctuations of the equipment will be reflected in the operating data and captured by the Agent, thereby minimizing the impact of equipment performance degradation on overall energy-saving optimization.

[0183] It should be noted that the description of reinforcement learning and transfer learning separately in this article is only for the reader's experience and to correctly understand the content of the present invention. It does not mean that in the present invention, the two algorithms run completely independently. In fact, the two algorithms will run separately, together, or modularly under an architecture according to the actual situation of the project. These situations are essentially still the methods described in the content of the present invention. In addition, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0184] It should be noted that the devices, units and methods disclosed in the embodiments of this article can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of this article. In this regard, each box in the flowchart or block diagram can represent a part of a module, program or code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system for performing a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0185] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0186] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0188] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0190] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0191] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0192] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0193] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A general air conditioning system energy-saving control method based on reinforcement transfer learning, characterized in that: The method comprises: The agent obtains the operating data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp; The operation data and the energy consumption data are used as inputs and input into the intelligent body for learning, and the basic parameters of the intelligent body are set. The basic parameters include a state set, an action set, an environment reward function, and an ε-greed strategy setting value. The basic parameters of the intelligent body are shown in the following formula: ； Wherein, S represents the state set of the agent, which includes the instantaneous cooling load of the system and the outdoor meteorological parameters; A represents the action set of the agent, which includes the operating state of the air-conditioning equipment and its current action setting parameters; r represents the reward function of the environment, which is the energy efficiency index of the air-conditioning system, and the system energy efficiency index includes but is not limited to one or more of COP, EER or PUE; p represents the state transition probability of the environment, and γ represents the discount factor; is the maximum Q value that can be obtained in the next state; According to the implementation parameters of the target air-conditioning system project 1 to be controlled, the agent determines whether the Q table of project 2 for migration can be found from the existing training library. If so, the state space and the action space are normalized to establish a correlation between the Q tables of different state and action spaces, and the normalized Q table is processed to complete the migration. The implementation parameters include but are not limited to the air-conditioning system load, equipment type, equipment quantity and connection method. If not, the agent is directly set according to the reinforcement learning requirements; The normalizing of the state space and the action space so as to establish associations between Q tables between different state and action spaces, and processing the normalized Q table specifically includes: The normalization formula is as follows: ； Wherein, x is the object data to be normalized, min and max are the maximum and minimum values ​​in the object data, respectively. As the reference value for normalization, divide the value of each state point s and action point a by the corresponding reference value and , and scaled to the interval [0,1], as shown below: , ； ， ； Among them, Q1 is the action of the air conditioning system at each state point in project 1, and Q2 is the action of the air conditioning system at each state point in project 2; After normalization, a Q value in the new Q table It is obtained by weighting the Q values ​​in the original Q table that are close to its index value. The specific formula is as follows: ； in, is the value of the normalized row index of the original Q table, that is, the state space is mapped to the row index; is the value of the normalized column index of the original Q table, that is, the action space is mapped to the column index; and is a parameter, and its value ∈(0,1) is used as the base to construct a function with the absolute value of the distance between the indices as the exponent. When the distance is 0, the function value is 1. When the distance increases, the function value decreases rapidly, and the speed of decrease is related to the value of the parameter; It constitutes the contribution weight of the Q value of the point to the new Q value required; is the normalization coefficient, which is used to scale the sum of all weights to 1. The calculation formula of the normalization coefficient is as follows: ； When migrating systems with different energy efficiency indicators, the Q value of the entire Q table is scaled according to the size of the reward value; The threshold value and step length parameters of the action set of the agent that has completed the migration are set, and the agent sends a control instruction to the target air-conditioning system, wherein the threshold value and step length parameters of the action set of the agent that has completed the migration are set specifically including: The control time interval shall not be less than 10 minutes per time; The control step size, i.e. the adjustment frequency, shall not be higher than 2Hz / time; The temperature adjustment interval shall not exceed 1℃ / time; The safety thresholds of chiller outlet water temperature, water pump frequency, and cooling tower fan frequency refer to the design parameters of the entire air-conditioning system and the design parameters of the corresponding equipment.

2. The energy-saving control method for a general air-conditioning system based on reinforcement transfer learning according to claim 1, characterized in that: The method further comprises: After the intelligent agent sends a control instruction to the target air-conditioning system, it synchronously obtains the operating parameters of the current timestamp and calculates the current system energy efficiency index. The system energy efficiency index is used as a reward value and is regarded by the intelligent agent as a basis for determining whether to store it in the Q table. The system energy efficiency index includes but is not limited to one or more of COP, EER or PUE.

3. The energy-saving control method for a general air-conditioning system based on reinforcement transfer learning according to claim 1, characterized in that: The method also includes: setting hyperparameters of reinforcement learning, including learning rate, discount coefficient, initial random coefficient epsilon, random coefficient attenuation, and debugging the reward value setting of the intelligent agent according to the actual needs of the target air-conditioning system project to be controlled.

4. The energy-saving control method for a general air-conditioning system based on reinforcement transfer learning according to claim 3, characterized in that: The debugging of the reward value setting of the intelligent agent according to the actual requirements of the target air conditioning system project to be controlled specifically includes: The system energy efficiency ratio COP is used as the reward value and the system energy efficiency ratio COP is standardized. The system energy efficiency ratio COP is obtained by the agent when issuing control instructions and is calculated based on the current operating parameters. Multiply the standardized system COP by the temperature difference correction factor. The specific formula is as follows: ； Among them, COP is the system energy efficiency ratio, Tcws is the cooling water main supply water temperature, and Tcwr is the cooling water main return water temperature.

5. An intelligent agent for energy-saving control of a general air-conditioning system based on reinforcement transfer learning, characterized in that: The intelligent agent comprises: The data acquisition module is used to obtain the operation data of the air-conditioning system and the energy consumption data of each air-conditioning device at the corresponding timestamp; The reinforcement learning module is used to input the operation data and the energy consumption data as input into the intelligent body for learning, and set the basic parameters of the intelligent body, wherein the basic parameters include a state set, an action set, an environment reward function, and an ε-greed strategy setting value, wherein the basic parameters of the intelligent body are shown in the following formula: ； Wherein, S represents the state set of the agent, which includes the instantaneous cooling load of the system and the outdoor meteorological parameters; A represents the action set of the agent, which includes the operating state of the air-conditioning equipment and its current action setting parameters; r represents the reward function of the environment, which is the energy efficiency index of the air-conditioning system, and the system energy efficiency index includes but is not limited to one or more of COP, EER or PUE; p represents the state transition probability of the environment, and γ represents the discount factor; is the maximum Q value that can be obtained in the next state; A transfer learning module is used to determine whether a Q table for project 2 to be transferred can be found from an existing training library according to the implementation parameters of the target air-conditioning system project 1 to be controlled. If so, the state space and the action space are normalized to establish a correlation between the Q tables of different state and action spaces, and the normalized Q table is processed to complete the transfer, wherein the implementation parameters include but are not limited to the air-conditioning system load, equipment type, equipment quantity and connection mode; Among them, the state space and the action space are normalized to establish a correlation between the Q tables between different state and action spaces, and the normalized Q table is processed specifically including: The normalization formula is as follows: ； Wherein, x is the object data to be normalized, min and max are the maximum and minimum values ​​in the object data, respectively. As the reference value for normalization, divide the value of each state point s and action point a by the corresponding reference value and , and scaled to the interval [0,1], as shown below: , ； ， ； Among them, Q1 is the action of the air conditioning system at each state point in project 1, and Q2 is the action of the air conditioning system at each state point in project 2; After normalization, a Q value in the new Q table It is obtained by weighting the Q values ​​in the original Q table that are close to its index value. The specific formula is as follows: ； in, is the value of the normalized row index of the original Q table, that is, the state space is mapped to the row index; is the value of the normalized column index of the original Q table, that is, the action space is mapped to the column index; and is a parameter, and its value ∈(0,1) is used as the base to construct a function with the absolute value of the distance between the indices as the exponent. When the distance is 0, the function value is 1. When the distance increases, the function value decreases rapidly, and the speed of decrease is related to the value of the parameter; It constitutes the contribution weight of the Q value of the point to the new Q value required; is the normalization coefficient, which is used to scale the sum of all weights to 1. The calculation formula of the normalization coefficient is as follows: ； When migrating systems with different energy efficiency indicators, the Q value of the entire Q table is scaled according to the size of the reward value; The instruction issuing module is used to limit the threshold and step parameters of the action set of the intelligent agent that has completed the migration, and issue a control instruction to the target air-conditioning system, wherein the setting of the threshold and step parameters of the action set of the intelligent agent that has completed the migration specifically includes: The control time interval shall not be less than 10 minutes per time; The control step size, i.e. the adjustment frequency, shall not be higher than 2Hz / time; The temperature adjustment interval shall not exceed 1℃ / time; The safety thresholds of chiller outlet water temperature, water pump frequency, and cooling tower fan frequency refer to the design parameters of the entire air-conditioning system and the design parameters of the corresponding equipment.

6. A smart device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 4.

Images