Air conditioner control method, system and equipment based on deep reinforcement learning and medium

Abstract

The invention discloses an air conditioner control method, system and device based on deep reinforcement learning and a medium, first environment state data is obtained and input into a pre-trained initial air conditioner strategy scheduling model, then a first operation strategy is obtained according to the first environment state data and a preset factor, and the first operation strategy is sent to the air conditioner control system. And then the first operation state of the air conditioner is adjusted according to the strategy, and the adjusted second environment state data are obtained again. And parameters of the initial air conditioner strategy scheduling model are optimized through a preset reward function, the first operation strategy and the second environment state data, and a target air conditioner strategy scheduling model is obtained. And the current environment state data are input into the target air conditioner strategy scheduling model, a current operation strategy can be generated, and the current operation state of the air conditioner is accurately adjusted according to the strategy. According to the embodiment, the environment state data can be obtained in real time, the operation strategy is dynamically adjusted, and therefore the response speed and flexibility of air conditioner control are effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of artificial intelligence technology, and in particular to air-conditioning control methods, systems, devices and media based on deep reinforcement learning. Background Art

[0002] In the prior art, the operation mode of air conditioning control system is relatively traditional and limited. Most traditional air conditioning control systems rely on fixed rules or preset models to adjust the temperature. This method may be slow to respond to complex and changing indoor and outdoor environments, and it is difficult to quickly adjust to the optimal state, resulting in reduced indoor comfort. In addition, many systems are often designed based on specific building models or assumptions, which may not achieve the expected control effect when applied to different buildings or environments. This limitation restricts the flexibility of air conditioning control systems. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiment of the present application provides an air-conditioning control method based on deep reinforcement learning, which can effectively improve the response speed and flexibility of air-conditioning control.

[0005] In a first aspect, an embodiment of the present application provides an air-conditioning control method based on deep reinforcement learning, including: obtaining first environmental state data, and inputting the first environmental state data into a pre-trained initial air-conditioning strategy scheduling model; obtaining a first operating strategy based on the first environmental state data and the preset factor; adjusting the first operating state of the air conditioner according to the first operating strategy to obtain a second operating state, and obtaining the adjusted second environmental state data; optimizing the parameters of the initial air-conditioning strategy scheduling model according to a preset reward function, the first operating strategy and the second environmental state data to obtain a target air-conditioning strategy scheduling model; obtaining current environmental state data, and inputting the current environmental state data into the target air-conditioning strategy scheduling model for strategy generation to obtain a current operating strategy; adjusting the current operating state of the air conditioner according to the current operating strategy.

[0006] In combination with the first aspect, in an embodiment of the present application, obtaining the first operating strategy based on the first environmental state data and the preset factor includes: converting the first environmental state data into an initial environmental state vector, normalizing and denoising the initial environmental state vector to obtain a target environmental state vector; generating an initial operating strategy probability distribution based on the target environmental state vector; adjusting the initial operating strategy probability distribution according to the preset factor to obtain a target operating strategy probability distribution, and in the target operating strategy probability distribution, selecting the operating strategy corresponding to the first probability as the first operating strategy according to the preset condition.

[0007] In combination with the first aspect, in an embodiment of the present application, the initial operating strategy probability distribution is adjusted according to a preset factor to obtain a target operating strategy probability distribution, including: obtaining an uncertainty index of the initial operating strategy probability distribution; constructing a preset index, and adjusting the preset factor according to the uncertainty index and the preset index to obtain an optimized preset factor; and adjusting the initial operating strategy probability distribution according to the optimized preset factor to obtain a target operating strategy probability distribution.

[0008] In combination with the first aspect, in an embodiment of the present application, the parameters of the initial air-conditioning strategy scheduling model are optimized according to the preset reward function, the first operating strategy and the second environmental state data to obtain the target air-conditioning strategy scheduling model, including: calculating the instant reward information according to the preset reward function and the second environmental state data; calculating the actual reward information according to the first operating strategy and the second environmental state data; constructing a first loss function, and calculating the first loss value between the instant reward information and the actual reward according to the first loss function; and updating the parameters of the air-conditioning strategy scheduling model according to the first loss value.

[0009] In combination with the first aspect, in an embodiment of the present application, the preset reward function includes a first indicator, a second indicator and a third indicator, and the types of the first indicator, the second indicator and the third indicator are different. The instant reward information is calculated based on the preset reward function and the second environmental state data, including: calculating a first performance value of the first indicator, a second performance value of the second indicator and a third performance value of the third indicator based on the second environmental state data; assigning a first weight to the first indicator, assigning a second weight to the second indicator, and assigning a third weight to the third indicator, and using the first weight, the second weight and the third weight, performing weighted calculation on the first performance value, the second performance value and the third performance value to obtain instant reward information.

[0010] In combination with the first aspect, in an embodiment of the present application, the first indicator is an energy consumption indicator, the second indicator is a comfort indicator, and the third indicator is an environmental indicator.

[0011] In combination with the first aspect, in an embodiment of the present application, the training steps of the pre-trained initial air-conditioning strategy scheduling model include: reading historical experience data, the historical experience data including third environmental state data, a third operating strategy, fourth environmental state data and historical instant reward information, the fourth environmental state data is obtained by adjusting the third environmental state data according to the third operating strategy, and the historical instant reward information is obtained according to the fourth environmental state data and the preset reward function; calculating the historical actual reward information according to the third operating strategy and the fourth environmental state data; constructing a second loss function, calculating the second loss value between the historical instant reward information and the historical actual reward according to the second loss function, and updating the parameters of the air-conditioning strategy scheduling model according to the second loss value.

[0012] In the second aspect, an embodiment of the present application provides an air-conditioning system based on deep reinforcement learning, which is applied to the air-conditioning control method as described above, including: a data processing module, used to obtain first environmental state data, and input the first environmental state data into a pre-trained initial air-conditioning policy scheduling model; a first strategy generation module, used to obtain a first operating strategy based on the first environmental state data and the preset factor; a first information processing module, used to adjust the first operating state of the air conditioner according to the first operating strategy to obtain a second operating state, and obtain the adjusted second environmental state data; a model generation module, used to optimize the parameters of the initial air-conditioning policy scheduling model according to a preset reward function, the first operating strategy and the second environmental state data to obtain a target air-conditioning policy scheduling model; a second strategy generation module, used to obtain current environmental state data, input the current environmental state data into the target air-conditioning policy scheduling model for strategy generation, and obtain the current operating strategy; a second information processing module, used to adjust the current operating state of the air conditioner according to the current operating strategy.

[0013] On the other hand, an embodiment of the present application further provides an electronic device, including:

[0014] at least one processor;

[0015] at least one memory for storing at least one program;

[0016] When at least one of the programs is executed by at least one of the processors, the air conditioning control method as described above is implemented.

[0017] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor, and the computer program executable by the processor is used to implement the air conditioning control method as described above when executed by the processor.

[0018] An air conditioning control method based on deep reinforcement learning provided in an embodiment of the present application first obtains first environmental state data and inputs it into a pre-trained initial air conditioning strategy scheduling model. Then, according to the first environmental state data and a preset factor, the model generates a first operation strategy. Then, the first operation state of the air conditioner is adjusted according to the strategy to obtain a second operation state, and the adjusted second environmental state data is obtained again. Through the preset reward function, the first operation strategy and the second environmental state data, the parameters of the initial air conditioning strategy scheduling model can be continuously iterated and optimized until the target air conditioning strategy scheduling model is obtained. Through deep learning and adaptation, the model can gradually master a more efficient and adaptable air conditioning control strategy, significantly improve the response speed and environmental adaptability, and ensure that the air conditioner can quickly adjust to the optimal state when facing a changing environment, thereby greatly improving the indoor comfort. The current environmental state data is input into the target air conditioning strategy scheduling model to generate the current operation strategy, and the current operation state of the air conditioner is accurately adjusted according to the strategy. The embodiment of the present application can obtain environmental state data in real time and dynamically adjust the operation strategy, so that it can flexibly respond to various indoor and outdoor environmental changes, and improve the wide applicability and flexibility of air conditioning control. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an air conditioning control method based on deep reinforcement learning provided in an embodiment of the present application;

[0020] Figure 2 The embodiment of this application provides Figure 1 Specific flow chart of step 120;

[0021] Figure 3 The embodiment of this application provides Figure 2 Specific flow chart of step 230;

[0022] Figure 4 The embodiment of this application provides Figure 1 A specific flow chart of step 140;

[0023] Figure 5 Schematic diagram of the training process of the initial air conditioning strategy scheduling model provided in the embodiment of the present application;

[0024] Figure 6 is a structural diagram of an air conditioning system based on deep reinforcement learning provided in an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] It should be noted that, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order from that in the flowchart. The terms "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can be implemented, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of narration, and are not used to limit the scope of the implementation of the present invention. The change or adjustment of the relative relationship should also be regarded as the scope of the implementation of the present invention without substantially changing the technical content.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] In the prior art, the operation mode of air conditioning control system is relatively traditional and limited. Traditional air conditioning control systems mostly rely on fixed rules or preset models to adjust the temperature. This method is powerless when facing complex and changeable indoor and outdoor environments, especially when dealing with non-standard or emergency situations, its efficiency is often greatly reduced. Although some systems try to introduce optimization algorithms to improve performance, most of these algorithms are determined at the beginning of the design and lack the ability to self-adjust and optimize over time and environmental changes. Therefore, when the external environment or indoor demand changes significantly, the response speed of these systems may be relatively slow, and it is difficult to quickly adjust to the optimal state, resulting in a decrease in indoor comfort.

[0030] In addition, existing systems also have shortcomings in energy management. They may fail to make full use of the collected environmental status data to optimize the control strategy, resulting in insufficient refinement and efficiency in energy use. Many systems are often designed based on specific building models or assumptions, which may not achieve the expected control effect when applied to different buildings or environments. This limitation restricts the wide applicability and flexibility of air conditioning control systems.

[0031] In addition, integrating different technical components into a unified air conditioning control system is also a challenge faced by existing technologies. Compatibility and collaboration issues between different components may lead to difficulties in system integration, which in turn affects the stability and performance of the entire system.

[0032] In view of this, an air conditioning control method, an air conditioning system, an electronic device and a computer-readable storage medium based on deep reinforcement learning are provided in an embodiment of the present application. The method first obtains the first environmental state data and inputs it into a pre-trained initial air conditioning strategy scheduling model. Then, according to the first environmental state data and the preset factor, the model generates a first operation strategy. Then, the first operation state of the air conditioner is adjusted according to the strategy to obtain the second operation state, and the adjusted second environmental state data is obtained again. Through the preset reward function, the first operation strategy and the second environmental state data, the parameters of the initial air conditioning strategy scheduling model can be continuously iterated and optimized until the target air conditioning strategy scheduling model is obtained. Through deep learning and adaptation, the model can gradually master a more efficient and adaptable air conditioning control strategy, significantly improve the response speed and environmental adaptability, and ensure that the air conditioner can quickly adjust to the optimal state when facing a changing environment, thereby greatly improving the indoor comfort. By inputting the current environmental state data into the target air conditioning strategy scheduling model, the current operation strategy can be generated, and the current operation state of the air conditioner can be accurately adjusted according to the strategy. The embodiments of the present application can obtain environmental status data in real time and dynamically adjust the operation strategy, so as to flexibly respond to various indoor and outdoor environmental changes and improve the wide applicability and flexibility of air-conditioning control.

[0033] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown, Figure 1 1 is a flow chart of an air conditioning control method based on deep reinforcement learning provided in an embodiment of the present application. The air conditioning control method includes but is not limited to steps 110 to 160.

[0035] Step 110: Acquire first environmental state data, and input the first environmental state data into a pre-trained initial air conditioning strategy scheduling model;

[0036] Step 120: Obtaining a first operation strategy according to the first environmental state data and a preset factor;

[0037] Step 130: adjusting the first operating state of the air conditioner according to the first operating strategy to obtain a second operating state, and obtaining adjusted second environmental state data;

[0038] Step 140: Optimizing the parameters of the initial air conditioning strategy scheduling model according to the preset reward function, the first operation strategy and the second environmental state data to obtain a target air conditioning strategy scheduling model;

[0039] Step 150: obtaining current environmental status data, inputting the current environmental status data into the target air conditioning strategy scheduling model to generate a strategy, and obtaining a current operation strategy;

[0040] Step 160: Adjust the current operating state of the air conditioner according to the current operating strategy.

[0041] Steps 110 to 160 are described in detail below.

[0042] In a feasible embodiment, the first environmental status data covers key parameters indoors and outdoors (for example, inside and outside a gymnasium), including but not limited to temperature, humidity, carbon dioxide concentration, light intensity, flow of people, and activity types (such as competition, training, or idle state). These data can be acquired in real time through various sensors and statistical equipment, which may include temperature sensors, humidity sensors, carbon dioxide concentration sensors, and professional equipment for counting flow of people. By flexibly selecting timed collection or triggering based on specific events, the latest environmental status data can be efficiently collected from the above sensors. This not only ensures the timeliness and accuracy of the data, but also provides a solid foundation for subsequent air conditioning strategy scheduling.

[0043] In a feasible embodiment, the initial air conditioning strategy scheduling model can be a model based on machine learning (such as neural network, decision tree, etc.) or deep learning (such as convolutional neural network CNN, recurrent neural network RNN, deep neural network DNN, etc.). The model has been pre-trained and can generate air conditioning operation strategies based on input environmental status data.

[0044] In a feasible embodiment, after the first environmental state data is input into a pre-trained initial air conditioning strategy scheduling model, the model can formulate a first operating strategy in combination with preset factors (such as temperature factors). Specifically, the first environmental state data is analyzed and interpreted according to a preset temperature factor (this factor may represent the user's preference for temperature, the temperature condition of the external environment, or the performance characteristics of the air conditioning equipment itself). By analyzing the relationship between the various environmental parameters and the temperature factor in the first environmental state data, a preliminary operating strategy (i.e., the first operating strategy) can be derived. This strategy may include multiple aspects such as the on / off state, working mode, wind speed, temperature setting, etc. of the air conditioning equipment, and aims to provide an air conditioning operation plan that is both comfortable and energy-saving according to the current environmental conditions and user needs.

[0045] See also Figure 2 , Figure 2 It is a specific flow chart of step 120 provided in an embodiment of the present application, and the flow may include but is not limited to steps 210 to 230.

[0046] Step 210: converting the first environment state data into an initial environment state vector, normalizing and denoising the initial environment state vector, and obtaining a target environment state vector;

[0047] Step 220: Generate an initial operation strategy probability distribution according to the target environment state vector;

[0048] Step 230: The initial operation strategy probability distribution is adjusted according to a preset factor to obtain a target operation strategy probability distribution. In the target operation strategy probability distribution, an operation strategy corresponding to the first probability is selected as the first operation strategy according to a preset condition.

[0049] It is worth noting that environmental data usually includes multiple types of information, such as temperature, humidity, air quality, etc. These data may come from different sensors or data sources with different units and dimensions. By integrating environmental data into a state vector, it helps to improve the performance and accuracy of the model and simplify the data management and analysis process. It should be noted that the state vector is a fixed-length vector containing all relevant information of environmental data. Each element corresponds to a specific environmental parameter. Before integrating the data, data cleaning is usually required, including removing outliers, filling missing values, etc.

[0050] In a feasible embodiment, due to the differences in units and dimensions of different environmental parameters, in order to ensure that the model can learn and understand these data more effectively, they can be subjected to a series of preprocessing, including denoising and normalization. Denoising aims to eliminate outliers and noise in the data to improve the accuracy and reliability of the data. Normalization converts environmental parameters of different dimensions and ranges to the same scale so that they have the same weight and influence in the model. Such preprocessing steps help the model better capture and utilize the features in the data, thereby generating more accurate air conditioning operation strategies.

[0051] In a feasible embodiment, in the process of generating the initial operation strategy probability distribution according to the target environment state vector, the model will analyze the target environment state vector, consider the impact of various environmental factors (such as temperature, humidity, air quality, etc.) on the current environment, and generate an initial operation strategy probability distribution based on these factors. This distribution represents the probability of each strategy being selected among different possible operation strategies. The size of the probability usually reflects the adaptability and effect of the strategy under the current environmental state. For example, in the air conditioning strategy scheduling model, if the target environment state vector shows that the current temperature is high and the humidity is moderate, then the model may generate a higher probability distribution for those operation strategies that can reduce the temperature, such as lowering the indoor temperature set point, increasing the air supply volume of the air conditioner, etc. At the same time, the impact of other factors (such as air quality, energy efficiency, etc.) on the strategy selection will also be considered, so as to obtain a comprehensive initial operation strategy probability distribution. Next, this probability distribution is adjusted according to the preset factor (such as the temperature factor) to obtain a target operation strategy that is more in line with actual needs. This adjustment process usually involves weighting, offsetting or renormalizing the probability distribution, so that the final selected strategy can meet the user's comfort while also achieving energy saving, environmental protection and other goals.

[0052] In a feasible embodiment, it is assumed that in a scenario, it is necessary to monitor and adjust the environmental conditions in the greenhouse, such as temperature, humidity, light intensity, etc., to ensure that the crops in the greenhouse can grow in the best environment. In this scenario, the temperature factor is an environmental factor that has a greater impact on crop growth. It is assumed that the environmental state data of the current greenhouse (i.e., the first environmental state data) includes: temperature is 25°C, humidity is 60%, and light intensity is 3000 lux. At the same time, the environmental parameter range suitable for crop growth is set: temperature is between 15-35°C, humidity is between 40%-80%, and light intensity is between 2000-4000 lux. After processing in step 210, a normalized and denoised target environmental state vector can be obtained, and its value is [0.5, 0.5, 0.5]. Further, it is assumed that there is an initial operation strategy probability distribution generated based on the initial air conditioning strategy scheduling model. For example, in this scenario, the model predicts that in the current environment, the probabilities of adopting strategies such as ventilation, humidification, and supplementary lighting are 0.3, 0.2, and 0.5, respectively. Given that the temperature factor has the most critical impact on crop growth, and the current temperature is already at 25°C in the optimal crop growth temperature range (24-26°C), there is no need to adjust the temperature significantly. Based on this, the probability of ventilation strategies that may cause temperature drops can be moderately reduced, while the probability of supplementary lighting strategies that may promote photosynthesis and crop growth can be increased. After the above adjustments, the probability distribution of the target operation strategy is adjusted to: ventilation 0.2 (reduced), humidification 0.2 (remains unchanged), supplementary lighting 0.6 (increased). Finally, in the current probability distribution of the target operation strategy, the probability of the supplementary lighting strategy is the highest, reaching 0.6. Therefore, the supplementary lighting strategy can be selected as the primary operation strategy in order to further optimize the growth environment of crops.

[0053] See also Figure 3 In a feasible embodiment, the process of adjusting the initial operating strategy probability distribution according to the preset factor in step 230 to obtain the target operating strategy probability distribution includes but is not limited to steps 310 to 330.

[0054] Step 310: Obtaining an uncertainty index of the probability distribution of the initial operation strategy;

[0055] Step 320: constructing a preset index, adjusting the preset factor according to the uncertainty index and the preset index, and obtaining an optimized preset factor;

[0056] Step 330: Adjust the initial operation strategy probability distribution according to the optimization preset factors to obtain the target operation strategy probability distribution.

[0057] In a feasible embodiment, the uncertainty index of the probability distribution of the initial operation strategy can be used to measure the degree of discreteness or uncertainty of the probability of each strategy in the distribution. Common uncertainty indicators include entropy and variance. Specifically, entropy is an indicator of the uncertainty of random variables in information theory. For probability distribution, the larger the entropy, the more uncertain the distribution and the closer the probabilities of each strategy; conversely, the smaller the entropy, the more certain the distribution, and the probability of one or some strategies is much higher than other strategies. Variance measures the degree of deviation of the probability of each strategy in the probability distribution from the average probability. The larger the variance, the more dispersed the distribution and the greater the probability difference between strategies; the smaller the variance, the more concentrated the distribution.

[0058] In a feasible embodiment, the preset index is set according to actual needs and can be used to guide the adjustment direction and target of the preset factors. These indicators may include user comfort, energy consumption, environmental impact, etc. For example, if the user pays more attention to comfort, a comfort index based on environmental parameters such as temperature and humidity can be set; if energy saving is desired, a preset index based on energy consumption can be set.

[0059] In a feasible embodiment, in the process of evaluating the rationality of the probability distribution of the current initial operation strategy according to the uncertainty index and the preset index, and adjusting the preset factor accordingly, if the uncertainty index is high (such as a large entropy), it may indicate that the distribution is too dispersed, and the preset factor needs to be adjusted to increase the probability of some strategies and reduce the probability of other strategies, thereby reducing uncertainty. At the same time, the specific value of the preset factor is optimized according to the preset index (such as comfort, energy consumption, etc.) so that the adjusted probability distribution is more in line with actual needs.

[0060] In a feasible embodiment, through steps 310 to 330, the initial operation strategy probability distribution can be adjusted according to the preset factors to obtain a target operation strategy probability distribution that better meets actual needs. This process is an iterative optimization process, and it may be necessary to adjust the preset factors and evaluate the uncertainty index and preset index multiple times until a satisfactory result is obtained.

[0061] In a feasible embodiment, the air conditioning device is configured and controlled according to the first operating strategy generated previously (for example, a strategy containing specific parameter settings, such as temperature setting value, wind speed, working mode, etc.). The air conditioning device starts to operate according to the configuration of the first operating strategy, thereby changing its current first operating state. This state may include parameters such as the operating temperature, humidity, and air supply volume. Through sensors or other monitoring devices, the adjusted operating state data of the air conditioning device are collected in real time or regularly, and these data constitute the second operating state of the air conditioner. The second operating state reflects the actual working condition of the air conditioner after the implementation of the first operating strategy. At the same time, it is also necessary to collect the state data of the environment where the air conditioner is located, which reflects the impact of the air conditioning operation on the environment. These data may include parameters such as temperature, humidity, and air quality, which may have changed after the implementation of the first operating strategy. The adjusted second environmental state data is an important basis for evaluating the effect of the first operating strategy, and they can be used for subsequent strategy optimization and adjustment.

[0062] In a feasible embodiment, by comparing and analyzing the environmental status data before and after adjustment with the preset reward function (such as improving comfort, reducing energy consumption and other indicators), the parameter configuration of the model can be continuously iterated and optimized until the most efficient control strategy is explored, thereby constructing a target air conditioning strategy scheduling model.

[0063] See also Figure 4 , Figure 4 It is a specific flow chart of step 140 , and the flow may include but is not limited to steps 410 to 440 .

[0064] Step 410: Calculate and obtain instant reward information according to a preset reward function and the second environment state data;

[0065] Step 420: Calculate actual reward information according to the first operation strategy and the second environment status data;

[0066] Step 430: construct a first loss function, and calculate a first loss value between the instant reward information and the actual reward according to the first loss function;

[0067] Step 440: Update the parameters of the air conditioning strategy scheduling model according to the first loss value.

[0068] In a feasible embodiment, the preset reward function refers to a predefined function used to evaluate the reward obtained by performing a certain action (such as the first operation strategy) under a given environmental state. The reward function may take into account multiple factors, such as indoor temperature comfort, energy consumption, humidity control, etc. The second environmental state data refers to the new environmental state data collected after the execution of the first operation strategy, such as temperature, humidity, outdoor temperature, etc. The immediate reward information refers to the immediate reward value calculated by inputting the second environmental state data into the preset reward function, which can be used to indicate the degree of improvement of the environmental state and the efficiency of energy consumption after the execution of the first operation strategy.

[0069] In a feasible embodiment, the preset reward function may include a first indicator, a second indicator, and a third indicator, and the types of the first indicator, the second indicator, and the third indicator are different. Then, the specific process of calculating the instant reward information according to the preset reward function and the second environment state data in step 310 includes:

[0070] Calculating a first performance value of the first indicator, a second performance value of the second indicator, and a third performance value of the third indicator according to the second environmental status data;

[0071] A first weight is assigned to the first indicator, a second weight is assigned to the second indicator, and a third weight is assigned to the third indicator. The first performance value, the second performance value, and the third performance value are weightedly calculated using the first weight, the second weight, and the third weight to obtain instant reward information.

[0072] In a feasible embodiment, the first indicator may be an energy consumption indicator (such as kilowatt-hour consumption), the second indicator may be a comfort indicator (such as temperature uniformity, wind speed suitability), and the third indicator may be an environmental indicator (such as indoor air quality). After obtaining the second environmental state data (including temperature, humidity, light intensity, PM2.5 concentration, energy consumption and other real-time data related to the environment), the performance value of the energy consumption indicator can be calculated based on the energy consumption in the second environmental state data. The performance value of the comfort indicator can be calculated based on environmental data such as temperature and humidity. The performance value of the environmental indicator can be calculated based on environmental data such as light intensity and PM2.5 concentration.

[0073] In a feasible embodiment, a weight value is assigned to the energy consumption indicator according to business needs. This weight reflects the importance of energy consumption in the overall reward calculation. A weight value is assigned to the comfort indicator, which also reflects the importance of comfort in the overall reward calculation. A weight value is assigned to the environmental indicator, which reflects the relative importance of environmental quality in the overall reward calculation. It is worth noting that the distribution of weights should be reasonable to ensure that the overall reward can reflect the combined impact of all indicators, and the sum of the weights should generally be 1 (or some other normalized value).

[0074] In a feasible embodiment, the first performance value of the energy consumption index, the second performance value of the comfort index and the third performance value of the environmental index are multiplied by their corresponding weights (the first weight, the second weight and the third weight), and then these weighted values ​​are added together. The sum is the instant reward information, which is a single value that integrates the three indicators of energy consumption, comfort and environmental quality, and is used to represent the instant reward obtained by executing a certain operating strategy under the current environmental conditions.

[0075] In a feasible embodiment, the actual reward information is a reward calculation based on actual business needs. For example, if the second environmental status data indicates that the indoor temperature is within a comfortable range and the energy consumption is below a certain threshold, a higher actual reward is given.

[0076] In a feasible embodiment, the first loss function refers to a function used to measure the difference between the immediate reward information and the actual reward. It can be a mean square error, cross entropy loss or other types of loss functions, depending on the type of reward information and the optimization goal of the model. The first loss value refers to the loss value calculated by inputting the immediate reward information and the actual reward information into the first loss function, which represents the error of the model in predicting the reward.

[0077] In a feasible embodiment, an optimization algorithm (such as gradient descent, stochastic gradient descent, Adam, etc.) can be used to update the parameters of the air conditioning strategy scheduling model according to the first loss value. By continuously iteratively optimizing the parameters of the model, the error of the model in predicting rewards is minimized, thereby obtaining a more accurate operation strategy.

[0078] In addition, see Figure 5 In a feasible embodiment, the training process of the pre-trained initial air conditioning strategy scheduling model in step 110 may include but is not limited to steps 510 to 530.

[0079] Step 510: reading historical experience data, the historical experience data including the third environment state data, the third operation strategy, the fourth environment state data and historical instant reward information, the fourth environment state data is obtained by adjusting the third environment state data according to the third operation strategy, and the historical instant reward information is obtained according to the fourth environment state data and a preset reward function;

[0080] Step 520: Calculate historical actual reward information according to the third operation strategy and the fourth environmental state data;

[0081] Step 530: construct a second loss function, calculate a second loss value between the historical instant reward information and the historical actual reward according to the second loss function, and update the parameters of the air conditioning strategy scheduling model according to the second loss value.

[0082] In a feasible embodiment, the third environmental state data records the indoor and outdoor environmental conditions at a certain point in the past, such as temperature, humidity, light intensity, etc. The third operating strategy refers to the air conditioning operating strategy historically adopted for the above environmental state data, including temperature setting, wind speed, working mode, etc. The fourth environmental state data refers to the change in environmental state after the execution of the third operating strategy, that is, the new environmental state actually achieved. The historical instant reward information is calculated based on the fourth environmental state data and the preset reward function, and is used to evaluate the quality of the current strategy, usually considering multiple indicators such as comfort and energy efficiency.

[0083] In a feasible embodiment, based on the third operation strategy and the fourth environmental state data, historical actual reward information can be further calculated. The purpose of this step is to quantify the actual effect of the strategy by comparing the changes in environmental state before and after the execution of the strategy and the actual impact of these changes on user comfort.

[0084] In a feasible embodiment, in order to continuously improve the accuracy and efficiency of the policy scheduling model, step 530 constructs a second loss function, which is intended to measure the deviation between the historical instant reward information and the historical actual reward (i.e., the second loss value). Through gradient descent or other optimization algorithms, the parameters of the model are gradually adjusted according to the calculated second loss value to minimize this deviation. This process is continuously iterated until the model can accurately predict and generate the optimal air conditioning operation strategy under a given environmental state.

[0085] Through the training process from step 510 to step 530, the initial air-conditioning strategy scheduling model is gradually optimized, improving its adaptability to different environmental conditions and the accuracy of strategy generation, laying a solid foundation for achieving efficient and intelligent air-conditioning control.

[0086] In a feasible embodiment, after obtaining the target air conditioning strategy scheduling model, the current environmental state data can be collected in real time, and the current environmental state data can be input into the target air conditioning strategy scheduling model for strategy generation to obtain the current operation strategy, and then the current operation state of the air conditioner can be adjusted according to the current operation strategy. By acquiring the environmental state data in real time and dynamically adjusting the operation strategy, various indoor and outdoor environmental changes can be flexibly responded to, thereby significantly improving the wide applicability and flexibility of the air conditioning system control. Taking the air conditioning system of the gymnasium as an example, the target air conditioning strategy scheduling model can be deployed on its control server to ensure that the model can run stably and efficiently on the server. Through a series of sensors and other equipment, the environmental state data in the gymnasium (including important information such as indoor temperature, humidity, outdoor weather conditions, and indoor personnel activities) can be captured in real time. Based on these data, the target air conditioning strategy scheduling model will calculate the optimal air conditioning operation strategy and apply these strategies to the air supply part of the air conditioning system. At the same time, the system will continuously monitor the environmental changes after the execution of the strategy to achieve dynamic adjustment and optimization, thereby forming a closed-loop control system. In this system, the model not only receives real-time environmental feedback, but also continuously adjusts and optimizes the strategy based on these feedbacks. In addition, the performance of the entire system can be monitored over a long period of time, and the target air-conditioning strategy scheduling model can be optimized according to actual needs to ensure that the operation of the air-conditioning system is always maintained in the best condition.

[0087] See also Figure 6 , Figure 6 600 is a structural diagram of an air conditioning system based on deep reinforcement learning provided in an embodiment of the present application, which can be applied to an air conditioning control method as in any of the previous embodiments. The air conditioning system 600 includes:

[0088] The data processing module 610 is used to obtain first environmental state data and input the first environmental state data into a pre-trained initial air conditioning strategy scheduling model;

[0089] A first strategy generating module 620, configured to obtain a first operation strategy according to the first environment state data and a preset factor;

[0090] The first information processing module 630 is used to adjust the first operating state of the air conditioner according to the first operating strategy to obtain a second operating state, and obtain the adjusted second environmental state data;

[0091] A model generation module 640 is used to optimize the parameters of the initial air conditioning strategy scheduling model according to a preset reward function, the first operation strategy and the second environmental state data to obtain a target air conditioning strategy scheduling model;

[0092] The second strategy generation module 650 is used to obtain current environmental status data, input the current environmental status data into the target air conditioning strategy scheduling model to generate a strategy, and obtain the current operation strategy;

[0093] The second information processing module 660 is used to adjust the current operating state of the air conditioner according to the current operating strategy.

[0094] It should be noted that since the air-conditioning system 600 of this embodiment can implement the air-conditioning control method of the previous embodiment, the air-conditioning system 600 of this embodiment and the air-conditioning control method of the previous embodiment have the same technical principles and the same beneficial effects. In order to avoid repetition, they will not be repeated here.

[0095] Reference Figure 7 The embodiment of the present application further discloses an electronic device, the electronic device 700 comprising:

[0096] at least one processor 701;

[0097] At least one memory 702, used to store at least one program;

[0098] When at least one program is executed by at least one processor 701 , the above air conditioning control method is implemented.

[0099] The embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program executable by a processor. When the computer program executable by the processor is executed by the processor, it is used to implement the air conditioning control method as described above.

[0100] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An air conditioning control method based on deep reinforcement learning, characterized in that: include: Acquire first environmental state data, and input the first environmental state data into a pre-trained initial air conditioning strategy scheduling model; Obtaining a first operation strategy according to the first environmental state data and the preset factor; Adjusting the first operating state of the air conditioner according to the first operating strategy to obtain a second operating state, and acquiring adjusted second environmental state data; Optimizing the parameters of the initial air conditioning strategy scheduling model according to a preset reward function, the first operation strategy, and the second environmental state data to obtain a target air conditioning strategy scheduling model; Acquire current environmental status data, input the current environmental status data into the target air conditioning strategy scheduling model to generate a strategy, and obtain a current operation strategy; The current operating state of the air conditioner is adjusted according to the current operating strategy.

2. The air conditioning control method based on deep reinforcement learning according to claim 1, characterized in that: The method of obtaining a first operating strategy based on the first environmental state data and the preset factors includes: converting the first environmental state data into an initial environmental state vector, normalizing and denoising the initial environmental state vector to obtain a target environmental state vector; generating an initial operating strategy probability distribution based on the target environmental state vector; adjusting the initial operating strategy probability distribution based on the preset factors to obtain a target operating strategy probability distribution, and in the target operating strategy probability distribution, selecting an operating strategy corresponding to a first probability as the first operating strategy based on preset conditions.

3. The air conditioning control method based on deep reinforcement learning according to claim 2, characterized in that: The adjusting the initial operating strategy probability distribution according to the preset factors to obtain the target operating strategy probability distribution includes: obtaining an uncertainty index of the initial operating strategy probability distribution; constructing a preset index, and adjusting the preset factors according to the uncertainty index and the preset index to obtain an optimized preset factor; and adjusting the initial operating strategy probability distribution according to the optimized preset factor to obtain a target operating strategy probability distribution.

4. The air conditioning control method based on deep reinforcement learning according to claim 1, characterized in that: The method of optimizing the parameters of the initial air-conditioning strategy scheduling model according to the preset reward function, the first operating strategy and the second environmental state data to obtain the target air-conditioning strategy scheduling model includes: calculating the instant reward information according to the preset reward function and the second environmental state data; calculating the actual reward information according to the first operating strategy and the second environmental state data; constructing a first loss function, and calculating the first loss value between the instant reward information and the actual reward according to the first loss function; and updating the parameters of the air-conditioning strategy scheduling model according to the first loss value.

5. The air conditioning control method based on deep reinforcement learning according to claim 4, characterized in that: The preset reward function includes a first indicator, a second indicator and a third indicator, and the types of the first indicator, the second indicator and the third indicator are different. The instant reward information is calculated according to the preset reward function and the second environmental state data, including: calculating a first performance value of the first indicator, a second performance value of the second indicator and a third performance value of the third indicator according to the second environmental state data; assigning a first weight to the first indicator, assigning a second weight to the second indicator, and assigning a third weight to the third indicator, and using the first weight, the second weight and the third weight to perform weighted calculation on the first performance value, the second performance value and the third performance value to obtain instant reward information.

6. The air conditioning control method based on deep reinforcement learning according to claim 5, characterized in that: The first indicator is an energy consumption indicator, the second indicator is a comfort indicator, and the third indicator is an environmental indicator.

7. The air conditioning control method based on deep reinforcement learning according to claim 1, characterized in that: The training steps of the pre-trained initial air-conditioning strategy scheduling model include: reading historical experience data, the historical experience data includes a third environment state data, a third operation strategy, a fourth environment state data and historical instant reward information, the fourth environment state data is obtained by adjusting the third environment state data according to the third operation strategy, and the historical instant reward information is obtained according to the fourth environment state data and the preset reward function; calculating the historical actual reward information according to the third operation strategy and the fourth environment state data; constructing a second loss function, calculating the second loss value between the historical instant reward information and the historical actual reward according to the second loss function, and updating the parameters of the air-conditioning strategy scheduling model according to the second loss value.

8. An air conditioning system based on deep reinforcement learning, characterized in that: The air conditioning control method according to any one of claims 1 to 7 comprises: A data processing module, used for acquiring first environmental state data, and inputting the first environmental state data into a pre-trained initial air conditioning strategy scheduling model; A first strategy generating module, used for obtaining a first operation strategy according to the first environmental state data and the preset factor; A first information processing module, configured to adjust the first operating state of the air conditioner according to the first operating strategy to obtain a second operating state, and obtain adjusted second environmental state data; A model generation module, used to optimize the parameters of the initial air conditioning strategy scheduling model according to a preset reward function, the first operation strategy and the second environmental state data to obtain a target air conditioning strategy scheduling model; A second strategy generation module is used to obtain current environmental status data, input the current environmental status data into the target air conditioning strategy scheduling model to generate a strategy, and obtain a current operation strategy; The second information processing module is used to adjust the current operating state of the air conditioner according to the current operating strategy.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the air conditioning control method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program executable by a processor is stored therein, and when the computer program executable by the processor is executed by the processor, it is used to implement the air conditioning control method as described in any one of claims 1 to 7.

Images