A method, device, equipment and storage medium for energy-saving control of a closed environment based on deep reinforcement learning

Abstract

The present invention proposes a method, device, equipment and storage medium for energy-saving control of a closed environment based on deep reinforcement learning. The method includes: building and pre-training a deep reinforcement learning model; updating the benchmark energy consumption regularly; obtaining and pre-processing the air-conditioning gear status data; inputting data into the model, and selecting the gear with the highest score as the best control; when the temperature of the closed environment exceeds the standard, starting the bottom-line control and obtaining the bottom-line status data; using the bottom-line data for model retraining; using the benchmark energy consumption and reward function to evaluate the prediction results, and using the loss function to iteratively optimize the model accordingly. When the benchmark energy consumption update time arrives, the benchmark energy consumption is obtained again. The present invention continuously learns and iteratively optimizes the deep reinforcement learning model by using the loss function, reward function and the benchmark energy consumption adjusted regularly, thereby realizing energy-saving control of the closed environment and optimizing the air-conditioning control strategy. This method not only improves energy utilization efficiency, but also reduces energy consumption.

Description

Technical Field

[0001] The present invention relates to the field of energy-saving control technology, and in particular to a closed environment energy-saving control method, device, equipment and storage medium based on deep reinforcement learning. Background Art

[0002] In the context of the ever-changing science and technology, social civilization is constantly progressing, and the public's expectations for the quality of life are also increasing. Especially in closed environments such as homes, office areas and storage spaces, air conditioning equipment has become a key facility to ensure environmental comfort. Traditional air conditioning control strategies mainly include constant temperature control, timer switches, and intelligent automatic adjustment. Among them, constant temperature control is achieved by setting a fixed temperature value. Its advantages are simple control and stable final temperature, but the energy consumption is relatively high, and there is a risk of deviation between the actual temperature and the set value, such as the actual temperature difference between 26°C in summer and the same gear in spring and autumn.

[0003] The timed switch strategy turns the air conditioner on or off during a preset time period to achieve energy saving. Compared with constant temperature control, this method has more advantages in energy saving and is relatively easy to operate. However, the timed switch strategy fails to fully consider the changes in actual temperature demand, which may cause the air conditioner to continue to run when it is not needed, or to be turned off when it is needed, resulting in unnecessary waste of energy. If the timing is set improperly, it may also cause the air conditioner to over-operate for a long time (such as when no one is using it) or under-operate (such as the air conditioner is not started before the user returns), which in turn affects energy efficiency and energy saving results.

[0004] The intelligent automatic adjustment mode dynamically adjusts the air conditioner operating gear according to the temperature sensor data in the closed space. It is the most technologically advanced and energy-saving method among the three strategies. It can flexibly adjust the air conditioner working state according to the actual situation, effectively avoiding unnecessary energy consumption. By dynamically adjusting the temperature and wind speed, a more comfortable temperature can be provided in different environments. However, during the temperature control process of this mode, when the actual temperature is higher or lower than the set value, it simply adjusts to a gear higher or lower than the preset temperature, failing to fully consider the impact of the current gear on subsequent control, that is, the optimal gear control strategy is not adopted, resulting in energy waste. Summary of the invention

[0005] The present invention aims to at least solve the technical problem of energy waste in the prior art, and particularly innovatively proposes an energy-saving control method for a closed environment based on deep reinforcement learning.

[0006] In order to achieve the above-mentioned object of the present invention, the present invention provides a closed environment energy-saving control method based on deep reinforcement learning, the method comprising:

[0007] S1. Building a deep reinforcement learning model and pre-training the deep reinforcement learning model;

[0008] S2. Obtaining a benchmark energy consumption, and regularly updating the benchmark energy consumption;

[0009] S3, obtaining state data of each gear control of the air conditioner, and preprocessing the state data;

[0010] S4, inputting the state data into a deep reinforcement learning model, obtaining the predicted scores of each gear of the air conditioner, and selecting the gear with the highest score as the optimal control gear;

[0011] S5. Collect temperature data of the closed environment. When the temperature data is greater than a threshold, start the bottom-line temperature control and obtain bottom-line status data during the bottom-line temperature control.

[0012] S6. Sending the bottom-line state data to the deep reinforcement learning model to retrain the deep reinforcement learning model;

[0013] S7. Perform reward evaluation on the prediction result of the deep reinforcement learning model using the benchmark energy consumption and the reward function, and iteratively optimize the deep reinforcement learning model using the loss function according to the reward evaluation result.

[0014] S8. When the benchmark energy consumption update time is reached, the benchmark energy consumption is re-acquired, and steps S3 to S7 are continued.

[0015] As an optional embodiment of the present invention, optionally, the expression of the reward function is:

[0016]

[0017]

[0018] in, represents the overall benefit value, Indicated in The reward value obtained during the first regulation is Indicates the total number of controls. represents the attenuation factor, , Indicates The number of times of regulation, Indicates Attenuation factor during regulation The value of Represents the reward value, Indicates the current air conditioning energy consumption, Indicates baseline energy consumption.

[0019] As an optional embodiment of the present invention, optionally, the loss function is expressed as:

[0020]

[0021] in, represents the loss value, represents the prediction score, Represents the reward value.

[0022] As an optional embodiment of the present invention, optionally, the status data includes current control gear data, current outdoor temperature data, current outdoor humidity data, energy consumption data of the last control to the current gear, current indoor temperature data, current indoor humidity data, current air-conditioning power data, current time point, indoor temperature data at the last control, indoor humidity data at the last control, and air-conditioning power data at the last control.

[0023] As an optional embodiment of the present invention, optionally, the expression of the deep reinforcement learning model is:

[0024]

[0025]

[0026]

[0027]

[0028] in, represents the prediction score, represents a deep reinforcement learning model, represents a piecewise linear function, represents the weight, represents the data input to the deep learning model, Indicates bias, Indicates the current control gear data. Indicates the current outdoor temperature data. Indicates the current outdoor humidity data. Indicates the current indoor temperature data. Indicates the current indoor humidity data. Indicates the current air conditioning power data. Indicates the indoor temperature data at the last adjustment. Indicates the indoor humidity data at the last adjustment. Indicates the air conditioning power data during the last adjustment. Indicates the current time point. Represents a collection of data, represents the predicted score for gear 1, Indicates gear position The prediction score of .

[0029] On the other hand, the present invention also provides a closed environment energy-saving control device based on deep reinforcement learning, the device comprising:

[0030] A collection module is used to collect the status data of each gear control of the air conditioner, and regularly collect the benchmark energy consumption and store the status data;

[0031] A preprocessing module, connected to the acquisition module, for preprocessing the status data;

[0032] A score calculation module, connected to the preprocessing module, is used to calculate the predicted scores of each gear of the air conditioner and select the best control gear with the highest predicted score;

[0033] The temperature monitoring module is connected to the score calculation module and is used to collect temperature data of the closed environment. When the temperature data is greater than a threshold, the bottom-line temperature control is started, and the bottom-line status data of the bottom-line temperature control is obtained. The bottom-line status data is sent to the score calculation module to optimize the score calculation module.

[0034] The beneficial effect of the present invention is that by using the loss function, reward function and timed adjusted benchmark energy consumption to continuously learn and iteratively optimize the deep reinforcement learning model, energy-saving control of the enclosed environment is achieved, and the air conditioning control strategy is optimized. This method not only improves energy utilization efficiency and reduces energy consumption, but also improves the user experience. In addition, in terms of bottom-line control, the present invention sets a temperature monitoring module. When the temperature of the enclosed environment exceeds the set threshold, the bottom-line control temperature is automatically started, and the bottom-line state data when the bottom-line control temperature is obtained is used to optimize the scoring calculation module. This bottom-line control mechanism can ensure the comfort of the environment under extreme conditions, and at the same time can improve the overall control effect through continuous learning and optimization.

[0035] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0037] Figure 1 This is a flow chart of Example 1 of the present invention.

[0038] Figure 2 It is a schematic diagram of the structure of the deep reinforcement learning model in Example 1 of the present invention.

[0039] Figure 3 It is a flow chart of the structure of the embodiment 1 of the present invention. DETAILED DESCRIPTION

[0040] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0041] Example 1

[0042] A closed environment energy-saving control method based on deep reinforcement learning, the method comprising:

[0043] S1. Building a deep reinforcement learning model and pre-training the deep reinforcement learning model;

[0044] like Figure 2 As shown, it should be noted that when constructing a deep reinforcement learning model, it is first necessary to set the architecture of the model according to the characteristics and dimensions of the data in the actual application scenario, as well as the control parameters required by the model. This process involves careful consideration and setting of the input data dimensions and the model output dimensions. In order to meet the functional requirements of the model and taking into account the limitations of the deployment environment, this model is specially designed to run on an industrial computer. Since the computing resources usually equipped with industrial computers are relatively limited, the design of the model must match it to ensure efficient operation. In order to balance the computational complexity and predictive performance of the model, this embodiment adopts a 5-layer neural network structure, which includes 3 hidden layers. In the setting of the hidden layer, each layer is configured with 256 neurons. Such a structure can provide sufficient nonlinear processing capabilities while keeping the overall model lightweight.

[0045] S2. Obtaining a benchmark energy consumption, and regularly updating the benchmark energy consumption;

[0046] like Figure 1 and 3As shown, it should be noted that the update of the benchmark energy consumption is a task that is executed according to a plan. In this embodiment, it is set to be updated every two hours, and the frequency is relatively high. This is because the fluctuation of ambient temperature may be quite significant during the day, and the benchmark energy consumption data at a specific moment cannot fully reflect the situation throughout the day. In order to solve this problem, this embodiment introduces a dynamic benchmark energy consumption strategy, that is, the benchmark power consumption data is updated every two hours. All energy saving rate calculations are based on this dynamically updated benchmark power consumption, so that the reward value can be calculated more accurately in the subsequent training process, so that the model can converge faster during the training process, improving the training effect and efficiency of the model.

[0047] S3, obtaining state data of each gear control of the air conditioner, and preprocessing the state data;

[0048] like Figure 1 and 3 It should be noted that the state data recorded by the air conditioner under different gear control states is actually a record of the specific state of each operation of the air conditioner. These state data include but are not limited to the current control gear of the air conditioner, the temperature and humidity of the outdoor environment, and the energy consumption from the last control to the current control process. In addition, it also includes the temperature and humidity of the current indoor environment, the current working power of the air conditioner, and the specific time point. In addition, there is also information such as the indoor temperature and humidity at the last control, the power at the time of control, etc. These data can reflect the changes and connections between the two control operations in detail, and can also evaluate the effect of the last control to determine whether it has a good effect. Specifically, by comparing the energy consumption from the last control to the current control with the benchmark energy consumption, the energy saving amount and energy saving rate can be calculated, which can not only be used as a criterion for judging the energy saving effect, but also provide reference and guidance for the subsequent use and control of the air conditioner. Specifically, the state data is preprocessed as follows: first, this embodiment cleans the collected original state data to remove abnormal values ​​or missing values ​​caused by equipment failure, data transmission errors, etc. Then, the cleaned data is standardized to eliminate the dimensional differences between different features, so that the model can better learn the intrinsic relationship between the data. In addition, this embodiment also sets a reasonable range for each type of collected data. When the collected data exceeds the preset reasonable range, the collected data will be discarded. After the data is qualified, each collected data is divided by the preset maximum value to achieve data normalization, so that the data is distributed in the range of 0 to 1, making the model more robust and easier to converge.

[0049] S4, inputting the state data into a deep reinforcement learning model, obtaining the predicted scores of each gear of the air conditioner, and selecting the gear with the highest score as the optimal control gear;

[0050] like Figure 2 As shown, it should be noted that in step S4, the deep reinforcement learning model receives the preprocessed state data as input and performs complex calculations and analysis through its internal neural network structure. Each neuron in the model performs a nonlinear transformation on the input data, and then outputs the predicted score for each air-conditioning gear through layer-by-layer superposition and weight adjustment. These scores represent the expected energy-saving effect and comfort of each gear control under the current environmental conditions. By comparing the scores of each gear, the model can automatically select the gear with the highest score as the best control gear, thereby realizing intelligent energy-saving control.

[0051] S5. Collect temperature data of the closed environment. When the temperature data is greater than a threshold, start the bottom-line temperature control and obtain bottom-line status data during the bottom-line temperature control.

[0052] like Figure 3 As shown, it should be noted that the temperature data of the air-conditioning environment is collected in real time through the temperature monitoring thread. If the real-time temperature data is greater than the preset threshold, this usually means that the temperature of the closed environment is too high, which may have an adverse effect on the user's comfort. At this time, the system will start the bottom-line temperature control strategy, which is an emergency measure designed to quickly reduce the indoor temperature to a more comfortable level. When the bottom-line temperature control is started, the system will record the bottom-line status data when the bottom-line temperature is controlled. These data include the indoor temperature, humidity, air conditioning working power, etc. at the beginning of the control, as well as the energy consumption during the control process. These data are of great significance for the subsequent analysis of the effect of the bottom-line control and the optimization of the control strategy.

[0053] S6. Sending the bottom-line state data to the deep reinforcement learning model to retrain the deep reinforcement learning model;

[0054] It should be noted that in step S6, the bottom-line state data is sent to the deep reinforcement learning model for retraining. This step is a key link in the continuous optimization and learning of the model. Since the bottom-line control is an unconventional operation that usually only occurs in extreme cases, the bottom-line state data contains special cases that are difficult for the model to encounter during conventional training. Adding these data to the training set allows the model to learn the control strategies in various situations more comprehensively, thereby improving the generalization and robustness of the model. During the retraining process, the model will re-evaluate the predicted scores of each gear based on the bottom-line state data and update its internal neural network weights. Through continuous iteration and optimization, the model can gradually adapt to more complex and changeable environmental conditions, thereby providing more accurate and effective control strategies. At the same time, since the bottom-line control is a high-energy consumption operation, the model will also pay more attention to the consideration of energy-saving effects during the retraining process to achieve more green and environmentally friendly air-conditioning control.

[0055] S7. Perform reward evaluation on the prediction result of the deep reinforcement learning model using the benchmark energy consumption and the reward function, and iteratively optimize the deep reinforcement learning model using the loss function according to the reward evaluation result.

[0056] It should be noted that in step S7, this embodiment uses the benchmark energy consumption and reward function to perform a detailed reward evaluation on the prediction results of the deep reinforcement learning model. This evaluation mechanism is the key to the model's ability to continuously optimize and improve itself. Specifically, this embodiment compares the model's prediction results with the actual control effect, calculates the actual energy saving rate, and compares it with the benchmark energy consumption to evaluate the model's performance in energy saving. According to the reward evaluation results, this embodiment uses the loss function to iteratively optimize the deep reinforcement learning model. The loss function is a key indicator in the model training process, which measures the gap between the model's prediction results and the actual results. By continuously adjusting the parameters and weights of the model, the loss function is gradually reduced, thereby improving the model's prediction accuracy and generalization ability. Through the iterative optimization of step S7, the deep reinforcement learning model can gradually adapt to various complex and changeable environmental conditions and provide a more accurate and effective air conditioning control strategy. This not only helps to improve the energy-saving effect of air conditioning, but also improves the user's comfort and satisfaction. At the same time, since the model has the ability to self-learn and optimize, it can also continue to improve and improve with the changes in time and environment.

[0057] S8. When the benchmark energy consumption update time is reached, the benchmark energy consumption is re-acquired, and steps S3 to S7 are continued.

[0058] It should be noted that after the baseline energy consumption is updated, the system will execute the complete process from step S3 to step S7 again. First, the system will collect the latest status data and perform preprocessing, including data cleaning, standardization, reasonable range screening and normalization, to ensure the quality and validity of the data. Then, the preprocessed status data is input into the deep reinforcement learning model, and the model will predict the scores of each air-conditioning gear based on these data, and select the gear with the highest score as the best control gear. During the control process, the system will continuously monitor the temperature data of the closed environment. If the temperature exceeds the preset threshold, the system will start the bottom-up control temperature strategy and record the bottom-up status data. These bottom-up status data will be sent back to the deep reinforcement learning model for retraining to improve the model's control ability under extreme conditions. After the retraining is completed, the system will use the new baseline energy consumption and reward function to reward the model's prediction results, and iteratively optimize the model using the loss function based on the evaluation results. Through this process, the model will be able to continuously learn and optimize to adapt to various complex and changing environmental conditions and provide more accurate and effective air-conditioning control strategies. Finally, when the benchmark energy consumption update time is reached again, the system will repeat the above process to ensure that the model is always based on the latest energy consumption data for regulation, prediction and optimization. This continuous update mechanism enables the entire system to continuously adapt to environmental changes and maintain efficient energy saving and comfort.

[0059] As an optional embodiment of the present invention, optionally, the expression of the reward function is:

[0060]

[0061]

[0062] in, represents the overall benefit value;

[0063] Indicated in The reward value obtained during the first regulation;

[0064] Indicates the total number of controls;

[0065] represents the attenuation factor, ;

[0066] Indicates Number of times of regulation;

[0067] Indicates Attenuation factor during regulation The value of

[0068] Represents the reward value;

[0069] Indicates the current air conditioning energy consumption;

[0070] Indicates baseline energy consumption.

[0071] It should be noted that there are many types of reward functions. Generally speaking, the setting of rewards can be considered from two dimensions: temperature and power consumption. However, in actual design, since the weights of these two parameters are difficult to accurately set based on human judgment alone, this embodiment proposes an innovative solution: a single, energy-based reward function. This function focuses on the consideration of energy consumption information, and over-temperature management is entrusted to a dedicated monitoring thread. Once an over-temperature situation is detected, the bottom-up mechanism will be triggered immediately to respond. Using this reward function strategy, this embodiment can learn and implement the optimal energy-saving control strategy. Unlike traditional strategies that tend to avoid over-temperature as much as possible, this reward function strategy can learn strategies that can achieve higher energy-saving effects even in over-temperature situations. For example, in some cases, the energy consumption of choosing to shut down after 12 minutes and supplemented by a 3-minute 26-degree high-wind cooling bottom-up solution may be lower than the air supply mode that runs continuously for 15 minutes. In addition, in order to ensure the stability and reliability of the reward function, this embodiment also introduces a truncation mechanism to ensure that the reward value is always kept within a reasonable range, thereby effectively reducing the impact of abnormal values ​​on system performance.

[0072] As an optional embodiment of the present invention, optionally, the loss function is expressed as:

[0073]

[0074] in, represents the loss value;

[0075] represents the prediction score;

[0076] Represents the reward value.

[0077] It should be noted that by using the loss function to measure the difference between the predicted results and the actual results, and by continuously iterating and updating the parameters, the model and the environment are continuously interacting and learning. After a period of learning, the model will be able to reach an optimal decision-making state, so as to better cope with various complex environmental challenges. This method can help the model continuously improve its performance in practical applications, achieve more accurate predictions and more effective decisions.

[0078] As an optional embodiment of the present invention, optionally, the status data includes current control gear data, current outdoor temperature data, current outdoor humidity data, energy consumption data of the last control to the current gear, current indoor temperature data, current indoor humidity data, current air-conditioning power data, current time point, indoor temperature data at the last control, indoor humidity data at the last control, and air-conditioning power data at the last control.

[0079] It should be noted that the comprehensiveness and accuracy of the state data are crucial to the prediction and control effects of the deep reinforcement learning model. In this embodiment, the state data covers information in multiple dimensions, including but not limited to the current control gear data, indoor and outdoor temperature and humidity data, energy consumption data, air conditioning power data, and time data. These data provide the model with rich environmental perception capabilities, enabling the model to more accurately predict the potential results under various control strategies and make the best decisions accordingly. In practical applications, these state data can be acquired in real time through various sensors and measuring devices, and input into the deep reinforcement learning model for processing and analysis. At the same time, in order to ensure the real-time and accuracy of the data, the system also needs to calibrate and maintain the sensors regularly to ensure that they can continue to provide high-quality data support.

[0080] As an optional embodiment of the present invention, optionally, the expression of the deep reinforcement learning model is:

[0081]

[0082]

[0083]

[0084]

[0085] in, represents the prediction score;

[0086] represents a deep reinforcement learning model;

[0087] represents a linear piecewise function;

[0088] represents weight;

[0089] Data representing the input of a deep learning model;

[0090] Indicates bias;

[0091] Indicates the current control gear data;

[0092] Indicates the current outdoor temperature data;

[0093] Indicates the current outdoor humidity data;

[0094] Indicates the current indoor temperature data;

[0095] Indicates the current indoor humidity data;

[0096] Indicates the current air conditioning power data;

[0097] Indicates the indoor temperature data at the last adjustment;

[0098] Indicates the indoor humidity data at the last adjustment;

[0099] Indicates the air conditioning power data at the last adjustment;

[0100] Indicates the current time point;

[0101] Represents a collection of data;

[0102] represents the predicted score of gear 1;

[0103] Indicates gear position The prediction score of .

[0104] As an optional embodiment of the present invention, optionally, the method further includes packaging and storing the state data.

[0105] It should be noted that during the operation of this embodiment, the system will record the state data of the control process in detail, organize and encapsulate it as valuable historical experience, and then store it in the corresponding queue, aiming to provide solid data support for the system's decision-making. Each control operation is accompanied by a transition from the old environmental state to the new environmental state. This embodiment will carefully encapsulate each change in the air conditioning control state. The encapsulated object contains detailed information about the control gear, outdoor temperature, outdoor humidity, energy consumption within the time range of the new and old state conversion, and specific information about the new and old states. Among these information, both the new and old states contain key data such as indoor temperature, indoor humidity, current air conditioning power, and current time point. Through the analysis of these state conversion data, a rich and accurate data source is provided for the intelligent body's decision-making.

[0106] Example 2

[0107] A closed environment energy-saving control device based on deep reinforcement learning, the device comprising:

[0108] A collection module is used to collect the status data of each gear control of the air conditioner, and regularly collect the benchmark energy consumption and store the status data;

[0109] The acquisition module includes a variety of sensors for collecting the current control gear data, current outdoor temperature data, current outdoor humidity data, energy consumption data of the last adjustment to the current gear, current indoor temperature data, current indoor humidity data, current air-conditioning power data, current time point, indoor temperature data at the last adjustment, indoor humidity data at the last adjustment and air-conditioning power data at the last adjustment, and sends the collected data to the preprocessing module for preprocessing.

[0110] A preprocessing module, connected to the acquisition module, for preprocessing the status data;

[0111] A score calculation module, connected to the preprocessing module, is used to calculate the predicted scores of each gear of the air conditioner and select the best control gear with the highest predicted score;

[0112] The scoring calculation module is the deep reinforcement learning model in Example 1.

[0113] The temperature monitoring module is connected to the score calculation module and is used to collect temperature data of the closed environment. When the temperature data is greater than a threshold, the bottom-line temperature control is started, and the bottom-line status data of the bottom-line temperature control is obtained. The bottom-line status data is sent to the score calculation module to optimize the score calculation module.

[0114] It should be noted that the device provides a data preprocessing module. The module provides functions such as data quality detection, data cleaning, data encapsulation and data normalization, providing the most basic data guarantee for the method, avoiding dirty data and incomplete data, and realizing the data normalization function.

[0115] An experience playback queue module is also provided in the acquisition module. During the operation of this device, the state data of the control process will be recorded, encapsulated into historical experience, and stored in the corresponding queue to provide data support for the decision-making system. Each control will cause the old environmental state to be converted to a new environmental state. Each change in the air-conditioning control state is encapsulated, and the encapsulated object includes the control gear, outdoor temperature, outdoor humidity, energy consumption within the time range of the new and old state conversion, and specific information of the new and old states. The new and old states include data such as indoor temperature, indoor humidity, current air-conditioning power, and current time point. These turntable conversion data provide a data source for the intelligent body's decision-making.

[0116] In addition, the device also provides a lightweight, continuously learning decision-making scoring calculation module based on deep reinforcement learning. The device provides functions such as pre-training, training, self-learning based on environmental interaction, decision-making, and abnormal monitoring of the decision-making scoring calculation module. When the device is started, the decision-making scoring calculation module will be automatically pre-trained to provide excellent control effects. During each adjustment, historical experience data will be learned so that the scoring calculation module can continuously learn better control strategies and make decisions on the next adjustment gear. The abnormal monitoring system provides an automatic correction function. When the scoring calculation module adopts the wrong gear and causes overheating, the system can correct the current wrong decision.

[0117] In summary, this device uses a data preprocessing module, an experience playback system, a lightweight, continuously learning decision scoring calculation module based on deep reinforcement learning, covering the entire process of data processing, data normalization, historical experience storage, model training, and decision-making of the scoring calculation module, meeting the real-time, personalization, continuous learning, and automatic correction requirements of air-conditioning control, while ensuring that the air-conditioning will not overheat while saving energy.

[0118] Example 3

[0119] A computer device comprising:

[0120] processor;

[0121] a memory for storing processor-executable instructions;

[0122] Among them, the processor is configured to implement a closed environment energy-saving control method based on deep reinforcement learning in Example 1 when executing the executable instructions.

[0123] It should be noted that the computer device includes: a processor, a memory, and the computer device may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0124] Among them, the processor is used to control the overall operation of the computer device to complete all or part of the steps in the above-mentioned big data-based factory equipment automated testing method.

[0125] The memory is used to store various types of data to support operations on the computer device, which data may include, for example, instructions for any application or method operating on the computer device, as well as application-related data; the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0126] The multimedia component may include a screen and an audio component, wherein the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals; for example, the audio component may include a microphone for receiving external audio signals, and the received audio signals may be further stored in a memory or sent via a communication component; the audio component also includes at least one speaker for outputting audio signals.

[0127] The I / O interface provides an interface between the processor and other interface modules, which may be a keyboard, a mouse, buttons, etc.; these buttons may be virtual buttons or physical buttons.

[0128] The communication component is used for wired or wireless communication between the computer device and other devices; wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, or a combination of one or more of them, so the corresponding communication component may include: Wi-Fi module, Bluetooth module, NFC module, mobile phone communication module.

[0129] As a preferred solution of this embodiment, the computer device can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned big data-based factory equipment automation testing method.

[0130] Example 4

[0131] A computer-readable storage medium, comprising:

[0132] a memory having a computer program stored thereon;

[0133] A processor is used to execute the program in the memory to implement a closed environment energy-saving control method based on deep reinforcement learning in Example 1.

[0134] It should be noted that the electronic device according to the embodiment of the present disclosure includes a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement any of the above-mentioned cloud resource task scheduling methods with efficient optimization when executing the executable instructions.

[0135] Here, it should be noted that the number of processors can be one or more. At the same time, the electronic device in the embodiment of the present disclosure may also include an input device and an output device. Among them, the processor, memory, input device and output device may be connected through a bus or in other ways, which are not specifically limited here.

[0136] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and various modules, such as: the program or module corresponding to the cloud resource task scheduling method with efficient optimization in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0137] The input device can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0138] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A closed environment energy-saving control method based on deep reinforcement learning, characterized in that: The method comprises: S1. Building a deep reinforcement learning model and pre-training the deep reinforcement learning model; S2. Obtaining a benchmark energy consumption, and regularly updating the benchmark energy consumption; S3, obtaining state data of each gear control of the air conditioner, and preprocessing the state data; S4, inputting the state data into a deep reinforcement learning model, obtaining the predicted scores of each gear of the air conditioner, and selecting the gear with the highest score as the optimal control gear; S5. Collect temperature data of the closed environment. When the temperature data is greater than a threshold, start the bottom-line temperature control and obtain bottom-line status data during the bottom-line temperature control. The bottom-line temperature control strategy is an emergency measure designed to quickly reduce the indoor temperature to a comfortable level. S6. Sending the bottom-line state data to the deep reinforcement learning model to retrain the deep reinforcement learning model; S7, using the benchmark energy consumption and the reward function to perform a reward evaluation on the prediction result of the deep reinforcement learning model, and using the loss function to iteratively optimize the deep reinforcement learning model according to the reward evaluation result; S8, when the benchmark energy consumption update time is reached, reacquire the benchmark energy consumption, and continue to execute steps S3 to S7; The expression of the reward function is: in, represents the overall benefit value, Indicated in The reward value obtained during the first regulation is Indicates the total number of controls. represents the attenuation factor, , Indicates The number of times of regulation, Indicates Attenuation factor during regulation The value of Represents the reward value, Indicates the current air conditioning energy consumption, represents the baseline energy consumption; The expression of the loss function is: in, represents the loss value, represents the prediction score, Represents the reward value; The status data includes current control gear data, current outdoor temperature data, current outdoor humidity data, energy consumption data of the last control to the current gear, current indoor temperature data, current indoor humidity data, current air conditioning power data, current time point, indoor temperature data at the last control, indoor humidity data at the last control, and air conditioning power data at the last control; The expression of the deep reinforcement learning model is: in, represents the prediction score, represents a deep reinforcement learning model, represents a piecewise linear function, represents the weight, represents the data input to the deep learning model, represents the bias, Indicates the current control gear data. Indicates the current outdoor temperature data. Indicates the current outdoor humidity data. Indicates the current indoor temperature data. Indicates the current indoor humidity data. Indicates the current air conditioning power data. Indicates the indoor temperature data at the last adjustment. Indicates the indoor humidity data at the last adjustment. Indicates the air conditioning power data during the last adjustment. Indicates the current time point. Represents a collection of data, represents the predicted score for gear 1, Indicates gear position The prediction score of .

2. A closed environment energy-saving control method based on deep reinforcement learning as claimed in claim 1, characterized in that: Preprocessing the state data in step S3 includes: S301, setting a threshold, and filtering out status data outside the threshold; S302: Normalize the state data.

3. A closed environment energy-saving control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The method further includes packaging and storing the state data.

4. A closed environment energy-saving control device based on deep reinforcement learning, characterized in that: The device comprises a closed environment energy-saving control method based on deep reinforcement learning according to any one of claims 1 to 3; The device also includes: A collection module is used to collect the status data of each gear control of the air conditioner, and regularly collect the benchmark energy consumption and store the status data; A preprocessing module, connected to the acquisition module, for preprocessing the status data; A score calculation module, connected to the preprocessing module, is used to calculate the predicted scores of each gear of the air conditioner and select the best control gear with the highest predicted score; The temperature monitoring module is connected to the score calculation module and is used to collect temperature data of the closed environment. When the temperature data is greater than a threshold, the bottom-line temperature control is started, and the bottom-line status data of the bottom-line temperature control is obtained. The bottom-line status data is sent to the score calculation module to optimize the score calculation module.

5. A computer device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement a closed environment energy-saving control method based on deep reinforcement learning as described in any one of claims 1 to 3 when executing the executable instructions.

6. A computer-readable storage medium, characterized in that: include: a memory having a computer program stored thereon; A processor, used to execute the program in the memory to implement a closed environment energy-saving control method based on deep reinforcement learning as described in any one of claims 1 to 3.

Images