Temperature control method and device based on offline reinforcement learning and related equipment

Abstract

The invention provides a temperature control method and device based on off-line reinforcement learning and related equipment, and relates to the technical field of industrial internet, the method comprises the steps that historical operation data of a temperature adjusting system is acquired, a historical data set is formed, and the temperature adjusting system comprises refrigeration equipment and a fan; the historical operation data comprises environment data, target control data and fan control data; training the reinforcement learning model offline based on the historical data set; target control data and environment data of the temperature adjusting system at the previous moment and the current moment are obtained in real time, and the trained reinforcement learning model is used for outputting a fan control scheme for temperature control. According to the invention, high-precision and low-delay temperature regulation control can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of industrial Internet technology, and in particular to a temperature control method, device and related equipment based on offline reinforcement learning. Background Art

[0002] In modern factory environments, temperature control systems are an important component in ensuring production efficiency, product quality, and stable equipment operation. Figure 1 As shown in the figure, due to the complex industrial field environment and variable loads, traditional temperature control methods (such as proportional-integral-derivative (PID) control) have the following major problems:

[0003] Traditional temperature control methods are highly parameter-dependent and difficult to adapt to dynamic environments: control methods such as PID require manual parameter adjustment and cannot automatically adapt to the nonlinearity, time-varying nature and disturbances of complex systems.

[0004] Industrial temperature control systems suffer from serious problems such as delayed response, over-regulation, or oscillation: especially in systems with large thermal inertia or drastic changes in hot and cold loads, such as electric arc furnaces and ice storage systems, over-regulation or system instability are prone to occur.

[0005] Lack of prediction and optimization capabilities: PID is based on current error control, does not predict future trends, and cannot consider optimization goals such as system energy efficiency and global performance.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0007] The present disclosure provides a temperature control method, apparatus, and related equipment based on offline reinforcement learning, which, at least to a certain extent, overcome the problem that temperature control methods in related technologies are difficult to adapt to dynamic environments.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0009] According to one aspect of the present disclosure, a temperature control method based on offline reinforcement learning is provided, comprising: acquiring historical operating data of a temperature control system to form a historical data set, wherein the temperature control system includes refrigeration equipment and a fan; the historical operating data includes environmental data, target control data, and fan control data; offline training a reinforcement learning model based on the historical data set; acquiring target control data and environmental data of the temperature control system at the previous moment and the current moment in real time, and using the trained reinforcement learning model to output a fan control scheme for temperature control.

[0010] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, offline training of a reinforcement learning model based on the historical data set includes: performing data processing on the historical data set to generate an input data set for the reinforcement learning model; determining a target offline reinforcement learning algorithm; and performing model training on the input data set based on the target offline reinforcement learning algorithm.

[0011] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, data processing is performed on the historical data set to generate an input data set for the reinforcement learning model, including: constructing a state vector based on the environmental data and the target control data; constructing an action vector based on the wind turbine control data; and generating the input data set for the reinforcement learning model based on the state vector and the action vector.

[0012] In some exemplary embodiments of the present disclosure, based on the aforementioned solution, a reward function of the reinforcement learning model is constructed based on the state vector and the action vector.

[0013] In some exemplary embodiments of the present disclosure, based on the aforementioned solution, determining a target offline reinforcement learning algorithm includes: determining the target offline reinforcement learning algorithm by the following formula:

[0014]

[0015] Among them, Q min represents the minimum value of the Q(s,a) function; s represents the state; a represents the action; α represents the hyperparameter; E s～D represents the expected value of state s; D represents the historical data set; log∑ α exp(Q(s,a)) represents the logarithm of the exponential sum of the Q values ​​of all actions a in state s; E a～D( a | s ) [Q(s,a)] represents the expected value of the Q value of action a in a given state s; TDloss represents the standard time difference loss term.

[0016] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the environmental data includes: temperature data of the target space, temperature data of the environmental space, the target control data is: controlling the temperature data of the target space to reach a preset value, and the fan control data includes: fan speed data.

[0017] According to another aspect of the present disclosure, a temperature control device based on offline reinforcement learning is also provided, including: a historical operation data acquisition module, used to acquire historical operation data of a temperature control system to form a historical data set, wherein the temperature control system includes refrigeration equipment and a fan; the historical operation data includes: environmental data, target control data and fan control data; a reinforcement learning model training module, used to train a reinforcement learning model offline based on the historical data set; a fan control scheme output module, used to acquire in real time the target control data and environmental data of the temperature control system at the previous moment and the current moment, and use the trained reinforcement learning model to output a fan control scheme for temperature control.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned temperature control methods based on offline reinforcement learning by executing the executable instructions.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned temperature control methods based on offline reinforcement learning is implemented.

[0020] According to another aspect of the present disclosure, a computer program product is further provided, comprising: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the computer program or instructions implements any of the above-mentioned temperature control methods based on offline reinforcement learning.

[0021] The embodiments of the present disclosure provide a temperature control method, device, and related equipment based on offline reinforcement learning. By combining offline training of reinforcement learning with adaptive adjustment of the temperature control system, the problems of traditional methods such as reliance on fixed parameters and response lag are overcome. The embodiments of the present disclosure can provide precise, stable, and low-latency control effects, and have low computing resource overhead, adapting to complex and dynamically changing industrial environments. At the same time, it also has strong generalization capabilities, which enables the embodiments of the present disclosure to be efficiently deployed in a variety of industrial scenarios, greatly improving the performance and flexibility of the temperature control system. This innovative temperature control method not only solves practical industrial problems, but also promotes the further application and popularization of reinforcement learning in industrial automation.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0024] Figure 1 A schematic diagram of the online reinforcement learning process in related technologies is shown;

[0025] Figure 2 A schematic diagram showing an exemplary application system architecture of a temperature control method based on offline reinforcement learning in an embodiment of the present disclosure is shown;

[0026] Figure 3 A schematic diagram of a temperature control method based on offline reinforcement learning in an embodiment of the present disclosure is shown;

[0027] Figure 4 A schematic diagram showing a simulation of a temperature control process in an embodiment of the present disclosure is shown;

[0028] Figure 5 A schematic diagram of a data set design according to an embodiment of the present disclosure is shown;

[0029] Figure 6 A schematic diagram of an offline reinforcement learning process in an embodiment of the present disclosure is shown;

[0030] Figure 7 A schematic diagram showing a temperature control result in an embodiment of the present disclosure is shown;

[0031] Figure 8 Another schematic diagram of temperature control results in an embodiment of the present disclosure is shown;

[0032] Figure 9 A schematic diagram of a temperature control device based on offline reinforcement learning in an embodiment of the present disclosure is shown;

[0033] Figure 10 A schematic diagram of an electronic device using a temperature control method based on offline reinforcement learning in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0036] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0037] Figure 2 FIG. 1 shows an exemplary application system architecture diagram to which the temperature control method based on offline reinforcement learning in the embodiment of the present disclosure can be applied. Figure 2 As shown, the system architecture may include a terminal device 201 , a network 202 and a server 203 .

[0038] The network 202 is a medium for providing a communication link between the terminal device 201 and the server 203, and can be a wired network or a wireless network.

[0039] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0040] The terminal device 201 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0041] Optionally, the client of the application installed in different terminal devices 201 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile phone client, a PC client, etc.

[0042] The server 203 may be a server that provides various services, such as a background management server that provides support for the devices operated by the user using the terminal device 201. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0043] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0044] Those skilled in the art will know that Figure 2 The number of terminal devices, networks, and servers in the embodiment is merely illustrative, and any number of terminal devices, networks, and servers may be provided based on actual needs. This embodiment of the present disclosure does not limit this.

[0045] Under the above system architecture, an embodiment of the present disclosure provides a temperature control method based on offline reinforcement learning, which can be executed by any electronic device with computing and processing capabilities.

[0046] In some embodiments, the temperature control method based on offline reinforcement learning provided in the embodiments of the present disclosure can be executed by the terminal device of the above-mentioned system architecture; in other embodiments, the temperature control method based on offline reinforcement learning provided in the embodiments of the present disclosure can be executed by the server in the above-mentioned system architecture; in other embodiments, the temperature control method based on offline reinforcement learning provided in the embodiments of the present disclosure can be implemented by the terminal device and server in the above-mentioned system architecture through interaction.

[0047] For ease of understanding, the following terms are explained in the embodiments of this disclosure:

[0048] Offline reinforcement learning (Offline RL): Offline reinforcement learning is a branch of reinforcement learning in which the agent learns the optimal strategy by using historical data before interacting with the environment. Unlike traditional reinforcement learning methods (online reinforcement learning), offline reinforcement learning does not require real-time interaction with the environment during training. Instead, it uses pre-collected datasets (e.g., historical environment state, action, reward pairs) for training and strategy learning.

[0049] Figure 3 A schematic diagram of a temperature control method based on offline reinforcement learning in an embodiment of the present disclosure is shown. The method includes the following steps:

[0050] S302, acquiring historical operating data of a temperature control system to form a historical data set. The temperature control system includes refrigeration equipment and fans. The historical operating data includes: environmental data, target control data, and fan control data.

[0051] It should be noted that the temperature control system in the embodiment of the present disclosure may refer to an equipment system for regulating the temperature of a certain space (such as a room, a machine room, or a carriage), which generally includes: a refrigeration device, a fan, and a control system. More specifically, the refrigeration device may be an air-conditioning compressor, a chiller, a heat pump, etc. The fan is a device for air circulation and air supply, such as an indoor fan coil unit, a fan assembly, etc. The control system is used to monitor the environment and adjust the operating status of the equipment; secondly, the historical operating data in the embodiment of the present disclosure refers to the various operating parameters and feedback information recorded during the long-term operation of the temperature control system. The historical operating data includes: the environment Data, target control data and fan control data, among which environmental data is status information describing the environment in which the temperature control system is located, which can usually include the current indoor temperature, outdoor temperature, humidity, etc.; target control data refers to the target value set by the user, that is, the control target to be achieved, which can usually include the set temperature, control mode, user demand level, etc. For example, the user wants the indoor temperature to be 26°C, and 26°C here is the target control data; in addition, the fan control data in the embodiment of the present disclosure refers to the control instructions received by the fan during operation or its actual operating status, which can usually include the wind speed set value, actual speed, etc.

[0052] S304: Offline training of a reinforcement learning model based on the historical data set.

[0053] It should be noted that the historical data set in the embodiment of the present disclosure refers to a large amount of operation and feedback data recorded during the operation of the temperature control system, which can generally include state, action, reward and next state, wherein the state is used to describe the current situation of the environment or system; the action is the operation performed by the intelligent agent (controller); the reward is a feedback signal used to measure the quality of the action; the next state is the new state after the action is executed; secondly, the offline training in the embodiment of the present disclosure refers to not interacting with the real environment, but only using existing historical data to train the model; in addition, the reinforcement learning model in the embodiment of the present disclosure is a machine learning model that learns the optimal strategy through a trial and error mechanism, which is generally composed of the following parts: Q function, policy network, target network, wherein the Q function evaluates the value of taking a certain action in a certain state; the policy network outputs the action to be taken in a given state; the target network is used to stabilize the training process. In more detail, in offline training, common models include Deep Q-Network (DQN) model, Conservative Q-Learning (CQL) model, Batch-Constrained Q Learning (Batch-Constrained For example, the embodiment of the present disclosure can use the CQL algorithm to train a temperature controller model so that it automatically adjusts the fan speed according to the indoor temperature and the set temperature to achieve optimal comfort and energy efficiency.

[0054] S306 , obtaining the target control data and environmental data of the temperature control system at the previous moment and the current moment in real time, and using the trained reinforcement learning model to output a fan control solution for temperature control.

[0055] It should be noted that the fan control scheme in the embodiment of the present disclosure refers to the specific control instructions output by the model, which are used to adjust the working state of the fan. In more detail, common control actions include fan start and stop, wind speed setting, etc. For example, the current set temperature is 26°C and the indoor temperature is 27°C. The fan control scheme output by the reinforcement learning model may be a suggestion to set the fan to a high speed to increase the speed.

[0056] The temperature control method based on offline reinforcement learning provided in the embodiments of the present disclosure first obtains historical operating data of the temperature control system, including environmental data, target control data, and fan control data, to form a historical data set; then, a reinforcement learning model is trained offline based on the historical data set; finally, the target control data and environmental data of the temperature control system at the previous and current moments are obtained in real time, and the trained reinforcement learning model is used to output a fan control scheme for temperature control. Compared with the problem that temperature control methods in the related art are difficult to adapt to dynamic environments, the embodiments of the present disclosure overcome the problems of traditional methods such as reliance on fixed parameters and response lag by combining offline reinforcement learning training with adaptive adjustment of the temperature control system. The embodiments of the present disclosure can provide accurate, stable, and low-latency control effects, have low computing resource overhead, adapt to complex and dynamically changing industrial environments, and also have strong generalization capabilities. This enables the embodiments of the present disclosure to be efficiently deployed in a variety of industrial scenarios, greatly improving the performance and flexibility of the temperature control system. This innovative temperature control method not only solves practical industrial problems, but also promotes the further application and popularization of reinforcement learning in industrial automation.

[0057] In some embodiments, as Figure 4 The temperature control system in the embodiment of the present disclosure shown includes a refrigeration device and a fan. First, the refrigeration device starts and generates a cold water flow, which then flows through the fan. Under the action of the fan, cold air is blown into the room, and the cold water exchanges heat with the air in the room, thereby achieving cooling control of the room.

[0058] In some embodiments, the embodiments of the present disclosure train a reinforcement learning model based on a historical data set offline, including: processing the historical data set to generate an input data set for the reinforcement learning model; determining a target offline reinforcement learning algorithm; and training the input data set based on the target offline reinforcement learning algorithm. Specifically, the embodiments of the present disclosure process the historical data set to obtain an input data set suitable for reinforcement learning model training, which can not only provide high-quality, structured training samples, but also facilitate the subsequent reinforcement learning model to understand the dynamics of the environment. In more detail, the data processing process of the embodiments of the present disclosure can include data cleaning processing to remove outliers and fill in missing values, normalization processing to make data of different dimensions comparable, extraction of the states of the previous moment and the current moment to construct a time series relationship, etc.; In addition, the embodiment of the present disclosure selects the target offline reinforcement learning algorithm to determine the learning method of the model and whether the final strategy is safe, stable, and energy-saving. By training the input data set through the target offline reinforcement learning algorithm, the model can learn to make intelligent decisions based on the current environmental state, which can replace or assist the traditional PID controller to achieve automatic, energy-saving, and stable temperature regulation.

[0059] In some embodiments, the disclosed embodiments process historical data sets to generate input data sets for a reinforcement learning model, including: constructing a state vector based on environmental data and target control data; constructing an action vector based on fan control data; and generating an input data set for a reinforcement learning model based on the state vector and the action vector. Specifically, the disclosed embodiments combine the current environmental data of the temperature control system (such as indoor and outdoor temperatures, etc.) and the target control data set (such as the set temperature) set by the user into a state vector to describe the current status of the temperature control system, abstract the complex system state into a numerical vector, and encode the fan control data (such as wind speed setting value, start and stop status) into an action vector to indicate what behavior the fan has taken in this state. The disclosed embodiments combine all collected state vectors and corresponding control actions into a pair of data samples to form a complete data structure, which serves as training data for the reinforcement learning model.

[0060] In some embodiments, the disclosed embodiment keeps the temperature of the refrigeration equipment constant and the temperature of the cold water flow stable, so it can be seen that the speed of the fan determines the cooling effect of the room. The disclosed embodiment designs the historical data set as follows Figure 5 As shown, the environmental data (e.g., indoor temperature (Room_temp), outdoor temperature (Outdoor_temp) of the day) and the target control data (e.g., target temperature (Target_temp)) are combined into a state vector (State) in reinforcement learning, and the fan control data (e.g., fan speed (Fan_speed)) is combined into an action vector (Action). Specifically, the state vector and action vector in the embodiment of the present disclosure can be written as:

[0061] State=np.array([Room_temp, Outdoor_temp, Target_temp]);

[0062] Action=np.array([Fan_speed]).

[0063] In some embodiments, the present disclosure constructs a reward function for a reinforcement learning model based on the state vector and the action vector. Specifically, the present disclosure designs the reward function as: Reward = (-1.0×abs(Error)+0.3×max(Delta_error, 0)-0.3×(Action_change)).

[0064] Among them, Error represents the difference between the current temperature and the target temperature. The larger the Error, the greater the penalty. Delta_error represents the difference between the previous Error and the current Error, and rewards the convergence trend. Action_change represents the change between two actions, and punishes action jumps.

[0065] In some embodiments, the present disclosure determines a target offline reinforcement learning algorithm, including: determining the target offline reinforcement learning algorithm by the following formula (1):

[0066]

[0067] Among them, Q min represents the minimum value of the Q(s,a) function; s represents the state; a represents the action; α represents the hyperparameter; E s～D represents the expected value of state s; D represents the historical data set; log∑ α exp(Q(s,a)) represents the logarithm of the exponential sum of the Q values ​​of all actions a in state s; E a～D(a|s) [Q(s,a)] represents the expected value of the Q value of action a in a given state s; TDloss represents the standard time difference loss term.

[0068] In some embodiments, the present disclosure uses the offline reinforcement learning CQL algorithm in the D3rlpy library, such as Figure 6 As shown in Figure 2, in offline reinforcement learning, there is only a static data set and no interaction with the environment. In this case, the standard Q-learning method will produce an overly high Q-value estimate, especially for those actions that the policy has never explored. This will cause the policy to favor actions that seem to have high rewards but are actually unknown or even wrong. Therefore, the CQL algorithm is used to conservatively estimate all actions that have not been observed in the data (i.e., reduce their Q-values) to avoid the policy from selecting them. CQL modifies the objective function of Q-learning as shown in formula (1): the first term is the conservative loss term, which prevents the Q-value of unseen actions from being higher than that of seen actions; the second term is the standard TD loss term (Bellman error); the hyperparameter α controls the degree of conservatism.

[0069] In some embodiments, the advantages of using offline reinforcement learning in the embodiments of the present disclosure are: avoiding online trial and error and improving safety: in high-risk environments such as manufacturing, autonomous driving, and energy control, direct online learning may have catastrophic consequences. Offline RL can learn based on historical data without the need for repeated on-site experiments; making full use of existing production data: industrial systems often accumulate a large amount of sensor, operation, and status historical data. Offline RL can directly use these logs for strategy optimization and increase data value; reducing experimental costs: no expensive simulation systems or repeated actual experiments are required, and it is suitable for applications where each experiment is costly.

[0070] In some embodiments, in order to apply the trained reinforcement learning model to a real control system, the embodiments of the present disclosure plan to deploy the model to a training platform, interact with field equipment through an industrial standard module bus communication protocol, and realize intelligent adjustment and control of the fan speed.

[0071] In more detail, during operation, the training platform collects the current room temperature, outdoor ambient temperature, and the set target temperature to form the current state vector (State) of the system. The reinforcement learning model deployed on the edge controller or industrial computing platform performs forward reasoning based on the input state and outputs an action vector as the control instruction for the fan. Because the sensor register range is different from the actual one, the reasoning result needs to be converted when transmitted to the execution module: Fan_speed = (Action × 27648) / 100.

[0072] In some embodiments, as Figure 7 and Figure 8 As shown, the temperature control with the target temperature of 15°C is realized twice by using reinforcement learning model reasoning. It can be observed that the temperature drops rapidly and stabilizes at around 15°C. Therefore, the advantages of the temperature control system using the temperature control method based on offline reinforcement learning in the embodiment of the present disclosure are: strong real-time performance: the module bus communication responds quickly and supports millisecond-level data interaction; high intelligence: the model automatically learns the control strategy based on historical experience and can adaptively adjust in different environments; flexible deployment: the model can be deployed on edge servers, industrial PCs or embedded controllers; scalable: more sensors (such as humidity, PM2.5) and control objects (such as heaters, curtains, etc.) can be connected to achieve multi-target control; low-latency reasoning: after the reinforcement learning model is deployed, action decisions can be completed at the millisecond level.

[0073] In some embodiments, the environmental data in the disclosed embodiments includes: target space temperature data; ambient space temperature data; target control data includes: controlling the target space temperature data to a preset value; and fan control data includes: fan speed data. Specifically, the disclosed embodiments can collect actual operating data from a cold storage training platform. The higher the wind speed, the more efficient the cold air delivery, allowing more low-temperature air to be introduced into the room, achieving precise and adjustable cooling and temperature control effects.

[0074] In some embodiments, compared with the related technologies, the embodiments of the present disclosure are simple to implement, low-cost, and inexpensive; the reasoning speed is fast, ensuring the real-time nature of the regulation; and the generalization ability is good, which can adapt to complex environmental changes.

[0075] Based on the same inventive concept, the present disclosure also provides a temperature control device based on offline reinforcement learning, as shown in the following embodiment. Since the principle of solving the problem in this device embodiment is similar to that in the above method embodiment, the implementation of this device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0076] Figure 9 A schematic diagram of a temperature control device based on offline reinforcement learning in an embodiment of the present disclosure is shown, the device comprising:

[0077] The historical operation data acquisition module 901 is used to acquire the historical operation data of the temperature control system to form a historical data set. The temperature control system includes refrigeration equipment and fans. The historical operation data includes: environmental data, target control data, and fan control data.

[0078] A reinforcement learning model training module 902 is used to train the reinforcement learning model offline based on the historical data set;

[0079] The fan control scheme output module 903 is used to obtain the target control data and environmental data of the temperature control system at the previous moment and the current moment in real time, and output the fan control scheme for temperature control using the trained reinforcement learning model.

[0080] In an embodiment of the present disclosure, a temperature control device based on offline reinforcement learning is provided. A historical operation data acquisition module acquires historical operation data of a temperature control system, including environmental data, target control data, and fan control data, to form a historical data set. A reinforcement learning model training module offline trains a reinforcement learning model based on the historical data set. A fan control solution output module acquires target control data and environmental data of the temperature control system at the previous and current moments in real time. The trained reinforcement learning model is used to output a fan control solution for temperature control. Compared to the problem that temperature control methods in related arts are difficult to adapt to dynamic environments, the present embodiment combines offline reinforcement learning training with adaptive adjustment of the temperature control system, overcoming the problems of traditional methods such as reliance on fixed parameters and response lag. The present embodiment can provide accurate, stable, and low-latency control effects, has low computing resource overhead, and adapts to complex and dynamically changing industrial environments. It also has strong generalization capabilities, which enables the present embodiment to be efficiently deployed in a variety of industrial scenarios, greatly improving the performance and flexibility of the temperature control system. This innovative temperature control method not only solves practical industrial problems, but also promotes the further application and popularization of reinforcement learning in industrial automation.

[0081] In some embodiments, the reinforcement learning model training module in the embodiments of the present disclosure is also used to process historical data sets to generate input data sets for the reinforcement learning model; determine the target offline reinforcement learning algorithm; and perform model training on the input data set based on the target offline reinforcement learning algorithm.

[0082] In some embodiments, the reinforcement learning model training module in the embodiments of the present disclosure is also used to construct a state vector based on environmental data and target control data; construct an action vector based on fan control data; and generate an input data set for the reinforcement learning model based on the state vector and action vector.

[0083] In some embodiments, the embodiments of the present disclosure construct a reward function of a reinforcement learning model based on a state vector and an action vector.

[0084] In some embodiments, the reinforcement learning model training module in the embodiments of the present disclosure is also used to determine the target offline reinforcement learning algorithm through the following formula (1).

[0085] In some embodiments, the environmental data in the embodiments of the present disclosure include: temperature data of the target space, temperature data of the environmental space, target control data is: controlling the temperature data of the target space to reach a preset value, and fan control data includes: fan speed data.

[0086] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0087] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-mentioned temperature control methods based on offline reinforcement learning by executing the executable instructions. Since the principle of solving the problem in this electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of this electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0088] Refer to the following Figure 10 1000 according to this embodiment of the present disclosure will be described. Figure 10 The electronic device 1000 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 10 As shown, electronic device 1000 is implemented as a general-purpose computing device. Components of electronic device 1000 may include, but are not limited to, the aforementioned at least one processing unit 1001, the aforementioned at least one storage unit 1002, and a bus 1003 connecting different system components (including storage unit 1002 and processing unit 1001).

[0090] The storage unit stores program codes, which can be executed by the processing unit 1001, so that the processing unit 1001 executes the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present disclosure.

[0091] In some embodiments, when the electronic device is used to control, for example, the temperature control method based on offline reinforcement learning described above, the processing unit 1001 may perform the following steps of the above method embodiment:

[0092] The historical operating data of the temperature control system is obtained to form a historical data set. The temperature control system includes refrigeration equipment and fans. The historical operating data includes: environmental data, target control data, and fan control data. The reinforcement learning model is trained offline based on the historical data set. The target control data and environmental data of the temperature control system at the previous and current moments are obtained in real time. The trained reinforcement learning model is used to output a fan control solution for temperature control.

[0093] The storage unit 1002 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 10021 and / or a cache memory unit 10022 , and may further include a read-only memory unit (ROM) 10023 .

[0094] The storage unit 1002 may also include a program / utility 10024 having a set (at least one) of program modules 10025, such program modules 10025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0095] Bus 1003 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0096] The electronic device 1000 may also communicate with one or more external devices 1004 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1000, and / or any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 1005. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1006. As shown, the network adapter 1006 communicates with other modules of the electronic device 1000 via a bus 1003. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0098] Based on the same inventive concept, the present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements any of the above-mentioned temperature control methods based on offline reinforcement learning. Because the principles for solving the problems in this computer-readable storage medium embodiment are similar to those in the above-mentioned method embodiment, the implementation of this computer-readable storage medium embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated here.

[0099] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0101] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0102] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0103] Based on the same inventive concept, the present disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the offline reinforcement learning-based temperature control method of any of the above-mentioned method embodiments. Since the principles for solving the problem in this computer program product embodiment are similar to those in the above-mentioned method embodiment, the implementation of this computer program product embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated here.

[0104] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0105] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0106] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0107] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A temperature control method based on offline reinforcement learning, characterized in that: include: Acquire historical operating data of a temperature control system to form a historical data set, wherein the temperature control system includes a refrigeration device and a fan; The historical operation data includes: environmental data, target control data and fan control data; Offline training of a reinforcement learning model based on the historical dataset; The target control data and environmental data of the temperature control system at the previous moment and the current moment are obtained in real time, and the fan control scheme is output using the trained reinforcement learning model to perform temperature control.

2. The temperature control method based on offline reinforcement learning according to claim 1, characterized in that: Offline training of a reinforcement learning model based on the historical data set includes: Processing the historical data set to generate an input data set for the reinforcement learning model; Determine the target offline reinforcement learning algorithm; Model training is performed on the input data set based on the target offline reinforcement learning algorithm.

3. The temperature control method based on offline reinforcement learning according to claim 2, characterized in that: Processing the historical data set to generate an input data set for the reinforcement learning model includes: constructing a state vector based on the environmental data and the target control data; constructing an action vector based on the fan control data; An input data set of the reinforcement learning model is generated according to the state vector and the action vector.

4. The temperature control method based on offline reinforcement learning according to claim 3, characterized in that: A reward function of the reinforcement learning model is constructed based on the state vector and the action vector.

5. The temperature control method based on offline reinforcement learning according to claim 2, characterized in that: Determine the target offline reinforcement learning algorithm, including: The target offline reinforcement learning algorithm is determined by the following formula: Among them, Q min represents the minimum value of the Q(s,a) function; s represents the state; a represents the action; α represents the hyperparameter; E s～D represents the expected value of state s; D represents the historical data set; log∑ α exp(Q(s,a)) represents the logarithm of the exponential sum of the Q values ​​of all actions a in state s; E a～D(a|s) [Q(s,a)] represents the expected value of the Q value of action a in a given state s; TDloss represents the standard time difference loss term.

6. The temperature control method based on offline reinforcement learning according to claim 1, characterized in that: The environmental data includes: temperature data of the target space and temperature data of the environmental space; the target control data is: controlling the temperature data of the target space to reach a preset value; and the fan control data includes: speed data of the fan.

7. A temperature control device based on offline reinforcement learning, characterized in that: include: A historical operation data acquisition module is used to acquire historical operation data of a temperature control system to form a historical data set, wherein the temperature control system includes a refrigeration device and a fan; The historical operation data includes: environmental data, target control data and fan control data; A reinforcement learning model training module, configured to train the reinforcement learning model offline based on the historical data set; The fan control scheme output module is used to obtain the target control data and environmental data of the temperature regulation system at the previous moment and the current moment in real time, and output the fan control scheme for temperature control using the trained reinforcement learning model.

8. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the temperature control method based on offline reinforcement learning according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the temperature control method based on offline reinforcement learning according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the temperature control method based on offline reinforcement learning according to any one of claims 1 to 6.

Images