Intelligent heating control method based on deep reinforcement learning and adaptive control

Abstract

The present invention relates to the technical field of heating network control, and discloses an intelligent heating control method based on deep reinforcement learning and adaptive control, comprising the following steps: S1: collecting outdoor environment data and heat load data of each heating station, and performing data preprocessing; S2: training a deep reinforcement learning model based on the outdoor environment data and heat load data; S3: formulating a heating strategy based on the deep reinforcement learning model; S4: executing the heating strategy through an adaptive control algorithm. The present invention improves energy utilization efficiency, reduces unnecessary energy loss, enhances the stability and reliability of the entire heating system, ensures heating quality, and optimizes energy distribution.

Description

Technical Field

[0001] The present invention relates to the technical field of heating pipe network regulation, and in particular to an intelligent heating regulation method based on deep reinforcement learning and adaptive control. Background Art

[0002] With the acceleration of urbanization and the popularization of environmental protection and energy-saving concepts, the intelligent and refined management of heating systems has become an inevitable trend in the development of the industry. Especially in the northern region, the centralized heating system bears the important responsibility of ensuring the heating needs of residents in winter. Its operating efficiency and energy utilization rate directly affect the city's energy consumption structure and environmental protection level. Modern heating systems have gradually developed from the initial simple temperature control to the use of automatic control systems, realizing the basic function of on-demand heating. The application of automation equipment such as sensors and controllers enables the heating system to monitor the changes in indoor and outdoor environmental temperatures in real time and adjust the working status of the heating equipment accordingly. At the same time, advanced data analysis technology has also been applied in the field of heating. Through the mining and analysis of historical data, future heat load demand can be predicted, thereby achieving a certain degree of predictive regulation.

[0003] However, traditional heating control modes are mostly based on fixed rules or simplified mathematical models, which are difficult to fully cope with complex and changeable external environmental conditions and changes in user heat demand. For example, sudden weather changes may cause the error of existing prediction models to increase, and the heating strategy cannot be adjusted in a timely and accurate manner, which in turn leads to the problem of oversupply or undersupply of heating. In addition, due to the lack of dynamic optimization mechanism, the system often finds it difficult to achieve the global optimal energy configuration, and the heating efficiency is greatly limited. Most of the current heating strategies rely on statically set thresholds and empirical rules, and do not have the ability of online learning and continuous optimization. Faced with rapidly changing environmental conditions and user behavior habits, this "one-size-fits-all" control method is likely to lead to waste of heating resources or imbalance between supply and demand. Because the control means are relatively single and not precise enough, there is often a large deviation between energy consumption and actual heat demand, which not only increases operating costs, but also aggravates environmental pollution. In summary, although the automation and informatization construction of the heating industry has made certain progress, it still faces many challenges in actual operation.

[0004] For example, a Chinese patent with the authorization announcement number CN110736129B discloses an intelligent balancing control system and method for urban heating pipe networks. The system includes: a first regulating component, a second regulating component, a plate heat exchanger unit and a control cabinet. By setting a first regulating valve on the primary water supply pipe, a first temperature sensor on the primary return water pipe, a second regulating valve and a second temperature sensor on the secondary water supply pipe, and a third regulating valve on the bypass pipe, each valve is adjusted according to the temperature value, and the regulation of the primary pipe network and the regulation of the secondary pipe network are separated. The regulation of the primary pipe network is realized by the return water temperature of the primary pipe network, and the regulation of the secondary pipe network is realized by the secondary water supply temperature. When the water supply temperature demand of the secondary network changes, the hydraulic condition of the primary network is not affected, and the hydraulic condition of the primary network is guaranteed to be stable, and the decoupled operation of the primary pipe network and the secondary pipe network is realized, solving the problem of hydraulic imbalance between the primary pipe network and the secondary pipe network.

[0005] For example, the patent application publication number CN108592173A discloses a method for controlling a heating network, including the following steps: intelligent monitoring information collection, corresponding module information, temperature information corresponding to the heating network, and pipeline information and heat source information in the heating network; obtaining the heating load of the heating network according to the current outdoor temperature information and the building information, and obtaining the actual value corresponding to the heating network at the minimum total energy consumption according to the heating load, the actual heat dissipation situation, the pipeline information and the heat source information. This invention can improve the disadvantages of traditional heating networks and reduce the total energy consumption of heating networks.

[0006] The above patents all have the problems raised by this background technology: it is difficult to fully cope with the complex and changeable external environmental conditions and changes in user heat demand, the control methods are relatively single and not precise enough, and there is often a large deviation between energy consumption and actual heat demand.

[0007] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgement or any form of suggestion that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention

[0008] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide an intelligent heating control method based on deep reinforcement learning and adaptive control to improve energy utilization efficiency and reduce unnecessary energy loss; enhance the stability and reliability of the entire heating system, ensure the heating quality and optimize energy distribution.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0010] An intelligent heating control method based on deep reinforcement learning and adaptive control comprises the following steps: S1: collecting outdoor environment data and heat load data of each heating station, and performing data preprocessing;

[0011] S2: training a deep reinforcement learning model based on the outdoor environment data and the heat load data;

[0012] S3: Formulate a heating strategy based on the deep reinforcement learning model;

[0013] S4: Execute the heating strategy through an adaptive control algorithm.

[0014] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, wherein: the outdoor environment data includes temperature, humidity, wind speed, and solar radiation intensity;

[0015] The heat load data represents the amount of heat that the heating station needs to provide in order to keep the indoor temperature of all users at the heating standard temperature.

[0016] The preprocessing includes missing value filling, outlier detection and replacement, data normalization, and data synchronization integration; wherein the method of data synchronization integration is to match the heat load data with the outdoor environment data according to time to form a time series corresponding to the heat load data and the outdoor environment data.

[0017] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, the method for training the deep reinforcement learning model is as follows:

[0018] S100: Setting the time period for heating control; constructing the state space, action set, and reward function; constructing the strategy network and initializing the parameters;

[0019] S200: Select an action from the action collection and execute it;

[0020] S300: Calculate the cumulative reward value and update the parameters of the strategy network;

[0021] S400: Entering the next time period, generating a new action set and updating the feature vector of the state space;

[0022] S500: Repeat steps S200-S400 until the cumulative reward value converges and the training of the policy network is completed; save the policy network and deploy the deep reinforcement learning model.

[0023] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, wherein: the time period is the minimum time unit for heating control;

[0024] The state space is composed of the outdoor environment data and heat load data;

[0025] The action set is dynamically generated based on the state space and time period, and the method is as follows:

[0026] For any time period A, extract the heat load data of each heating station in the time period corresponding to time period A in each year of the previous n years, where n is a positive integer; calculate the average of the n heat load data of the i-th heating station, denoted as E Ai , the value range of i is 1, 2, ..., m, where m represents the number of thermal power stations; set an arithmetic progression containing k elements, denoted as Q, Q = {q1, q2, ..., q k}; where q1 is the smallest element in the sequence Q, and its value range is (0, 1); q k is the largest element in the sequence Q, and its value range is (1, 2); then for the time period A, the action set A a The form is as follows:

[0027] A a ={a1,a2,......,a k};

[0028] The specific form of each action is as follows:

[0029] a j = {q j ·E A1 ,q j ·E A2 ,......,q j ·E Am};

[0030] Among them, a j represents the jth action in the action set, where the value of j ranges from 1, 2, ..., k; q j Represents the jth element in the arithmetic sequence Q.

[0031] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, the calculation formula of the reward function is as follows:

[0032]

[0033] Among them, s j represents the characteristic vector of the state space at the jth time period; a j represents the action performed in the jth time period; R(s j ,a j) indicates that the eigenvector in the state space is s j When , execute action a j The reward function value; α is the weight parameter, β is the proportional coefficient; E ij represents the heat load supply allocated to the i-th thermal power station in the j-th time period; represents the heat load demand of the i-th thermal power station in the j-th time period; E j It represents the total heat production of the thermal power plant in the jth time period, and the calculation formula is as follows:

[0034]

[0035] Among them, η i It represents the heat loss coefficient when the thermal power plant delivers heat load to the i-th thermal power station.

[0036] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, wherein: the characteristic vector of the state space is composed of historical outdoor environment data, real-time outdoor environment data, and historical heat load data of each thermal power station; the updating method is as follows: let the current time period entered be B, extract the heat load data and outdoor environment data of each thermal power station in the time period corresponding to the time period B in each year of the previous n years, and obtain real-time outdoor environment data to form the characteristic vector of the state space corresponding to the time period B.

[0037] As a preferred scheme of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, the method for formulating the heating strategy is as follows: at the beginning of each heating control time period, real-time outdoor environmental parameters are collected and input into the deep reinforcement learning model, and the model automatically generates an action collection and outputs the action with the highest selection probability; based on the action with the highest selection probability, the heat load supply delivered by the thermal power plant to each thermal power station in the current heating control time period is determined, and the total heat production of the thermal power plant in the current heating control time period is calculated.

[0038] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, the execution of the heating strategy specifically includes adaptively controlling the total heat production of the thermal power plant and adaptively controlling the heat load supply of the thermal power plant to each thermal power station; wherein, the method for adaptively controlling the total heat production of the thermal power plant is as follows:

[0039] Real-time collection of fuel consumption;

[0040] Design a PID controller for a thermal power plant and calculate the error term e0(t) using the following formula:

[0041] e0(t)=E j-E(t);

[0042] Among them, E(t) represents the actual heat production at the current moment, and the calculation formula is as follows:

[0043] E(t)=η0·m(t)·H v ;

[0044] Among them, η0 represents the thermal efficiency of the boiler; m(t) represents the mass of fuel currently consumed; H v Indicates the calorific value of the fuel;

[0045] The control output u0(t) at the current moment is calculated by the PID algorithm, where u0(t) represents the adjusted supply amount of fuel;

[0046] Based on u0(t), the fuel supply is adjusted through hardware devices;

[0047] The actual heat production is continuously monitored and the fuel supply is adjusted until the actual heat production reaches the total heat production.

[0048] As a preferred solution of the intelligent heating control method based on deep reinforcement learning and adaptive control described in the present invention, the method of adaptively controlling the heat load supply amount delivered by the thermal power plant to each thermal power station is as follows:

[0049] Real-time collection of primary water supply temperature, primary return water temperature, and primary water supply instantaneous flow rate of each thermal power station;

[0050] Design a PID controller for each thermal power station and calculate the error term e of the PID controller of the i-th thermal power station i (t), the formula is as follows:

[0051] e i (t) = E ij -E i (t);

[0052] Among them, E i (t) represents the actual heat load value obtained by the i-th thermal power station; the calculation formula is as follows:

[0053] E i (t) = c·(T ri -T si )·Q i (t);

[0054] Where c is the specific heat capacity of water; T ri represents the primary return water temperature of the ith thermal power station; T si represents the primary water supply temperature of the i-th thermal power station; Q i(t) represents the total amount of primary water supply of the i-th thermal power station from the beginning of the current time period to the current moment, which is obtained by integrating the instantaneous flow of primary water supply over time;

[0055] The control output u of each thermal power station at the current moment is calculated by the PID algorithm i (t), where u i (t) represents the branch valve opening and circulating pump frequency of the i-th thermal power station;

[0056] Continuously monitor the actual heat load value obtained by each thermal power station, and adjust the branch valve opening and circulation pump frequency until the actual heat load value obtained by each thermal power station reaches the allocated heat load supply.

[0057] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0058] Applying deep reinforcement learning to the heating control scenario can achieve the transformation from passive on-demand heating to active prediction and optimization of heating strategies, significantly improving the intelligence level of the heating system. Through deep reinforcement learning, macro heating strategy planning is completed, taking into account complex factors such as seasonality, weather changes, and predicted user needs; adaptive control can quickly respond to the current actual situation, overcome the limitations of a single technology in heating control, reduce the complexity of the deep reinforcement learning model, reduce its burden, and speed up the convergence of the network; the organic combination of the two can not only meet the long-term stable and efficient operation of the heating system, but also take into account the short-term flexible control needs.

[0059] The heating strategy derived from deep learning can more accurately match the actual heat demand, avoid overheating or underheating, thereby improving energy efficiency, reducing unnecessary energy loss, and contributing to energy conservation and emission reduction. The use of adaptive control algorithms to implement heating strategies can dynamically adjust the total heat output of thermal power plants and the heat load supply received by each thermal power station according to actual conditions, enhancing the stability and reliability of the entire heating system, ensuring the quality of heating, and optimizing energy distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0061] Figure 1 A flow chart of the intelligent heating control method based on deep reinforcement learning and adaptive control provided by the present invention;

[0062] Figure 2A flow chart of a method for training a deep reinforcement learning model for formulating a heating strategy provided by the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0064] Example 1

[0065] This embodiment introduces an intelligent heating control method based on deep reinforcement learning and adaptive control. Figure 1 , the method comprises the following steps:

[0066] An intelligent heating control method based on deep reinforcement learning and adaptive control includes the following steps:

[0067] S1: Collect outdoor environment data and heat load data of each heating station, and perform data preprocessing;

[0068] The outdoor environmental data include temperature, humidity, wind speed, and solar radiation intensity; the outdoor environmental data can be used to predict the heat load demand of each thermal power station. Temperature is one of the factors that most significantly affects the heat load demand. Lower outdoor temperatures will lead to an increase in heating heat load. During the heating season, high humidity will make the human body feel colder and increase people's demand for heating. Wind affects the convective heat transfer on the outer surface of the building. Increased wind speed will accelerate the heat dissipation process of the building's outer shell, resulting in increased heat loss. Solar radiation has a direct impact on the heat gain of a building, especially for buildings with large glass curtain walls or south-facing windows. Sunlight can significantly increase indoor temperatures and reduce heating needs.

[0069] The heat load data represents the amount of heat that the heating station needs to provide in order to keep the indoor temperature of all users at the heating standard temperature.

[0070] The preprocessing includes missing value filling, outlier detection and replacement, data normalization, and data synchronization integration;

[0071] The method of data synchronization and integration is to match the heat load data with the outdoor environment data according to time, so as to form a time series of the heat load data and the outdoor environment data.

[0072] The missing value filling method is one of linear interpolation, polynomial interpolation, and prediction based on time series model;

[0073] The data normalization method is one of the minimum-maximum normalization and Z-score standardization; since outdoor environmental parameters and heat load data often have different dimensions and numerical ranges, they need to be converted to the same scale so that the deep reinforcement learning model can process all input features. After completing the above preprocessing steps, the collected data can be used as an effective input for the deep reinforcement learning model, thereby building an intelligent model that can adjust the heating strategy according to real-time environmental conditions.

[0074] S2: Based on the outdoor environment data and heat load data, train a deep reinforcement learning model; refer to Figure 2 , the method is as follows:

[0075] S100: Setting the time period for heating control; constructing the state space, action set, and reward function; constructing the strategy network and initializing the parameters;

[0076] The time period is the minimum time unit for heat supply regulation; for example, one day is set as a time period;

[0077] The state space is composed of the outdoor environment data and heat load data;

[0078] The action set is dynamically generated based on the state space and time period, and the method is as follows:

[0079] For any time period A, extract the heat load data of each heating station in the time period corresponding to time period A in each year of the previous n years, where n is a positive integer; calculate the average of the n heat load data of the i-th heating station, denoted as E Ai , the value range of i is 1, 2, ..., m, where m represents the number of thermal power stations; set an arithmetic progression containing k elements, denoted as Q, Q = {q1, q2, ..., q k}; where q1 is the smallest element in the sequence Q, and its value range is (0, 1); q k is the largest element in the sequence Q, and its value range is (1, 2); then for the time period A, the action set A a The form is as follows:

[0080] A a ={a1,a2,......,a k};

[0081] The specific form of each action is as follows:

[0082] a j = {q j ·E A1 ,q j ·E A2 ,......,q j·E Am};

[0083] Among them, a j represents the jth action in the action set, where the value of j ranges from 1, 2, ..., k; q j represents the jth element in the arithmetic sequence Q;

[0084] The above scheme sets actions based on the historical heat load demand of each thermal power station as a reference. For example, the optional actions of the i-th thermal power station are set to 0.8 times, 0.9 times, 1.0 times, 1.1 times, and 1.2 times the historical average heat load demand. The policy network optimizes the actions set based on historical data by comparing the real-time outdoor environmental parameters with the historical outdoor environmental parameters, thereby improving the training speed of the deep reinforcement learning model.

[0085] The calculation formula of the reward function is as follows:

[0086]

[0087] Among them, s j represents the characteristic vector of the state space at the jth time period; a j represents the action performed in the jth time period; R(s j ,a j ) indicates that the eigenvector in the state space is s j When , execute action a j The reward function value; α is the weight parameter, β is the proportional coefficient, which are set by technicians in this field according to actual needs; E ij represents the heat load supply allocated to the i-th thermal power station in the j-th time period, which is determined by the action selected to be executed in the j-th time period; represents the heat load demand of the i-th thermal power station in the j-th time period, which is determined according to the heat load data in the state space; E j It represents the total heat production of the thermal power plant in the jth time period, which is calculated based on the heat load supply of each thermal power station. The formula is as follows:

[0088]

[0089] Among them, η i represents the heat loss coefficient when the thermal power plant delivers heat load to the i-th thermal power station, which is determined experimentally by technicians in this field;

[0090] This reward function takes into account the total heat output of the thermal power plant and the supply and demand balance of each thermal power station. While saving total energy consumption, it also balances the uniformity of the heat load supply of each thermal power station to the greatest extent.

[0091] The policy network includes an input layer, a hidden layer, and an output layer; the input layer is used to input the feature vector of the state space; the hidden layer is used to further extract the features of the state space; the output layer is used to generate the selection probability of each action in the action collection under the current state space. The output is converted into a probability distribution using the softmax function to ensure that the sum of the selection probabilities of all actions is 1;

[0092] S200: Select an action from the action collection and execute it;

[0093] The method for selecting an action is as follows:

[0094] Inputting the feature vector of the current state space into the policy network to obtain the selection probability of each action in the action set under the current state space;

[0095] Set a threshold parameter ε, with a value range of (0, 0.15];

[0096] Generate a random number r in the range of [0, 1]; if r is greater than or equal to ε, execute the action with the highest selection probability; if r is less than ε, randomly select an action from the action collection and execute it;

[0097] S300: Calculate the cumulative reward value and update the parameters of the strategy network;

[0098] The calculation formula of the cumulative reward value is as follows:

[0099]

[0100] Among them, R N represents the current cumulative reward value; N represents the number of actions that have been executed; β represents the discount factor, β j represents the jth power of the discount factor β; R(s j ,a j ) represents the state space s j Next, perform action a j The reward value of j is 1, 2, ..., N.

[0101] The calculation formula for updating the parameters of the policy network is as follows:

[0102]

[0103] Among them, δ represents any parameter in the policy network; Indicates that the function in the brackets is used to find the gradient of δ; η is the learning rate; L N is the objective function, and the calculation formula is as follows:

[0104] L N =ln(p(s N ,aN )·R N );

[0105] Among them, p(s N ,a N ) indicates that in the environment state s N Next action a N The probability of selection.

[0106] S400: Entering the next time period, generating a new action set and updating the feature vector of the state space;

[0107] The characteristic vector of the state space is composed of historical outdoor environment data, real-time outdoor environment data, and historical heat load data of each thermal power station; the method for updating the characteristic vector of the state space is as follows: let the current time period be B, extract the heat load data and outdoor environment data of each thermal power station in the time period corresponding to time period B in each year of the previous n years, and obtain real-time outdoor environment data to form the characteristic vector of the state space corresponding to time period B.

[0108] During the training phase of the deep reinforcement learning model, the real-time outdoor environment data is actually acquired by reading the outdoor environment data of the corresponding time period in the state space; after the model training is completed, during the actual deployment and application process, the real-time outdoor environment data is collected in real time through sensors and other detection equipment.

[0109] S500: Repeat steps S200-S400 until the cumulative reward value converges and the training of the policy network is completed; save the policy network and deploy the deep reinforcement learning model.

[0110] After multiple iterations, the cumulative reward value tends to be stable and no longer fluctuates significantly. In other words, the cumulative reward value is considered to have converged, and the policy network is able to make a decision that maximizes the cumulative reward value.

[0111] S3: Formulate a heating strategy based on the deep reinforcement learning model; the method is as follows:

[0112] At the beginning of each heating regulation time period, real-time outdoor environmental parameters are collected and input into the deep reinforcement learning model, and the model automatically generates a collection of actions and outputs the action with the highest selection probability; based on the action with the highest selection probability, the heat load supply delivered by the thermal power plant to each thermal power station in the current heating regulation time period is determined, and the total heat production of the thermal power plant in the current heating regulation time period is calculated.

[0113] S4: Executing the heating strategy through an adaptive control algorithm, including adaptively controlling the total heat output of the thermal power plant and adaptively controlling the heat load supply of the thermal power plant to each thermal power station.

[0114] The method for adaptively controlling the total heat output of a thermal power plant is as follows:

[0115] Real-time collection of fuel consumption;

[0116] Design a PID controller for a thermal power plant and calculate the error term e0(t) using the following formula:

[0117] e0(t)=E j -E(t);

[0118] Among them, E(t) represents the actual heat production at the current moment, and the calculation formula is as follows:

[0119] E(t)=η0·m(t)·H v ;

[0120] Among them, η0 represents the thermal efficiency of the boiler; it is the efficiency of the boiler in converting the heat energy released by fuel combustion into water supply heat; m(t) represents the mass of fuel currently consumed; H v Indicates the calorific value of the fuel; it is the amount of heat released when a unit mass of fuel is completely burned, usually given in the fuel product manual;

[0121] The control output u0(t) at the current moment is calculated by the PID algorithm, where u0(t) represents the adjusted supply amount of fuel;

[0122] Based on u0(t), the fuel supply is adjusted through hardware devices; the control output signal u0(t) is mapped to the actual adjustment range of the fuel supply, and the fuel supply is accurately controlled by adjusting the combustion controller or the servo motor to drive the fuel valve;

[0123] Continuously monitor the actual heat production and adjust the fuel supply until the actual heat production reaches the total heat production;

[0124] The parameters of the PID controller are initially set through on-site debugging and empirical formulas based on the boiler response characteristics and system stability requirements, and are adjusted regularly.

[0125] The method for adaptively controlling the heat load supply of the thermal power plant to each thermal power station is as follows:

[0126] The primary water supply temperature, primary return water temperature and primary water supply instantaneous flow rate of each thermal power station are collected in real time; the heating network data is obtained in real time by installing flow meters and thermometers on the main pipeline network.

[0127] Design a PID controller for each thermal power station and calculate the error term e of the PID controller of the i-th thermal power station i (t), the formula is as follows:

[0128] e i (t) = Eij -E i (t);

[0129] Among them, E i (t) represents the actual heat load value obtained by the i-th thermal power station; the calculation formula is as follows:

[0130] E i (t) = c·(T ri -T si )·Q i (t);

[0131] Where c is the specific heat capacity of water; T ri represents the primary return water temperature of the ith thermal power station; T si represents the primary water supply temperature of the i-th thermal power station; Q i (t) represents the total amount of primary water supply of the i-th thermal power station from the beginning of the current time period to the current moment, which is obtained by integrating the instantaneous flow of primary water supply over time;

[0132] The control output u of each thermal power station at the current moment is calculated by the PID algorithm i (t), where u i (t) represents the branch valve opening and circulating pump frequency of the i-th thermal power station;

[0133] Each branch valve is equipped with an electric actuator, which can remotely adjust the valve opening; the circulating water pump has a frequency conversion control function, which can adjust its speed; by adjusting the branch valve opening and the circulating pump frequency, the distribution ratio of the primary water supply from the thermal power station to each thermal power station can be adjusted, and then the distribution ratio of the heat load supply of each thermal power station can be adjusted.

[0134] Continuously monitor the actual heat load value obtained by each thermal power station, and adjust the branch valve opening and circulation pump frequency until the actual heat load value obtained by each thermal power station reaches the allocated heat load supply.

[0135] In deep reinforcement learning, combining the advantages of adaptive control can help improve the performance and convergence speed of the algorithm. Adaptive control can solve some underlying control problems by real-time monitoring and adjusting system parameters, especially when facing changes in the dynamic characteristics of the system, it can perform local optimization faster and maintain the stability of the system.

[0136] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. An intelligent heating control method based on deep reinforcement learning and adaptive control, characterized in that: The following steps are involved: S1: Collect outdoor environment data and heat load data of each heating station, and perform data preprocessing; S2: training a deep reinforcement learning model based on the outdoor environment data and the heat load data; The method for training the deep reinforcement learning model is as follows: S100: Setting the time period for heating control; constructing the state space, action set, and reward function; constructing the strategy network and initializing the parameters; The time period is the minimum time unit for heat supply regulation; The state space is composed of the outdoor environment data and heat load data; The action set is dynamically generated based on the state space and time period, and the method is as follows: for any time period A, extract the heat load data of each thermal power station in the time period corresponding to the time period A in each year of the previous n years, where n is a positive integer; average the n heat load data of the i-th thermal power station, denoted as E Ai , the value range of i is 1, 2, ..., m, where m represents the number of thermal power stations; set an arithmetic progression containing k elements, denoted as Q, Q = {q1, q2, ..., q k }; where q1 is the smallest element in the sequence Q, and its value range is (0, 1); q k is the largest element in the sequence Q, and its value range is (1, 2); then for the time period A, the action set A a The form is as follows: <h2 style=";text-align:left;direction:ltr">A<h2 style=";text-align:left;direction:ltr"> a <h2 style=";text-align:left;direction:ltr"> (a1,a2,......,a)<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr">}; The specific form of each action is as follows: to j ={q j ·AND A1 ,q j ·AND A2 ,......,q j ·AND Am }; Among them, a j represents the jth action in the action set, where the value of j ranges from 1, 2, ..., k; q j represents the jth element in the arithmetic sequence Q; The calculation formula of the reward function is as follows: Among them, s j represents the characteristic vector of the state space at the jth time period; a j represents the action performed in the jth time period; R(s j ,a j ) indicates that the eigenvector in the state space is s j When , execute action a j The reward function value; α is the weight parameter, β is the proportional coefficient; E ij represents the heat load supply allocated to the i-th thermal power station in the j-th time period; represents the heat load demand of the i-th thermal power station in the j-th time period; E j It represents the total heat production of the thermal power plant in the jth time period, and the calculation formula is as follows: Among them, η i It represents the heat loss coefficient when the thermal power plant delivers heat load to the i-th thermal power station; S200: Select an action from the action collection and execute it; S300: Calculate the cumulative reward value and update the parameters of the strategy network; S400: Entering the next time period, generating a new action set and updating the feature vector of the state space; S500: Repeat steps S200-S400 until the cumulative reward value converges and the training of the policy network is completed; save the policy network and deploy the deep reinforcement learning model; S3: Formulate a heating strategy based on the deep reinforcement learning model; S4: Execute the heating strategy through an adaptive control algorithm.

2. The intelligent heating control method based on deep reinforcement learning and adaptive control according to claim 1, characterized in that: The outdoor environment data includes temperature, humidity, wind speed, and solar radiation intensity; The heat load data represents the amount of heat that the heating station needs to provide to keep the indoor temperature of all users at the heating standard temperature; The preprocessing includes missing value filling, outlier detection and replacement, data normalization, and data synchronization integration; wherein the method of data synchronization integration is to match the heat load data with the outdoor environment data according to time to form a time series corresponding to the heat load data and the outdoor environment data.

3. The intelligent heating control method based on deep reinforcement learning and adaptive control according to claim 2, characterized in that: The characteristic vector of the state space is composed of historical outdoor environmental data, real-time outdoor environmental data, and historical heat load data of each thermal power station; the updating method is as follows: let the current time period be B, extract the heat load data and outdoor environmental data of each thermal power station in the time period corresponding to time period B in each year of the previous n years, and obtain real-time outdoor environmental data to form the characteristic vector of the state space corresponding to time period B.

4. The intelligent heating control method based on deep reinforcement learning and adaptive control according to claim 3 is characterized in that: The method for formulating a heating strategy is as follows: at the beginning of each heating regulation time period, real-time outdoor environmental parameters are collected and input into the deep reinforcement learning model, and the model automatically generates a collection of actions and outputs the action with the highest selection probability; based on the action with the highest selection probability, the heat load supply delivered by the thermal power plant to each thermal power station in the current heating regulation time period is determined, and the total heat production of the thermal power plant in the current heating regulation time period is calculated.

5. The intelligent heating control method based on deep reinforcement learning and adaptive control according to claim 4, characterized in that: Executing the heating strategy specifically includes adaptively controlling the total heat output of the thermal power plant and adaptively controlling the heat load supply of the thermal power plant to each thermal power station; wherein the method for adaptively controlling the total heat output of the thermal power plant is as follows: Real-time collection of fuel consumption; Design a PID controller for a thermal power plant and calculate the error term e0(t) using the following formula: e0(t)=E j -E(t); Among them, E(t) represents the actual heat production at the current moment, and the calculation formula is as follows: E(t)=η0·m(t)·H v ; Among them, η0 represents the thermal efficiency of the boiler; m(t) represents the mass of fuel currently consumed; H v Indicates the calorific value of the fuel; The control output u0(t) at the current moment is calculated by the PID algorithm, where u0(t) represents the adjusted supply amount of fuel; Based on u0(t), the fuel supply is adjusted through hardware devices; The actual heat production is continuously monitored and the fuel supply is adjusted until the actual heat production reaches the total heat production.

6. The intelligent heating control method based on deep reinforcement learning and adaptive control according to claim 5, characterized in that: The method for adaptively controlling the heat load supply of the thermal power plant to each thermal power station is as follows: Real-time collection of primary water supply temperature, primary return water temperature, and primary water supply instantaneous flow rate of each thermal power station; Design a PID controller for each thermal power station and calculate the error term e of the PID controller of the i-th thermal power station i (t), the formula is as follows: e i (t)=E ij -E i (t); Among them, E i (t) represents the actual heat load value obtained by the i-th thermal power station; the calculation formula is as follows: E i (t)=c·(T ri -T si )·Q i (t); Where c is the specific heat capacity of water; T ri represents the primary return water temperature of the i-th thermal power station; T si represents the primary water supply temperature of the i-th thermal power station; Q i (t) represents the total amount of primary water supply of the i-th thermal power station from the beginning of the current time period to the current moment, which is obtained by integrating the instantaneous flow of primary water supply over time; The control output u of each thermal power station at the current moment is calculated by the PID algorithm i (t), where u i (t) represents the branch valve opening and circulating pump frequency of the i-th thermal power station; Continuously monitor the actual heat load value obtained by each thermal power station, and adjust the branch valve opening and circulation pump frequency until the actual heat load value obtained by each thermal power station reaches the allocated heat load supply.

Images