A heating network balancing method based on deep reinforcement learning
Through the heating two-network balance method based on deep reinforcement learning, the DQN algorithm is used to generate a balance control plan sequence and combine it with adaptive fuzzy control, the problem of inflexible regulation of traditional heating systems is solved, efficient and energy-saving heating services are achieved, and energy-saving heating services are reduced, energy consumption and power loss are reduced.
Patent Information
- Application Number
- CN202411760416.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-08
AI Technical Summary
It is difficult for traditional heating systems to achieve flexible regulation, resulting in low heating efficiency and serious energy waste. Especially during the peak winter season, heating demand has increased significantly, which puts forward higher requirements for the balance and regulation of heating systems.
The heating two-network balance method based on deep reinforcement learning is adopted, and the parameters of the heating two-network system are trained and learned through the DQN algorithm to generate a balance control plan sequence for different load grading, and dynamically adjust it according to real-time temperature parameters during the operation period, and correct it in combination with adaptive fuzzy control.
It has achieved a dynamic balance of demand with a higher sensitivity, reduced the power loss in the heating season of building valves, unit valves, and inlet valves in the second network, solved the problem of difficulty in obtaining power, and could not implement the business scenarios where batteries need to be used, fully tapped the energy-saving potential of the heating system, reduce energy consumption, and save costs.
Smart Images
Figure CN119778777A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of heating, and specifically, relates to a heating secondary network balancing method based on deep reinforcement learning. Background Art
[0002] With the acceleration of urbanization and the increase in energy consumption, traditional heating systems are facing many challenges. Traditional heating systems mainly use a single heating grid, which is difficult to achieve flexible regulation, often leading to problems such as low heating efficiency, serious energy waste, and high system operating costs. Especially during the winter peak period, the demand for heating increases significantly, which puts higher requirements on the balance and regulation of the heating system. At present, the secondary network balance field of the centralized heating industry mainly adopts static test solutions. However, since the heating season of the secondary network circulating pump cannot be changed, the operating conditions of the secondary network circulating pump are generally set according to the heating load in the deep cold period, resulting in the system in the early and late cold periods operating under the "big horse pulling a small cart" condition, which results in a large waste of energy. In addition, as a more advanced research direction in the field of secondary network balance in the centralized heating industry in recent years, the dynamic balance control scheme has shown significant results in experimental tests and can improve all the shortcomings of the static test scheme. However, in actual project applications, the following problems exist: the building valves and household valves in the dynamic balance control scheme need to be adaptively and repeatedly adjusted according to the operating conditions of the secondary network circulation pump. The frequency is high, resulting in a large number of regulating valves, an increase in overall energy consumption, and partial offset of the energy-saving effect; it cannot be implemented for business scenarios where it is difficult to obtain electricity and batteries need to be used. Summary of the invention
[0003] The purpose of the present invention is to provide a heating network balancing method based on deep reinforcement learning, so as to solve the problems of low energy efficiency of the heating system control method and the inability of the control strategy to adapt to the project engineering application, improve the energy efficiency of the heating system, the adaptability and real-time performance of the control strategy, and better adapt to actual engineering applications. The present invention adopts the distributed isolation design concept of "training learning" and "operation optimization" of the deep reinforcement learning algorithm to construct a smart heating network balancing method, thereby realizing efficient and energy-saving heating services, real-time monitoring and control technology, and optimizing the heating process. It is more flexible, adaptable and real-time than the dynamic balancing method, and provides a more energy-efficient control strategy.
[0004] The present invention adopts the following technical solution: a heat supply secondary network balancing method based on deep reinforcement learning, characterized in that it includes the following steps:
[0005] S1: Plan generation. The plan generation is performed during the commissioning period of the heating season. The load classification is set according to the parameters of the initial cold, deep cold, late cold, outdoor temperature, and indoor temperature. According to the requirements of different load classifications, the parameters of the heating network system are continuously trained and learned using the DQN algorithm, and a balance control plan sequence for different load classifications is generated offline.
[0006] S2: Plan call, the plan call is during the heating season, according to the parameters of early cold, deep cold, late cold, outdoor temperature, and indoor temperature as screening conditions, the corresponding balance control plan sequence under the current load classification is selected and issued for execution;
[0007] S3: Adaptive control, which is to use decentralized adaptive fuzzy control to make corrections according to feedback deviations after calling the balance control plan sequence, and set feedback deviation start dead zone according to the load characteristics of different heating network systems;
[0008] In step S1, the DQN algorithm is used to continuously train and learn the parameters of the heat supply network balance, including the evaluation network Q(s,a;w) and the target network Q(s,a;w) generated by the neural network. - ), where s is the load classification status at that moment, a is the action taken, i.e., adjusting the frequency of the secondary heating network circulation pump or adjusting the secondary side water supply temperature, w and w - is the training parameter of the neural network;
[0009] The evaluation network Q(s,a;w) is responsible for controlling the heating network system and collecting parameters for empirical training. During training, the weight w of the evaluation network Q(s,a;w) is copied to the weight w of the target network after a certain number of training. - .
[0010] Furthermore, in the step S1, the DQN algorithm training model is used to learn the secondary network circulation pump frequency and the secondary side water supply temperature when the energy consumption is lowest.
[0011] Furthermore, the DQN algorithm training model also includes an experience pool, which is composed of data interacting in the experience training of the evaluation network Q(s,a;w) and the control heating network system.<s,a,r,s′> , where s is the state of the load classification of the evaluation network at that moment, a is the action taken, r is the corresponding feedback reward for executing action a in the current state s, and s′ is the state of the load classification at the next moment.
[0012] Furthermore, in the step S1, the DQN algorithm is an offline strategy algorithm, which adopts a greedy strategy to store the collected data for use in subsequent DQN algorithm training.
[0013] Furthermore, when the evaluation network Q(s,a;w) is trained with the DQN algorithm, the target network Q(s,a;w - ) parameters remain stable within a certain number of training times.
[0014] The beneficial effects of the present invention are as follows: the present invention proposes a heating network balancing method based on deep reinforcement learning, which utilizes the advantages of the self-supervised machine learning method that does not require specific environmental model parameters and acquires knowledge through interaction with the environment, as well as the advantages of its offline strategy, and adopts the distributed isolation design concept of "training learning" and "operation optimization" to construct a heuristic plan technology route in the field of intelligent heating network balancing: plan generation-plan call-adaptive control. It can achieve a dynamic balance of demand and supply with high sensitivity. At the same time, compared with dynamic balance control schemes with similar effects, it reduces the power loss of building valves, unit valves, and household valves in the heating season of the second network, solves the problem that business scenarios that require the use of batteries due to difficulty in obtaining electricity cannot be implemented, and fully taps the energy-saving potential of the heating system, reduces energy consumption, and saves costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 The steps of the heating network balancing method based on deep reinforcement learning;
[0016] Figure 2 It is the control block diagram of adaptive fuzzy control, i.e. fuzzy control PID control method;
[0017] Figure 3 This is the framework diagram of the DQN algorithm. DETAILED DESCRIPTION
[0018] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0019] like Figure 1 The figure shows the steps of the secondary heating network balancing method based on deep reinforcement learning, which includes: plan generation, plan calling, and adaptive control.
[0020] The plan generation is during the commissioning period of the heating season. The load classification is set according to the parameters of the initial cold, deep cold, late cold, outdoor temperature, and indoor temperature. According to the requirements of different load classifications, the parameters of the heating network system are continuously trained and learned using the DQN algorithm, and a balance control plan sequence for different load classifications is generated offline.
[0021] Plan calling, the plan calling is during the heating season operation period, according to the parameters of early cold, deep cold, late cold, outdoor temperature, and indoor temperature as screening conditions, the corresponding balance control plan sequence under the current load classification is selected and issued for execution;
[0022] Adaptive control, the adaptive control is to use decentralized adaptive fuzzy control to make corrections based on feedback deviation after calling the balance control plan sequence for execution. According to the load characteristics of different heating network systems, the feedback deviation start-up dead zone is set to avoid over-regulation and cause power waste similar to dynamic balance control schemes.
[0023] It should be noted that decentralization means dispersing centralized control to more nodes, using distributed storage and computing power, and the rights and obligations of the entire network nodes are the same, so as to achieve distributed storage, recording and updating of data. The key to decentralization is to reduce the monopoly and control of a single center and increase the robustness and self-organization of the system.
[0024] like Figure 2 The figure shows the control block diagram of adaptive fuzzy control, i.e., fuzzy control PID control method, in which three PID parameters can be adjusted in real time. Fuzzy logic is used to optimize the PID parameters in real time according to fuzzy rules, thus overcoming the shortcoming that traditional PID parameters cannot be adjusted in real time.
[0025] Set the feedback deviation start dead zone, that is, according to different load grades, set the deviation range not exceeding 10%. When the predicted load differs from the actual feedback load by more than 10%, it is necessary to re-forecast the load, obtain a new load grade based on the new load, and further obtain the operating strategy under the new load grade.
[0026] Among them, after the load classification is divided, the DQN algorithm is used to train and learn the frequency of the secondary network circulation pump and the secondary side water supply temperature when the energy consumption is the lowest. The state space is the operating frequency range of the secondary network circulation pump and the reasonable range of the secondary side water supply temperature.
[0027] Since DQN is an offline strategy algorithm, we can use an ε-greedy strategy to balance exploration and utilization when collecting data, store the collected data, and use it in subsequent training. The selection function π′(s) of the greedy strategy is to select the action that can maximize the action value, and its expression is:
[0028]
[0029] The specific implementation method of the ε-greedy method is to first let the program generate a random number in the interval [0,1] from a uniform distribution. If the value is less than the preset 1-ε, then select the action that maximizes the action value. Otherwise, an action is randomly selected, where s is the state at that moment and a is the action taken (i.e., the frequency of the circulation pump of the secondary heating network and the water supply temperature on the secondary side).
[0030] DQN uses a neural network to generate the function Q(s,a;w), where s is the state at that moment, a is the action taken (i.e., the frequency of the secondary heating network circulation pump and the secondary side water supply temperature), and w is the training parameter of the neural network. When training the network, the action value estimation is related to the weight w. When the weight changes, the action value estimation will also change.
[0031] DQN uses two identical networks, namely the evaluation network Q(s,a;w) and the target network Q(s,a;w - ), the evaluation network is responsible for controlling the agent and collecting experience, and the target network is used to calculate y t , that is, y t =r t +Υ·max a Q(s t+1 ,a;w - ), where y t is the target value of the Q network; r t is the value obtained based on actual observation; max a Q(s t+1 ,a;w - ) is based on the Q network in state s t+1 The estimated value made at the time; Υ discount factor.
[0032] During the update process, only the weight w of the evaluation network Q(s,a;w) is updated, and the weight w of the target network is - Keep it unchanged. After a certain number of updates, copy the updated evaluation network weights to the target network for the next batch of updates, so that the target network can also be updated. Since the target value of the return is relatively fixed during a period of time when the target network does not change, the introduction of the target network increases the stability of learning.
[0033] Among them, the function Q(s,a;w) is generated by a neural network, namely the Q network, where s is the state at that moment, a is the action taken (i.e. the frequency of the secondary heating network circulation pump and the secondary side water supply temperature), and w is the training parameter of the neural network. The input of the neural network is the state s, and the output is the score Q value of all actions a (i.e. the value of the action). The specific steps are as follows:
[0034] 1) Initialization
[0035] a. Initialize playback memory D, whose capacity is N;
[0036] b. Initialize the action-value function Q with random weights (i.e. initialize the neural network parameters);
[0037] c. Initialize the parameter values, learning rate α, discount factor Υ, exploration rate ε;
[0038] 2) Environmental Interaction Cycle
[0039] a. Select an action based on the current state: According to the ε-greedy strategy, we can randomly select an action a with probability ε t (Explore new action space) Select the action a that maximizes Q in the current state with probability 1-ε t ,Right now
[0040] b. Execute the currently selected action and observe the next state s t+1 and the reward r for choosing this action t , get the next state and immediate reward;
[0041] c. t ,a t ,r t ,s t+1 ) is stored in playback memory D;
[0042] d. Sampling from replay memory and updating the network: When the number of samples in the replay memory D reaches a certain threshold M, a batch (s i ,a i ,r i ,s i+1 ), i = 1, ..., b, b is the batch size; for each sample, calculate the target value y i , if s i+1 is a terminal state, then y i =r i ,otherwise where w - are the parameters of the target network (the target network is used for stable training, and its parameter update is slower than that of the evaluation network). The mean square error loss function is used Perform a gradient descent update on the evaluation network, where w is the parameter of the evaluation network and b is the batch size.
[0043] e. Update the target network: Periodically copy the parameters of the main network to the target network (for example, update once every C steps).
[0044] like Figure 2 The following is the framework diagram of the DQN algorithm. First, the agent continuously interacts with the environment to obtain interaction data.<s,a,r,s′> Store it in the experience pool. When there is enough data in the experience pool, randomly take out a certain size of data from the experience pool, use the current network to calculate the predicted value of Q, use the target network to calculate the target value of Q, and then calculate the loss function between the two. Use gradient descent to update the current network parameters. After repeating several times, copy the parameters of the current network to the target network.
[0045] like Figure 3, a specific implementation method includes the following steps:
[0046] Action a=(f,T), where f is the frequency of the secondary heating network circulation pump, and T is the secondary side water supply temperature. The frequency of the secondary heating network circulation pump and the secondary side water supply temperature are set in a reasonable range, that is, the frequency of the secondary network circulation pump f∈[30Hz,50Hz], and the secondary side water supply temperature T∈[30℃,40℃].
[0047] The reward function calculation method comes from two parts, namely r t =P t -E t , where P t It is inversely proportional to the degree of deviation from the comfortable temperature (18℃-24℃), that is, the absolute value of the temperature difference multiplied by the inverse proportional coefficient -20, where E t is the energy consumption of the circulation pump at the current moment.
[0048] For example, the state s at the first moment 1 The primary side water supply temperature is 60℃, the primary side return water temperature is 40℃, the outdoor temperature is -5℃, and the current average indoor temperature of users is 19℃.
[0049] The first action selected is a 1 =(40,35). After executing this action, the current reward r is calculated according to the reward function. 1 = -273, this moment (s 1 ,a 1 ,r 1 ,s 2 ) is put into the replay memory D, and the interaction data at different times are obtained in turn. With more training and exploration under different states, the strategy will be continuously optimized. When the maximum number of iterations is reached, the optimal strategy calculated is a=(38,36).
Claims
1. A heating network balancing method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Plan generation. The plan generation is performed during the commissioning period of the heating season. The load classification is set according to the parameters of the initial cold, deep cold, late cold, outdoor temperature, and indoor temperature. According to the requirements of different load classifications, the parameters of the heating network system are continuously trained and learned using the DQN algorithm, and a balance control plan sequence for different load classifications is generated offline. S2: Plan call, the plan call is during the heating season, according to the parameters of early cold, deep cold, late cold, outdoor temperature, and indoor temperature as screening conditions, the corresponding balance control plan sequence under the current load classification is selected and issued for execution; S3: Adaptive control, which is to use decentralized adaptive fuzzy control to make corrections according to feedback deviations after calling the balance control plan sequence, and set feedback deviation start dead zone according to the load characteristics of different heating network systems; In step S1, the DQN algorithm is used to continuously train and learn the parameters of the heat supply network balance, including the evaluation network Q(s,a;w) and the target network generated by the neural network. Where s is the load classification status at that moment, a is the action taken, i.e. adjusting the frequency of the secondary heating network circulation pump or adjusting the secondary side water supply temperature, w and w - is the training parameter of the neural network; The evaluation network Q(s,a;w) is responsible for controlling the heating network system and collecting parameters for empirical training. During training, the weight w of the evaluation network Q(s,a;w) is copied to the weight w of the target network after a certain number of training. - .
2. According to claim 1, a method for balancing the heating network based on deep reinforcement learning is characterized in that: In the step S1, the DQN algorithm training model is used to learn the secondary network circulation pump frequency and the secondary side water supply temperature when the energy consumption is the lowest.
3. According to claim 2, a method for balancing the heating network based on deep reinforcement learning is characterized in that: The DQN algorithm training model also includes an experience pool, which is composed of data interacting in the experience training of the evaluation network Q(s,a;w) and the control heating network system.<s,a,r,s′> , where s is the state of the load classification of the evaluation network at that moment, a is the action taken, r is the corresponding feedback reward for executing action a in the current state s, and s′ is the state of the load classification at the next moment.
4. A method for balancing the heating network based on deep reinforcement learning according to claim 3, characterized in that: In the step S1, the DQN algorithm is an offline strategy algorithm, which adopts a greedy strategy to store the collected data for use in subsequent DQN algorithm training.
5. The method for balancing the heating network based on deep reinforcement learning according to claim 1, characterized in that: When the evaluation network Q(s,a;w) is trained empirically by the DQN algorithm, the target network Q(s,a;w - ) parameters remain stable within a certain number of training times.