A heat exchange station control method based on reinforcement learning

Abstract

The application discloses a heat exchange station control method based on reinforcement learning, and mainly comprises two parts, wherein the first part is a simulation environment model of the heat exchange station learned based on a generative adversarial method, and the second part is a control strategy training of the heat exchange station using a PPO reinforcement learning technology, so that a good heat exchange station control method is obtained. The simulation environment model of the heat exchange station is constructed based on historical data of the heat exchange station, and a generative adversarial method is adopted to learn the simulation environment model, so that the simulation environment can obtain a good simulation effect in a state that has not appeared in the historical data. Then, the simulation environment model learned by the generative adversarial structure is used to train the control strategy by using the PPO reinforcement learning technology. Since the training process of the PPO reinforcement learning technology is relatively stable and the training process variance is small, the obtained control strategy can well complete the control target given by the design reward function, and the temperature of the control result will not lag.

Description

Technical Field

[0001] This invention relates to a heat exchange station control method based on reinforcement learning, which belongs to the application of reinforcement learning in the energy field of computer technology and is particularly suitable for heating scenarios. Background Technology

[0002] A heat exchange station is a place where heat is concentrated and exchanged. It transmits high-temperature hot water or steam generated by a thermal power plant to various residential communities, transferring the heat to the community's pipe network to provide heating for residents. Based on the heating method, they are divided into direct supply stations and indirect supply stations. A heat exchange station system mainly consists of two parts: the primary network and the secondary network. The primary network provides heat to the entire system. Hot water enters the system's heat exchangers through the primary network pipes, where it releases heat energy; this portion of hot water is called primary network supply water. The secondary network supplies heat to users. Water in the secondary network pipes absorbs heat in the heat exchangers. This heat-absorbed water flows through the heating pipes, providing heat to users; this state of water is called secondary network effluent. After flowing through the heat exchange station, this water returns; this portion is called secondary network return water. Generally, controlling the secondary network return water temperature within a target temperature range is considered to indicate that the entire system is well controlled. Traditional heat exchange station control methods generally include manual control and Proportional Integral-Derivative (PID) control. However, manual control of heat exchange stations is labor-intensive and relies entirely on the operator's experience, making it prone to temperature instability and subsequent failure to meet heating standards for users. PID controllers struggle to balance dynamic and static performance requirements; their integral saturation and derivative components amplify noise, limiting their control effectiveness. Furthermore, traditional error estimation methods are overly simplistic and prone to causing control system malfunctions. Therefore, a good heat exchange station control method must fully consider both the external environment and the current state of the heat exchange station itself to make reasonable control decisions, avoiding instability and lag.

[0003] In recent years, with the widespread application of deep learning in numerous fields, deep reinforcement learning technology has achieved a series of results in areas such as robot control and games. However, the application of reinforcement learning also has significant limitations. This is because reinforcement learning improves its control strategy by continuously interacting with the environment, which means that many application scenarios lack the necessary training environment and data. For example, in some scenarios, the cost of trial and error can be extremely high, hindering the improvement of the strategy's control effectiveness. Therefore, most reinforcement learning techniques cannot be directly applied to real-world scenarios, and current mainstream reinforcement learning techniques require a continuously interactive simulation environment, which varies from heat exchanger to heat exchanger. Therefore, there is an urgent need for a method that can apply reinforcement learning to the real-world scenarios of heat exchangers, providing a flexible and robust approach for heat exchanger control. Summary of the Invention

[0004] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] To address the problems and shortcomings of existing technologies, this invention aims to provide a heat exchanger station control method based on reinforcement learning. This method primarily achieves its goal through a learning process involving establishing a simulation environment model, training a reinforcement learning control strategy, and transferring the learning to the real environment. This constructs a simulation environment for the heat exchanger station, overcoming the high trial-and-error costs inherent in reinforcement learning techniques in practical applications, thus providing a more flexible and robust control method. A simulation environment model for the heat exchanger station is constructed based on historical data, and a generative adversarial approach (GAP) is used to learn this model, enabling the simulation environment to achieve good simulation results even in situations not present in the historical data. Then, the simulation environment model learned through the GAP is used to train the control strategy using PPO reinforcement learning. The resulting strategy effectively achieves the control objective given by the design reward function, ensuring that the control effect is not lagging, thereby solving the problems mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention discloses a heat exchange station control method based on reinforcement learning, which mainly includes the following steps:

[0008] Step 1: Determine the control objectives that the heat exchange station needs to achieve. The control objectives include the state S and action a of the heat exchange station.

[0009] Step 2: Obtain and collect historical data executed by the heat exchange station;

[0010] Step 3: Based on the historical data of the heat exchange station, a simulation environment model of the heat exchange station is output using generative adversarial learning;

[0011] Step 4: Set different reward functions according to the control objectives of the heat exchange station;

[0012] Step 5: Train the PPO agent using PPO reinforcement learning technology based on the simulation environment model and reward function;

[0013] Step 6: Utilize the trained PPO agent to perform real-time control of the heat exchange station.

[0014] Furthermore, the state S of the heat exchange station mentioned in step 1 is represented as follows:

[0015] S=<temp1,temp2,temp3,temp4,p1,p2,p3,flow>,

[0016] Where temp1 represents the outside temperature, temp2 represents the primary network supply water temperature, temp3 represents the secondary network return water temperature, temp4 represents the secondary network outlet water temperature, p1 represents the pressure after primary network descaling, p2 represents the pressure after secondary network descaling, p3 represents the pressure at the secondary network outlet, and flow represents the secondary network return water flow rate.

[0017] Furthermore, the action 'a' of the heat exchange station mentioned in step 1 is represented as follows:

[0018] a = <v,pump>

[0019] Where v represents the set percentage of the regulating valve opening, and pump represents the set value of the circulating water pump speed.

[0020] Furthermore, in step 2, historical data executed by the heat exchange station is acquired and collected, specifically including the following steps:

[0021] Step 2.1: Collect historical data on the state S and action a of the heat exchange station at various times;

[0022] Step 2.2 involves performing simple processing on the collected historical data;

[0023] Step 2.3: Arrange the historical data in chronological order.

[0024] Furthermore, in step 3, generative adversarial learning is used based on the historical data of the heat exchange station to output a simulation environment model of the heat exchange station, specifically including the following steps:

[0025] Step 3.1: Define a generator and a discriminator respectively;

[0026] Step 3.2, change the current state S of the heat exchange station. t and action a t The generator outputs the predicted state for the next time step.

[0027] Step 3.3, predict the state for the next time step. The input is fed into the discriminator to calculate the reward signal. Then it is passed to the generator;

[0028] Step 3.4, based on the reward signal The generator is updated and trained using the PPO reinforcement learning method;

[0029] Step 3.5: Update the discriminator using gradient descent based on the binary classification loss function;

[0030] Step 3.6, until the training is completed, output the simulation environment model of the heat exchange station.

[0031] If traditional adversarial learning methods are used, the generator does not employ reinforcement learning. The generator only considers the effect of a single step, not the effect of the entire trajectory, resulting in compound errors. In step five, when reinforcement learning is used to learn the control strategy, the prediction of the environment in the first few steps is relatively accurate during the interaction between the strategy and the environment. However, as the simulation environment continues to calculate, the accumulated errors become increasingly large, making the simulation environment's prediction of the next environmental change inaccurate. This affects the reinforcement learning training process. In the heat exchange station scenario, if a generator using traditional generative adversarial learning is used as the simulation environment for the heat exchange station, the compound errors will cause the predicted state value S = <temp1,temp2,temp3,temp4,p1,p2,p3,flow> to differ significantly from the true value when training the reinforcement learning strategy. This results in poor performance of the heat exchange station control strategy learned through reinforcement learning. This problem can be effectively solved by using the reinforcement learning technique of the present invention to train the generator. Compared with the method of training the generator using supervised learning, the present invention considers long-term benefits in the learning process, rather than just single-step benefits. In step 3.4, the generator is trained using a reinforcement learning method instead of a supervised learning method, utilizing reward signals. The generator learns by maximizing the gains on the overall trajectory of each heat exchange station, rather than the gains from each individual step of the heat exchange station. This allows the generator to fully consider performance on the entire trajectory, rather than performance on a single step, thus solving the problem of compound errors. As a result, when training the control strategy using reinforcement learning, the state values ​​predicted by the simulation environment are not much different from the actual values, and the trained control strategy is closer to the real control strategy of the heat exchange station.

[0032] Furthermore, in step 4, different reward functions are set according to the control objectives of the heat exchange station to adapt to different scenarios, specifically including the following:

[0033] If the temperature of the secondary network return water is controlled within a certain target temperature range, the reward function can be set as: R(S) t )=|temp target -temp3|, where temp target Here, temp3 represents the target temperature, and temp3 is the current secondary network return water temperature. If energy consumption is taken as the control objective, the reward function can be set as: R(S t The reward function R(S) = |water + power|, where water represents the water cost of the current heat exchange station and power represents the electricity cost of the current heat exchange station. If there are multiple control objectives, the reward function R(S) can be defined by weighted summation. t )=ω1R1+ω2R2+...+ω n R n .

[0034] Furthermore, in step 5, PPO reinforcement learning is used to train the PPO agent based on the simulation environment model and reward function. The PPO agent adopts the Actor-Critic framework, and the specific training steps are as follows:

[0035] Step 5.1, initialize the Actor and Critic networks;

[0036] Step 5.2, in the Actor network, based on the current state S of the heat exchange station t Calculate the action a currently being performed by the heat exchange station. t ;

[0037] Step 5.3: Calculate the state S of the heat exchange station at the next moment based on the simulation environment model. t+1 ;

[0038] Step 5.4: Calculate the reward signal R based on the reward function;

[0039] Step 5.5: Save and update the actions from steps 5.2 to 5.4 to the Actor and Critic networks;

[0040] Step 5.6: When the Actor and Critic networks reach the training count, complete the training and output the PPO agent control strategy.

[0041] Furthermore, the simulation environment model obtained in step 3 can also be used for early warning, specifically including the following steps:

[0042] Step 7.1: Calculate the predicted value of the heat exchange station at the next moment for the simulation environment model;

[0043] Step 7.2: Read the current sensor data of the heat exchange station to obtain the true value at the next moment.

[0044] Step 7.3: Compare the predicted value with the actual value. If the difference between them exceeds the set value, repeat steps 3 to 5 to retrain the control strategy and simulation environment model of the PPO agent.

[0045] Furthermore, the loss function for PPO reinforcement learning is expressed as follows:

[0046] L CLIP (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)A t )],

[0047] Where θ represents the parameters of the PPO agent neural network, and r t (θ) is the ratio of the probability that the PPO agent selects the current action to the probability that the PPO agent selects this action when this set of data records is available. t It is the dominance function.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] This invention provides a heat exchanger station control method based on reinforcement learning, which can flexibly model the factors to be considered as inputs to the control strategy, making the factors considered in the control strategy more flexible and diverse. This invention mainly consists of two parts: the first part is to learn the simulation environment model of the heat exchanger station based on a generative adversarial approach (GAP); the second part uses PPO reinforcement learning technology to train the control strategy of the heat exchanger station, thereby obtaining a better heat exchanger station control method. In learning the simulation environment model of the heat exchanger station, the historical data of the heat exchanger station is used to construct the simulation environment model. In order to apply the GAP method used in this invention to the heat exchanger station scenario, this invention modifies the GAP method by defining the generator input as the current state S of the heat exchanger station. t And action a, the output is the state at the next moment: S t+1 =G(S t ,a t In the generator's learning process, the generator is treated as an agent in reinforcement learning, and the generator G(S) is... t ,a t The predicted state S of the heat exchange station at the next moment. t+1 As the action taken at time t during reinforcement learning, the action S is executed. t+1 Afterwards, the agent receives the reward signal log2(D(S)). t+1 Using this reward signal, a reinforcement learning algorithm is employed to update the generator. The generator can generate states beyond historical data, allowing the simulation environment to achieve good simulation results under conditions not present in historical data. After obtaining the simulation environment model, PPO reinforcement learning technology is used to train the control strategy for the heat exchange station. Because the training process of PPO reinforcement learning is relatively stable, has low variance, and is insensitive to parameter settings, it can learn a good control method for the heat exchange station. The learned control method achieves more stable temperature control and eliminates lag compared to manual control and PID control methods. Furthermore, due to the flexible setting of the reward function, objectives can be set during the learning process of the control agent to achieve effective control of the heat exchange station or energy conservation. Attached Figure Description

[0050] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application.

[0051] In the attached diagram:

[0052] Figure 1 This is a flowchart illustrating the overall steps in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall process structure in an embodiment of the present invention;

[0054] Figure 3 : This is a flowchart illustrating the steps of learning the simulation environment model in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the simulation environment model learning process in an embodiment of the present invention.

[0056] Figure 5 This is a flowchart illustrating the steps involved in finding the PPO intelligent agent in this embodiment of the invention.

[0057] Figure 6 This is a schematic diagram of the PPO agent training process in an embodiment of the present invention. Detailed Implementation

[0058] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0059] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0060] This invention discloses a heat exchange station control method based on reinforcement learning. The following will describe this disclosure in detail with reference to the accompanying drawings and embodiments.

[0061] Reference Figures 1 to 2 As shown, the main steps include the following:

[0062] Step 1: Determine the control objectives that the heat exchange station needs to achieve. The control objectives include the state S and action a of the heat exchange station.

[0063] Step 2: Obtain and collect historical data executed by the heat exchange station;

[0064] Step 3: Based on the historical data of the heat exchange station, a simulation environment model of the heat exchange station is output using generative adversarial learning.

[0065] Step 4: Set different reward functions according to the control objectives of the heat exchange station;

[0066] Step 5: Train the PPO agent using PPO reinforcement learning techniques based on the simulation environment model and reward function;

[0067] Step 6: Utilize the trained PPO agent to perform real-time control of the heat exchange station.

[0068] Specifically, the first step is to determine the control objectives that the heat exchange station needs to achieve, which include the station's state S and actions a. Historical data on the heat exchange station's execution is acquired; this historical data reflects the control objectives set at each previous time point, i.e., the station's state S and actions a at each previous time point. Based on this historical data, generative adversarial learning is employed to output a simulation environment model S of the heat exchange station. t+1 =φ(S t ,a t ). Among them, S t This represents the state of the heat exchange station at time t, while a t This represents the action of the heat exchange station at time t. Then, different reward functions are set according to the control objectives of the heat exchange station, and these reward functions are used as reward signals in the PPO reinforcement learning process. The reward function is determined according to specific needs to be applicable to different scenarios. Next, based on the simulation environment model φ(S) of the heat exchange station... t ,a t The PPO agent is trained using PPO reinforcement learning and the reward function to improve its control capability in the heat exchange station in the simulation environment. The trained PPO agent can then be used as the control strategy for the heat exchange station to perform real-time control.

[0069] Furthermore, the state S of the heat exchange station in step 1 includes both the external state and the internal state of the heat exchange station, and is defined as follows:

[0070] S=<temp1,temp2,temp3,temp4,p1,p2,p3,flow>,

[0071] Where temp1 represents the ambient temperature, temp2 represents the primary network supply water temperature, temp3 represents the secondary network return water temperature, temp4 represents the secondary network outlet water temperature, p1 represents the pressure after primary network sludge removal, p2 represents the pressure after secondary network sludge removal, p3 represents the pressure at the secondary network outlet pipe, and flow represents the secondary network return water flow rate. That is, the ambient temperature temp1 represents the external state of the heat exchange station, while the other variables represent factors affecting the internal state of the heat exchange station. The action 'a' of the heat exchange station is defined as...

[0072] a = <v,pump>

[0073] Where v represents the set percentage of the regulating valve opening, and pump represents the set value of the circulating water pump speed.

[0074] Specifically, in step 2, historical data executed by the heat exchange station is acquired and collected, including the following steps:

[0075] Step 2.1: Collect historical data on the state S and action a of the heat exchange station at various times;

[0076] Step 2.2 involves performing simple processing on the collected historical data;

[0077] Step 2.3: Arrange the historical data in chronological order.

[0078] Specifically, to construct an environment suitable for training reinforcement learning techniques, a simulation model of the heat exchange station needs to be built using historical data. When the historical data coverage is comprehensive, supervised learning can be used to train this simulation model. However, when the historical data coverage is incomplete, the simulation model learned through supervised learning cannot generalize well to situations not present in the data. Since reinforcement learning techniques continuously explore during training, they may encounter states not found in the historical data. Therefore, the policy learned in the simulation environment obtained through supervised learning is incomplete. Thus, we use historical data from the heat exchange station to construct the simulation environment model, enabling the simulation environment to achieve good simulation results for states not present in the historical data.

[0079] Reference Figure 3 and Figure 4 As shown, in step 3, generative adversarial learning is used based on the historical data of the heat exchange station to output a simulation environment model of the heat exchange station. This specifically includes the following steps:

[0080] Step 3.1: Define a generator and a discriminator respectively;

[0081] Step 3.2, change the current state S of the heat exchange station. t and action a t The input is given to the generator, which outputs the predicted state for the next time step.

[0082] Step 3.3: Predict the state for the next time step. The input is fed into the discriminator to calculate the reward signal. Then pass it to the generator;

[0083] Step 3.4, based on the reward signal The PPO reinforcement learning method is used to update the training generator;

[0084] Step 3.5: Update the trained discriminator using gradient descent based on the binary classification loss function;

[0085] Step 3.6 continues until the training is complete, at which point the simulation environment model of the heat exchange station is output.

[0086] Specifically, to address the shortcomings of supervised learning in learning simulation environments, this invention employs a generative adversarial approach to learn the simulation environment model. When using a generative adversarial learning structure to learn the simulation environment model, the generator can generate states beyond historical data during training, allowing the simulation environment model to achieve good simulation results in states not present in historical data. Therefore, using the simulation environment model learned through the generative adversarial structure to train a policy using reinforcement learning techniques enables the resulting policy to adapt well to real-world situations.

[0087] First, a generator and a discriminator need to be defined separately; that is, the simulation environment is learned through adversarial training of the generator and discriminator. Defining the generator is equivalent to defining a neural network φ(S,a). Assume that the state of the heat exchanger at a certain moment is S. t The action taken at this moment is a. t , will S t and a t When input into the generator, it is represented as φ(S) t ,a t In this process, the calculated result is the predicted state of the heat exchange station at the next moment. This can be represented as The discriminator is a classifier, denoted as D(S), which can be expressed as... Where S→{0,1}. The judge is used to determine whether the input has the same distribution as the original dataset. If the distribution is the same, the output is 1; if the distribution is different, the output is 0.

[0088] The predicted state of the heat exchange station at the next moment. The input is fed into the discriminator to calculate the reward signal. Passed to the generator, it is represented as Since the generator is updated using the PPO reinforcement learning algorithm, we will use the reward signal calculated in the previous step. The discriminator is then incorporated into the PPO reinforcement learning process. Next, the training process based on the discriminator is a supervised learning process, using a standard binary classification loss function for updating. That is, the discriminator updates its network parameters using gradient descent based on the binary classification loss function. Training ends when the termination condition is met. Typically, the number of training iterations is used as the termination condition, ultimately resulting in a simulation environment model of a heat exchange station. After training, the discriminator can distinguish whether the distribution of the input data matches the original dataset, and the defined generator can be directly used as the simulation environment. Throughout the training process, the generator and discriminator improve synchronously. Ultimately, the generator can well fit the dataset distribution shifts, and a simulation environment for a heat exchange station can be constructed using the generator.

[0089] In step 4, different reward functions are set according to the control objectives of the heat exchange station to adapt to different scenarios. The specific settings of the reward functions are as follows:

[0090] To control the secondary network return water temperature around a certain target temperature, the reward function can be set as follows:

[0091] R(S t )=|temp target -temp3|

[0092] Where, temp target tem3 represents the target temperature, while tem3 represents the current secondary network return water temperature.

[0093] If energy consumption is to be used as the control objective, the reward function can be set to...

[0094] R(S t ) = |water + power|,

[0095] Where water represents the water cost currently consumed by the heat exchange station, and power represents the electricity cost currently consumed by the heat exchange station.

[0096] When there are multiple control objectives, the reward function can be defined by weighted summation.

[0097] R(S t )=ω1R1+ω2R2+...+ω n R n .

[0098] After obtaining a simulation model, this invention uses PPO reinforcement learning technology to train the control strategy for the heat exchange station. Because the training process of PPO reinforcement learning is relatively stable, has low variance, and is insensitive to parameter settings, it can learn a good control method for the heat exchange station. Furthermore, the learned control method is more stable than manual control and PID control methods and does not exhibit lag. (Refer to...) Figure 5 and Figure 6 As shown, in step 5, PPO reinforcement learning is used to train the PPO agent based on the simulation environment model and reward function. The PPO agent adopts the Actor-Critic framework, and the specific training steps are as follows:

[0099] Step 5.1, initialize the Actor and Critic networks;

[0100] Step 5.2, in the Actor network, based on the current state S of the heat exchange station t Calculate the action a currently being performed by the heat exchange station. t ;

[0101] Step 5.3: Calculate the state S of the heat exchange station at the next moment based on the simulation environment model. t+1 ;

[0102] Step 5.4: Calculate the reward signal R based on the reward function;

[0103] Step 5.5: Save and update the actions from steps 5.2 to 5.4 to the Actor and Critic networks;

[0104] Step 5.6: When the Actor and Critic networks reach the training count, complete the training and output the PPO agent control strategy.

[0105] Specifically, we will use the Actor-Critic framework for the PPO reinforcement learning method. At the beginning of the process, the Actor network and Critic network are initialized. The Actor network is configured based on the current state S. t Calculate the action a to be executed t The Critic network is based on action a t The feedback after execution evaluates the actions performed by the Actor network. The Actor network evaluates the actions based on the current state S. t Calculate the currently executing action a t That is, a t =π(S) t Based on the simulation environment model obtained in step 3, the state of the heat exchange station at the next moment is represented as S. t+1 =φ(S t ,a t According to the reward function R(S) in step 4t ) Calculate the reward signal R. Then save the actions in steps 5.2 to 5.4 as <S t , a t , R, S t+1 , ρ>, where ρ is the probability that the current Actor selects the currently executed action. Then use this saved action to update the Actor network and the Critic network. When the Actor network and the Critic network reach the number of training times, the training ends. Otherwise, the Actor network continues to execute actions in the new state and repeats this operation.

[0106] Furthermore, the loss function of PPO reinforcement learning is expressed as

[0107] L CLIP (θ) = E t , min(r t (∈) A t , clip(r t (θ), 1 - ∈, 1 + ∈) A t ) -

[0108] where θ is the parameter of the PPO agent neural network. r t (θ) is a correction term, which is the ratio of the probability that the current PPO agent selects the current action to the probability that the PPO agent selects this action when this set of data is recorded. clip(a, b, c) is a function, that is, when the value of a is between b and c, the output of the function is a; if a > b, the output of the function is b; if a < c, the output of the function is c. The loss function of PPO reinforcement learning is the minimum loss function for updating the Actor network. The role of this function is to limit the influence of r t (θ) on the update within a reasonable range. A t is the advantage function. A t is defined as A t = R(S t ) + γV(S t+1 ) - V(S t ), and V(S t ) is a value function defined during the PPO reinforcement learning training process, that is, the Critic network. The minimum loss function for updating the Critic network is expressed as The loss function of this PPO reinforcement learning can ensure that in each update of the PPO method, the change in the neural network parameters is within a reasonable range and will not change too much.

[0109] The simulation environment model obtained in step 3 can also be used for early warning, which specifically includes the following steps:

[0110] Step 7.1: Calculate the predicted value of the heat exchange station at the next moment for the simulation environment model;

[0111] Step 7.2: Read the current sensor data of the heat exchange station to obtain the true value at the next moment.

[0112] Step 7.3: Compare the predicted value with the actual value. If the difference between them exceeds the set value, repeat steps 3 to 5 to retrain the control strategy and simulation environment model of the PPO agent.

[0113] Specifically, in step 3, we obtain a simulation environment model to predict the state of the heat exchange station at the next moment. In the actual control process, each step predicts the state value of the next step, thus obtaining the predicted value y = φ(S) at the next moment. t ,a t Then, read the sensor data from the current heat exchange station to obtain the actual value at the next moment, and compare the predicted value at the next moment with the actual value obtained at the next moment. If the difference between the two is too large, i.e., |yS t+1 When |>ε, it indicates a deviation between the simulation environment and the real environment, requiring retraining of the simulation model and the strategy model. Therefore, repeat steps 3 through 5 to ultimately obtain a control scheme adapted to the new environment.

[0114] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A heat exchange station control method based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Determine the control objectives that the heat exchange station needs to achieve. The control objectives include the status of the heat exchange station. and actions ; Step 2: Obtain and collect historical data executed by the heat exchange station; Step 3: Based on the historical data of the heat exchange station, a simulation environment model of the heat exchange station is output using generative adversarial learning; Step 4: Set different reward functions according to the control objectives of the heat exchange station; Step 5: Train the PPO agent using PPO reinforcement learning technology based on the simulation environment model and reward function; Step 6: Utilize the trained PPO agent to perform real-time control of the heat exchange station; In step 3, based on the historical data of the heat exchange station, a simulation environment model of the heat exchange station is output using generative adversarial learning, specifically including the following steps: Step 3.1: Define a generator and a discriminator respectively; Step 3.2: Change the current status of the heat exchange station. and actions The generator outputs the predicted state for the next time step. ; Step 3.3, predict the state for the next time step. The input is fed into the discriminator to calculate the reward signal. Then it is passed to the generator; Step 3.4, based on the reward signal The generator is updated and trained using the PPO reinforcement learning method; Step 3.5: Update the discriminator using gradient descent based on the binary classification loss function; Step 3.6, until the training is completed, output the simulation environment model of the heat exchange station; In step 4, different reward functions are set according to the control objectives of the heat exchange station to adapt to different scenarios, including the following: If the temperature of the secondary network return water is controlled within a certain target temperature range, the reward function can be set as follows: ， in, Represents the target temperature. Let be the current secondary network return water temperature; if energy consumption is taken as the control objective, the reward function can be set as: in, This represents the water cost currently consumed by the heat exchange station. This represents the electricity cost currently consumed by the heat exchange station; if there are multiple control objectives, a weighted summation method can be used to define the reward function. ; In step 5, PPO reinforcement learning is used to train the PPO agent based on the simulation environment model and reward function. The PPO agent adopts the Actor-Critic framework, and the specific training steps are as follows: Step 5.1, initialize the Actor and Critic networks; Step 5.2, in the Actor network, based on the current state of the heat exchange station Calculate the action currently being performed by the heat exchange station. ; Step 5.3: Calculate the state of the heat exchange station at the next moment based on the simulation environment model. ; Step 5.4: Calculate the reward signal based on the reward function. ; Step 5.5: Save and update the actions from steps 5.2 to 5.4 to the Actor and Critic networks; Step 5.6: When the Actor and Critic networks reach the training count, complete the training and output the PPO agent control strategy.

2. The heat exchange station control method based on reinforcement learning according to claim 1, characterized in that: The state of the heat exchange station described in step 1 Represented as, in, Represents the external temperature. Represents the primary water supply temperature. This represents the return water temperature of the secondary network. This represents the outlet water temperature of the secondary network. This represents the pressure after one round of net cleaning. This represents the pressure after secondary network decontamination. This represents the pressure at the outlet of the secondary network water pipe. This represents the return water flow rate of the secondary network.

3. The heat exchange station control method based on reinforcement learning according to claim 2, characterized in that: The operation of the heat exchange station described in step 1 Represented as, in, This represents the percentage of the control valve opening setting. This represents the setpoint for the circulating water pump speed.

4. The heat exchange station control method based on reinforcement learning according to claim 3, characterized in that, Step 2 involves acquiring and collecting historical data from the heat exchange station, specifically including the following steps: Step 2.1: Collect the status of the heat exchange station at various times. and actions Historical data; Step 2.2 involves performing simple processing on the collected historical data; Step 2.3: Arrange the historical data in chronological order.

5. The heat exchange station control method based on reinforcement learning according to claim 1, characterized in that, The simulation environment model obtained in step 3 can also be used for early warning, specifically including the following steps: Step 7.1: Calculate the predicted value of the heat exchange station at the next moment for the simulation environment model; Step 7.2: Read the current sensor data of the heat exchange station to obtain the true value at the next moment. Step 7.3: Compare the predicted value with the actual value. If the difference between them exceeds the set value, repeat steps 3 to 5 to retrain the control strategy and simulation environment model of the PPO agent.

6. The heat exchange station control method based on reinforcement learning according to claim 1, characterized in that: The loss function for PPO reinforcement learning is expressed as follows: in, These are the parameters of the PPO agent neural network. It is the ratio of the probability that the PPO agent will choose the current action to the probability that the PPO agent will choose this action when this set of data records is available. It is the dominance function.

Images