Heat exchanger real-time control method and system, electronic device, and storage medium

Abstract

The application provides a heat exchanger real-time control method and system, an electronic device and a storage medium, and belongs to the technical field of intelligent control. The method comprises the following steps: acquiring a wort real-time temperature value and a wort target temperature value at a current time; inputting the wort real-time temperature value and the wort target temperature value into a PID controller to obtain a control amount; performing real-time control on the heat exchanger based on the control amount; and wherein the control parameters of the PID controller are determined by a reinforcement learning network that has been trained. The heat exchanger real-time control method and system, the electronic device and the storage medium provided by the application can improve the control accuracy of the heat exchanger.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and more specifically, relates to a real-time control method and system for heat exchangers, electronic devices, and storage media. Background Technology

[0002] In beer brewing, wort is one of the key media requiring careful processing, and its temperature needs to be precisely controlled multiple times. The heat exchanger is the core equipment for achieving this temperature control. Current technology typically uses a PID controller to control the heat exchanger, but adjusting its control parameters relies heavily on manual experience, resulting in low temperature control accuracy, which negatively impacts wort quality and brewing efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a real-time control method and system for heat exchangers, electronic equipment, and storage medium to improve the control accuracy of heat exchangers.

[0004] A first aspect of this application provides a real-time control method for a heat exchanger, comprising:

[0005] Obtain the current real-time temperature value of the wort and the target temperature value of the wort;

[0006] The real-time temperature value and target temperature value of the wort are input into the PID controller to obtain the control quantity;

[0007] Real-time control of the heat exchanger is performed based on the control quantity;

[0008] The control parameters of the PID controller are determined through a trained reinforcement learning network, which includes an action network and a value network. The loss function of the reinforcement learning network is determined in the following ways:

[0009] The instantaneous reward function is determined based on the first difference between the real-time temperature value of the wort and the target temperature value of the wort at the first moment, and the second difference between the control quantity output by the PID controller at the second moment and the control quantity output at the current moment; the second moment is the moment before the first moment.

[0010] The overall reward function for a set time period is determined based on the instant reward.

[0011] The advantage estimate is determined based on the overall reward function and the output data of the value network.

[0012] The shearing loss function is calculated based on the immediate reward function, the overall reward function, and the advantage estimate.

[0013] The entropy regularization term is determined based on the probability distribution of each action output by the action network.

[0014] The loss function of the reinforcement learning network is determined based on the shearing loss function, the loss function of the value network, and the entropy regularization term.

[0015] A second aspect of this application provides a real-time control system for a heat exchanger, comprising:

[0016] The data acquisition module is used to acquire the real-time temperature value of the wort and the target temperature value of the wort at the current moment.

[0017] The PID control module is used to input the real-time temperature value and the target temperature value of the wort into the PID controller to obtain the control quantity;

[0018] Temperature control module, used for real-time control of heat exchanger based on the control quantity;

[0019] The control parameters of the PID controller are determined through a trained reinforcement learning network, which includes an action network and a value network. The loss function of the reinforcement learning network is determined in the following ways:

[0020] The instantaneous reward function is determined based on the first difference between the current real-time temperature value of wort and the target temperature value of wort, and the second difference between the control quantity output by the PID controller at the previous moment and the control quantity output at the current moment.

[0021] The overall reward function for a set time period is determined based on the instant reward.

[0022] The advantage estimate is determined based on the overall reward function and the output data of the value network.

[0023] The shearing loss function is calculated based on the immediate reward function, the overall reward function, and the advantage estimate.

[0024] The entropy regularization term is determined based on the probability distribution of each action output by the action network.

[0025] The loss function of the reinforcement learning network is determined based on the shearing loss function, the loss function of the value network, and the entropy regularization term.

[0026] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described real-time control method for a heat exchanger.

[0027] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described real-time control method for a heat exchanger.

[0028] The beneficial effects of the real-time control method and system for heat exchangers, electronic equipment, and storage medium provided in this application are as follows:

[0029] This application's embodiments dynamically adjust the control parameters of the PID controller in real time based on a reinforcement learning network, providing a faster fitting speed while reducing frequent and drastic fluctuations and lowering machine wear. Furthermore, the reinforcement learning neural network of this invention is based on a PID model, resulting in more stable control performance. Compared to black-box pure RL solutions, the output of the reinforcement neural network can be interpreted. Moreover, pure RL solutions rely on larger and more complex neural network structures to capture nonlinear relationships in heat exchange systems, leading to low training efficiency and high computational complexity. In contrast, the PID-based neural network has a simple structure, millisecond-level computation time, and can respond quickly. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic flowchart of a real-time control method for a heat exchanger provided in an embodiment of this application;

[0032] Figure 2 An overall architecture diagram of a heat exchanger control system provided in an embodiment of this application;

[0033] Figure 3 A decay comparison diagram corresponding to different discount factors provided in an embodiment of this application;

[0034] Figure 4 A structural block diagram of a real-time control system for a heat exchanger provided in an embodiment of this application;

[0035] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0038] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a real-time control method for a heat exchanger provided in an embodiment of this application. The method can be executed by an electronic device and may include:

[0039] S101: Obtain the current real-time temperature value of the wort and the target temperature value of the wort.

[0040] In this embodiment, the real-time temperature value of the wort at the current moment can be obtained in real time by a temperature sensor. The target temperature value of the wort at the current moment can be stored in the electronic device in advance or can be manually input.

[0041] S102: Input the real-time temperature value and target temperature value of the wort into the PID controller to obtain the control quantity; wherein, the control parameters of the PID controller are determined through a trained reinforcement learning network, which includes an action network and a value network. The loss function of the reinforcement learning network is determined in the following ways:

[0042] The instantaneous reward function is determined based on the first difference between the real-time temperature value of the wort and the target temperature value of the wort at the first moment, and the second difference between the control quantity output by the PID controller at the second moment and the control quantity output at the current moment; the second moment is the moment before the first moment.

[0043] The overall reward function for a given time period is determined based on immediate rewards.

[0044] The advantage estimate is determined based on the output data of the overall reward function and the value network.

[0045] The shearing loss function is calculated based on the immediate reward function, the overall reward function, and the advantage estimate.

[0046] The entropy regularization term is determined based on the probability distribution of each action output by the action network;

[0047] The loss function of a reinforcement learning network is determined based on the shearing loss function, the loss function of the value network, and the entropy regularization term.

[0048] Please refer to Figure 2 In this embodiment, a PID controller can be used for real-time control of the heat exchanger. Specifically, the real-time temperature value PV(t) and the target temperature value SP(t) of the wort can be input into the PID controller. The PID controller calculates the first difference e(t) between the real-time temperature value PV(t) and the target temperature value SP(t), and outputs the corresponding control quantity u(t) to control the opening degree of the valve of the heat exchanger or the power of the pump, so that the real-time temperature value PV(t) of the wort reaches the target temperature value SP(t).

[0049] The control parameters of the PID controller are determined through a trained reinforcement learning network. In this embodiment, the reinforcement learning network outputs a reward signal based on the difference SP(t) between the real-time wort temperature PV(t) and the target wort temperature. Subsequent reinforcement learning will adjust its weight parameters based on this reward signal. The immediate reward for reinforcement learning at each time t is defined as:

[0050] ;

[0051] in:

[0052] : Indicates the difference between the heat exchange output temperature and the set temperature at time point t (the first moment);

[0053] : This indicates the difference between the current PID output and the output at the previous time point (second time point) at time t;

[0054] Temperature error weighting coefficient;

[0055] : Control the output difference weighting coefficient;

[0056] This instant reward The larger the error (e(t)) or the more intense the control action ( ), The smaller the value, the higher the penalty. This allows the neural network to naturally learn to minimize tracking error and control its actions during reinforcement learning. Furthermore, an overall reward signal over a certain period can be formed based on the immediate reward at each moment, which can then be used for subsequent calculations.

[0057] In actual training, this embodiment adjusts the model according to its actual performance. and This is to balance temperature error with the degree of control.

[0058] The loss function of a reinforcement learning network is calculated through the following steps:

[0059] 1. Overall Algorithm Framework

[0060] a. Definition of Environment and Strategy

[0061] i. State space: The 128-bit feature vector output by the encoder;

[0062] ii. Action Space: The mean and standard deviation of the output of the policy network (the first two regression heads of the decoder) are parameterized Gaussian distributions and sampled.

[0063] iii. Action execution: Input to the PID controller to obtain the control signal The environment returns to the next state. With instant rewards ;

[0064] b. Reinforcement learning objective: Maximize the total discounted reward (or equivalently minimize the negative reward), using the Proximal Policy Optimization (PPO) algorithm to adjust the weight parameters of the Actor network. Weight parameters of the value network (Critic network) Perform iterative optimization.

[0065] 2. Action sampling and logarithmic probability calculation

[0066] a. Obtain parameters from the Actor network:

[0067] ;

[0068] in, This represents the mean of the action. The logarithm represents the standard deviation of the action. It should be noted that in this embodiment, all logs are based on the natural logarithm e.

[0069] b. Calculate the standard deviation:

[0070] ;

[0071] c. Sampling action:

[0072] ;

[0073] ;

[0074] in, It follows a standard multivariate normal Gaussian distribution. For one The identity matrix.

[0075] d. Calculate the logarithmic probability:

[0076] ;

[0077] The above formula yields a scalar quantity, which is used for subsequent ratio calculations. And the calculation of the loss function.

[0078] 3. Rollout:

[0079] Obtain the feature vector at the current time point from the encoder. .according to Use the decoder to get Calculate sampling and calculate Calculated using the Critic network .

[0080] Will Input to the PID controller, calculate The input environment and reward system are obtained and ,Record Collect trajectory data of the complete length TT and put it into the experience buffer.

[0081] 4. Calculate the overall reward function With advantage estimation :

[0082] First, based on the instantaneous reward at each moment, an overall reward signal can be formed over a certain period of time. Specifically, the overall reward function within a set time period can be calculated using the first formula; the first formula is:

[0083] ;

[0084] in, This represents the overall reward function for a given time period, where T represents the given time period. This is a discount factor used to adjust the behavior of reinforcement learning networks, thereby balancing the weights of short-term and long-term reward signals. For example, if it's necessary to adjust the temperature to the target value more quickly, it can be... The value increased by 0.3, therefore, in the early stage This will be more important than what follows. Or perhaps, more stable temperature fluctuations are needed. The value was lowered (0.03) to ensure later... They also have relatively high weights. During encoder pre-training, the discount factor is set to 0, making the reward signal equally important at any given time point, thus ensuring the model conforms to the characteristics of the base model. Only during fine-tuning, the discount factor is adjusted according to the actual scenario requirements to achieve different scenario needs. The decay comparison diagrams corresponding to different discount factors are shown below. Figure 3 As shown.

[0085] Then calculate the advantage estimate using the following formula. :

[0086] ;

[0087] 5. PPO Loss Function

[0088] Calculate for each time step t:

[0089] (1) Importance sampling ratio:

[0090] ;

[0091] (2) Clipped Surrogate:

[0092] ;

[0093] In this embodiment, the core idea of ​​the pruning strategy objective is to allow for small increases or decreases in probability when estimating the improvement of the new strategy using old samples, while preventing extreme single-step changes through truncation (i.e., the effect of min), thereby improving the stability and efficiency of training.

[0094] when When the new strategy is superior to the existing one, the goal is to improve the success rate of continued execution. The action, that is The probability. And if It has exceeded (1+ε) Then hope Restricted to (1+ε) Because it continues to increase This can lead to excessively large single-step changes, resulting in instability. Similarly, when When it is worse than the existing strategy, A value that is too small can also lead to instability, so it is limited to (1-ε). .

[0095] By adding upper and lower bounds to the changes generated in a single step through the shearing strategy objective, stability can be improved.

[0096] (3) Value Network Loss

[0097] ;

[0098] (4) Entropy regularization term (exploration encouraged)

[0099] ;

[0100] In this embodiment, by adding an entropy regularization term, a controllable fluctuation, or entropy, is added to the loss. If the entropy is too small, the policy will tend to a fixed state, that is, a fixed set of actions (i.e.,... This will account for the majority of the probability. If the strategy converges too early to "fixed output individual"... A deterministic strategy (with low entropy) lacks opportunities to try other actions and may miss better solutions. Therefore, this embodiment encourages exploration of different approaches by increasing entropy, that is, increasing the uncertainty of actions. This is to assess the impact on the system, thereby increasing the likelihood of finding a better solution and avoiding convergence to a local optimum.

[0101] (5) Total loss

[0102] ;

[0103] Hyperparameter settings: .

[0104] S103: Real-time control of the heat exchanger based on control variables.

[0105] In this embodiment, the opening degree of the heat exchanger valve or the power of the pump are controlled based on the control quantity u(t) so that the real-time temperature value PV(t) of the wort reaches the target temperature value SP(t) of the wort.

[0106] As can be seen from the above, this embodiment, based on a reinforcement learning network to dynamically adjust the control parameters of the PID controller in real time, can provide a faster fitting speed while reducing frequent and drastic fluctuations and lowering machine wear. Furthermore, the reinforcement learning neural network of this invention, based on a PID model, makes the control effect more stable. Compared to the black-box pure RL scheme, the output of the reinforcement neural network can be interpreted. Moreover, the pure RL scheme relies on a larger and more complex neural network structure to capture the nonlinear relationships in the heat exchange system, resulting in low training efficiency and high computational complexity. In contrast, the PID-based neural network has a simple structure, a computation time in the millisecond range, and can respond quickly.

[0107] In one embodiment of this application, the real-time control method for a heat exchanger further includes:

[0108] A mapping relationship between the control output of the PID controller and the real-time temperature value of the wort is constructed based on historical data of the heat exchanger.

[0109] A simulation environment for a heat exchanger is constructed based on mapping relationships, and a reinforcement learning network is trained based on this simulation environment.

[0110] In this embodiment, the reinforcement learning neural network is first pre-trained in a simulated environment. By applying different simulated environments (set temperature, liquid temperature, heat exchanger efficiency, etc.), the reinforcement learning network is able to output reasonable PID parameters.

[0111] Existing heat exchanger simulations require detailed parameters of the actual heat exchanger to construct a complex physical model for simulation. These simulation methods require many parameters, are computationally complex, and many physical quantities (pressure, flow rate, etc.) are unnecessary in this scenario, consuming unnecessary computational resources. This embodiment establishes a mapping relationship between the control quantities output by the PID controller (including the opening degree of the chilled water valve or the power of the pump) and the target wort temperature by using historical data of the heat exchanger, thus creating a simulation environment that simplifies the complexity of the model.

[0112] In one embodiment of this application, the SOPDT model is used to construct a mapping relationship between the control quantity output by the PID controller and the real-time temperature value of the wort based on historical data of the heat exchanger; the real-time control method for the heat exchanger further includes:

[0113] The attribute parameters of the SOPDT model are fitted using the differential evolution method.

[0114] In this embodiment, the historical operating data of the heat exchanger includes the output wort temperature and the corresponding chilled water valve / pump power. A SOPDT model can be used to construct the mapping relationship between the control output of the PID controller and the real-time wort temperature. The calculation formula for the SOPDT model is as follows:

[0115] ;

[0116] : Process output variable (wheat wort output temperature);

[0117] : Process input variables (cold water valve opening / pump power);

[0118] Process gain;

[0119] Damping coefficient;

[0120] Time constant;

[0121] Dead time;

[0122] In this embodiment, the four parameters of SOPDT can be fitted based on the PV(t) and u(t) data of the actual heat exchanger using the Differential Evolution method.

[0123] In one embodiment of this application, the attribute parameters of the SOPDT model include process gain, damping coefficient, time constant, and dead time; the attribute parameters of the SOPDT model fitted based on the differential evolution method include:

[0124] Calculate the cross-correlation between the control output of the PID controller at the current moment and the wort temperature value at each of the multiple delay times; wherein the wort temperature values ​​at the multiple delay times are the wort temperature values ​​at different delay times after the current moment.

[0125] The delay time with the largest corresponding cross-correlation information is used as the center value of the search range for dead time;

[0126] The search range for dead time is determined based on the center value and the preset interval length;

[0127] Obtain the search ranges corresponding to the process gain, damping coefficient, and time constant, respectively;

[0128] Based on the search ranges corresponding to process gain, damping coefficient, time constant, and dead time, the attribute parameters of the SOPDT model are fitted using the differential evolution method.

[0129] In this embodiment, cross-correlation can be used to quickly estimate... The approximate range is determined in order to set reasonable upper and lower limits (search range) for differential evolution.

[0130] The cross-correlation function measures the similarity between two signals at different time lags. In reality, PV(t) and u(t) are represented as discrete data points, expressed as... If the two sets of sequences have the same length, N, then .

[0131] Based on this, The process for determining the search scope is as follows:

[0132] (1) Data preprocessing:

[0133] For the original input , Normalize the discrete data by removing the mean:

[0134] ;

[0135] ;

[0136] here, and This represents the average value of the data points.

[0137] The calculation method is as follows:

[0138] ;

[0139] ;

[0140] (2) Cross-correlation calculation

[0141] Calculate normalized cross-correlation:

[0142] , ;

[0143] if , ,but Set to 0;

[0144] This refers to the number of data points after which a change in u will be reflected in PV.

[0145] For each value of l, calculate This forms a sequence.

[0146] (3) Peak Finding

[0147] Find the index with the highest cross-correlation:

[0148] ;

[0149] Corresponding time delay estimate:

[0150] ;

[0151] in The sampling interval is the time interval between two sampling points.

[0152] (4) Determine Search scope:

[0153] After estimating the approximate delay through cross-correlation, this value is used as the center of the search range, and then a tolerance bandwidth is added or subtracted. As a differential evolution method The search range for this parameter is:

[0154] ;

[0155] After obtaining After determining the approximate range, we fit all the parameters of SOPDT using differential evolution.

[0156] In one embodiment of this application, the reinforcement learning network adopts an encoder-decoder network structure; the weight parameters of the encoder are initialized in the following manner:

[0157] Add a regression network after the encoder;

[0158] Collect multiple sets of sample data, each set of sample data including the time series data of the first difference and the attribute parameters of the heat exchanger;

[0159] The encoder and regression network are trained based on multiple sets of sample data to initialize the weight parameters of the encoder.

[0160] In this embodiment, an encoder-decoder structure is used to construct the reinforcement learning model architecture. For the encoder part, because temporal information needs to be captured, this embodiment employs a simplified Transformers architecture. This network is trained through a simple regression task, predicting the current process's gain, damping coefficient, time constant, and dead time based on the temporal difference between the process output variables and the set expected variables. The regression task enables the encoder to understand the process characteristics from the input time series.

[0161] Then we use a multilayer perceptron (MLP) decoder to output the process variables, set the expected variables and the timing series of the previous PID output to output the parameters of the next PID controller.

[0162] In this embodiment, the weight parameters of the encoder are initialized through a regression task. Specifically, a simple MLP regression network is added after the encoder to predict the process gain, damping coefficient, time constant, and dead time of the system to which the input time series data belongs, based on the input time series data.

[0163] The weight parameters are initialized through a regression training task. Compared to random initialization and subsequent adjustment of parameters through reinforcement learning, initializing parameters through a regression task and then fine-tuning them in subsequent reinforcement learning yields better training results and stronger training stability.

[0164] For example, a regression network architecture could be: :

[0165] ;

[0166] In this embodiment, the system's operating data is collected by simulating the environment using various different heat exchangers. The encoder and regression network described above are then used to predict the system's four attribute parameters. The training algorithm uses AdamW, which evaluates the difference between the model's output four parameters and the true parameters using the root mean square error (RMSE). The model parameters are then updated via backpropagation based on the error.

[0167] In subsequent training, this embodiment will adjust the weight parameters of both the encoder and decoder simultaneously using a group strategy based on feedback from the reward system. The encoder's learning rate will be artificially lowered, and only the encoder parameters will be fine-tuned, with a focus on training the decoder model parameters.

[0168] In one embodiment of this application, the real-time temperature value of the wort and the target temperature value of the wort are input into a PID controller to obtain a control quantity, including:

[0169] The control quantity output by the PID control is calculated using the second formula; the second formula is:

[0170] ;

[0171] in,

[0172] ;

[0173] ;

[0174] = ;

[0175] This represents the control quantity obtained in the k-th control cycle. The control quantity obtained in the (k-1)th control cycle The control quantity obtained in the (k-2)th control cycle; Represents the derivative filter coefficients. This represents the proportional gain of the PID controller. The integral time represents the proportional gain of the PID controller. The derivative coefficient represents the proportional gain of the PID controller; Indicates the control period. This represents the real-time temperature value of the wort obtained by sampling during the k-th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-1)th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-2)th control cycle.

[0176] In this embodiment, for the heat exchanger control scenario, the general PID controller can be improved. The existing general PID (distance PID) calculation formula is as follows:

[0177] ;

[0178] in,

[0179] : pid output;

[0180] Output deviation;

[0181] Set the target output (Set Point);

[0182] : Actual process output (Process Variable);

[0183] : Controller gain;

[0184] Integral time constant;

[0185] : Derivative time constant;

[0186] In this embodiment, it is considered that if the integral term continues to accumulate even when the valve is already fully open (0%) or fully open (100%), it may lead to the problem of "integral wind-up". This will make the controller sluggish when desaturating, slow down the adjustment, and worsen the system response.

[0187] To address the aforementioned issues, this embodiment transforms the distance PID controller into a speed PID controller through a single integration:

[0188] ;

[0189] The speed PID controller consists of three parts: Proportional: ;

[0190] Integral: ;

[0191] Differential: ;

[0192] In the above formula for calculating speed PDI, since the current output is only related to the current deviation, the integral part will no longer accumulate.

[0193] Furthermore, the continuous PID controller is transformed into a discrete form to facilitate calculation:

[0194] ;

[0195] In the proportional and derivative parts, the error between the previous and subsequent sampling points will be involved:

[0196] ;

[0197] ；

[0198] In the two formulas above, because the controller's set temperature (SP) is involved, a sudden and drastic change in SP will affect the values ​​of these two parts, causing fluctuations and instability in the PID output. Therefore, this embodiment removes SP from these two parts. The modified PID is as follows:

[0199] ;

[0200] In addition, this embodiment also introduces a derivative filter to smooth the output of the PID controller, so as to prevent sudden changes from causing instability and risks to the overall system.

[0201] ;

[0202] Where α is the filter coefficient.

[0203] The final discrete velocity PID with derivative filter is as follows:

[0204] ;

[0205] ;

[0206] ;

[0207] ;

[0208] ;

[0209] ;

[0210] ;

[0211] ;

[0212] ;

[0213] ;

[0214] ;

[0215] The output of the PID controller represents the valve opening (closed - fully open) or the pump power (closed - maximum power). The range is limited to 0-100.

[0216] Corresponding to the real-time control method of the heat exchanger in the above embodiment, Figure 4 This is a structural block diagram of a real-time control system for a heat exchanger provided according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 4 The real-time control system 20 for the heat exchanger includes: a data acquisition module 21, a PID control module 22, and a temperature control module 23.

[0217] Among them, the data acquisition module 21 is used to acquire the real-time temperature value of the wort and the target temperature value of the wort at the current moment;

[0218] PID control module 22 is used to input the real-time temperature value of wort and the target temperature value of wort into the PID controller to obtain the control quantity;

[0219] Temperature control module 23 is used for real-time control of the heat exchanger based on control variables;

[0220] The control parameters of the PID controller are determined through a trained reinforcement learning network, which includes an action network and a value network. The loss function of the reinforcement learning network is determined in the following ways:

[0221] The instantaneous reward function is determined based on the first difference between the real-time temperature value of the wort and the target temperature value of the wort at the first moment, and the second difference between the control quantity output by the PID controller at the second moment and the control quantity output at the current moment; the second moment is the moment before the first moment.

[0222] The overall reward function for a given time period is determined based on immediate rewards.

[0223] The advantage estimate is determined based on the output data of the overall reward function and the value network.

[0224] The shearing loss function is calculated based on the immediate reward function, the overall reward function, and the advantage estimate.

[0225] The entropy regularization term is determined based on the probability distribution of each action output by the action network;

[0226] The loss function of a reinforcement learning network is determined based on the shearing loss function, the loss function of the value network, and the entropy regularization term.

[0227] In one embodiment of this application, the PID control module 22 is specifically used for:

[0228] A mapping relationship between the control output of the PID controller and the real-time temperature value of the wort is constructed based on historical data of the heat exchanger.

[0229] A simulation environment for a heat exchanger is constructed based on mapping relationships, and a reinforcement learning network is trained based on this simulation environment.

[0230] In one embodiment of this application, the SOPDT model is used to construct a mapping relationship between the control quantity output by the PID controller and the real-time temperature value of the wort based on historical data of the heat exchanger; the PID control module 22 is further used for:

[0231] The attribute parameters of the SOPDT model are fitted using the differential evolution method.

[0232] In one embodiment of this application, the attribute parameters of the SOPDT model include process gain, damping coefficient, time constant, and dead time; the PID control module 22 is further used for:

[0233] Calculate the cross-correlation between the control output of the PID controller at the current moment and the wort temperature value at each of the multiple delay times; wherein the wort temperature values ​​at the multiple delay times are the wort temperature values ​​at different delay times after the current moment.

[0234] The delay time with the largest corresponding cross-correlation information is used as the center value of the search range for dead time;

[0235] The search range for dead time is determined based on the center value and the preset interval length;

[0236] Obtain the search ranges corresponding to the process gain, damping coefficient, and time constant, respectively;

[0237] Based on the search ranges corresponding to process gain, damping coefficient, time constant, and dead time, the attribute parameters of the SOPDT model are fitted using the differential evolution method.

[0238] In one embodiment of this application, the reinforcement learning network adopts an encoder-decoder network structure; the PID control module 22 is specifically used for:

[0239] Add a regression network after the encoder;

[0240] Collect multiple sets of sample data, each set of sample data including the time series data of the first difference and the attribute parameters of the heat exchanger;

[0241] The encoder and regression network are trained based on multiple sets of sample data to initialize the weight parameters of the encoder.

[0242] In one embodiment of this application, the PID control module 22 is specifically used for:

[0243] The overall reward function for a given time period is determined by calculating using the first formula; the first formula is:

[0244] ;

[0245] in, The overall reward function for a given time period, where T represents the given time period. This represents the instant reward function.

[0246] In one embodiment of this application, the PID control module 22 is specifically used for:

[0247] The control quantity output by the PID control is calculated using the second formula; the second formula is:

[0248] ;

[0249] in,

[0250] ;

[0251] ;

[0252] = ;

[0253] This represents the control quantity obtained in the k-th control cycle. The control quantity obtained in the (k-1)th control cycle The control quantity obtained in the (k-2)th control cycle; Represents the derivative filter coefficients. This represents the proportional gain of the PID controller. The integral time represents the proportional gain of the PID controller. The derivative coefficient represents the proportional gain of the PID controller; Indicates the control period. This represents the real-time temperature value of the wort obtained by sampling during the k-th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-1)th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-2)th control cycle.

[0254] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 5 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of the data acquisition module 21, PID control module 22, and temperature control module 23 are shown.

[0255] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0256] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0257] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store preset constants such as wort target temperature values.

[0258] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the real-time control method for heat exchangers provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0259] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0260] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0261] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0262] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0263] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connections shown or discussed may be indirect coupling or communication connections through some interfaces or units, or they may be electrical, mechanical, or other forms of connection.

[0264] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0265] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0266] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time control method for a heat exchanger, characterized in that, include: Obtain the current real-time temperature value of the wort and the target temperature value of the wort; The real-time temperature value and target temperature value of the wort are input into the PID controller to obtain the control quantity; Real-time control of the heat exchanger is performed based on the control quantity; The control parameters of the PID controller are determined through a trained reinforcement learning network, which includes an action network and a value network. The loss function of the reinforcement learning network is determined in the following ways: The instantaneous reward function is determined based on the first difference between the real-time temperature value of the wort and the target temperature value of the wort at the first moment, and the second difference between the control quantity output by the PID controller at the second moment and the control quantity output at the current moment; the second moment is the moment before the first moment. The overall reward function within a set time period is determined based on the instant reward function. The advantage estimate is determined based on the overall reward function and the output data of the value network. The shearing loss function is calculated based on the immediate reward function, the overall reward function, and the advantage estimate. The entropy regularization term is determined based on the probability distribution of each action output by the action network. The loss function of the reinforcement learning network is determined based on the shearing loss function, the loss function of the value network, and the entropy regularization term. The real-time temperature value and target temperature value of the wort are input into the PID controller to obtain the control variables, including: The control quantity output by the PID control is calculated using the second formula; the second formula is: in, = This represents the control quantity obtained in the k-th control cycle. The control quantity obtained in the (k-1)th control cycle The control quantity obtained in the (k-2)th control cycle; Represents the derivative filter coefficients. This represents the proportional gain of the PID controller. The integral time represents the proportional gain of the PID controller. The derivative coefficient represents the proportional gain of the PID controller; Indicates the control period. This represents the real-time temperature value of the wort obtained by sampling during the k-th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-1)th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-2)th control cycle.

2. The real-time control method for a heat exchanger as described in claim 1, characterized in that, Also includes: A mapping relationship between the control quantity output by the PID controller and the real-time temperature value of the wort is constructed based on the historical data of the heat exchanger. A simulation environment for the heat exchanger is constructed based on the mapping relationship, and the reinforcement learning network is trained based on the simulation environment.

3. The real-time control method for a heat exchanger as described in claim 2, characterized in that, Using the SOPDT model, a mapping relationship between the control quantity output by the PID controller and the real-time temperature value of the wort is constructed based on the historical data of the heat exchanger; The real-time control method for the heat exchanger also includes: The attribute parameters of the SOPDT model are fitted using the differential evolution method.

4. The real-time control method for a heat exchanger as described in claim 3, characterized in that, The attribute parameters of the SOPDT model include process gain, damping coefficient, time constant, and dead time; the attribute parameters of the SOPDT model fitted by the differential evolution method include: Calculate the cross-correlation between the control quantity output by the PID controller at the current moment and the wort temperature value at each of the multiple delay times; wherein the wort temperature values ​​at the multiple delay times are the wort temperature values ​​at different delay times after the current moment. The delay time with the largest corresponding cross-correlation information is used as the center value of the dead time search range; The search range for the dead time is determined based on the center value and the preset interval length; Obtain the search ranges corresponding to the process gain, the damping coefficient, and the time constant, respectively; Based on the search ranges corresponding to the process gain, the damping coefficient, the time constant, and the dead time, the attribute parameters of the SOPDT model are fitted using the differential evolution method.

5. The real-time control method for a heat exchanger as described in claim 1, characterized in that, The reinforcement learning network adopts an encoder-decoder network structure; the encoder's weight parameters are initialized in the following way: Add a regression network after the encoder; Collect multiple sets of sample data, each set of sample data including the time series data of the first difference and the attribute parameters of the heat exchanger; The encoder and the regression network are trained based on the multiple sets of sample data to initialize the weight parameters of the encoder.

6. The real-time control method for a heat exchanger as described in claim 1, characterized in that, Determining the overall reward function within a set time period based on the instant reward function includes: The overall reward function for a given time period is calculated using a first formula; the first formula is: in, This represents the overall reward function for a given time period, where T represents the given time period. Indicates the discount factor. This represents the instant reward function.

7. A real-time control system for a heat exchanger, characterized in that, include: The data acquisition module is used to acquire the real-time temperature value of the wort and the target temperature value of the wort at the current moment. The PID control module is used to input the real-time temperature value and the target temperature value of the wort into the PID controller to obtain the control quantity; Temperature control module, used for real-time control of heat exchanger based on the control quantity; The control parameters of the PID controller are determined through a trained reinforcement learning network, which includes an action network and a value network. The loss function of the reinforcement learning network is determined in the following ways: The instantaneous reward function is determined based on the first difference between the current real-time temperature value of wort and the target temperature value of wort, and the second difference between the control quantity output by the PID controller at the previous moment and the control quantity output at the current moment. The overall reward function within a set time period is determined based on the instant reward function. The advantage estimate is determined based on the overall reward function and the output data of the value network. The shearing loss function is calculated based on the immediate reward function, the overall reward function, and the advantage estimate. The entropy regularization term is determined based on the probability distribution of each action output by the action network. The loss function of the reinforcement learning network is determined based on the shearing loss function, the loss function of the value network, and the entropy regularization term. The PID control module is specifically used for: The control quantity output by the PID control is calculated using the second formula; the second formula is: in, = This represents the control quantity obtained in the k-th control cycle. The control quantity obtained in the (k-1)th control cycle The control quantity obtained in the (k-2)th control cycle; Represents the derivative filter coefficients. This represents the proportional gain of the PID controller. The integral time represents the proportional gain of the PID controller. The derivative coefficient represents the proportional gain of the PID controller; Indicates the control period. This represents the real-time temperature value of the wort obtained by sampling during the k-th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-1)th control cycle. This represents the real-time temperature value of the wort obtained by sampling during the (k-2)th control cycle.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images