Reinforcement learning control method for organic Rankine cycle based on virtual-real transfer of agent model

Abstract

The present invention discloses a reinforcement learning control method for an organic Rankine cycle based on virtual-real migration of a proxy model, comprising the following steps: (1) in the presence of disturbances in the residual heat operating condition, changing the set values ​​of the organic Rankine cycle system at regular intervals and collecting historically accumulated closed-loop operation data; (2) using the collected data to train a system dynamics model, which is used as a proxy model for the organic Rankine cycle system; (3) using the proxy model to construct a virtual simulation environment for reinforcement learning pre-training, setting multiple set values ​​within a reinforcement learning episode and adding time-varying residual heat disturbances to train a reinforcement learning agent; (4) migrating the structure and parameters of the reinforcement learning agent to an actual organic Rankine cycle control system, performing fine-tuning and continuing training; and (5) using the reinforcement learning agent for process control of the actual organic Rankine cycle system. The present invention can significantly improve training efficiency and effectively solve the problems of slow system response and time-consuming computation.

Description

Technical Field

[0001] The present invention belongs to the field of industrial process control, and in particular relates to an organic Rankine cycle reinforcement learning control method based on virtual-real migration of an agent model. Background Art

[0002] The Organic Rankine Cycle (ORC) is considered an effective low-temperature waste heat recovery technology. It can utilize various low-boiling-point organic compounds as working fluids to recycle a wide range of low-temperature waste heat. Its simple structure and high heat recovery efficiency have made it a research hotspot in low-temperature waste heat recovery technology. Furthermore, the ORC is widely used in geothermal power generation, solar power generation, and biomass power generation.

[0003] For example, the Chinese patent document with publication number CN105626175A discloses an organic Rankine cycle power generation system, including a cycle power generation main loop and an expander expansion gas regulation loop; in the cycle power generation main loop, the working fluid outlet of the high-pressure evaporator is connected to the main gas inlet of the expander, the exhaust port of the expander is connected to the inlet of the condenser, the outlet of the condenser is connected to the inlet of the high-pressure working fluid pump, the outlet of the high-pressure working fluid pump is connected to the working fluid inlet of the high-pressure evaporator, and the output end of the expander is connected to the generator; in the expander expansion gas regulation loop, the inlet of the low-pressure working fluid pump is connected to the outlet of the condenser, the outlet of the low-pressure working fluid pump is connected to the working fluid inlet of the low-pressure evaporator, the working fluid outlet of the low-pressure evaporator is connected to the expansion gas regulation port of the expander, the expansion gas regulation port of the expander is connected to the inlet of the first heat exchange pipeline of the regenerator, and the first heat exchange pipeline outlet of the regenerator is connected to the inlet of the condenser.

[0004] ORC is a multivariable coupled, time-varying, nonlinear system. Fluctuations in waste heat conditions can easily cause the system state to deviate from the set value or even violate constraints. Therefore, a controller is required during cyclic operation to ensure the normal operation of the ORC and meet the constraints. A good control system design not only ensures the stable and reliable operation of the ORC system under varying waste heat conditions and extends the service life of system components, but also improves the waste heat recovery rate and maximizes the utilization of waste heat resources. Therefore, the control system is an important component of an ORC-based low-temperature waste heat recovery system. Achieving a reliable and high-performance control system design plays a vital role in the promotion and application of ORC technology.

[0005] Deep reinforcement learning (DRL) is a highly sought-after technology, with research on DRL surging in both academia and industry. However, DRL's true application in production and its potential to create value for businesses and society still faces significant challenges. First, its implementation remains limited to the realm of games, where trial and error is a given. For real-world industrial processes and autonomous driving, training is prone to safety issues and the high cost of trial and error makes direct training on physical objects impossible. Second, reinforcement learning suffers from low exploration efficiency, often resulting in a significant amount of useless experience, resulting in prolonged training times and even failure to converge. Furthermore, reinforcement learning suffers from poor generalization performance, meaning trained agents can only make correct decisions within a single environment. Even slight changes in the environment can significantly degrade the agent's decision-making performance.

[0006] In the field of control of organic Rankine cycle systems, there is currently no research work on methods for virtual-to-real transfer training of controllers based on agent models and reinforcement learning. Summary of the Invention

[0007] The present invention provides an organic Rankine cycle reinforcement learning control method based on virtual-real migration of a proxy model. For an organic Rankine cycle system with slow response speed and inability to perform reinforcement learning training directly on the physical object, the proxy model pre-training method can be used to greatly improve the training efficiency while ensuring the safety of the actual system, and quickly obtain a reinforcement learning controller with excellent control effect.

[0008] An organic Rankine cycle reinforcement learning control method based on virtual-real migration of an agent model includes:

[0009] (1) When there is a disturbance in the waste heat operating conditions, the set value of the organic Rankine cycle system is changed at regular intervals to collect historical accumulated closed-loop operation data;

[0010] (2) Using the collected closed-loop operation data to train a system dynamics model based on neural network prediction, and using it as a proxy model of the organic Rankine cycle system, verify the accuracy and consistency of the proxy model and the actual system under closed-loop control; if the proxy model meets the requirements, perform step (3); otherwise, repeat step (2);

[0011] (3) Using the agent model to construct a virtual simulation environment for reinforcement learning pre-training, multiple set values ​​are set in a reinforcement learning episode and time-varying residual heat disturbance is added to train the reinforcement learning agent to control the agent model; if the training effect meets the preset conditions, step (4) is executed, otherwise step (3) is repeated;

[0012] (4) Migrate the structure and parameters of the reinforcement learning agent pre-trained in the virtual simulation environment to the actual organic Rankine cycle control system, and perform fine-tuning and continue training until the ideal control effect is achieved on the actual system;

[0013] (5) The trained reinforcement learning agent is used for process control of actual organic Rankine cycle system.

[0014] Furthermore, in step (1), the closed-loop operation data accumulated historically is collected as follows:

[0015] Collect several system state quantities that change with time under various operating conditions, including: evaporator outlet pressure P e , evaporator outlet temperature T oe , mass flow rate of working fluid m ai , condensing fluid temperature T ai , working fluid pump speed u, superheat value SH, tracking error SH-SH set and control signal m; the collected system state sequence will be used as input and output data for training the system dynamics model based on neural network prediction.

[0016] The specific process of step (2) is:

[0017] (2-1) The seven state quantities P at time t e , T oe , m ai , T ai ,u,SH,SH-SH set The control signal m is input into the neural network prediction model, and the state quantity at time t+1 is used as a label for training to obtain the system dynamics model, which is used as the proxy model of the organic Rankine cycle system;

[0018] The structure of the neural network prediction model is as follows: the first layer is LSTM with 80 neurons, a dropout layer is added to prevent overfitting, and another layer of LSTM is added with 100 neurons, and another dropout layer is added to prevent overfitting.

[0019] (2-2) Under closed-loop control, the trained proxy model and the actual system are given the same initial state and controller parameters. The accuracy and consistency of the prediction model are verified by comparing the control effects of the two. If the model prediction effect does not meet expectations, repeat step (2-1).

[0020] Preferably, in step (3), a PPO algorithm is used to train a reinforcement learning agent to control the proxy model.

[0021] The specific process of step (3) is:

[0022] (3-1) According to the range of state changes in the historical data of the actual organic Rankine cycle system, set the upper and lower limits of the reinforcement learning state and the termination conditions of the reinforcement learning training episode;

[0023] (3-2) The action of the reinforcement learning agent is used as the control signal m, and the action space is set according to the range of the control signal in the actual organic Rankine cycle control system;

[0024] (3-3) In a reinforcement learning episode, the set value of the agent model control system is randomly changed at intervals, and a time-varying residual heat disturbance is added to improve the generalization performance after migration; according to the error signal SH-SH set Set discrete and continuous rewards, continuously train and adjust parameters until the reinforcement learning agent can take accurate actions to quickly reduce errors and achieve the ideal control effect.

[0025] According to the error signal SH-SH set The specific settings for discrete and continuous rewards are:

[0026] Set the discrete part reward as: when SH-SH set When the absolute value of is less than 1, a positive reward is given, and the rest are negative rewards. The smaller the error interval, the greater the reward; the continuous part reward is reward = -5*(SH-SH set ) 2 -0.5(u t -u t-1 ).

[0027] Step (4) specifically includes:

[0028] The reinforcement learning agent pre-trained in the virtual simulation environment constructed by the agent model is transferred to the control system of the actual organic Rankine cycle. The parameter settings of the reinforcement learning reward, algorithm parameters, and episode termination conditions are further adjusted according to the deviation of the control effect between the actual system and the virtual simulation environment. Training is continued on the real object until the reinforcement learning agent can achieve the same set value tracking performance in the actual organic Rankine cycle control system.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The present invention uses the data generated by the actual organic Rankine cycle system to build a model. The resulting proxy model can quickly predict the state at the next moment based on the state and control signal at the previous moment, effectively solving the problems of slow response and time-consuming calculation of the actual system.

[0031] (2) The present invention performs pre-training on the proxy model without interacting with the actual environment, which ensures security and greatly improves the training speed.

[0032] (3) The agent model of the present invention is obtained by training a system dynamics model based on neural network prediction, which can predict states that have not appeared in historical data during the reinforcement learning training process, thereby improving the generalization performance of the reinforcement learning agent in multimodal control situations.

[0033] (4) The present invention uses the PPO algorithm to train the proxy model and the actual ORC system, which reduces the frequency of hyperparameter debugging, effectively improves the training success rate, and accelerates model convergence.

[0034] (5) The method proposed in this invention can be easily extended to any industrial control system, providing a new idea for promoting the implementation of reinforcement learning in the field of industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of the organic Rankine cycle reinforcement learning control method based on virtual-real migration of the agent model of the present invention;

[0036] Figure 2 A structural diagram of an organic Rankine cycle system according to an embodiment of the present invention;

[0037] Figure 3 This is a structural diagram of a neural network prediction model in an embodiment of the present invention;

[0038] Figure 4 This is a prediction effect diagram of the neural network prediction model in an embodiment of the present invention;

[0039] Figure 5 This is a control effect diagram of migrating the reinforcement learning agent to the actual system in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0041] like Figure 1 As shown in the figure, an organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model is proposed. The structure of the organic Rankine cycle system is shown in the figure. Figure 2 As shown, it mainly consists of five components: evaporator 1, expander 2, condenser 3, liquid storage tank 4 and booster pump 5.

[0042] The specific control process includes the following steps:

[0043] S01, collect data from the actual system: add Gaussian white noise to the closed-loop control system of the organic Rankine cycle under the residual heat disturbance condition of 13.44kg / s, randomly change the set value every 100 seconds, and collect 5000 operating data; the evaporator outlet pressure P e , evaporator outlet temperature T oe , working fluid mass flow mai, condensing fluid temperature T ai , working fluid pump speed u, superheat value SH, tracking error SH-SH set and control signal m.

[0044] S02, training neural network prediction model: 5000 running data are divided into 4000 as training set and 1000 as test set. Construct neural network prediction model, the first layer is LSTM, the number of neurons is 80, add dropout layer to prevent overfitting, add another layer of LSTM, the number of neurons is 100, add dropout layer to prevent overfitting, its structure is as follows Figure 3 As shown. e , T oe , m ai , T ai ,u,SH,SH-SH set and control signal m as input to the neural network, and the P at time t+1 is e , T oe , m ai , T ai ,u,SH,SH-SH set as labels for training.

[0045] After training 1000 times, we draw a test set to test and draw a comparison chart between the predicted results and the actual state values, such as Figure 4 As shown, the mean square error is calculated. Under closed-loop control, the actual system and the prediction model are given the same PI parameters and setpoints, and the same initial state is input. After both respond for 200 seconds, a curve of the superheat value versus time is plotted to verify the accuracy of the model.

[0046] In S03, the more accurate neural prediction model is used as the proxy model of the organic Rankine cycle system. The gym module in Python is used to build a reinforcement learning environment. The upper limit of the state space is set to [2500, 500, 20, 600, 30.0, 30.0, 15], the lower limit is set to [1800, 200, 0.0, 0.0, 10, 10, -15], and the action space is [10 to 30]. The superheat value is set to SH set =24 as the control target, when SH<10 or SH>30, done=true; set the discrete part reward as: when SH-SH setWhen the absolute value of is less than 1, a positive reward is given, and the rest are negative rewards. The smaller the error interval, the greater the reward; the continuous part reward is reward = -5*(SH-SH set ) 2 -0.5(u t -u t-1 ), the PPO algorithm is used to train the reinforcement learning intelligent agent control agent model to make the superheat value quickly reach the set requirements and reduce the error.

[0047] S04, migrate the reinforcement learning agent that achieves the ideal control effect on the agent model to the actual organic Rankine cycle control system, and set the superheat value to SH set =24, directly controlling the actual organic Rankine cycle system.

[0048] Control effect such as Figure 5 As shown in the figure, the reward setting is adjusted according to the deviation of the control effect on the agent model and the actual object, and the PPO algorithm is continued to be used for training and parameter adjustment. After a small number of training episodes, the reinforcement learning agent can quickly eliminate the error in the actual organic Rankine cycle system and make the superheat value meet the set requirements.

[0049] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model, characterized in that: include: (1) When there is a disturbance in the waste heat operating conditions, the set value of the organic Rankine cycle system is changed at regular intervals to collect historical accumulated closed-loop operation data; (2) Using the collected closed-loop operation data to train a system dynamics model based on neural network prediction, and using it as a proxy model of the organic Rankine cycle system, verify the accuracy and consistency of the proxy model and the actual system under closed-loop control; if the proxy model meets the requirements, perform step (3); otherwise, repeat step (2); (3) Using the agent model to construct a virtual simulation environment for reinforcement learning pre-training, multiple set values ​​are set in a reinforcement learning episode and time-varying residual heat disturbance is added to train the reinforcement learning agent to control the agent model; if the training effect meets the preset conditions, step (4) is executed, otherwise step (3) is repeated; (4) Migrate the structure and parameters of the reinforcement learning agent pre-trained in the virtual simulation environment to the actual organic Rankine cycle control system, and perform fine-tuning and continue training until the ideal control effect is achieved on the actual system; (5) The trained reinforcement learning agent is used for process control of actual organic Rankine cycle system.

2. The organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model according to claim 1 is characterized in that: In step (1), the closed-loop operation data accumulated historically is collected as follows: Collect several system state quantities that change with time under various operating conditions, including: evaporator outlet pressure P e , evaporator outlet temperature T oe , mass flow rate of working fluid m ai , condensing fluid temperature T ai , working fluid pump speed u, superheat value SH, tracking error SH-SH set and control signal m; the collected system state sequence will be used as input and output data for training the system dynamics model based on neural network prediction.

3. The organic Rankine cycle reinforcement learning control method based on agent model virtual-real migration according to claim 2 is characterized in that: The specific process of step (2) is: (2-1) The seven state quantities P at time t e , T oe , m ai , T ai ,u,SH,SH-SH set The control signal m is input into the neural network prediction model, and the state quantity at time t+1 is used as a label for training to obtain the system dynamics model, which is used as the proxy model of the organic Rankine cycle system; (2-2) Under closed-loop control, the trained proxy model and the actual system are given the same initial state and controller parameters. The accuracy and consistency of the prediction model are verified by comparing the control effects of the two. If the model prediction effect does not meet expectations, repeat step (2-1).

4. The organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model according to claim 3 is characterized in that: In step (2-1), the structure of the neural network prediction model is as follows: the first layer is LSTM, the number of neurons is 80, a dropout layer is added to prevent overfitting, and another layer of LSTM is added, the number of neurons is 100, and another dropout layer is added to prevent overfitting.

5. The organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model according to claim 1, characterized in that: In step (3), the PPO algorithm is used to train a reinforcement learning agent to control the proxy model.

6. The organic Rankine cycle reinforcement learning control method based on agent model virtual-real migration according to claim 1 is characterized in that: The specific process of step (3) is: (3-1) According to the range of state changes in the historical data of the actual organic Rankine cycle system, set the upper and lower limits of the reinforcement learning state and the termination conditions of the reinforcement learning training episode; (3-2) The action of the reinforcement learning agent is used as the control signal m, and the action space is set according to the range of the control signal in the actual organic Rankine cycle control system; (3-3) In a reinforcement learning episode, the set value of the agent model control system is randomly changed at intervals, and a time-varying residual heat disturbance is added to improve the generalization performance after migration; according to the error signal SH-SH set Set discrete and continuous rewards, continuously train and adjust parameters until the reinforcement learning agent can take accurate actions to quickly reduce errors and achieve the desired control effect, where SH represents the superheat value.

7. The organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model according to claim 6, characterized in that: In step (3-3), according to the error signal SH-SH set The specific settings for discrete and continuous rewards are: Set the discrete part reward as: when SH-SH set When the absolute value of is less than 1, a positive reward is given, and the rest are negative rewards. The smaller the error interval, the greater the reward; the continuous part reward is reward = -5*(SH-SH set ) 2 -0.5(u t -u t-1 ).

8. The organic Rankine cycle reinforcement learning control method based on virtual-real migration of agent model according to claim 1, characterized in that: Step (4) specifically includes: The reinforcement learning agent pre-trained in the virtual simulation environment constructed by the agent model is transferred to the control system of the actual organic Rankine cycle. The parameter settings of the reinforcement learning reward, algorithm parameters, and episode termination conditions are further adjusted according to the deviation of the control effect between the actual system and the virtual simulation environment. Training is continued on the real object until the reinforcement learning agent can achieve the same set value tracking performance in the actual organic Rankine cycle control system.

Images