Multi-input multi-output system-oriented dual-network auxiliary reinforcement learning controller training method

Abstract

The embodiment of the invention provides a multi-input multi-output system-oriented dual-network auxiliary reinforcement learning controller training method and a multi-input multi-output system-oriented dual-network auxiliary reinforcement learning controller training system. The method comprises the following steps: constructing a simulation environment for replacing real environment interaction based on historical operation data of a multi-input multi-output system; determining an original state vector in the historical operation data and an incidence matrix among state variables at different sampling time points in the historical operation data, and performing state diagram reconstruction on the original state vector based on the incidence matrix to obtain an enhanced state feature; in a simulation environment, the enhanced state features are input into a reinforcement learning controller constructed by a strategy-value network, a predictive control action signal and a reward signal are output, the reward signal is used as training guidance for training, and the reinforcement learning controller used for adjusting system manipulated variables in a real environment is obtained. According to the embodiment of the invention, the cost and risk caused by interaction with a real system are avoided, the optimization control cost is reduced, and the stability and safety are improved.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent control technology, and in particular to a dual-network assisted reinforcement learning controller training method, system, electronic device, and storage medium for multiple-input multiple-output systems. Background Technology

[0002] Multiple-input multiple-output (MIO) control systems are widely used in complex industrial processes. They typically feature multiple manipulated variables coupled with multiple controlled variables, resulting in highly nonlinear and uncertain system dynamics. In such systems, each input variable simultaneously affects multiple output variables, making it difficult for single-loop control strategies to effectively coordinate different control objectives. This can easily lead to increased energy consumption, violation of operational constraints, or decreased system stability. Therefore, achieving optimal control while meeting safety constraints is a key issue in MIO industrial process control.

[0003] Control of multiple-input multiple-output (MIMO) systems relies on traditional PID (Proportional Integral Derivative) control or linear model predictive control. However, designing a PID controller suitable for MIMO systems is quite complex, and its control performance has limitations. While model predictive control methods demonstrate superior performance, they are highly dependent on the accuracy of the system model, which limits their practical applications.

[0004] Existing technologies attempt to use artificial intelligence to control multiple-input multiple-output systems. For example, the DDPG (Deep Deterministic Policy Gradient) algorithm combines the function approximation capability of deep learning with the efficient decision-making characteristics of policy gradient methods, enabling it to learn the optimal control policy in a continuous action space.

[0005] In the process of realizing this invention, this application found that the following problems still exist in the related technologies where the DDPG algorithm is directly applied to industrial processes: First, the algorithm training requires a lot of interaction with the environment, and direct interaction with real multiple-input multiple-output systems is costly and risky; Second, the standard DDPG algorithm has limited ability to perceive the complex coupling relationship between system states, which affects the accuracy and robustness of the control strategy; Finally, the exploration efficiency and stability during the training process still need to be further improved. Summary of the Invention

[0006] In order to at least address the problems existing in the aforementioned technologies.

[0007] In a first aspect, embodiments of the present invention provide a dual-network assisted reinforcement learning controller training method for multiple-input multiple-output (MIMO) systems, comprising: Based on the historical operating data of the multiple-input multiple-output system, a simulation environment is constructed to replace the interaction with the real environment; The original state vector in the historical operation data is determined, as well as the correlation matrix between each state variable at different sampling time points in the historical operation data. Based on the correlation matrix, the original state vector is reconstructed into a state diagram to obtain enhanced state features. The state variables include optimization variables and controlled variables. In the simulation environment, the enhanced state features are input to a reinforcement learning controller constructed from a policy-value network, which outputs a predicted control signal and a reward signal. The reward signal is used as training guidance to train the reinforcement learning controller. After training, a reinforcement learning controller for real-time adjustment of system manipulation variables for real-world environments is obtained.

[0008] Secondly, embodiments of the present invention provide a dual-network assisted reinforcement learning controller training system for multiple-input multiple-output (MIMO) systems, comprising: The simulation environment construction module is used to build a simulation environment that replaces the interaction with the real environment based on the historical operating data of the multi-input multi-output system. The state diagram reconstruction module is used to determine the original state vector in the historical operation data and the correlation matrix between each state variable at different sampling time points in the historical operation data. Based on the correlation matrix, the original state vector is reconstructed to obtain enhanced state features. The state variables include optimization variables and controlled variables. The training module is used in the simulation environment to input the enhanced state features into a reinforcement learning controller constructed by a policy-value network, output a predicted control signal and a reward signal, use the reward signal as training guidance to train the reinforcement learning controller, and after training, obtain a reinforcement learning controller for real-time adjustment of the manipulated variables of a multi-input multi-output system for real-world environments.

[0009] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the dual-network-assisted reinforcement learning controller training method for multiple-input multiple-output systems according to any embodiment of the present invention.

[0010] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems according to any embodiment of the present invention.

[0011] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems according to any embodiment of the present invention.

[0012] The beneficial effects of this invention are as follows: By designing a reward function centered on system operating costs, the agent is guided to learn operational strategies, and experiments have shown that this significantly reduces the operating costs of multiple-input multiple-output (MIMO) systems. Through state diagram reconstruction, the agent can more deeply perceive the coupling relationships between system states, thereby making more coordinated control decisions, effectively suppressing fluctuations in key state variables, and improving the stability and security of the system. The use of a dual-network simulation environment allows the reinforcement learning training process to be conducted entirely in a virtual environment, avoiding the costs and risks associated with interacting with real systems, and greatly promoting the application of the algorithm in industrial scenarios. Overall, the reinforcement learning control method of this invention exhibits smaller reward fluctuations and more stable control actions after training convergence, and has stronger adaptability to complex working conditions and dynamic changes. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems provided in an embodiment of the present invention; Figure 2 This is a process flow diagram of a three-phase regenerative thermal oxidizer for a dual-network assisted reinforcement learning controller training method for a multi-input multi-output system, provided by an embodiment of the present invention. Figure 3 This is a schematic diagram of the loss function curve for training a dual-network environment model of a dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems provided in an embodiment of the present invention. Figure 4 This is a schematic diagram comparing the predicted and actual values ​​of the environment model in a dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems, provided by an embodiment of the present invention. Figure 5 This is a schematic diagram of the reward curve of a reinforcement learning controller during the training process of a dual-network assisted reinforcement learning controller training method for a multiple-input multiple-output system provided in an embodiment of the present invention. Figure 6 This is an overall flowchart of a dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems provided by an embodiment of the present invention; Figure 7 This is a schematic diagram of the valve opening curves under the reinforcement learning strategy of a dual-network assisted reinforcement learning controller training method for a multi-input multi-output system provided in an embodiment of the present invention. Figure 8 This is a schematic diagram of the fuel flow rate under the reinforcement learning strategy of a dual-network assisted reinforcement learning controller training method for a multiple-input multiple-output system provided in an embodiment of the present invention. Figure 9 This is a schematic diagram of the temperature state change curve under the reinforcement learning strategy of a dual-network assisted reinforcement learning controller training method for a multiple-input multiple-output system provided in an embodiment of the present invention. Figure 10 This is a schematic diagram of the structure of a dual-network assisted reinforcement learning controller training system for multiple-input multiple-output systems provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of an embodiment of an electronic device for training a dual-network-assisted reinforcement learning controller for a multiple-input multiple-output system, as provided in one embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] This application describes a multi-input multi-output (MIMO) system that controls multiple manipulated variables in an industrial process to achieve intelligent control. Examples of MIMO systems include: three-phase regenerative thermal oxidizers, once-through boiler-turbine units, and multi-stage continuous stirred tanks. The three-phase regenerative thermal oxidizer achieves different operating states by controlling multiple valves.

[0017] A once-through boiler-turbine unit generates steam by continuously heating water and directly producing steam to efficiently drive a steam turbine for power generation. It requires the control of coal feed rate, water feed rate, valve opening degree, steam pressure, etc., to achieve efficient driving of the steam turbine for power generation.

[0018] Multistage continuous stirred tank reactors optimize mixing and reaction efficiency by connecting multiple stirred reactors in series. This requires controlling combinations of rotating shafts, stirring shafts, and rotors to achieve the stirring and reaction of materials. In the following embodiments, a Regenerative Thermal Oxidizer (RTO) system is specifically used as an example to describe in detail the reinforcement learning optimization controller training method and system for multi-input multi-output systems proposed in this invention. However, this invention is not limited to this application scenario; similarly, it can also be applied to the aforementioned DC boiler-turbine units and multistage continuous stirred tank reactors, among other scenarios.

[0019] like Figure 1 The diagram shows a flowchart of a dual-network assisted reinforcement learning controller training method for a multiple-input multiple-output (MIMO) system according to an embodiment of the present invention, including the following steps: S11: Based on the historical operating data of the multiple-input multiple-output system, construct a simulation environment to replace the interaction with the real environment; S12: Determine the original state vector in the historical operation data, and the correlation matrix between each state variable at different sampling time points in the historical operation data. Based on the correlation matrix, reconstruct the state diagram of the original state vector to obtain enhanced state features, wherein the state variables include optimization variables and controlled variables. S13: In the simulation environment, the enhanced state features are input to the reinforcement learning controller constructed by the policy-value network, and the predicted control signal and reward signal are output. The reward signal is used as training guidance to train the reinforcement learning controller. After training, a reinforcement learning controller for real-time adjustment of system manipulation variables for real environment is obtained.

[0020] In this embodiment, a three-phase regenerative thermal oxidizer will be used as an example for explanation. The process flow of the three-phase regenerative thermal oxidizer is as follows: Figure 2 As shown, the dilution air is mixed with the exhaust gas through a dilution air valve and then input into the regenerator chamber by the main fan and distributor. The three-phase regenerator thermal oxidizer processes the gas in three operating states: In operating state 1, the exhaust gas enters the first regenerator chamber (or regenerator) on the left, is preheated, and then enters the combustion chamber for oxidation and decomposition at high temperature. The purified high-temperature gas enters the second regenerator chamber in the middle to release heat, while the third regenerator chamber on the right is backflushed with purified gas to remove residual exhaust gas.

[0021] In operating state 2, the exhaust gas switches to the second regenerator chamber, where it is preheated before entering the combustion chamber for oxidation. The purified high-temperature gas then enters the third regenerator chamber to release heat, while the first regenerator chamber undergoes backflushing.

[0022] In operating state 3, the exhaust gas enters the third regenerator chamber, is preheated, and then enters the combustion chamber for oxidation. The purified high-temperature gas enters the first regenerator chamber to release heat, and the second regenerator chamber undergoes backflushing. Through the cyclical operation of these three operating states, exhaust gas purification and heat recovery are achieved. The controller trained in this method adjusts the valve opening of the three-phase regenerable thermal oxidizer in real time to regulate the above operating states.

[0023] For step S11, during the operation of the three-phase regenerative thermal oxidizer described above, data acquisition and preprocessing are performed. Monitoring points are configured for each node of each step. Historical operating data are collected from multiple monitoring points of the three-phase RTO system, key variables are selected, and a simulation environment that replaces the interaction with the real environment is constructed.

[0024] Similarly, data acquisition and preprocessing are also carried out during the operation of the DC boiler-turbine unit. The operation process of the DC boiler-turbine unit will not be described in detail here. Monitoring points are configured at each node of each step. At the same time, historical operating data (such as coal feed rate, water feed rate, valve opening degree, steam pressure, etc.) are collected from multiple monitoring points to screen out key variables and then construct a simulation environment to replace the interaction with the real environment.

[0025] If the operation of a multi-stage continuous stirred tank is being carried out, data acquisition and preprocessing are also performed. Historical operating data (e.g., shaft, stirring shaft, rotor combination, etc.) are collected from multiple monitoring points, key variables are selected, and a simulation environment that replaces the interaction with the real environment is constructed.

[0026] As one implementation method, constructing a simulation environment to replace real-world interaction based on historical operating data of the multiple-input multiple-output system includes: Historical operating data of a three-phase regenerative thermal oxidizer is collected, and manipulated variables, optimized variables, controlled variables, and disturbance variables are extracted from the historical operating data. The manipulated variables, optimization quantities, controlled quantities, and disturbance variables are normalized to eliminate the influence of dimensions. A simulation environment with a dual neural network is constructed, with the optimized quantity and the controlled quantity as the primary and secondary objectives of training. The manipulated variable and the perturbation variable are used as inputs to the simulation environment of the dual neural network to obtain the prediction results of the optimized quantity and the controlled quantity, respectively. Based on the primary and secondary objectives and the prediction results, the simulation environment is fitted and trained to obtain a simulation environment that replaces the interaction of the real environment.

[0027] The manipulated variables include the opening degree of the fresh air valve, the opening degree of the fuel valve, and the opening degree of the heat bypass valve; the optimized quantity includes the fuel flow rate; and the controlled quantity includes the combustion chamber temperature and the temperatures of multiple heat storage chambers.

[0028] In this embodiment, key variables in the historical operating data include: manipulated variables (e.g., fresh air valve opening, fuel valve opening, and heat bypass valve opening), optimized variables (e.g., fuel flow rate), controlled variables (e.g., combustion chamber temperature and the temperatures of the upper and lower parts of the three regenerators), and disturbance variables (variables not listed above). The manipulated variables, optimized variables, controlled variables, and disturbance variables are normalized to eliminate dimensional effects and accelerate model convergence. (For a once-through boiler-turbine unit environment, manipulated variables include valve opening, optimized variables include coal feed rate and water feed rate, controlled variables include steam pressure, and the others are disturbance variables. These can be adapted and adjusted according to different environments. Other environmental data can be obtained from the above steps and will not be elaborated further here.)

[0029] Specifically, data from 146 monitoring points were collected from the industrial three-phase RTO system to form a dataset, and 34 key variables were selected. These include the opening of the fresh air valve (abbreviated as FV201, and the English letters in parentheses thereafter are corresponding abbreviations) as manipulated variables, the opening of the fuel valve (FCV401) and the opening of the heat bypass valve (FV202) as manipulated variables, the fuel quantity (OIL) as optimized variables, the combustion chamber temperature (TT211) as controlled variables, and the temperature measurement points of six regenerators (TT203, TT204, TT206, TT207, TT209, TT210). The remaining 23 variables were used as disturbance variables.

[0030] All the key variables mentioned above are normalized to the [0,1] interval to eliminate the influence of dimensions. The normalization steps are as follows: (1) (1) Among them, the first one in the formula The original data to be normalized, the rest (Bold) is the matrix composed of all the original data. For the normalized data, This is a correction term used to prevent the denominator from being zero. Through the above method, the normalized manipulated variable, the optimized variable, the controlled variable, and the disturbance variable are obtained.

[0031] The manipulated variables, optimized variables, controlled variables, and disturbance variables are obtained. These data are then used for dual-neural network environment modeling. The same steps can be used for environment modeling of once-through boiler-turbine units and multi-stage continuous stirred tank reactors.

[0032] In one implementation, the dual neural network consists of an independent first neural network and a second neural network. The first neural network and the second neural network adopt a three-layer feedforward neural network. The neurons in the input layer include multiple manipulated variables and multiple perturbation variables. The neurons in the output layer of the first neural network are the optimization variables, and the neurons in the output layer of the second neural network are the controlled variables.

[0033] In this embodiment, two independent neural network models are constructed, using normalized manipulated variables and perturbation variables as network inputs. Both neural networks employ a three-layer feedforward neural network structure, with 26 neurons in the input layer (3 manipulated variables and 23 perturbation variables) and 128 neurons in the hidden layer, using the ReLU activation function. Furthermore, the two independent neural networks use the optimized variable (fuel flow rate) and the controlled variable (temperature at various points) as their network outputs, respectively. Specifically, the output layer of the first neural network has one neuron, corresponding to the optimized variable, i.e., fuel flow rate; the output layer of the second neural network has seven neurons, corresponding to the controlled variable, i.e., seven temperature states.

[0034] During the training process, the normalized optimization quantity and the controlled quantity are used as the true value of the target. The manipulated variable and the perturbation variable are input into the above neural network model. The output layer of the first neural network is used to determine the predicted optimization quantity, and the output layer of the second neural network is used to determine the predicted controlled quantity. The predicted optimization quantity and the predicted controlled quantity are used as the predicted value. The mean squared error is used as the loss function during the training process, as shown in the following formula (2): (2) in, For the sample size mentioned above, For the true value, This is the predicted value. The parameters can be updated using the Adam optimizer, with the learning rate set to 0.001 and the batch size to 64.

[0035] Using the above method, dual neural networks can accurately fit the dynamic characteristics of the RTO system, constructing a simulation environment that meets the accuracy requirements, thus replacing the costly interaction with the real environment.

[0036] like Figure 3 As shown, the loss function steadily decreases during training, indicating good network convergence. For the loss curves of both neural networks, convergence is achieved in approximately 10 iterations. The temperature prediction loss converges relatively slowly. Figure 4 The comparison between the model's predicted values ​​and the actual values ​​is shown. It can be seen that the predicted curves are highly consistent with the actual curves. Only at some turning points with sharp changes, due to the rapid changes in the dynamic characteristics of the system, are there slight deviations in the prediction results, but the overall tracking effect is good.

[0037] For step S12, after training the dual neural network, state diagram reconstruction and feature enhancement are also performed. The original state vector in the historical operating data and the correlation matrix between state variables at different sampling time points within the historical operating data are determined. Based on the correlation matrix, the original state vector is reconstructed into a state diagram.

[0038] As one implementation method, determining the original state vector in the historical operating data, and the correlation matrix between state variables at different sampling time points within the historical operating data, includes: The Pearson correlation coefficient among the state variables is determined by the matrix elements. The constructed correlation matrix R: (3) Among them, the and They represent the first The and the first The state variables, the and They represent the first The and the first The average value of each state variable at all sampling time points. This indicates the total number of sampled data points.

[0039] In this implementation, the Pearson correlation coefficients between each pair of state variables, such as the controlled and optimized variables, from historical operational data are used to form a correlation matrix. This correlation matrix is ​​then multiplied by the original state vector to reconstruct the state. This process embeds the linear correlation information between variables into the state representation, generating enhanced state features with stronger expressive power, thus helping the agent understand the coupling relationships within the system.

[0040] Specifically, the Pearson correlation coefficients between various state variables are calculated based on historical operating data. For state variables... and Its correlation coefficient The calculation is as described in formula (3) above. For example, in this embodiment, it can be set as follows: , and They represent the first The and the first The average value of each state variable at all sampling time points.

[0041] By calculating each pair of state variables in the system, the correlation matrix of the following formula (4) is finally constructed. : (4) The correlation matrix With the original state vector Multiplying them together achieves the state reconstruction of the following formula (5): (5).

[0042] in, The original state vector is 8-dimensional. This is the reconstructed enhanced state feature. This process embeds the correlation information between variables into the state representation through linear transformation, effectively extracting the system's higher-order features and enhancing the expressive power of the coupling relationships between state variables. The enhanced state features are obtained through the above method.

[0043] For step S13, this method constructs a reinforcement learning controller with a policy-value network architecture, which is trained in a simulation environment. Generally, the policy network, typically employing a multilayer perceptron or convolutional neural network, is responsible for predicting the optimal action probability distribution for a given state, i.e., "how to control". It is implemented through a neural network (such as a multilayer perceptron or convolutional network), taking the state as input (such as the enhanced state features obtained in the previous steps) and outputting the probability of continuous valve opening control actions. The value network is responsible for evaluating the final value (i.e., reward) of a given state, helping to determine the quality of the current situation. It is also constructed using a neural network, with an output being a scalar value representing the expected quality of the state.

[0044] The enhanced state features determined in the above steps are input into the reinforcement learning controller of the policy-value network architecture. In the policy network, the determined continuous valve opening control action is output. Further, the value network determines the corresponding reward Q value based on the above continuous valve opening control action and the real-time state of the three-phase RTO in the current control system, which is used to evaluate the long-term merits of the above continuous valve opening control action.

[0045] Specifically, in the reinforcement learning controller used in a three-phase regenerative thermal oxidizer, both the policy network and the value network employ a three-layer fully connected network structure. The policy network's input layer has 8 neurons that receive reconstructed state features, its hidden layer has 256 neurons using the ReLU activation function, and its output layer has 3 neurons using the Tanh activation function to output the normalized opening actions of three valves (fuel valve, dilution air valve, and heat bypass valve). The value network's input layer has 11 neurons, including 8 state features and 3 actions, its hidden layer has 256 neurons using the ReLU activation function, and its output layer has 1 neuron that outputs a Q-value estimate.

[0046] In one implementation, the reward signal r includes the variable to be optimized, the upper limit constraint of the state, and the lower limit constraint of the state, as shown in the following formula (6): (6) Among them, the The variable to be optimized is... Represents the state constraint variables, the and This represents the lower and upper limits of the state constraint variable. When the constrained state is within a preset safety range, the reward is zero; when it deviates from the preset safety range, a negative reward is generated. The k... i This represents the weighting coefficient of the reward.

[0047] In this embodiment, regarding training guidance, the reinforcement learning controller employs a multi-objective fusion reward function design. This function consists of three weighted components: a fuel consumption reward based on the ReLU function, which proportionalizes negative reward values ​​with fuel flow to guide the system towards energy-saving operation; a combustion chamber temperature reward based on a smooth cutoff function, where the reward is zero when the temperature is within the safe range of 760℃ to 1000℃, and a negative reward is generated if it deviates, ensuring stable and controlled core temperature; and a regenerator temperature reward based on a variant of the ReLU function, which applies a negative penalty when the temperature of any regenerator exceeds the upper limit of 900℃ to prevent damage to the regenerator due to overheating. Through this comprehensive design, the controller can achieve the core optimization objective of minimizing fuel consumption while ensuring the safe and stable operation of the system.

[0048] In practical implementation, the training objective of the reinforcement learning optimization controller used in the multiple-input multiple-output system is guided by a reward function. This reward function is designed with three parts, targeting fuel consumption, combustion chamber temperature, and regenerator temperature respectively. The total reward function... It consists of the sum of the following parts, as shown in the following formula (7): (7) in, For fuel flow rate, and These represent the combustion chamber temperature and the regenerator temperature, respectively. and This indicates the lower and upper limits of the safe temperature range for the heat storage chamber. This is the upper limit of the combustion chamber temperature. When the temperature is within the preset safe range, the reward is zero, and when it deviates from this range, a negative reward is generated. This refers to the weighting coefficients for each part of the reward.

[0049] Similarly, in the scenarios of DC boiler-turbine units and multi-stage continuous stirred tanks, the key data can be mapped to the variables to be optimized and the upper and lower limits of the state for calculation.

[0050] Furthermore, as an implementation method, to ensure the stability and efficiency of the training process, the reinforcement learning controller introduces three key mechanisms: an experience replay mechanism, a soft update strategy mechanism for independent policy networks and value networks, and a linear decay exploration mechanism.

[0051] In this embodiment, firstly, an experience replay mechanism is used to store the interaction data between the agent and the environment. Random sampling is used to break the temporal correlation between data and improve learning efficiency. Secondly, separate target networks are established for the policy network and the value network, and the parameters are gradually synchronized by soft updating the policy to effectively suppress the fluctuation of Q-value estimation and enhance training stability. The target network is a network with the same structure as the main network (such as the policy network) but with independent parameters. It does not directly participate in action selection but is used to calculate the target Q-value, thus providing a stable training signal and avoiding oscillations caused by frequent updates to the main network parameters. Soft updates are achieved through a small learning rate (soft update coefficient). This enables progressive parameter synchronization. For example, in implementation, one could use... ,in, These are the main network parameters. These are the target network parameters. This strategy allows the target network parameters to gradually converge towards the main network, rather than being suddenly replaced, thus smoothing the training process.

[0052] Finally, a linearly decaying exploration rate strategy is designed to maintain a high exploration rate in the early stages of training to fully explore the action space, and then gradually reduce it to a predetermined threshold to ensure that the strategy eventually converges to the optimal solution.

[0053] The exploration strategy employs linear decay, as shown in the following formula (8): (8) Among them, the initial exploration rate End of exploration rate decay steps , This represents the current number of training steps. This design ensures that the movement space is fully explored in the early stages of training and that the strategy is stably converged to the optimal strategy in the later stages of training.

[0054] Figure 5 The convergence curve of the reward function of the reinforcement learning controller during training is shown. The three stages of the training process can be clearly observed from the trend of the curve.

[0055] In the early stages of training, the reward value fluctuates significantly and remains at a low level, reflecting the agent's exploration phase of the action space. During this period, the agent accumulates experience through random exploration, gradually building an understanding of the environment and reward mechanism. The fluctuation of the reward value precisely reflects the agent's process of trying different control strategies and exploring the optimal solution.

[0056] As training progresses into the middle stage, the reward value begins to show a steady upward trend. This indicates that the agent has begun to learn effective control strategies and is continuously optimizing its decision-making process. The steady increase in the reward value shows that the agent is gradually mastering a balanced strategy for achieving energy-saving goals while ensuring the safe operation of the system.

[0057] In the later stages of training, the reward function curve tends to stabilize and converge to a high level. This indicates that the agent has learned a near-optimal control strategy and can stably achieve multi-objective optimization control under complex conditions. The final reward value stabilizes at a high level, validating the sufficiency and effectiveness of the training process.

[0058] After training, the trained reinforcement learning controller is deployed to the three-phase RTO control system. Based on the real-time system status, the controller outputs the optimal valve opening control signal, achieving real-time, intelligent optimization control. The flowchart of the overall steps described above is as follows: Figure 6 As shown.

[0059] This implementation demonstrates that by designing a reward function centered on reducing fuel consumption, the agent learns energy-saving operating strategies, and experiments have shown that this significantly reduces total fuel consumption. Through state diagram reconstruction, the agent can more deeply perceive the coupling relationships between system states, thereby making more coordinated control decisions, effectively suppressing fluctuations in key temperature parameters, and improving system stability and safety. In applications such as DC boiler-turbine units and multi-stage continuous stirred tank reactors, it can reduce the consumption of water, coal, and other energy sources. The use of a dual-network simulation environment allows the reinforcement learning training process to be conducted entirely in a virtual environment, avoiding the costs and risks associated with interacting with real systems, and greatly promoting the algorithm's practical application in industrial scenarios. Overall, the reinforcement learning control method presented here exhibits smaller reward fluctuations and more stable valve control actions after training convergence, demonstrating stronger adaptability to complex operating conditions and dynamic changes.

[0060] As one implementation, after obtaining a reinforcement learning controller for real-time adjustment of system manipulation variables in a real environment, the method further includes: The reinforcement learning controller determines the control signal for real-time adjustment of the multiple-input multiple-output system. Receive the input fine-tuning control signal and determine the fine-tuning control signal as the adaptation and optimization signal; The reinforcement learning controller is trained based on the loss function of the predicted control signal and the adaptation tuning signal until a preset target is reached, thus obtaining a reinforcement learning controller adapted to the multiple-input multiple-output system.

[0061] In this embodiment, considering that the reinforcement learning controller is trained in a simulation environment, the following experiments have verified its excellent performance in terms of valve control stability, fuel economy, and temperature stability. To further enable the reinforcement learning controller trained in the simulation environment to better adapt to various oxidizers in real-world scenarios, taking the three-phase regenerative thermal oxidizer described above as an example, this design takes into account that some new three-phase regenerative thermal oxidizers may not have been used for long and have insufficient historical operating data. Therefore, historical operating data from another three-phase regenerative thermal oxidizer with the exact same configuration and model that has been in use for a long time can be used to construct a simulation environment for that configuration. However, this method still considers that new and old three-phase regenerative thermal oxidizers are not the same unit, and there may still be some errors. Considering this situation, this method further specifically trains the reinforcement learning controller trained in this cross-device-defined simulation environment. After deploying the trained reinforcement learning controller to the three-phase RTO control system, it outputs valve opening control signals. Based on the actual conditions of different three-phase regenerative thermal oxidizers, operators may make slight adjustments to the valve opening control signal output. The valve is then fine-tuned again, and an adaptation tuning signal is input. At this point, a loss function is determined based on the adaptation tuning signal and the output valve opening control signal. This loss function is then used to further optimize and fine-tune the reinforcement learning controller. Through reinforcement learning-fine-tuning training, control optimization for various real-world three-phase regenerative thermal oxidizers is achieved.

[0062] Experiments illustrate this method by applying the trained reinforcement learning controller to a real-world multi-input multi-output system, such as a three-phase regenerative thermal oxidizer system, for testing and evaluation. The test lasted 10,000 seconds, and the overall performance of the control system was comprehensively evaluated through monitoring and analysis of key operating parameters.

[0063] In terms of valve control performance, such as Figure 7 As shown, the reinforcement learning controller demonstrated excellent control quality. The opening changes of the three control valves (fresh air valve FV201, fuel valve FCV401, and heat bypass valve FV202) were smooth and coordinated, and the valve movement amplitude was effectively suppressed, avoiding equipment wear and system disturbances caused by frequent large-scale adjustments. In particular, the fuel valve FCV401 achieved a stable fuel supply to the combustion chamber through precise fine-tuning, demonstrating the controller's accurate grasp of the system's dynamic characteristics.

[0064] In terms of energy efficiency, such as Figure 8 As shown, the data displays fuel flow monitoring data. During a 10,000-second test cycle, the total fuel consumption of the system was 108.63 kg. Analysis of the fuel flow curve reveals that the controller can intelligently adjust the fuel supply based on the real-time system status, significantly reducing fuel consumption while ensuring processing efficiency, demonstrating excellent energy-saving performance.

[0065] like Figure 9 The temperature control performance was demonstrated. Monitoring data showed that the combustion chamber temperature remained stable within the industrial safety range with low temperature fluctuation variance, reflecting the controller's excellent temperature stability. Simultaneously, the temperature distribution at the six heat storage chamber temperature measuring points (TT203, TT204, TT206, TT207, TT209, TT210) was uniform, and the temperature variance at each measuring point remained at a low level, indicating that the system's heat distribution was balanced and the heat storage medium was effectively protected.

[0066] In summary, the graph reconstruction reinforcement learning algorithm-based oxidizer control optimization strategy provided by this method exhibits excellent performance in terms of valve control stability, fuel economy, and temperature stability. In particular, it provides an effective technical solution for the intelligent optimization control of three-phase regenerative thermal oxidizer systems, and has significant industrial application value and promising prospects.

[0067] like Figure 10 The diagram shown is a structural schematic of a reinforcement learning optimization controller training system for a multiple-input multiple-output (MIMO) system according to an embodiment of the present invention. The system can execute the dual-network assisted reinforcement learning controller training method for MIMO systems described in any of the above embodiments and is configured in a terminal.

[0068] This embodiment provides a reinforcement learning optimization controller training system 10 for a multiple-input multiple-output system, which includes: a simulation environment construction module 11, a state diagram reconstruction module 12, and a training module 13.

[0069] The simulation environment construction module 11 is used to construct a simulation environment that replaces the interaction with the real environment based on the historical operating data of the multi-input multi-output system. The state diagram reconstruction module 12 is used to determine the original state vector in the historical operating data and the correlation matrix between each state variable at different sampling time points in the historical operating data. Based on the correlation matrix, the original state vector is reconstructed into a state diagram to obtain enhanced state features. The state variables include optimization variables and controlled variables. The training module 13 is used to input the enhanced state features into the reinforcement learning controller constructed by the policy-value network in the simulation environment, output the predicted control signal and the reward signal, and use the reward signal as training guidance to train the reinforcement learning controller. After training, a reinforcement learning controller for real-time adjustment of the manipulated variables of the multi-input multi-output system is obtained for the real environment.

[0070] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the dual-network-assisted reinforcement learning controller training method for multiple-input multiple-output systems in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: Based on the historical operating data of the multiple-input multiple-output system, a simulation environment is constructed to replace the interaction with the real environment; The original state vector in the historical operation data is determined, as well as the correlation matrix between each state variable at different sampling time points in the historical operation data. Based on the correlation matrix, the original state vector is reconstructed into a state diagram to obtain enhanced state features. The state variables include optimization variables and controlled variables. In the simulation environment, the enhanced state features are input to a reinforcement learning controller constructed from a policy-value network, which outputs a predicted control signal and a reward signal. The reward signal is used as training guidance to train the reinforcement learning controller. After training, a reinforcement learning controller for real-time adjustment of system manipulation variables for real-world environments is obtained.

[0071] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems in any of the above method embodiments is executed.

[0072] Figure 11This is a schematic diagram of the hardware structure of an electronic device for a dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems, as provided in another embodiment of this application. Figure 11 As shown, the device includes: One or more processors 1110 and memory 1120, Figure 11 Taking a processor 1110 as an example, the device for training a dual-network-assisted reinforcement learning controller for a multiple-input multiple-output system may further include an input device 1130 and an output device 1140.

[0073] The processor 1110, memory 1120, input device 1130, and output device 1140 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0074] The memory 1120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the dual-network-assisted reinforcement learning controller training method for MIMO systems in this embodiment. The processor 1110 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 1120, thereby implementing the dual-network-assisted reinforcement learning controller training method for MIMO systems described in the above embodiment.

[0075] The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1120 may optionally include memory remotely located relative to the processor 1110, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0076] Input device 1130 can receive input numerical or character information. Output device 1140 may include display devices such as a display screen.

[0077] The one or more modules are stored in the memory 1120. When executed by the one or more processors 1110, they execute the dual-network assisted reinforcement learning controller training method for multiple-input multiple-output systems in any of the above method embodiments.

[0078] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0079] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the dual-network-assisted reinforcement learning controller training method for multiple-input multiple-output systems according to any embodiment of this invention.

[0081] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0082] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0083] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0084] (4) Other electronic devices with data processing functions.

[0085] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-network-assisted reinforcement learning controller training method for multiple-input multiple-output (MIMO) systems, comprising: Based on the historical operating data of the multiple-input multiple-output system, a simulation environment is constructed to replace the interaction with the real environment; The original state vector in the historical operation data is determined, as well as the correlation matrix between each state variable at different sampling time points in the historical operation data. Based on the correlation matrix, the original state vector is reconstructed into a state diagram to obtain enhanced state features. The state variables include optimization variables and controlled variables. In the simulation environment, the enhanced state features are input to a reinforcement learning controller constructed from a policy-value network, which outputs a predicted control signal and a reward signal. The reward signal is used as training guidance to train the reinforcement learning controller. After training, a reinforcement learning controller for real-time adjustment of system manipulation variables for real-world environments is obtained.

2. The dual-network assisted reinforcement learning controller training method according to claim 1, characterized in that, The multi-input multi-output system includes: a three-phase regenerative thermal oxidation furnace; Based on the historical operating data of the multiple-input multiple-output system, a simulation environment is constructed to replace the interaction with the real environment, including: Historical operating data of a three-phase regenerative thermal oxidizer is collected, and manipulated variables, optimized variables, controlled variables, and disturbance variables are extracted from the historical operating data. A simulation environment with a dual neural network is constructed, with the optimized quantity and the controlled quantity as the primary and secondary objectives of training. The manipulated variable and the perturbation variable are used as inputs to the simulation environment of the dual neural network to obtain the prediction results of the optimized quantity and the controlled quantity, respectively. Based on the primary and secondary objectives and the prediction results, the simulation environment is fitted and trained to obtain a simulation environment that replaces the interaction of the real environment.

3. The dual-network assisted reinforcement learning controller training method according to claim 2, characterized in that, After extracting the manipulated variables, optimized variables, controlled variables, and disturbance variables from the historical operating data, the method further includes: The manipulated variables, optimization quantities, controlled quantities, and disturbance variables are normalized to eliminate the influence of dimensions.

4. The dual-network assisted reinforcement learning controller training method according to claim 2, characterized in that, The dual neural network consists of an independent first neural network and a second neural network. The first neural network and the second neural network adopt a three-layer feedforward neural network. The neurons in the input layer include multiple manipulated variables and multiple perturbation variables. The neurons in the output layer of the first neural network are the optimized variables, and the neurons in the output layer of the second neural network are the controlled variables. The manipulated variables include the opening degree of the fresh air valve, the opening degree of the fuel valve, and the opening degree of the heat bypass valve; the optimized quantity includes the fuel flow rate; and the controlled quantity includes the combustion chamber temperature and the temperatures of multiple heat storage chambers.

5. The dual-network assisted reinforcement learning controller training method according to claim 1, characterized in that, The determination of the original state vector in the historical operating data, and the correlation matrix between state variables at different sampling time points within the historical operating data, includes: The Pearson correlation coefficient among the state variables is determined by the matrix elements r. ij The constructed correlation matrix R: ， Among them, the and They represent the first i The and the first j The state variables, the and They represent the first i The and the first j The average value of each state variable at all sampling time points. P This indicates the total number of sampled data points.

6. The dual-network assisted reinforcement learning controller training method according to claim 4, characterized in that, The reward signal r includes the variable to be optimized, the upper limit constraint of the state, and the lower limit constraint of the state: ， Among them, the The variable to be optimized is... Represents the state constraint variables, the and This represents the lower and upper limits of the state constraint variable. When the constrained state is within a preset safety range, the reward is zero; when it deviates from the preset safety range, a negative reward is generated. The k... i This represents the weighting coefficient of the reward.

7. The dual-network assisted reinforcement learning controller training method according to claim 1, characterized in that, The training mechanism for training the reinforcement learning controller using the reward signal as training guidance includes: an experience replay mechanism, a soft update strategy mechanism for independent policy networks and value networks, and a linear decay exploration mechanism.

8. The dual-network assisted reinforcement learning controller training method according to claim 1, characterized in that, The multi-input multi-output system also includes: a DC boiler-turbine unit and a multi-stage continuous stirring vessel.

9. The dual-network assisted reinforcement learning controller training method according to claim 1, characterized in that, After obtaining a reinforcement learning controller for real-time adjustment of the manipulator variables of a multi-input multi-output system for a real-world environment, the method further includes: The reinforcement learning controller determines the control signal for real-time adjustment of the multiple-input multiple-output system. Receive the input fine-tuning control signal and determine the fine-tuning control signal as the adaptation and optimization signal; The reinforcement learning controller is trained based on the loss function of the predicted control signal and the adaptation tuning signal until a preset target is reached, thus obtaining a reinforcement learning controller adapted to the multiple-input multiple-output system.

10. A dual-network-assisted reinforcement learning controller training system for multiple-input multiple-output (MIMO) systems, comprising: The simulation environment construction module is used to build a simulation environment that replaces the interaction with the real environment based on the historical operating data of the multi-input multi-output system. The state diagram reconstruction module is used to determine the original state vector in the historical operation data and the correlation matrix between each state variable at different sampling time points in the historical operation data. Based on the correlation matrix, the original state vector is reconstructed to obtain enhanced state features. The state variables include optimization variables and controlled variables. The training module is used in the simulation environment to input the enhanced state features into a reinforcement learning controller constructed by a policy-value network, output a predicted control signal and a reward signal, use the reward signal as training guidance to train the reinforcement learning controller, and after training, obtain a reinforcement learning controller for real-time adjustment of manipulator variables of a multi-input multi-output system for real-world environments.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-9.

12. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-9.

13. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-9.

Images