DMC catalytic tower operation state regulation and control method based on reinforcement learning

Abstract

The invention discloses a DMC catalytic tower operation state regulation and control method based on reinforcement learning, and relates to the technical field of chemical process control, and the method comprises the steps: inputting standardized state data into an initial operation strategy model, carrying out strategy reasoning and action selection, and generating a standardized state vector and a preliminary action suggestion; performing slow frequency strategy calculation on the standardized state vector and the preliminary action suggestion to generate a slow frequency control action; adjusting the opening degree of a feeding valve, the position of a tower pressure adjusting valve and the flow of cooling water through slow frequency control action, and generating a new standardized state vector and an actual action log; performing online parameter updating on the initial operation strategy model by using the new standardized state vector and the actual action log to generate an updated operation strategy model; and reasoning the new standardized state vector by using the updated operation strategy model to generate a real-time regulation and control instruction. According to the invention, the regulation and control precision and adaptability are improved, and the experiment cost and risk are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of chemical process control, in particular to a DMC catalytic column operating state regulation method based on reinforcement learning. BACKGROUND

[0002] In the field of chemical process control, as an important multiphase reaction device, the precise control of the operating state of the catalytic column has a direct impact on the reaction yield, selectivity and energy consumption. With the development of computer technology, control theory and process modeling methods, multivariable control strategies have gradually become an important means of chemical production regulation. Dynamic Matrix Control (DMC) as an advanced control method based on predictive models, through dynamic updating of process historical data and predictive models, realizes accurate adjustment of multiple inputs and multiple outputs, and has been widely used in petrochemical, fine chemical and pharmaceutical industries.

[0003] The existing DMC control method still has certain limitations in practical application, such as limited adaptability to multivariable coupling and nonlinear characteristics under complex working conditions, and lack of rapid adaptive adjustment mechanism when facing dynamic changes or abnormal disturbances. To some extent, it may affect the yield optimization and energy consumption control of the catalytic column, and can provide stable slow frequency control output, but for rapidly changing operating states, the strategy updating and real-time regulation capability still has room for improvement. The existing technology still has complexity and hysteresis problems in integrating historical operation data and real-time state information to generate high-precision control instructions, and it is difficult to achieve comprehensive optimization of multivariable and multi-working conditions. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a DMC catalytic column operating state regulation method based on reinforcement learning to solve the problem of high-precision adaptive regulation under multi-working condition and multi-variable operation of the catalytic column.

[0006] To solve the above technical problems, the present application provides the following technical solutions: The present application provides a DMC catalytic column operating state regulation method based on reinforcement learning, which comprises, Collecting catalytic column operating state data and preprocessing to generate standardized state data, constructing a digital simulation environment and performing reinforcement learning training to generate an initial operating strategy model; Inputting the standardized state data into the initial operating strategy model, performing strategy reasoning and action selection to generate a standardized state vector and a preliminary action suggestion; Slow frequency strategy calculation is performed on the standardized state vector and the preliminary action suggestion to generate slow frequency control actions; A new standardized state vector and an actual action log are generated by adjusting the opening of the feed valve, the position of the tower pressure regulating valve and the cooling water flow through slow frequency control actions; An updated operation strategy model is generated by performing online parameter updating on the initial operation strategy model using the new standardized state vector and the actual action log; Real-time control instructions are generated by reasoning the new standardized state vector using the updated operation strategy model.

[0007] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the catalytic tower operation state data includes tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, reactant and product concentration.

[0008] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the generation of standardized state data includes the following specific steps, Anomaly value detection and filtering processing are performed on the catalytic tower operation state data to generate cleaned operation data; The cleaned operation data is normalized and mapped to a unified dimension range to generate normalized data; The normalized data is feature encoded and state vector constructed, historical state information is integrated and integrity is checked to generate standardized state data.

[0009] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the generation of the initial operation strategy model includes the following specific steps, Based on the standardized state data, a digital simulation environment is established, the catalytic tower operation state data is digitally modeled, and virtual operation data is generated; The virtual operation data is input into the reinforcement learning training process, and the policy parameters are iteratively adjusted through action exploration, state feedback and reward function evaluation to generate a candidate operation strategy set; The candidate operation strategy set is comprehensively evaluated and optimized to generate an initial operation strategy model.

[0010] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the generation of the standardized state vector and the preliminary action suggestion includes the following specific steps, The standardized state data is loaded into the initial operation strategy model according to the time window and variable group, and the state embedding vector sequence is constructed through feature embedding and time sequence encoding; The state embedding vector sequence is inferred and calculated to generate action tendency scores, and a candidate action set is selected through state constraint and historical sequence analysis; Select a preliminary action suggestion from the candidate action set, and integrate it with the state embedding vector sequence to generate a standardized state vector and a preliminary action suggestion.

[0011] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the slow frequency control action is generated, and the specific steps are as follows, Joint analysis is performed on the standardized state vector and the preliminary action suggestion, and the future operation variables are predicted based on the standardized state data to generate a future state sequence. The future state sequence and the preliminary action suggestion are subjected to multi-objective constraint analysis to form a multi-objective constraint optimization scheme, and the candidate slow frequency control action set is generated by solving the reward function. The candidate slow frequency control action set is subjected to timing smoothing and executability verification to generate a slow frequency control action.

[0012] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the slow frequency control action is generated, and the specific steps are as follows, According to the future state sequence and the preliminary action suggestion, the future offset is calculated, and the temperature constraint, pressure constraint, material balance constraint and energy balance constraint are constructed according to the allowed boundary to generate a multi-objective constraint input set. The multi-objective constraint input set is input into the nonlinear programming method to solve the feasibility of the preliminary action suggestion and generate a robust candidate action set. The robust candidate action set is subjected to multi-objective function solving and Pareto frontier analysis to generate a multi-objective constraint optimization scheme.

[0013] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the slow frequency control action is generated, and the specific steps are as follows, The slow frequency control action is mapped and converted into the set values of the feed valve opening, tower pressure regulating valve position and cooling water flow to generate executable control instructions. According to the executable control instructions, the feed valve opening, tower pressure regulating valve position and cooling water flow are adjusted in real time to generate a new standardized state vector. The new standardized state vector and the executable control instructions are recorded and associated to generate an actual action log.

[0014] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the slow frequency control action is generated, and the specific steps are as follows, The new standardized state vector and the actual action log are cleaned, normalized and feature encoded to generate standardized training data; The standardized training data is input into the initial operation strategy model, the policy parameters are adjusted online through error back propagation and gradient update, and a temporary updated policy parameter is generated; The convergence of the temporary updated policy parameters is verified and the safety constraints are checked to generate an updated operation strategy model.

[0015] As a preferred scheme of the DMC catalytic tower operation state regulation method based on reinforcement learning, the specific steps of generating the real-time regulation instruction are as follows, The new standardized state vector is embedded into the state embedding sequence, and input into the updated operation strategy model for inference calculation to generate the action priority distribution; The action priority distribution is subjected to standardized state data constraint and historical state correlation analysis, and the executable action candidate set is selected and evaluated and sorted to generate the regulation action; The regulation action is mapped to the set value of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow, and the real-time regulation instruction is generated.

[0016] The present application has the following advantages: by establishing a digital simulation environment based on standardized state data and performing reinforcement learning training, an initial operation strategy model is generated, which realizes policy optimization for the complex operation characteristics of the catalytic tower in a virtual environment, thereby constructing an adaptive strategy foundation, improving the regulation precision and adaptability, and reducing the experimental cost and risk. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Fig. 1 Flowchart of the DMC catalytic tower operation state regulation method based on reinforcement learning.

[0019] Fig. 2 Flowchart for generating standardized state vector and preliminary action suggestion.

[0020] Fig. 3 Flowchart for generating slow frequency control action.

[0021] Fig. 4 Flowchart for generating real-time regulation instruction. DETAILED DESCRIPTION

[0022] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0023] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0024] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is separate or alternative to other embodiments.

[0025] Reference Figs. 1-4 For one embodiment of the present application, the embodiment provides a DMC catalytic column operating state regulation method based on reinforcement learning, comprising the following steps: S1, collecting catalytic column operating state data and preprocessing to generate standardized state data, constructing a digital simulation environment and performing reinforcement learning training to generate an initial operating strategy model.

[0026] S1.1, the catalytic column operating state data includes tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, reactant and product concentration.

[0027] Specifically, the distributed temperature sensor is used to continuously collect the tower top temperature and tower bottom temperature data at the top and bottom of the catalytic column, and the sampling period is 5 seconds as an example; the pressure transmitter is used to collect the real-time tower pressure data in the tower; the flow meter is used to measure the feed flow and cooling water flow, and the measurement accuracy is ±0.5% as an example; the online analyzer is used to sample and detect the reactant fluid to obtain the reactant concentration and product concentration, and the catalytic column operating state data is integrated.

[0028] S1.2, the catalytic column operating state data is subjected to outlier detection and filtering processing to generate cleaned operating data.

[0029] Specifically, time series analysis is performed on the column top temperature, column bottom temperature, column pressure, feed flow rate, cooling water flow rate, reactant concentration, and product concentration in the catalytic column operating state data, the change rate between adjacent sampling points is calculated by the sliding window method, and the example window length is 5 sampling points; when the change rate of the catalytic column operating state data exceeds the range of three times the standard deviation of the mean value in the same window, the catalytic column operating state data is marked as an abnormal value; the abnormal value is replaced by the median filtering method, and the median value in the adjacent window is smoothed; the digital low-pass filtering method is used to filter the processed column top temperature, column bottom temperature, column pressure, feed flow rate, cooling water flow rate, reactant concentration, and product concentration; the filtered column top temperature, column bottom temperature, column pressure, feed flow rate, cooling water flow rate, reactant concentration, and product concentration are re-aligned according to the time stamp to generate cleaned operating data.

[0030] It should also be noted that the digital low-pass filtering method is a processing method that suppresses signal components above a set cutoff frequency by applying a filter on a discrete time series, thereby retaining low-frequency signal characteristics.

[0031] S1.3, normalize the cleaned operating data and map to a unified dimension range to generate normalized data.

[0032] Specifically, the linear minimum-maximum normalization method is used to calculate the proportion coefficient according to the difference between the maximum value and the minimum value by statistically analyzing the value range in the cleaned operating data; the parameters such as column top temperature, column bottom temperature, column pressure, feed flow rate, cooling water flow rate, reactant concentration, and product concentration are linearly scaled one by one, so that the data values of each parameter are mapped to a unified value interval, for example, within the range of [0, 1]; the boundary check method is used to detect whether the normalized result exceeds the dimension range, and the interpolation correction is performed on the abnormal points to generate normalized data.

[0033] It should also be noted that the specific steps for detecting whether the normalized result exceeds the dimension range are as follows: after completing the linear minimum-maximum normalization, the normalized results of each parameter in the normalized data are traversed in turn to detect whether the normalized results are within the preset dimension range; if the normalized result is less than 0 or greater than 1, the normalized result is marked as dimension out-of-limit data; the boundary correction processing is performed on the dimension out-of-limit data, and the normalized result is adjusted to 0 when it is less than 0, and adjusted to 1 when it is greater than 1; the overall statistical characteristics of the corrected normalized data are recalculated to confirm that all parameters of the normalized data are within the unified dimension range, so as to ensure the effectiveness and consistency of the normalized data, wherein the dimension range refers to the value interval allowed by the normalized data under the unified dimension, for example, the data within the range of [0, 1] indicates that each operating parameter has been mapped to a unified scale.

[0034] S1.4, feature coding on normalized data and state vector construction, integrating historical state information and performing integrity check, generating standardized state data.

[0035] Specifically, when feature coding on normalized data, first, the normalized data is respectively processed by numerical mapping and vectorization according to time window and variable group, to generate feature vectors; the feature vectors are spliced according to time sequence, and historical state information is integrated, the integrity of the state vector sequence composed of normalized data and historical state information is checked, missing values are supplemented and abnormal values are replaced by linear interpolation, and standardized state vectors are constructed, to generate standardized state data.

[0036] It should also be noted that historical state information refers to continuous observation data sequences of tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, reactant concentration and product concentration within a certain time window before the current time point; Variable group refers to a set formed by classifying indicators with similar functions or properties in the catalytic tower running state data, for example, temperature variable group can include tower top temperature and tower bottom temperature, flow variable group can include feed flow and cooling water flow, concentration variable group can include reactant concentration and product concentration, and the data in each variable group is uniformly processed according to the group in feature coding and vectorization processing, to preserve the correlation and time sequence characteristics between various indicators.

[0037] S1.5, based on standardized state data, establish a digital simulation environment, and digitally model the catalytic tower running state data to generate virtual running data.

[0038] Specifically, based on standardized state data, the parameters of catalytic tower running state data such as tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, reactant and product concentration are digitally modeled, and each parameter is mapped to virtual running data that can be simulated according to time sequence and operating conditions, and numerical simulation method is used to calculate the change trend of each operating parameter under different operating conditions, to generate virtual running data.

[0039] It should also be noted that numerical simulation method is to establish mathematical model of physical heat transfer, mass transfer, chemical reaction kinetics and other processes involved in the running process of catalytic tower by computer, to discretize continuous partial differential equations or algebraic equations (for example, using finite difference method, finite volume method or finite element method), and to calculate the change of each operating parameter with time and operating condition by using iterative solution, numerical integration or time step advancement method, to predict the operating behavior of tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, and reactant and product concentration, to generate virtual running data; Working condition refers to the division and description of the operating environment or operating state that may affect the change of process parameters in the operation of the catalytic tower, such as different feed flow levels, cooling water flow settings, tower pressure regulation levels and reaction load conditions. Each working condition corresponds to a specific combination of operating parameters, which is used to reflect the change rule of the operating parameters of the catalytic tower under different operating conditions in the process of digital modeling and numerical simulation.

[0040] S1.6, input the virtual operation data into the reinforcement learning training process, adjust the policy parameters iteratively through action exploration, state feedback and reward function evaluation, and generate a candidate operation strategy set.

[0041] Specifically, the virtual operation data is input into the reinforcement learning training process in time sequence, an action candidate set is generated for the catalytic tower operation state data at each time step, an action is selected and executed according to the action exploration strategy, the state feedback after action execution is recorded, the state feedback and action execution result are input into the reward function for performance evaluation, the policy parameters are adjusted according to the reward function output, the policy parameters are continuously optimized through iterative calculation and gradient update, until a candidate operation strategy set covering different working conditions is generated.

[0042] It should be noted that the policy parameters are a set of adjustable numerical values used to define the action selection rules and state response relationships in the reinforcement learning operation strategy; the reward function is an evaluation standard used to quantify the degree of action selection in reinforcement learning, which guides the optimization of the strategy by assigning values to the action results.

[0043] S1.7, comprehensive evaluation and optimization of the candidate operation strategy set to generate an initial operation strategy model.

[0044] Specifically, the candidate operation strategy set is evaluated according to the preset evaluation index, the control effect, stability and response speed of the candidate operation strategy set under different catalytic tower operating conditions are evaluated, and the evaluation results of the candidate operation strategy set are recorded; the candidate operation strategy set is sorted according to the evaluation results of the candidate operation strategy set, the candidate operation strategy set with performance lower than the predetermined performance standard is removed, and a multi-objective scoring method is used to optimize the policy parameters of the remaining candidate operation strategy set, including preliminary action suggestion weight adjustment and standardized state vector feedback correction to form optimized policy parameters; the optimized policy parameters are associated with the corresponding standardized state vectors of each time step, and the policy parameters and state vectors are associated and combined in sequence to generate an initial operation strategy model.

[0045] It should be noted that the specific steps of the preset evaluation index include: determining the evaluation dimensions of the control accuracy, response speed, stability and energy consumption of the catalytic tower operation; setting quantitative standards for each evaluation dimension, such as tower top temperature and tower bottom temperature deviation, adjustment action completion time, state fluctuation amplitude, and feed and cooling water flow consumption; assigning weights to each evaluation dimension, such as control accuracy 0.4, response speed 0.3, stability 0.2, and energy consumption 0.1; constructing a comprehensive scoring function of the candidate operation strategy set according to the quantitative standards and weights for performance evaluation and sorting; The predetermined performance standard refers to the specific performance requirement preset in the catalytic tower operation control to ensure the operation efficiency, stability and safety, such as reaction product yield not less than 95%, tower pressure fluctuation not more than ±0.5 MPa, and tower top temperature change rate not more than 2 degrees Celsius per minute, etc., for measuring whether the candidate operation strategy set meets the operation requirements; The multi-objective scoring method refers to the method of comprehensively quantitatively evaluating the performance of the candidate operation strategy set under multiple evaluation indexes (such as control effect, stability and response speed, etc.). Usually, by assigning weights to each index or using vector scoring method, the scores of each index are weighted or normalized to obtain a single comprehensive score or a score vector, so as to facilitate the comparison of the advantages and disadvantages of different candidate operation strategy sets, and guide the strategy parameter optimization and selection.

[0046] It should be noted that the dynamic iteration of state-action-reward in the digital simulation environment using the reinforcement learning algorithm can autonomously explore the optimal control path under virtual working conditions, can avoid the safety risks and energy waste caused by direct testing on actual devices, and realizes intelligent modeling and strategy learning of complex catalytic reaction processes.

[0047] S2, input the standardized state data into the initial operation strategy model, perform strategy reasoning and action selection, and generate standardized state vectors and preliminary action suggestions.

[0048] S2.1, load the standardized state data into the initial operation strategy model according to the time window and variable group, construct the state embedding vector sequence through feature embedding and time sequence coding.

[0049] Specifically, the standardized state data is loaded into the initial operation strategy model in turn according to the set time window length and variable group classification order, feature embedding operation is performed on the standardized state data in each time window, and tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, reactant concentration and product concentration, etc. are mapped into high-dimensional feature vectors; time sequence coding is performed on the high-dimensional feature vectors in time sequence, and the time dependence relationship and the correlation information between variables are coded to generate the state embedding vector of each time step; all time steps of the state embedding vector are arranged in time sequence to construct the state embedding vector sequence.

[0050] It should also be noted that setting the length of the time window refers to determining the number of standardized state data included in each time window according to the number of consecutive time steps when processing standardized state data, for example, each time window includes the top temperature, bottom temperature and other data of the past 5 collection time points; the variable group classification order refers to arranging each variable according to the pre-defined order of the catalyst tower operating state data category, for example, arranging the data in the order of top temperature, bottom temperature, tower pressure, then feed flow, cooling water flow, reactant concentration and product concentration, to ensure the consistency and relevance of the data order in the feature embedding and time sequence encoding process.

[0051] S2.2, the state embedding vector sequence is calculated, the action tendency score is generated, and the candidate action set is screened out through state constraint and historical sequence analysis.

[0052] Specifically, the state embedding vector sequence is input into the initial operation strategy model in chronological order, and the neural network layer of the initial operation strategy model is used for inference calculation to obtain the action tendency score corresponding to each possible action; the action tendency score is applied to the pre-set state constraint condition, and the action that violates the constraint condition is removed; at the same time, the action tendency score is combined with the historical state vector sequence to analyze the relevance of the previous action and the current action, for example, calculating the weighted influence of the previous action on the current action, and removing the action that does not meet the historical state relevance; the remaining actions are sorted according to the tendency score to generate a candidate action set, which may include the top five actions with the highest tendency score in the example.

[0053] It should also be noted that the pre-set state constraint condition refers to setting the allowable change range or boundary value for the top temperature, bottom temperature, tower pressure, feed flow, cooling water flow and other key operating parameters according to the safety and process requirements of the catalytic tower operation, which is used to remove control actions that may cause over-standard or instability during action selection, to ensure that the generated candidate action set meets the safety and process feasibility requirements; wherein setting the allowable change range or boundary value refers to the upper and lower limits or acceptable fluctuation interval pre-defined for the operating state variables of the catalytic tower, which is used to constrain the state change during action selection and strategy inference to ensure that the action execution does not exceed the safety or process allowable range; The historical state vector sequence refers to a set of standardized state vectors recorded at several previous time points in chronological order during the operation of the catalytic tower, which is used to reflect the dynamic evolution trend and time-dependent characteristics in strategy inference and action selection to assist in judging the feasibility of the current action and optimizing future control decisions; The neural network layer refers to a layer of calculation units composed of a plurality of artificial neurons, each neuron receives the input of the previous layer, processes the output signal through weight weighting, bias adjustment and nonlinear activation function, and is used for feature extraction, pattern recognition or mapping transformation of the input data, so as to realize the inference calculation from the input to the output.

[0054] S2.3, selecting a preliminary action suggestion from the candidate action set and integrating with the state embedding vector sequence to generate a standardized state vector and a preliminary action suggestion.

[0055] Specifically, the actions in the candidate action set are sorted according to the action tendency score, and the actions that meet the conditions are selected by combining the preset state constraint condition and the historical state vector sequence, the preliminary action suggestion is selected from the filtered actions, and the preliminary action suggestion is integrated with the state embedding vector sequence in time sequence and variable group to form the standardized state vector and the preliminary action suggestion, for example, the selected action value is mapped to the state embedding vector at the corresponding time step, to generate a standardized state vector and a preliminary action suggestion that can completely represent the current running state and action decision.

[0056] For example, the preset state constraint condition can limit the tower top temperature to be in the range of 350-370K, the tower pressure to be in the range of 1.2-1.5MPa, and the feed flow rate to be in the range of 100-120kg / h, and the historical state vector sequence records the action values of the previous 5 time steps, if a certain candidate action will cause the tower top temperature to exceed 370K at the corresponding time step, or there is a significant conflict with the actions of the previous 5 time steps (for example, the continuous large increase in the feed flow rate causes instability), the action will be eliminated; the preliminary action suggestion that meets the constraint condition and has reasonable association with the historical action is selected from the remaining candidate actions.

[0057] S3, performing slow frequency strategy calculation on the standardized state vector and the preliminary action suggestion to generate a slow frequency control action.

[0058] S3.1, performing joint analysis on the standardized state vector and the preliminary action suggestion, and predicting future running variables based on the standardized state data to generate a future state sequence.

[0059] Specifically, the standardized state vector and the preliminary action suggestion are jointly analyzed in time sequence and variable group, the relevant features of each state variable and action variable are extracted, and the tower top temperature, tower bottom temperature, tower pressure, feed flow rate, cooling water flow rate, reactant and product concentration at future time steps are predicted step by step based on historical state information and autoregressive moving average method to generate a future state sequence, for example, the running variables at subsequent time steps are predicted based on the standardized state vector and the preliminary action suggestion at the previous time step to generate a future state sequence.

[0060] It should be noted that the autoregressive moving average method is a time series modeling method, which predicts future data by using the autoregressive relationship of historical data and the moving average relationship of errors. The relevant features refer to the numerical relationship, change trend, time delay effect and mutual dependence between the state variables (such as the tower top temperature, tower bottom temperature, tower pressure, feed flow, cooling water flow, reactant and product concentration) and the action variables (such as the valve opening degree and flow rate set in the preliminary action suggestion).

[0061] S3.2, multi-objective constraint analysis of future state sequence and preliminary action suggestion, forming a multi-objective constraint optimization scheme, and solving through a reward function to generate a candidate slow frequency control action set.

[0062] Specifically, the future state sequence and the preliminary action suggestion are subjected to multi-objective constraint analysis according to the constraint conditions of each operating index, including setting the allowed variation range of the tower top temperature, the tower bottom temperature, the tower pressure, the feed flow and the cooling water flow, and combining with the process limit of the reactant and product concentration to form a multi-objective constraint optimization scheme; according to the preset reward function, the action sequence in the multi-objective constraint optimization scheme is evaluated, and the action that can satisfy each constraint condition and optimize the reward function value is selected to generate a candidate slow frequency control action set, for example, each candidate action contains the specific set value of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow.

[0063] It should be noted that the preset reward function is a mathematical expression for quantitatively evaluating the execution result of the candidate action or strategy in the digital simulation environment, which assigns numerical rewards or penalties according to the degree of deviation of the key operating indexes such as the tower top temperature, the tower bottom temperature, the tower pressure, the feed flow, the cooling water flow and the reactant and product concentration from the expected target, and is used to strengthen the learning process and converge the strategy to the direction of efficient and stable operation.

[0064] S3.2.1, calculate the future offset according to the future state sequence and the preliminary action suggestion, and construct temperature constraints, pressure constraints, material balance constraints and energy balance constraints according to the allowed boundary to generate a multi-objective constraint input set.

[0065] Specifically, according to the future state sequence and the preliminary action suggestion, the values of the tower top temperature, the tower bottom temperature, the tower pressure, the feed flow, the cooling water flow, the reactant concentration and the product concentration are extracted step by step, and the offset of each variable relative to the current state is calculated; for each variable, the offset is compared with the allowed boundary to generate temperature constraints, pressure constraints, material balance constraints and energy balance constraints, respectively, and the multi-objective constraint input set is generated in a unified arrangement according to the time step order and variable type.

[0066] It should be noted that the allowable boundary refers to the safe range or limit range of the control variables of the catalytic tower during operation, for example, the allowable boundary of the feed flow is 10-50 cubic meters / hour, the allowable boundary of the tower pressure is 1.0-2.5 megapascals, and the allowable boundary of the tower top temperature is 250-350 degrees Celsius, which is used to ensure that the control action is within the executable and safe range.

[0067] S3.2.2, input the multi-objective constraint input set into the nonlinear programming method, and perform feasibility solving on the preliminary action suggestion to generate a robust candidate action set.

[0068] Specifically, after arranging the multi-objective constraint input set by time step and variable dimension, the nonlinear programming method is input, and the feasibility of each preliminary action suggestion is solved, and the executable range under the temperature constraint, pressure constraint, material balance constraint and energy balance constraint is calculated; the action results under different disturbance scenarios are compared item by item, the action combinations that meet the temperature constraint, pressure constraint, material balance constraint and energy balance constraint in each disturbance scenario are calculated, the action combinations that are feasible under all disturbance scenarios are screened out, and are sorted according to the time sequence of the action occurrence, to generate a robust candidate action set arranged in time sequence.

[0069] It should be noted that the nonlinear programming method is a method for solving problems containing nonlinear relationships by using mathematical optimization methods to obtain the variable values that make the objective function optimal; The executable range refers to the variable value range of the catalytic tower control operation that can be reliably implemented under actual mechanical and process conditions, for example, the executable range of the feed valve opening degree is 0%-100%, the executable range of the cooling water flow is 5-30 cubic meters / hour, and the executable range of the tower pressure regulating valve position is 0-90 degrees, which is used to ensure that the control action can be actually executed and will not exceed the mechanical capacity or safety limit.

[0070] S3.2.3, multi-objective function solving and Pareto front analysis are performed on the robust candidate action set to generate a multi-objective constraint optimization scheme.

[0071] Specifically, the non-dominated sorting genetic algorithm is used to calculate the multi-objective function values of the conversion rate, energy consumption and safety index for each action combination in the robust candidate action set, and the non-dominated solution is identified through Pareto front analysis; the identified non-dominated solution is filtered according to the temperature constraint, pressure constraint, material balance constraint and energy balance constraint, and is sorted according to the priority order of each objective function to generate a multi-objective constraint optimization scheme.

[0072] It should be noted that the non-dominated sorting genetic algorithm is a multi-objective optimization method based on the evolutionary idea, through the non-dominated sorting and congestion comparison of the population individuals, the optimal solution set on the Pareto frontier is found by selecting, crossing and mutating generation by generation.

[0073] S3.3, timing smoothing and executability checking are performed on the candidate slow frequency control action set to generate a slow frequency control action.

[0074] Specifically, the candidate slow frequency control action set is smoothed in time sequence order, the sliding average method is used to adjust the numerical change of each action at consecutive time points, and the mutation and discontinuous points are eliminated. The smoothed action is checked according to the executable range of the tower top temperature, the tower bottom temperature, the tower pressure, the feed flow, the cooling water flow, and the concentrations of reactants and products, and the action exceeding the allowed boundary is eliminated or adjusted to generate a slow frequency control action satisfying the timing continuity and operation constraints.

[0075] It should be noted that the slow frequency control action refers to a medium-low frequency control instruction with a control execution frequency lower than the real-time control layer and higher than the long-term scheduling layer. The typical frequency range is generally, for example, 0.01 Hz-0.1 Hz (i.e., executed once every 10-100 seconds). In industrial applications, the slow frequency control action is mainly used for dynamic optimization and trend correction of slow-changing processes such as catalytic tower temperature, tower pressure, and material flow. By using a lower execution frequency, smooth transition is achieved, mechanical wear and energy consumption fluctuations caused by frequent valve action are reduced, and a stable state reference is provided for the real-time control layer, thereby achieving a dynamic balance between control accuracy, mechanical life, and operation economy in industrial continuous production.

[0076] S4, the slow frequency control action is used to adjust the feed valve opening, the tower pressure regulating valve position, and the cooling water flow to generate a new standardized state vector and an actual action log.

[0077] S4.1, the feed valve opening, the tower pressure regulating valve position, and the cooling water flow are obtained by closed-loop mapping and adjusting the slow frequency control action.

[0078] Specifically, the slow frequency control action is classified according to the control variables, and the control ranges of the feed valve opening, the tower pressure regulating valve position, and the cooling water flow are correspondingly adjusted. The slow frequency control action is mapped to the adjustment instruction of each control variable, for example, the value of the slow frequency control action is mapped to the executable range of the feed valve opening 0%-100%, the executable range of the tower pressure regulating valve position 0-90 degrees, and the executable range of the cooling water flow 5-30 cubic meters / hour through proportional conversion. The feed valve opening, the tower pressure regulating valve position, and the cooling water flow are adjusted in closed loop, and the mapping relationship is corrected according to the feedback of the actual executed state vector to generate the feed valve opening, the tower pressure regulating valve position, and the cooling water flow.

[0079] It should also be noted that the control variable classification refers to the way of classifying and managing each adjustable control parameter in the operation of the catalytic tower according to the function or adjustment object, so as to respectively map and adjust different types of variables in the control strategy; for example, the slow frequency control action is divided into three categories: feed valve opening degree category, tower pressure regulating valve position category and cooling water flow category, each category corresponds to a specific control device and adjustment range, so as to generate corresponding adjustment instructions and execution actions for each control variable.

[0080] S4.2, the slow frequency control action is mapped in a closed loop to generate set values of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow, and executable control instructions are generated.

[0081] Specifically, the slow frequency control action is mapped to the executable range of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow according to the control variable classification, for example, the slow frequency control action value is mapped to the set values of the feed valve opening degree 0%~100%, the tower pressure regulating valve position 0~90 degrees and the cooling water flow 5~30 cubic meters / hour through proportional conversion; the mapped feed valve opening degree, tower pressure regulating valve position and cooling water flow are sequentially checked in a closed loop with the standardized state data, and the mapping relationship is corrected according to the standardized state data feedback to generate executable control instructions.

[0082] S4.3, according to the executable control instructions, the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow are adjusted in real time to generate a new standardized state vector.

[0083] Specifically, according to the executable control instructions, the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow are sequentially adjusted to the instruction set values; in the adjustment process, the tower top temperature, the tower bottom temperature, the tower pressure, the feed flow, the cooling water flow, the reactant concentration and the product concentration are collected in real time, the corresponding variables in the state vector are updated according to the time sequence, and a new standardized state vector is generated.

[0084] It should also be noted that the instruction set value is a specific operation parameter obtained by closed-loop mapping according to the slow frequency control action calculation result, which is used to directly adjust the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow, and is an executable value for achieving the control target, for example, the feed valve opening degree is 50%, the tower pressure regulating valve position is 45 degrees, and the cooling water flow is 20 cubic meters / hour.

[0085] S4.4, the new standardized state vector and the executable control instructions are recorded and associated to generate an actual action log.

[0086] Specifically, the new standardized state vector and executable control instructions are recorded in chronological order, the feed valve opening, tower pressure regulating valve position, cooling water flow and corresponding standardized state vector are extracted, the standardized state vector at each time is associated with the corresponding executable control instruction, and an actual action log is generated, for example, the standardized state vector value at a certain time is recorded as [0.45, 0.62, 0.51, 0.48, 0.55], the corresponding feed valve opening is 50%, the tower pressure regulating valve position is 45 degrees, and the cooling water flow is 20 cubic meters / hour.

[0087] S5, using the new standardized state vector and the actual action log, the initial operation strategy model is updated online to generate an updated operation strategy model.

[0088] S5.1, the new standardized state vector and the actual action log are cleaned, normalized and feature encoded to generate standardized training data.

[0089] Specifically, the feed valve opening, tower pressure regulating valve position, cooling water flow and corresponding standardized state vector recorded in the actual action log are extracted in chronological order, abnormal values are removed according to the set abnormality detection rule, continuous data is normalized, the normalized feed valve opening, tower pressure regulating valve position, cooling water flow and standardized state vector are feature encoded to generate feature vectors of uniform dimension, and the feature vectors at each time point are arranged in sequence to generate standardized training data.

[0090] It should also be noted that the abnormality detection rule is a judgment standard formulated for the change characteristics of the feed valve opening, tower pressure regulating valve position, cooling water flow and standardized state vector. When the change amplitude of continuous time series data exceeds three times the standard deviation of the mean value in the same window, or the single-point data exceeds the historical observation range, the data at the time point is marked as abnormal for cleaning and removal. Through the abnormality detection rule, it can be ensured that the data sequence used for training is continuous and does not contain sudden abnormalities, maintaining the reliability and consistency of the data.

[0091] S5.2, the standardized training data is input into the initial operation strategy model, and the policy parameters are adjusted online through error back propagation and gradient update to generate temporary updated policy parameters.

[0092] Specifically, the standardized training data is loaded into the input layer of the initial operation strategy model in chronological order and variable groups to generate a prediction output, and the actual action label corresponding to the standardized training data is calculated for error, the expression is: ; Wherein, represents the prediction error at the th time step, represents the number of action dimensions, represents the actual action value at the i-th time step and the j-th action dimension in the standardized training data, represents the action value predicted by the initial running policy model at the i-th time step and the j-th action dimension, represents the time step index, represents the action dimension index; The gradient is calculated by back propagation according to the error, and the expression is: ; wherein, represents the update amount of the policy parameter, represents the policy parameter in the initial running policy model, represents the learning rate, for example, 0.01, represents the error of the gradient of the policy parameter , and represents the partial derivative; According to the gradient information, the policy parameter in the initial running policy model is iteratively updated to generate the temporary updated policy parameter, for example, the learning rate is 0.01 to adjust the gradient step, until a cycle of updating is completed.

[0093] S5.3, the convergence of the temporary updated policy parameter is verified and the safety constraint is checked, and the updated running policy model is generated.

[0094] ​​​​Specifically, the temporary update strategy parameter is input into the convergence verification process, the reserved verification set in the standardized training data is used to perform forward inference on the temporary update strategy parameter, the error sequence of a plurality of consecutive training rounds is calculated, and the moving average and relative change rate of the error are calculated to determine the convergence (for example, the relative change rate value is 0.001), and at the same time, the Euclidean norm method is used to calculate the gradient norm of the strategy parameter; after the convergence determination is completed, the temporary update strategy parameter that passes the convergence determination is used to perform forward rolling simulation on a plurality of representative working conditions in a digital simulation environment, the operating indexes such as the top temperature, the bottom temperature, the tower pressure, the feed flow, the cooling water flow, the reactant concentration and the product concentration obtained by simulation are checked item by item according to the set allowable boundary and executable range, and it is verified whether the action amplitude of the simulation action sequence exceeds the executable range of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow; when the convergence verification and the safety constraint verification both pass, the temporary update strategy parameter is used as the final strategy parameter, an updated running strategy model is formed, and output; when any one of the verifications fails, the temporary update strategy parameter is subjected to rollback or strategy parameter projection processing, for example, linear interpolation according to the rollback coefficient 0.5 and the previous parameter or clipping of the out-of-limit parameter, and the convergence verification and the safety constraint verification are repeated until the requirements are met, and the updated running strategy model is generated.

[0095] S6, using the updated running strategy model, a new standardized state vector is inferred to generate real-time control instructions.

[0096] S6.1, the new standardized state vector is subjected to feature embedding, mapped into a state embedding sequence, and input into the updated running strategy model for inference calculation to generate an action priority distribution.

[0097] Specifically, the new standardized state vector is loaded according to the set time window length and variable group classification order and subjected to feature embedding operation, and the feature vector of each time step is output; the feature vectors of each time step are time-sequentially encoded and spliced to generate a state embedding sequence; the state embedding sequence is input into the updated running strategy model and subjected to forward inference calculation to obtain the original score value of each candidate action; the original score value is normalized by softmax calculation to generate an action priority distribution.

[0098] It should also be noted that the softmax calculation is to map each element of the input vector through the exponential function and normalize it, so that each element value of the output vector is between 0 and 1 and the sum is 1, which is used to represent the probability distribution.

[0099] S6.2, the action priority distribution is subjected to standardized state data constraint and historical state correlation analysis, and an executable action candidate set is screened out and evaluated and sorted to generate a control action.

[0100] Specifically, the action priority distribution is traversed by action entry one by one, for each action entry, the action entry is mapped into control set values of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow rate, and it is determined whether the mapped control set values are within the executable range and the allowable boundary, if the control set values exceed the executable range or the allowable boundary, it is marked as a violation item and counted into the violation count; For each action entry, a historical state correlation score is calculated, specifically, the matching degree of the action and the subsequent state change is calculated using a correlation coefficient or a regression coefficient with the historical state vector sequence and the corresponding past action record as samples, and is normalized into a compatibility score; then for each action entry, an evaluation score is calculated according to pre-defined evaluation dimensions, including a control accuracy index, a response speed index, a stability index and an energy consumption index, wherein the control accuracy index is based on the deviation estimated value of the mapped control set value on the future state sequence, the response speed index is based on the statistical value of the corresponding completion time of the action entry in the historical state vector sequence, the stability index is based on the fluctuation amplitude of the historical state vector sequence, and the energy consumption index is based on the scalar sum of the feed flow rate and the cooling water flow rate; The action priority, the violation count, the compatibility score and the score of each evaluation dimension are combined into a comprehensive score according to predetermined weights; all action entries are sorted in descending order according to the comprehensive score, and a certain number of action entries at the top of the sorting are selected as an executable action candidate set, and after the executable action candidate set is reviewed for executability and checked for time sequence consistency, a control action is generated.

[0101] It should be noted that the predetermined weight refers to a proportionality coefficient set according to the influence degree of each evaluation dimension on the overall operation performance, for example, the control accuracy weight is 0.4, the response speed weight is 0.3, the stability weight is 0.2 and the energy consumption weight is 0.1; The correlation coefficient or the regression coefficient is used to quantify the strength and direction of the linear relationship between two variables, the correlation coefficient represents the numerical size and positive or negative direction of the correlation between variables, and the regression coefficient represents the average change amplitude of the dependent variable corresponding to a unit change of the independent variable.

[0102] S6.3, mapping the control action into the set values of the feed valve opening degree, the tower pressure regulating valve position and the cooling water flow rate to generate real-time control instructions.

[0103] Specifically, the control action is read by action number one by one, for each control action, the weight parameters in the action vector are analyzed, and the feed valve control component, the tower pressure regulating valve control component and the cooling water flow control component are extracted respectively; the numerical range in the action vector is normalized to calculate the corresponding set value of each component, the expression is: ; Wherein, represents the first The set value of each control component, Indicates the first The minimum permissible setting value for each control component. Indicates the first The maximum permissible setting value for each control component. This represents the normalized weight coefficients corresponding to the control action vector. Indicates the index of the control component; The obtained feed valve opening setting value, tower pressure regulating valve position setting value, and cooling water flow setting value are sorted by timestamp and combined into a real-time control parameter set; the real-time control parameter set is encoded into a unified control data structure and the execution time information is added to generate a real-time control command.

[0104] In summary, this invention achieves strategy optimization of the complex operating characteristics of the catalytic tower in a virtual environment by establishing a digital simulation environment based on standardized state data and conducting reinforcement learning training to generate an initial operating strategy model. This constructs an adaptive strategy foundation, improves control accuracy and adaptability, and reduces experimental costs and risks.

[0105] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for regulating the operating state of a DMC catalytic column based on reinforcement learning, characterized in that: include, Collect the operating status data of the catalytic tower, preprocess it to generate standardized status data, build a digital simulation environment, and perform reinforcement learning training to generate an initial operating strategy model; Standardized state data is input into the initial running strategy model to perform strategy reasoning and action selection, generating standardized state vectors and preliminary action suggestions; Slow-frequency strategy calculations are performed on the standardized state vector and preliminary action suggestions to generate slow-frequency control actions; By adjusting the opening of the feed valve, the position of the tower pressure regulating valve, and the cooling water flow rate through slow-frequency control actions, new standardized state vectors and actual action logs are generated. By using the new standardized state vector and actual action logs, the initial running strategy model is updated online to generate the updated running strategy model; By using the updated operational strategy model, reasoning is performed on the new standardized state vector to generate real-time control commands.

2. The method of claim 1, wherein the method is based on reinforcement learning. The operating status data of the catalytic tower includes the top temperature, bottom temperature, tower pressure, feed flow rate, cooling water flow rate, and the concentrations of reactants and products.

3. The method of claim 1, wherein the method is based on reinforcement learning. The specific steps for generating standardized state data are as follows: Outlier detection and filtering are performed on the catalytic tower operating status data to generate post-cleaning operating data. The cleaned running data is normalized and mapped to a unified dimension range to generate normalized data. Normalized data is feature-encoded and state vectors are constructed. Historical state information is integrated and its integrity is verified to generate standardized state data.

4. The method of claim 1, wherein the method is based on reinforcement learning. The specific steps for generating the initial running strategy model are as follows: Based on standardized status data, a digital simulation environment is established to digitally model the catalytic tower operating status data and generate virtual operating data. The virtual running data is input into the reinforcement learning training process. Through action exploration, state feedback and reward function evaluation, the policy parameters are iteratively adjusted to generate a set of candidate running policies. A comprehensive evaluation and optimization of the candidate operating strategy set is performed to generate an initial operating strategy model.

5. The method of claim 1, wherein the method is based on reinforcement learning. The specific steps for generating the standardized state vector and preliminary action suggestions are as follows. Standardized state data is loaded into the initial running strategy model according to time windows and variable groups, and a state embedding vector sequence is constructed through feature embedding and temporal encoding. Inference calculations are performed on the state embedding vector sequence to generate action tendency scores, and candidate action sets are selected through state constraints and historical sequence analysis. Preliminary action suggestions are selected from the candidate action set and integrated with the state embedding vector sequence to generate standardized state vectors and preliminary action suggestions.

6. The method of claim 1, wherein the method is based on reinforcement learning. The specific steps for generating the slow-frequency control action are as follows. The standardized state vector and preliminary action suggestions are jointly analyzed, and the future operating variables are predicted by combining the standardized state data to generate a future state sequence. The future state sequence and preliminary action suggestions are subjected to multi-objective constraint analysis to form a multi-objective constraint optimization scheme, which is then solved by the reward function to generate a set of candidate slow-frequency control actions. The candidate slow-frequency control action set is subjected to timing smoothing and executability verification to generate slow-frequency control actions.

7. The method of claim 6, wherein the method further comprises: determining a current state of the DMC catalytic column; and determining a target state of the DMC catalytic column based on the current state of the DMC catalytic column. The process of performing multi-objective constraint analysis on the future state sequence and preliminary action suggestions to form a multi-objective constraint optimization scheme involves the following specific steps. According to the future state sequence and the preliminary action recommendation, a future offset is calculated, and a multi-objective constraint input set is generated according to the temperature constraint, the pressure constraint, the material balance constraint and the energy balance constraint according to the allowable boundary; The multi-objective constraint input set is input into a nonlinear programming method to solve the feasibility of the preliminary action recommendation, and a robust candidate action set is generated; The multi-objective function is solved and the Pareto frontier is analyzed for the robust candidate action set, and a multi-objective constraint optimization scheme is generated.

8. The method of claim 1, wherein the method is based on reinforcement learning. The new standardized state vector and the actual action log are generated, and the specific steps are as follows, The slow frequency control action is closed loop mapped and converted into the set value of the feed valve opening, the tower pressure regulating valve position and the cooling water flow, and an executable control instruction is generated; According to the executable control instruction, the feed valve opening, the tower pressure regulating valve position and the cooling water flow are adjusted in real time, and a new standardized state vector is generated; The new standardized state vector and the executable control instruction are recorded and associated to generate an actual action log.

9. The method of claim 1, wherein the method is based on reinforcement learning. The updated running strategy model is generated, and the specific steps are as follows, The new standardized state vector and the actual action log are cleaned, normalized and feature encoded to generate standardized training data; The standardized training data is input into the initial running strategy model, and the policy parameters are adjusted online through error back propagation and gradient update to generate temporary updated policy parameters; The convergence of the temporary updated policy parameters is verified and the safety constraint is checked to generate an updated running strategy model.

10. The method of claim 1, wherein the method is based on reinforcement learning. The real-time regulation instruction is generated, and the specific steps are as follows, The new standardized state vector is feature embedded and mapped into a state embedding sequence, and input into the updated running strategy model for inference calculation to generate an action priority distribution; The action priority distribution is analyzed for standardized state data constraint and historical state correlation, and an executable action candidate set is selected and evaluated and sorted to generate a regulation action; The regulation action is mapped into the set value of the feed valve opening, the tower pressure regulating valve position and the cooling water flow to generate a real-time regulation instruction.

Images