A process industry control method based on offline learning and online constraint fine-tuning

Abstract

The present application relates to a kind of process industry control method based on offline learning and online constraint fine tuning.Process industry control method includes the historical operation data of the controlled object, constructs the offline data set including state variable, control variable, state transition information and constraint information;Then based on the offline data set, initial control strategy is learned offline to obtain the initial strategy that meets the distribution of historical feasible control behavior;Then the initial strategy is deployed to actual control object, and a small amount of online fine tuning is carried out in combination with real-time operation data;In online control phase, the safety constraint check and correction of strategy output action are carried out, when candidate action does not meet preset safety constraint, action clipping, feasible region projection or back control are executed.The method can reduce the risk of online exploration while improving the adaptability of control system to working condition change and external disturbance, and can balance control accuracy, stability and safety, and is suitable for closed-loop control of water tank liquid level system and other process industry objects.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for process industries, and in particular to a process industry control method based on offline learning and online constraint fine-tuning. It is a closed-loop control method that combines a small amount of online interactive data from actual operation for constrained fine-tuning, and is applicable to process industry objects such as level control, flow control, pressure control, and temperature control. Background Technology

[0002] Process industries are widely present in sectors such as chemical engineering, metallurgy, energy, power generation, water treatment, pharmaceuticals, and equipment manufacturing. Their production processes typically feature continuous operation, strong dynamic coupling, complex sources of disturbance, and stringent operational constraints. For typical process variables such as level, flow rate, temperature, and pressure, control systems not only need to meet requirements for tracking accuracy and response speed, but also need to consider equipment operational safety, energy consumption constraints, smoothness of action, and adaptability to changing operating conditions. Therefore, achieving high-performance control while ensuring safe and stable operation has always been a crucial technical challenge in the process industry.

[0003] Existing control methods in process industries mostly employ proportional-integral-derivative (PID) control, model predictive control (MMC), and rule-based control. These methods can achieve good control performance under conditions of known models and relatively stable operating conditions. However, in actual industrial processes, the controlled objects often face problems such as model uncertainty, time-varying parameters, frequent operating condition changes, and significant external disturbances. This leads to limitations in the control accuracy, adaptability, and multi-constraint handling capabilities of traditional control methods. Especially when the operating environment of the object changes significantly, control strategies relying solely on fixed parameters or fixed models struggle to simultaneously achieve speed, stability, and safety.

[0004] With the development of data-driven control and reinforcement learning technologies, automatically optimizing control strategies using historical operational data and real-time interactive data has gradually become a new technological approach. Reinforcement learning methods can continuously adjust control strategies through interaction with the environment, exhibiting strong adaptive potential for complex nonlinear objects. However, traditional online reinforcement learning typically requires extensive trial and error for strategy improvement. Process industry objects have high safety requirements and significant costs associated with online trial and error. Directly conducting large-scale exploration on actual equipment can easily lead to deterioration in control performance and even trigger risks such as liquid level exceeding limits, equipment exceeding limits, and sudden changes in behavior. Therefore, it is difficult to directly apply it to real industrial processes.

[0005] On the other hand, process industry sites typically accumulate a large amount of historical operating data. This data reflects the state changes and control behaviors of equipment under different operating conditions and disturbances, providing a data foundation for strategy learning. While some existing data-driven methods can train control models or strategies using offline data, strategies learned purely offline are often limited by the coverage of historical data, and are prone to insufficient adaptability when operating conditions change, setpoints are adjusted, or external disturbances intensify. Without a subsequent online correction mechanism, the control strategy struggles to maintain good field performance. Simultaneously, if the online update phase lacks safety constraints on control actions and limits on the magnitude of strategy updates, it can easily lead to excessively rapid strategy drift, affecting system stability.

[0006] Therefore, existing technologies suffer from at least the following problems: First, relying solely on online reinforcement learning carries high risks in online exploration and struggles to meet industrial safety requirements; second, control strategies obtained solely through offline learning have limited adaptability and are ill-suited to long-term responses to changes in operating conditions and external disturbances; and third, during the execution of control actions and the updating of strategies, there is a lack of safety verification, action correction, and rollback mechanisms closely integrated with the operational constraints of the process industry. Therefore, it is necessary to propose a process industry control method that balances offline data utilization, online adaptability, and operational safety to address these technical problems. Summary of the Invention

[0007] To address this, the present invention provides a process industry control method based on offline learning and online constraint fine-tuning. By utilizing historical operating data for offline strategy learning, an initial control strategy that can be directly deployed is obtained. During actual operation, constrained fine-tuning is performed by combining a small amount of online interactive data. At the same time, safety constraint verification, action correction, and backoff mechanisms are introduced during the execution of control actions, thereby reducing the risk of online exploration, improving the adaptability of the control system to changes in operating conditions and external disturbances, and balancing control accuracy, operational stability, and safety.

[0008] To address the aforementioned technical problems, this invention provides a process industry control method based on offline learning and online constraint fine-tuning, comprising: Historical operating data of controlled objects in the process industry are collected. The historical operating data includes the current state quantity, control action, next state quantity and constraint information within the control cycle, and an offline dataset is constructed based on the historical operating data. Based on the control objectives and operational constraints of the controlled objects in the process industry, a control task model is established to determine the state space, action space, performance evaluation indicators, and safety constraint set. The initial control strategy is learned offline using the offline dataset to obtain an initial control strategy suitable for deployment on actual objects, wherein the output of the initial control strategy is constrained by the distribution of historical feasible control behaviors. The initial control strategy is deployed on the controlled objects in the process industry. During actual operation, candidate control actions are generated based on the real-time collected current state quantities and the input of the initial control strategy. The candidate control actions are checked for safety constraints. When the candidate control action meets the safety constraints, it is applied to the controlled object as the final execution action. When the candidate control action does not meet the safety constraints, the candidate control action is modified to obtain the final execution action, or the final execution action is output by the preset backup controller. Based on a small amount of interactive data collected during online operation, the initial control strategy is fine-tuned online to improve its adaptability to changes in operating conditions and external disturbances.

[0009] In one embodiment of the present invention, determining the state space, action space, performance evaluation index, and set of safety constraints includes: The state variables include one or more of the following: current value of the controlled variable, setpoint, control deviation, rate of change of deviation, historical control variables, and disturbance estimate. The performance evaluation indicators include one or more of the following: tracking error, overshoot, settling time, steady-state error, control energy consumption, and control action smoothness. The set of safety constraints includes one or more of the following: the range of controlled variable values, the magnitude of control quantity, the rate of change of control quantity, and the equipment operating boundary constraints.

[0010] In one embodiment of the present invention, the process industry controlled object is a liquid level control object; the liquid level control object is a single-tank liquid level system, a dual-tank liquid level system, or a multi-tank coupled liquid level system; Offline data samples are represented as follows: ; in, Indicates time The state variables, Indicates time Controlling actions, This represents the real-time performance evaluation value. This represents the state quantity at the next moment. This indicates information related to constraints. This represents the total number of offline samples; The state variables are: ; in, This is the current liquid level value. The target liquid level value, For level tracking error, The amount of error change, This is the actuator control quantity at the previous control moment. This refers to the disturbance estimate or the inflow / outflow state quantity; Control action is defined as the valve opening adjustment amount or pump speed adjustment amount; control action is defined as the control increment. ; The actual execution control quantity satisfies: .

[0011] In one embodiment of the present invention, a control task model is established based on the control objectives and operational constraints of the controlled object in the process industry, including: Based on the requirements of the liquid level control task, construct an instant evaluation function: ; in, These are weighting coefficients greater than zero, used to characterize the importance of liquid level error, error change, control energy consumption, and the smoothness of action changes, respectively. The liquid level system must at least meet the following safety constraints: ; in, and These are the lower and upper limits of the allowable liquid level, respectively. and These refer to the allowable range of actuator control quantities; To control the maximum variation of the quantity in a single cycle; Alternatively, a soft constraint penalty term can be introduced to enhance constraint awareness during the policy learning phase, defining the constraint cost function as: ; in, To constrain the penalty weights.

[0012] In one embodiment of the present invention, offline learning of the initial control strategy using the offline dataset includes: The initial control policy is obtained by limiting the degree to which the control policy deviates from the distribution of actionable actions in the offline dataset, thereby reducing the probability of the policy outputting high-risk actions in the early stages of actual deployment, wherein: Offline learning objectives for: ; in, These represent optimization terms related to control performance, used to improve cumulative control benefits; This represents a behavioral constraint, used to constrain the deviation between the policy output and historical control behavior. This indicates penalties related to safety constraints; and These are the weighting coefficients.

[0013] The behavioral constraints are: ; in, For control strategies, Indicates time The state variables, Indicates time Controlling actions, Indicates the offline dataset State-action pairs obtained from mid-sample Perform mathematical expectation calculation. This refers to an offline dataset composed of historical operational data. Behavioral constraints limit the deviation between the policy output and historical actions in offline data, making the initial control policy learned offline closer to historical feasible control behaviors.

[0014] In one embodiment of the present invention, during actual operation, candidate control actions are generated based on the real-time collected current state variables and the input of the initial control strategy, including: During actual operation of the control system, the current control state is collected. Input to the initial control strategy or the current control strategy after fine-tuning. Candidate control actions are obtained. : ; If the control action is in incremental form, then candidate execution control variables are obtained. : ; in, Indicates the current policy version number.

[0015] In one embodiment of the present invention, when the candidate control action does not meet the safety constraints, the candidate control action is modified to obtain the final execution action, or the final execution action is output by a preset safety controller, including: Modifying the candidate control action includes any one or more of the following methods: Amplitude clipping and rate-of-change limitation are applied to candidate control actions. Based on the state prediction results of the controlled object, the candidate control actions are projected into the feasible domain that satisfies the safety constraints. Replace candidate control actions according to preset safety rules; The preset minimum control controller is any one of a rule-based controller, a proportional-integral-derivative controller, or a model prediction controller.

[0016] In one embodiment of the present invention, amplitude clipping and rate-of-change limitation of candidate control actions are performed, including: The following formula is used to constrain the amplitude and rate of change of candidate actions: ; in, Represents the clipping function; Indicates time Candidate execution control variables output by the control strategy; This represents the control quantity after applying amplitude constraints to the candidate execution control quantity; This represents a safe execution control quantity that simultaneously satisfies both amplitude constraints and rate of change constraints. and These represent the lower and upper limits of the allowable control quantity, respectively. Indicates time The actual execution control quantity; This indicates the maximum allowable change in the control quantity within adjacent control cycles; Based on the state prediction results of the controlled object, candidate control actions are projected into the feasible domain that satisfies safety constraints, including: Let the safe feasible region be... Then the corrected safety action satisfies: ; in, Represents the safe and feasible domain Any selectable control value within; That is, under the premise of satisfying the constraints, the safe action that is closest to the candidate action is selected as the final action to be executed.

[0017] In one embodiment of the present invention, online constraint fine-tuning of the initial control strategy includes: The online constraint fine-tuning is performed when a preset trigger condition is met, and the trigger condition includes one or a combination of the following: The control error exceeds the set error threshold for multiple consecutive control cycles; The amplitude of the external disturbance exceeds the set threshold. The operating condition of the controlled object has changed; The deviation between the control strategy output and the actual control requirements exceeds the set deviation threshold; During online fine-tuning, the update frequency, number of steps per update, and maximum allowable parameter change are limited to suppress excessively rapid policy drift. After completing the fine-tuning of online constraints, the performance of the updated control strategy is evaluated. Only when the updated control strategy meets the preset safety and control performance indicators will it be replaced by the current online operating strategy.

[0018] In one embodiment of the present invention, fine-tuning the initial control strategy through online constraints further includes: When the triggering conditions are met, the interaction data during the online operation will be added to the online experience pool. In this process, only data that passes the security constraint verification or data that has been filtered out for anomalies is added to the experience pool to avoid anomalies from adversely affecting policy updates. The objective function for online fine-tuning is as follows: ; in, This represents the loss term obtained by sampling from the offline dataset. This represents the loss term based on new online samples. This indicates a regularization term for parameter variation. This represents the weight coefficient of new online samples. Indicates the weight of the regularization term; The parameter change regularization term is: ; in, and These represent the current strategy parameters and the previous strategy parameters, respectively.

[0019] The technical solution of the present invention has the following advantages compared with the prior art: This invention introduces a two-stage control mechanism that combines offline learning with online constraint fine-tuning. It makes full use of existing historical operating data in the process industry, reducing the need for online trial and error in real objects while obtaining deployable initial control strategies. This effectively reduces the exploration risks associated with directly applying traditional online reinforcement learning to industrial objects.

[0020] This invention performs safety constraint verification and correction on candidate control actions during the online control phase, and switches to a preset backup controller when necessary. It can organically combine the output of reinforcement learning strategy with the operational constraints of the process industry, effectively avoiding risks such as liquid level exceeding limits, control quantity exceeding limits, and sudden action changes, thereby improving the safety and stability of the control system operation.

[0021] This invention enables the initial control strategy to continuously adapt to changes in operating conditions, changes in object parameters, and external disturbances through a constrained fine-tuning mechanism based on a small amount of online interactive data. Compared with purely offline control strategies, it has better field adaptability and dynamic performance; compared with purely online reinforcement learning methods, it significantly reduces sample requirements and deployment risks.

[0022] This invention can simultaneously achieve multiple performance indicators such as tracking accuracy, overshoot suppression, motion smoothness, and energy consumption control. It is applicable to various process industrial control scenarios such as liquid level, flow rate, temperature, and pressure, and has good engineering application value and promotion prospects. Attached Figure Description

[0023] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0024] Figure 1 This is a flowchart illustrating the overall process control method for industrial processes based on offline learning and online constraint fine-tuning according to the present invention.

[0025] Figure 2 This is a closed-loop control block diagram for the online control stage in this invention.

[0026] Figure 3 This is a schematic diagram of the collaborative mechanism between offline learning and online constraint fine-tuning in this invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0028] This embodiment uses a single-tank level control system as an example to illustrate a process industry control method based on offline learning and online constraint fine-tuning. The method is also applicable to dual-tank level systems, multi-tank coupled level systems, and other process industry control objects such as flow rate, pressure, and temperature.

[0029] In this embodiment, the controlled object is a single water tank level control system, and its control objective is to enable the water tank level to quickly and smoothly track the target level under the condition of changes in the set value and external disturbances, and to meet the operational constraints such as the level safety boundary, the actuator action range and the action change rate limit.

[0030] Reference Figures 1 to 3 As shown, the method includes the following steps: S101, Constructing an offline dataset

[0031] First, historical operating data of the water tank system is collected. This historical operating data can come from manual operation records, traditional controller operation records, historical experimental data, or simulation operation data. Each data sample includes at least the current state variable, control action, next state variable, and constraint information. Offline data samples can be represented as follows:

[0032] in, Indicates time The state variables, Indicates time Controlling actions, This represents the real-time performance evaluation value. This represents the state quantity at the next moment. This indicates information related to constraints. This represents the total number of offline samples.

[0033] In this embodiment, the state variables are preferably configured as follows:

[0034] in, This is the current liquid level value. The target liquid level value, For level tracking error, The amount of error change, This is the actuator control quantity at the previous control moment. These are disturbance estimates or inflow / outflow state quantities.

[0035] The control action can be defined as a valve opening adjustment or a pump speed adjustment. In a preferred embodiment, the control action is defined as a control increment:

[0036] The actual execution control quantity satisfies:

[0037] Using incremental control improves control smoothness and facilitates constraints on the rate of change of motion.

[0038] To ensure the quality of offline data, the collected historical operational data can be preprocessed, including removing outliers, filling in missing values, normalizing, classifying and labeling by operating condition, and extracting samples near constraint boundaries. Preferably, the stable operation segment, the setpoint switching segment, and the disturbance recovery segment can be marked separately to improve the generalization ability of the strategy under different operating conditions in the subsequent offline learning stage.

[0039] S102. Establish the control task model and constraint model. Based on the requirements of the liquid level control task, a control performance evaluation index is constructed. To balance liquid level tracking accuracy, overshoot suppression, smooth operation, and control energy consumption, the following real-time evaluation function is adopted in this embodiment:

[0040] in, These are weighting coefficients greater than zero, used to characterize the importance of liquid level error, error variation, control energy consumption, and the smoothness of action changes, respectively. These weights can be set according to the process requirements of the actual controlled object.

[0041] In this embodiment, the liquid level system satisfies at least the following safety constraints:

[0042] in, and These are the lower and upper limits of the allowable liquid level, respectively. and These refer to the allowable range of actuator control quantities; To control the maximum variation of the quantity in a single cycle.

[0043] In some implementations, a soft constraint penalty term can be further introduced to enhance constraint awareness during the policy learning phase. For example, the constraint cost function can be defined as:

[0044] in, To constrain the penalty weights.

[0045] S103. Learning the initial control strategy based on offline datasets. After obtaining the offline dataset, the initial control strategy is learned offline based on the historical operating data to obtain an initial control strategy suitable for field deployment. The core requirement of this initial control strategy is to maintain consistency with the historical distribution of possible actions while learning control patterns, thereby reducing the probability of outputting high-risk actions in the early stages of deployment.

[0046] In this invention, the offline learning objective can be written in the following form:

[0047] in, These represent optimization terms related to control performance, used to improve cumulative control benefits; This represents a behavioral constraint, used to constrain the deviation between the policy output and historical control behavior. This indicates penalties related to safety constraints; and These are the weighting coefficients.

[0048] The behavioral constraint term is preferably written as:

[0049] That is, by limiting the deviation between the policy output and the historical actions in the offline data, the initial control policy learned offline is made closer to the historical feasible control behavior.

[0050] In a preferred embodiment, the offline learning module can be implemented using an actor-critic-based offline reinforcement learning framework; more preferably, it can be implemented using an offline reinforcement learning algorithm with behavioral constraints. After offline learning, an initial control policy is obtained. And it was deployed in a real liquid level control system.

[0051] S104. Generate candidate control actions during the online operation phase. During actual operation of the control system, the current liquid level control status is collected. Input to the initial control strategy or the current control strategy after fine-tuning. Candidate control actions are obtained:

[0052] If the control action is in incremental form, then the candidate execution control variables are obtained:

[0053] in, Indicates the current policy version number.

[0054] S105. Perform safety constraint verification and correction on candidate control actions. To ensure that the online control process meets industrial safety requirements, this invention incorporates a safety constraint processing module between strategy output and control execution. This module determines whether candidate actions satisfy state constraints and action constraints; if satisfied, they are executed directly; otherwise, the candidate actions are modified, and if necessary, the system switches to a backup controller.

[0055] In one implementation, the candidate actions are first limited in terms of amplitude and rate of change:

[0056] in, Represents the clipping function; Indicates time Candidate execution control variables output by the control strategy; This represents the control quantity after applying amplitude constraints to the candidate execution control quantity; This represents a safe execution control quantity that simultaneously satisfies both amplitude constraints and rate of change constraints. and These represent the lower and upper limits of the allowable control quantity, respectively. Indicates time The actual execution control quantity; This indicates the maximum allowable change in the control quantity within adjacent control cycles; In another implementation, candidate actions can be projected into the safe and feasible region based on the state prediction results of the controlled object. Let the safe and feasible region be... Then the corrected safety action satisfies:

[0057] in, Represents the safe and feasible domain Any selectable control value within; That is, under the premise of satisfying the constraints, the safe action that is closest to the candidate action is selected as the final action to be executed.

[0058] In a further implementation, if the current liquid level is detected to be close to the boundary, it is predicted that the liquid level may exceed the boundary at the next moment, or the control action correction exceeds the threshold, then the system switches to the bottom protection controller. For example, PID controllers or model predictive controllers, outputting a safety margin control action. :

[0059] otherwise:

[0060] The above design effectively combines reinforcement learning control strategies with industrial safety mechanisms, avoiding problems such as liquid level exceeding limits, excessive valve action, or equipment exceeding operating limits in actual operation.

[0061] S106. Fine-tuning constraints based on a small amount of online data. To improve the strategy's adaptability to changes in operating conditions and disturbances, this embodiment further introduces an online constraint fine-tuning mechanism. This online constraint fine-tuning employs a hybrid update method combining online and offline data, and only online interaction data that has passed safety constraint verification, or online interaction data that has been filtered out for anomalies, is added to the experience data pool for strategy updates.

[0062] Specifically, online fine-tuning is not performed in every control cycle, but rather preferably when preset trigger conditions are met. These trigger conditions may include one or a combination of the following: 1. The control error exceeds the set error threshold for multiple consecutive control cycles; 2. The set value has changed significantly; 3. The amplitude of external disturbances exceeds the set threshold. 4. The object's operating mode has changed; 5. The deviation between the control strategy output and the actual control requirements exceeds the set deviation threshold.

[0063] When the triggering conditions are met, the interaction data during the online operation will be added to the online experience pool. Preferably, only data that passes security constraint verification or data that has been filtered out for anomalies is added to the experience pool to avoid anomaly data from adversely affecting policy updates.

[0064] The objective function for online fine-tuning can be written as:

[0065] in, This represents the loss term obtained by sampling from the offline dataset. This represents the loss term based on new online samples. This indicates a regularization term for parameter variation. This represents the weight coefficient of new online samples. This indicates the weight of the regularization term.

[0066] The parameter change regularization term can be written as:

[0067] in, and These represent the current strategy parameters and the previous strategy parameters, respectively. By imposing constraints on changes in strategy parameters, it is possible to prevent the strategy from drifting too quickly during online fine-tuning, thereby improving the stability of the online control process.

[0068] In some implementations, the update frequency, number of steps per update, and maximum allowable parameter change can be limited to suppress excessive policy drift. For example, an update may be performed only once after a fixed control window has ended, or only a small number of gradient iterations may be performed, thereby achieving "small sample, low frequency, limited update".

[0069] S107, Update Strategy Acceptance and Replacement After completing the fine-tuning of online constraints, the performance of the updated control strategy is evaluated. Only when the updated control strategy meets the preset safety and control performance indicators will it be replaced by the current online operating strategy.

[0070] Specifically, after online fine-tuning is completed, the new strategy is not directly replaced by the online running strategy. Instead, an acceptance test is conducted first. Acceptance indicators may include whether tracking error has decreased, whether overshoot is under control, whether the number of constraint violations has decreased, and whether changes in control variables are stable.

[0071] If the updated strategy If the preset security and control performance indicators are met, the current online operating strategy is used instead; otherwise, the original strategy is retained and operation continues. This process can be represented as:

[0072] The above-mentioned acceptance mechanism can further prevent strategy degradation caused by insufficient online samples or local anomalies.

[0073] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A process industry control method based on offline learning and online constraint fine-tuning, characterized in that, include: Historical operating data of controlled objects in the process industry are collected. The historical operating data includes the current state quantity, control action, next state quantity and constraint information within the control cycle, and an offline dataset is constructed based on the historical operating data. Based on the control objectives and operational constraints of the controlled objects in the process industry, a control task model is established to determine the state space, action space, performance evaluation indicators, and safety constraint set. The initial control strategy is learned offline using the offline dataset to obtain an initial control strategy suitable for deployment on actual objects, wherein the output of the initial control strategy is constrained by the distribution of historical feasible control behaviors. The initial control strategy is deployed on the controlled objects in the process industry. During actual operation, candidate control actions are generated based on the real-time collected current state quantities and the input of the initial control strategy. The candidate control actions are checked for safety constraints. When the candidate control action meets the safety constraints, it is applied to the controlled object as the final execution action. When the candidate control action does not meet the safety constraints, the candidate control action is modified to obtain the final execution action, or the final execution action is output by the preset backup controller. Based on a small amount of interactive data collected during online operation, the initial control strategy is fine-tuned online to improve its adaptability to changes in operating conditions and external disturbances.

2. The process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, Define the state space, action space, performance evaluation metrics, and set of safety constraints, including: The state variables include one or more of the following: current value of the controlled variable, setpoint, control deviation, rate of change of deviation, historical control variables, and disturbance estimate. The performance evaluation indicators include one or more of the following: tracking error, overshoot, settling time, steady-state error, control energy consumption, and control action smoothness. The set of safety constraints includes one or more of the following: the range of controlled variable values, the magnitude of control quantity, the rate of change of control quantity, and the equipment operating boundary constraints.

3. The process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, The controlled object in the process industry is a liquid level control object; the liquid level control object is a single-tank liquid level system, a dual-tank liquid level system, or a multi-tank coupled liquid level system. Offline data samples are represented as follows: ； in, Indicates time The state variables, Indicates time Controlling actions, This represents the real-time performance evaluation value. This represents the state quantity at the next moment. This indicates information related to constraints. This represents the total number of offline samples; The state variables are: ； in, This is the current liquid level value. The target liquid level value, For level tracking error, The amount of error change, This is the actuator control quantity at the previous control moment. This refers to the disturbance estimate or the inflow / outflow state quantity; Control action is defined as the valve opening adjustment amount or pump speed adjustment amount; control action is defined as the control increment. ； The actual execution control quantity satisfies: 。 4. The process industry control method based on offline learning and online constraint fine-tuning according to claim 3, characterized in that, Based on the control objectives and operational constraints of the controlled objects in the process industry, a control task model is established, including: Based on the requirements of the liquid level control task, construct an instant evaluation function: ； in, These are weighting coefficients greater than zero, used to characterize the importance of liquid level error, error change, control energy consumption, and the smoothness of action changes, respectively. The liquid level system must at least meet the following safety constraints: ； in, and These are the lower and upper limits of the allowable liquid level, respectively. and These refer to the allowable range of actuator control quantities; To control the maximum variation of the quantity in a single cycle; Alternatively, a soft constraint penalty term can be introduced to enhance constraint awareness during the policy learning phase, defining the constraint cost function as: ； in, To constrain the penalty weights.

5. The process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, The initial control policy is learned offline using the aforementioned offline dataset, including: The initial control policy is obtained by limiting the degree to which the control policy deviates from the distribution of actionable actions in the offline dataset, thereby reducing the probability of the policy outputting high-risk actions in the early stages of actual deployment, wherein: Offline learning objectives for: ； in, These represent optimization terms related to control performance, used to improve cumulative control benefits; This represents a behavioral constraint, used to constrain the deviation between the policy output and historical control behavior. This indicates penalties related to safety constraints; and These are the weighting coefficients; The behavioral constraints are: ； in, For control strategies, Indicates time The state variables, Indicates time Controlling actions, Indicates the offline dataset State-action pairs obtained from mid-sample Perform mathematical expectation calculation. This represents an offline dataset composed of historical operational data; behavioral constraints limit the deviation between the policy output and historical actions in the offline data, making the initial control policy learned offline closer to historical feasible control behaviors.

6. The process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, In actual operation, based on the real-time collected current state variables and the input of the initial control strategy, candidate control actions are generated, including: During actual operation of the control system, the current control state is collected. Input to the initial control strategy or the current control strategy after fine-tuning. Candidate control actions are obtained. : ； If the control action is in incremental form, then candidate execution control variables are obtained. : ； in, Indicates the current policy version number.

7. The process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, When the candidate control action does not meet the safety constraints, the candidate control action is modified to obtain the final execution action, or the final execution action is output by the preset backup controller, including: Modifying the candidate control action includes any one or more of the following methods: Amplitude clipping and rate-of-change limitation are applied to candidate control actions. Based on the state prediction results of the controlled object, the candidate control actions are projected into the feasible domain that satisfies the safety constraints. Replace candidate control actions according to preset safety rules; The preset minimum control controller is any one of a rule-based controller, a proportional-integral-derivative controller, or a model prediction controller.

8. The process industry control method based on offline learning and online constraint fine-tuning according to claim 5, characterized in that, Amplitude clipping and rate-of-change limitation are applied to candidate control actions, including: The following formula is used to constrain the amplitude and rate of change of candidate actions: ； in, Represents the clipping function; Indicates time Candidate execution control variables output by the control strategy; This represents the control quantity after applying amplitude constraints to the candidate execution control quantity; This represents a safe execution control quantity that simultaneously satisfies both amplitude constraints and rate of change constraints. and These represent the lower and upper limits of the allowable control quantity, respectively. Indicates time The actual execution control quantity; This represents the maximum allowable change in the control quantity within adjacent control cycles; based on the state prediction results of the controlled object, candidate control actions are projected into the feasible region that satisfies safety constraints, including: Let the safe feasible region be... Then the corrected safety action satisfies: ； in, Represents the safe and feasible domain Any selectable control value within; That is, under the premise of satisfying the constraints, the safe action that is closest to the candidate action is selected as the final action to be executed.

9. The process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, Online constraint fine-tuning of the initial control strategy includes: The online constraint fine-tuning is performed when a preset trigger condition is met, and the trigger condition includes one or a combination of the following: The control error exceeds the set error threshold for multiple consecutive control cycles; The amplitude of the external disturbance exceeds the set threshold. The operating condition of the controlled object has changed; The deviation between the control strategy output and the actual control requirements exceeds the set deviation threshold; During online fine-tuning, the update frequency, number of steps per update, and maximum allowable parameter change are limited to suppress excessively rapid policy drift. After completing the fine-tuning of online constraints, the performance of the updated control strategy is evaluated. Only when the updated control strategy meets the preset safety and control performance indicators will it be replaced by the current online operating strategy.

10. A process industry control method based on offline learning and online constraint fine-tuning according to claim 1, characterized in that, Online constraint fine-tuning of the initial control strategy further includes: When the triggering conditions are met, the interaction data during the online operation will be added to the online experience pool. In this process, only data that passes the security constraint verification or data that has been filtered out for anomalies is added to the experience pool to avoid anomalies from adversely affecting policy updates. The objective function for online fine-tuning is as follows: ； in, This represents the loss term obtained by sampling from the offline dataset. This represents the loss term based on new online samples. This indicates a regularization term for parameter variation. This represents the weight coefficient of new online samples. Indicates the weight of the regularization term; The parameter change regularization term is: ； in, and These represent the current strategy parameters and the previous strategy parameters, respectively.

Images