Method, device, equipment and medium for generating control action execution suggestion of process industry control system

Abstract

The application discloses a control action execution suggestion generation method and device of a process industry control system, equipment and a medium, relates to the technical field of artificial intelligence, and comprises the following steps: taking a verified world model of the process industry control system as a target world model; inputting a strategy training sample into a strategy model, outputting a control action, and processing the control action; inputting the processed control action into the target world model to obtain a multi-step rolling prediction result; screening a target strategy model; inputting real-time running data of the process industry control system into the target strategy model, outputting a target control action, verifying the target control action, and generating a control action execution suggestion of the process industry control system based on a verification result and the target control action, so as to control the process industry control system, avoid online trial and error in a real scene and online solution of a long-time domain optimization problem in each cycle, and improve the accuracy of control action execution suggestion generation of the process industry control system.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, equipment, and medium for generating control action execution suggestions for process industry control systems. Background Technology

[0002] Training on control strategies in the process industry faces the following challenges: (1) Online trial and error is not allowed in real-world settings: If reinforcement learning agents explore in real devices, it may cause water quality to exceed standards, equipment to exceed limits, product quality to fluctuate, energy consumption to be abnormal or safety risks to occur.

[0003] (2) Traditional online optimization calculation and maintenance costs are high: Model predictive control and other methods usually require online solving of optimization problems in each control cycle, which places high demands on model accuracy, constraint modeling and real-time performance of the solver. When faced with multivariable, long time delay, nonlinear and frequent operating condition switching, online solving is prone to slowing down or requires a lot of manual parameter tuning.

[0004] (3) Ordinary supervised learning strategies lack feedback on action consequences: Directly fitting historical operator actions can obtain a fast strategy, but it can only imitate historical behavior and cannot actively optimize future control effects through objective functions. It is also difficult to handle new target settings that rarely appear in history.

[0005] (4) Most existing world model reinforcement learning solutions are geared towards games, robots or general tasks, and have not disclosed single-step target interfaces, dynamic reachability gating, action change constraints, constraint boundaries and one-click deployment output mechanisms suitable for process industries.

[0006] As can be seen from the above, how to avoid online trial and error in real-world scenarios and solve long-term optimization problems online in each cycle, reduce the risk of the strategy driving the system to exceed limits in order to track the target, and improve the accuracy of the control action execution suggestions generated by the process industry control system are problems that need to be solved in this field. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for generating control action execution suggestions for process industry control systems. This method avoids real-world online trial and error and the need for solving long-term time-domain optimization problems online in each cycle, reduces the risk of the strategy driving the system beyond its limits to track the target, and improves the accuracy of generating control action execution suggestions for process industry control systems. The specific solution is as follows: In a first aspect, this application discloses a method for generating control action execution suggestions for a process industry control system, including: Construct a world model of the process industry control system, verify the world model, and use the verified world model as the target world model; Obtain policy training samples, input the policy training samples into the policy model, output control actions, process the control actions, and obtain processed control actions. The processed control action is input into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step. The state vector of the next rolling step is used as the current state in the strategy training sample. Then, the process jumps to the step of inputting the strategy training sample into the strategy model until the preset termination condition is met to obtain the multi-step rolling prediction result. Based on the multi-step rolling prediction results, an objective function is constructed, and the objective function is used to update the model parameters of the strategy model to obtain the updated strategy model. The updated strategy model is validated to obtain validation metrics. Based on the validation metrics, the target strategy model is selected from the updated strategy model. The real-time operating data of the process industry control system is input into the target strategy model, the target control action is output, the target control action is verified, the verification result is obtained, and based on the verification result and the target control action, the control action execution suggestion of the process industry control system is generated so that the process industry control system can be controlled based on the control action execution suggestion.

[0008] Optionally, the world model for constructing a process industry control system includes: Construct a set of time-series operation variables for the process industry control system; Based on the set of time-series runtime variables, a set of bidirectional potential hybrid edges and a hybrid risk context for the set of bidirectional potential hybrid edges are determined; the hybrid risk context is the hybrid risk information corresponding to the set of bidirectional potential hybrid edges. Using the set of time-series operating variables and the hybrid risk context, the initial world model is trained to obtain the world model of the process industry control system.

[0009] Optionally, the world model for constructing a process industry control system includes: Acquire time-series data of the process industry control system, and determine a target state variable based on the time-series data; the target state variable is the state variable of the next cycle of the current cycle. Generate parent variable aggregation values ​​for each of the target state variables; Perform single-step state prediction on the aggregated amount of the parent variable to obtain the state prediction result for the next cycle; Using the state prediction result of the next cycle as the initial state, multi-step rolling prediction is performed on the time series data to obtain the multi-step rolling prediction result. Using the target state variable, the state prediction result of the next cycle, and the multi-step rolling prediction result, the initial world model is trained to obtain the world model of the process industry control system.

[0010] Optionally, the step of obtaining policy training samples, inputting the policy training samples into the policy model, outputting control actions, and processing the control actions to obtain processed control actions includes: Obtain strategy training samples for the process industry control system; the strategy training samples include current state, target setpoint, constraints, historical actions, external operating conditions, and historical error information; The policy training samples are input into the policy model, and the output control action is generated. Based on preset upper and lower limits for control actions, the control actions are trimmed. Based on the preset maximum change in motion, the rate of change of the control motion after trimming is limited. The control action after the rate of change is limited is smoothed to obtain the processed control action.

[0011] Optionally, the step of inputting the processed control action into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step, using the state vector of the next rolling step as the current state in the strategy training sample, and then jumping to the step of inputting the strategy training sample into the strategy model, until a preset termination condition is met to obtain a multi-step rolling prediction result, includes: The threshold for the number of rolling steps is determined based on the control cycle and process response time of the process industry control system. The current state, external conditions, and post-processing control actions in the strategy training samples are input into the target world model for prediction, thereby obtaining the state vector of the process industry control system in the next rolling step. The state vector of the next scrolling step is used as the current state in the policy training sample. The policy training sample is updated using the state vector of the next scrolling step and the processed control action. Then, the process jumps to the step of inputting the policy training sample into the policy model until the current number of scrolling steps is greater than the number of scrolling steps threshold, and the multi-step scrolling prediction result is obtained.

[0012] Optionally, the step of constructing an objective function based on the multi-step rolling prediction results, and using the objective function to update the model parameters of the policy model to obtain the updated policy model, includes: An objective function is constructed based on the multi-step rolling prediction results; the objective function includes target tracking loss, safety constraint loss, motion smoothing loss, and dynamic reachability loss; Based on the objective function, the policy model is backpropagated, and the model parameters of the policy model are updated to obtain the updated policy model.

[0013] Optionally, the step of inputting real-time operating data of the process industry control system into the target strategy model, outputting target control actions, verifying the target control actions, obtaining verification results, and generating control action execution suggestions for the process industry control system based on the verification results and the target control actions includes: Acquire real-time operating data from process industry control systems; The real-time running data is sorted and normalized to obtain processed data; The processed data is input into the target policy model for forward reasoning to obtain the forward reasoning result. The forward reasoning result is then restored to obtain the restored action. The restored action is preprocessed to obtain the preprocessed action; The preprocessed actions are validated for legality, rate of change, and constraints to obtain the validation results. If the validation results show that the validity validation, rate of change validation, and constraint validation all pass, then control action execution suggestions for the process industry control system will be generated.

[0014] Secondly, this application discloses a control action execution suggestion generation device for a process industry control system, comprising: The verification module is used to construct a world model of the process industry control system, verify the world model, and use the verified world model as the target world model. The processing module is used to acquire policy training samples, input the policy training samples into the policy model, output control actions, process the control actions, and obtain processed control actions. The multi-step rolling prediction module is used to input the processed control action into the target world model for prediction, obtain the state vector of the process industry control system in the next rolling step, use the state vector of the next rolling step as the current state in the strategy training sample, and then jump to the step of inputting the strategy training sample into the strategy model until the preset termination condition is met, and obtain the multi-step rolling prediction result. The model parameter update module is used to construct an objective function based on the multi-step rolling prediction results, and use the objective function to update the model parameters of the strategy model to obtain the updated strategy model. The filtering module is used to verify the updated strategy model, obtain verification indicators, and filter out the target strategy model from the updated strategy model based on the verification indicators. The execution suggestion generation module is used to input the real-time operating data of the process industry control system into the target strategy model, output the target control action, verify the target control action, obtain the verification result, and generate control action execution suggestions for the process industry control system based on the verification result and the target control action, so as to control the process industry control system based on the control action execution suggestions.

[0015] Thirdly, this application discloses an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned method for generating control action execution suggestions for a process industry control system.

[0016] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned method for generating control action execution suggestions for a process industry control system.

[0017] Beneficial Effects: This application provides a method for generating control action execution suggestions for process industry control systems. The method includes constructing a world model of the process industry control system, validating the world model, and using the validated world model as the target world model to avoid direct experimentation on real systems; obtaining strategy training samples, inputting the training samples into the strategy model, outputting control actions, processing the control actions to obtain processed control actions, thus improving the stability and feasibility of the training process; inputting the processed control actions into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step, using the state vector of the next rolling step as the current state in the strategy training samples, and then jumping to the step of inputting the strategy training samples into the strategy model until a preset termination condition is met, obtaining multi-step rolling prediction results. This effectively improves the strategy's ability to cope with process lag and accumulated disturbances. Based on multi-step rolling prediction... The test results are used to construct an objective function, which is then used to update the model parameters of the strategy model, resulting in an updated strategy model that improves the accuracy and consistency of control decisions over long time scales. The updated strategy model is then validated to obtain validation indices. Based on these indices, a target strategy model is selected from the updated strategy models to avoid overfitting or accidentally excellent models, ensuring that the finally selected strategy has more stable generalization performance and reliability under different operating conditions. Real-time operating data from the process industry control system is input into the target strategy model, outputting target control actions. These target control actions are then validated to obtain validation results. Based on the validation results and the target control actions, control action execution suggestions for the process industry control system are generated. These suggestions enable control of the process industry control system, ensuring that actual actions are adopted or adjusted within a safe and feasible range, thus improving the controllability and safety during the deployment phase. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This application discloses a flowchart of a method for generating control action execution suggestions for a process industry control system. Figure 2 A detailed flowchart is generated for the control action execution suggestions of a process industry control system disclosed in this application; Figure 3 Generate a system structure diagram for the control action execution suggestions of a process industry control system disclosed in this application; Figure 4 This is a schematic diagram of a control action execution suggestion generation device for a process industry control system disclosed in this application. Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The training of control strategies in the process industry faces the following problems: (1) Online trial and error is not allowed in real-world settings: If reinforcement learning agents explore in real devices, it may cause water quality to exceed standards, equipment to exceed limits, product quality to fluctuate, energy consumption to be abnormal or safety risks to occur. (2) Traditional online optimization calculation and maintenance costs are high: Model predictive control and other methods usually require online solution of optimization problems in each control cycle, which has high requirements for model accuracy, constraint modeling and real-time performance of solvers. When faced with multivariable, long lag, nonlinear and frequent switching of operating conditions, online solution is easy to slow down or require a lot of manual parameter tuning. (3) Ordinary supervised learning strategies lack action consequence feedback: Directly fitting historical operator actions can obtain fast strategies, but can only imitate historical behavior, cannot actively optimize future control effects through objective functions, and is also difficult to handle new target settings that rarely appear in history. (4) Most existing world model reinforcement learning schemes are geared towards games, robots or general tasks, and have not disclosed single-step target interfaces, dynamic reachability gating, action change constraints, constraint boundaries and one-click deployment output mechanisms suitable for the process industry. As can be seen from the above, how to avoid online trial and error in real-world scenarios and solve long-term optimization problems online in each cycle, reduce the risk of the strategy driving the system to exceed limits in order to track the target, and improve the accuracy of the control action execution suggestions generated by the process industry control system are problems that need to be solved in this field.

[0022] See Figure 1 As shown in the figure, this invention discloses a method for generating control action execution suggestions for a process industry control system, which may specifically include: Step S11: Construct a world model of the process industry control system, verify the world model, and use the verified world model as the target world model.

[0023] In this embodiment, there are two methods for constructing a world model of a process industry control system. The first method is to construct a set of time-series operating variables of the process industry control system; based on the set of time-series operating variables, determine a set of bidirectional potential hybrid edges and a hybrid risk context of the set of bidirectional potential hybrid edges; the hybrid risk context is hybrid risk information corresponding to the set of bidirectional potential hybrid edges; and use the set of time-series operating variables and the hybrid risk context to train the initial world model to obtain a world model of the process industry control system.

[0024] The second method involves: acquiring time-series data of the process industry control system; determining target state variables based on the time-series data; the target state variables being the state variables of the next cycle after the current cycle; generating parent variable aggregation quantities for each target state variable; performing single-step state prediction on the parent variable aggregation quantities to obtain the state prediction result for the next cycle; using the state prediction result of the next cycle as the initial state, performing multi-step rolling prediction on the time-series data to obtain multi-step rolling prediction results; and training the initial world model using the target state variables, the state prediction results of the next cycle, and the multi-step rolling prediction results to obtain the world model of the process industry control system.

[0025] A world model of a process industry control system can be constructed using either of the two methods described above.

[0026] Step S12: Obtain policy training samples, input the policy training samples into the policy model, output control actions, process the control actions, and obtain processed control actions.

[0027] In this embodiment, a strategy training sample for the process industry control system is obtained. The strategy training sample includes the current state, target setpoint, constraints, historical actions, external operating conditions, and historical error information. The strategy training sample is input into the strategy model, and control actions are output. The control actions are trimmed based on preset upper and lower limits. The trimmed control actions are subject to a rate-of-change limitation based on a preset maximum change in action. The control actions with the rate-of-change limitation are smoothed to obtain the processed control actions.

[0028] After obtaining the target world model, the world model parameters are fixed. The deployment strategy output is then validated against action upper and lower limits, rate of change, and hard constraints.

[0029] In this step, the policy training samples include the current state. Target set value Constraints including upper or lower limits Historical actions External operating conditions In addition to historical error information, the policy sample, as well as the deviation between the state and the target, the amount of deviation change, the cumulative deviation, the constraint margin, the historical maximum deviation of the constraint, the amount of action change, and the action acceleration, are used as policy input features. Then, these policy input features are input into the policy model to output control actions.

[0030] Among them, the current state describes the instantaneous operating position of the device, the target setpoint describes the desired quality, energy consumption or output level, the constraints describe the safety upper limit, equipment load and emission boundary, the historical actions are used to calculate the rate of change of actions and the inertia of actions, and the external operating conditions are used to distinguish between load, raw material and environmental changes.

[0031] Let the strategy model be The strategy model in the first Output of each rolling step: ; in, In the first The control action output by the strategy model during each control cycle. For strategy parameters, In the first The policy input features received by the policy model during each control cycle.

[0032] The input features of the strategy do not require all items to be exactly the same, but they must adhere to uniform generation rules. For each controlled indicator, the system must at least construct the current value, target value, target deviation, deviation change, and cumulative deviation; for each constrained indicator, the system must at least construct the current margin, historical maximum risk of exceeding limits, and distance from the safety boundary; for each action, the system must at least construct the action of the previous cycle, the change in action, and the allowable range of change. This allows the same strategy training framework to be reused in different process industry projects.

[0033] Before entering policy training, the target world model needs to undergo read-only verification. The system records the target world model file hash, normalized parameter version, variable pattern version, and test set acceptance metrics, and checks during policy training to ensure these parameters have not been updated by the optimizer. If the gradient of the target world model's parameters is not blocked, or if the normalized version is inconsistent with the policy input version, policy training is aborted.

[0034] In this step, the upper and lower limits of the control action are derived from the capabilities of the field equipment and the safety boundary, while the maximum change in action is derived from the actuator speed, impact constraints, or operating procedures.

[0035] Step S13: Input the processed control action into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step. Use the state vector of the next rolling step as the current state in the strategy training sample, and then jump to the step of inputting the strategy training sample into the strategy model until the preset termination condition is met to obtain the multi-step rolling prediction result.

[0036] In this embodiment, the rolling step threshold is determined based on the control cycle and process response time of the process industry control system; the current state, external operating conditions, and post-processing control actions in the strategy training sample are input into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step; the state vector of the next rolling step is used as the current state in the strategy training sample, and the strategy training sample is updated using the state vector of the next rolling step and the post-processing control actions; then, the process jumps to the step of inputting the strategy training sample into the strategy model until the current rolling step is greater than the rolling step threshold, thus obtaining the multi-step rolling prediction result.

[0037] In this step, let the target world model be... Parameters of the target world model Fixed, target world model rolling prediction: ; in, The first prediction for the target world model The state vector at time t, The first prediction for the target world model The state vector at time t, For the first The vector of external operating conditions at time t.

[0038] Scrolling step threshold The number of steps for the target world model is determined based on the control cycle and process response time.

[0039] Step S14: Construct an objective function based on the multi-step rolling prediction results, and use the objective function to update the model parameters of the strategy model to obtain the updated strategy model.

[0040] In this embodiment, an objective function is constructed based on the multi-step rolling prediction results; the objective function includes target tracking loss, safety constraint loss, action smoothing loss, and dynamic reachability loss; based on the objective function, the policy model is backpropagated, and the model parameters of the policy model are updated to obtain the updated policy model.

[0041] Target tracking and safety constraints are represented by a single-step objective function, using a controlled index. A safety constraint indicator ,Target Upper limit For example, the dynamic reachability margin is: ; in, For the first Dynamic reachability margin at any given moment For the first The target setting value at any time, As constraints, For the first Controlled indicators at any given time For the first Safety constraints at all times.

[0042] Accessibility gating is as follows: ; in, For the first The accessibility threshold at any given time. Accessibility-gated temperature, This is the Sigmoid activation function. The reachability gating temperature is used to control the smoothness of reachability gating transitions and is determined based on policy stability.

[0043] Gating when the target approaches or exceeds the safe reach range This increases the risk factor, shifting the focus from simply tracking the target to prioritizing the suppression of safety indicators. The single-step loss is: ; in, For the first The single-step target loss value at time t. For safety target reference values, For security penalty weighting, For motion smoothing weights, For the first Controlling actions at all times For the first Controlling actions at specific moments.

[0044] The objective function for multi-step policy training is: ; in, Let the objective function of the policy model be... For the first The single-step target loss value at time t. Regularize the weights for action changes. For the first Controlling actions at specific moments.

[0045] Among them, the safety penalty weight is higher than the target tracking weight. The target tracking weight is configured based on the importance of process quality or operational goals, while the action smoothing weight is configured based on equipment lifespan and energy consumption constraints.

[0046] because It is differentiable for policy parameters. The gradient can be calculated; since The gradient is fixed, and only updates the policy model, while the target world model remains a validated virtual environment.

[0047] Step S15: Validate the updated strategy model to obtain validation metrics, and select the target strategy model from the updated strategy models based on the validation metrics.

[0048] During actual training, the system first warms up the policy using historical actions or conservative control actions to ensure that the policy output falls within the executable range of the current environment; then it enters the differentiable rolling training phase. Each training batch randomly selects historical moments and target conditions, and the policy output actions first pass through the action projection module. The action projection module sequentially performs upper and lower limit clipping, maximum rate of change constraint, and action smoothing to obtain executable actions for rolling the target world model.

[0049] During the rolling process, the system retains the predicted state, action, target error, constraint margin, and loss component for each step. If the predicted state at a certain step touches the hard safety boundary, subsequent rolling will still be executed, but that step and its subsequent steps will be given a higher safety penalty weight, enabling the policy to learn to avoid constraints in advance, rather than correcting actions only after exceeding the limit.

[0050] To prevent the policy from exploiting errors in the target world model, the system sets a confidence region constraint for the target world model. If the predicted state deviates from the training data distribution during the rolling process, or if key constraint variables enter the low confidence interval of the target world model, the system marks that segment of the trajectory as a low-confidence trajectory. Low-confidence trajectories can reduce the policy update weights or be used only for safety penalties instead of for target tracking rewards.

[0051] After training, the system does not directly adopt the strategy from the last iteration. Instead, it selects a strategy version based on validation metrics to filter out the target strategy model. Validation metrics include at least the target interval hit rate, average constraint violation, action change rate, final deviation, and long-term rolling stability. Only when the updated strategy model meets the thresholds under multiple target conditions can it be used as the target strategy model and then enter the deployment and solidification phase.

[0052] The outputs during the strategy model training phase include strategy training curves, rolling trajectories, target deviation indices, safety limit violation indices, action change indices, target interval hit rate, constraint violation rate, and convergence time. The system also outputs strategy response curves under different target settings to determine whether the strategy possesses target-conditional generalization ability.

[0053] Step S16: Input the real-time operating data of the process industry control system into the target strategy model, output the target control action, verify the target control action, obtain the verification result, and generate control action execution suggestions for the process industry control system based on the verification result and the target control action, so as to control the process industry control system based on the control action execution suggestions.

[0054] In this embodiment, real-time operating data of the process industry control system is acquired; the real-time operating data is sorted and normalized to obtain processed data; the processed data is input into the target strategy model for forward inference to obtain forward inference results; the forward inference results are restored to obtain restored actions; the restored actions are preprocessed to obtain preprocessed actions; the preprocessed actions are subjected to legality verification, rate of change verification, and constraint verification to obtain verification results; if the verification results show that the legality verification, rate of change verification, and constraint verification all pass, a control action execution suggestion for the process industry control system is generated so that the process industry control system can be controlled based on the control action execution suggestion.

[0055] The output of the target strategy model deployment phase includes the current control action or action increment, action pruning records, change rate limit records, safety constraint verification results, and necessary manual review markers. The target strategy model can be solidified into a general inference format, enabling the field system to perform strategy forward inference only once per control cycle, reducing the burden of online optimization.

[0056] If the prediction error of the target world model exceeds a threshold under a certain operating condition, the system will prohibit the automatic adoption of the strategy output under that condition and switch to conservative control or manual review. If the strategy output action touches the hard constraint boundary, the system will perform pruning, smoothing, or rollback.

[0057] The deployment includes at least the strategy model parameters, normalization parameters, variable order table, action upper and lower limit table, maximum action change table, target condition description, constraint configuration, and version verification information. During field invocation, the system first constructs the input according to the variable order table, then performs normalization and strategy inference, and finally restores the action to the field physical quantity unit. If the field input lacks necessary variables or the variable order is mismatched, the system will not output any automatic control actions.

[0058] Safety post-processing is performed in three layers. The first layer is action legality verification, ensuring that the action is within the equipment's capabilities; the second layer is rate of change verification, ensuring that actions in adjacent cycles will not impact the equipment; the third layer is constraint prediction verification, using a target world model or lightweight constraint model to check for potential short-term limit exceedance risks that the action may cause. Only when all three layers of verification pass will the action be output as an automatic suggestion; otherwise, a conservative action, a rollback action, or a manual review mark will be output.

[0059] Taking wastewater treatment aeration control as an example, the system first constructs a world model using the set of time-series operating variables or time-series data from the process industry control system. This model enables the system to predict future dissolved oxygen, effluent ammonia nitrogen, and energy consumption based on influent load, historical aeration rate, dissolved oxygen, ammonia nitrogen, sludge concentration, and water temperature. After rolling verification, the world model is frozen, resulting in the target world model.

[0060] During the strategy training phase, the strategy inputs include current dissolved oxygen, predicted effluent ammonia nitrogen, target ammonia nitrogen, target deviation, deviation changes, historical operation of aeration valves or blowers, and energy consumption constraints. The strategy output is the aeration action for the next control cycle. The target world model extrapolates several future cycles, with the single-step objective function simultaneously penalizing deviations in effluent indicators, excessive energy consumption, rapid changes in action, and exceeding safety boundaries. The loss is backpropagated to the strategy model, allowing the strategy learning to strike a trade-off between target water quality and energy consumption, resulting in the target strategy model.

[0061] During the deployment phase, the target strategy model outputs aeration action suggestion once per cycle, which is then verified against the upper and lower limits of the blower frequency and the maximum rate of change. If a sudden change in the influent load leads to insufficient confidence in the world model, the system outputs a manual review flag or reverts to the existing control logic.

[0062] The technical solution of this application is also applicable to ammonia denitrification, chemical temperature control, pump station water level, energy station load distribution, fermentation feeding and metallurgical process control.

[0063] In this embodiment, the specific process for generating control action execution suggestions for the process industry control system is as follows: Figure 2 As shown, it is divided into three stages: world model preparation, policy rolling training, and deployment output. The first two steps involve constructing and validating a world model of the process industry control system. Only after the world model is validated are its parameters fixed to obtain the target world model. The target world model is no longer involved in parameter updates to prevent the strategy training process from altering the already accepted virtual process environment.

[0064] Steps three through seven constitute the differentiable rolling trajectory generation stage. The system uses policy samples, along with the deviation between the state and the target, the change in deviation, the cumulative deviation, the constraint margin, the historical maximum deviation of the constraint, the change in action, and the action acceleration, as policy input features. These policy input features are then input into the policy model, which outputs control actions or action increments. After pruning, rate-of-change limiting, and smoothing, the target world model predicts the state vector for the next rolling step and continues to input the state vector for the next rolling step, forming a multi-step rolling prediction result.

[0065] Steps eight through ten constitute the strategy update and deployment solidification phase. In each rolling step, the system constructs an objective function and backpropagates it along the differentiable trajectory of the target world model to the strategy model parameters. Based on validation metrics, it selects the target strategy model. Subsequently, the target strategy model is exported as a deployment model. During online runtime, forward inference is performed only once per control cycle, followed by secure post-processing to output control action execution suggestions.

[0066] In this embodiment, the system structure diagram for generating control action execution suggestions for the process industry control system is as follows: Figure 3 As shown, it includes a world model validation module, a policy training module, an objective and constraint module, and a deployment output module. This invention adopts a "model first, then freeze, then train, then deploy" implementation approach: The world model verification module adopts a chain of first building the world model, then freezing the world model, and then training the strategy. This allows the strategy training to be carried out in a verified virtual process environment, avoiding online trial and error in real-world situations and preventing the strategy training from distorting the world model in order to reduce the target loss. The strategy training module backpropagates the future control target loss to the strategy model through multi-step rolling prediction results, enabling the strategy to not only imitate historical actions, but also actively optimize the future state according to the target and constraints. The target and constraint module is designed for a single-step target interface for the process industry, unifying target tracking, safety limit violation, motion smoothing, constraint margin, and dynamic reachability into a differentiable loss; dynamic reachability gating is introduced to automatically increase the priority of safety constraints when the target approaches the safety boundary or becomes unreachable, reducing the risk that the strategy will drive the system to exceed the limit in order to track the target; The deployment output module requires only one forward inference during deployment, avoiding the need for online solving of long time-domain optimization problems in each cycle, making it suitable for industrial control systems with high real-time requirements.

[0067] As can be seen, this application provides a method for generating control action execution suggestions for process industry control systems. The method includes: constructing a world model of the process industry control system; validating the world model; using the validated world model as the target world model to avoid direct experimentation on real systems; obtaining strategy training samples; inputting the strategy training samples into the strategy model to output control actions; processing the control actions to obtain processed control actions, thereby improving the stability and feasibility of the training process; inputting the processed control actions into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step; using the state vector of the next rolling step as the current state in the strategy training samples; then jumping to the step of inputting the strategy training samples into the strategy model until a preset termination condition is met, obtaining multi-step rolling prediction results, effectively improving the strategy's ability to cope with process lag and cumulative disturbances; and based on the multi-step rolling prediction results... An objective function is constructed and used to update the model parameters of the strategy model, resulting in an updated strategy model that improves the accuracy and consistency of control decisions over long time scales. The updated strategy model is then validated to obtain validation indices. Based on these indices, a target strategy model is selected from the updated strategy models to avoid overfitting or accidentally excellent models, ensuring that the finally selected strategy has more stable generalization performance and reliability under different operating conditions. Real-time operating data from the process industry control system is input into the target strategy model, outputting target control actions. These actions are then validated to obtain validation results. Based on the validation results and the target control actions, control action execution suggestions for the process industry control system are generated. These suggestions enable control of the process industry control system, ensuring that actual actions are adopted or adjusted within a safe and feasible range, thus improving the controllability and safety during the deployment phase.

[0068] See Figure 4 As shown in the figure, an embodiment of the present invention discloses a control action execution suggestion generation device for a process industry control system, which may specifically include: Verification module 11 is used to construct a world model of the process industry control system, verify the world model, and use the verified world model as the target world model. Processing module 12 is used to acquire policy training samples, input the policy training samples into the policy model, output control actions, process the control actions, and obtain processed control actions. The multi-step rolling prediction module 13 is used to input the processed control action into the target world model for prediction, obtain the state vector of the process industry control system in the next rolling step, use the state vector of the next rolling step as the current state in the strategy training sample, and then jump to the step of inputting the strategy training sample into the strategy model until the preset termination condition is met, and obtain the multi-step rolling prediction result. The model parameter update module 14 is used to construct an objective function based on the multi-step rolling prediction results, and use the objective function to update the model parameters of the strategy model to obtain the updated strategy model. The filtering module 15 is used to verify the updated strategy model, obtain verification indicators, and filter out the target strategy model from the updated strategy model based on the verification indicators. The execution suggestion generation module 16 is used to input the real-time operation data of the process industry control system into the target strategy model, output the target control action, verify the target control action, obtain the verification result, and generate control action execution suggestions for the process industry control system based on the verification result and the target control action, so as to control the process industry control system based on the control action execution suggestions.

[0069] In some specific embodiments, the verification module 11 may specifically include: The timing operation variable set construction module is used to construct the timing operation variable set of the process industry control system; The hybrid risk context determination module is used to determine the bidirectional potential hybrid edge set and the hybrid risk context of the bidirectional potential hybrid edge set based on the time-series running variable set; the hybrid risk context is the hybrid risk information corresponding to the bidirectional potential hybrid edge set. The first training module is used to train the initial world model using the set of time-series operating variables and the hybrid risk context to obtain the world model of the process industry control system.

[0070] In some specific embodiments, the verification module 11 may specifically include: The target state variable determination module is used to acquire time-series data of the process industry control system and determine the target state variable based on the time-series data; the target state variable is the state variable of the next cycle of the current cycle. The parent variable aggregation quantity generation module is used to generate parent variable aggregation quantities for each of the target state variables; The single-step state prediction module is used to perform single-step state prediction on the aggregate quantity of the parent variable to obtain the state prediction result for the next period. The multi-step rolling prediction module is used to perform multi-step rolling prediction on the time series data, taking the state prediction result of the next cycle as the initial state, to obtain the multi-step rolling prediction result. The second training module is used to train the initial world model using the target state variable, the state prediction result of the next cycle, and the multi-step rolling prediction result to obtain the world model of the process industry control system.

[0071] In some specific embodiments, the processing module 12 may specifically include: The strategy training sample acquisition module is used to acquire strategy training samples for the process industry control system; the strategy training samples include the current state, target setpoint, constraints, historical actions, external operating conditions, and historical error information. The strategy training sample input module is used to input the strategy training sample into the strategy model and output control actions; The trimming module is used to trim the control actions based on preset upper and lower limits of control actions; The rate of change limiting module is used to limit the rate of change of the cut control action based on the preset maximum amount of action change. The smoothing module is used to smooth the control action after the rate of change is limited, so as to obtain the processed control action.

[0072] In some specific embodiments, the multi-step rolling prediction module 13 may specifically include: The rolling step threshold determination module is used to determine the rolling step threshold based on the control cycle and process response time of the process industry control system. The state vector output module for the next rolling step is used to input the current state, external conditions, and post-processing control actions from the strategy training samples into the target world model for prediction, so as to obtain the state vector of the process industry control system in the next rolling step. The multi-step rolling prediction result determination module is used to take the state vector of the next rolling step as the current state in the policy training sample, update the policy training sample using the state vector of the next rolling step and the processed control action, and then jump to the step of inputting the policy training sample into the policy model until the current number of rolling steps is greater than the number of rolling steps threshold, and obtain the multi-step rolling prediction result.

[0073] In some specific embodiments, the model parameter update module 14 may specifically include: The objective function construction module is used to construct an objective function based on the multi-step rolling prediction results; the objective function includes target tracking loss, safety constraint loss, action smoothing loss, and dynamic reachability loss; The backpropagation module is used to backpropagate the policy model based on the objective function and update the model parameters of the policy model to obtain the updated policy model.

[0074] In some specific embodiments, the execution suggestion generation module 16 may specifically include: The real-time operation data acquisition module is used to acquire real-time operation data of the process industry control system; The data sorting and normalization processing module is used to sort and normalize the real-time running data to obtain processed data. The forward reasoning module is used to input the processed data into the target policy model for forward reasoning, obtain the forward reasoning result, restore the forward reasoning result, and obtain the restored action; The preprocessing module is used to preprocess the restored action to obtain the preprocessed action; The verification module is used to perform legality verification, rate of change verification, and constraint verification on the preprocessed actions, and obtain the verification results. The control action execution suggestion generation module is used to generate control action execution suggestions for the process industry control system if the verification results show that the legality verification, rate of change verification, and constraint verification all pass.

[0075] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the control action execution suggestion generation method for a process industrial control system executed by an electronic device as disclosed in any of the foregoing embodiments.

[0076] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0077] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0078] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the control action execution suggestion generation method for the process industry control system disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the control action execution suggestion generation device of the process industry control system from external devices, and may also include data collected by its own input / output interface 25.

[0079] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0080] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the steps of the method for generating control action execution suggestions for the process industry control system disclosed in any of the foregoing embodiments.

[0081] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0082] The foregoing has provided a detailed description of the method, apparatus, equipment, and medium for generating control action execution suggestions for a process industry control system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating control action execution suggestions for a process industry control system, characterized in that, include: Construct a world model of the process industry control system, verify the world model, and use the verified world model as the target world model; Obtain policy training samples, input the policy training samples into the policy model, output control actions, process the control actions, and obtain processed control actions. The processed control action is input into the target world model for prediction to obtain the state vector of the process industry control system in the next rolling step. The state vector of the next rolling step is used as the current state in the strategy training sample. Then, the process jumps to the step of inputting the strategy training sample into the strategy model until the preset termination condition is met to obtain the multi-step rolling prediction result. Based on the multi-step rolling prediction results, an objective function is constructed, and the objective function is used to update the model parameters of the strategy model to obtain the updated strategy model. The updated strategy model is validated to obtain validation metrics. Based on the validation metrics, the target strategy model is selected from the updated strategy model. The real-time operating data of the process industry control system is input into the target strategy model, the target control action is output, the target control action is verified, the verification result is obtained, and based on the verification result and the target control action, the control action execution suggestion of the process industry control system is generated so that the process industry control system can be controlled based on the control action execution suggestion.

2. The method for generating control action execution suggestions for a process industry control system according to claim 1, characterized in that, The world model for constructing process industry control systems includes: Construct a set of time-series operation variables for the process industry control system; Based on the set of time-series runtime variables, a set of bidirectional potential hybrid edges and a hybrid risk context for the set of bidirectional potential hybrid edges are determined; the hybrid risk context is the hybrid risk information corresponding to the set of bidirectional potential hybrid edges. Using the set of time-series operating variables and the hybrid risk context, the initial world model is trained to obtain the world model of the process industry control system.

3. The method for generating control action execution suggestions for a process industry control system according to claim 1, characterized in that, The world model for constructing process industry control systems includes: Acquire time-series data of the process industry control system, and determine a target state variable based on the time-series data; the target state variable is the state variable of the next cycle of the current cycle. Generate parent variable aggregation values ​​for each of the target state variables; Perform a single-step state prediction on the aggregated amount of the parent variable to obtain the state prediction result for the next cycle; Using the state prediction result of the next cycle as the initial state, multi-step rolling prediction is performed on the time series data to obtain the multi-step rolling prediction result. Using the target state variable, the state prediction result of the next cycle, and the multi-step rolling prediction result, the initial world model is trained to obtain the world model of the process industry control system.

4. The method for generating control action execution suggestions for a process industry control system according to claim 1, characterized in that, The process of acquiring policy training samples, inputting the policy training samples into a policy model, outputting control actions, and processing the control actions to obtain processed control actions includes: Obtain strategy training samples for the process industry control system; the strategy training samples include current state, target setpoint, constraints, historical actions, external operating conditions, and historical error information; The policy training samples are input into the policy model, and the output control action is generated. Based on preset upper and lower limits for control actions, the control actions are trimmed. Based on the preset maximum change in motion, the rate of change of the control motion after trimming is limited. The control action after the rate of change is limited is smoothed to obtain the processed control action.

5. The method for generating control action execution suggestions for a process industry control system according to claim 1, characterized in that, The process involves inputting the processed control action into the target world model for prediction, obtaining the state vector of the process industry control system in the next rolling step, using the state vector of the next rolling step as the current state in the strategy training sample, and then jumping to the step of inputting the strategy training sample into the strategy model until a preset termination condition is met, resulting in a multi-step rolling prediction result, including: The threshold for the number of rolling steps is determined based on the control cycle and process response time of the process industry control system. The current state, external conditions, and post-processing control actions in the strategy training samples are input into the target world model for prediction, thereby obtaining the state vector of the process industry control system in the next rolling step. The state vector of the next scrolling step is used as the current state in the policy training sample. The policy training sample is updated using the state vector of the next scrolling step and the processed control action. Then, the process jumps to the step of inputting the policy training sample into the policy model until the current number of scrolling steps is greater than the number of scrolling steps threshold, and the multi-step scrolling prediction result is obtained.

6. The method for generating control action execution suggestions for a process industry control system according to claim 1, characterized in that, The step of constructing an objective function based on the multi-step rolling prediction results, and using the objective function to update the model parameters of the policy model to obtain the updated policy model includes: An objective function is constructed based on the multi-step rolling prediction results; the objective function includes target tracking loss, safety constraint loss, motion smoothing loss, and dynamic reachability loss; Based on the objective function, the policy model is backpropagated, and the model parameters of the policy model are updated to obtain the updated policy model.

7. The method for generating control action execution suggestions for a process industry control system according to any one of claims 1 to 6, characterized in that, The process involves inputting real-time operating data of the process industry control system into the target strategy model, outputting target control actions, verifying the target control actions, obtaining verification results, and generating control action execution suggestions for the process industry control system based on the verification results and the target control actions. This includes: Acquire real-time operating data from process industry control systems; The real-time running data is sorted and normalized to obtain processed data; The processed data is input into the target policy model for forward reasoning to obtain the forward reasoning result. The forward reasoning result is then restored to obtain the restored action. The restored action is preprocessed to obtain the preprocessed action; The preprocessed actions are validated for legality, rate of change, and constraints to obtain the validation results. If the validation results show that the validity validation, rate of change validation, and constraint validation all pass, then control action execution suggestions for the process industry control system will be generated.

8. A control action execution suggestion generation device for a process industry control system, characterized in that, include: The verification module is used to construct a world model of the process industry control system, verify the world model, and use the verified world model as the target world model. The processing module is used to acquire policy training samples, input the policy training samples into the policy model, output control actions, process the control actions, and obtain processed control actions. The multi-step rolling prediction module is used to input the processed control action into the target world model for prediction, obtain the state vector of the process industry control system in the next rolling step, use the state vector of the next rolling step as the current state in the strategy training sample, and then jump to the step of inputting the strategy training sample into the strategy model until the preset termination condition is met, and obtain the multi-step rolling prediction result. The model parameter update module is used to construct an objective function based on the multi-step rolling prediction results, and use the objective function to update the model parameters of the strategy model to obtain the updated strategy model. The filtering module is used to verify the updated strategy model, obtain verification indicators, and filter out the target strategy model from the updated strategy model based on the verification indicators. The execution suggestion generation module is used to input the real-time operating data of the process industry control system into the target strategy model, output the target control action, verify the target control action, obtain the verification result, and generate control action execution suggestions for the process industry control system based on the verification result and the target control action, so as to control the process industry control system based on the control action execution suggestions.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method for generating control action execution suggestions for a process industry control system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the method for generating control action execution suggestions for a process industry control system as described in any one of claims 1 to 7.

Images