Industrial control system anomaly detection method based on digital twinning and reinforcement learning

Abstract

The invention discloses an industrial control system anomaly detection method based on digital twinborn and reinforcement learning, and aims to solve the problems that in the prior art, abnormal samples are scarce, the false alarm rate is high under a dynamic working condition, and the generalization ability of a single detection model is insufficient. Generating synthetic data by simulating anomalies in the twinborn body, and mixing the synthetic data with normal data to form a training set after space-time and distribution alignment; based on the real-time data, generating a time sequence prediction residual error, a distribution reconstruction error and classification confidence multi-source evidence in parallel; constructing a reinforcement learning environment, constructing a state through multi-source evidence and the like, and guiding an agent to learn a fusion strategy through an asymmetric reward function; and training the intelligent agent to obtain a detection model and performing online deployment. According to the method, the multi-source evidence is adaptively fused through the reinforcement learning agent, so that the robustness, the accuracy and the adaptive capability of anomaly detection under the dynamic working condition are remarkably improved, and missing report and false report are effectively reduced.

Description

Technical Field

[0001] This application belongs to the field of information security and intelligent detection technology of industrial control systems, specifically involving an anomaly detection method for industrial control systems that integrates digital twins and deep reinforcement learning. Background Technology

[0002] With the continuous development of the Industrial Internet, Industrial Control Systems (ICS) are gradually evolving from traditional closed and isolated architectures to open, interconnected, and cross-domain integrated systems. While introducing capabilities such as remote operation and maintenance, cloud-edge collaboration, data sharing, and business interconnection, the network boundaries are becoming blurred, the attack surface is significantly expanding, and the systems are vulnerable to the penetration and long-term latency of complex attacks such as Advanced Persistent Threats (APTs), thus threatening production safety, stable equipment operation, and the continuity of critical infrastructure.

[0003] Anomaly detection, as a crucial technical means for the security protection of industrial control systems, can continuously monitor the system's operating status and detect potential anomalies even in situations lacking clear attack characteristics, where attack methods are constantly evolving, or where prior rules are difficult to cover. However, industrial control systems are widely used in critical fields such as petrochemicals, power energy, and rail transportation, where safety and continuity requirements are extremely high. Due to constraints such as on-site safety production requirements, low incidence of actual attack events, high costs of fault / attack reproduction, and difficulties in anomaly sample labeling, existing anomaly detection methods generally face the following problems in engineering implementation:

[0004] 1. Scarcity of abnormal samples and prominent class imbalance: Industrial field systems are in normal operation most of the time. The number of real fault or attack samples is small, the acquisition cost is high and they are difficult to label. This leads to a serious imbalance in the training samples of the detection model based on supervised learning. The model is prone to bias towards the normal class, resulting in problems such as high false negative rate and insufficient generalization ability.

[0005] 2. Insufficient adaptability to dynamic operating conditions and high false alarm rate: Industrial control systems typically exhibit dynamic changes under multiple operating conditions, such as start-up and shutdown, load fluctuations, process switching, and equipment aging. Existing detection methods based on static thresholds or fixed rules struggle to accurately characterize normal fluctuation boundaries under operating condition switching and process disturbances. This can easily lead to misjudging normal operating condition changes as abnormalities, resulting in a higher false alarm rate, reduced alarm reliability, and impacted on-site handling efficiency.

[0006] Therefore, how to effectively model the dynamic operating laws of industrial control systems under multiple operating conditions under the condition of scarce abnormal samples, and to build an anomaly detection mechanism that can adaptively adjust with operating conditions and drift, so as to reduce false alarms and false negatives and improve detection robustness and deployability, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] This application aims to overcome the shortcomings of the prior art and provide an anomaly detection method for industrial control systems based on digital twins and reinforcement learning. This method generates high-quality synthetic anomaly data through digital twin technology to alleviate the problem of sample scarcity, and introduces a reinforcement learning mechanism to adaptively fuse multi-source detection evidence, thereby achieving more robust and accurate anomaly detection under dynamic operating conditions. The anomaly detection method for industrial control systems based on digital twins and reinforcement learning of this application includes the following steps:

[0008] S1. Obtain entity operation data through acquisition units deployed in the field layer and control layer of the industrial control system, and construct the digital twin of the industrial control system by adopting a hybrid driving method that combines the mechanism model of process partitioning and the data-driven model.

[0009] S2. Simulate various abnormal scenarios in the digital twin to generate a synthetic abnormal sequence, and perform spatiotemporal data alignment and distribution alignment processing on the synthetic abnormal sequence to obtain an enhanced sample set. Mix the enhanced sample set with historical normal data to form a supervised training set.

[0010] S3. Input the real-time running data into the feature extraction module to generate time-series prediction evidence, distribution reconstruction evidence, and classification confidence evidence in parallel.

[0011] S4. Define the state space, action space and asymmetric reward function of the agent. The state space includes at least the original sensor data, prediction residual, reconstruction error and classification confidence.

[0012] S5. Train the agent in a simulated environment using the supervised training set to learn an adaptive fusion strategy for multi-source evidence and obtain a target anomaly detection model.

[0013] S6. Deploy the trained target anomaly detection model on an edge computing node for real-time anomaly detection, and fine-tune the parameters of the digital twin online using real-time data when preset conditions are met.

[0014] In one embodiment, the hybrid driving method specifically includes:

[0015] The structure of the industrial control system is divided into regions according to the process. In each region, a deterministic differential equation mechanism model is established for the subprocesses that follow clear physical laws. Based on the topological connection and process flow structure, the mechanism models of each region are solved jointly.

[0016] Establish data-driven models based on deep neural networks for subprocesses with strong nonlinearity and complex mechanisms;

[0017] The outputs of the mechanistic model and the data-driven model are calibrated using Kalman filtering or weighted fusion algorithms to obtain theoretical baseline values ​​for the key state variables of the system.

[0018] In one embodiment, the abnormal scenario simulation includes at least one parameterized simulation method, the parameterized simulation method including:

[0019] Perform a sensor spoofing attack by superimposing a bias term onto the selected sensor channel. or linear drift term ;

[0020] Perform a replay attack and record historical normal time windows. The data sequence is mapped to the current time window. ;

[0021] False data injection is performed on specific control loops, and the state vector is modified according to the topology to make it satisfy linear constraints and deviate from the real physical state.

[0022] Configure actuator jamming fault settings, set the actuator feedback value to remain constant within a preset time interval, or set the actuator response gain to zero;

[0023] To set up process disturbance faults, external disturbance variables or superimposed step signals that conform to a preset statistical distribution are introduced into the state equation of the digital twin.

[0024] In one embodiment, the step of performing spatiotemporal data alignment and distribution alignment processing on the synthesized anomalous sequence specifically includes:

[0025] Data domain adaptation includes time synchronization and statistical matching. Time synchronization aligns the time grid of the simulation data with the sampling timestamps of the actual acquired data through resampling. Statistical matching corrects the marginal statistics of the simulation data through feature-level affine transformations, satisfying the following:

[0026]

[0027] in The original simulation signal, The mean and standard deviation of the simulation data. and These represent the mean and standard deviation of the actual historical data under the corresponding operating conditions.

[0028] In one embodiment, the temporal prediction evidence is obtained by predicting historical data using a Long Short-Term Memory (LSTM) network and calculating the prediction residual; the distribution reconstruction evidence is obtained by reconstructing the current data using a Variational Autoencoder (VAE) and calculating the reconstruction error; and the classification confidence evidence is obtained by outputting the anomaly probability of real-time data using a classification model pre-trained on the supervised training set.

[0029] In one embodiment, the state space satisfy:

[0030]

[0031] in, This is the normalized sensor observation vector at the current time t; This represents the prediction residual vector based on time correlation. This is the reconstruction error vector based on distribution consistency; This refers to the contextual statistical features within the sliding time window or the hidden state of the previous time step. Represents vector norm or element-wise absolute value operation.

[0032] In one embodiment, the action space is a discrete action space or a continuous action space, wherein the discrete action space satisfies It directly outputs binary classification results, where 0 represents normal and 1 represents abnormal; the continuous action space satisfies... Output anomaly probability value or dynamically adjusted detection threshold. .

[0033] In one embodiment, the asymmetric reward function is based on the agent's actions. With real labels The combination of values ​​of the asymmetric reward function The design is as follows:

[0034]

[0035] in, For agent actions, The true labels for the samples; This means that the penalty weight for missed detections is much greater than the penalty weight for false alarms, in order to meet the high security integrity level requirements of industrial control systems.

[0036] In one embodiment, the agent is trained using the Proximal Policy Optimization (PPO) algorithm. The agent employs an Actor-Critic network architecture and updates its policy by interacting with the environment, sampling, and optimizing a total loss function that includes a shearing objective function, a value function loss, and an entropy regularization term.

[0037] In one embodiment, satisfying the preset condition means that the target anomaly detection model determines that the current state is normal and the output anomaly confidence level is lower than the safety threshold.

[0038] Online incremental learning is performed on the data-driven model in the digital twin using the high-confidence normal samples at this time;

[0039] The trained agent is deployed on edge computing nodes to perform online inference and output anomaly detection results in real time. When the agent determines that the system is normal and the confidence level meets the preset threshold, the model parameters of the digital twin are fine-tuned online using real-time data to adapt to system drift.

[0040] Therefore, this application has the following beneficial effects:

[0041] This application provides an anomaly detection method for industrial control systems based on digital twins and reinforcement learning, comprising the following steps: acquiring entity operation data through acquisition units deployed at the field layer and control layer of the industrial control system; constructing a digital twin of the industrial control system using a hybrid driving approach combining a process-differentiated mechanism model and a data-driven model; simulating multiple anomaly scenarios in the digital twin to generate a synthetic anomaly sequence, and performing spatiotemporal data alignment and distribution alignment processing on the synthetic anomaly sequence to obtain an enhanced sample set; mixing the enhanced sample set with historical normal data to form a supervised training set; and inputting the real-time operation data into a specific... The feature extraction module generates temporal prediction evidence, distribution reconstruction evidence, and classification confidence evidence in parallel. It defines the agent's state space, action space, and asymmetric reward function. The state space includes at least the original sensor data, prediction residuals, reconstruction errors, and classification confidence. The agent is trained in a simulated environment using the supervised training set to learn an adaptive fusion strategy for multi-source evidence, resulting in a target anomaly detection model. The trained target anomaly detection model is deployed on edge computing nodes for real-time anomaly detection, and online parameter fine-tuning of the digital twin is performed using real-time data when preset conditions are met. This application discloses an anomaly detection method for industrial control systems based on digital twins and reinforcement learning. It aims to address the problems of scarce anomaly samples, high false alarm rates under dynamic operating conditions, and insufficient generalization ability of single detection models in existing technologies. The method involves constructing a mechanism- and data-driven digital twin; simulating anomalies to generate synthetic data within the twin, which is then spatiotemporally and distributionally aligned and mixed with normal data to form a training set; generating multi-source evidence such as time-series prediction residuals, distribution reconstruction errors, and classification confidence based on real-time data in parallel; constructing a reinforcement learning environment, using multi-source evidence to build states, and guiding the agent to learn a fusion strategy using an asymmetric reward function; training the agent to obtain a detection model and deploying it online. This application significantly improves the robustness, accuracy, and adaptability of anomaly detection under dynamic operating conditions by adaptively fusing multi-source evidence through a reinforcement learning agent, effectively reducing false negatives and false alarms.

[0042] The technical solution of this application achieves three key breakthroughs in anomaly detection in industrial control systems by introducing hierarchical mechanism modeling and global physical constraints, significantly surpassing traditional data-driven methods:

[0043] First, it overcomes the "curse of dimensionality" and error accumulation. Compared with existing techniques that directly use end-to-end black-box models or simple cascaded models, which easily lead to training difficulties and progressively amplified errors, this scheme divides the complex system into multiple independent units according to the process flow and models them separately, significantly reducing the complexity of a single model. Furthermore, by solving a topological network simultaneously, it utilizes physical conservation laws such as mass and energy to form hard constraints at the global level, enabling real-time mutual verification of the states of each unit and automatic error correction, thus ensuring the physical consistency and numerical stability of the entire system's state output.

[0044] Secondly, it deeply integrates physical laws to accurately identify covert attacks. Purely data-driven models rely on statistical correlations and are easily deceived by carefully constructed attacks that conform to statistical characteristics but violate physical principles (such as advanced spoofing). This solution constructs a unit mechanism model based on first principles. Any attack that violates fundamental physical constraints such as the conservation of mass and energy (e.g., an increase in feed flow rate while the tank level remains unchanged) will induce significant equation residuals during the global simultaneous solution process. This achieves highly robust anomaly detection based on physical laws, a fundamental defensive capability that purely data-driven methods lack.

[0045] Finally, the model possesses physical interpretability, supporting precise fault location. When an anomaly is detected, the system can quickly locate the specific "process unit" or "connection relationship" of the residual anomaly based on the topology model, directly indicating potential root causes such as "reactor internal leakage," "pipeline blockage," or "sensor tampering." This achieves a leap from "detecting anomalies" to "locating the cause," greatly improving the efficiency of operation and maintenance response and the accuracy of decision-making, and meeting the stringent requirements of high-security industrial scenarios for the interpretability and operability of the detection system. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a system flowchart of an anomaly detection method for industrial control systems based on digital twins and reinforcement learning;

[0048] Figure 2 This is an overall process architecture diagram of an embodiment of this application;

[0049] Figure 3 It is an adversarial data augmentation flowchart for digital twins of petroleum refining and fractionation units;

[0050] Figure 4 This is a flowchart of multi-source evidence generation and state construction using a catalytic cracking fractionation unit as an example;

[0051] Figure 5 This is a block diagram of a dynamic fusion mechanism based on deep reinforcement learning, as described in this application embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0054] To address the core shortcomings of existing technologies in industrial control system anomaly detection, such as scarce anomaly samples, high false alarm rates under dynamic operating conditions, and insufficient generalization ability of single models, this application proposes an innovative solution integrating digital twins and deep reinforcement learning. First, a hybrid-driven digital twin is constructed to faithfully reproduce the physical industrial control system in a virtual space. Based on this, adversarial simulation is performed to parameterize and generate synthetic anomaly data with various attack and fault modes. This data undergoes rigorous spatiotemporal and statistical distribution alignment processing to ensure consistency with real-world data distribution, effectively alleviating the challenges of scarce real-world anomaly samples and class imbalance, and providing a high-quality, sufficient data foundation for subsequent model training.

[0055] Secondly, to overcome the limitations of a single detection model, the method designs a parallel multi-source evidence generation mechanism. It utilizes a Long Short-Term Memory Network (LSTM) to capture temporal prediction residuals, a Variational Autoencoder (VAE) to capture data distribution reconstruction errors, and supplements this with a classification model trained on augmented data to provide prior confidence. These three sources of evidence provide anomaly indications from different dimensions—temporal logic, variable correlation, and category discrimination—forming a complementary detection perspective and significantly reducing the risk of missed detections due to model blind spots.

[0056] To address the challenge of setting thresholds under dynamic operating conditions, this method introduces Deep Reinforcement Learning (DRL) as an adaptive fusion decision-maker. Multi-source evidence, raw data, and contextual features are collectively constructed into a state space. An asymmetric reward function, centered on severe penalties for missed detections, is designed to train the agent to learn dynamic fusion weights and decision-making strategies. This enables the system to adaptively adjust the decision boundaries based on the real-time operating context, effectively distinguishing between genuine anomalies and normal fluctuations under complex operating conditions, significantly reducing the false alarm rate.

[0057] Finally, by deploying an online self-supervised update mechanism, the digital twin is continuously fine-tuned using high-confidence normal samples, enabling the entire system to evolve collaboratively with equipment aging and operational condition drift, maintaining long-term detection stability and adaptability. This application systematically improves the robustness, accuracy, and engineering deployability of industrial control anomaly detection through the synergistic effects of digital twin-generated data, comprehensive perception of multi-source evidence, reinforcement learning-based intelligent decision-making, and online closed-loop continuous optimization.

[0058] This application provides an anomaly detection method for industrial control systems based on digital twins and reinforcement learning, including steps S1 to S6, as described above. Figure 1 , Figure 1 This is a system flowchart of an anomaly detection method for industrial control systems based on digital twins and reinforcement learning.

[0059] S1. Obtain entity operation data through acquisition units deployed in the field layer and control layer of the industrial control system, and construct the digital twin of the industrial control system by adopting a hybrid driving method that combines the mechanism model of process partitioning and the data-driven model.

[0060] S2. Simulate various abnormal scenarios in the digital twin to generate a synthetic abnormal sequence, and perform spatiotemporal data alignment and distribution alignment processing on the synthetic abnormal sequence to obtain an enhanced sample set. Mix the enhanced sample set with historical normal data to form a supervised training set.

[0061] S3. Input the real-time running data into the feature extraction module to generate time-series prediction evidence, distribution reconstruction evidence, and classification confidence evidence in parallel.

[0062] S4. Define the state space, action space and asymmetric reward function of the agent. The state space includes at least the original sensor data, prediction residual, reconstruction error and classification confidence.

[0063] S5. Train the agent in a simulated environment using the supervised training set to learn an adaptive fusion strategy for multi-source evidence and obtain a target anomaly detection model.

[0064] S6. Deploy the trained target anomaly detection model on an edge computing node for real-time anomaly detection, and fine-tune the parameters of the digital twin online using real-time data when preset conditions are met.

[0065] Specifically, in this embodiment, the industrial control system anomaly detection method based on digital twins and reinforcement learning provided in this application embodiment refers to... Figure 1The system flowchart shown includes the following six core steps S1 to S6, forming a complete technical process from data construction and model training to online deployment and adaptation. It aims to significantly improve the robustness and accuracy of industrial control systems under abnormal sample scarcity and dynamic operating conditions.

[0066] Step S1: Digital Twin Construction

[0067] First, data acquisition units deployed at the field layer (such as sensors and actuators) and control layer of the industrial control system acquire multi-source time-series data reflecting the operating state of the physical entity in real time, including but not limited to key process variables such as temperature, pressure, flow rate, liquid level, and vibration. Based on this, a hybrid driving paradigm of mechanistic model + data-driven model is adopted to construct a high-fidelity digital twin corresponding to the physical system. Specifically, for sub-processes with clear physical laws and analytical expressions (such as mass conservation and energy balance), a deterministic mechanistic model based on differential equations is constructed; for dynamic links with strong nonlinearity, complex coupling, and difficult to accurately describe mechanisms (such as valve characteristics and wear and aging effects), historical normal operation data is used to train a deep learning model, represented by a long short-term memory network, to establish a data-driven model. Furthermore, through Kalman filtering or adaptive weighted fusion algorithms, the outputs of the mechanistic model and the data-driven model are calibrated and fused to generate a set of theoretical benchmark values ​​for key state variables that accurately reflect the system's expected behavior under normal mechanistic constraints, providing a reliable reference baseline for subsequent anomaly simulation, evidence generation, and drift adaptation.

[0068] Furthermore, the digital twin is constructed using a process partitioning mechanism model through a hierarchical and systematic modeling method, which specifically includes the following three steps:

[0069] First, the process system is partitioned and topology mapped. This method, based on the process flow diagram (P&ID) and actual physical connections of the industrial control system, decouples the complex overall process flow into multiple physically clear and functionally independent process units, such as reactor units, heat exchanger units, distillation column units, or storage tank units. Simultaneously, by analyzing the piping and signal connections between units, a topology network describing the overall system's interconnections is constructed. This network clearly defines the transmission direction and coupling relationships of material flow, energy flow, and control signal flow between units, providing a structural framework for subsequent model integration.

[0070] Secondly, unit mechanism modeling is performed. For each independent process unit, a mathematical model of its internal dynamic characteristics is established strictly according to first principles. This is usually expressed as a set of deterministic differential equations, the core of which includes: mass conservation equations based on material inflow and outflow, energy conservation equations considering heat exchange and reaction heat, and momentum balance equations involving pressure and flow resistance. The parameters of the equations (such as volume and heat transfer coefficient) are derived from equipment design data, thus establishing a local dynamic model with clear physical meaning for each unit.

[0071] Finally, a global simultaneous solution is implemented. Based on the system topology network constructed in the first step, the boundary conditions of all independent unit models in the second step (with the output of the upstream unit serving as the input of the downstream unit) are precisely correlated and coupled. This integrates the scattered unit equations into a coupled state-space equation system describing the dynamics of the entire system. By simultaneously solving this large equation system using numerical methods (such as the Euler method and the Runge-Kutta method), the key state variables (such as temperature, pressure, liquid level, and concentration) of all units at any given time can be calculated simultaneously.

[0072] Step S2: Data Augmentation and Alignment Based on Digital Twin

[0073] To address the extreme scarcity of realistic anomaly samples in industrial scenarios, this step involves adversarial simulation within a constructed digital twin. This parametrically simulates various typical industrial control attack modes (such as sensor spoofing attacks, replay attacks, and fake data injection attacks) and equipment failure scenarios (such as valve jamming, actuator failure, and drift faults). After generating the initial synthetic anomaly sequence, a two-stage alignment process is performed to eliminate discrepancies between simulation and reality: first, spatiotemporal alignment, using resampling and interpolation techniques to align the simulation data's time grid to the timestamps of the actual acquired data, ensuring temporal phase consistency; second, distribution alignment (domain adaptation), adjusting the statistical distribution of the simulation data through feature-level affine transformations. Specifically, for each feature dimension, the mean μsim and standard deviation σsim of its simulation data, as well as the mean μreal and standard deviation σreal of the corresponding real historical normal data under the operating conditions, are calculated using the formula:

[0074]

[0075] A transformation is performed to make the synthetic data consistent with the distribution of real data in terms of marginal statistics. Finally, the aligned and enhanced anomalous sample set is mixed with a large amount of historical normal data to construct a relatively balanced and high-quality supervised training set, laying the data foundation for the subsequent training of supervised models and reinforcement learning agents.

[0076] Step S3: Generation of evidence for multi-source heterogeneous anomalies

[0077] To comprehensively capture anomaly signs from different dimensions, this step designs a parallel feature extraction module to process the real-time incoming runtime data and simultaneously generate three complementary types of anomaly evidence:

[0078] Evidence for temporal prediction: Utilizing a pre-trained long short-term memory network branch, sensor readings at the current or next moment are predicted based on recent historical window data, and the residual vector between the actual observation and the predicted value is calculated. This evidence is used to identify sudden anomalies or slow drifts that violate the laws of temporal evolution.

[0079] Distributed reconstruction evidence: Using a pre-trained variational autoencoder branch, the multidimensional sensor data at the current time step is encoded and reconstructed, and the error vector between the original data and the reconstructed data is calculated. This evidence is used to detect hidden anomalies that disrupt normal correlations and joint distribution consistency among variables.

[0080] Classification confidence evidence: A lightweight supervised classifier (such as XGBoost) is pre-trained using the mixed training set obtained in step S2, and the probability score of belonging to the abnormal category is directly output for real-time data. These three sources of evidence serve as prior criteria for decision-making and can be referenced during the fusion process. They provide different characterizations of the degree of anomaly from the perspectives of temporal relevance, data distribution consistency, and class separability.

[0081] Step S4: Constructing a reinforcement learning fusion decision-making environment

[0082] The problem of multi-evidence fusion and final anomaly determination is formalized as a Markov decision process, thereby constructing a training and decision-making environment for reinforcement learning agents:

[0083] State space St: The agent's observation state is composed of multi-source evidence and contextual information, specifically the original sensor observation vector. Predicting the disability Reconstruction error Classification confidence and contextual features that include sliding window statistical features (such as mean, variance, and trend). The concatenated vector, i.e.

[0084]

[0085] Action space It can be designed as a discrete space (e.g.) ∈{0,1}, where 0 represents normal and 1 represents abnormal) or a continuous space (such as outputting the probability value of abnormality). (∈[0,1]), this embodiment prefers discrete actions to directly trigger alarms.

[0086] The asymmetric reward function Rt, designed to align with the safety-first principle of industrial control systems, imposes a significantly higher penalty on false negatives (FN) than on false positives (FP). An example design is as follows: a reward of +1 for correct detection (TP / TN), a penalty of -1 for false positives, and a heavy penalty of -10 for false negatives. This design guides the agent to adopt a conservative strategy under uncertain conditions, prioritizing the anomaly detection rate.

[0087] Step S5: Agent Training and Policy Optimization

[0088] The supervised training set (including labels) constructed in step S2 is used to simulate an interactive reinforcement learning environment. In this environment, a deep reinforcement learning agent based on an Actor-Critic architecture is trained using the Proximal Policy Optimization (PPO) algorithm. The agent (Actor network) outputs an action probability distribution based on its current state St, interacts with the environment, receives a reward Rt, and transitions to the next state St+1. The Critic network evaluates the state value to assist in policy updates. During training, the network parameters are iteratively optimized by maximizing the pruned policy objective function, minimizing the value function error, and adding an entropy regularization term to encourage exploration. Ultimately, the agent learns to adaptively weigh various types of evidence in situations where multi-source evidence may conflict or be uncertain, outputting the optimal anomaly detection decision, thus obtaining the target anomaly detection model, i.e., the trained policy network.

[0089] Step S6: Online Monitoring and Adaptive Updates

[0090] The trained and converged target anomaly detection model is deployed to edge computing nodes or industrial servers close to the data source to achieve low-latency real-time anomaly detection. During online operation, real-time data flows through step S3 to generate multi-source evidence and construct a state St, which is then directly output as "normal / abnormal" after being input into the model. To further improve the system's long-term adaptability, a self-supervised update mechanism is introduced: when the model determines the current state as normal (At=0) and the classification confidence Cconf is lower than the preset safety threshold εsafe (i.e., belonging to a high-confidence normal sample), the system includes this sample in a clean sample pool. Samples from this pool are periodically used to perform online incremental learning or parameter fine-tuning of the data-driven model in the digital twin, enabling the digital twin to evolve synchronously with concept drift caused by equipment aging and gradual changes in operating conditions. In this way, the baseline model of the entire detection system is continuously updated, effectively suppressing the increase in false alarm rate caused by model mismatch and ensuring its long-term detection performance and stability in complex and dynamic industrial environments. This method creatively solves the problem of scarce abnormal data through digital twin technology, and achieves dynamic optimal fusion of multi-source evidence by leveraging the adaptive decision-making capabilities of reinforcement learning. It forms an industrial control system anomaly detection solution that combines high efficiency in sample utilization, comprehensive detection dimensions, adaptive decision-making, and long-term operational robustness.

[0091] like Figure 2 The diagram illustrates the overall architecture of this application in a specific embodiment, representing the complete process from physical entity data acquisition, digital twin construction and data augmentation, multi-branch evidence generation to reinforcement learning fusion decision-making and online adaptive updates. In this specific implementation, the fractionation unit of a Fluid Catalytic Cracking (FCC) unit is preferred as an example for illustration.

[0092] The data layer corresponds to steps S1 and S2 in the embodiment. Specifically, the "real-time data stream" of the "industrial control system" comes from the fractionation unit, which includes multi-dimensional time-series data such as top temperature, pressure, and reflux flow rate. The "digital twin" constructs a hybrid model based on this data and mechanistic knowledge (such as material balance). By simulating attacks and faults such as sensor spoofing and valve jamming in the twin, "abnormal simulation data" is generated. Subsequently, "data alignment" is performed through timestamp alignment and affine transformation based on mean-standard deviation, so that the distribution of simulation data closely matches the real "historical normal data", ultimately forming a "hybrid dataset" for training.

[0093] The model layer corresponds to step S3, where it receives the real-time data stream processed by a sliding window. The LSTM network in the unsupervised model ensemble generates prediction errors, and the VAE model generates reconstruction errors; together, they constitute temporal and distributional evidence. The supervised classification model (such as XGBoost pre-trained using a mixed dataset) outputs a confidence score indicating that the current data is an anomaly. This evidence, along with the contextual features derived from sliding window statistics (such as mean and variance), is concatenated into a comprehensive state vector, fully representing the instantaneous state of the system.

[0094] The dynamic fusion layer corresponds to steps S4 and S5. The core of this layer is a "deep reinforcement learning controller" (such as a PPO agent based on an Actor-Critic architecture). It receives the state vector from the model layer, and its internal policy network outputs "dynamic fusion weights" and a "dynamic fusion policy," which determines how to balance prediction error, reconstruction error, and confidence score to make the final judgment. The judgment result (normal / abnormal) is evaluated according to an asymmetric "reward calculation" rule (e.g., the penalty for false negatives is much higher than for false positives), and the controller policy is continuously iterated and improved through an "optimization update" mechanism (such as gradient updates in PPO), thereby adaptively outputting more robust "abnormal detection results" under varying conditions. The entire process achieves a complete closed loop from data simulation and evidence extraction to intelligent adaptive decision-making.

[0095] In one embodiment, the hybrid driving method specifically includes:

[0096] The structure of the industrial control system is divided into regions according to the process. In each region, a deterministic differential equation mechanism model is established for the subprocesses that follow clear physical laws. Based on the topological connection and process flow structure, the mechanism models of each region are solved jointly.

[0097] Establish data-driven models based on deep neural networks for subprocesses with strong nonlinearity and complex mechanisms;

[0098] The outputs of the mechanistic model and the data-driven model are calibrated using Kalman filtering or weighted fusion algorithms to obtain theoretical baseline values ​​for the key state variables of the system.

[0099] Specifically, in this embodiment, the core of the hybrid driving method lies in integrating prior physical knowledge with data-driven capabilities to construct a high-fidelity digital twin. Its implementation comprises three levels:

[0100] First, for subprocesses in industrial control systems with clear physical laws that can be precisely described by mathematical equations, such as material conservation in reactors, energy balance in heat exchangers, or liquid level dynamics in storage tanks, mechanistic models based on deterministic differential equations (sets) are established. These models originate from first principles, have clear physical meaning and good extrapolation properties, and provide an interpretable theoretical framework for system behavior.

[0101] Secondly, for dynamic processes in the system that exhibit strong nonlinearity, strong coupling, or whose mechanisms are not yet fully understood, such as complex mass and heat transfer effects, non-ideal actuator characteristics (e.g., dead zones, hysteresis), and components significantly affected by random noise, deep neural networks are used for modeling. Network structures with temporal modeling capabilities, such as long short-term memory networks or gated recurrent units, are preferred. These networks are trained using historical normal operation data to learn the complex mapping relationships between multidimensional inputs and key state variables, thus supplementing the shortcomings of pure mechanistic models.

[0102] Finally, since the mechanistic model may contain simplification errors, and the data-driven model may be limited by the training data, the outputs of both need to be co-calibrated. This embodiment uses Kalman filtering or an adaptive weighted fusion algorithm to achieve this process. Kalman filtering, through state estimation and updating, optimally fuses the predicted values ​​of the mechanistic model and the "observed" values ​​of the data-driven model, taking into account process noise and observation noise. Weighted fusion dynamically allocates fusion weights based on the model's confidence level or historical performance under different operating conditions. Through this calibration step, a more accurate and robust theoretical baseline value for the system's key state variables is finally output. This baseline value not only serves as the core output of the digital twin but also provides a "normal" reference baseline for anomaly simulation in subsequent steps, providing a basis for comparison in evidence generation, thus achieving a unity of mechanistic reliability and data adaptability.

[0103] In one embodiment, the abnormal scenario simulation includes at least one parameterized simulation method, the parameterized simulation method including:

[0104] Perform a sensor spoofing attack by superimposing a bias term onto the selected sensor channel. or linear drift term ;

[0105] Perform a replay attack and record historical normal time windows. The data sequence is mapped to the current time window. ;

[0106] False data injection is performed on specific control loops, and the state vector is modified according to the topology to make it satisfy linear constraints and deviate from the real physical state.

[0107] Configure actuator jamming fault settings, set the actuator feedback value to remain constant within a preset time interval, or set the actuator response gain to zero;

[0108] To set up process disturbance faults, external disturbance variables or superimposed step signals that conform to a preset statistical distribution are introduced into the state equation of the digital twin.

[0109] Specifically, in this embodiment, to construct a high-quality synthetic anomaly dataset, various parameterizable anomaly scenario simulations were implemented in the digital twin. These simulations aim to reproduce typical malicious attack and failure modes in real industrial control environments, and their core methods include the following three:

[0110] Firstly, sensor spoofing attack simulation. This method involves selecting one or more key sensor channels (such as temperature and pressure sensors) and programmatically superimposing artificial interference terms onto their real or simulated output readings. Specifically, it takes two forms: one is to superimpose a fixed or slowly varying deviation term δ, causing the measured value to continuously deviate from the true physical quantity; the other is to superimpose a time-dependent linear drift term. This simulates the process of sensors gradually becoming inaccurate. This attack aims to disrupt the control loop's perception of the true state, potentially causing the controller to make dangerous decisions based on erroneous information.

[0111] Secondly, replay attack simulation. This method simulates an attacker stealing and repeatedly sending historically normal data to mask the current anomaly. In practice, the system first extracts a segment of length from the historically normal data database. The data sequence. Then, this recorded historical sequence is completely or partially "replayed" and mapped onto the current simulation's time window. This replaces the actual dynamic data for that time period. This causes the monitoring system to observe a seemingly normal but actually stagnant or cyclical false state, thus bypassing detection based on instantaneous thresholds.

[0112] Third, simulated fake data injection attacks. This is a more covert and complex attack simulation. Instead of randomly altering data, it systematically and collaboratively modifies multiple related state vectors or measurements based on an understanding of the specific control loop topology and inherent physical / logical constraints (material balance, energy conservation, control equations) of the industrial control system. The altered fake data still satisfies these linear or nonlinear constraints at a local or global level, making it difficult to detect during initial consistency-based verification. However, overall, it deviates from the true physical operating state, aiming for long-term latency and slow destruction.

[0113] To construct a more comprehensive hybrid training set, the anomaly scenario simulation in the digital twin further covers typical equipment failure patterns. Its parameterized simulation methods mainly include the following two categories to achieve flexible and controllable reproduction of physical world failures:

[0114] Firstly, the actuator jamming fault setting. Mechanical jamming of actuators (control valves, pumps) is a common fault. This embodiment simulates this fault by parameterizing its behavioral model: one approach is to force the actuator's position feedback value (such as valve opening) to remain a constant value within a preset time interval, regardless of changes in control commands, thus simulating the condition where the actuator is jammed in a certain position. A more refined approach is to simulate a complete loss of driving capability and no response to control signals by setting the actuator's response gain to upstream controller output commands to zero. Both parameters (jamming time window, jamming value, gain attenuation factor) can be flexibly adjusted to generate jamming fault sequences of different severity and duration.

[0115] Secondly, process disturbance fault setting. This method simulates dynamic deviations caused by abnormal changes in internal system parameters or unknown external disturbances. Specifically, this is achieved by introducing external disturbance variables conforming to a preset statistical distribution into the state equations of the digital twin. For example, a Gaussian white noise term with zero mean and adjustable variance can be added to the material or energy balance equation to simulate abnormal disturbances with enhanced randomness. Another strategy is to superimpose a deterministic step signal or ramp signal onto the differential equation of a key state variable (such as reaction temperature or inlet flow rate) to simulate persistent deviations caused by sudden changes in equipment performance (such as increased fouling in heat exchangers) or the sudden intervention of external disturbances. The amplitude, duration, and statistical characteristics of the disturbance can all be parameterized.

[0116] In one embodiment, the step of performing spatiotemporal data alignment and distribution alignment processing on the synthesized anomalous sequence specifically includes:

[0117] Data domain adaptation includes time synchronization and statistical matching. Time synchronization aligns the time grid of the simulation data with the sampling timestamps of the actual acquired data through resampling. Statistical matching corrects the marginal statistics of the simulation data through feature-level affine transformations, satisfying the following:

[0118]

[0119] in The original simulation signal, and The mean and standard deviation of the simulation data. and These represent the mean and standard deviation of the actual historical data under the corresponding operating conditions.

[0120] Specifically, in this embodiment, performing data alignment processing on the synthetic anomaly sequences generated by the digital twin simulation is a crucial domain adaptation step. This step aims to eliminate the inherent differences between the virtual simulation environment and the real physical system, ensuring that the synthetic data can be effectively used to train realistic detection models. This processing mainly includes two continuous and complementary steps: time synchronization and statistical matching.

[0121] Time synchronization primarily addresses the time misalignment issue caused by the inconsistency between the simulation step size and the actual data acquisition cycle. It involves realigning the data sequences output from continuous or discrete time grids within the simulation program with the actual sampling timestamps of the data acquisition units in the real industrial control system. Common methods include linear interpolation or spline interpolation, which estimate the values ​​that the simulated signal should have at the actual sampling time, thus ensuring strict synchronization between the synthesized sequence and the real data stream on the time axis and avoiding erroneous timing characteristics introduced by phase deviations.

[0122] Statistical matching aims to further correct the differences in probability distribution characteristics between simulated and real data. Due to factors such as model simplification and imperfections in noise models, the statistical characteristics of simulated data (such as amplitude range and fluctuation scale) often differ from those of measured data. Therefore, a feature-level affine transformation is used for calibration. For each data feature dimension to be aligned (e.g., a simulation sequence of a specific sensor), the mean μ_sim and standard deviation σ_sim of the simulated data in that dimension are first calculated, along with the mean μ_real and standard deviation σ_real of a large amount of real historical normal data under the corresponding operating conditions. Subsequently, the formula is applied to standardize and rescale each original simulated data point x_sim, transforming it into a new data point with the target statistical characteristics. This transformation essentially translates and scales the distribution of the simulated data, matching its first-order (mean) and second-order (standard deviation) statistical moments with the real data, thereby significantly reducing the distribution offset between the simulation and real domains and enhancing the generalization ability of subsequent data-driven models.

[0123] like Figure 3As shown in the diagram, this illustrates the digital twin data augmentation process for a petroleum refining fractionation unit. The "control system entities" on the left represent the physical fractionation units (towers, heat exchangers, pumps, valves, etc.), whose operational data drives subsequent modeling. The middle layer is the core modeling stage: the "mechanism model" establishes differential equations based on mass / energy conservation; the "data model" (such as LSTM) learns complex nonlinear dynamics; and the two are integrated to construct a high-fidelity "digital twin process model." In this twin model, parameterized anomalies such as "replay attacks," "spoof data injection," and "actuator failures" are injected to simulate and generate diverse synthetic anomaly data (i.e., multiple "user data" modules in the diagram). These simulated data undergo data alignment processing (time synchronization and distribution matching) and are ultimately mixed with real historical normal data to form a high-quality hybrid dataset for subsequent model training.

[0124] In one embodiment, the temporal prediction evidence is obtained by predicting historical data using a Long Short-Term Memory (LSTM) network and calculating the prediction residual; the distribution reconstruction evidence is obtained by reconstructing the current data using a Variational Autoencoder (VAE) and calculating the reconstruction error; and the classification confidence evidence is obtained by outputting the anomaly probability of real-time data using a classification model pre-trained on the supervised training set.

[0125] Specifically, in this embodiment, the generation of multi-source heterogeneous anomaly evidence is achieved through three parallel and complementary technical branches, aiming to comprehensively capture system anomalies from different dimensions.

[0126] First, the generation of time-series predictive evidence relies on a Long Short-Term Memory (LSTM) network. This branch is trained using normal time-series sensor data (such as temperature and pressure sequences) within a historical sliding time window to learn the inherent time dependencies and dynamic evolution patterns among system variables. During online operation, recent historical data is input into the LSTM network to obtain its predicted values ​​for each sensor reading at the current moment. Subsequently, the prediction residual vector is obtained by calculating the element-wise absolute difference between the actual observations and the network predictions. This residual directly quantifies the degree to which the system's real-time behavior deviates from the learned time-series patterns, effectively indicating sudden anomalies or slow drifts that violate temporal causality.

[0127] Secondly, the evidence for distribution reconstruction is provided by the Variational Autoencoder (VAE) branch. During the training phase, the VAE learns from a large amount of normal data to construct a low-dimensional latent probability model capable of characterizing the normal joint distribution among multiple variables. In online applications, the multidimensional observation data at the current time step is input into the VAE, mapped to the latent space by the encoder, and then reconstructed back into the original data space by the decoder, yielding the reconstructed output. The difference between the original input and the reconstructed output is calculated to obtain the reconstruction error vector. Because the VAE excels at reconstructing data that conforms to its learned distribution, when the input data anomalies disrupt the normal statistical association between variables, its reconstruction will produce significant errors, thus revealing anomalies based on distribution consistency.

[0128] Finally, the classification confidence evidence is generated by a pre-trained supervised classification model. This model (e.g., a gradient boosting decision tree) is trained using a mixed training set containing sufficiently synthetic anomalous samples constructed in the preceding steps, learning to distinguish between normal and anomalous patterns. For real-time input observational data, the model directly outputs a probability score representing its anomalous category, serving as a prior confidence score based on historical experience and pattern matching.

[0129] In one embodiment, the state space satisfy:

[0130]

[0131] in, This is the normalized sensor observation vector at the current time t; This represents the prediction residual vector based on time correlation. This is the reconstruction error vector based on distribution consistency; This refers to the contextual statistical features within the sliding time window or the hidden state of the previous time step. Represents vector norm or element-wise absolute value operation.

[0132] Specifically, in this embodiment, the state space of the reinforcement learning agent... This state vector is meticulously constructed as a high-dimensional feature vector, comprehensively integrating the system's current observations, multi-dimensional anomalies, and operational context information, providing ample basis for the agent's decision-making. This state vector is composed of the following five core components using the concatenation function Concat():

[0133] first This represents the sensor observation vector after normalization at the current time t. It contains real-time readings from various sensors in the industrial control system (such as temperature, pressure, and flow transmitters). Normalization aims to eliminate scale differences caused by different physical dimensions, bringing the data to the same order of magnitude.

[0134] It is a prediction residual vector based on time correlation, and its calculation method is as follows: Here It is the prediction of the current moment by the Long Short-Term Memory network based on historical data, with the symbol... This represents the absolute value of the difference element by element. This vector quantifies the deviation between actual observations and ideal values ​​predicted based on historical time series patterns, and is direct evidence for capturing time series anomalies.

[0135] It is the reconstruction error vector based on distribution consistency, and its calculation method is as follows: Here This is the reconstruction output of the variational autoencoder based on the current observation data. This vector reflects the difference between the actual data and the reconstructed data generated from the normal data distribution, and is used to reveal anomalies that disrupt statistical associations and joint distributions between variables.

[0136] It is the classification confidence score output by a pre-trained supervised classification model. It is usually a scalar probability value, representing the prior confidence level that the current data is judged as abnormal, and provides a direct reference for decision-making based on historical pattern learning.

[0137] The state space St represents contextual features, providing additional environmental and historical information needed for decision-making. It can contain statistical features of observed data within a sliding time window (such as mean, variance, and trend) to characterize recent operational status; in some network architectures, it can also be the hidden state of the agent or prediction model from the previous time step, used to convey temporal memory information. By fusing these five types of information, the state space St provides the agent with a comprehensive decision-making view that simultaneously perceives the current situation, temporal violations, distribution deviations, prior judgments, and operational context.

[0138] like Figure 4As shown in the figure, this illustrates the multi-source evidence generation and state construction process using a petroleum catalytic cracking fractionation unit as an example. The process begins with a base model library, where a pre-trained LSTM prediction model receives actual time-series data (such as tower top temperature sequences) from digital twins or the field, outputs predicted values, and calculates the difference between these predictions and actual observations, forming the LSTM prediction error, which serves as evidence of time-series anomalies. Simultaneously, simulated data from the data dictionary and real-time data are input into the VAE reconstruction model. The model outputs reconstructed values ​​and calculates the reconstruction error to capture data distribution anomalies. Furthermore, a supervised classifier (such as XGBoost) analyzes the real-time data, directly outputting anomaly confidence scores to provide prior judgment criteria. On the other hand, environmental context data (such as current load and operating condition indicators) is processed into contextual features (such as sliding window statistics). Finally, the prediction error, reconstruction error, anomaly confidence, and contextual features are concatenated and fused to form a comprehensive state vector characterizing the system's operating status. This vector serves as the input to the subsequent deep reinforcement learning agent, supporting its adaptive fusion and anomaly decision-making.

[0139] In one embodiment, the action space is a discrete action space or a continuous action space, wherein the discrete action space satisfies It directly outputs binary classification results, where 0 represents normal and 1 represents abnormal; the continuous action space satisfies... Output anomaly probability value or dynamically adjusted detection threshold. .

[0140] Specifically, in this embodiment, the action space of the reinforcement learning agent is designed as two selectable types to adapt to different application requirements and deployment scenarios.

[0141] The first type is the discrete action space, i.e. At each decision-making moment, the intelligent agent directly outputs a binary classification result. An action value of "0" indicates that the current system state is "normal," requiring no alarm trigger; an action value of "1" indicates an "abnormal" state, initiating the corresponding alarm and security response mechanism. This design is intuitive and efficient, with clear decision results, facilitating direct integration with existing industrial control security system alarm interfaces. It is suitable for scenarios with extremely high real-time requirements and where explicit "yes / no" responses are needed.

[0142] The second type is continuous action space, i.e. Under this setting, the agent's output is a continuous numerical value between 0 and 1. This value can be interpreted in two ways: first, directly as a probability estimate that the current state is "abnormal," providing operations personnel with continuous quantitative risk assessment; second, as a dynamically adjusted detection threshold. This is used to adjust the decision boundaries of subsequent fixed classifiers in real time. The continuous action space can express richer and more refined decision information, allowing the system to implement risk-level early warning or adaptive sensitivity adjustment, suitable for complex or gradual anomaly detection tasks requiring decision-making. The specific action space used can be selected and configured based on the actual industrial control system's safety level, false alarm tolerance, and specific requirements for decision interpretability.

[0143] In one embodiment, the asymmetric reward function is based on the agent's actions. With real labels The combination of values ​​of the asymmetric reward function The design is as follows:

[0144]

[0145] in, For agent actions, The true labels for the samples; This means that the penalty weight for missed detections is much greater than the penalty weight for false alarms, in order to meet the high security integrity level requirements of industrial control systems.

[0146] Specifically, in this embodiment, the design of the asymmetric reward function is the core mechanism guiding the training of the reinforcement learning agent. This function is based on the actions output by the agent. (e.g., determining normal or abnormal) The actual label corresponding to the current state. The matching relationship between them is used to calculate the immediate reward Rt. The specific rule is designed as follows: when the agent makes a correct judgment, i.e. = A positive reward of +rcorrect is given to reinforce correct decision-making. When the agent makes a false positive (FalsePositive), i.e., At=1 (determined as anomalous) but Lt=0 (actually correct), a penalty is applied. When an agent experiences a false negative, that is... =0 (normal) but When the value is 1 (actual anomaly), a much harsher penalty is imposed. .

[0147] The key to this design lies in the asymmetry of the penalty weights, i.e., the mandatory requirement. This asymmetry makes the negative cost of missed detections far outweigh that of false positives. This asymmetry profoundly influences the agent's policy learning direction: when faced with uncertainty, the agent, in order to minimize cumulative penalties and maximize long-term rewards, tends to adopt a more conservative or vigilant strategy. The agent learns to prioritize triggering alarms (identifying anomalies) rather than risking letting a potential real threat slip by in situations of ambiguous evidence. This mechanism ensures that the trained anomaly detection model fundamentally implements the core principle of "better to false positives than false negatives" in industrial safety, thereby effectively meeting the mandatory safety requirements of low false negative rates for anomaly detection systems in critical infrastructures such as oil, chemical, and power industries, and improving overall protection levels.

[0148] In one embodiment, the agent is trained using a proximal policy optimization algorithm. The agent employs an Actor-Critic network architecture and updates its policy by interacting with the environment, sampling, and optimizing a total loss function that includes a shearing objective function, a value function loss, and an entropy regularization term.

[0149] Specifically, in this embodiment, the training of the agent employs a proximal policy optimization algorithm. This algorithm is based on an Actor-Critic architecture, comprising two cooperating neural networks: the Actor network acts as the policy function, responsible for outputting the probability distribution of decision actions (such as "normal" or "abnormal") based on the current system state (a state vector incorporating multi-source evidence); the Critic network acts as the value function, responsible for evaluating the long-term expected return of the current state, providing directional guidance for policy updates.

[0150] The core of training is optimizing a composite total loss function, which consists of three parts: first, a critical shearing objective function, which ensures training stability and prevents policy mutations by constraining the update magnitude of the old and new policies; second, a value function loss, used to train the Critic network to more accurately estimate state values; and third, an entropy regularization term, used to encourage the policy to maintain a certain degree of exploratory activity and avoid prematurely getting trapped in local optima. The agent interacts with a simulated environment built on digital twins and hybrid datasets, collecting data trajectories of states, actions, rewards, and the next state. Using this data, the total loss function is iteratively optimized using gradient descent methods (such as the Adam optimizer), thereby simultaneously updating the parameters of both the Actor and Critic networks. Specifically, this includes:

[0151] Construct an Actor-Critic dual-network structure. The Actor network takes a state vector as input. Output action probability distribution ; Critic network input Output state value estimation Both networks employ a multilayer perceptron structure, such as two hidden layers (64 neurons each, Tanh activation function) and an output layer.

[0152] The agent interacts with a simulated environment driven by a hybrid training set, and at each step t, it adjusts its actions according to the current policy. Select Action Receive rewards and the next state and the trajectory data ( Store it in the experience buffer.

[0153] Based on the shearing objective function to limit the policy update magnitude, and combining the value function loss and the entropy regularization term to construct the total loss function, an example is shown below:

[0154]

[0155] in MSE can be used. The policy entropy is iteratively updated using the Adam optimizer. The process continues until the cumulative reward or key detection indicators converge, resulting in the target anomaly detection model.

[0156] like Figure 5 As shown in the figure, this diagram illustrates a deep reinforcement learning dynamic fusion decision-making mechanism applied to a fractionation unit. The core is a deep reinforcement learning fusion decision-maker, whose input is a state vector representing the system's operational status (including prediction error, reconstruction error, anomaly confidence, and contextual features). Internally, this decision-maker employs an Actor-Critic network architecture: the Actor network (policy network) generates an action probability distribution—that is, a preference policy for different pieces of evidence—through a Softmax layer, and outputs the final dynamic fusion weights after weight normalization; the Critic network (value network) evaluates the value of the current state, guiding policy updates.

[0157] In the feedback optimization stage, the output dynamic fusion weights are combined with the raw output scores from the base models (LSTM, VAE, classifier). A comprehensive anomaly score is generated through weighted summation. This score then enters the dynamic fusion and feedback execution threshold judgment module, which makes a final determination (normal / abnormal) based on a preset or dynamically adjusted threshold, and outputs the detection result.

[0158] Simultaneously, this result is used in reward calculation. Based on the asymmetric reward function (e.g., a heavy penalty of -10 for missed detections and a light penalty of -1 for false positives), the system calculates the reward value for the current decision and feeds it back to the DRL decision-maker, driving the update and optimization of its Actor-Critic network parameters. This closed-loop process enables the system to adaptively learn the optimal fusion strategy online, thereby achieving accurate and robust anomaly detection under the complex and variable operating conditions of the fractionation unit.

[0159] In one embodiment, satisfying the preset condition means that the target anomaly detection model determines that the current state is normal and the output anomaly confidence level is lower than the safety threshold.

[0160] Online incremental learning is performed on the data-driven model in the digital twin using the high-confidence normal samples at this time;

[0161] The trained agent is deployed on edge computing nodes to perform online inference and output anomaly detection results in real time. When the agent determines that the system is normal and the confidence level meets the preset threshold, the model parameters of the digital twin are fine-tuned online using real-time data to adapt to system drift.

[0162] Specifically, in this embodiment, the preset condition is defined as a dual-requirement screening mechanism to ensure the security and effectiveness of online updates. Specifically, this condition includes: first, the target anomaly detection model (i.e., the trained reinforcement learning agent) deployed on edge computing nodes determines the current real-time data as "normal"; second, the anomaly prior confidence score output by the model in parallel must be lower than a pre-set, extremely stringent safety threshold ε_safe (e.g., 0.05). This dual verification aims to rigorously screen out high-confidence normal samples, thereby minimizing the introduction of potential anomalies or uncertain samples into the update process and preventing model contamination or misleading.

[0163] Once the samples meet the above conditions, the system will automatically trigger an online adaptive update process. The core of this process is to use these high-confidence, normal samples to perform online incremental learning or parameter fine-tuning on key components within the digital twin—the data-driven model (e.g., an LSTM network for state prediction). This process typically employs mini-batch gradient descent or online learning algorithms to progressively adjust model parameters with a small learning rate. In this way, the predictive model in the digital twin can continuously learn from conceptual drift caused by equipment aging, seasonal changes, and the slow decline of catalyst activity, ensuring that the calculation of theoretical baseline values ​​remains synchronized with the actual "normal" behavior of the physical system.

[0164] The closed-loop self-updating mechanism enables field-deployed detection systems to maintain high detection accuracy and low false alarm rates over the long term without frequent, costly manual recalibration or model retraining. The digital twin, evolving in tandem with the physical system, ensures the timeliness and accuracy of the theoretical references that form the cornerstone of the entire detection algorithm, thereby guaranteeing the long-term effectiveness and operational adaptability of the multi-source evidence generation and reinforcement learning-based decision-making processes.

[0165] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0166] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0167] It should be particularly noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, or of course, by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting abnormality of an industrial control system based on digital twinning and reinforcement learning, characterized in that, Includes the following steps: S1. Obtain entity operation data through acquisition units deployed in the field layer and control layer of the industrial control system, and construct the digital twin of the industrial control system by adopting a hybrid driving method that combines the mechanism model of process partitioning and the data-driven model. S2. Simulate various abnormal scenarios in the digital twin to generate a synthetic abnormal sequence, and perform spatiotemporal data alignment and distribution alignment processing on the synthetic abnormal sequence to obtain an enhanced sample set. Mix the enhanced sample set with historical normal data to form a supervised training set. S3. Input the real-time running data into the feature extraction module to generate time-series prediction evidence, distribution reconstruction evidence, and classification confidence evidence in parallel. S4. Define the state space, action space and asymmetric reward function of the agent. The state space includes at least the original sensor data, prediction residual, reconstruction error and classification confidence. S5. Train the agent in a simulated environment using the supervised training set to learn an adaptive fusion strategy for multi-source evidence and obtain a target anomaly detection model. S6. Deploy the trained target anomaly detection model on an edge computing node for real-time anomaly detection, and fine-tune the parameters of the digital twin online using real-time data when preset conditions are met.

2. The method of claim 1, wherein, In step S1, the hybrid driving method specifically includes: The structure of the industrial control system is divided into regions according to the process. In each region, a deterministic differential equation mechanism model is established for the subprocesses that follow clear physical laws. Based on the topological connection and process flow structure, the mechanism models of each region are solved jointly. Establish data-driven models based on deep neural networks for subprocesses with strong nonlinearity and complex mechanisms; The outputs of the mechanistic model and the data-driven model are calibrated using Kalman filtering or weighted fusion algorithms to obtain theoretical baseline values ​​for the key state variables of the system.

3. The method of claim 1, wherein, In step S2, the abnormal scenario simulation includes at least one parameterized simulation method, which includes: Performing a sensor spoofing attack, superimposing a bias term on selected sensor channels or a linear drift term ; Performing a replay attack, recording a historical normal time window of data sequences and mapping it to a current time window ; False data injection is performed on specific control loops, and the state vector is modified according to the topology to make it satisfy linear constraints and deviate from the real physical state. Configure actuator jamming fault settings, set the actuator feedback value to remain constant within a preset time interval, or set the actuator response gain to zero; To set up process disturbance faults, external disturbance variables or superimposed step signals that conform to a preset statistical distribution are introduced into the state equation of the digital twin.

4. The method of claim 1, wherein, Step S2, which involves performing spatiotemporal data alignment and distribution alignment on the synthesized anomalous sequence, specifically includes: Data domain adaptation includes time synchronization and statistical matching. Time synchronization aligns the time grid of the simulation data with the sampling timestamps of the actual acquired data through resampling. Statistical matching corrects the marginal statistics of the simulation data through feature-level affine transformations, satisfying the following: ； wherein is the original simulation signal, is the mean and standard deviation of the simulation data, and are the mean and standard deviation of the real historical data, respectively.

5. The method of claim 1, wherein, In step S3, the temporal prediction evidence is obtained by predicting historical data using a Long Short-Term Memory (LSTM) network and calculating the prediction residual; the distribution reconstruction evidence is obtained by reconstructing the current data using a Variational Autoencoder (VAE) and calculating the reconstruction error; and the classification confidence evidence is obtained by outputting the anomaly probability of real-time data using a classification model pre-trained on the supervised training set.

6. The method according to claim 1, characterized in that, In step S4, the state space satisfies: ； wherein, is the normalized sensor observation vector at the current time instant t; is the prediction residual vector based on time correlation; is the reconstruction error vector based on distribution consistency; is the context statistical feature within the sliding time window or the hidden state at the previous time instant, denotes the vector norm or element-wise absolute value operation.

7. The method according to claim 1, characterized in that, In step S4, the action space is a discrete action space or a continuous action space, wherein the discrete action space satisfies , and directly outputs a binary classification result, 0 representing normal and 1 representing abnormal. The continuous action space satisfies , outputting an abnormal probability value or a dynamically adjusted detection threshold .

8. The method according to claim 1, characterized in that, In step S4, the asymmetric reward function takes values in combination with the real label of the agent action In step S4, the asymmetric reward function takes values in combination with the real label is designed as follows: ； wherein, is the agent action, is the sample true label; That is, the penalty weight of the false negative is much larger than that of the false positive to meet the high safety integrity level requirement of the industrial control system.

9. The method of claim 1, wherein, In step S5, the agent is trained using the Proximal Policy Optimization (PPO) algorithm. The agent adopts an Actor-Critic network architecture and updates its policy by interacting with the environment, sampling, and optimizing the total loss function, which includes the shearing objective function, the value function loss, and the entropy regularization term.

10. The method of claim 1, wherein, In step S6, satisfying the preset condition means that the target anomaly detection model determines that the current state is normal and the output anomaly confidence level is lower than the safety threshold. Online incremental learning is performed on the data-driven model in the digital twin using the high-confidence normal samples at this time; The trained agent is deployed on edge computing nodes to perform online inference and output anomaly detection results in real time. When the agent determines that the system is normal and the confidence level meets the preset threshold, the model parameters of the digital twin are fine-tuned online using real-time data to adapt to system drift.

Images