Industrial control system abnormal behavior identification method and device

Through reinforcement learning and domain adaptive BERT model, a system that can efficiently detect abnormal behavior in complex industrial control environments is built, which solves the problems of incomplete rule coverage and inappropriate assumptions in the existing technology, and achieves efficient and accurate abnormal behaviors Identify.

CN118898041AInactive Publication Date: 2024-11-05BEIJING EASYNETWORKS TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411243604.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing industrial control system abnormal behavior recognition methods rely on artificially defined rules and thresholds, which are difficult to completely cover the diversity and variability in complex systems, affecting the recognition accuracy, and the assumption premise based on statistical analysis may not meet the actual situation, resulting in Degraded detection performance.

Method used

Using reinforcement learning method to build an abnormal behavior recognition model, the prediction results of sample behavior characteristics are used as actions, and the accuracy of the prediction results is rewarded. The domain adaptive BERT model and self-coding model are used to extract and reduce dimensionality features to build a complex environment A medium and efficient system for detecting abnormal behaviors.

Benefits of technology

It realizes efficient abnormal behavior detection in complex and dynamic industrial control environments, adaptively responds to environmental changes, improves prediction accuracy and efficiency, reduces equipment failure rate, and improves the safety and stability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118898041A_ABST
    Figure CN118898041A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of abnormal behavior recognition, and provides an industrial control system abnormal behavior recognition method and device, and the method comprises the steps: extracting behavior characteristics of to-be-recognized operation data in an industrial control system; inputting the behavior characteristics into an abnormal behavior recognition model to obtain an abnormal behavior recognition result output by the abnormal behavior recognition model; the abnormal behavior recognition model is obtained by taking the prediction result of the sample behavior characteristics as an action, taking the accuracy degree of the prediction result as an award and performing reinforcement learning. According to the method, the prediction result of the sample behavior characteristics serves as an action, the accuracy degree of the prediction result serves as an award, the abnormal behavior recognition model is constructed in a reinforcement learning mode, the prediction strategy can be continuously optimized in the interaction process with the environment, and efficient abnormal behavior detection can be kept in the complex and dynamic industrial control environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal behavior recognition, and in particular to a method and device for recognizing abnormal behavior of an industrial control system. Background Art

[0002] Industrial Control System (ICS) abnormal behavior recognition refers to the use of various sensor data and monitoring information to identify abnormal or abnormal operating behaviors in industrial control systems through algorithms and models. Industrial control system abnormal behavior recognition can help predict equipment failures, optimize production processes, or prevent potential safety issues.

[0003] At present, most of the detection of abnormal behavior relies on pre-defined rules and thresholds. By monitoring and comparing the parameters and status of industrial control systems, when certain rules are triggered or parameters exceed the set range, an abnormal alarm will be issued. However, this method relies on manually defined rules and thresholds, which require professionals to manually set and update. It requires experts to have an in-depth understanding of the system and abnormal behavior to ensure the accuracy and coverage of the rules. However, industrial control systems are often very complex, and the diversity and variability of abnormal behaviors are also very large. It is difficult to fully cover all situations by relying solely on manually defined rules, which in turn affects the accuracy of abnormal behavior identification. Summary of the invention

[0004] The present invention provides an abnormal behavior identification method and device for an industrial control system, which are used to solve the defects in the prior art.

[0005] The present invention provides a method for identifying abnormal behavior of an industrial control system, comprising the following steps: Extract behavioral features of operating data to be identified in industrial control systems; Inputting the behavior feature into an abnormal behavior recognition model to obtain an abnormal behavior recognition result output by the abnormal behavior recognition model; The abnormal behavior recognition model is obtained by reinforcement learning, taking the prediction results of sample behavior characteristics as actions and the accuracy of the prediction results as rewards.

[0006] According to a method for identifying abnormal behavior of an industrial control system provided by the present invention, the step of acquiring the abnormal behavior identification model comprises: Randomly extracting experience data from the experience playback buffer, the experience data is used to characterize the current operating state of the industrial control system, the prediction result of the sample behavior characteristics under the current operating state, the accuracy of the prediction result under the current operating state, and the next operating state of the industrial control system after executing the behavior corresponding to the prediction result; Determining a target expected return value based on the accuracy of the prediction result corresponding to the next operating state and the accuracy of the prediction result under the current operating state; Based on the target expected return value, updating the return value of the predicted result under the current operating state; Based on the reward value of the prediction result under the current operating state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model.

[0007] According to an abnormal behavior identification method for an industrial control system provided by the present invention, the initial model includes multiple branches; The method of updating the initial model parameters based on the reward value of the prediction result under the current operating state and the target expected reward value to obtain the abnormal behavior recognition model includes: Based on the empirical data, determining occlusion information, wherein the occlusion information is used to characterize key points of interest for each branch; Determine a loss value based on the reward value of the prediction result under the current operating state and the target expected reward value; Based on the loss value and the occlusion information, parameters of each branch of the initial model are updated respectively to obtain the abnormal behavior recognition model.

[0008] According to a method for identifying abnormal behavior of an industrial control system provided by the present invention, the step of extracting behavioral features of operating data to be identified in the industrial control system includes: Extracting initial behavior features from the operation data to be identified; Performing dimensionality reduction processing on the initial behavior feature to obtain the behavior feature.

[0009] According to a method for identifying abnormal behavior of an industrial control system provided by the present invention, the step of extracting initial behavior features from the operation data to be identified comprises: Extracting initial behavior features from the to-be-identified operation data based on a feature extraction model; The feature extraction model is obtained by fine-tuning the pre-trained model using a fine-tuning task, wherein the fine-tuning task includes at least one of an industrial control system log classification task, a system event prediction task, and an event association relationship reasoning task.

[0010] According to an abnormal behavior identification method for an industrial control system provided by the present invention, the dimensionality reduction processing of the initial behavior feature to obtain the behavior feature includes: Based on the feature dimensionality reduction model, the initial behavior feature is subjected to dimensionality reduction processing to obtain the behavior feature; The feature dimensionality reduction model is constructed based on the encoder in the autoencoder model, and the autoencoder model includes an encoder and a decoder. The encoder is used to reduce the dimensionality of the initial behavior features of the sample to obtain the sample behavior features, and the decoder is used to restore the sample behavior features to the original dimensional features.

[0011] The present invention also provides an abnormal behavior identification device for an industrial control system, comprising the following modules: An extraction unit, used to extract behavioral features of operating data to be identified in the industrial control system; An identification unit, used for inputting the behavior feature into an abnormal behavior identification model to obtain an abnormal behavior identification result output by the abnormal behavior identification model; The abnormal behavior recognition model is obtained by reinforcement learning, taking the prediction results of sample behavior characteristics as actions and the accuracy of the prediction results as rewards.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for identifying abnormal behavior of an industrial control system as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for identifying abnormal behavior of an industrial control system as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for identifying abnormal behavior of an industrial control system.

[0015] The method and device for identifying abnormal behavior of an industrial control system provided by the present invention use the prediction results of sample behavior characteristics as actions and the accuracy of the prediction results as rewards, and construct an abnormal behavior recognition model through reinforcement learning. The method and device can continuously optimize the prediction strategy during the interaction with the environment, and can maintain efficient abnormal behavior detection in a complex and dynamic industrial control environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1It is a flow chart of the abnormal behavior identification method of the industrial control system provided by the present invention.

[0018] Figure 2 It is a flow chart of the abnormal behavior recognition model construction method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the abnormal behavior identification device of the industrial control system provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] When identifying abnormal behaviors in industrial control systems, they often rely on pre-defined rules and thresholds to detect abnormal behaviors. By monitoring and comparing the parameters and states of industrial control systems, abnormal alarms are issued when certain rules are triggered or when parameters exceed the set range. However, this method relies on manually defined rules and thresholds, which require professionals to manually set and update them. It also requires experts to have an in-depth understanding of the system and abnormal behaviors to ensure the accuracy and coverage of the rules. However, industrial control systems are often very complex, and the diversity and variability of abnormal behaviors are also very large. It is difficult to fully cover all situations by relying solely on manually defined rules, which in turn affects the accuracy of abnormal behavior identification.

[0023] In addition, there are also methods based on statistical analysis of historical data of industrial control systems to identify abnormal behaviors. By building models to describe the statistical characteristics of normal behaviors, when new data differ significantly from the established models, they are considered abnormal behaviors. However, methods based on statistical analysis are usually based on some assumptions, such as data obeying a specific probability distribution, abnormal behaviors having specific statistical characteristics, etc. In actual industrial control systems, the distribution of data and the characteristics of abnormal behaviors may change due to the complexity of the system, and these assumptions may not be fully met, thus affecting the detection performance. On the other hand, methods based on statistical analysis usually require a certain understanding and assumptions about the data distribution of industrial control systems. If the data distribution changes or the assumptions are inaccurate, the detection performance may be reduced.

[0024] In addition, there are also methods that use machine learning algorithms to build deep learning models for abnormal behavior identification, such as learning and extracting features from large-scale real-time data, identifying patterns and rules of abnormal behavior, and providing early warning prompts. However, abnormal behavior detection methods based on machine learning usually require feature extraction or feature selection of raw data. Since data in industrial control systems may have high dimensions, complex correlations and nonlinear characteristics, it may not be able to fully capture the characteristics of the data; if the distribution of training data is inconsistent with the data in the actual deployment environment, the generalization ability of the model may be reduced, and abnormal behavior in actual industrial control systems may not be accurately detected.

[0025] The present invention provides a method for identifying abnormal behaviors of industrial control systems. The method uses a BERT model based on domain adaptive fine-tuning and reinforcement learning to identify abnormal behaviors of industrial control systems, realize monitoring of operating data, predict abnormal behaviors of equipment and parameter changes in various industry scenarios, adaptively respond to dynamically changing environments and potential threats, improve the accuracy and efficiency of predictions, enhance the safety and stability of industrial control systems, reduce equipment failure rates, and improve production continuity.

[0026] Figure 1 FIG. 1 is a flow chart of the abnormal behavior identification method of the industrial control system provided by the present invention. Figure 1 As shown, the method includes step 110 and step 120.

[0027] Step 110: extracting behavioral features of the operating data to be identified in the industrial control system.

[0028] Here, in the industrial control system, the operation data to be identified can be understood as the status data and operation data of each device in the industrial control system. By monitoring and analyzing these data, abnormal situations can be detected and responded to in a timely manner to ensure the safe, stable and efficient operation of the equipment. Among them, the operation data to be identified can include operation data (such as equipment switching, starting, stopping, parameter adjustment and other operations), operation status change data (such as changes in equipment operation status, such as speed changes, load changes, etc.), production index change data (such as changes in indicators such as output, quality, energy consumption, etc. during the production process), etc.

[0029] Optionally, the operating data to be identified can be collected through sensor data, such as an industrial control system can use various sensors to collect real-time data (temperature, pressure, flow, etc.) during equipment operation; the operating data to be identified can also be obtained through controller records, such as the controller recording the equipment's operation history, parameter settings, and system status changes.

[0030] In addition, the behavior features of the operation data to be identified are used to characterize key information of each device, such as device status information, device operation information, etc. Optionally, the behavior features can be extracted from the operation data to be identified by a pre-trained deep neural network (such as a BERT model).

[0031] Step 120: input the behavior feature into the abnormal behavior recognition model to obtain an abnormal behavior recognition result output by the abnormal behavior recognition model; The abnormal behavior recognition model is obtained through reinforcement learning, with the predicted results of sample behavior characteristics as actions and the accuracy of the predicted results as rewards.

[0032] Specifically, the abnormal behavior recognition model is used to detect abnormal behaviors or events in the running data to be recognized. It is obtained through reinforcement learning, with the prediction results of sample behavior features as actions and the accuracy of the prediction results as rewards. The accuracy of the prediction results refers to the correctness and precision of the prediction or classification made by the abnormal behavior recognition model for the input sample behavior features.

[0033] That is to say, the abnormal behavior recognition model in the embodiment of the present invention is a reinforcement learning network, which learns the optimal prediction strategy by interacting with the industrial control system environment. The reinforcement learning network includes four parts, namely, state, action, reward, and environment. Among them, the state is the operating state of the industrial control system at a certain moment; the action is the decision made by the abnormal behavior recognition model in the current state (that is, the prediction result of the sample behavior characteristics in the current state), that is, "normal" or "abnormal"; the reward can be defined based on the accuracy of the prediction result, and a positive reward is given for correctly identifying normal behavior, a higher positive reward is given for correctly identifying abnormal behavior, a negative reward is given for misjudging normal behavior as abnormal, and a higher negative reward is given for misjudging abnormal behavior as normal; the environment is the operating data and state of the industrial control system.

[0034] The abnormal behavior identification method for industrial control systems provided by the embodiment of the present invention takes the prediction results of sample behavior characteristics as actions and the accuracy of the prediction results as rewards, and constructs an abnormal behavior identification model through reinforcement learning. It can continuously optimize the prediction strategy during the interaction with the environment and maintain efficient abnormal behavior detection in complex and dynamic industrial control environments.

[0035] Based on the above embodiment, the step of acquiring the abnormal behavior recognition model includes: Randomly extracting experience data from the experience playback buffer, the experience data is used to characterize the current operating state of the industrial control system, the prediction results of the sample behavior characteristics under the current operating state, the accuracy of the prediction results under the current operating state, and the next operating state of the industrial control system after executing the behavior corresponding to the prediction results; Determine the target expected return value based on the accuracy of the prediction results corresponding to the next operating state and the accuracy of the prediction results under the current operating state; Based on the target expected return value, update the return value of the predicted result under the current running state; Based on the reward value of the prediction result under the current operating state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model.

[0036] Specifically, the experience replay buffer can be understood as a buffer for storing historical experience data. Randomly sampling small batches of experience data from the experience replay buffer can reduce the correlation between adjacent samples in training, thereby helping to prevent sample sequence correlation during training and improve the stability of model training. In addition, randomly sampling experience data can make more efficient use of computing resources, that is, different samples can be processed in parallel, thereby speeding up the training process.

[0037] The experience replay buffer can be used to store previous experience tuples, which usually include state (current operating state of the industrial control system, state), action (prediction result of sample behavior characteristics under the current operating state, action), reward (accuracy of prediction result, reward), next state, etc. (next operating state of the industrial control system after executing the behavior corresponding to the prediction result, next_state), and whether the state corresponding to the current experience tuple is the terminal state (done). During training, experience data is randomly extracted from the experience replay buffer to update the model parameters. After updating the model parameters, new experience tuples can be added to the experience replay buffer for continued use in the next round of training.

[0038] In addition, the target expected return value is determined based on the accuracy of the prediction result corresponding to the next running state (i.e., the reward of the next running state) and the accuracy of the prediction result in the current running state (i.e., the reward of the current running state). The target expected return value is used to guide the update of the return value of the prediction result in the current running state. The larger the return value, the greater the probability of selecting the prediction result as the action in the current running state. Optionally, the return value of the prediction result of the current running state can be updated based on the model learning rate. Among them, the target expected return value can be a target Q value, and the return value of the prediction result in the current running state can be the output of the Q value function head. The output of each Q value function head is used to estimate the uncertainty of the Q value function, guide the exploration process, and then tend to explore states and actions that the current model is not very sure about.

[0039] After determining the reward value of the prediction result and the target expected reward value under the current running state, the loss value can be determined based on the difference between the two, and the initial model parameters can be updated based on the loss value to obtain the abnormal behavior recognition model. The initial model can be a pre-trained model.

[0040] Based on any of the above embodiments, the initial model includes multiple branches; Based on the reward value of the prediction result under the current running state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model, including: Based on empirical data, determine the occlusion information, which is used to characterize the key points of concern to each branch; Determine the loss value based on the return value of the predicted result under the current running state and the target expected return value; Based on the loss value and occlusion information, the parameters of each branch of the initial model are updated respectively to obtain the abnormal behavior recognition model.

[0041] Specifically, the initial model can be constructed based on the Bootstrapped DQN (Bootstrapped Deep Q-Network) network. The Bootstrapped DQN network contains a shared convolutional layer and multiple independent output heads (bootstrap heads). Each output head can be regarded as an independent score, and each output head corresponds to an estimate of a Q-value function.

[0042] After obtaining the experience data from the experience playback buffer, the experience data can be feature analyzed to determine the key features corresponding to each branch (that is, the key points that each branch focuses on). For example, the key features corresponding to each branch can be evaluated based on the feature selection algorithm. According to the key features corresponding to each branch, the corresponding occlusion information is generated, and the occlusion information is used to define the attention allocation or feature selection strategy of each branch.

[0043] At the same time, based on the difference between the reward value of the predicted result in the current running state and the target expected reward value, the loss value can be determined. According to the task of each branch, different loss functions can be used to calculate the corresponding loss value. For example, the cross entropy loss function can be used for classification tasks, or the mean square error loss function can be used for regression tasks.

[0044] After determining the occlusion information and loss value, the key points of concern for each branch can be determined according to the occlusion information, and a corresponding weight can be assigned to each branch, and the loss value can be weighted based on the weight to obtain the weighted loss corresponding to each branch, and the parameters of each branch can be updated based on the weighted loss. After the training reaches a preset round or until the predetermined performance target is reached, the abnormal behavior recognition model can be obtained. Among them, the Q network parameters corresponding to each Q value function head can be updated based on the weighted loss.

[0045] Based on any of the above embodiments, extracting behavioral features of the operating data to be identified in the industrial control system includes: Extracting initial behavior features from the operation data to be identified; The initial behavior features are processed by dimensionality reduction to obtain behavior features.

[0046] Considering that the initial behavior features extracted from the running data to be identified are usually high-dimensional feature vectors, a large amount of computing resources and a long training time are usually required during the training process. The embodiment of the present invention performs dimensionality reduction processing on the initial behavior features, thereby significantly reducing the dimension of the data, reducing the computational complexity, and improving the efficiency of the algorithm.

[0047] Optionally, redundant and irrelevant features in the initial behavior features can be removed, and key features can be retained to achieve dimensionality reduction of the initial behavior features, improve the training effect and prediction performance of the model, alleviate the problems of overfitting and poor generalization ability, and improve the performance and stability of the model. Among them, principal component analysis, feature selection and other methods can be used to remove redundant and irrelevant features in the initial behavior features, retain key features, and achieve dimensionality reduction of the initial behavior features.

[0048] Based on any of the above embodiments, extracting initial behavior features from the operation data to be identified includes: Based on the feature extraction model, the initial behavior features are extracted from the running data to be identified; The feature extraction model is obtained by fine-tuning the pre-trained model using a fine-tuning task, where the fine-tuning task includes at least one of an industrial control system log classification task, a system event prediction task, and an event correlation relationship reasoning task.

[0049] Specifically, the feature extraction model is obtained by fine-tuning the pre-trained model using the fine-tuning task. The pre-trained model is a neural network model pre-trained on large-scale data, which is usually used to solve various natural language processing tasks. These models learn the common features and patterns of language, such as grammatical structure, semantic relationship and contextual understanding, by self-supervising learning on a large amount of text data. Among them, the pre-trained model can include the BERT model. The BERT model learns from text data through unsupervised learning in the pre-training stage without the need for manual annotation tasks, which enables the BERT model to capture a large amount of language knowledge and general language laws. The fine-tuned BERT model can learn important features and patterns in industrial control system data and capture rich contextual information. This ability is also effective when processing industrial control system data, and can generate high-dimensional feature representations with semantic information.

[0050] In addition, fine-tuning refers to applying the pre-trained model to the fine-tuning task and adjusting it according to the dataset of the fine-tuning task to optimize the performance of the model. The purpose of fine-tuning is to use the general language representation learned by the pre-trained model on large-scale data to quickly adapt to the needs of the fine-tuning task.

[0051] Among them, the fine-tuning task includes at least one of the industrial control system log classification task, the system event prediction task and the event association relationship reasoning task. The industrial control system log classification task is used to enable the model to identify the event type to which the log belongs; the system event prediction task is used to enable the model to predict the next possible event of the current event; the event association relationship reasoning task is used to enable the model to determine whether two events are related.

[0052] Based on any of the above embodiments, the initial behavior features are subjected to dimensionality reduction processing to obtain behavior features, including: Based on the feature dimensionality reduction model, the initial behavior features are reduced in dimension to obtain the behavior features; The feature dimensionality reduction model is constructed based on the encoder in the autoencoder model. The autoencoder model includes an encoder and a decoder. The encoder is used to reduce the dimensionality of the initial behavior features of the sample to obtain the sample behavior features, and the decoder is used to restore the sample behavior features to the original dimensional features.

[0053] Here, the autoencoder model is an unsupervised learning neural network model, which consists of two parts: an encoder and a decoder. It is mainly used for feature learning and reconstruction of data. The encoder is responsible for mapping the input data (such as initial behavior features) to a low-dimensional representation of a latent space. This process is usually achieved by reducing the dimension of the network or applying specific dimensionality reduction techniques (such as principal component analysis). The output of the encoder is the sample behavior features after dimensionality reduction. The decoder receives the latent representation output by the encoder (i.e., the features after dimensionality reduction) and tries to map it back to the dimensional space of the original input to reconstruct data as close to the original input as possible. The goal of the decoder is to minimize the reconstruction error so that the decoded data can retain the key features of the original data.

[0054] The high-dimensional initial behavior features extracted by the fine-tuned BERT model usually require a lot of computing resources and a long training time during the training process. Feature dimensionality reduction can significantly reduce the dimension of the data, thereby reducing the computational complexity and improving the efficiency of the algorithm.

[0055] Figure 2 is a flow chart of the abnormal behavior recognition model construction method provided by the present invention, such as Figure 2 As shown, the method includes: First, data collection and data preprocessing are performed. If the BERT model needs to be fine-tuned, the industrial control system log classification task, system event prediction task, and event correlation relationship reasoning task are used as fine-tuning tasks, and the BERT model is fine-tuned to obtain a feature extraction model.

[0056] When it is necessary to train a feature dimensionality reduction model, the output of the feature extraction model is used as the input of the autoencoder model. The autoencoder model includes an encoder and a decoder. The encoder is responsible for mapping the input data X to a low-dimensional representation in a latent space. The decoder receives the latent representation output by the encoder (i.e., the reduced-dimensional feature Z) and tries to map it back to the dimensional space of the original input to reconstruct data X that is as close as possible to the original input. After the training is completed, the encoder is exported as a feature dimensionality reduction model.

[0057] Next, an initial model is obtained, which may be obtained by initializing a Bootstrapped DQN network, and the initial model includes multiple branches. For the Bootstrapped DQN network, a branch can be understood as a bootstrap head, and each head corresponds to a Q-value function head.

[0058] Create an experience replay buffer, which is used to store the experience tuples (state, action, reward, next_state, done) of the agent's interaction with the environment. During the training process, small batches of experience data will be randomly sampled from the experience replay buffer for training to reduce the correlation between samples and improve training efficiency and stability.

[0059] Then, the network is trained. In each training round, the following steps are performed: S1. At the beginning of each round, randomly select a Q-value function head as the Q-value function of the current round and use the greedy strategy to select actions based on the currently selected Q-value function head; S2, execute the selected action and observe the reward and next state returned by the environment; S3, store the experience tuple (state, action, reward, next_state, done) into the experience playback buffer and record the occlusion information; S4, randomly sampling a small batch of experience data from the experience playback buffer and updating the network parameters corresponding to each Q-value function head according to the occlusion information; S5, regularly update the initial model parameters, the target network is used to stabilize the training process; S6. Use the output of each Q-value function head to estimate the uncertainty of the Q-value function and guide the exploration process to tend to explore those states and actions that the current model is less certain about.

[0060] The training ends when the preset number of rounds is reached or when the predetermined performance target is reached. After the training is completed, the performance of the abnormal behavior recognition model on the test set can be used for final evaluation.

[0061] The abnormal behavior identification method of industrial control system provided by the embodiment of the present invention realizes real-time monitoring of the operation status of industrial control system by combining the domain adaptive BERT model with the reinforcement learning network, and quickly and accurately warns of potential abnormal behavior, getting rid of the influence of the traditional industrial control system abnormal behavior detection method on the reliance on expert experience and the subjective judgment of operation and maintenance personnel; solves the problem of incomplete coverage of detection rules of the traditional industrial control system abnormal behavior detection method, and avoids the omission of abnormal behavior; does not rely on specific data distribution assumptions, through the domain adaptive BERT model and reinforcement learning, does not rely on specific data distribution assumptions, so it is more flexible and adaptable. It can fully learn the correlation between different system operation data and abnormal states, and avoid the influence of individual erroneous data on abnormal behavior detection. Use the relevant data of the industrial control system to construct a specific fine-tuning task, and fine-tune the pre-trained BERT model, so that the BERT model can master the specific feature patterns of the industrial control system data and fully capture the characteristics of the industrial control system data.

[0062] The abnormal behavior identification device for an industrial control system provided by the present invention is described below. The abnormal behavior identification device for an industrial control system described below and the abnormal behavior identification method for an industrial control system described above can be referenced to each other.

[0063] Based on any of the above embodiments, Figure 3 Schematic diagram of the structure of the abnormal behavior identification device for industrial control system provided by the present invention. Figure 3 As shown, the device comprises: An extraction unit 310 is used to extract behavioral features of the operating data to be identified in the industrial control system; The identification unit 320 is used to input the behavior feature into the abnormal behavior identification model to obtain the abnormal behavior identification result output by the abnormal behavior identification model; The abnormal behavior recognition model is obtained through reinforcement learning, with the predicted results of sample behavior characteristics as actions and the accuracy of the predicted results as rewards.

[0064] Based on any of the above embodiments, the step of acquiring the abnormal behavior recognition model includes: Randomly extracting experience data from the experience playback buffer, the experience data is used to characterize the current operating state of the industrial control system, the prediction results of the sample behavior characteristics under the current operating state, the accuracy of the prediction results under the current operating state, and the next operating state of the industrial control system after executing the behavior corresponding to the prediction results; Determine the target expected return value based on the accuracy of the prediction results corresponding to the next operating state and the accuracy of the prediction results under the current operating state; Based on the target expected return value, update the return value of the predicted result under the current running state; Based on the reward value of the prediction result under the current operating state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model.

[0065] Based on any of the above embodiments, the initial model includes multiple branches; Based on the reward value of the prediction result under the current running state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model, including: Based on empirical data, determine the occlusion information, which is used to characterize the key points of concern to each branch; Determine the loss value based on the return value of the predicted result under the current running state and the target expected return value; Based on the loss value and occlusion information, the parameters of each branch of the initial model are updated respectively to obtain the abnormal behavior recognition model.

[0066] Based on any of the above embodiments, extracting behavioral features of the operating data to be identified in the industrial control system includes: Extracting initial behavior features from the operation data to be identified; The initial behavior features are processed by dimensionality reduction to obtain behavior features.

[0067] Based on any of the above embodiments, extracting initial behavior features from the operation data to be identified includes: Based on the feature extraction model, the initial behavior features are extracted from the running data to be identified; The feature extraction model is obtained by fine-tuning the pre-trained model using a fine-tuning task, where the fine-tuning task includes at least one of an industrial control system log classification task, a system event prediction task, and an event correlation relationship reasoning task.

[0068] Based on any of the above embodiments, the initial behavior features are subjected to dimensionality reduction processing to obtain behavior features, including: Based on the feature dimensionality reduction model, the initial behavior features are reduced in dimension to obtain the behavior features; The feature dimensionality reduction model is constructed based on the encoder in the autoencoder model. The autoencoder model includes an encoder and a decoder. The encoder is used to reduce the dimensionality of the initial behavior features of the sample to obtain the sample behavior features, and the decoder is used to restore the sample behavior features to the original dimensional features.

[0069] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the abnormal behavior identification method of the industrial control system, the method comprising: extracting the behavior characteristics of the operating data to be identified in the industrial control system; inputting the behavior characteristics into the abnormal behavior identification model to obtain the abnormal behavior identification result output by the abnormal behavior identification model; the abnormal behavior identification model is obtained by reinforcement learning with the prediction result of the sample behavior characteristics as the action and the accuracy of the prediction result as the reward.

[0070] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the industrial control system abnormal behavior identification method provided by the above-mentioned methods, and the method includes: extracting the behavior characteristics of the operating data to be identified in the industrial control system; inputting the behavior characteristics into an abnormal behavior identification model to obtain an abnormal behavior identification result output by the abnormal behavior identification model; the abnormal behavior identification model is obtained by reinforcement learning with the prediction result of the sample behavior characteristics as the action and the accuracy of the prediction result as the reward.

[0072] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the industrial control system abnormal behavior identification method provided by the above-mentioned methods, the method comprising: extracting the behavior characteristics of the operating data to be identified in the industrial control system; inputting the behavior characteristics into an abnormal behavior identification model to obtain an abnormal behavior identification result output by the abnormal behavior identification model; the abnormal behavior identification model is obtained by reinforcement learning with the prediction result of the sample behavior characteristics as the action and the accuracy of the prediction result as the reward.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying abnormal behavior of an industrial control system, characterized in that: include: Extract behavioral features of operating data to be identified in industrial control systems; Inputting the behavior feature into an abnormal behavior recognition model to obtain an abnormal behavior recognition result output by the abnormal behavior recognition model; The abnormal behavior recognition model is obtained by reinforcement learning, using the prediction results of sample behavior characteristics as actions and the accuracy of the prediction results as rewards; The step of acquiring the abnormal behavior recognition model includes: Randomly extracting experience data from the experience playback buffer, the experience data is used to characterize the current operating state of the industrial control system, the prediction result of the sample behavior characteristics under the current operating state, the accuracy of the prediction result under the current operating state, and the next operating state of the industrial control system after executing the behavior corresponding to the prediction result; Determining a target expected return value based on the accuracy of the prediction result corresponding to the next operating state and the accuracy of the prediction result under the current operating state; Based on the target expected return value, updating the return value of the predicted result under the current operating state; Based on the reward value of the prediction result under the current operating state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model.

2. The method for identifying abnormal behavior of an industrial control system according to claim 1, characterized in that: The initial model includes multiple branches; The method of updating the initial model parameters based on the reward value of the prediction result under the current operating state and the target expected reward value to obtain the abnormal behavior recognition model includes: Based on the empirical data, determining occlusion information, wherein the occlusion information is used to characterize key points of interest for each branch; Determine a loss value based on the reward value of the prediction result under the current operating state and the target expected reward value; Based on the loss value and the occlusion information, parameters of each branch of the initial model are updated respectively to obtain the abnormal behavior recognition model.

3. The method for identifying abnormal behavior of an industrial control system according to any one of claims 1 to 2, characterized in that: The extracting behavior characteristics of the operation data to be identified in the industrial control system includes: Extracting initial behavior features from the operation data to be identified; Performing dimensionality reduction processing on the initial behavior feature to obtain the behavior feature.

4. The method for identifying abnormal behavior of an industrial control system according to claim 3, characterized in that: The extracting of initial behavior features from the to-be-identified operation data includes: Extracting initial behavior features from the to-be-identified operation data based on a feature extraction model; The feature extraction model is obtained by fine-tuning the pre-trained model using a fine-tuning task, wherein the fine-tuning task includes at least one of an industrial control system log classification task, a system event prediction task, and an event association relationship reasoning task.

5. The method for identifying abnormal behavior of an industrial control system according to claim 3, characterized in that: The performing dimensionality reduction processing on the initial behavior feature to obtain the behavior feature includes: Based on the feature dimensionality reduction model, the initial behavior feature is subjected to dimensionality reduction processing to obtain the behavior feature; The feature dimensionality reduction model is constructed based on the encoder in the autoencoder model, and the autoencoder model includes an encoder and a decoder. The encoder is used to reduce the dimensionality of the initial behavior features of the sample to obtain the sample behavior features, and the decoder is used to restore the sample behavior features to the original dimensional features.

6. An abnormal behavior identification device for an industrial control system, characterized in that: include: An extraction unit, used to extract behavioral features of operating data to be identified in the industrial control system; An identification unit, used for inputting the behavior feature into an abnormal behavior identification model to obtain an abnormal behavior identification result output by the abnormal behavior identification model; The abnormal behavior recognition model is obtained by reinforcement learning, using the prediction results of sample behavior characteristics as actions and the accuracy of the prediction results as rewards; The step of acquiring the abnormal behavior recognition model includes: Randomly extracting experience data from the experience playback buffer, the experience data is used to characterize the current operating state of the industrial control system, the prediction result of the sample behavior characteristics under the current operating state, the accuracy of the prediction result under the current operating state, and the next operating state of the industrial control system after executing the behavior corresponding to the prediction result; Determining a target expected return value based on the accuracy of the prediction result corresponding to the next operating state and the accuracy of the prediction result under the current operating state; Based on the target expected return value, updating the return value of the predicted result under the current operating state; Based on the reward value of the prediction result under the current operating state and the target expected reward value, the initial model parameters are updated to obtain the abnormal behavior recognition model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for identifying abnormal behavior of an industrial control system as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying abnormal behavior of an industrial control system as claimed in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying abnormal behavior of an industrial control system as claimed in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Network transaction information identification method and system

    CN115099299A

  • Hydropower station hoist monitoring method and system based on deep learning

    CN117908456A

  • Network anomaly intrusion detection model training method, detection method and device

    CN118018426A

  • Data processing method based on speech recognition model

    CN118411983A

  • Operation management method and system for efficient and energy-saving air conditioner room

    CN118517769A

Cited By

  • Method and system for processing equipment performance statistical data

    CN119357849A