Anomaly detection method for fine chemical production process

Abstract

The present application belongs to the field of computer and chemical cross-disciplinary multi-monitoring point monitoring abnormality detection, and proposes an abnormality detection method for fine chemical production process, an abnormality recognition intelligent agent model is randomly sampled in the data set and obtains the current state, the abnormality recognition intelligent agent model gives the action according to the current state; the reinforcement learning training environment returns the most abnormal next time state according to the data set, and takes the next time state as the input of the abnormality recognition intelligent agent model at the next time step; the reinforcement learning training environment gives the corresponding reward according to the judgment of the abnormality recognition intelligent agent model on the data; the abnormality recognition intelligent agent model learns the network weight according to the neuron, and obtains the optimal network weight of the abnormality recognition intelligent agent model for abnormality detection. The method of the present application firstly combines the pulse neural network with the deep reinforcement learning to complete the abnormality detection task in the chemical production scene, and has low detection power consumption and wide use scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of anomaly detection in computer and chemical cross-disciplinary multi-monitoring point monitoring, and particularly relates to an anomaly detection method for fine chemical production process. BACKGROUND

[0002] Fine chemical production is a highly automated process, and its production equipment is composed of various sensors and actuators to achieve automated operation. Historically, this task was done by experts who pre-set thresholds for each sensor. If the measured value exceeds the threshold defined by the relevant expert, the system will consider that it is not running as expected. In the chemical production process, the generation of abnormal data can lead to problems such as increased defective rate, material loss, and even threats to personal safety and the environment. Therefore, chemical data anomaly detection technology is of great significance for early detection of abnormal situations in production, ensuring the safe, stable and efficient operation of the production process, and improving product quality and yield. However, traditional methods rely on expert judgment and lack the combination of intelligent judgment and data characteristics. Overall, chemical data has the following three characteristics: 1) Multivariate: Chemical processes involve multiple variables, which increases the complexity of anomaly detection and requires consideration of the relationship and mutual influence between multiple variables. 2) High dimensionality: Data in chemical processes often contains a large number of variables and is often continuously changing. This makes anomaly detection need to handle a large amount of data and effectively handle high-dimensional data space. 3) Uncertainty of prior information: In the chemical process, prior information may be incomplete or inaccurate, which increases the difficulty of anomaly detection.

[0003] Based on the above characteristics, existing chemical abnormality detection methods focus more on unsupervised algorithms, which use the vast amount of data recorded by enterprise production line sensors to build, fit, and predict models. This method follows the trend of the times and has many emerging technologies, such as linear regression, logistic regression, decision tree, support vector machine, random forest, Boosting, etc. Compared with traditional methods, this algorithm has great advantages, but at the same time, this model also has some shortcomings. First, this model ultimately boils down to a data model, and the data collected is usually normal data. After training the model with correct data, when encountering abnormal data, it may also be mistaken for normal data, i.e. poor generalization ability; second, since there are more than one type of abnormality in actual chemical production, the existing method cannot effectively judge when encountering multiple abnormal types of data, making the abnormality detection in the chemical environment have limitations; third, there is a lot of noise in actual chemical production, and a large amount of repeated noise will significantly reduce the detection accuracy for unsupervised algorithms. Therefore, how to learn the basic patterns of normal behavior using a small amount of labeled data based on a large amount of unlabeled data and detect abnormalities is one of the hotspots of research in recent years. Compared with unsupervised methods, weakly supervised methods can significantly reduce the labeling cost while maintaining high detection performance.

[0004] Weakly supervised methods focus on using the underlying information in labeled anomalies to train model parameters to explore potential abnormal features. Although these methods perform well in related fields, their performance often depends on the quality of a small amount of labeled data. When faced with a situation where known abnormal features cannot cover all data features, their performance and generalization ability will be greatly reduced. In addition, these methods perform particularly poorly when dealing with noisy and contaminated scene information. These problems hinder the application of this type of method in actual chemical production to some extent.

[0005] To solve these problems, some researchers have shifted their attention to reinforcement learning (RL). Reinforcement learning is a new learning paradigm in which an agent interacts with the environment by learning a strategy to maximize rewards or achieve specific goals. The recently proposed detection method DPLAN based on reinforcement learning has achieved good performance in multiple anomaly detection tasks and has autonomous learning and adaptive capabilities. However, this method still cannot effectively detect anomalies in actual chemical production scenarios, mainly because the sampling method used has limitations and cannot effectively explore all abnormal types, the method used has weak resistance to noise in complex environments, resulting in poor stability of the method, and there are a large number of invalid alarms, which cannot meet the actual scene requirements.

[0006] Therefore, based on the investigation and analysis of the existing chemical production process anomaly detection method, it is found that the existing method has problems such as poor convergence and accuracy of the model for large noise and non-stationary actual production process variable data. Based on this situation, first, we design a model combining reinforcement learning and pulse neural network to solve the problem that the existing method has weak noise resistance. Compared with the existing method, our method can combine reinforcement learning to learn autonomously, detect unknown advantages and the noise resistance of the pulse neural network, accurately identify anomalies in actual chemical production, secondly, by designing a normalized cosine similarity algorithm, the feature dimension of the chemical data is reduced and the correlation between the dimension features is reduced, which effectively improves the generalization of the algorithm in dealing with complex chemical data; finally, by designing a reasonable reward algorithm and data division, our method can fully utilize the small amount of prior information in chemical production, and through data representation and promotion, the ability to identify unknown types of anomalies is realized. The experimental results prove that the present application can effectively identify multiple types of anomalies in the production process, reduce the false positive rate, and realize the deployment of a multivariate monitoring system in actual chemical production processes. SUMMARY

[0007] The purpose of the present application is to detect anomalies in the monitoring points of complex chemical production processes, and to achieve efficient and accurate anomaly detection results by innovatively using pulse neural networks and reinforcement learning methods.

[0008] The technical solution of the present application is an anomaly detection method for fine chemical production processes, which constructs a data set , a reinforcement learning training environment and an anomaly recognition agent model; the anomaly recognition agent model performs random sampling and obtains the current state s for the first time in the data set t ; the anomaly recognition agent model gives an action a t or a 0 according to the current state s 1 ; the reinforcement learning training environment returns the next time state s that is most abnormal according to the data set t+1 , and takes the next time state s t+1 as the input of the anomaly recognition agent model at the next time step; the reinforcement learning training environment gives the corresponding reward according to the judgment of the anomaly recognition agent model at time t for the data; in each time step t, the anomaly recognition agent model learns the network weight according to the LIF neuron and the IF neuron, the LIF neuron and the IF neuron are used as the activation function, and the anomaly recognition agent model with the optimal network weight is obtained for anomaly detection.

[0009] The construction process of the data set is as follows;

[0010] Capture the delay behavior, nonlinear behavior between M sensors in chemical production process, map the behavior information of M sensors to storage medium to form historical data, manually mark part of the abnormal data, convert the data into table type standardized data set Wherein The labeled abnormal data set is divided into And unlabeled data set

[0011] The reinforcement learning training environment improves the reward function and sampling function based on the existing algorithm DQN; the improved sampling function is a multi-angle abnormal deviation sampling function; the multi-angle abnormal deviation sampling function is defined as:

[0012]

[0013]

[0014] S dis ={i 1:5}where score dis (i1)≥…≥score dis (i5)

[0015] S LOF ={j 1:5}where score LOF (j1)≥…≥score LOF (j5)

[0016] S iForest ={k 1:5}where score iForest (k1)≥…≥score iForest (k5)

[0017] S intersection =S dis ∩S LOF ∩S iForest

[0018]

[0019] Wherein, φ(s t ) represents a random sampling function in the labeled abnormal data set , p represents the probability of sampling from And ; s t+1 Indicates the state at the next time; ψ(s t ) represents finding Anomaly bias function, score dis (s t ) is the anomaly score calculated by the cosine similarity algorithm; norm cos_sim (s,s t ) represents the distance between the current state s and other states in the data set after orthogonalization, i, j, k respectively represent the current data set Calculate S dis , S LOF , S iForest , the potential anomaly with the highest anomaly score; S intersection represents the intersection between the potential anomalies with the highest anomaly scores, and the anomaly score function S LOF , S iForest is obtained by inputting the data set into two classical unsupervised algorithms LOF, iForest, and through the detection results of the unsupervised algorithm;

[0020] The anomaly score score cos_sim of the normalized cosine similarity algorithm norm dis , the anomaly score score LOF of the LOF algorithm, and the anomaly score score iForest of the isolation forest algorithm are combined to measure the potential anomalies of the data set according to the combined score; for each algorithm, the top five abnormal states with the highest anomaly scores and their corresponding scores S dis , S LOF , S iFore are calculated; the abnormal states and the corresponding scores are combined into a final state space set; the three state spaces are crossed, and when the intersection is not empty, the state with the largest total anomaly score is selected as the state information s t+1 of the reinforcement learning training environment, otherwise, the state with the highest anomaly score in the normalized cosine similarity algorithm is selected as the state information s t+1 of the reinforcement learning training environment;

[0021] The improved reward function considers external rewards and internal rewards. The external reward function is designed for the labeled anomaly data set , and the internal reward function is designed for the unlabeled data set . Specifically, for the internal reward function, the unsupervised algorithm COPOD is introduced, and the detection result score(s t ) of the unsupervised algorithm is used to guide the agent to explore the environment. Our improved reward function can be defined as:

[0022]

[0023] threshold is a threshold value, the external reward assigned to each action of the agent depends on its current state and action. If the anomaly identification agent model selects a 1 action in step t and the state is in , it will get a larger reward, indicating that the anomaly identification agent model correctly identifies the anomaly in . For the intrinsic reward, the effect of the unsupervised method is adjusted by a threshold value. Here, score(s t ) represents the anomaly score of the current state s t . If the anomaly score is higher than the set threshold, the current state s t may have potential anomalies. When the anomaly identification agent model correctly identifies the anomaly, it will get a large reward. On the contrary, if the anomaly score is lower than the threshold, the current state s t is considered normal, and the anomaly identification agent model will receive a small reward to encourage exploration behavior.

[0024] The anomaly identification agent model construction process is as follows;

[0025] An anomaly identification agent model ADSD is constructed based on a spiking neural network; the agent aims to find the optimal execution action a 0 or a 1 to maximize the task-specific reward under the set reinforcement learning training environment setting; the anomaly identification agent model ADSD includes an input layer, a fully connected layer, and an output layer, wherein:

[0026] The input layer receives state information from the reinforcement learning training environment; the input layer includes an IF neuron;

[0027] The IF neuron acts as an encoder and receives standard data format input in the data set; when the total amount of input signals exceeds the threshold value, the IF neuron generates a single timing pulse output signal; the mathematical model of the IF neuron is represented as:

[0028]

[0029] w j represents the weight of the input signal, x j represents the size of the input signal, and θ represents the threshold value;

[0030] The full connection layer receives the time sequence pulse output signal from the input layer;LIF neuron is used as the activation function of the full connection layer, when a LIF neuron receives the time sequence pulse output pulse of other neurons, the membrane potential is accumulated according to the current neural network connection structure, and the LIF neuron is gradually reduced to the resting potential through a function decaying with time;The form of LIF neuron is defined as follows:

[0031]

[0032] V represents the membrane potential, E L represents the resting potential, I(t) represents the time sequence pulse output signal received by the neuron, R m represents the membrane resistance, τ m represents the membrane capacitance;

[0033] The output layer outputs the action value function Q to obtain the action at the current moment, and inputs into the reinforcement learning training environment;The action value function Q and the loss function are consistent with the existing algorithm DQN, which is specifically shown as follows:

[0034]

[0035]

[0036] Wherein Q * (s,a) represents that after taking action a in state s, r t represents the immediate reward obtained at time step t, γ is the discount factor (0<γ<1), and the loss function evaluates the performance of the current action value function Q parameter θ i ; ρ(·) is the state behavior distribution in the experience replay buffer, ε is the experience replay buffer of the environment;By obtaining the optimal action value function Q, the optimal decision is finally obtained.

[0037] The beneficial results of the application are as follows:

[0038] 1. Since the state of the process parameters shows dynamicity, nonlinearity, continuity and periodicity along with the production process, and the chemical reaction has a large time lag, the industrial parameters have strong correlation. Based on the characteristics of the process parameters of the chemical production line, a new abnormality detection method for fine chemical production process is proposed to improve the adaptability to complex structure abnormal data tasks. To our knowledge, we are the first to combine pulse neural network and deep reinforcement learning to complete the abnormality detection task in the chemical production scene.

[0039] 2. Low power consumption

[0040] The pulse neural network uses a pulsed way to transmit information, compared with the continuous activity of the traditional neural network, the neuron of the pulse neural network consumes little energy in the non-activated state, and the neuron is activated and consumes large energy only when the pulse transmission occurs. Energy efficiency optimization, the sparse coding and event-driven principle are used in the activation function, so that the activation times of the neuron are reduced, thereby reducing the overall power consumption. In addition, through the specific neuromorphic hardware, the power consumption of the pulse neural network can be reduced to 1% compared with the artificial neural network, which makes the application suitable for deployment on edge devices, such as mobile phones, watches and other devices, which greatly improves the use possibility of the method in the actual environment.

[0041] 3. Wide use scenarios

[0042] The method has obvious advantages for the following types of data: a) unknown abnormal type data, because it is difficult to comprehensively collect abnormal information in chemical production, there are a large number of unknown abnormal type data, due to the lack of prior information of such abnormality, it is difficult to identify such abnormality; b) abnormal data close to normal data characteristics; c) abnormal data with a large number of correlations between data characteristics; d) normal data close to abnormal data (false positives). BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A structure diagram of an abnormality detection method for fine chemical production process.

[0044] Figure 2 An abnormality result display of an embodiment of the application. DETAILED DESCRIPTION

[0045] The main technical points of the application are: 1) introducing a pulse neuron to filter a large amount of noise in the chemical production data; 2) reducing data correlation to amplify abnormal data characteristics based on multi-angle sampling function through feature dimension scaling; 3) designing an abnormality recognition intelligent agent based on the actual production environment of the chemical industry. The method described in the application is suitable for complex systems similar to the chemical industry, such as water plant abnormal attacks, power plant abnormal operation, server abnormal attacks, and abnormal mechanical operation. An abnormality detection method for fine chemical production process, which realizes the judgment of abnormal conditions in the chemical production process.

[0046] An abnormality detection method for fine chemical production process, comprising the following steps:

[0047] Step 1, constructing a data set required for training

[0048] The delay and non-linear behavior between M sensors (assuming that the chemical production process involves M sensors) are captured by RP (Recursive plot), the behavior information of the M sensors is mapped to a storage medium to form historical data, and a part of the abnormal data is manually marked, and then the data is converted into tabular standardized data wherein is divided into a labeled abnormal data set and an unlabeled data set

[0049] Step 2, constructing a reinforcement learning training environment

[0050] The reinforcement learning training environment improves the reward function and the sampling function based on the existing algorithm DQN. In order to enable the abnormal identification intelligent agent model to effectively explore the unlabeled abnormal, we designed a multi-angle abnormal deviation sampling function in this part. Generally, the sampling method based on distance can only find the most obvious abnormal, and often falls into local optimum. In order to alleviate the influence of this problem, we use a multi-angle abnormal deviation function to represent the potential of a group of abnormal points, which is of great significance to explore the abnormal space. The improved sampling function is defined as:

[0051]

[0052]

[0053] S dis ={i 1:5}where score dis (i1)≥…≥score dis (i5)

[0054] S LOF ={j 1:5}where score LOF (j1)≥…≥score LOF (j5)

[0055] S iForest ={k 1:5}where score iForest (k1)≥…≥score iForest (k5)

[0056] S intersection =S dis ∩S LOF ∩S iForest #

[0057]

[0058] where φ(s t ) denotes a random sampling function in the labeled anomaly dataset . The anomaly scores score cos_sim of the normalized cosine similarity algorithm norm dis , score LOF of the LOF algorithm, and score iForest of the isolation forest algorithm are combined to measure potential anomalies comprehensively. In particular, each score is pre-processed to achieve normalization. For each algorithm, the top five anomaly states (i,j,k) and their corresponding scores (S dis ,S LpF ,S iForest ) are computed. These special states and their scores are combined to form a final state space set, which is then intersected. If the intersection is not empty, the state with the largest anomaly score sum is selected as the state information s t+1 for the reinforcement learning training environment. Otherwise, the state with the highest anomaly score in the normalized cosine similarity algorithm is selected as s t+1 . This design helps the agent to access as many anomaly types as possible in the state space.

[0059] Step 3, building an anomaly identification agent model

[0060] The anomaly identification agent model is built based on spiking neural networks, aiming to find the optimal execution action under various environmental settings to maximize task-specific rewards. Among them, the IF (integrated-and-fire) and LIF (leaky integrated-and-fire) spiking neurons are selected in the SNN. The IF neuron acts as an encoder, which receives signal inputs from other neurons, accumulates them, and generates a single output signal as long as the total input signal exceeds the threshold. Its mathematical model can be represented as:

[0061]

[0062] w j denotes the weight of the input signal, x j denotes the size of the input signal, and θ denotes the threshold. In addition, the LIF neuron is used as a decision maker. When a LIF neuron receives pulses from other neurons, these pulses will be scaled according to the learned synaptic weights. By summing all the scaled pulses, the depolarization process is achieved. At the same time, through a function that decays over time, the membrane potential can gradually return to the polarization state. Therefore, the LIF is defined as follows:

[0063]

[0064] V represents the membrane potential, E L represents the resting potential, I(t) represents the input current received by the neuron, R m represents the membrane resistance, and τ m represents the membrane capacitance. In particular, V th is set to infinity, so that the neuron does not generate a pulse output, ensuring that the output value of the Q function is continuous.

[0065] Step 4, the process of anomaly identification

[0066] Through steps 2 and 3, the reinforcement learning training environment required for training and the anomaly identification agent model are obtained, and then the anomaly is inferred; the specific process is: the agent obtains the current state s t , gives the action after judgment; the next time state s t+1 of the most suspected anomaly is returned through the reinforcement learning training environment designed in step 2, and appropriate rewards are given according to the agent's judgment of the data at this time, so that it can learn to distinguish abnormal data.

[0067] 1. Overall process

[0068] (1) Preprocess the data set collected by the production line Among them represents a small-scale abnormal data set, represents a large-scale normal data set;

[0069] (2) Initialize the parameters in the anomaly identification agent model network, including the action value function Q function, the target function, the network weight, the experience replay pool, etc.

[0070] (3) The agent obtains data and selects actions, and continuously iterates training according to the reward feedback from the environment, updates the network weight according to the current training step, calculates the corresponding reward of the current state, calculates the network parameters and loss function, and continuously iterates until the training is completed.

[0071] (4) Deploy the trained model to the corresponding platform, and the model judges the data anomaly according to the current data.

[0072] 2. Anomaly identification agent model network training parameter settings

[0073] The anomaly detection smart agent model ADSD is trained for 20,000 steps with 10,000 warm-up steps and the target-network is updated every 10,000 steps. Each episode contains 2,000 steps. An episode is only considered complete when 2,000 steps have been completed. The parameters θ of the reward function are updated at the end of each episode (i.e. 2,000 steps). The experience replay queue size is set to 100,000, the minibatch size is set to 32, the discount factor γ is set to 0.99, and the learning rate is set to 0.00025.

Claims

1. An anomaly detection method for fine chemical production processes, characterized in that, Building a dataset , Reinforcement learning training environment and anomaly detection agent model; the anomaly detection agent model is based on the dataset The first random sampling is performed to obtain the current state. The anomaly detection intelligent agent model is based on the current state. Give an action ; Reinforcement learning training environment based on dataset Return to the most abnormal state in the next time step. and the state at the next moment. As the input to the next time step of the anomaly detection agent model; the reinforcement learning training environment provides corresponding rewards based on the anomaly detection agent model's judgment of the data at time t; in each time step t, the anomaly detection agent model learns network weights based on LIF neurons and IF neurons, with the LIF neurons and IF neurons serving as activation functions, to obtain an anomaly detection agent model with optimal network weights for anomaly detection; The reinforcement learning training environment is based on the existing DQN algorithm with improved reward and sampling functions; The improved sampling function is a multi-angle anomaly deviation sampling function; the multi-angle anomaly deviation sampling function is defined as follows: ； in, Indicates anomalies in labeled datasets The random sampling function in Indicates from respectively and The probability of sampling in the middle; Indicates the state at the next moment; Indicates searching Abnormal bias function of potential abnormal states The anomaly score calculated by the cosine similarity algorithm; This represents the distance between the current state s and other states in the dataset after orthogonalization. These represent the current dataset. calculate At that time, the potential abnormality with the highest abnormality score; The anomaly score function represents the intersection of potential anomalies with the highest anomaly scores. , It is by passing the dataset The input is fed into two classic unsupervised algorithms, LOF and iForest, and the results are obtained through the detection of the unsupervised algorithms. Normalized cosine similarity algorithm abnormal scores Outlier score of LOF algorithm Anomaly scores in the Isolation Forest algorithm Combination, based on the combined score measure dataset Potential anomalies; for each algorithm, calculate the top five anomaly states with the highest anomaly scores and their corresponding scores. , , Abnormal states and their corresponding scores form the final state space set; when the intersection of the three state spaces is not empty, the state with the largest sum of abnormal scores is selected as the state information of the reinforcement learning training environment. Otherwise, the state with the highest anomaly score in the normalized cosine similarity algorithm is selected as the state information of the reinforcement learning training environment. ; The improved reward function considers both external and internal rewards; the external reward function is for labeled anomaly datasets. The design incorporates an internal reward function tailored to the unlabeled dataset. Design; For the intrinsic reward function, an unsupervised algorithm, COPOD, is introduced, and the detection results of the unsupervised algorithm are used... Given a reward; the improved reward function is defined as: ； The external reward assigned to each action of the anomaly detection agent model is based on its current state and action, with the threshold being used as the threshold. The process of constructing the anomaly detection intelligent agent model is as follows; An anomaly detection agent model ADSD is constructed based on a spiking neural network; the agent's goal is to find the optimal action to perform under a given reinforcement learning training environment. To maximize task-specific rewards; the anomaly detection agent model ADSD consists of an input layer, a fully connected layer, and an output layer, wherein: The input layer receives state information from the reinforcement learning training environment; the input layer includes IF neurons. The IF neuron acts as an encoder, receiving standard data format input from the dataset. When the total amount of input signal exceeds a threshold, the IF neuron generates a single time-sequential pulse output signal. The mathematical model of the IF neuron is expressed as: ； The weights of the input signal are represented. Indicates the magnitude of the input signal. Indicates the threshold; The fully connected layer receives the temporal pulse output signal from the input layer. Using LIF neurons as the activation function of the fully connected layer, when a LIF neuron receives the temporal pulse output signal from other neurons, it accumulates the temporal pulse signal according to the current neural network connection structure to generate a membrane potential. The LIF neuron uses a time-decaying function to gradually reduce the membrane potential until it eventually returns to the resting potential. The formal definition of a LIF neuron is as follows: ； Represents membrane potential. Represents the resting potential. The output signal represents the time-series pulse received by the neuron. Represents film resistance. Represents film capacitance; The output layer outputs the action value function Q to obtain the action at the current time, and inputs it into the reinforcement learning training environment; the action value function Q and the loss function are consistent with the existing algorithm DQN, specifically as follows: ;in Indicates the state Take action Then follow the optimal strategy get, This represents the immediate reward obtained at time step t. Discount factor (0 < < 1), the loss function evaluates the value of the current action function Q parameters Performance; For the distribution of state behavior in the experience replay buffer, This serves as a buffer for replaying environmental experiences; by obtaining the optimal action value function Q, the optimal decision is ultimately achieved. The dataset The construction process is as follows; Capturing the chemical production process The delay and nonlinear behavior between individual sensors will The behavioral information of each sensor is mapped to a storage medium to form historical data. A portion of the abnormal data is manually labeled, and the data is converted into a standardized tabular dataset. ,in It was divided into labeled anomaly datasets. and unlabeled datasets , .

Images