Information system data anomaly detection method based on unsupervised learning and memory fusion

Abstract

The present application relates to a kind of information system data anomaly detection method based on unsupervised learning and memory fusion, solve the current method containing less labeled sample and sample category distribution imbalance, historical data characteristics are difficult to be well utilized.The method includes the process of information system data acquisition and data preprocessing, the process of constructing the network structure of data anomaly detection model, the process of training data anomaly detection model, the process of verifying data anomaly detection model, the process of testing data anomaly detection model and the process of applying data anomaly detection model.The present application is applicable to time series data anomaly detection field and computer technology application field, and the method is beneficial to improve the accuracy of information system data anomaly detection, to improve the reliability of information system.

Description

Technical Field

[0001] This invention belongs to the fields of time-series data anomaly detection and computer technology application, specifically relating to a data anomaly detection method for information systems based on unsupervised learning and memory fusion. Background Technology

[0002] With the rapid development of information technology and artificial intelligence, various industries are constantly evolving towards digitalization, informatization, and intelligentization. This leads to increasingly complex information system application software, generating massive amounts of data during operation. Monitoring key data is crucial for better understanding the overall operational status of the information system. However, manual data monitoring is tedious, burdensome, and prone to oversights. Furthermore, it faces challenges such as the large volume of time-series data, the limited number of labeled samples, and the unbalanced distribution of sample types. The long operating time of information systems may also make it difficult to effectively utilize historical data features. Therefore, an efficient and automated method for detecting data anomalies in information systems is essential. Summary of the Invention

[0003] This invention provides a data anomaly detection method for information systems based on unsupervised learning and memory fusion, which solves the problems of current methods having few labeled samples, unbalanced sample types, and difficulty in effectively utilizing historical data features.

[0004] The technical solution of this invention is: A data anomaly detection method for information systems based on unsupervised learning and memory fusion includes the following processes: information system data acquisition and preprocessing; constructing the network structure of a data anomaly detection model; training the data anomaly detection model; validating the data anomaly detection model; testing the data anomaly detection model; and applying the data anomaly detection model. The steps of this invention are as follows: The first step is information system data collection, which involves selecting data with significant monitoring value from the sensor data, command data, communication data, status data, process data, and log data of the information system for collection. The second step is to preprocess the collected data. First, the data is encoded and de-identified. Then, it is normalized. Finally, four different transformations are used to enhance the data to obtain the final dataset. The third step is to construct the network structure of the data anomaly detection model, which mainly consists of four parts: encoder, self-supervised module, memory fusion module, and decoder. The fourth step is to train the anomaly detection model using the training set, including forward propagation, loss calculation, backpropagation, and weight update. The fifth step is to evaluate the data anomaly detection model using the validation set data, including the forward propagation and calculation of validation metrics. Based on the evaluation results, different strategies are selected. If the evaluation results are unsatisfactory, hyperparameters such as batch size, training epochs, dropout rate, and learning rate are adjusted, and the model is retrained in the fourth step. If the evaluation results are good, the model is moved to the sixth step. The sixth step is to test the data anomaly detection model using the test set data, including the forward propagation and calculation of validation metrics. The seventh step is to deploy the trained data anomaly detection model to the actual production environment, process real data, and continuously monitor the performance of the data anomaly detection model. Step 8, End.

[0005] This invention is applicable to the fields of time-series data anomaly detection and computer technology applications. This method helps to improve the accuracy of data anomaly detection in information systems, thereby improving the reliability of information systems. Attached Figure Description

[0006] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the anomaly detection model. Detailed Implementation

[0007] The specific content and implementation of the present invention will be further described below with reference to the accompanying drawings and specific embodiments: like Figure 1 As shown, an information system data anomaly detection method based on unsupervised learning and memory fusion includes the following processes: information system data acquisition and preprocessing; constructing the network structure of a data anomaly detection model; training the data anomaly detection model; validating the data anomaly detection model; testing the data anomaly detection model; and applying the data anomaly detection model. The steps of this invention are as follows: The first step is information system data acquisition. From the information system's sensor data, command data, communication data, status data, process data, and log data, select N data points with significant monitoring value for acquisition. The sampling period is T, each sample contains L sampling points, and the size of each sample is L. N. Normal data collection is conducted when the information system is in normal working condition, including different working modes and scenarios. Abnormal data collection is conducted when the software is in an abnormal state, including problems such as sensor malfunctions, information processing malfunctions, configuration file malfunctions, communication malfunctions, and human error.

[0008] The second step is data preprocessing. The collected data needs to be preprocessed. First, the data is encoded and de-identified. Then, it is normalized. Finally, four different transformations are used to enhance the data to obtain the final dataset.

[0009] Normalization is performed using the Min-Max Normalization method, where... The data before normalization. This is the normalized data.

[0010]

[0011] Data augmentation can be performed using the following data transformation methods: (1) Noise transformation: Gaussian noise is added to the data.

[0012] (2) Smoothing transformation: The data is smoothed by using a low-pass filter SavitzkyGolay (SG).

[0013] (3) Reverse transformation: The sample is transformed in reverse along the time dimension to obtain a sample with the opposite time direction.

[0014] (4) Negative transformation: The sample is transformed in the negative direction along the dimension of the value to obtain a mirror image of the input data.

[0015] The enhanced samples are divided into training, validation, and test sets. The training set contains only normal samples and is used for training the unsupervised learning model; the validation and test sets contain both normal and abnormal samples and are used for validating and testing the model.

[0016] The third step is to construct the network structure of the data anomaly detection model. The data anomaly detection model mainly consists of four parts: an encoder, a self-supervised module, a memory fusion module, and a decoder. The overall structure of the network model is as follows: Figure 2 As shown.

[0017] The encoder consists of a padding layer and two Conv2D+ReLU+Maxpooling layers, which can extract latent feature tensors.

[0018] The self-supervised module consists of a Conv2D+Sigmoid layer, a Flatten layer, two fully connected layers, and a Softmax layer, which outputs binary classification results. The self-supervised module establishes a binary classification auxiliary task, leveraging this task to mine its own supervisory information from large-scale unsupervised data. Because the training set only contains normal data, the data itself implicitly contains binary classification labels. Training the binary classification network and encoder with this implicit supervisory information helps the encoder learn representations valuable for downstream tasks.

[0019] The memory fusion module takes as input the features extracted by the encoder, reshapes these features to obtain new features, denoted as... A memory matrix is ​​used to record multiple patterns of normal features. This memory matrix is ​​denoted as... During the training phase, the memory matrix can be updated through a reconstruction loss function, forcing it to record normal features. Reconstructing anomalies using normal patterns retrieved from the memory matrix will lead to higher reconstruction errors. With memory matrix The previous matching degree is denoted as Match degree The calculation method is as follows:

[0020]

[0021] in, , Indicates the degree of matching The One value, Representing the memory matrix The Line, i.e., the first Individual memory patterns. Set matching degree. For memory matrix The features containing historical normal pattern memories obtained by performing weighted summation are denoted as follows: .

[0022]

[0023] Finally, the extracted features are... and features that contain normal historical patterns of memory The memory fusion module outputs the result by performing a Concat operation and a Reshape operation.

[0024] The decoder consists of three Conv2DTranspose+ReLU layers, one Conv2Dtranspose+Sigmoid layer, and one Cropping layer, which can obtain the input. Reconstruction .

[0025] The fourth step is to train the anomaly detection model. This involves training the model using the training set, including forward propagation, loss calculation, backpropagation, and weight updates.

[0026] (1) The forward propagation process is as follows: As input to the encoder, the data is downsampled by the encoder module to extract latent feature tensors. The feature tensors output by the encoder are then fed into the self-supervised module and the memory fusion module, respectively. Inputting the feature tensors extracted by the encoder into the self-supervised module outputs a binary classification result. Inputting the feature tensors extracted by the encoder into the memory fusion module fuses the currently extracted features with the memory features of historical normal patterns, resulting in a fused feature of current and historical information. The fused feature extracted by the memory fusion module is then input into the decoder to obtain the final input. Reconstruction .

[0027] (2) The process of calculating the loss is as follows: the total error of the model Due to reconstruction error and self-supervised error It consists of two parts:

[0028] This represents all trainable parameters of the entire model. The weight parameters represent the loss function.

[0029] By comparing the prediction results output by the self-supervised module with the latent labels and calculating the cross-entropy loss, the self-supervised error can be obtained. .

[0030]

[0031] in, The number of self-supervised learning classes (including normal and abnormal data). =2). and These are the hidden label and the predicted probability, respectively.

[0032] Reconstruction error Calculated using mean squared error (MSE):

[0033] If the reconstruction error is less than a threshold, it is considered normal data; otherwise, it is judged as abnormal data.

[0034] (3) The backpropagation and weight update process is as follows: The gradient of the loss function with respect to the weights of each layer is calculated according to the chain rule, and the gradient is propagated backward layer by layer from the output layer. The Adam optimizer is used, and the learning rate is set to... The network weight parameters are adjusted based on the gradient to reduce loss.

[0035] Step 5: Validate the data anomaly detection model The model is evaluated using validation set data, including forward propagation and the calculation of validation metrics. Different strategies are selected based on the evaluation results. If the evaluation results are unsatisfactory, hyperparameters such as batch size, epochs, dropout rate, and learning rate are adjusted, and the model is retrained in step four. If the evaluation results are good, step six is ​​performed. The forward propagation process is consistent with that in step four.

[0036] The process of calculating validation metrics is as follows: The main metrics used to evaluate the performance of the algorithm are accuracy, precision, recall, and F1 score. These metrics are introduced below.

[0037] Accuracy represents the proportion of correctly classified samples out of the total number of samples. The formula is as follows:

[0038] Precision, also known as accuracy, represents the proportion of samples that are predicted to be positive but are actually positive. The formula for calculating precision is as follows:

[0039] Recall, also known as the full detection rate, represents the proportion of actual positive samples out of the total number of positive samples predicted to be positive. The formula is as follows:

[0040] The F1 score is a weighted average of precision and recall. The formula for calculating the F1 score is as follows:

[0041] Precision reflects the model's ability to distinguish negative samples; the higher the precision, the stronger the model's ability to distinguish negative samples. Recall reflects the model's ability to identify positive samples; the higher the recall, the stronger the model's ability to identify positive samples. The F1 score is a combination of both; the higher the F1 score, the more robust the model.

[0042] Step 6: Test the anomaly detection model. Use the test set data to test the model, including forward propagation and calculating validation metrics, to obtain the performance metrics of the anomaly detection model. The forward propagation process is consistent with that in Step 4. The process of calculating validation metrics is consistent with that in Step 5.

[0043] Step 7: Apply the data anomaly detection model. Deploy the trained model to the actual production environment, process real data, and continuously monitor model performance.

[0044] Step 8, End.

Claims

1. A method for detecting data anomalies in information systems based on the fusion of unsupervised learning and memory, characterized in that, The method includes the following steps: The first step is information system data collection, which involves selecting data with significant monitoring value from the sensor data, command data, communication data, status data, process data, and log data of the information system for collection. The second step is to preprocess the collected data. First, the data is encoded and de-identified. Then, it is normalized. Finally, different transformations are used to enhance the data to obtain the final dataset. The third step is to construct the network structure of the data anomaly detection model, which consists of four parts: encoder, self-supervised module, memory fusion module, and decoder. The fourth step is to train the anomaly detection model using the training set, including forward propagation, loss calculation, backpropagation, and weight update. The fifth step involves evaluating the anomaly detection model using the validation set data. This includes forward propagation and calculating validation metrics. Based on the evaluation results, different strategies are selected. If the evaluation results are unsatisfactory, the batch size, epochs, dropout rate, and learning rate are adjusted, and the model is retrained in the fourth step. If the evaluation results are good, the process proceeds to the sixth step. The sixth step is to test the data anomaly detection model using the test set data, including the forward propagation and calculation of validation metrics. The seventh step is to deploy the trained model to the actual production environment, process real data, and continuously monitor the model performance. Step 8, End.

2. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, In the first step, the sampling period is T, each sample contains L sampling points, and the size of each sample is L. N, Normal data collection is carried out when the information system is in normal working condition, including different working modes and scenarios; Abnormal data collection is carried out when the software is in an abnormal state, including sensor abnormalities, information processing abnormalities, configuration file abnormalities, communication abnormalities, and human error.

3. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, The data preprocessing process described in the second step includes: Normalization is performed using the min-max normalization method, where The data before normalization. For the normalized data: Data augmentation can be performed using the following data transformation methods: (1) Noise transformation: Gaussian noise is added to the data; (2) Smoothing transformation: The data is smoothed using a low-pass filter SavitzkyGolay (SG); (3) Reverse transformation: The sample is transformed in reverse along the time dimension to obtain a sample with the opposite time direction; (4) Negative transformation: The sample is transformed in the negative direction along the dimension of the value to obtain a mirror image of the input data.

4. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 3, characterized in that: The data augmentation samples are divided into training set, validation set and test set. The training set contains only normal samples and is used for training the unsupervised learning model. The validation set and test set contain normal samples and abnormal samples and are used for model validation and testing.

5. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, In the third step: the network structure of the data anomaly detection model includes four parts: encoder, self-supervised module, memory fusion module and decoder. The encoder consists of a padding layer and two Conv2D+ReLU+Maxpooling layers, which can extract latent feature tensors; The self-supervised module includes a Conv2D+Sigmoid layer, a Flatten layer, two fully connected layers, and a Softmax layer, which can output binary classification results. The self-supervised module establishes a binary classification auxiliary task and uses the auxiliary task to mine its own supervision information from large-scale unsupervised data. Since the training set only contains normal data, the data itself implicitly contains binary classification labels. By using the implicit supervision information to train the binary classification network and encoder, it is beneficial for the encoder to learn representations that are valuable for downstream tasks. The input to the memory fusion module is the feature extracted by the encoder. This feature is then reshaped to obtain a new feature, denoted as... A memory matrix is ​​used to record multiple patterns of normal features. This memory matrix is ​​denoted as... During the training phase, the memory matrix is ​​updated through a reconstruction loss function, forcing it to record normal features. Reconstructing anomalies using normal patterns retrieved from the memory matrix will lead to higher reconstruction errors. With memory matrix The previous matching degree is denoted as degree of matching The calculation method is as follows: in, , Indicates the degree of matching The One value, Representing the memory matrix The Line, i.e., the first Individual memory patterns, set matching degree For memory matrix The features containing historical normal pattern memories obtained by performing weighted summation are denoted as follows: ; Finally, the extracted features are... and features that contain normal historical patterns of memory The memory fusion module output is obtained by performing a Concat operation and a Reshape operation. The decoder consists of three Conv2DTranspose+ReLU layers, one Conv2Dtranspose+Sigmoid layer, and one Cropping layer, and can obtain the input... Reconstruction .

6. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, In the fourth step, the forward propagation process includes: Will As input to the encoder, the data is downsampled by the encoder module to extract latent feature tensors. The output feature tensors are then fed into the self-supervised module and the memory fusion module. The self-supervised module outputs a binary classification result, while the memory fusion module fuses the currently extracted features with historical normal pattern memories to obtain a fused feature. This fused feature is then input into the decoder to obtain the final input. Reconstruction .

7. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, In the fourth step, the process of calculating the loss includes: Overall error of the model Due to reconstruction error and self-supervised error It consists of two parts: This represents all trainable parameters of the entire model. The weight parameters represent the loss function; The prediction results output by the self-supervised module are compared with the hidden labels, and the cross-entropy loss is calculated to obtain the self-supervised error. : in, The number of self-supervised learning classes. and These are the hidden labels and the predicted probabilities, respectively. Reconstruction error Calculated using mean square error (MSE): If the reconstruction error is less than a threshold, it is considered normal data; otherwise, it is judged as abnormal data.

8. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 7, characterized in that, The number of self-supervised learning classes includes both normal and abnormal data. =2.

9. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, In the fourth step, the backpropagation and weight update process is as follows: The gradient of the loss function with respect to the weights of each layer is calculated according to the chain rule. The gradient is propagated backward layer by layer from the output layer. The Adam optimizer is used, and the learning rate is set to... The network weight parameters are adjusted based on the gradient to reduce the loss.

10. The information system data anomaly detection method based on unsupervised learning and memory fusion according to claim 1, characterized in that, The process of calculating the verification index includes: The performance of the algorithm is evaluated using several metrics, including accuracy, precision, recall, and F1 score.

Images