Multi-network Anomaly Recognition Method Based on Denoising Autoencoder

Abstract

The present invention is a method for multi-network anomaly recognition based on denoising autoencoders, belonging to the fields of big data and information security. The method comprises the following steps: S1: Sampling network traffic data to obtain an input data matrix; S2: Dividing the input data matrix into blocks; S3: Using the Dropout layer of the encoder in the denoising autoencoder network to perform denoising processing on each sub-matrix according to a probability, and then encoding to obtain an overall sample; S4: Randomly extracting sub-matrices, and encoding them using the encoder in the encoding network to obtain local samples; S5: Using historical data for training and testing; S6: Using the trained network to generate overall samples and local samples as inputs respectively, and using a classifier to determine whether the data is abnormal data. The method of the present invention gives specific denoising probabilities and random sampling probabilities to each block in a statistical sense, which is beneficial to improving the accuracy of intrusion detection.

Description

Technical Field

[0001] The present invention relates to a method for identifying multiple network anomalies based on denoising autoencoders, belonging to the fields of big data and information security, and is particularly applicable to the identification of multiple network anomalies based on denoising autoencoders. Background Art

[0002] With the popularization of the Internet and the increasing number of network-connected devices, the security of the Internet faces great threats. On the one hand, in the era of big data, the Internet records a lot of privacy information such as the food, clothing, housing, and transportation of many users and the core data of enterprises; on the other hand, most users have a very weak awareness of data privacy protection, and the security threats and attacks on the Internet are also increasing continuously. According to the statistics of the Internet Security Threat Report, "In June 2021, the number of terminals infected with Trojan or botnet malicious programs was more than 1.9 million; the number of domestic websites tampered with was 8,747, etc." It can be seen that the current network security situation is becoming increasingly severe.

[0003] In order to dynamically detect network intrusions, the application of intrusion detection systems plays a significant role in network security; as an active security defense technology, it is the second line of defense for maintaining network security. Intrusion detection systems are divided into misuse detection systems and anomaly detection systems. The misuse detection system analyzes network connections, monitors the characteristics of network traffic, and discovers network abnormal behaviors by comparing with a known attack feature library. When the behaviors of monitored users or systems match the characteristics of abnormal behaviors in the database, the detection system determines their behaviors as attacks. The anomaly detection system is based on the principle of statistical analysis. First, it analyzes the data of normal behaviors, defines the baseline or normal state of the network, and then monitors network behaviors, comparing its abnormal state with the normal state to find abnormal behaviors. Currently, complex network traffic affects the accuracy of intrusion detection; moreover, network intrusion attack traffic is large and forms are diverse, requiring real-time responses to such attacks. The accuracy of intrusion detection is a measure of the intrusion detection ability, and reducing the false negative rate and false positive rate is the key to improving the intrusion detection ability and also a necessary condition for effectively preventing intrusions. Document [1] presents a method of "identifying and classifying network abnormal behaviors based on small samples of network intrusion data". This method can ensure a certain degree of accuracy based on the denoising autoencoder network while significantly reducing the amount of data, but blind denoising will lead to the loss of key features. At the same time, due to the lack of reference to the original information, it will lead to deviations in the convergence positions of identification and classification.

[0004] [1] Luo Shuxiang. Research on the Identification and Classification of Network Abnormal Behaviors Based on Small Samples of Network Intrusion [D]. Shanghai Normal University, 2022. DOI: 10.27312 / d.cnki.gshsu.2022.001976. Summary of the Invention

[0005] In view of this, the present invention provides a multi-network anomaly recognition method based on denoising autoencoders, aiming to perform denoising in a targeted manner according to statistical laws and adding part of the original information as a reference, which is beneficial to improving the accuracy of intrusion detection.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The multi-network anomaly recognition method based on denoising autoencoders includes the following steps:

[0008] S1: Sampling the network traffic data to obtain an input data matrix;

[0009] S2: Dividing the input data matrix into N sub-matrices;

[0010] S3: Using the Dropout layer of the encoder in the denoising autoencoder network to perform denoising processing on each sub-matrix according to a probability, and then using a fully connected layer to connect and encode to obtain an overall sample;

[0011] S4: Randomly extracting sub-matrices from the N sub-matrices according to a probability, and using the encoder in the encoding network to perform encoding processing on them to obtain local samples;

[0012] S5: Using the historical data to divide into a training set and a validation set, using the training set to train the denoising autoencoder network and the encoding network, and using the validation set to test the denoising autoencoder network and the encoding network;

[0013] S6: Using the trained denoising autoencoder network and encoding network to generate an overall sample and a local sample as inputs respectively, and using a classifier to determine whether the data is abnormal data;

[0014] The denoising autoencoder network consists of an encoder and a decoder; wherein the encoder in the denoising autoencoder network is a convolutional neural network containing a Dropout layer, and the decoder in the denoising autoencoder network is a convolutional neural network; the encoding network consists of an encoder and a decoder, and the encoder and decoder in the encoding network are convolutional neural networks.

[0015] Further, the number N of sub-matrices in step S2 is obtained by an expert system according to historical records, or can be obtained by training a deep learning network.

[0016] Preferably, dividing the input data matrix into N sub-matrices can be done by dividing according to a fixed sliding window size, and there can be partial data overlap between adjacent sub-matrices.

[0017] Further, the depth of the convolutional neural network and the size of the convolutional kernel in the denoising autoencoder network and the encoding network can be determined according to their respective specific functions and usage environments.

[0018] Further, the noise reduction probability of each sub-matrix described in step S3 is different; the noise reduction probability p corresponding to the i-th sub-matrix i = ω i ·p; where p is the average noise reduction probability, and ω i is the weight, both of which are obtained by training using a deep learning network.

[0019] Further, step S4 is specifically as follows: (1) Randomly select sub-matrices from the N sub-matrices as local samples according to the noise reduction probability corresponding to each sub-matrix, where is rounded up; (2) Establish an encoding network and train it using historical data; (3) Use the encoder in the trained encoding network to encode and process the input data to obtain local samples.

[0020] Further, testing and training the denoising autoencoder network and the encoding network described in step S5 requires the use of the decoders of the denoising autoencoder network and the encoding network, so as to perform gradient descent training through the loss function; after the training is completed, the decoders of the denoising autoencoder network and the encoding network will no longer play a role in the generation of the overall sample and the local sample.

[0021] Preferably, the accuracy can be appropriately sacrificed to accelerate the classification speed. Step S6 is specifically as follows: (1) Connect the local sample and the overall sample using a fully connected layer as the input of the anomaly classifier; (2) Use a binary classifier as the anomaly classifier; (3) Train the anomaly classifier using the historical data with labels; (4) Use the trained anomaly classifier to classify and identify new data.

[0022] Preferably, to ensure accuracy, step S6 can specifically also be: (1) Use the local sample and the overall sample as the inputs of the anomaly classifier respectively; (2) Connect two parallel binary classifiers in series with a binary classifier as the anomaly classifier; (3) Train the anomaly classifier using the historical data with labels; (4) Use the trained anomaly classifier to classify and identify new data.

[0023] Preferably, before using the local sample and the overall sample described in step S6 as inputs, an attention mechanism network (LTSM) can be used to assign weights to them respectively, so that the features are more distinct.

[0024] The beneficial effects of the present invention are as follows: The present invention provides a multi-network anomaly recognition method based on denoising autoencoders. After dividing the data into blocks, local samples and global samples are obtained by using random sampling and denoising autoencoders respectively, and then a classifier is used to accurately classify them. Specific denoising probabilities and random sampling probabilities can be given to each block in a statistical sense, which is beneficial to improving the accuracy of intrusion detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to illustrate the objectives and technical solutions of the present invention, the following drawings are provided for description:

[0026] Figure 1 It is a flowchart of the present invention;

[0027] Figure 2 It is an architecture diagram of the denoising autoencoder network of the present invention;

[0028] Figure 3 It is a schematic diagram in Embodiment 1 of the present invention;

[0029] Figure 4 It is a schematic diagram of Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] To make the objectives and technical solutions of the present invention clearer and more understandable, the present invention will be described in detail below with reference to the drawings and embodiments.

[0031] Embodiment 1: For the NSL-KDD dataset, the attributes of each piece of data in the dataset samples are represented by numerical and character-valued feature values, and it contains 43 features: 41 of which refer to the traffic input itself, and the last two are labels (normal or attack) and scores (the severity of the traffic input itself); among the 41 features, 3 are character-valued features, and the character-valued features need to be numericalized and normalized. 20% of the data in the NSL-KDD dataset is used as the validation set, and the remaining normal traffic is used as the training set. The validation set is input into the trained model to obtain the distribution of reconstruction errors, so as to determine the optimal threshold, and the performance evaluation of the model is completed for the validation set. In order to accurately implement intrusion detection, the present invention provides a "multi-network anomaly recognition method based on denoising autoencoders"; it includes the following steps:

[0032] S1: Sort each piece of data in the NSL-KDD dataset to obtain the corresponding input data matrix.

[0033] S2: Set the sliding window size and step length, and divide the input data matrix into N sub-matrices.

[0034] S3: Use the Dropout layer of the encoder in the denoising autoencoder network to perform denoising processing on each sub-matrix according to a probability respectively, and then use the fully connected layer to connect and encode to obtain the global sample.

[0035] The noise reduction probability of each of the described sub - matrices is different; the noise reduction probability p corresponding to the i - th sub - matrix i = ω i ·p; where p is the average noise reduction probability, and ω i is the weight, both of which are obtained by training using a deep learning network.

[0036] S4: Randomly select sub - matrices from the N sub - matrices according to the probability, and use the encoder in the encoding network to perform encoding processing on them to obtain local samples.

[0037] Specifically: (1) Randomly select sub - matrices from the N sub - matrices as local samples according to the noise reduction probability corresponding to each sub - matrix, where is the ceiling function; (2) Establish an encoding network and train it using historical data; (3) Use the encoder in the trained encoding network to perform encoding processing on the input data to obtain local samples.

[0038] S5: Divide the historical data into a training set and a validation set, use the training set to train the denoising auto - encoding network and the encoding network, and use the validation set to test the denoising auto - encoding network and the encoding network.

[0039] The testing and training of the denoising auto - encoding network and the encoding network need to use the decoders of the denoising auto - encoding network and the encoding network in order to perform gradient descent training through the loss function; when the training is completed, the decoders of the denoising auto - encoding network and the encoding network will no longer play a role in the generation of the overall sample and the local sample.

[0040] S6: Use the trained denoising auto - encoding network and encoding network to generate the overall sample and the local sample as inputs respectively, and use the classifier to determine whether the data is abnormal data.

[0041] Specifically: (1) Connect the local sample and the overall sample into a whole using a fully - connected layer as the input of the anomaly classifier; (2) Use a binary classifier as the anomaly classifier; (3) Train the anomaly classifier using the historical data with labels; (4) Use the trained anomaly classifier to classify and identify new data.

[0042] The method of the present invention can independently assign the noise reduction probability, can effectively eliminate noise interference, and at the same time, combined with the original information, can make the analysis more accurate.

[0043] Embodiment 2: For the case of Embodiment 1, in order to better reflect the data characteristics and improve the recognition accuracy, on the basis of Embodiment 1, the present invention provides an improved "multiple network anomaly recognition method based on denoising autoencoding", wherein the same parts of specific steps S1 to S6 as those in Embodiment 1 will not be elaborated here.

[0044] The difference is that step S6 can specifically be: (1) using the local samples and the overall samples as the inputs of the anomaly classifier respectively; (2) connecting two parallel binary classifiers in series with a binary classifier as the anomaly classifier; (3) training the anomaly classifier using the historical data with labels; (4) using the trained anomaly classifier to classify and recognize new data.

[0045] Before the local samples and the overall samples described in step S6 are used as inputs, the attention mechanism network (LTSM) can be used to assign weights to them respectively so that the features are more distinct.

[0046] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A method for multi-network anomaly recognition based on denoising auto-encoder, characterized in that The method comprises the following steps: S1: Sampling the network traffic data to obtain an input data matrix; S2: Dividing the input data matrix into N sub-matrices; S3: Using the Dropout layer in the encoder of the denoising autoencoder network to perform denoising processing on each sub-matrix according to a probability respectively, and then connecting them with a fully connected layer for encoding to obtain an overall sample; S4: Randomly extracting sub-matrices from the N sub-matrices according to a probability, and using the encoder in the encoding network to perform encoding processing on them to obtain local samples; S5: Using the historical data to be divided into a training set and a validation set, using the training set to train the denoising autoencoder network and the encoding network, and using the validation set to test the denoising autoencoder network and the encoding network; S6: Using the trained denoising autoencoder network and encoding network to generate an overall sample and a local sample as inputs respectively, and using a classifier to determine whether the data is abnormal data; The described denoising autoencoder network is composed of an encoder and a decoder; wherein the encoder in the denoising autoencoder network is a convolutional neural network containing a Dropout layer, and the decoder in the denoising autoencoder network is a convolutional neural network; the described encoding network is composed of an encoder and a decoder, and the encoder and decoder in the encoding network are convolutional neural networks.

2. The method for multi-network anomaly recognition based on denoising autoencoder according to claim 1, wherein The number N of sub-matrices described in step S2 is obtained by an expert system according to historical records, or can also be obtained by training with a deep learning network.

3. The method for multi-network anomaly recognition based on denoising autoencoder according to claim 1, wherein, The convolutional neural networks in the described denoising autoencoder network and encoding network can determine the network depth and the size of the convolutional kernel according to their respective specific functions and usage environments.

4. The method for multi-network anomaly recognition based on denoising auto-encoding according to claim 1, wherein The noise reduction probability of each sub-matrix described in step S3 is different; the noise reduction probability p corresponding to the i-th sub-matrix i = ω i ·p; where p is the average noise reduction probability, and ω i is the weight, both of which are obtained by training using a deep learning network.

5. The method for multi-network anomaly recognition based on denoising auto-encoder according to claim 1, wherein, Step S4 is specifically as follows: (1) Randomly select sub-matrices from the N sub-matrices as local samples according to the noise reduction probability corresponding to each sub-matrix, where is rounded up; (2) Establish an encoding network and train it using historical data; (3) After encoding and processing the input data using the encoder in the trained encoding network, obtain local samples.

6. The method for multi-network anomaly recognition based on denoising auto-encoding according to claim 1, characterized in that The specific step S6 is: (1) Connecting the local sample and the overall sample into a whole using a fully connected layer as the input of the anomaly classifier; (2) Using a binary classifier as the anomaly classifier; (3) Training the anomaly classifier using the historical data with labels; (4) Using the trained anomaly classifier to classify and identify new data.

7. The method for identifying multiple network anomalies based on denoising autoencoding according to claim 1, wherein The specific step S6 can also be: (1) Using the local sample and the overall sample as the inputs of the anomaly classifier respectively; (2) Using two parallel binary classifiers in series with a binary classifier as the anomaly classifier; (3) Training the anomaly classifier using the historical data with labels; (4) Using the trained anomaly classifier to classify and identify new data.

Images