A semi-supervised network traffic anomaly detection method and device based on a stacked autoencoder

Abstract

The application discloses a semi-supervised network flow anomaly detection method and device based on a stacked autoencoder, and the method comprises the following steps: selecting a training data set and randomly sampling two-by-two to form a sample pair set; training a two-stage stacked autoencoder by using the training data set to obtain a feature extractor; extracting and fusing the features of the sample pairs in the sample pair set to train an anomaly score generator; randomly extracting samples from the training data set and the data to be detected to form a sample pair to obtain an anomaly score; and determining whether the data is abnormal according to an anomaly reference value and the anomaly score. The application can effectively reduce the influence of insufficient labeled samples on the model performance and generalization ability, and improve the precision of the model in network flow data anomaly detection by randomly combining labeled samples and unlabeled samples, fully utilizing the existing labeled information to expand the number of labeled samples for model training in the case of insufficient labeled data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network traffic anomaly detection, and particularly relates to a semi-supervised network traffic anomaly detection method and device based on a stacked autoencoder. BACKGROUND

[0002] In the current digital era, the network has become an indispensable part of people's life and work. With the widespread popularity of the Internet and the continuous expansion of its applications, network traffic is showing a rapid growth trend. However, the rapid increase in network traffic has also caused a series of security problems, among which network traffic anomaly detection has become one of the key research directions in the field of network security.

[0003] Network traffic anomaly refers to the situation where the traffic behavior in the network deviates significantly from the normal mode. These anomalies may be caused by various factors, such as network attacks, malicious software attacks, device failures, network configuration errors, etc. If these anomalies cannot be detected and handled in a timely manner, it may cause network performance degradation, service interruption, data leakage, and other serious consequences, causing huge losses to individuals and enterprises.

[0004] Traditional network security technologies mainly focus on firewalls, intrusion detection systems, etc. However, these technologies can usually only detect known attack patterns, and are not effective against new and unknown attack methods. Network traffic anomaly detection technology is a data analysis-based approach that monitors and analyzes network traffic in real time to explore potential abnormal behavior.

[0005] Deep learning is a machine learning technology that relies on neural network structures, and its main purpose is to explore the internal laws of sample data. Compared with traditional machine learning models, deep learning has higher efficiency in model construction and feature extraction, and can achieve higher accuracy in processing large-scale data and complex structure data. According to the difference in training methods, traffic anomaly detection methods can be divided into supervised learning, unsupervised learning, and semi-supervised learning. Among them, supervised learning requires all training data to have labels, while unsupervised learning does not rely on label information at all. Semi-supervised learning is between the two, using a large amount of unlabeled data and a small amount of labeled data to train together. In actual application scenarios, abnormal conditions are usually rare, and it is difficult and costly to obtain sufficient labeled abnormal data, which may cause data imbalance problems and affect the training effectiveness of the model. SUMMARY

[0006] The application aims to provide a semi-supervised network traffic anomaly detection method and device based on a stacked autoencoder, which can effectively reduce the influence of insufficient labeled anomaly labels on model performance and generalization ability, and improve the accuracy of network traffic data anomaly detection by randomly combining normal samples, abnormal samples and unlabeled samples to obtain sample pairs, using a stacked autoencoder to extract features from the data, and fully combining existing label information to determine the degree of data anomaly.

[0007] To achieve the above purpose, the application adopts the following technical solutions:

[0008] In a first aspect, the application provides a semi-supervised network traffic anomaly detection method based on a stacked autoencoder, comprising:

[0009] Obtaining a network traffic dataset for preprocessing and dividing it into a training dataset and a validation dataset, randomly sampling the training dataset two by two to form sample pairs, obtaining a sample pair set, and setting different sample pair abnormal reference values; the training dataset includes an unlabeled data set, a labeled abnormal data set and a labeled normal data set;

[0010] Training a two-level stacked autoencoder using the training dataset to obtain a feature extractor;

[0011] Using the trained feature extractor to extract features of the sample pairs in the sample pair set and fuse them, and using the result as input to train a traffic anomaly detection model to obtain an anomaly scorer;

[0012] Randomly selecting one sample from the unlabeled data set, labeled abnormal data set and labeled normal data set, and combining it with the network traffic data to be detected to form a sample pair, which is input into the anomaly scorer after feature extraction and fusion to obtain an anomaly score;

[0013] According to the abnormal reference value and the calculated anomaly score, it is determined whether the network traffic data to be detected is abnormal.

[0014] Preferably, the network traffic dataset is obtained and divided into a training dataset and a validation dataset, and the training dataset is randomly sampled two by two to form sample pairs to obtain a sample pair set, comprising:

[0015] Obtaining a network traffic dataset , , satisfying , wherein the labels of the first data are unknown, denoted as an unlabeled data set , the middle is a labeled abnormal data, denoted as a labeled abnormal set , and the last The data items are labeled as normal data, and are denoted as the labeled normal set. ;

[0016] From the set , , 10% of the data was randomly selected from each dataset as validation data, and the remaining data was used as the training dataset.

[0017] Two samples are randomly selected from the training dataset. and Composition of sample pairs Repeat this process to obtain a set of sample pairs. , This represents the number of sample pairs.

[0018] Preferably, setting abnormal reference values ​​for different sample pairs includes:

[0019] Based on the unlabeled data set , Set of marked anomalous data and labeled normal data sets Obtained through random sampling combination and Six combinations, with abnormal reference values ​​set as follows: , , , , and ,in, .

[0020] Preferably, the step of training a two-level stacked autoencoder using the training dataset to obtain a feature extractor includes:

[0021] A two-level stacked autoencoder is constructed using sparse autoencoders and ordinary autoencoders. The constructed two-level stacked autoencoder is trained using a training dataset, as follows:

[0022] Randomly select one data point from the training dataset. ,Will The input is fed into a sparse autoencoder to obtain... Reconstructed data and characteristics ;

[0023] Through the loss function calculate and To minimize the reconstruction error, optimize the parameters of the sparse autoencoder model with the goal of minimizing the training loss, and stop training after the loss converges.

[0024] Features Input into a regular autoencoder to obtain Reconstructed data and characteristics ;

[0025] Through the loss function calculate The reconstruction error is minimized by optimizing the parameters of the ordinary autoencoder model to reduce the training loss. Training is stopped after the loss converges, resulting in a well-trained two-stage stacked autoencoder, which is used as a feature extractor.

[0026] Preferably, the loss function is expressed as follows:

[0027] ,

[0028] in, It is a sparse metric function. For sparse autoencoder model parameters, For sparsity penalty parameters, For regularization parameters;

[0029] ,

[0030] in, These are parameters for a typical autoencoder model. This is the regularization parameter.

[0031] Preferably, the step of extracting and fusing features from sample pairs in the sample pair set using the trained feature extractor, and then using these features as input to train the traffic anomaly detection model to obtain an anomaly scorer, includes:

[0032] The sample pairs were processed using two identical feature extractors respectively. Feature extraction was performed on two samples, and the following methods were used: Feature fusion is performed to obtain sample pairs. Features ;

[0033] A multilayer perceptron is used as the traffic anomaly detection model to fuse features. As input, and randomly put If one dimension is set to 0, the outlier score of the output sample pair is denoted as . ;

[0034] Using loss function The traffic anomaly detection model is trained to optimize the training model parameters with the goal of minimizing the training loss. Training is stopped after the loss converges, and the anomaly scorer is obtained.

[0035] Preferably, the step of determining whether the network traffic data to be detected is abnormal based on the abnormal reference value and the calculated abnormal score includes:

[0036] From the set of labeled anomalies Unlabeled data sets and labeled normal data sets A sample is randomly selected from each of the samples, denoted as . , and , respectively with the network traffic data to be detected Composition data pairs , and After feature extraction and fusion, the fused features are obtained. , , The network traffic data to be detected is obtained by preprocessing the traffic packet data using a packet capture tool.

[0037] Fusion features , , The data are input into the anomaly scorer, which outputs anomaly scores. , , ;

[0038] outlier scores and , and Abnormal reference value , and Compare, if If so, the reference judgment for the corresponding input data pair is normal; if Then the reference judgment for the corresponding input data pair is abnormal;

[0039] outlier scores and , and Abnormal reference value , and Compare, if If so, the reference judgment for the corresponding input data pair is normal; if If so, the reference judgment for the corresponding input data pair is abnormal;

[0040] outlier scores and , and Abnormal reference value , and is compared, if , the reference for the corresponding input data pair is determined to be normal; if , the reference for the corresponding input data pair is determined to be abnormal;

[0041] If two or more references are determined to be normal, the network traffic data to be detected is determined to be normal, otherwise, the network traffic data to be detected is determined to be abnormal.

[0042] In a second aspect, the present application provides a semi-supervised network traffic anomaly detection device based on a stacked autoencoder, which is used to implement the semi-supervised network traffic anomaly detection method based on a stacked autoencoder described above, and the device comprises:

[0043] A data preprocessing module is configured to obtain a network traffic dataset for preprocessing and dividing into a training dataset and a validation dataset, randomly sampling two by two to form sample pairs from the training dataset to obtain a sample pair set, and setting abnormal reference values for different sample pairs; the training dataset comprises an unlabeled data set, a labeled abnormal data set, and a labeled normal data set;

[0044] A feature extractor training module is configured to train a two-level stacked autoencoder using the training dataset to obtain a feature extractor;

[0045] An anomaly scorer training module is configured to extract features of sample pairs in the sample pair set using the trained feature extractor and fuse the features, and use the fused features as input to train a traffic anomaly detection model to obtain an anomaly scorer;

[0046] A scoring module is configured to randomly select one sample from the unlabeled data set, the labeled abnormal data set, and the labeled normal data set, respectively, and form a sample pair with the network traffic data to be detected, and input the sample pair into the anomaly scorer after feature extraction and fusion to obtain an abnormal score;

[0047] A result output module is configured to determine whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated abnormal score.

[0048] In a third aspect, the present application provides a computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the semi-supervised network traffic anomaly detection methods based on a stacked autoencoder described above.

[0049] In a fourth aspect, the present application provides a computing device comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any one of the methods of the semi-supervised network traffic anomaly detection based on stacked autoencoder according to the above.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] 1. The semi-supervised network traffic anomaly detection method based on stacked autoencoder provided by the present application can effectively reduce the influence of insufficient labeled samples on the model performance and generalization ability, and improve the precision of the model in network traffic data anomaly detection, by randomly combining normal samples, abnormal samples and unlabeled samples to obtain sample pairs, fully utilizing the existing label information to expand the number of labeled samples for model training.

[0052] 2. The semi-supervised network traffic anomaly detection method based on stacked autoencoder provided by the present application can effectively mine more rich representation forms and abnormal patterns of data, reduce the influence of high-dimensional data on detection effect, and further improve the precision of detection results, by extracting features of network traffic data through stacked autoencoder.

[0053] 3. The semi-supervised network traffic anomaly detection method based on stacked autoencoder provided by the present application has strong scalability, as the abnormal score calculator and other modules can be replaced by other models. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of the semi-supervised network traffic anomaly detection method based on stacked autoencoder provided by the present application is shown in the figure.

[0055] Figure 2 A flowchart of the network traffic data anomaly detection in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but not as a limitation of the present application.

[0057] It should be noted that, in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0058] It should be emphasized that the terms "comprises / comprising" when used in this specification are taken to specify the presence of stated features, integers, steps or components but do not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0059] Embodiments of the present application will be described hereinafter with reference to the accompanying drawings. In the drawings, like reference numerals designate identical or similar parts, or identical or similar steps.

[0060] It should be emphasized here that the step labels mentioned hereinafter are not a limitation on the order of the steps, but it should be understood that the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0061] The first embodiment of the present application provides a semi-supervised network traffic anomaly detection method based on stacked autoencoder, referring to Figure 1 , comprising:

[0062] 1) Select a traffic dataset, the network traffic data in the traffic dataset should include IP, port, protocol, payload size, duration, service type, etc. information, preprocess the dataset and divide it into a training dataset and a validation set, wherein the training dataset includes an unlabeled data set, a labeled abnormal data set and a labeled normal data set;

[0063] 2) Randomly sample the training dataset two by two to form a sample pair set, and set different abnormal reference values for different sample pairs;

[0064] 3) Train a two-level stacked autoencoder composed of a sparse autoencoder and a general autoencoder using the training dataset to obtain a feature extractor;

[0065] 4) Use the trained feature extractor to extract the features of the sample pairs in the sample pair set and fuse them, and use the result as input to train a traffic anomaly detection model to obtain an anomaly scorer;

[0066] 5) After preprocessing the traffic packet data obtained from packet capture tools such as Wireshark, the network traffic data to be detected is obtained, a sample is randomly selected from the unlabeled data set, the labeled abnormal data set and the labeled normal data set, and a sample pair is formed with the data to be detected. After feature extraction and fusion, input the anomaly scorer to obtain an abnormal score;

[0067] 6) Determine whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated abnormal score.

[0068] In the embodiments of the present application, the traffic dataset is randomly sampled two by two to form a sample pair set, which is as follows:

[0069] Given traffic data set , meet . Where the first The label of the data is unknown, denoted as an unlabeled data set , the middle It is labeled abnormal data, denoted as labeled abnormal set , and the last The data is labeled normal data, denoted as labeled normal set .

[0070] Randomly select 10% of the data from the set , , As test data , the remaining data as the training set , that is . Randomly take two samples and From the training data set Form a sample pair , repeat this process to obtain a sample pair set , The number of sample pairs is

[0071] In the embodiment of the application, the abnormal reference value of different sample pairs is set according to the known label information combined with the sample category combination, including:

[0072] According to the unlabeled data set , the labeled abnormal data set And the labeled normal data set , through random sampling combination, six kinds of combinations can be obtained and , Indicates that the sample pair composed of two data in set , Indicates that the sample pair composed of data in set and set , Indicates that the sample pair composed of data in set and set , Indicates that the sample pair composed of two data in set , Indicates that the sample pair composed of data in set and set , and the abnormal reference values are respectively set as , , , , and wherein, .

[0073] In the embodiment of the present application, the two-stage stacked autoencoder composed of a sparse autoencoder and a general autoencoder uses a training data set The two-stage stacked autoencoder constructed is trained to obtain a feature extractor, and the specific process is as follows:

[0074] S21: from a data is randomly selected is input into the sparse autoencoder to obtain the reconstruction data and the feature ;

[0075] S22: the reconstruction error of and is calculated by a loss function ;

[0076] S23: S21 to S22 are repeated, and the model parameters are optimized with the goal of minimizing the training loss, and the training is stopped when the loss converges;

[0077] S24: the feature obtained in S21 is input into the autoencoder to obtain the reconstruction data and the feature ;

[0078] S25: the reconstruction error of is calculated by a loss function ;

[0079] S26: S24 to S25 are repeated, and the model parameters are optimized with the goal of minimizing the training loss, and the training is stopped when the loss converges, obtaining a trained two-stage stacked autoencoder;

[0080] S27: two identical two-stage stacked autoencoders are used to extract features from two samples in a sample pair respectively, and the features are fused using product to obtain the feature of the sample pair .

[0081] The above process is represented as follows:

[0082] ,

[0083] ,

[0084] ,​​

[0085] ,

[0086] wherein, represents the original input data, , respectively represent the features extracted using the sparse autoencoder and the ordinary autoencoder, , respectively represent the sparse autoencoder and the ordinary autoencoder model parameters, , respectively represent the sparse autoencoder and the ordinary autoencoder reconstructed data.

[0087] First, the sparse autoencoder is trained using the loss function After the training is completed, the features extracted by the sparse autoencoder are taken as the input, and the ordinary autoencoder is trained using the loss function .

[0088] The finally trained feature extractor is represented as:

[0089] ,

[0090] wherein, and are parameters obtained by training, is the input sample, is the finally extracted feature.

[0091] The loss function , is represented as follows:

[0092] ,

[0093] wherein, is a sparsity measurement function, is a training parameter, is a sparsity penalty parameter, is a regularization parameter;

[0094] ,

[0095] wherein, is a training parameter, is a regularization parameter.

[0096] In the embodiment of the application, the feature extracted by the trained feature extractor is used to train a traffic anomaly detection model to obtain an anomaly scorer;

[0097] ​S31: respectively using the above trained two-level stacked autoencoder on each sample in the sample pair set in the sample pair, feature extraction is performed, and the features are fused using to obtain the features of the sample pair ; the features are input into the constructed traffic anomaly detection model;

[0098] S32, the traffic anomaly detection model uses a multi-layer perceptron, takes the fused features as input, and randomly sets a certain dimension of to 0 to improve the model generalization, and outputs the anomaly score of the sample pair, denoted as ;

[0099] S33, the loss function is used to train the traffic anomaly detection model, is expressed as:

[0100] ,

[0101] wherein is the anomaly score obtained by the traffic anomaly detection model, is the abnormal reference value of the sample pair, is the training parameter, is the regularization parameter;

[0102] S34: repeating S31 to S33, and optimizing the training parameters with the goal of minimizing the training loss, stopping training when the loss converges, and obtaining the anomaly scorer.

[0103] In the embodiment of the application, one sample is randomly extracted from the unmarked data set, the labeled abnormal data set and the labeled normal data set, respectively, to form a sample pair with the to-be-detected data, after feature extraction and fusion, the anomaly scorer is input to obtain the anomaly score, and whether the to-be-detected data is abnormal is judged according to the abnormal reference value and the calculated anomaly score, see Figure 2 , the specific process is as follows:

[0104] S41: randomly selecting one sample from , and , denoted as , and , respectively, to form data pairs , and with the to-be-detected data , after feature extraction and fusion, the fused features , ,​​ ;

[0105] S42: input the fusion features , , into the anomaly scorers respectively, and output anomaly scores , , ;

[0106] S43: compare the anomaly scores with the anomaly reference values , and of , and , if , the reference is determined as normal for the corresponding input data pair; if , the reference is determined as abnormal for the corresponding input data pair;

[0107] S44: compare the anomaly scores with the anomaly reference values , and of , and , if , the reference is determined as normal for the corresponding input data pair; if , the reference is determined as abnormal for the corresponding input data pair;

[0108] S45: compare the anomaly scores with the anomaly reference values , and of , and , if , the reference is determined as normal for the corresponding input data pair; if , the reference is determined as abnormal for the corresponding input data pair;

[0109] S46: if two or more references are determined as normal, the data is determined as normal, otherwise as abnormal.

[0110] Based on the above inventive concept, the application further provides a semi-supervised network traffic anomaly detection device based on a stacked autoencoder, which comprises:

[0111] The data preprocessing module is configured to obtain a network traffic dataset for preprocessing and dividing into a training dataset and a verification dataset, randomly sampling two-by-two to form sample pairs from the training dataset, obtaining a sample pair set, and setting abnormal reference values for different sample pairs.

[0112] The feature extractor training module is configured to train a two-level stacked autoencoder composed of a sparse autoencoder and a normal autoencoder using the training dataset, and obtain a feature extractor.

[0113] The anomaly scorer training module is configured to extract features of the sample pairs in the sample pair set using the trained feature extractor and fuse the features, and use the fused features as input to train a traffic anomaly detection model to obtain an anomaly scorer.

[0114] The scoring module is configured to randomly select one sample from the unlabeled data set, the labeled abnormal data set and the labeled normal data set, respectively, to form a sample pair with the network traffic data to be detected, and input the sample pair into the feature extractor and the anomaly scorer after feature extraction and fusion to obtain an anomaly score.

[0115] The result output module is configured to determine whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated anomaly score.

[0116] The specific working processes of the above-described modules can refer to the corresponding processes in the foregoing method embodiments, and will not be described here. The division of the modules is only a logical functional division, and another division mode can be used in actual implementation, for example, multiple modules can be combined or integrated into another system.

[0117] Based on the same inventive concept, the present application provides a computer system, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program implements the steps of the semi-supervised network traffic anomaly detection method based on the stacked autoencoder when loaded into the processor.

[0118] Based on the same inventive concept, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the semi-supervised network traffic anomaly detection method based on the stacked autoencoder when executed by a processor.

[0119] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one

[0120] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination of flowcharts and / or blocks in the flowcharts can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0121] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination of flowcharts and / or blocks in the flowcharts can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination of flowcharts and / or blocks in the flowcharts can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0123] Finally, it should be noted that the above-described embodiments are merely intended for describing and illustrating, not limiting, the technical solutions of the present application, and although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and any modifications or equivalent replacements made without departing from the spirit and scope of the present application should be covered within the protection scope of the claims of the present application.

Claims

1. A semi-supervised network traffic anomaly detection method based on stacked autoencoder, characterized in that, The method comprises the following steps: obtaining a network traffic dataset for preprocessing and dividing the network traffic dataset into a training dataset and a verification dataset, randomly sampling two-by-two sample pairs from the training dataset to obtain a sample pair set, and setting abnormal reference values for different sample pairs; the training dataset comprises an unlabeled data set, a labeled abnormal data set, and a labeled normal data set; training a two-level stacked autoencoder using the training dataset to obtain a feature extractor, which comprises the following steps: constructing a two-level stacked autoencoder using a sparse autoencoder and a normal autoencoder, and training the constructed two-level stacked autoencoder using the training dataset, as follows: Randomly select one data point from the training dataset. ,Will The input is fed into a sparse autoencoder to obtain... Reconstructed data and characteristics ; By a loss function Computing With Reconstruction error of the sparse auto-encoder model parameters is optimized to minimize the training loss, and the training is stopped after the loss converges. features input into a regular autoencoder, obtaining reconstructed data and features ; By a loss function The reconstruction error of The ordinary autoencoder model parameters are optimized to minimize the training loss, and the training is stopped after the loss converges, obtaining the trained two-level stacked autoencoder as a feature extractor; extracting features of sample pairs in the sample pair set using the trained feature extractor and fusing the features, and inputting the fused features into a traffic anomaly detection model to obtain an abnormal score generator, which comprises the following steps: The sample pairs were processed using two identical feature extractors respectively. Feature extraction was performed on two samples, and the following methods were used: Feature fusion is performed to obtain sample pairs. Features ; A multi-layer perceptron is used as the flow anomaly detection model to fuse features as input, and randomly set a certain dimension of to 0, output the anomaly score of the sample pair, denoted as ; Using a loss function The traffic anomaly detection model is trained to optimize the training model parameters with the objective of minimizing the training loss, and the training is stopped after the loss converges, obtaining an anomaly scorer; randomly selecting one sample from each of the unlabeled data set, the labeled abnormal data set, and the labeled normal data set, and inputting the selected samples and the network traffic data to be detected into the abnormal score generator after feature extraction and fusion to obtain an abnormal score; determining whether the network traffic data to be detected is abnormal according to the abnormal reference values and the calculated abnormal scores.

2. The semi-supervised network traffic anomaly detection method based on stacked autoencoder according to claim 1, characterized in that, The method comprises the following steps: Acquiring network traffic dataset , , satisfying wherein the first items of data are unknown, denoted as an unlabeled data set , the middle items are labeled abnormal data, denoted as a labeled abnormal set , and the last items of data are labeled normal data, denoted as a labeled normal set ; From the set , , 10% of the data were randomly selected as validation data, and the remaining data were used as the training data set. randomly take two samples from the training dataset and form a sample pair , repeat this process to obtain a set of sample pairs , is the number of sample pairs.

3. The semi-supervised network traffic anomaly detection method based on stacked autoencoder according to claim 2, characterized in that, The method comprises the following steps: According to the unlabeled data set , the labeled abnormal data set , and the labeled normal data set , six combinations are obtained by random sampling combination and , and the abnormal reference values are respectively set as , , , , and , wherein .

4. The semi-supervised network traffic anomaly detection method based on stacked autoencoder according to claim 3, characterized in that, The loss function is expressed as follows: ， wherein, is a sparsity measure function, is a sparse autoencoder model parameter, is a sparsity penalty parameter, is a regularization parameter; ， wherein, are the common autoencoder model parameters, is a regularization parameter.

5. The semi-supervised network traffic anomaly detection method based on stacked autoencoder according to claim 3, characterized in that, The method comprises the following steps: a sample is randomly selected from the labeled abnormal data set , the unlabeled data set , and the labeled normal data set , respectively, and is recorded as , , and , respectively, to form data pairs , , and with the network traffic data to be detected , respectively, and after feature extraction and fusion, the fusion features , , are obtained; the network traffic data to be detected is obtained by using a packet capturing tool to obtain traffic packet data for preprocessing The fused features , , are input into the anomaly scorer, respectively, and an anomaly score , , is output. Abnormal score With , And Abnormal reference value , And Comparison, if , the reference decision for the corresponding input data pair is normal; if , the reference decision for the corresponding input data pair is abnormal; Abnormal score With , and Abnormal reference value , and Compare, if , the reference decision for the corresponding input data pair is normal; if , the reference decision for the corresponding input data pair is abnormal; Abnormal score With , and Abnormal reference value , and Compare, if , the reference decision for the corresponding input data pair is normal; if , the reference decision for the corresponding input data pair is abnormal; If two or more reference values determine that the network traffic data to be detected is normal, the network traffic data to be detected is determined to be normal; otherwise, the network traffic data to be detected is determined to be abnormal.

6. A semi-supervised network traffic anomaly detection apparatus based on stacked autoencoders, characterized in that, The device for implementing the semi-supervised network traffic anomaly detection method based on a stacked autoencoder comprises the following modules: a data preprocessing module for obtaining a network traffic dataset for preprocessing and dividing the network traffic dataset into a training dataset and a verification dataset, randomly sampling two-by-two sample pairs from the training dataset to obtain a sample pair set, and setting abnormal reference values for different sample pairs; the training dataset comprises an unlabeled data set, a labeled abnormal data set, and a labeled normal data set; a feature extractor training module for training a two-level stacked autoencoder using the training dataset to obtain a feature extractor; an abnormal score generator training module for extracting features of sample pairs in the sample pair set using the trained feature extractor and fusing the features, and inputting the fused features into a traffic anomaly detection model to obtain an abnormal score generator; a scoring module for randomly selecting one sample from each of the unlabeled data set, the labeled abnormal data set, and the labeled normal data set, and inputting the selected samples and the network traffic data to be detected into the abnormal score generator after feature extraction and fusion to obtain an abnormal score; a result output module for determining whether the network traffic data to be detected is abnormal according to the abnormal reference values and the calculated abnormal scores.

7. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method of any of claims 1-6. The one or more programs, when executed by a computing device, cause the computing device to perform any of the methods of semi-supervised network traffic anomaly detection based on stacked autoencoders recited in claims 1-5.

8. A computing device, comprising: comprise, one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods of semi-supervised network traffic anomaly detection based on stacked autoencoders recited in claims 1-5.

Images