Semi-supervised network traffic anomaly detection method and device based on stacked auto-encoders

Abstract

The invention discloses a semi-supervised network traffic anomaly detection method and device based on a stacked auto-encoder, and the method comprises the steps: selecting a training data set, and randomly sampling every two training data sets to form a sample pair, thereby obtaining a sample pair set; training a secondary stacked auto-encoder by adopting the training data set to obtain a feature extractor; extracting and fusing the features of the sample pairs in the sample pair set, and training to obtain an abnormal scoring device; randomly extracting samples from the training data set to form a sample pair with the to-be-detected data, and obtaining an abnormal score; and judging whether the data is abnormal according to the abnormal reference value and the abnormal score. According to the method, the labeled samples and the unlabeled samples are randomly combined, and the existing label information is fully utilized to expand the number of the labeled samples for model training under the condition that label data are scarce, so that the influence of insufficient labeled samples on the model performance and generalization ability can be effectively reduced; and the network flow data anomaly detection precision of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic anomaly detection, and in particular to a semi-supervised network traffic anomaly detection method and device based on stacked autoencoders. Background Art

[0002] In today's digital age, the Internet has become an indispensable part of people's lives and work. With the widespread popularization of the Internet and the continuous expansion of its applications, network traffic has shown a rapid growth trend. However, the sharp increase in network traffic has also caused a series of security issues, among which network traffic anomaly detection has become one of the key research directions in the field of network security.

[0003] Network traffic anomalies refer to situations where the traffic behavior in the network deviates significantly from the normal pattern. These anomalies may be caused by a variety of factors, such as network attacks, malware attacks, equipment failures, network configuration errors, etc. If these anomalies are not detected and handled in time, they may lead to serious consequences such as reduced network performance, service interruptions, data leakage, etc., causing huge losses to individuals and enterprises.

[0004] Traditional network security technologies mainly focus on firewalls, intrusion detection systems, etc. However, these technologies can usually only detect known attack patterns, and are powerless against new and unknown attack methods. Network traffic anomaly detection technology is a method based on data analysis, which explores potential abnormal behaviors through real-time monitoring and analysis of network traffic.

[0005] As a machine learning technology based on neural network structure, deep learning aims to explore the inherent laws of sample data. Compared with traditional machine learning models, deep learning has higher efficiency in model construction and feature extraction, and can achieve higher accuracy when processing large-scale data and complex structure data. According to the difference in training methods, traffic anomaly detection methods can be divided into supervised learning, unsupervised learning and semi-supervised learning. Among them, supervised learning requires all training data to have labels, while unsupervised learning does not rely on label information at all. Semi-supervised learning is between the two, and it uses a large amount of unlabeled data and a small amount of labeled data for joint training. In actual application scenarios, abnormal conditions are usually rare. It is not only difficult but also costly to obtain sufficient labeled abnormal data, which may cause data imbalance and affect the training effect of the model. Summary of the invention

[0006] The purpose of the present invention is to propose a semi-supervised network traffic anomaly detection method and device based on stacked autoencoders. By randomly combining normal samples, abnormal samples and unlabeled samples to obtain sample pairs, the stacked autoencoder is used to extract features of the data, and the existing label information is fully combined to judge the degree of abnormality of the data. The method can effectively reduce the impact of the insufficient number of labeled abnormal labels on the model performance and generalization ability, and improve the accuracy of the model in detecting anomalies in network traffic data.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a semi-supervised network traffic anomaly detection method based on a stacked autoencoder, comprising:

[0009] A network traffic data set is obtained for preprocessing and divided into a training data set and a validation data set, the training data set is randomly sampled to form sample pairs, a sample pair set is obtained, and abnormal reference values ​​of different sample pairs are set; the training data set includes an unlabeled data set, a labeled abnormal data set, and a labeled normal data set;

[0010] Using the training data set to train a two-level stacked autoencoder to obtain a feature extractor;

[0011] The trained feature extractor is used to extract features of sample pairs in the sample pair set and fuse them, and the features are used as input to train a traffic anomaly detection model to obtain an anomaly scorer;

[0012] A sample is randomly selected from each of the unlabeled data set, the labeled abnormal data set, and the labeled normal data set to form a sample pair with the network traffic data to be detected, and after feature extraction and fusion, the sample is input into an anomaly scorer to obtain an anomaly score;

[0013] Whether the network traffic data to be detected is abnormal is determined according to the abnormal reference value and the calculated abnormal score.

[0014] Preferably, the network traffic data set is obtained and divided into a training data set and a validation data set, and the training data set is randomly sampled to form sample pairs, to obtain a sample pair set, including:

[0015] Get network traffic dataset , ,satisfy , among which The label of the item data is unknown, recorded as an unlabeled data set ,middle Item is the labeled abnormal data, recorded as the labeled abnormal set ,at last The data item is labeled normal data, recorded as the labeled normal set ;

[0016] From the collection , , 10% of the data are randomly selected as verification data, and the remaining data are used as training data sets;

[0017] Randomly select two samples from the training data set and Composition sample pair Repeat this process to obtain a set of sample pairs , is the number of sample pairs.

[0018] Preferably, the setting of abnormal reference values ​​for different sample pairs includes:

[0019] Based on the unlabeled data set , a collection of labeled anomaly data and labeled normal data set , obtained by random sampling combination and There are six combinations, and the abnormal reference values ​​are set as , , , , and ,in, .

[0020] Preferably, the adopting of the training data set to train a two-level stacked autoencoder to obtain a feature extractor comprises:

[0021] A sparse autoencoder and a common autoencoder are used to form a two-level stacked autoencoder, and a training data set is used to train the constructed two-level stacked autoencoder as follows:

[0022] Randomly select a piece of data from the training data set ,Will Input into the sparse autoencoder to obtain Reconstructed data and Features ;

[0023] Through the loss function calculate and The reconstruction error is calculated, and the sparse autoencoder model parameters are optimized to minimize the training loss. The training is stopped after the loss converges.

[0024] The characteristics Input into the ordinary autoencoder to obtain Reconstructed data and Features ;

[0025] Through the loss function calculate The reconstruction error is taken into account, and the parameters of the ordinary autoencoder model are optimized with the goal of minimizing the training loss. The training is stopped after the loss converges, and the trained two-level stacked autoencoder is obtained as the feature extractor.

[0026] Preferably, the loss function is expressed as follows:

[0027] ,

[0028] in, is a sparse metric function, are the sparse autoencoder model parameters, is the sparse penalty parameter, is the regularization parameter;

[0029] ,

[0030] in, are the parameters of the common autoencoder model, is the regularization parameter.

[0031] Preferably, the feature extractor that has been trained is used to extract features of sample pairs in the sample pair set and fuse them, and this is used as input to train a traffic anomaly detection model to obtain an anomaly scorer, including:

[0032] Use two identical feature extractors to extract the sample pairs The two samples in the feature extraction are used, and The feature fusion is performed by multiplication to obtain the sample pair Features ;

[0033] A multi-layer perceptron is used as a traffic anomaly detection model to fuse features As input, and randomly Set one dimension of to 0 and output the abnormal score of the sample pair, denoted as ;

[0034] Using loss function The traffic anomaly detection model is trained to optimize the training model parameters with the goal of minimizing the training loss. The training is stopped after the loss converges to obtain the anomaly scorer.

[0035] Preferably, judging whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated abnormal score includes:

[0036] From the labeled anomaly data set , unlabeled data set and labeled normal data set A sample is randomly selected from each of the , and , respectively with the network traffic data to be detected Composition data pair , and After feature extraction and fusion, the fusion feature is obtained , , ; The network traffic data to be detected is obtained by using a packet capture tool to obtain traffic packet data and preprocessing it;

[0037] The fusion features , , Input into the anomaly scorer respectively and output the anomaly score , , ;

[0038] The anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if Then the reference for the corresponding input data pair is judged as abnormal;

[0039] The anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal;

[0040] The anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal;

[0041] If two or more references are judged to be normal, the network traffic data to be detected is judged to be normal, otherwise it is abnormal.

[0042] In a second aspect, the present invention provides a semi-supervised network traffic anomaly detection device based on a stacked autoencoder, which is used to implement the above-mentioned semi-supervised network traffic anomaly detection method based on a stacked autoencoder, and the device includes:

[0043] The data preprocessing module is used to obtain a network traffic data set for preprocessing and divide it into a training data set and a verification data set, randomly sample the training data set to form sample pairs, obtain a sample pair set, and set abnormal reference values ​​for different sample pairs; the training data set includes an unlabeled data set, a labeled abnormal data set, and a labeled normal data set;

[0044] A feature extractor training module, used to train a two-level stacked autoencoder using the training data set to obtain a feature extractor;

[0045] An anomaly scorer training module, used to extract features of sample pairs in the sample pair set using the trained feature extractor and fuse them, and use them as input to train a traffic anomaly detection model to obtain an anomaly scorer;

[0046] A scoring module is used to randomly select a sample from the unlabeled data set, the labeled abnormal data set and the labeled normal data set, respectively, to form a sample pair with the network traffic data to be detected, and after feature extraction and fusion, input the sample into an anomaly scorer to obtain an anomaly score;

[0047] The result output module is used to determine whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated abnormal score.

[0048] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform any of the above-mentioned semi-supervised network traffic anomaly detection methods based on stacked autoencoders.

[0049] In a fourth aspect, the present invention provides a computing device comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above-mentioned semi-supervised network traffic anomaly detection methods based on stacked autoencoders.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The semi-supervised network traffic anomaly detection method based on stacked autoencoders provided by the present invention obtains sample pairs by randomly combining normal samples, abnormal samples and unlabeled samples, and makes full use of the existing label information to expand the number of labeled samples for model training, which can effectively reduce the impact of insufficient labeled samples on model performance and generalization ability, and improve the accuracy of the model in detecting anomalies in network traffic data;

[0052] 2. The semi-supervised network traffic anomaly detection method based on stacked autoencoders provided by the present invention extracts features from network traffic data through stacked autoencoders, which can effectively mine richer representation forms and abnormal patterns of the data, reduce the impact of high-dimensional data on the detection effect, and further improve the accuracy of the detection results;

[0053] 3. The semi-supervised network traffic anomaly detection method based on stacked autoencoders provided by the present invention can replace modules such as anomaly scorers with other models and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of a semi-supervised network traffic anomaly detection method based on stacked autoencoders provided by the present invention;

[0055] Figure 2 The figure is a flowchart of network traffic data anomaly detection in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0057] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0058] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0059] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0060] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0061] The first embodiment of the present invention provides a semi-supervised network traffic anomaly detection method based on stacked autoencoders, see Figure 1 ,include:

[0062] 1) Select a traffic data set. The network traffic data in the traffic data set should include information such as IP, port, protocol, load size, duration, service type, etc. The data set is preprocessed and divided into a training data set and a validation set. The training data set includes an unlabeled data set, a labeled abnormal data set, and a labeled normal data set;

[0063] 2) Randomly sample the training data set into sample pairs, obtain a sample pair set, and set abnormal reference values ​​for different sample pairs;

[0064] 3) Using the training data set to train a two-level stacked autoencoder consisting of a sparse autoencoder and a normal autoencoder, a feature extractor is obtained;

[0065] 4) Use the trained feature extractor to extract and fuse the features of the sample pairs in the above sample pair set, and use this as input to train the traffic anomaly detection model to obtain the anomaly scorer;

[0066] 5) Preprocess the traffic packet data obtained from packet capture tools such as Wireshark to obtain the network traffic data to be detected, randomly select a sample from the unlabeled data set, the labeled abnormal data set, and the labeled normal data set, and form a sample pair with the data to be detected. After feature extraction and fusion, input it into the anomaly scorer to obtain the anomaly score;

[0067] 6) Determine whether the network traffic data to be detected is abnormal based on the abnormal reference value and the calculated abnormal score.

[0068] In the embodiment of the present invention, the flow data set is randomly sampled to form sample pairs, and a sample pair set is obtained, which is specifically as follows:

[0069] Given a traffic dataset ,satisfy . Among them The label of the item data is unknown, recorded as an unlabeled data set ,middle Item is the labeled abnormal data, recorded as the labeled abnormal set ,at last The data item is labeled normal data, recorded as the labeled normal set .

[0070] Randomly select 10% of the data from sets U, A, and N as test data The remaining data is used as the training set ,Right now From the training dataset Randomly select two samples from and Composition sample pair Repeat this process to obtain a set of sample pairs , is the number of sample pairs.

[0071] In the embodiment of the present invention, the abnormal reference values ​​of different sample pairs are set according to the known label information combined with the sample category combination, including:

[0072] Based on the unlabeled data set , a collection of labeled anomaly data and labeled normal data set , through random sampling combination, we can get, and Six combinations, Representing a collection A sample pair consisting of two data in , Representing a collection and collection The sample pairs composed of data in Representing a collection and collection The sample pairs composed of data in Representing a collection A sample pair consisting of two data in , Representing a collection and collection The sample pairs composed of data in , their abnormal reference values ​​are set as , , , , and ,in, .

[0073] In the embodiment of the present invention, a two-stage stacked autoencoder consisting of a sparse autoencoder and a common autoencoder is used to train a data set. The constructed two-level stacked autoencoder is trained to obtain a feature extractor. The specific process is as follows:

[0074] S21: From Randomly select a piece of data ,Will Input into the sparse autoencoder to obtain Reconstructed data and Features ;

[0075] S22: Through the loss function calculate and The reconstruction error of

[0076] S23: Repeat S21 to S22, and optimize the model parameters with the goal of minimizing the training loss, and stop training after the loss converges;

[0077] S24: The features obtained from S21 Input into the autoencoder and obtain Reconstructed data and Features ;

[0078] S25: Through the loss function calculate The reconstruction error of

[0079] S26: Repeat S24 to S25, and optimize the model parameters with the goal of minimizing the training loss. Stop the training after the loss converges to obtain a trained two-level stacked autoencoder.

[0080] S27: Use two identical secondary stacked autoencoders to encode the sample pairs separately The two samples in the feature extraction are used, and The feature fusion is performed by multiplication to obtain the sample pair Features .

[0081] The above process is expressed as follows:

[0082] ,

[0083] ,

[0084] ,

[0085] ,

[0086] in, represents the original input data, , Respectively represent the features extracted by sparse autoencoder and ordinary autoencoder, , Respectively represent the sparse autoencoder and ordinary autoencoder model parameters, , They represent the data reconstructed by sparse autoencoder and ordinary autoencoder respectively.

[0087] First, use the loss function Train the sparse autoencoder. After training, use the features extracted by the sparse autoencoder to As input, and use the loss function Training a vanilla autoencoder.

[0088] The final trained feature extractor is expressed as:

[0089] ,

[0090] in, and The parameters are obtained through training. is the input sample, is the final extracted feature.

[0091] Loss Function , , which is expressed as follows:

[0092] ,

[0093] in, is a sparse metric function, is the training parameter, is the sparse penalty parameter, is the regularization parameter;

[0094] ,

[0095] in, is the training parameter, is the regularization parameter.

[0096] In the embodiment of the present invention, the features extracted by the trained feature extractor are used to train the traffic anomaly detection model to obtain an anomaly scorer;

[0097] S31: Extract features from each sample in the sample pair in the sample pair set using the above-trained two-level stacked autoencoder, and use The feature fusion is performed by multiplication to obtain the sample pair Features ; The feature Input into the constructed traffic anomaly detection model;

[0098] S32, the traffic anomaly detection model uses a multi-layer perceptron to fuse features As input, and randomly One dimension of is set to 0 to improve the generalization of the model, and the output is the abnormal score of the sample pair, denoted as ;

[0099] S33. Use loss function Train the traffic anomaly detection model. It is expressed as follows:

[0100] ,

[0101] in, is the anomaly score obtained by the traffic anomaly detection model, is the abnormal reference value of the sample pair, is the training parameter, is the regularization parameter;

[0102] S34: Repeat S31 to S33, and optimize the training parameters with the goal of minimizing the training loss. Stop the training after the loss converges to obtain the anomaly scorer.

[0103] In the embodiment of the present invention, a sample is randomly selected from each of the unlabeled data set, the labeled abnormal data set, and the labeled normal data set to form a sample pair with the data to be detected. After feature extraction and fusion, the sample is input into the anomaly scorer to obtain an anomaly score. In addition, whether the data to be detected is abnormal is determined based on the abnormal reference value and the calculated anomaly score. Figure 2 , the specific process is as follows:

[0104] S41: From , and A sample is randomly selected from each of the , and , respectively with the data to be detected Composition data pair , and After feature extraction and fusion, the fusion feature is obtained , , ;

[0105] S42: Fusion features , , Input into the anomaly scorer respectively and output the anomaly score , , ;

[0106] S43: Anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal;

[0107] S44: Anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal;

[0108] S45: Anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal;

[0109] S46: If two or more references are judged to be normal, the data is judged to be normal, otherwise it is abnormal.

[0110] Based on the above inventive concept, the present invention also provides a semi-supervised network traffic anomaly detection device based on stacked autoencoders, the device comprising:

[0111] The data preprocessing module is used to obtain a network traffic data set for preprocessing and divide it into a training data set and a verification data set, randomly sample the training data set to form sample pairs, obtain a sample pair set, and set abnormal reference values ​​for different sample pairs; the training data set includes an unlabeled data set, a labeled abnormal data set, and a labeled normal data set;

[0112] A feature extractor training module, used for training a two-level stacked autoencoder consisting of a sparse autoencoder and a common autoencoder using a training data set to obtain a feature extractor;

[0113] An anomaly scorer training module is used to use the trained feature extractor to extract and fuse the features of the sample pairs in the sample pair set, and use the features as input to train the traffic anomaly detection model to obtain an anomaly scorer;

[0114] The scoring module is used to randomly select a sample from each of the unlabeled data set, the labeled abnormal data set and the labeled normal data set, and form a sample pair with the network traffic data to be detected. After feature extraction and fusion, the sample is input into the anomaly scorer to obtain an anomaly score.

[0115] The result output module is used to determine whether the network traffic data to be detected is abnormal based on the abnormal reference value and the calculated abnormal score.

[0116] The specific working process of each module described above can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. The module division is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system.

[0117] Based on the same inventive concept, the present invention provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the above-mentioned semi-supervised network traffic anomaly detection method based on stacked autoencoders are implemented.

[0118] Based on the same inventive concept, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned semi-supervised network traffic anomaly detection method based on stacked autoencoders.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A semi-supervised network traffic anomaly detection method based on stacked autoencoders, characterized in that: include: A network traffic data set is obtained for preprocessing and divided into a training data set and a validation data set, the training data set is randomly sampled to form sample pairs, a sample pair set is obtained, and abnormal reference values ​​of different sample pairs are set; the training data set includes an unlabeled data set, a labeled abnormal data set, and a labeled normal data set; Using the training data set to train a two-level stacked autoencoder to obtain a feature extractor; The trained feature extractor is used to extract features of sample pairs in the sample pair set and fuse them, and the features are used as input to train a traffic anomaly detection model to obtain an anomaly scorer; A sample is randomly selected from each of the unlabeled data set, the labeled abnormal data set, and the labeled normal data set to form a sample pair with the network traffic data to be detected, and after feature extraction and fusion, the sample is input into an anomaly scorer to obtain an anomaly score; Whether the network traffic data to be detected is abnormal is determined according to the abnormal reference value and the calculated abnormal score.

2. According to the semi-supervised network traffic anomaly detection method based on stacked autoencoders according to claim 1, it is characterized in that: The network traffic data set is obtained and divided into a training data set and a verification data set, and the training data set is randomly sampled in pairs to form sample pairs, to obtain a sample pair set, including: Get network traffic dataset , ,satisfy , among which the former The label of the item data is unknown, recorded as an unlabeled data set ,middle Item is the labeled abnormal data, recorded as the labeled abnormal set ,at last The data item is labeled normal data, recorded as the labeled normal set ; From the collection , , 10% of the data are randomly selected as verification data, and the remaining data are used as training data sets; Randomly select two samples from the training data set and Composition sample pair Repeat this process to obtain a set of sample pairs , is the number of sample pairs.

3. According to claim 2, a semi-supervised network traffic anomaly detection method based on stacked autoencoders is characterized in that: The setting of abnormal reference values ​​for different sample pairs includes: Based on the unlabeled data set , a collection of labeled anomaly data and labeled normal data set , obtained by random sampling combination and There are six combinations, and the abnormal reference values ​​are set as , , , , and ,in, .

4. According to claim 3, a semi-supervised network traffic anomaly detection method based on stacked autoencoders is characterized in that: The method of using the training data set to train a two-level stacked autoencoder to obtain a feature extractor includes: A sparse autoencoder and a common autoencoder are used to form a two-level stacked autoencoder, and a training data set is used to train the constructed two-level stacked autoencoder as follows: Randomly select a piece of data from the training data set ,Will Input into the sparse autoencoder to obtain Reconstructed data and Features ; Through the loss function calculate and The reconstruction error is calculated, and the sparse autoencoder model parameters are optimized to minimize the training loss. The training is stopped after the loss converges. The characteristics Input into the ordinary autoencoder to obtain Reconstructed data and Features ; Through the loss function calculate The reconstruction error is taken into account, and the parameters of the ordinary autoencoder model are optimized with the goal of minimizing the training loss. The training is stopped after the loss converges, and the trained two-level stacked autoencoder is obtained as the feature extractor.

5. According to claim 4, a semi-supervised network traffic anomaly detection method based on stacked autoencoders is characterized in that: The loss function is expressed as follows: ， in, is a sparse metric function, are the sparse autoencoder model parameters, is the sparse penalty parameter, is the regularization parameter; ， in, are the parameters of the common autoencoder model, is the regularization parameter.

6. According to claim 4, a semi-supervised network traffic anomaly detection method based on stacked autoencoders is characterized in that: The feature extractor that has been trained is used to extract features of sample pairs in the sample pair set and fuse them, and the features are used as input to train a traffic anomaly detection model to obtain an anomaly scorer, including: Use two identical feature extractors to extract the sample pairs The two samples in the feature extraction are used, and The feature fusion is performed by multiplication to obtain the sample pair Features ; A multi-layer perceptron is used as a traffic anomaly detection model to fuse features As input, and randomly Set one dimension of to 0 and output the abnormal score of the sample pair, denoted as ; Using loss function The traffic anomaly detection model is trained to optimize the training model parameters with the goal of minimizing the training loss. The training is stopped after the loss converges to obtain the anomaly scorer.

7. A semi-supervised network traffic anomaly detection method based on stacked autoencoders according to claim 6, characterized in that: The determining whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated abnormal score includes: From the labeled anomaly data set , unlabeled data set and labeled normal data set A sample is randomly selected from each of the , and , respectively, with the network traffic data to be detected Composition data pair , and After feature extraction and fusion, the fusion feature is obtained , , ; The network traffic data to be detected is obtained by using a packet capture tool to obtain traffic packet data and preprocessing it; The fusion features , , Input into the anomaly scorer respectively and output the anomaly score , , ; The anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if Then the reference for the corresponding input data pair is judged as abnormal; The anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal; The anomaly score and , and Abnormal reference value , and For comparison, if , then the reference for the corresponding input data pair is judged to be normal; if , then the reference of the corresponding input data pair is judged as abnormal; If two or more references are judged to be normal, the network traffic data to be detected is judged to be normal, otherwise it is abnormal.

8. A semi-supervised network traffic anomaly detection device based on stacked autoencoders, characterized in that: The device is used to implement the semi-supervised network traffic anomaly detection method based on stacked autoencoders according to any one of claims 1 to 7, the device comprising: The data preprocessing module is used to obtain a network traffic data set for preprocessing and divide it into a training data set and a verification data set, randomly sample the training data set to form sample pairs, obtain a sample pair set, and set abnormal reference values ​​for different sample pairs; the training data set includes an unlabeled data set, a labeled abnormal data set, and a labeled normal data set; A feature extractor training module, used to train a two-level stacked autoencoder using the training data set to obtain a feature extractor; An anomaly scorer training module, used to extract features of sample pairs in the sample pair set using the trained feature extractor and fuse them, and use them as input to train a traffic anomaly detection model to obtain an anomaly scorer; A scoring module is used to randomly select a sample from the unlabeled data set, the labeled abnormal data set and the labeled normal data set, respectively, to form a sample pair with the network traffic data to be detected, and after feature extraction and fusion, input the sample into an anomaly scorer to obtain an anomaly score; The result output module is used to determine whether the network traffic data to be detected is abnormal according to the abnormal reference value and the calculated abnormal score.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the semi-supervised network traffic anomaly detection methods based on stacked autoencoders according to claims 1 to 7.

10. A computing device, characterized in that include, One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the semi-supervised network traffic anomaly detection methods based on stacked autoencoders according to claims 1 to 7.

Images