Network traffic detection method, system, electronic device and storage medium

Abstract

The present invention discloses a network traffic detection method, system, electronic device and storage medium. The method comprises extracting flow-level features from captured network traffic and performing unique-hot encoding on the features to obtain a multi-dimensional feature vector, using the kmeans algorithm to cluster the multi-dimensional feature vector into benign traffic and malicious traffic respectively, and generating sub-class labels; using a multi-centroid contrast autoencoder to reduce the dimension of the multi-dimensional feature vector to obtain an embedded feature vector; using a multi-layer perceptron to perform binary classification on the embedded feature vector, obtaining the benign class confidence and the malicious class confidence of the embedded feature vector, and determining the class corresponding to the high confidence as the predicted class; based on the sub-class centroid and confidence of the embedded feature vector, screening samples to be detected, and using the screened samples to be detected to optimize the multi-centroid contrast autoencoder and the multi-layer perceptron. The method of the present invention has a continuous and efficient ability to distinguish normal traffic from malicious traffic.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a network flow detection method, system, electronic equipment and storage medium. Background Art

[0002] In recent years, the network security situation has intensified, and detecting malicious intrusion behaviors in the network is an important challenge. Network Intrusion Detection System (NIDS) is a feasible solution to protect the target network from malicious attacks. Most previous research works use known data as prior knowledge to learn intrusion detection models to enable intrusion detection systems, and then deploy them in actual network environments to detect intrusions. That is, the statistical characteristics and label distribution of network traffic data will not change, but in real network environments, the characteristics and distribution of traffic will change over time. For example, the behavior of users in email systems and social networks may change over time, resulting in changes in data distribution. This phenomenon is called "concept drift". In addition, new vulnerabilities and attack categories continue to emerge, such as 0-day and N-day; the environment in which data is generated and the way data is collected may also change. The above situations will cause decision bias in the intrusion detection system, ignoring new patterns and abnormal behaviors that appear in the new data distribution, resulting in model aging. Therefore, only considering stable network environments and known attacks can no longer meet the actual application needs of effective network intrusion detection.

[0003] The processing methods of concept drift are mainly divided into passive processing methods and active processing methods. When dealing with concept drift, the passive processing method does not rely on the explicit concept drift detection mechanism, but adapts to the potential changes in the data stream by continuously adjusting the learning model. Existing methods include single learner methods and ensemble learning methods. When dealing with concept drift, the single learner method adapts to the changes in the data stream by controlling the complexity of the model. The advantages of this type of method are low computational overhead and the ability to adapt to changes in the data stream in real time. In the decision tree model, leaf nodes are incrementally replaced or decision trees are scanned regularly to detect and adapt to concept drift; in the Bayesian model, the forgetting mechanism is implemented to quickly adjust the attribute weights. Ensemble learning methods rely on the two basic points of diversity and adaptability to deal with concept drift and ensure a good combination of base learners. Block-based ensemble methods adapt to concept drift by creating new base classifiers for each data block or updating existing classifiers; online ensemble methods select appropriate data block sizes by learning and discarding each training sample separately; in addition, the weight coefficient of each classifier can be dynamically weighted and adjusted. However, passive learning methods take a long time to adapt to the new distribution and are not effective in dealing with noisy data and class imbalance. Active processing methods rely on the detected concept drift trigger mechanism to adjust the learning model to adapt to changes in the data stream when dealing with concept drift. The core is to accurately judge the time and severity of concept drift so that the model can be adjusted accordingly. Existing methods include ensemble learning-based methods, window-based methods, and neural network-based methods. The core idea of ​​ensemble learning-based methods is to use the prediction results of multiple models to make the final classification decision through voting or averaging, thereby reducing the performance degradation of a single model due to concept drift. However, this method has a high time complexity and requires more resources. Window-based methods track changes in data by maintaining one or two windows, and rely on statistical properties of the data in the window, such as mean or variance, to determine whether concept drift occurs. However, this method assumes that the data meets a certain distribution, but in reality, network data is generally messy. Summary of the invention

[0004] In view of the above problems, the present invention provides a network traffic detection method, system, electronic device and storage medium, aiming to have the ability to continuously and efficiently distinguish normal traffic from malicious traffic.

[0005] According to a first aspect of an embodiment of the present disclosure, a network traffic detection method is provided, the method comprising the following steps:

[0006] Extract flow-level features from the captured network traffic, perform one-hot encoding on the flow-level features to obtain a multi-dimensional feature vector, use the kmeans algorithm to cluster the multi-dimensional feature vector into benign traffic and malicious traffic, and generate sub-class labels;

[0007] Using a multi-centroid contrast autoencoder to reduce the dimension of the multi-dimensional feature vector to obtain an embedded feature vector;

[0008] Using a multi-layer perceptron to perform binary classification on the embedded feature vector, obtaining a benign class confidence and a malicious class confidence of the embedded feature vector, and determining the class corresponding to a high confidence as the predicted class;

[0009] The samples to be detected are screened based on the subclass centroids and confidences of the embedded feature vectors, and the screened samples to be detected are used to optimize the multi-centroid contrast autoencoder and the multi-layer perceptron.

[0010] In some embodiments, the kmeans algorithm sets the k value range to 2-20, calculates the silhouette coefficients corresponding to different k values, saves the k value corresponding to the maximum silhouette coefficient, and generates the corresponding subclass label.

[0011] In some embodiments, the loss function expression of the multi-centroid contrast autoencoder is as follows:

[0012] ,

[0013] in, represents the original sample characteristics, express The expected value of express The reconstructed sample features, is the loss coefficient, Representation sample and samples The expected value of , , , All samples and samples Category-related parameters, Representation sample and samples The distance between the embedded feature vectors, The + in the Greater than the preset When taking value, is 0, In Indicates when Greater than the preset When taking value, is 0.

[0014] In some embodiments, the possibility of the sample to be detected being screened is specifically as follows: based on the benign class and malicious class predicted by the multi-layer perceptron, the centroids of each subclass of the benign class and the malicious class are calculated respectively, and a benign class centroid closest to the sample to be detected and a malicious class centroid closest to the sample to be detected are found, and then the minimum ratio between the two closest distances is calculated; the confidence of the output of the sample to be detected is predicted by the multi-layer perceptron to obtain the uncertainty probability of the sample to be detected, and the minimum ratio is added to the uncertainty probability to constitute the selection possibility of the sample to be detected.

[0015] In some embodiments, the selection possibility of the sample to be detected is expressed as follows:

[0016] ,

[0017] in, represents the distance between the sample to be detected and the nearest benign class centroid in the embedding space, represents the distance between the sample to be detected and the nearest malicious class centroid in the embedding space, It represents the embedded feature vector of the sample to be tested after passing through the multi-centroid contrast autoencoder. It is the confidence of the output of the embedded feature vector after passing through the multi-layer perceptron.

[0018] According to a second aspect of an embodiment of the present disclosure, a network traffic detection system is provided, the system comprising:

[0019] A network traffic preprocessing module is used to extract flow-level features from the captured network traffic, perform one-hot encoding on the flow-level features to obtain a multi-dimensional feature vector, use the kmeans algorithm to cluster the multi-dimensional feature vector into benign traffic and malicious traffic, and generate sub-class labels;

[0020] A network traffic representation module, used for reducing the dimension of the multi-dimensional feature vector using a multi-centroid contrast autoencoder to obtain an embedded feature vector;

[0021] A network traffic detection module, used to use a multi-layer perceptron to perform binary classification on the embedded feature vector, obtain the benign class confidence and the malicious class confidence of the embedded feature vector, and determine the class corresponding to the high confidence as the predicted class;

[0022] An adaptation module is used to screen samples to be detected based on the subclass centroids and confidence levels of the embedded feature vectors, and to optimize the multi-centroid contrast autoencoder and the multi-layer perceptron using the screened samples to be detected.

[0023] In some embodiments, the k value range is set to 2-20 in the kmeans algorithm in the network traffic preprocessing module, the silhouette coefficients corresponding to different k values ​​are calculated, the k value corresponding to the maximum silhouette coefficient is saved and the corresponding subclass label is generated.

[0024] In some embodiments, the possibility of the sample to be detected in the adaptation module being screened is specifically as follows: based on the benign class and malicious class predicted by the multi-layer perceptron, the centroids of each subclass of the benign class and the malicious class are calculated respectively, and a benign class centroid closest to the sample to be detected and a malicious class centroid closest to the sample to be detected are found, and then the minimum ratio between the two closest distances is calculated; the confidence of the output of the sample to be detected is predicted by the multi-layer perceptron to obtain the uncertainty probability of the sample to be detected, and the minimum ratio is added to the uncertainty probability to constitute the selection possibility of the sample to be detected.

[0025] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned network traffic detection method when executing the program.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the above-mentioned network traffic detection method are implemented.

[0027] The disclosed embodiments provide a network traffic detection method, system, electronic device and storage medium. The method of the present invention is an active learning method based on a multi-centroid comparison autoencoder, which can effectively reduce the dimension of network traffic features based on the multi-centroid comparison autoencoder, and screen samples that affect data distribution for fine-tuning, thereby improving the detection efficiency of the model; the multi-centroid comparison autoencoder backbone model is used to represent network traffic features, and the statistical characteristics of the traffic are fully utilized to obtain a more effective feature representation. In addition, the method of the present invention proposes an effective sample collection method, which combines the nearest centroid ratio and uncertainty to directly screen out the sample set that affects the distribution difference, which is conducive to further fine-tuning. Further, the method of the present invention allows the distribution characteristics of network traffic to be continuously learned in a flexible manner using multi-centroid comparison autoencoders, nearest centroid ratios and uncertainty sampling, without forgetting the learned distribution characteristics, thereby protecting the target network system from malicious attacks. In summary, the method of the present invention has good dimensionality reduction and representation capabilities for high-dimensional features, and has a continuous and efficient discrimination capability for normal traffic and malicious traffic.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0030] Figure 1 It is a schematic diagram of the flow of a network traffic detection method in an embodiment of the present invention;

[0031] Figure 2 is an overall block diagram of a network traffic detection method according to an embodiment of the present invention;

[0032] Figure 3 is a flow chart of preprocessing original network traffic in an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of the structure of a network traffic detection system in an embodiment of the present invention;

[0034] Figure 5 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0036] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0037] The present invention provides a network traffic detection method, system, electronic device and storage medium, and provides the following embodiments:

[0038] The neural network-based method selects different neural networks as the backbone network detection samples and sets different detectors to determine whether concept drift occurs, generating the final detection results. Due to the efficient detection capability of neural networks, this active processing method is considered to be a feasible solution for concept drift research.

[0039] like Figure 1 As shown, the network traffic detection method includes the following steps:

[0040] S1. Extract flow-level features from captured network traffic, perform one-hot encoding on the flow-level features to obtain a multidimensional feature vector, use the kmeans algorithm to cluster the multidimensional feature vector into benign traffic and malicious traffic, and generate benign subclass labels and malicious subclass labels;

[0041] S2. Using a multi-centroid contrast autoencoder to reduce the dimension of the multi-dimensional feature vector to obtain an embedded feature vector;

[0042] S3, using a multilayer perceptron to perform binary classification on the embedded feature vector, obtaining the benign class confidence and the malicious class confidence of the embedded feature vector, and determining the class corresponding to the high confidence as the predicted class;

[0043] S4. Filter the samples to be detected based on the subclass centroid and confidence of the embedded feature vector, and use the filtered samples to be detected to optimize the multi-centroid comparison autoencoder and the multi-layer perceptron to improve the detection efficiency of the model at the next moment.

[0044] In the specific implementation process, the method of the present invention is an active learning method based on multi-centroid contrast autoencoder, such as Figure 2 As shown in the figure, firstly, the flow-level features of the captured network traffic are extracted, the features are one-hot encoded, and the benign and malicious traffic are clustered respectively by the kmeans algorithm, the traffic is subdivided and subclass labels are generated; then, the multi-centroid contrast autoencoder is used and the subclass labels are used to reduce the dimension of the feature vector to generate an embedded feature vector; then, the embedded feature vector is classified into two categories using a multi-layer perceptron and the confidence of the benign and malicious classes is output respectively. Then, based on the subclass centroid of the generated embedded feature vector and the generated confidence, the uncertainty samples are selected, and the selected samples are used to fine-tune the model to improve the detection efficiency of the model at the next moment.

[0045] In S2, the network traffic samples are converted into feature vectors, and the benign and malicious classes are sub-classified through unsupervised learning methods to generate sub-labels for the corresponding samples, while maintaining the time series relationship between network traffic data packets. Figure 3 As shown. Specifically, a network traffic feature extraction tool is used to extract flow-level statistical features from network traffic pcap and pcapng traffic packets. Next, a one-hot encoding tool is used to one-hot encode the statistical features and obtain an m-dimensional feature vector. Finally, the kmeans algorithm is used to cluster the feature vectors of the benign class N and the malicious class A respectively. The k value range in the kmeans algorithm is set to 2-20, and the silhouette coefficients corresponding to different k values ​​are calculated. The k value and subclass label sy corresponding to the maximum silhouette coefficient are saved, and the corresponding subclass labels N1, N2, N3, A1, A2, A3… are generated.

[0046] In order to more efficiently reduce the dimension of the feature vector and preserve the characteristic information of the sample, this method proposes a multi-centroid contrast autoencoder as the backbone network architecture for feature dimensionality reduction. The multi-centroid contrast autoencoder is used to reduce the dimension of the m-dimensional feature vector and obtain an n-dimensional embedded feature vector. The network layer of the encoder follows m-256-64-16-n, and the network layer of the decoder follows n-16-64-256-m. The neural network structure of the encoder has m nodes in the first layer, which is consistent with the dimension of the input m-dimensional feature vector, and then decreases successively to obtain n (m>>n) dimensional embedded features; the neural network structure of the decoder is exactly the opposite of the encoder. Because the main purpose of the autoencoder is to retain the important feature information of the sample through dimensionality reduction, the more similar the sample reconstructed by the decoder is to the original sample, the more it can be said that the embedded feature vector retains important feature information. In addition, dropout is added to each layer of the encoder to prevent overfitting. The loss function expression of the multi-centroid contrast autoencoder is as follows:

[0047] ,

[0048] in, represents the original sample characteristics, express The expected value of express The reconstructed sample features, is the loss coefficient, Representation sample and samples The expected value of , , , All samples and samples Category-related parameters, Representation sample and samples The distance between the embedded feature vectors, The + in the Greater than the preset When taking value, is 0, In Indicates when Greater than the preset When taking value, is 0. Specifically, when the sample and samples When they are benign and malicious, The value is 1, is 10, Representation sample and samples The distance between the embedded feature vectors is greater than 10. The value is 0. When the sample and samples When it is a benign or malicious class and belongs to different benign subclasses or malicious subclasses, The value is 1, otherwise, The value is 0, is 5, Representation sample and samples The distance between the embedded feature vectors is greater than 5. The value is 0. is the loss coefficient. The first term of the loss function is the reconstruction loss, which ensures that the loss between the original feature and the reconstructed feature is as small as possible. The second term of the loss function is the contrast loss between the benign class and the malicious class. During the training process, it ensures that the distance between samples in the benign class and the malicious class is small, the distance between different classes is large, and the loss coefficient is The third term of the loss function is the contrast loss between the same benign class or malicious class. When two samples are both benign or malicious, if they belong to the same subclass, the smaller the distance, the smaller the loss; otherwise, if they belong to different subclasses, the larger the distance, the smaller the loss, and the loss coefficient . It is beneficial to the detection process of the network traffic detection module.

[0049] In the network traffic detection task, the present invention uses a multi-layer perceptron to classify the embedded feature vector into two categories. During the training process, the multi-layer perceptron is trained with the embedded feature vector of the training set, and the label is the original label set y. During the detection process, the embedded feature vector of the test set is passed through the multi-layer perceptron to output the benign class confidence and the malicious class confidence respectively. The two confidences are compared and the class with high confidence is predicted.

[0050] In the drift adaptation task, the present invention combines the nearest centroid ratio and the classifier uncertainty method to screen out samples to be detected. The screened samples need to be manually labeled. After the manual labeling is completed, all samples are used to fine-tune the model and improve the detection effect of the model. In the specific detection process, the possibility of the sample to be detected being screened is specifically as follows: based on the benign and malicious classes predicted by the multi-layer perceptron, the centroids of each subclass of the benign and malicious classes are calculated respectively, and a benign class centroid closest to the sample to be detected and a malicious class centroid closest to the sample to be detected are found, and then the minimum ratio between the two closest distances is calculated; the confidence of the output of the sample to be detected is obtained by using the multi-layer perceptron to predict the uncertainty probability of the sample to be detected, and the minimum ratio is added to the uncertainty probability to form the selection possibility of the sample to be detected.

[0051] The selection possibility of the sample to be tested is expressed as follows:

[0052] ,

[0053] in, represents the distance between the sample to be detected and the nearest benign class centroid in the embedding space, represents the distance between the sample to be detected and the nearest malicious class centroid in the embedding space, It represents the embedded feature vector of the sample to be tested after passing through the multi-centroid contrast autoencoder. is the confidence of the embedded feature vector after passing through the multi-layer perceptron. The first term of the function is the nearest centroid ratio, which tries to select samples to be detected that hover between the benign class and the malicious class in the embedding space, which helps to further clarify the decision boundary of the model during fine-tuning. The second term of the function is the uncertainty probability, which selects samples that the multi-layer perceptron is uncertain about during classification.

[0054] Finally, the samples to be tested are sorted in descending order of selection probability, and the first N samples are combined with the training set to fine-tune the model. Try to use as little manual labeling overhead as possible to improve the detection efficiency of the model at the next moment.

[0055] Another embodiment is used to illustrate a network traffic detection system, such as Figure 4 As shown, the system 400 includes:

[0056] The network traffic preprocessing module 410 is used to extract flow-level features from the captured network traffic, perform one-hot encoding on the flow-level features to obtain a multi-dimensional feature vector, use the kmeans algorithm to cluster the multi-dimensional feature vector into benign traffic and malicious traffic, and generate benign subclass labels and malicious subclass labels;

[0057] A network traffic representation module 420, configured to reduce the dimension of the multi-dimensional feature vector using a multi-centroid contrast autoencoder to obtain an embedded feature vector;

[0058] The network traffic detection module 430 is used to use a multi-layer perceptron to perform binary classification on the embedded feature vector, obtain the benign class confidence and the malicious class confidence of the embedded feature vector, and determine the class corresponding to the high confidence as the predicted class;

[0059] The adaptation module 440 is used to screen the samples to be detected based on the subclass centroid and confidence of the embedded feature vector, and use the screened samples to be detected to optimize the multi-centroid contrast autoencoder and the multi-layer perceptron to improve the detection efficiency of the model at the next moment.

[0060] In the kmeans algorithm in the network traffic preprocessing module 410, the k value range is set to 2-20, the silhouette coefficients corresponding to different k values ​​are calculated, the k value corresponding to the maximum silhouette coefficient is saved, and the corresponding subclass label is generated.

[0061] The possibility of the sample to be detected in the adaptation module 440 being screened is specifically as follows: based on the benign class and malicious class predicted by the multi-layer perceptron, the centroids of each subclass of the benign class and the malicious class are calculated respectively, and a benign class centroid closest to the sample to be detected and a malicious class centroid closest to the sample to be detected are found, and then the minimum ratio between the two closest distances is calculated; the confidence of the output of the sample to be detected is predicted by the multi-layer perceptron to obtain the uncertainty probability of the sample to be detected, and the minimum ratio is added to the uncertainty probability to form the selection possibility of the sample to be detected.

[0062] In addition to the upper module, the system 400 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0063] The other specific working processes of the network traffic detection system 400 refer to the description of the above-mentioned basic network traffic detection method embodiment, which will not be repeated here.

[0064] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 5 The architecture of the computing device shown is implemented. Figure 5 The architecture of the computing device is shown. Figure 5 As shown, a computer system 510, a system bus 530, one or more CPUs 540, an input / output 520, a memory 550, etc. The memory 550 can store various data or files used for computer processing and / or communication and program instructions including the network traffic detection method of the embodiment executed by the CPU. Figure 5 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 5 One or more components in the system. The memory 550, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the network traffic detection method in the embodiment of the present invention (for example, the network traffic preprocessing module 410, the network traffic representation module 420, the network traffic detection module 430 and the adaptation module 440 in the network traffic detection system 400). One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 550, that is, to implement the above-mentioned network traffic detection method, which includes the following steps:

[0065] Extract flow-level features from the captured network traffic, perform one-hot encoding on the flow-level features to obtain a multidimensional feature vector, use the kmeans algorithm to cluster the multidimensional feature vector into benign traffic and malicious traffic, and generate benign subclass labels and malicious subclass labels;

[0066] Using a multi-centroid contrast autoencoder to reduce the dimension of the multi-dimensional feature vector to obtain an embedded feature vector;

[0067] Using a multi-layer perceptron to perform binary classification on the embedded feature vector, obtaining a benign class confidence and a malicious class confidence of the embedded feature vector, and determining the class corresponding to a high confidence as the predicted class;

[0068] The samples to be detected are screened based on the subclass centroid and confidence of the embedded feature vector, and the screened samples to be detected are used to optimize the multi-centroid contrast autoencoder and the multi-layer perceptron to improve the detection efficiency of the model at the next moment.

[0069] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the network traffic detection method provided in any embodiment of the present invention.

[0070] The memory 550 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 550 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include a memory remotely arranged relative to one or more CPUs 540, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0071] The input / output 520 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 520 may also include a display device such as a display screen.

[0072] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the network traffic detection method described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0073] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0074] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0075] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned network traffic detection method embodiment and will not be repeated here.

[0076] In summary, the technical solutions provided by the above embodiments, a network traffic detection method, system, electronic device and storage medium, the network traffic detection method, system, electronic device and storage medium provided by the embodiments of the present disclosure, the method of the present invention is an active learning method based on a multi-centroid contrast autoencoder, which can effectively reduce the dimension of network traffic features based on the multi-centroid contrast autoencoder, and screen samples that affect data distribution for fine-tuning, thereby improving the detection efficiency of the model; the backbone model of the multi-centroid contrast autoencoder is used to represent the network traffic features, and the statistical characteristics of the traffic are fully utilized to obtain a more effective feature representation. In addition, the method of the present invention proposes an effective sample collection method, which combines the nearest centroid ratio and uncertainty to directly screen out the sample set that affects the distribution difference, which is conducive to further fine-tuning. Further, the method of the present invention allows the distribution characteristics of network traffic to be continuously learned in a flexible manner by using multi-centroid contrast autoencoders, nearest centroid ratios and uncertainty sampling, without forgetting the learned distribution characteristics, thereby protecting the target network system from malicious attacks. In summary, the method of the present invention has good dimensionality reduction and representation capabilities for high-dimensional features, and has a continuous and efficient discrimination capability for normal traffic and malicious traffic.

[0077] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0078] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A network traffic detection method, characterized in that: The method comprises the following steps: Extract flow-level features from the captured network traffic, perform one-hot encoding on the flow-level features to obtain a multi-dimensional feature vector, use the kmeans algorithm to cluster the multi-dimensional feature vector into benign traffic and malicious traffic, and generate sub-class labels; Using a multi-centroid contrast autoencoder to reduce the dimension of the multi-dimensional feature vector to obtain an embedded feature vector; Using a multi-layer perceptron to perform binary classification on the embedded feature vector, obtaining a benign class confidence and a malicious class confidence of the embedded feature vector, and determining the class corresponding to a high confidence as the predicted class; Screening samples to be detected based on the subclass centroid and confidence of the embedded feature vector, and optimizing the multi-centroid contrast autoencoder and the multi-layer perceptron using the screened samples to be detected; The possibility of the sample to be detected being screened is specifically as follows: based on the benign class and malicious class predicted by the multi-layer perceptron, the centroids of each subclass of the benign class and the malicious class are calculated respectively, and a benign class centroid closest to the sample to be detected and a malicious class centroid closest to the sample to be detected are found, and then the minimum ratio between the two closest distances is calculated; the confidence of the output of the sample to be detected is predicted by the multi-layer perceptron to obtain the uncertainty probability of the sample to be detected, and the minimum ratio is added to the uncertainty probability to form the selection possibility of the sample to be detected; The selection possibility of the sample to be detected is expressed as follows: , in, represents the distance between the sample to be detected and the nearest benign class centroid in the embedding space, represents the distance between the sample to be detected and the nearest malicious class centroid in the embedding space, It represents the embedded feature vector of the sample to be tested after passing through the multi-centroid contrast autoencoder. It is the confidence of the output of the embedded feature vector after passing through the multi-layer perceptron.

2. The network traffic detection method according to claim 1, characterized in that: In the kmeans algorithm, the k value range is set to 2-20, the silhouette coefficients corresponding to different k values ​​are calculated, the k value corresponding to the maximum silhouette coefficient is saved, and the corresponding subclass labels are generated.

3. The network traffic detection method according to claim 1, characterized in that: The loss function expression of the multi-centroid contrast autoencoder is as follows: , in, represents the original sample characteristics, express The expected value of express The reconstructed sample features, is the loss coefficient, Representation sample and samples The expected value of , , , All samples and samples Category-related parameters, Representation sample and samples The distance between the embedded feature vectors, The + in the Greater than the preset When taking value, is 0, In Indicates when Greater than the preset When taking value, is 0.

4. A network traffic detection system, characterized in that: The system comprises: A network traffic preprocessing module is used to extract flow-level features from the captured network traffic, perform one-hot encoding on the flow-level features to obtain a multi-dimensional feature vector, use the kmeans algorithm to cluster the multi-dimensional feature vector into benign traffic and malicious traffic, and generate sub-class labels; A network traffic representation module, used for reducing the dimension of the multi-dimensional feature vector using a multi-centroid contrast autoencoder to obtain an embedded feature vector; A network traffic detection module, used to use a multi-layer perceptron to perform binary classification on the embedded feature vector, obtain the benign class confidence and the malicious class confidence of the embedded feature vector, and determine the class corresponding to the high confidence as the predicted class; An adaptation module, used for screening samples to be detected based on the subclass centroids and confidences of the embedded feature vectors, and optimizing the multi-centroid contrast autoencoder and the multi-layer perceptron using the screened samples to be detected; The possibility of the sample to be detected being screened in the adaptation module is specifically as follows: based on the benign class and malicious class predicted by the multi-layer perceptron, the centroids of each subclass of the benign class and the malicious class are calculated respectively, and a benign class centroid closest to the sample to be detected and a malicious class centroid closest to the sample to be detected are found, and then the minimum ratio between the two closest distances is calculated; the confidence of the output of the sample to be detected is predicted by the multi-layer perceptron to obtain the uncertainty probability of the sample to be detected, and the minimum ratio is added to the uncertainty probability to form the selection possibility of the sample to be detected; The selection possibility of the sample to be detected is expressed as follows: , in, represents the distance between the sample to be detected and the nearest benign class centroid in the embedding space, represents the distance between the sample to be detected and the nearest malicious class centroid in the embedding space, It represents the embedded feature vector of the sample to be tested after passing through the multi-centroid contrast autoencoder. It is the confidence of the output of the embedded feature vector after passing through the multi-layer perceptron.

5. The network traffic detection system according to claim 4, characterized in that: In the kmeans algorithm in the network traffic preprocessing module, the k value range is set to 2-20, the silhouette coefficients corresponding to different k values ​​are calculated, the k value corresponding to the maximum silhouette coefficient is saved, and the corresponding subclass labels are generated.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the network traffic detection method according to any one of claims 1 to 3 are implemented.

7. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the network traffic detection method according to any one of claims 1 to 3 are implemented.

Images