Network traffic processing method, apparatus, device, and storage medium

Abstract

The present disclosure relates to the technical field of computer, and particularly relates to a network traffic processing method and device, equipment and storage medium. The method comprises the following steps: obtaining real-time traffic characteristics corresponding to real-time sampling traffic, wherein the real-time sampling traffic is traffic data obtained by sampling the network traffic in real time; inputting the real-time traffic characteristics into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is obtained by training a sample feature set, and the sample feature set comprises at least one sample traffic characteristic; when the real-time sampling traffic is determined as abnormal traffic according to the detection result, updating the real-time traffic characteristics to the sample feature set, wherein the updated sample feature set is used for retraining the anomaly detection model. The present disclosure is used to solve the defects of low efficiency and low accuracy of network traffic detection in the prior art.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a network traffic processing method, apparatus, device, and storage medium. Background Art

[0002] With the continuous development of the internet, the volume of network interaction data has exploded, and at the same time, a significant amount of abnormal network traffic, such as fake order traffic, has also emerged. Existing technologies for network traffic detection first require computing a large amount of network traffic data. Based on this computational effort, the characteristics of the abnormal traffic can be derived to ultimately identify the abnormal traffic. This process requires that each piece of traffic data be highly dimensional. Consequently, when the system performs network traffic detection, the computational process consumes significant time and processing resources, leading to the risk of detecting anomalies and reducing the efficiency and accuracy of network traffic detection. Summary of the Invention

[0003] The present disclosure provides a network traffic processing method, apparatus, device and storage medium to address the defects of low efficiency and low accuracy of network traffic detection in the prior art.

[0004] The present disclosure provides a network traffic processing method, comprising: obtaining real-time traffic features corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; inputting the real-time traffic features into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is obtained by training a sample feature set, and the sample feature set includes at least one sample traffic feature; when it is determined according to the detection result that the real-time sampled traffic is abnormal traffic, updating the real-time traffic features to the sample feature set, wherein the updated sample feature set is used to retrain the anomaly detection model.

[0005] According to a network traffic processing method provided by the present disclosure, the anomaly detection model is obtained based on decision tree training; the training process of the anomaly detection model is as follows: obtaining a first information entropy of the sample feature set; for each of the sample traffic features in the sample feature set: dividing the sample feature set according to the sample traffic features to obtain a sample feature subset corresponding to the sample feature traffic; obtaining a second information entropy of the sample feature subset; obtaining an information gain corresponding to each of the sample traffic features based on the first information entropy and the second information entropy corresponding to each of the sample traffic features; and obtaining the anomaly detection model based on the decision tree based on the information gain.

[0006] According to a network traffic processing method provided by the present disclosure, the anomaly detection model based on the decision tree is obtained according to the information gain, including: for each of the sample traffic features in the sample feature set: obtaining a feature decision tree corresponding to the sample traffic feature; obtaining the number of leaf nodes of the feature decision tree; obtaining the feature selection probability corresponding to each of the sample traffic features based on the information gain and the number of leaf nodes respectively corresponding to each of the sample traffic features; calculating the difference between the feature selection probabilities respectively corresponding to any two of the sample traffic features; when it is determined that the difference is less than a difference threshold, fusing the two sample traffic features corresponding to the difference; and obtaining the anomaly detection model based on the fused sample traffic features.

[0007] According to a network traffic processing method provided by the present disclosure, the updating of the real-time traffic features to the sample feature set includes: adding the real-time traffic features as the new sample feature set to the sample feature set, wherein the sample feature set corresponds to a preset number of set samples; based on the sampling time corresponding to the sample traffic features, deleting the sample traffic features in the sample feature set that are the oldest from the current time one by one, until the real-time number of the sample traffic features in the sample feature set reaches the number of set samples, thereby obtaining the updated sample feature set.

[0008] According to a network traffic processing method provided by the present disclosure, before obtaining the first information entropy of the sample feature set, it also includes: based on the correlation between each of the sample traffic features in the sample feature set and the label feature, removing the sample traffic features whose correlation is lower than the correlation threshold to obtain a first sample feature set; deleting redundant features in the first sample feature set according to the decision tree corresponding to the removed sample feature set to obtain a second sample feature set; calculating the accurate influence corresponding to each of the sample traffic features in the second sample feature set; based on the accurate influence, selecting at least one optimal sample traffic feature in the second sample feature set to obtain an optimal feature set; updating the sample feature set through the optimal feature set, wherein the anomaly detection model is trained through the updated sample feature set.

[0009] According to a network traffic processing method provided by the present disclosure, the calculation of the accurate influence corresponding to each of the sample traffic features in the second sample feature set includes: based on the second sample feature set, obtaining the set accuracy of anomaly detection when describing network traffic; for each of the sample traffic features in the second sample feature set: removing the sample traffic feature from the second sample feature set to obtain a third sample feature set; based on the third sample feature set, obtaining the feature accuracy of anomaly detection when describing network traffic; and calculating the accurate influence corresponding to each of the sample traffic features based on the feature accuracy and the set accuracy.

[0010] According to a network traffic processing method provided by the present disclosure, based on the accurate influence, at least one optimal sample traffic feature in the second sample feature set is selected to obtain the optimal feature set, including: determining the sample traffic feature with the smallest accurate influence as the optimal sample traffic feature, and removing the optimal sample traffic feature from the second sample combination; repeatedly calculating the accurate influence corresponding to each sample traffic feature in the second sample feature set until the determined optimal sample traffic feature reaches a preset optimal number of samples, thereby obtaining the optimal feature set.

[0011] The present disclosure also provides a network traffic processing device, including: an acquisition module, used to obtain real-time traffic characteristics corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; a detection module, used to input the real-time traffic characteristics into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is obtained by training a sample feature set, and the sample feature set includes at least one sample traffic feature; an update module, used to update the real-time traffic characteristics to the sample feature set when determining that the real-time sampled traffic is abnormal traffic based on the detection result, wherein the updated sample feature set is used to retrain the anomaly detection model.

[0012] The present disclosure also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the network traffic processing methods described above is implemented.

[0013] The present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the network traffic processing method described above is implemented.

[0014] The network traffic processing method, apparatus, device, and storage medium provided by the present disclosure obtain the real-time traffic features corresponding to the real-time sampled traffic, input the real-time traffic features into a preset anomaly detection model, and the anomaly detection model outputs the detection results. When the real-time sampled traffic is determined to be abnormal traffic based on the detection results, the real-time traffic features are updated to a sample feature set, and the updated sample feature set is used to retrain the anomaly detection model. In this process, the real-time traffic features obtained by real-time sampling of the network traffic can realize real-time detection of the network traffic. At the same time, the real-time traffic features are processed by a preset anomaly detection model. The traffic is detected in real time by the anomaly detection model, avoiding the problem of low detection efficiency caused by parallel processing of a large amount of network traffic. In addition, if the real-time sampled traffic is abnormal traffic, the sample feature set is updated by the real-time traffic features and the anomaly detection model is retrained, thereby ensuring the accuracy of the anomaly detection model in detecting the ever-changing network traffic and further improving the efficiency of network detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 This is one of the flow charts of the network traffic processing method provided by the present disclosure;

[0017] Figure 2 This is a schematic diagram of the SDN network architecture provided by the present disclosure;

[0018] Figure 3 This is a schematic diagram of the sample feature set update principle provided by the present disclosure;

[0019] Figure 4 It is a schematic diagram of the network traffic processing architecture provided by the present disclosure;

[0020] Figure 5 This is the second flow chart of the network traffic processing method provided by the present disclosure;

[0021] Figure 6 It is a structural diagram of the network traffic processing device provided by the present disclosure;

[0022] Figure 7 It is a structural diagram of the electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the embodiments of the present disclosure.

[0024] The network traffic processing method provided by the present disclosure can be implemented by a software algorithm. The software algorithm implementing the method can be implemented on any device with data processing function, such as a server, a computer or a smart mobile device. Figure 1-Figure 5 The present invention describes a method for processing network traffic.

[0025] In one embodiment, if Figure 1 As shown, the process steps for implementing the network traffic processing method are as follows:

[0026] Step 101: Acquire real-time traffic characteristics corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time.

[0027] Obtain real-time traffic characteristics corresponding to real-time sampled traffic, where the real-time sampled traffic is traffic data obtained by sampling network traffic in real time.

[0028] In this embodiment, network traffic refers to network interaction data presented in the form of data streams. The network traffic is sampled in real time, and then feature extraction is performed on the collected real-time sampled traffic to obtain real-time traffic features.

[0029] In one embodiment, when extracting preliminary features from real-time sampled traffic, the preliminary traffic features obtained may contain redundant data or missing data due to noise interference and other factors during the extraction process. Therefore, after obtaining preliminary traffic features through real-time sampling, the preliminary traffic features can be preprocessed and data padded. Specifically, preprocessing can be a specific processing method such as filtering, and data padded can be a specific processing method such as interpolation. After preprocessing and data padded, the preliminary traffic features corresponding to the actual sampled traffic are obtained.

[0030] In one embodiment, the real-time traffic characteristics are represented in vector form to facilitate subsequent processing and calculation of the real-time traffic characteristics.

[0031] In one embodiment, network traffic detection can be achieved under various network architectures. For example, the network architecture of Software Defined Network (SDN) breaks through the bottlenecks of poor scalability and high management difficulty of traditional networks, and can achieve flexible management of the network. At the same time, the SDN network architecture realizes the decoupling of data forwarding and control. The SDN network is divided into a three-layer architecture: application plane, control plane, and data plane. The process of network traffic detection and processing is implemented in the application plane of the SDN network architecture.

[0032] Specifically, such as Figure 2 As shown in Figure 1, an SDN network architecture consists of an application plane, a control plane, and a data plane. In an SDN architecture, the centralized control plane and the distributed data plane are separated. The SDN control plane collects real-time network status through the southbound interface, exposes it, and notifies upper-layer applications in the application plane through the northbound interface. Furthermore, through the southbound and northbound interfaces, it translates upper-layer applications into lower-level rules or device hardware instructions (such as flow table installation or deletion) and distributes them to network devices in the data plane. The southbound interface complies with the Open Flow protocol.

[0033] When implementing network traffic processing in the application plane, an SDN anomaly detection system can be pre-configured. First, the system collects and processes data through northbound and southbound interfaces, samples network traffic transmitted on the data plane, parses the collected data packets, and extracts features to obtain real-time traffic features. These real-time traffic features are then represented as vectors. Anomalies are detected on the real-time traffic features using an anomaly detection model. If the detection results indicate that the real-time sampled traffic is anomalous, the anomalous traffic is discarded, undergoing other anomaly processing. Simultaneously, based on the anomalous real-time traffic features, the anomaly detection model is retrained, and the next extracted real-time traffic features are then detected using the retrained anomaly detection model.

[0034] In step 102, the real-time traffic feature is input into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is trained by a sample feature set, and the sample feature set includes at least one sample traffic feature.

[0035] In this embodiment, the anomaly detection model is used to detect real-time feature traffic and output detection results. These detection results include normal traffic and abnormal traffic. Furthermore, when the real-time sampled traffic is abnormal, a pre-trained anomaly category can also be output. The anomaly detection model must be pre-trained using a sample feature set, which includes at least one sample traffic feature. Furthermore, the sample feature set can also include an anomaly category label corresponding to each sample traffic feature.

[0036] In one embodiment, the anomaly detection model is obtained based on decision tree training. Specifically, the anomaly detection model training process is as follows: obtaining a first information entropy of a sample feature set; for each sample flow feature in the sample feature set: dividing the sample feature set by the sample flow feature to obtain a sample feature subset corresponding to the sample feature flow; obtaining a second information entropy of the sample feature subset; based on the first information entropy and the second information entropy corresponding to each sample flow feature, obtaining an information gain corresponding to each sample flow feature; and based on the information gain, obtaining a decision tree-based anomaly detection model.

[0037] In this embodiment, the decision tree is a common machine learning method and a very commonly used classification method. It is a supervised learning method. Commonly used decision tree models include the decision tree induction algorithm (ID3) model and the data mining algorithm (C4.5) model. Due to the discretization of network traffic under the SND network architecture, the ID3 algorithm is preferably used to construct an anomaly detection model. The ID3 algorithm uses the rate of decrease of information entropy as the criterion for selecting test attributes, that is, at each node, the attribute with the highest information gain that has not yet been used for division is selected as the division criterion, and then this process is continued until the generated decision tree can perfectly classify the training samples.

[0038] Furthermore, the ID3 algorithm selects the sample flow feature with the largest information gain based on continuous iteration to divide the spanning tree. Any sample flow feature f i The corresponding information gain Gain(T,f i ) is calculated as follows:

[0039]

[0040] Where T represents the sample feature set; f i represents the i-th sample traffic feature in the sample feature set T, where i is a positive integer; E(T) represents the first information entropy, i.e., the information entropy of the sample feature set T; Represents the second information entropy, that is, the sample flow feature f i Divide the sample feature set T and obtain the sample feature flow f iAfter dividing at least one sample feature subset, the information entropy of each sample feature subset.

[0041] When the difference between the first information entropy and the second information entropy is larger, the information gain Gain(T,f i ) is larger, indicating that the sample flow characteristic f i The greater the role it plays in the set partitioning process.

[0042] In one embodiment, due to the discrete nature of network traffic characteristics in networks such as SDN network architectures, the leaf nodes of the decision tree may be too large during the partitioning process. Therefore, when obtaining an anomaly detection model based on a decision tree based on information gain, for each sample traffic feature in the sample feature set: a feature decision tree corresponding to the sample traffic feature is obtained; the number of leaf nodes in the feature decision tree is obtained; based on the information gain and the number of leaf nodes corresponding to each sample traffic feature, the feature selection probability corresponding to each sample traffic feature is obtained; the difference between the feature selection probabilities corresponding to any two sample traffic features is calculated; when the difference is determined to be less than a difference threshold, the two sample traffic features corresponding to the difference are fused; and based on the fused sample traffic features, an anomaly detection model is obtained.

[0043] In this embodiment, the feature selection probability of the sample traffic features is used to determine whether there is similarity in the ability to distinguish categories, so that similar sample traffic features are merged to reduce the number of leaf nodes.

[0044] Specifically, the sample flow characteristics f i The corresponding feature selection probability attrSele(T,f i ) is calculated as follows:

[0045]

[0046] Among them, Gain(T,f i ) represents the sample flow characteristics f i The corresponding information gain, Lnum(T,f i ) represents the sample traffic feature f i The number of leaf nodes in the generated decision tree.

[0047] In the decision tree division process, for two different sample flow features f in the sample feature set T i And sample traffic characteristics f j , where j is a positive integer, i≠j, if the sample flow characteristic f i The corresponding information gain Gain(T,f i ), and the sample flow characteristics f j The corresponding information gain Gain(T,f j) are the same, then the sample flow characteristics f are calculated separately i The corresponding feature selection probability attrSele(T,f i ), and sample traffic characteristics f j The corresponding feature selection probability attrSele(T,f j ). From the above formula (2), we can see that when the numerator is the same, the smaller the denominator is, the larger the value of the feature selection probability is, and the greater the probability that the corresponding sample flow feature is selected is.

[0048] If attrSele(T,f i ) and attrSele(T,f j ) is similar, it means that the sample flow characteristics f i And sample traffic characteristics f j If the ability to distinguish categories is similar, the two features can be fused to obtain the fused sample traffic feature u. The details are as follows:

[0049] u=f i +f j (3);

[0050] Among them, u, f i and f j All of them can be represented by vectors, so the fusion process can be completed through calculation.

[0051] Furthermore, the sample flow characteristics f i And sample traffic characteristics f j When comparing the similarity of the ability to distinguish categories, the sample flow feature f i And sample traffic characteristics f j The difference is compared with the preset difference threshold a. Specifically, if the following conditions are met:

[0052] attrSele(T,f i )-attrSele(T,f j )

[0053] Then the sample flow characteristic f i And sample traffic characteristics f j Feature fusion can be performed.

[0054] It should be noted that the difference threshold a can be set according to actual conditions and needs. For example, the difference threshold a is set to 0.06. The scope of protection of the present disclosure is not limited to the specific value of the difference threshold.

[0055] ​In one embodiment, in order to further improve the effect of model training, feature selection is performed on the sample traffic features in the sample feature set. Specifically, before obtaining the first information entropy of the sample feature set, based on the correlation between each sample traffic feature in the sample feature set and the label feature, the sample traffic features with correlations lower than the correlation threshold are removed to obtain a first sample feature set; according to the decision tree corresponding to the removed sample feature set, the redundant features in the first sample feature set are deleted to obtain a second sample feature set; the accurate influence corresponding to each sample traffic feature in the second sample feature set is calculated; based on the accurate influence, at least one optimal sample traffic feature in the second sample feature set is selected to obtain an optimal feature set; the sample feature set is updated using the optimal feature set, wherein the anomaly detection model is trained using the updated sample feature set.

[0056] In this embodiment, for each sample flow feature in the sample feature set, a correlation analysis is first performed with the label feature. For any sample flow feature in the sample feature set, when the similarity with the preset label feature reaches a preset similarity threshold, the sample flow feature is removed from the sample feature set. More specifically, the main purpose of this process is to find the feature subset that is most relevant to the label. Preferably, mutual information is used to measure the similarity. Mutual information is an important indicator for measuring feature correlation, which represents a nonlinear estimate of feature correlation. Symmetric uncertainty is obtained by normalizing the mutual information to the entropy between feature values, or the entropy between the feature and the target class. It compensates for the deviation caused by information gain for features with a large number of different values.

[0057] In this embodiment, redundant features in the first sample feature set are deleted based on the decision tree corresponding to the removed sample feature set. Specifically, the calculation of redundant features is simplified using a minimum decision tree. Preferably, the redundancy between any two sample traffic features can be measured by the mutual information value, thereby determining whether any two sample traffic features are redundant, thereby obtaining the second sample feature set.

[0058] In this embodiment, based on the accuracy influence, at least one optimal sample flow feature in the second sample feature set is selected to obtain an optimal feature set.

[0059] In one embodiment, the accurate impact corresponding to each sample traffic feature in the second sample feature set is calculated. The specific implementation process is as follows: based on the second sample feature set, the set accuracy of anomaly detection when describing network traffic is obtained; for each sample traffic feature in the second sample feature set: the sample traffic feature is removed from the second sample feature set to obtain a third sample feature set; based on the third sample feature set, the feature accuracy of anomaly detection when describing network traffic is obtained; based on the feature accuracy and the set accuracy, the accurate impact corresponding to each sample traffic feature is calculated.

[0060] In this embodiment, based on the degree of influence of each sample traffic feature on the anomaly detection performance, network optimization is performed on the feature subset to obtain the optimal feature set for better detection of abnormal traffic. Selecting more sample traffic features can further improve the efficiency and accuracy of anomaly detection, which is particularly important for anomaly detection.

[0061] In this embodiment, the entire sample feature set is first used as a standard for measuring feature influence, and the sample feature set used as the standard is denoted as F′. For any sample flow feature f in the sample feature set, i , the impact on the accuracy of traffic identification I(f i ), the calculation formula is as follows:

[0062]

[0063] Among them, P F′ It represents the set accuracy, that is, the accuracy of anomaly detection when the sample feature set F′ is used to describe the network traffic; Indicates the removal of sample flow feature f i The feature accuracy after f i The accuracy of anomaly detection when the removed sample feature set describes network traffic.

[0064] It can be seen from the above formula (5) that the greater the accuracy influence, the smaller the influence of the feature on the accuracy of traffic data recognition.

[0065] In one embodiment, after calculating the accurate influence corresponding to each sample flow feature, at least one optimal sample flow feature in the second sample feature set is selected based on the accurate influence to obtain the optimal feature set. The specific process is as follows: the sample flow feature with the smallest accurate influence is determined as the optimal sample flow feature, and the optimal sample flow feature is removed from the second sample combination; the step of repeatedly calculating the accurate influence corresponding to each sample flow feature in the second sample feature set is repeated until the determined optimal sample flow feature reaches the preset optimal number of samples to obtain the optimal feature set.

[0066] In this embodiment, based on the accuracy influence, feature selection is continuously performed from the sample feature set. When the optimal sample flow feature is determined from the second sample feature set for the first time, the sample flow feature with the smallest accuracy influence is determined as the optimal sample flow feature, that is, the first selected sample flow feature is min[I(f i )], add the optimal sample flow feature to the optimal feature set F z The optimal sample flow feature is removed from the second sample combination, and then the accurate influence corresponding to each of the remaining sample flow features in the second sample feature set is calculated again, and the sample flow feature with the smallest accurate influence is again determined as the optimal sample flow feature, and so on.

[0067] Furthermore, with the optimal feature set F z The number of features in the algorithm will continue to iterate and increase, and the detection accuracy of traffic anomalies will show a trend of rising and then falling. Therefore, the optimal feature set F is calculated. z When there are n sample traffic features, calculate the corresponding abnormal traffic detection accuracy, which is recorded as C n If the feature C obtained later n >C n+1 >C n+2 When the optimal feature set F z When there are n sample traffic features in , it is considered that the required optimal feature set has been obtained and the system selection is stopped. At this time, the number of features determined by feature selection is n.

[0068] Step 103: When it is determined according to the detection result that the real-time sampled traffic is abnormal traffic, the real-time traffic features are updated to the sample feature set, wherein the updated sample feature set is used to retrain the anomaly detection model.

[0069] In this embodiment, the detection results output by the anomaly detection model are used to determine whether the real-time sampled traffic is abnormal. If it is not abnormal, the network traffic is resampled. If it is abnormal, two aspects are simultaneously processed: first, the abnormal traffic is eliminated and intercepted, and second, the real-time traffic features corresponding to the abnormal traffic are updated to the sample feature set to retrain the anomaly detection model. Through this retraining process, the anomaly detection model continuously learns the characteristics of the latest abnormal traffic, and can better adapt to real-time changes in network traffic.

[0070] In this embodiment, the process of retraining the anomaly detection model is the same as the training process provided in the above embodiment, except that the specific features included in the sample feature set are different, which will not be repeated here.

[0071] In one embodiment, in order to further reduce the system overhead, the real-time traffic features are updated to the sample feature set. The specific implementation process is as follows: the real-time traffic features are added to the sample feature set as a new sample feature set, where the sample feature set corresponds to a preset number of set samples; based on the sampling time corresponding to the sample traffic features, the sample traffic features in the sample feature set that are the oldest from the current time are deleted one by one until the real-time number of sample traffic features in the sample feature set reaches the number of set samples, and the updated sample feature set is obtained.

[0072] In this embodiment, as the network traffic detection process continues, more and more abnormal traffic will be detected. If the sample feature set is not processed, the number of sample traffic features in the sample feature set will increase, which will lead to more and more resources being occupied in the process of retraining the anomaly detection model. Figure 3 As shown, Figure 3 The circles in the figure represent the sampling points corresponding to the sample flow characteristics, arranged in chronological order. Black circles represent sampling points where anomalies actually occurred. A set sample count is pre-set based on needs and actual conditions. A single sampling operation can collect real-time flow samples from one or more sampling points. If b sampling points were collected during the most recent sampling at time t, and the pre-set set sample count is b + r, the sample flow sets corresponding to the earliest b sampling points are deleted to ensure that the number of features in the sample feature set remains at b + r, thus preventing the sample feature set from becoming too large.

[0073] In one embodiment, when based on the SDN network architecture, the process of network traffic processing is based on the following Figure 4 The network traffic processing architecture is implemented. Specifically, when processing network traffic, the switches in the data plane and the controllers in the control plane sample and extract the features in the flow table items in real time, collect traffic information, and preliminarily construct the sampling traffic features in the form of vectors to obtain the detection vector V′. The detection vector V′ is subjected to data preprocessing to obtain the vector corresponding to the real-time sampling features. The real-time sampling features are input into the anomaly detection model to determine whether the real-time sampling traffic is abnormal traffic. If not, the flow table entries are directly sent down through the controller to execute the flow rules. If so, the sample feature set required for the anomaly detection model training is updated. After the feature selection of the sample feature set is performed based on correlation and influence (FSCI) algorithm, the anomaly detection model is retrained, and the basic data of the trained anomaly detection model is saved. Then, the original anomaly detection model is updated through the retrained anomaly detection model.

[0074] In this embodiment, in order to ensure the stable operation of the system and reduce the CPU consumption of the machine, when processing network traffic, only when abnormal traffic is detected will the exception processing module be started to eliminate and process the abnormal data, and the anomaly detection model will be retrained at the same time, which can reduce the burden and consumption of the processing process.

[0075] In one embodiment, if Figure 5 As shown in the figure, based on the SDN network architecture, the SDN network traffic processing steps are as follows:

[0076] Step 501, real-time traffic collection;

[0077] Step 502: Data preprocessing. Extract and encode the real-time sampled traffic, and preprocess and fill the initially extracted traffic feature data.

[0078] Step 503: SDN network traffic representation, i.e., constructing a corresponding vector based on the real-time traffic features obtained after preprocessing;

[0079] Step 504: Anomaly detection. Input the real-time traffic characteristics into a preset anomaly detection model to obtain the detection results output by the anomaly detection model.

[0080] Step 505, determining whether the real-time sampling flow is abnormal flow based on the detection result, if so, executing step 506, if not, executing step 501;

[0081] Step 506: Exception handling, retrain the anomaly detection model, and execute step 501.

[0082] Specifically, the attack source address is imported into the exception handling module, abnormal traffic is processed, and data containing attack sources or abnormal traffic is discarded to prevent abnormal network data from entering normal servers. At the same time, feature selection and model update are started to update the sample feature set, and the anomaly detection model is retrained. The retrained model data is updated to the anomaly detection model to form a new anomaly detection model. After the anomaly processing is completed, the network traffic can be collected again.

[0083] Through the above steps, the anomaly detection model can be continuously updated to better intercept new abnormal traffic, thus achieving a complete abnormal traffic detection process. Construct an SDN-based abnormal traffic detection system to detect the collected traffic, then output the detection results and issue alarms, so that administrators can take appropriate measures to manage the network.

[0084] The network traffic processing method provided by the present disclosure obtains the real-time traffic features corresponding to the real-time sampled traffic, inputs the real-time traffic features into a preset anomaly detection model, and the anomaly detection model outputs the detection results. When the real-time sampled traffic is determined to be abnormal traffic based on the detection results, the real-time traffic features are updated to the sample feature set, and the updated sample feature set is used to retrain the anomaly detection model. In this process, the real-time traffic features obtained by real-time sampling of the network traffic can realize real-time detection of the network traffic. At the same time, the real-time traffic features are processed by the preset anomaly detection model. The traffic is detected in real time by the anomaly detection model, avoiding the problem of low detection efficiency caused by parallel processing of a large amount of network traffic. In addition, if the real-time sampled traffic is abnormal traffic, the sample feature set is updated by the real-time traffic features, and the anomaly detection model is retrained, thereby ensuring the accuracy of the anomaly detection model in detecting the ever-changing network traffic and further improving the efficiency of network detection.

[0085] Furthermore, the method provided by this disclosure can achieve early interception and exception handling of abnormal traffic, reducing the system's concurrency and ensuring system stability when facing large amounts of abnormal traffic. At the same time, it provides an anomaly detection model that changes with the increasing number of network attack types and the changing abnormal network traffic. Specifically, it utilizes semantic feature extraction of network traffic to implement the network anomaly detection model, ensuring the system's accuracy in detecting constantly changing abnormal traffic, making it more suitable for high-traffic systems such as e-commerce.

[0086] The network traffic processing device provided by the embodiment of the present disclosure is described below. The network traffic processing device described below and the network traffic processing method described above can be referred to in correspondence with each other. Figure 7 As shown, the network traffic processing device includes:

[0087] An acquisition module 601 is configured to acquire real-time traffic characteristics corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time;

[0088] Detection module 602, configured to input real-time traffic features into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is trained using a sample feature set, the sample feature set including at least one sample traffic feature;

[0089] The updating module 603 is used to update the real-time traffic features to the sample feature set when it is determined that the real-time sampled traffic is abnormal traffic according to the detection result, wherein the updated sample feature set is used to retrain the abnormality detection model.

[0090] In one embodiment, the network traffic processing device further includes a training module 604 for training an anomaly detection model, and the process is as follows: obtaining a first information entropy of a sample feature set; for each sample traffic feature in the sample feature set: dividing the sample feature set by the sample traffic feature to obtain a sample feature subset corresponding to the sample feature traffic; obtaining a second information entropy of the sample feature subset; obtaining an information gain corresponding to each sample traffic feature based on the first information entropy and the second information entropy corresponding to each sample traffic feature; obtaining an anomaly detection model based on a decision tree based on the information gain. The anomaly detection model is obtained based on decision tree training;

[0091] In one embodiment, the training module 604 is used to, for each sample flow feature in the sample feature set: obtain a feature decision tree corresponding to the sample flow feature; obtain the number of leaf nodes of the feature decision tree; obtain the feature selection probability corresponding to each sample flow feature based on the information gain and the number of leaf nodes corresponding to each sample flow feature; calculate the difference in the feature selection probabilities corresponding to any two sample flow features; when it is determined that the difference is less than a difference threshold, fuse the two sample flow features corresponding to the difference; and obtain an anomaly detection model based on the fused sample flow features.

[0092] In one embodiment, the update module 603 is used to add the real-time traffic features as a new sample feature set to the sample feature set, where the sample feature set corresponds to a preset number of set samples; based on the sampling time corresponding to the sample traffic features, the sample traffic features in the sample feature set that are the oldest from the current time are deleted one by one until the real-time number of sample traffic features in the sample feature set reaches the number of set samples, thereby obtaining an updated sample feature set.

[0093] In one embodiment, the training module 604, before obtaining the first information entropy of the sample feature set, also includes: based on the correlation between each sample flow feature in the sample feature set and the label feature, removing the sample flow features whose correlation is lower than the correlation threshold to obtain the first sample feature set; deleting the redundant features in the first sample feature set according to the decision tree corresponding to the removed sample feature set to obtain the second sample feature set; calculating the accurate influence corresponding to each sample flow feature in the second sample feature set; based on the accurate influence, selecting at least one optimal sample flow feature in the second sample feature set to obtain the optimal feature set; updating the sample feature set through the optimal feature set, wherein the anomaly detection model is trained through the updated sample feature set.

[0094] In one embodiment, the training module 604 is used to obtain the set accuracy of anomaly detection when describing network traffic based on the second sample feature set; for each sample traffic feature in the second sample feature set: remove the sample traffic feature from the second sample feature set to obtain a third sample feature set; based on the third sample feature set, obtain the feature accuracy of anomaly detection when describing network traffic; based on the feature accuracy and the set accuracy, calculate the accurate impact corresponding to each sample traffic feature.

[0095] In one embodiment, the training module 604 is used to determine the sample flow feature with the smallest accuracy influence as the optimal sample flow feature, and remove the optimal sample flow feature from the second sample combination; repeatedly calculate the steps of the accuracy influence corresponding to each sample flow feature in the second sample feature set until the determined optimal sample flow feature reaches the preset optimal number of samples, thereby obtaining the optimal feature set.

[0096] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other via the communication bus 704. The processor 701 may call the logic instructions in the memory 703 to execute a network traffic processing method, which includes: obtaining real-time traffic features corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; inputting the real-time traffic features into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is obtained by training a sample feature set, and the sample feature set includes at least one sample traffic feature; when the real-time sampled traffic is determined to be abnormal traffic according to the detection result, the real-time traffic features are updated to the sample feature set, wherein the updated sample feature set is used to retrain the anomaly detection model.

[0097] In addition, the logic instructions in the above-mentioned memory 703 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0098] On the other hand, the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the network traffic processing method provided by the above methods, and the method includes: obtaining real-time traffic characteristics corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; inputting the real-time traffic characteristics into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is obtained by training a sample feature set, and the sample feature set includes at least one sample traffic feature; when the real-time sampled traffic is determined to be abnormal traffic according to the detection result, the real-time traffic characteristics are updated to the sample feature set, wherein the updated sample feature set is used to retrain the anomaly detection model.

[0099] On the other hand, the present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned network traffic processing methods, the methods comprising: obtaining real-time traffic features corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; inputting the real-time traffic features into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is obtained by training a sample feature set, and the sample feature set includes at least one sample traffic feature; when the real-time sampled traffic is determined to be abnormal traffic according to the detection result, the real-time traffic features are updated to the sample feature set, wherein the updated sample feature set is used to retrain the anomaly detection model.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0101] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A network traffic processing method, characterized in that: include: Acquire real-time traffic characteristics corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; Inputting the real-time traffic feature into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is trained by a sample feature set, and the sample feature set includes at least one sample traffic feature; When it is determined according to the detection result that the real-time sampled traffic is abnormal traffic, updating the real-time traffic feature to the sample feature set, wherein the updated sample feature set is used to retrain the abnormality detection model; The anomaly detection model is obtained based on decision tree training; The training process of the anomaly detection model is as follows: Obtaining a first information entropy of the sample feature set; For each of the sample flow features in the sample feature set: dividing the sample feature set according to the sample flow feature to obtain a sample feature subset corresponding to the sample feature flow; obtaining a second information entropy of the sample feature subset; Obtaining information gain corresponding to each of the sample flow features according to the first information entropy and the second information entropy corresponding to each of the sample flow features; Acquire the anomaly detection model based on the decision tree according to the information gain; The step of obtaining the anomaly detection model based on the decision tree according to the information gain includes: For each of the sample flow features in the sample feature set: obtaining a feature decision tree corresponding to the sample flow feature; obtaining the number of leaf nodes of the feature decision tree; Based on the information gain and the number of leaf nodes corresponding to each of the sample flow features, obtaining a feature selection probability corresponding to each of the sample flow features; Calculating the difference between the feature selection probabilities corresponding to any two of the sample flow features; When it is determined that the difference is less than the difference threshold, fusing the two sample flow features corresponding to the difference; The anomaly detection model is obtained based on the fused sample traffic features.

2. The network traffic processing method according to claim 1, characterized in that: The updating of the real-time traffic feature to the sample feature set includes: Adding the real-time traffic feature as a new sample feature set to the sample feature set, wherein the sample feature set corresponds to a preset number of set samples; Based on the sampling time corresponding to the sample flow feature, the sample flow features that are the oldest from the current time in the sample feature set are deleted one by one until the real-time number of the sample flow features in the sample feature set reaches the number of set samples, thereby obtaining the updated sample feature set.

3. The network traffic processing method according to claim 1, wherein: Before obtaining the first information entropy of the sample feature set, the method further includes: Based on the correlation between each of the sample flow features in the sample feature set and the label feature, the sample flow features having a correlation lower than a correlation threshold are removed to obtain a first sample feature set; Deleting redundant features in the first sample feature set according to the decision tree corresponding to the removed sample feature set to obtain a second sample feature set; Calculating the accurate influence corresponding to each of the sample traffic features in the second sample feature set; Based on the accurate influence, selecting at least one optimal sample flow feature in the second sample feature set to obtain an optimal feature set; The sample feature set is updated using the optimal feature set, wherein the anomaly detection model is trained using the updated sample feature set.

4. The network traffic processing method according to claim 3, characterized in that: The calculating of the accurate influence corresponding to each of the sample traffic features in the second sample feature set includes: Based on the second sample feature set, obtaining a set accuracy of anomaly detection when describing network traffic; For each of the sample traffic features in the second sample feature set: removing the sample traffic feature from the second sample feature set to obtain a third sample feature set; and obtaining feature accuracy for anomaly detection when describing network traffic based on the third sample feature set; Based on the feature accuracy and the set accuracy, the accurate influence corresponding to each of the sample traffic features is calculated.

5. The network traffic processing method according to claim 3, characterized in that: The selecting, based on the accurate influence, at least one optimal sample flow feature in the second sample feature set to obtain an optimal feature set includes: determining the sample flow feature with the smallest accuracy influence as the optimal sample flow feature, and removing the optimal sample flow feature from the second sample combination; Repeat the step of calculating the accurate influence corresponding to each of the sample flow characteristics in the second sample feature set until the determined optimal sample flow characteristics reach a preset optimal number of samples, thereby obtaining the optimal feature set.

6. A network traffic processing device, characterized in that: include: An acquisition module, configured to acquire real-time traffic characteristics corresponding to real-time sampled traffic, wherein the real-time sampled traffic is traffic data obtained by sampling network traffic in real time; a detection module, configured to input the real-time traffic feature into a preset anomaly detection model to obtain a detection result output by the anomaly detection model, wherein the anomaly detection model is trained by a sample feature set, and the sample feature set includes at least one sample traffic feature; an updating module, configured to update the real-time traffic feature to the sample feature set when determining that the real-time sampled traffic is abnormal traffic according to the detection result, wherein the updated sample feature set is used to retrain the abnormality detection model; The training module is used to train the anomaly detection model. The process is as follows: obtain the first information entropy of the sample feature set; for each sample flow feature in the sample feature set: divide the sample feature set by the sample flow feature to obtain the sample feature subset corresponding to the sample feature flow; obtain the second information entropy of the sample feature subset; based on the first information entropy and the second information entropy corresponding to each sample flow feature, obtain the information gain corresponding to each sample flow feature; based on the information gain, obtain the anomaly detection model based on the decision tree. The anomaly detection model is obtained based on the decision tree training. The training module is also used to obtain, for each sample flow feature in the sample feature set: a feature decision tree corresponding to the sample flow feature; obtain the number of leaf nodes of the feature decision tree; obtain the feature selection probability corresponding to each sample flow feature based on the information gain and the number of leaf nodes corresponding to each sample flow feature; calculate the difference in the feature selection probabilities corresponding to any two sample flow features; when it is determined that the difference is less than a difference threshold, fuse the two sample flow features corresponding to the difference; and obtain an anomaly detection model based on the fused sample flow features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the network traffic processing method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the network traffic processing method according to any one of claims 1 to 5 is implemented.

Images