A network traffic data enhancement method for sample imbalance
By clustering the network traffic data set to calculate the sparse factor and difference weight, and generating new samples with SMOTE algorithm, the sample imbalance problem is solved, the data set quality and model generalization ability are improved, and the data enhancement effect is improved.
Patent Information
- Application Number
- CN202210325598.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-30
AI Technical Summary
When processing network traffic data in the prior art, the sample imbalance problem leads to insufficient generalization capabilities of the model. Oversampling methods such as SMOTE and Borderline SMOTE algorithms have insufficient sample distribution sparseness and boundary sample information enhancement, resulting in classification accuracy and data synthesis. Not high quality.
By performing K-Means clustering on the original network traffic dataset, the sparse factors between the minority class samples and the difference between the majority class samples are calculated, the initial weight is assigned, and new samples are synthesized based on SMOTE, and the data enhancement process is optimized to improve the data quality.
Effectively select samples at sparse distribution and boundary, improve the quality of data sets and model generalization capabilities, and improve the data enhancement effect.
Smart Images

Figure CN114781492B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of traffic data enhancement, and in particular to a network traffic data enhancement method for sample imbalance. Background Art
[0002] Artificial intelligence refers to the use of certain algorithms to enable machines to simulate human intelligence or even surpass human intelligence, while machine learning represents the process of using algorithms to guide computers to use known data to derive appropriate models and use this model to make judgments on new situations. So far, machine learning methods have been widely used in network intrusion detection. Throughout the entire machine learning process, the most important thing is data. Whether a machine learning model is robust and has high generalization ability is inseparable from the quality of the training data set.
[0003] In the era of big data, the statistics and analysis of network traffic data have become more important. Traffic data collected in real network environments in real time can be used for intrusion identification with the help of machine learning methods. The sample quality of the network traffic data set will directly determine the performance of identification. Network traffic data is difficult to obtain and most data often have little difference from each other, so there is often a serious imbalance in data categories. Oversampling data enhancement methods are a common method for solving the problem of sample imbalance. This type of method samples minority class samples to increase the number of minority class samples, and ultimately achieves the effect of balancing the majority class and minority class in network traffic data. A large number of existing oversampling methods are based on the SMOTE algorithm, which directly uses random sampling to select samples, and sparsely distributed samples are easily ignored. In addition, samples located at the boundary have a higher misclassification rate, and their information also needs to be enhanced.
[0004] Compared with general image and language text data, network traffic data is difficult to obtain and most of the data is often very small. Therefore, there is often a serious imbalance in data categories, which ultimately makes the generalization ability of the trained model insufficient. Data augmentation is a feasible solution to the above problem. Its essence is to increase the data in the data set by introducing prior knowledge, thereby improving the generalization ability of the model. Common data augmentation methods used to improve category imbalance include undersampling methods, oversampling methods, data set expansion, cost-sensitive learning, etc. Among them, oversampling is a relatively effective and fast method to solve the problem of unbalanced network traffic data samples. The representative algorithms are SMOTE and ADASYN. Its essence is to increase the number of minority class data samples by sampling minority class samples, and finally achieve the effect of balancing the majority class and minority class in network traffic data.
[0005] In order to understand the development status of the prior art, the existing patents and documents were searched, compared and analyzed, and the following technical information with high relevance to the present invention was screened out:
[0006] Patent Solution 1: CN112036515A provides an oversampling method, device and electronic device based on the SMOTE algorithm. The method includes: obtaining a historical sample data set, determining positive and negative samples and their corresponding numbers; determining majority class sample data and minority class sample data, and performing data vectorization processing; using a divergent point monitoring method to filter target sample data from the minority class sample data set; based on the SMOTE algorithm, oversampling the target sample data to generate a specific number of new sample data; and obtaining an expanded minority class sample data set based on the generated new sample data and the original minority class sample data. While optimizing the sampling method, this method solves the problem of data imbalance, improves the accuracy of model prediction, and effectively reduces the deviation introduced by data imbalance. Disadvantages: This solution is based on the traditional SMOTE algorithm for data synthesis, but it is worth noting that in the clustering results, samples at and near the boundaries are usually more likely to be misclassified than samples far from the boundaries. Therefore, before data synthesis, more attention should be paid to those minority class samples at and near the boundaries to enhance their information.
[0007] Patent Solution 2: CN111832664A, a method for equalizing and diagnosing fault samples of power transformers based on Borderline SMOTE, discloses a method for equalizing and diagnosing fault samples of power transformers based on Borderline SMOTE. The equalization method includes the steps of searching for minority samples, classifying minority samples, and generating new samples. The fault diagnosis method also includes a fault diagnosis step. This method increases the number of minority samples near the boundary samples, reduces the misjudgment rate of boundary samples, and thus improves the classification accuracy. It is suitable for the processing of non-balanced data sets by various artificial intelligence algorithms, and can be directly transplanted and extended to classification algorithms, with strong universality and generalization. Defects: The oversampling data enhancement method of this scheme is based on the Borderline-SMOTE algorithm, focusing on enhancing the sample information of the boundary. It identifies which samples are at the boundary between the minority class and the majority class according to the number of similar samples of the minority class sample neighbors, and then generates minority class samples at the boundary. However, due to the direct use of random sampling for sample selection, the data before oversampling often has uneven distribution, and sparsely distributed samples are easily ignored. Summary of the invention
[0008] Aiming at the problem that network traffic data is difficult to obtain and most of the data are often very small in difference, thus often resulting in serious imbalance of data categories, the present invention proposes a network traffic data enhancement method for sample imbalance.
[0009] The technical solution adopted by the present invention is:
[0010] A network traffic data enhancement method for sample imbalance includes the following steps:
[0011] S1, clustering the original network traffic data set, and dividing the clustering result clusters into a minority sample set and a majority sample set according to the number of traffic data in the clustering result clusters;
[0012] S2. Calculate the Euclidean distance between all minority class samples in each minority class sample set, and obtain the Euclidean distance matrix. According to the Euclidean distance matrix, obtain the distance ratio of each minority class sample to other minority class samples in each minority class sample set, and normalize it to obtain the sparse factor between each minority class sample.
[0013] S3, calculating the Euclidean distances from all minority class samples in each minority class sample set to all majority class samples in the majority class sample set, obtaining a Euclidean distance matrix, obtaining the distance ratio from the minority class samples to the majority class samples according to the Euclidean distance matrix, and performing normalization to obtain the difference between the minority class samples and the majority class samples;
[0014] S4. According to the sparse factors between minority samples in each minority sample set and the differences between minority samples and majority samples, the minority samples are given initial weights, and new samples are synthesized based on SMOTE to generate a new data set.
[0015] Furthermore, step S1 specifically includes:
[0016] S11, clustering the original network traffic data set through the K-Means clustering algorithm to obtain a certain number of clustering result clusters;
[0017] S12. For any clustering result cluster c k , when the number of samples contained in the cluster is less than or equal to the threshold θ, it is marked as the minority class sample set c k,min ; When the number of samples contained in the cluster is greater than the threshold θ, it is marked as the majority class sample set c k,maj .
[0018] Furthermore, step S2 specifically includes:
[0019] S21. For each minority class sample set c k,min , calculate the Euclidean distance between all minority class samples and get the Euclidean distance matrix
[0020]
[0021] In the formula, n is the minority class sample set c k,min The number of samples in ;
[0022] S22. According to the Euclidean distance matrix Get the minority class sample set c k,min For each minority class sample x i The distance ratio to other minority samples
[0023] Wherein, step S22 specifically includes:
[0024] S221. Calculate the minority class sample set c k,min Any minority class sample x i The sum of the distances to all other minority class samples, that is, the Euclidean distance matrix Sum each row of elements in express;
[0025] S222. Calculate the minority class sample set c k,min The sum of the distances between all minority class samples in , that is, the Euclidean distance matrix The upper triangular elements of express;
[0026] S223. Calculate the minority class sample set c k,min Any minority class sample x i The ratio of the sum of distances to all other minority class samples to the sum of distances between all minority class samples That is, calculate the Euclidean distance matrix The sum of the elements in the i-th row of is divided by the Euclidean distance matrix The sum of the upper triangular elements of ;
[0027]
[0028] S23, when Standard Deviation When it is not 0, the z-score method is used to Standardize and get the standardized score The 3σ criterion is used to clean noise during the standardization process, and the data range used to calculate the sparse factor is locked within Then Perform normalization calculation, that is, calculate The proportion of minority class samples in the sum of standardized scores is obtained as the minority class sample x i Sparsity factor i ;when Standard Deviation When it is 0, directly Perform normalization calculation to obtain the minority class sample xi The sparse factor of is calculated as follows:
[0029]
[0030]
[0031] S24. Repeat S21-S23 to obtain the sparse factors of all minority class samples in all minority class sample sets.
[0032] Furthermore, step S3 specifically includes:
[0033] S31. For a minority class sample set c k,min , first calculate all minority class samples to all majority class sample sets c k,maj The Euclidean distance of all majority class samples in the , and the Euclidean distance matrix is obtained
[0034]
[0035] In the formula, n is the minority class sample set c k,min The number of samples in, m is the set of all majority class samples c k,maj The number of all majority class samples in ;
[0036] S32. Obtain minority class sample set c k,min The minority class samples x i The distance ratio to the majority class samples Get the difference between the minority class samples and the majority class samples;
[0037] Wherein, step S32 specifically includes:
[0038] S321. Calculate the minority class sample set c k,min Any minority class sample x i To the majority class sample set c k,maj The sum of the distances of all majority class samples in, that is, the Euclidean distance matrix Sum each row of elements in express;
[0039] S322. Calculate the minority class sample set c k,min All minority class samples to majority class samples set c k,maj The sum of the distances of all majority class samples in , that is, the Euclidean distance matrix The sum of all elements in express;
[0040] S323. Calculate the minority class sample set c k,min Any minority class sample xi To the majority class sample set c k,maj The ratio of the sum of the distances of all majority class samples to the sum of the distances of all minority class samples to all majority class samples That is, calculate the Euclidean distance matrix The sum of the elements in the i-th row of is divided by the Euclidean distance matrix The sum of all elements in ;
[0041]
[0042] S33, when Standard Deviation When it is not 0, the z-score method is used to Standardize and get the standardized score The 3σ criterion is used to clean noise during the standardization process, and the data range used to calculate the sparse factor is locked within Then Perform normalization calculation, that is, calculate The proportion of minority class samples in the sum of standardized scores is obtained as the minority class sample x i Diversity between samples of the majority class i ;when Standard Deviation When it is 0, Perform normalization calculation to obtain the minority class sample x i Diversity between samples from the majority class i ; The calculation formula is as follows:
[0043]
[0044]
[0045] S34, repeat S31-S33 to obtain the difference of all minority class samples in all minority class sample sets;
[0046] Furthermore, step S4 specifically includes:
[0047] S41. According to any minority class sample set c k,min The sparse factor and difference of the minority class samples in the image are normalized using the Sigmoid function to give the minority class samples an initial weight, so that the surrounding sparse and boundary samples have a higher weight. i ;
[0048]
[0049] S42, randomly select from the weighted samples, synthesize new samples based on the SMOTE algorithm, and use triangular centroid sampling to sample the minority class sample set c k,min Perform sample synthesis;
[0050] S43. Repeat S41-S42 until all minority class sample sets have completed sample synthesis, and obtain a data-enhanced data set.
[0051] Compared with the prior art, the present invention has the following advantages:
[0052] (1) After clustering the original data set, the data is often unevenly distributed. The present invention helps select samples from sparse areas by calculating the sparsity of minority class samples in the clustering results.
[0053] (2) After clustering the original data set, the present invention calculates the difference between the minority class samples and the majority class samples in the clustering results to help select samples at the boundary. The sparsity and difference of any minority class sample are used as standards for objective weighting and then data synthesis is performed, which effectively improves the quality of the generated data.
[0054] (3) Through the oversampling data enhancement method proposed in the present invention, samples with sparse distribution and samples at the boundary of the data set can be selected more effectively, thereby improving the quality of generated data. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.
[0056] Figure 2 This is the original network traffic data diagram.
[0057] Figure 3 This is the data enhancement result diagram. DETAILED DESCRIPTION
[0058] In the example of the present invention, the public data set KDDCUP99 is used as the original input data. Figure 1 The specific analysis steps are as follows:
[0059] S1. Perform K-Means clustering (k=3) on the original network traffic data set to divide the minority class sample set and the majority class sample set in the clustering result; specifically, the following steps are included:
[0060] S11, clustering the original network traffic data set through the K-Means clustering algorithm to obtain a certain number of clustering result clusters;
[0061] S12. For any clustering result cluster c k, when the number of samples contained in the cluster is less than or equal to the threshold θ, it is marked as the minority class sample set c k,min ; When the number of samples contained in the cluster is greater than the threshold θ, it is marked as the majority class sample set c k,maj .
[0062] The original network traffic dataset is as follows Figure 2 The division results are shown in Table 1:
[0063] <![CDATA[c k ]]> <![CDATA[num(c k )]]> <![CDATA[c 1,maj ]]> 1321 <![CDATA[c 2,min ]]> 12 <![CDATA[c 3,min ]]> 6
[0064] Table 1. Division of majority and minority classes in clustering
[0065] S2. Calculate the Euclidean distance between all minority class samples in each minority class sample set, and obtain the Euclidean distance matrix. According to the Euclidean distance matrix, obtain the distance ratio of each minority class sample to other minority class samples in each minority class sample set, and normalize it to obtain the sparsity between each minority class sample.
[0066] Step S2 specifically includes:
[0067] S21. For each minority class sample set c k,min , first calculate the Euclidean distance between all minority class samples to obtain the Euclidean distance matrix
[0068]
[0069] In the formula, n is the minority class sample set c k,min The number of samples in ;
[0070] S22. According to the Euclidean distance matrix Get the minority class sample set c k,min For each minority class sample x i The distance ratio to other minority samples
[0071] Wherein, step S22 specifically includes:
[0072] S221. Calculate the minority class sample set c k,min Any minority class sample x i The sum of the distances to all other minority class samples, that is, the Euclidean distance matrix Sum each row of elements in express;
[0073] S222. Calculate the minority class sample set c k,min The sum of the distances between all minority class samples in , that is, the Euclidean distance matrix The upper triangular elements of express;
[0074] S223. Calculate the minority class sample set c k,min Any minority class sample x i The ratio of the sum of distances to all other minority class samples to the sum of distances between all minority class samples That is, calculate the Euclidean distance matrix The sum of the elements in the i-th row of is divided by the Euclidean distance matrix The sum of the upper triangular elements of ;
[0075]
[0076] S23, when Standard Deviation When it is not 0, the z-score method is used to Standardize and get the standardized score The 3σ criterion is used for noise cleaning during the standardization process, and the data range used to calculate the sparsity is locked within Then Perform normalization calculation, that is, calculate The proportion of minority class samples in the sum of standardized scores is obtained as the minority class sample x i Sparsity i ;when Standard Deviation When it is 0, directly Perform normalization calculation to obtain the minority class sample x i The sparsity of ; the calculation formula is as follows:
[0077]
[0078]
[0079] S24. Repeat S21-S23 to obtain the sparsity of all minority class samples in all minority class sample sets.
[0080] The sparsity of minority class samples is calculated by formulas (1), (2) and (3). The results are shown in Table 2 below:
[0081]
[0082] Table 2. Sparsity record table of minority class samples
[0083] S3, calculating the Euclidean distances from all minority class samples in each minority class sample set to all majority class samples in the majority class sample set, obtaining a Euclidean distance matrix, obtaining the distance ratio from the minority class samples to the majority class samples according to the Euclidean distance matrix, and performing normalization to obtain the difference between the minority class samples and the majority class samples;
[0084] Step S3 specifically includes:
[0085] S31. For a minority class sample set c k,min , first calculate all minority class samples to all majority class sample sets c k,maj The Euclidean distance of all majority class samples in the , and the Euclidean distance matrix is obtained
[0086]
[0087] In the formula, n is the minority class sample set c k,min The number of samples in, m is the set of all majority class samples c k,maj The number of all majority class samples in ;
[0088] S32. Obtain minority class sample set c k,min The minority class samples x i The distance ratio to the majority class samples Get the difference between the minority class samples and the majority class samples;
[0089] Wherein, step S32 specifically includes:
[0090] S321. Calculate the minority class sample set c k,min Any minority class sample x i To the majority class sample set c k,maj The sum of the distances of all majority class samples in, that is, the Euclidean distance matrix Sum each row of elements in express;
[0091] S322. Calculate the minority class sample set c k,min All minority class samples to majority class samples set c k,maj The sum of the distances of all majority class samples in , that is, the Euclidean distance matrix The sum of all elements in express;
[0092] S323, calculate the minority class sample set c k,min Any minority class sample x i To the majority class sample set c k,majThe ratio of the sum of the distances of all majority class samples to the sum of the distances of all minority class samples to all majority class samples That is, calculate the Euclidean distance matrix The sum of the elements in the i-th row of is divided by the Euclidean distance matrix The sum of all elements in ;
[0093]
[0094] S33, when Standard Deviation When it is not 0, the z-score method is used to Standardize and get the standardized score The 3σ criterion is used to clean noise during the standardization process, and the data range used to calculate the sparse factor is locked within Then Perform normalization calculation, that is, calculate The proportion of minority class samples in the sum of standardized scores is obtained as the minority class sample x i Diversity between samples of the majority class i ;when Standard Deviation When it is 0, Perform normalization calculation to obtain the minority class sample x i Diversity between samples of the majority class i ; The calculation formula is as follows:
[0095]
[0096]
[0097] S34, repeat S31-S33 to obtain the difference of all minority class samples in all minority class sample sets; calculate the difference of minority class samples by formula (4), (5) and (6), and the result is shown in Table 3 below:
[0098]
[0099] Table 3. Difference record table of minority class samples
[0100] S4. According to the sparsity between minority samples and the difference between minority samples and majority samples in each minority sample set, the minority samples are given initial weights, and new samples are synthesized based on SMOTE to generate a new data set.
[0101] Step S4 specifically includes:
[0102] S41. According to any minority class sample set ck,min The sparsity and diversity of minority samples are normalized using the Sigmoid function to assign initial weights to minority samples, so that sparse samples around and at the boundary get higher weights;
[0103]
[0104] S42, randomly select from the weighted samples, synthesize new samples based on the SMOTE algorithm, and use triangular centroid sampling to sample the minority class sample set c k,min Perform sample synthesis;
[0105] S43, repeat S41-S42 until all minority class sample sets have completed sample synthesis, and obtain the data set after data enhancement. The original data set contains a total of 1339 samples, and the data set after oversampling data enhancement contains a total of 2643 samples. The data enhancement result is shown in the figure Figure 3 shown.
[0106] Formula (7) is used to assign initial weights to minority samples. The results are shown in Table 4:
[0107]
[0108] Table 4 Minority class sample weight record table
Claims
1. A network traffic data enhancement method for sample imbalance, characterized in that: The following steps are involved: S1, clustering the original network traffic data set, and dividing the clustering result clusters into a minority sample set and a majority sample set according to the number of traffic data in the clustering result clusters; S2. Calculate the Euclidean distance between all minority class samples in each minority class sample set, and obtain the Euclidean distance matrix. According to the Euclidean distance matrix, obtain the distance ratio of each minority class sample to other minority class samples in each minority class sample set, and normalize it to obtain the sparse factor between each minority class sample. S3, calculating the Euclidean distances from all minority class samples in each minority class sample set to all majority class samples in the majority class sample set, obtaining a Euclidean distance matrix, obtaining the distance ratio from the minority class samples to the majority class samples according to the Euclidean distance matrix, and performing normalization to obtain the difference between the minority class samples and the majority class samples; S4, assign initial weights to minority samples according to the sparse factors between minority samples in each minority sample set and the differences between minority samples and majority samples, and synthesize new samples based on SMOTE to generate a new data set; Wherein, step S2 specifically includes: S21. For each minority class sample set c k,mi n, calculate the Euclidean distance between all minority class samples and obtain the Euclidean distance matrix In the formula, n is the minority class sample set c k,mi The number of samples in n; S22. According to the Euclidean distance matrix Get the minority class sample set c k,min For each minority class sample x i The proportion of distances to other minority samples Wherein, step S22 specifically includes: S221. Calculate the minority class sample set c k,min Any minority class sample x i The sum of the distances to all other minority class samples, that is, the Euclidean distance matrix Sum each row of elements in express; S222. Calculate the minority class sample set c k,mi The sum of the distances between all minority class samples in n, that is, the Euclidean distance matrix The upper triangular elements of express; S223. Calculate the minority class sample set c k,min Any minority class sample x i The ratio of the sum of distances to all other minority class samples to the sum of distances between all minority class samples That is, calculate the Euclidean distance matrix The sum of the elements in the i-th row of is divided by the Euclidean distance matrix The sum of the upper triangular elements of ; S23, when Standard Deviation When it is not 0, the z-score method is used to Standardize and get the standardized score The 3σ criterion is used to clean noise during the standardization process, and the data range used to calculate the sparse factor is locked within Then Perform normalization calculation, that is, calculate The proportion of minority class samples in the sum of standardized scores is obtained as the minority class sample x i The sparse factor Sparsityi; when Standard Deviation When it is 0, directly Perform normalization calculation to obtain the minority class sample x i The sparse factor of is calculated as follows: S24, repeat S21-S23 to obtain the sparse factors of all minority class samples in all minority class sample sets; Wherein, step S3 specifically includes: S31. For a minority class sample set c k,mi n, first calculate all minority class samples to all majority class sample sets c k,maj The Euclidean distance of all majority class samples in the matrix is obtained. In the formula, n is the minority class sample set c k,min The number of samples in, m is the set of all majority class samples c k,maj The number of all majority class samples in ; S32. Obtain minority class sample set c k,min The minority class samples x i The distance ratio to the majority class samples Get the difference between the minority class samples and the majority class samples; Wherein, step S32 specifically includes: S321. Calculate the minority class sample set c k,min Any minority class sample x i To the majority class sample set c k,maj The sum of the distances of all majority class samples in , that is, for the Euclidean distance matrix Sum each row of elements in express; S322. Calculate the minority class sample set c k,min All minority class samples to majority class samples set c k,maj The sum of the distances of all majority class samples in , that is, the Euclidean distance matrix The sum of all elements in express; S323, calculate the minority class sample set c k,min Any minority class sample x i To the majority class sample set c k,maj The ratio of the sum of the distances of all majority class samples to the sum of the distances of all minority class samples to all majority class samples That is, calculate the Euclidean distance matrix The sum of the elements in the i-th row of is divided by the Euclidean distance matrix The sum of all elements in ; S33, when Standard Deviation When it is not 0, z-SCore is used to Standardize and get the standardized score The 3σ criterion is used to clean noise during the standardization process, and the data range used to calculate the sparse factor is locked within Then Perform normalization calculation, that is, calculate The proportion of minority class samples in the sum of standardized scores is obtained as the minority class sample x i Diversity between samples of the majority class i ;when Standard Deviation When it is 0, Perform normalization calculation to obtain the minority class sample x i Diversity between samples of the majority class i ; The calculation formula is as follows: S34, repeat S31-S33 to obtain the difference of all minority class samples in all minority class sample sets; Wherein, step S4 specifically includes: S41. According to any minority class sample set c k,mi The sparse factors and differences of the minority class samples in n are normalized using the Sigmoid function to give the minority class samples an initial weight, so that the surrounding sparse and boundary samples have a higher weight. i ; S42, randomly select from the weighted samples, synthesize new samples based on the SMOTE algorithm, and use triangular centroid sampling to sample the minority class sample set c k,min Perform sample synthesis; S43. Repeat S41-S42 until all minority class sample sets have completed sample synthesis, and obtain a data-enhanced data set.
2. The network traffic data enhancement method for sample imbalance according to claim 1 is characterized in that: Step S1 specifically includes: S11, clustering the original network traffic data set through the K-Means clustering algorithm to obtain a certain number of clustering result clusters; S12. For any clustering result cluster c k , when the number of samples contained in the cluster is less than or equal to the threshold θ, it is marked as the minority class sample set c k,mi n; when the number of samples contained in the cluster is greater than the threshold θ, it is marked as the majority class sample set c k,maj .
