A method for constructing vulnerability attack database based on similarity measurement

Abstract

The present invention provides a method for constructing a vulnerability attack database based on similarity measurement. The method comprises: step 1, preprocessing the data set to obtain a data set with higher quality and stronger representativeness; step 2, obtaining the optimal feature subset of vulnerability attack by combining feature dimension reduction method; step 3, training multiple SVM classifiers to form a classification model, and completing the classification and identification of vulnerability attack; step 4, analyzing the classified vulnerability attack through mixed similarity measurement to determine whether it should be stored in the database; step 5, constructing a vulnerability attack database according to the result of step 4. Experimental results show that the proposed method has a certain effect on the constructed vulnerability attack database, and a vulnerability attack database for network traffic can be constructed, which has a certain effect on future research.

Description

Technical Field

[0001] The invention belongs to the technical field of network flow vulnerability attacks and relates to a vulnerability attack database construction method based on similarity measurement. Background Art

[0002] Today's society is developing rapidly, and the Internet has become one of the most important information technologies. As the scale of the network continues to grow, the traffic data is also showing an explosive growth trend. In addition, various applications in the network are emerging in an endless stream, which not only facilitates our lives, but also brings many network security and network management problems. At the same time, there are various vulnerability attacks and malicious traffic in the network, which greatly threaten network security. However, there are many studies on vulnerability databases, but there is no relatively complete vulnerability attack database for network traffic. The current compilation of vulnerability attacks is also scattered, which is not conducive to the analysis and detection of vulnerability attacks. Various vulnerability attacks exist in the current network, which poses a great security risk to the current network environment.

[0003] Vulnerability attacks are also a type of malicious traffic. Attackers will use existing vulnerabilities to complete malicious functions. In order to effectively manage the network, it is necessary to use the method of extracting vulnerability attack features in network traffic to accurately identify malicious traffic. Network traffic classification refers to classifying malicious traffic based on the corresponding attribute characteristics of various types of traffic from a large amount of network traffic data mixed with various categories, and then improving the vulnerability attack database after analysis of the classified data, and finally building a vulnerability attack database for network traffic. Kolosnjaji B et al. analyzed the API call sequence of the data, used deep learning methods to classify it, and judged the malicious attributes of the data at the same time. They obtained relatively good classification results through a hybrid neural network structure with two convolutional layers and one recursive layer; Li et al. improved the active learning malicious code detection method. They proposed a sample labeling strategy, which is a sample selection strategy based on the maximum distance and the minimum estimated risk. Compared with the method without active learning, the detection performance of the algorithm is better proved by experiments; Wang et al. proposed a hybrid model combining a deep autoencoder and a convolutional neural network, and used CNN-S and CNN-P structures for training to prevent overfitting problems. Experiments have shown that the accuracy of this hybrid model has been significantly improved and the training time has also been reduced to a certain extent. Summary of the invention

[0004] Based on the feature dimensionality reduction method in the prior art, there is still room for improvement in the accuracy of vulnerability attack identification. At the same time, the vulnerability attacks after classification are not reasonably utilized, and there is no relatively complete vulnerability attack database for reference and utilization in the network traffic direction. Therefore, the present invention proposes a vulnerability attack database construction method based on similarity measurement to solve the above problems.

[0005] The present invention provides a vulnerability attack database construction method based on similarity measurement, comprising:

[0006] Step 1: preprocess the data set to obtain a more representative data set;

[0007] Step 2, obtain the optimal feature subset of vulnerability attack by combining feature dimension reduction method;

[0008] Step 3: Train multiple SVM classifiers to form a classification model to complete the classification and identification of vulnerability attacks;

[0009] Step 4: After the classification, the vulnerability attacks are analyzed by hybrid similarity measurement to determine whether they should be stored in the database;

[0010] Step 5: Build a vulnerability attack database based on the results of step 4.

[0011] In the first aspect, the above step 1 specifically includes:

[0012] Step 1.1, the data scale in network traffic is very large. There will be a large amount of dirty data in the original data set, which may affect the operating efficiency of the classification algorithm. Therefore, data preprocessing can save analysis time. Data preprocessing is mainly divided into two steps: data sampling and data normalization.

[0013] Step 1.2, use stratified sampling method to sample data. First, stratify the original sample according to the rules, and then randomly sample each layer. Use random method to extract an exact number of samples from each layer after stratification. In the sampling process, we can use equal proportion method for sampling, or we can use unequal proportion method for sampling.

[0014] Step 1.3, data normalization is to map the data to a smaller numerical range according to the rules to speed up the data analysis process. The normalization method used is minimum-maximum normalization, and both the training set and the test set must be normalized to prevent some eigenvalues ​​from being too large or too small. This method is used to map the sample dimension to between [-1, 1], and the mapping function is:

[0015]

[0016] Among them, min represents the minimum value of the samples in the calculated dimension, and max represents the maximum value of the samples in the calculated dimension.

[0017] In the second aspect, the above step 2 specifically includes:

[0018] Step 2.1, filter by ReliefF algorithm, calculate the correlation between features and sample categories, the larger the calculated value, the larger the corresponding weight, which also means that the feature attribute has a greater impact on the recognition result of the sample, when the obtained weight is higher than the preset threshold, the feature is retained, otherwise the feature is deleted, and a feature subset is obtained, and the relevant feature selection CFS algorithm is further filtered on the feature subset. CFS evaluates the value of the feature subset, retains the features with high value, deletes the features with low value, and obtains the first feature subset;

[0019] Step 2.2, use the heuristic sequence forward search strategy on the first feature subset to further select features for a second time to obtain an approximately optimal feature subset. First, based on the empty set, select one or several features that can improve the classification accuracy and add them to the candidate feature subset. Repeat this process until the number of features in the candidate feature subset reaches the total number of features, and obtain the second feature subset;

[0020] Step 2.3, perform principal component analysis feature extraction on the second feature subset, calculate the covariance matrix C of the sample space S, and calculate its orthogonal matrix P and eigenvalues ​​λ1,λ2,…,λ through the covariance matrix C n (where λ1≥λ2≥Λ≥λ n ), set the upper limit of the cumulative contribution rate, and obtain the standard orthogonal vector u by calculation i , the sample space after dimensionality reduction is U=u1,u2,…,u m Then we get the principal component feature y=(u1,u2,…,u m ) T x i , and the optimal feature subset is obtained.

[0021] In the third aspect, the above step 3 specifically includes:

[0022] Step 3.1, the optimal feature subset obtained after combining feature dimensionality reduction of the training set is used as the input of the multi-classification SVM classification algorithm, and the Gaussian function is selected as the kernel function to complete the training and obtain the vulnerability attack classification model;

[0023] Step 3.2: classify and identify the test set.

[0024] In a fourth aspect, the above step 4 specifically includes:

[0025] Similarity measurement is a measurement method that comprehensively evaluates the similarity between two things. The closer the two things are, the greater the similarity measurement between them. The classified vulnerability attacks are compared with the vulnerability attacks in the database to determine whether they are repeated with the content in the database. The object of comparison is the feature set of the vulnerability attack.

[0026] When the lengths of the compared feature sets are not much different, the Jaccard similarity coefficient is used for comparative analysis. Assuming there are two sets A and B, the Jaccard similarity coefficient of the two sets is the proportion of the number of elements in the intersection of the two sets in the union of the two sets. The calculation formula is as follows:

[0027]

[0028] When the lengths of the compared feature sets differ greatly, cosine similarity is used. For text type matching calculations, attribute vectors A and B are word frequency vectors in the document. In the comparison process, cosine similarity can also be seen as a method to normalize the file length. In the case of information retrieval, word frequency cannot be negative, so the cosine similarity of two samples ranges from 0 to 1. At the same time, the angle between the frequency vectors of two words cannot exceed 90 degrees.

[0029] In a fifth aspect, the above step 5 specifically includes:

[0030] Check the current vulnerability attack database, and create the database and corresponding data tables if it does not exist;

[0031] When the database exists, it is determined whether the current vulnerability attack should be stored in the database. If the data already exists in the database, the data is deleted.

[0032] When the database exists and the data needs to be stored in the vulnerability attack database, the format of the data is standardized, and a new data is added to the database according to the format in the database.

[0033] The beneficial effects of the present invention are:

[0034] 1. In terms of feature dimensionality reduction, the combined feature dimensionality reduction method proposed further eliminates redundant features, selects a feature set that can better represent the characteristics of the data set, and solves the existing high-dimensional feature space problem, thereby obtaining the optimal feature subset and combining it with the SVM classifier to complete the classification and identification of vulnerability attacks. The combined feature dimensionality reduction method can further improve the performance of classification and identification.

[0035] 2. The hybrid similarity measurement selects different similarity measurement methods based on the different lengths of the compared vulnerability attack feature sets. Considering different comparison situations effectively reduces the redundancy of the vulnerability attack database and also plays a certain role in shortening the experimental time. The experimental results show that the proposed method has a certain effect on the constructed vulnerability attack database, and can construct a vulnerability attack database for network traffic, which has a certain effect on future research. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of vulnerability attack database construction;

[0037] Figure 2 It is a flow chart of combined feature dimensionality reduction of the present invention;

[0038] Figure 3 It is the experimental data information used in the present invention;

[0039] Figure 4 It is a schematic diagram comparing the experimental results of the combined feature dimensionality reduction method, Filter feature selection, Wrapper feature selection, and PCA feature extraction;

[0040] Figure 5 It is a schematic diagram comparing the experimental results of the mixed similarity coefficient, the Jaccard similarity coefficient, the cosine similarity and the Dice distance. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0042] The purpose of the present invention is to provide a vulnerability attack database construction method based on similarity measurement for vulnerability attacks in network traffic, effectively realize the classification and identification of vulnerability attacks, build a complete vulnerability attack database, and conduct sufficient experiments to prove the feasibility and effectiveness of the method. Specifically:

[0043] A vulnerability attack database construction method based on similarity measurement, comprising:

[0044] Step 1: preprocess the data set to obtain a more representative data set;

[0045] Step 2, obtain the optimal feature subset of vulnerability attack by combining feature dimension reduction method;

[0046] Step 3: Train multiple SVM classifiers to build a classification model to complete the classification and identification of vulnerability attacks;

[0047] Step 4: Use hybrid similarity measurement to determine whether the classified vulnerability attacks should be stored in the database;

[0048] Step 5: Build a vulnerability attack database based on the results of step 4.

[0049] Further, the step 1 specifically includes:

[0050] Step 1.1, the data scale in network traffic is very large. There will be a lot of dirty data in the original data set, which may affect the efficiency of the classification algorithm. Therefore, preprocessing the data first can save analysis time. Data preprocessing is mainly divided into two steps: data sampling and data normalization;

[0051] Step 1.2, use stratified sampling method to sample data, first stratify the original sample according to the rules, then randomly sample each layer, and use random method to extract the exact number of samples from each layer after stratification, and use equal proportion method or unequal proportion method to sample in the sampling process;

[0052] Step 1.3, data normalization is to map the data to a smaller numerical range according to the rules to speed up the data analysis process. The normalization method used is minimum-maximum normalization, and both the training set and the test set must be normalized to prevent some eigenvalues ​​from being too large or too small. This method is used to map the sample dimension to between [-1, 1], and the mapping function is:

[0053]

[0054] Among them, min represents the minimum value of the samples in the calculated dimension, and max represents the maximum value of the samples in the calculated dimension.

[0055] Further, the step 2 specifically includes:

[0056] Step 2.1, filter by ReliefF algorithm, calculate the correlation between features and sample categories, the larger the calculated value, the larger the corresponding weight, which also means that the feature attribute has a greater impact on the recognition result of the sample, when the obtained weight is higher than the preset threshold, the feature is retained, otherwise the feature is deleted, and a feature subset is obtained, and the relevant feature selection CFS algorithm is further filtered on the feature subset. CFS evaluates the value of the feature subset, retains the features with high value, deletes the features with low value, and obtains the first feature subset;

[0057] Step 2.2, using the heuristic sequential forward search strategy on the first feature subset to further select features for a second time, and obtain an approximately optimal feature subset. First, based on the empty set, select one or several features that can improve the classification accuracy and add them to the candidate feature subset. Repeat this process until the number of features in the candidate feature subset reaches the total number of features, and obtain the second feature subset.

[0058] Step 2.3, perform principal component analysis feature extraction on the second feature subset, calculate the covariance matrix C of the sample space S, and calculate its orthogonal matrix P and eigenvalues ​​λ1,λ2,…,λ through the covariance matrix C n (where λ1≥λ2≥Λ≥λ n ), set the upper limit of the cumulative contribution rate, and obtain the standard orthogonal vector u by calculation i , the sample space after dimensionality reduction is U=u1,u2,…,u m Then we get the principal component feature y=(u1,u2,…,u m ) T x i , and the optimal feature subset is obtained.

[0059] Further, the step 3 specifically includes:

[0060] Step 3.1, the optimal feature subset obtained after combining feature dimensionality reduction of the training set is used as the input of the multi-classification SVM classification algorithm, and the Gaussian function is selected as the kernel function to complete the training and obtain the vulnerability attack classification model;

[0061] Step 3.2: classify and identify the test set.

[0062] Further, the step 4 specifically includes:

[0063] Step 4.1, similarity measurement is a measurement method to comprehensively evaluate the similarity between two things. The closer the two things are, the greater the similarity measurement between them. Conversely, the farther the two things are, the smaller the similarity measurement between them. Compare the classified vulnerability attacks with the vulnerability attacks in the database to determine whether they are repeated in the database. The object of comparison is the feature set of the vulnerability attack;

[0064] Step 4.2, when the lengths of the compared feature sets are not much different, the Jaccard similarity coefficient is used for comparative analysis. Assuming there are two sets A and B, the Jaccard similarity coefficient of the two sets is the proportion of the number of elements in the intersection of the two sets in the union of the two sets. The calculation formula is as follows:

[0065]

[0066] Step 4.3, when the lengths of the compared feature sets differ greatly, cosine similarity is used. For text type matching calculations, attribute vectors A and B are word frequency vectors in the document. In the comparison process, cosine similarity can also be regarded as a method to normalize the file length. In the case of information retrieval, word frequency cannot be negative, so the cosine similarity of two samples ranges from 0 to 1. At the same time, the angle between the frequency vectors of two words cannot exceed 90 degrees.

[0067] Further, the step 5 specifically includes:

[0068] Step 5.1, check the current vulnerability attack database, if it does not exist, create the database and the corresponding data table;

[0069] Step 5.2, when the database exists, determine whether the current vulnerability attack should be stored in the database. If the data already exists in the database, delete the data.

[0070] Step 5.3, when the database exists and the data needs to be stored in the vulnerability attack database, standardize the format of the data and add a new data in the database according to the format in the database.

[0071] like Figure 2 As shown, the combined feature dimensionality reduction method of the present invention comprises:

[0072] Step 1: Preprocess the initial data set through data sampling and data normalization to obtain a data set with small scale but strong representativeness.

[0073] Step 2: Filter-type feature selection is performed on the preprocessed data set, including the ReliefF algorithm based on the distance criterion and the related feature selection CFS method.

[0074] In step 2 above, the steps of the above Filter-style feature selection are as follows:

[0075] Step 2.1, the steps of the ReliefF algorithm are as follows: (1) randomly select any sample s in the training set D; (2) select any sample s of the same type as sample s in D r Select n nearest neighbor samples r from the samples, and calculate the distance L between sample s and sample r sr ; (3) In a different type D from sample s t Select n nearest neighbor samples t from the samples, and calculate the distance L between sample s and sample t st ; (4) Compare L sr and L st Size: If L sr <L stIf the sample has an impact on the sample classification, it means that the weight of the corresponding sample attribute needs to be increased. On the contrary, it means that the sample has a small impact on the sample classification result, that is, the weight of the corresponding sample attribute needs to be reduced. (5) Repeat the above process m times, calculate the average weight of each feature, arrange the obtained weights from large to small, and retain the features whose weights are ahead of the set threshold, thereby obtaining a feature subset. The following formula is used to update the feature weight value:

[0076]

[0077] Among them, w(x) represents the weight value of feature x, s represents the random selection of any sample in the training set D, and r Select n nearest neighbor samples r from the samples of different types D t Select n nearest neighbor samples t, p(D t ) indicates category D t The ratio of , d(x,s,r) is the difference between sample s and sample r in the xth feature, and m represents the number of cycles.

[0078] The CFS algorithm removes redundant feature attributes to obtain a better feature subset. The search strategy uses the best-first search strategy. CFS evaluates the value of the feature subset. In this method, the following formula is used to calculate the attribute correlation:

[0079]

[0080] Among them, k represents the number of features contained in the feature subset, is the average correlation between features and categories, Represents the average correlation between features. A good feature subset should contain features that are highly correlated with the category, but the features are not correlated with each other. Therefore, when the correlation between features and categories is high and the correlation between features is low, the value of M is relatively large, and the classification effect of the feature subset is also better.

[0081] Step 3: Perform Wrapper feature selection on the feature subset in the previous step.

[0082] The steps of the above Wrapper feature selection are as follows:

[0083] 1), initial candidate feature subset F0;

[0084] 2) Select i features from the feature subset F after the Filter-style feature selection and add them to the initial candidate feature subset F0 to generate a new candidate feature subset F1, and calculate the classification accuracy A1 of the data set on F1;

[0085] 3) Select i features from the remaining features in F and add them to F1 to generate a new candidate feature subset F2, and calculate the classification accuracy A2 on F2;

[0086] 4) Compare A1 and A2. If A2>A1, it means that the accuracy on the candidate feature subset F2 is higher. Then change F1 so that F1=F2. Otherwise, do not change F1.

[0087] 5) Update the value of F2 according to the above steps, and then calculate the new value of A2 and compare it with A1 until the end condition is met.

[0088] Step 4: Perform principal component analysis (PCA) feature extraction on the data set after feature selection.

[0089] The steps of the above PCA feature extraction are as follows:

[0090] 1), the covariance matrix C of the sample space S.

[0091] Assume there are N samples S∈R n , each sample can be represented as S i =[s i1 ,…,s in ] T ∈R n , the mean vector is denoted as V, S i =[s 1i ,Λ,s ni ] T ∈R n is the corresponding eigenvector, and the covariance matrix is ​​calculated as follows:

[0092]

[0093] 2) Calculate its orthogonal matrix P and eigenvalues ​​λ1,λ2,…,λ through the covariance matrix C n (where λ1≥λ2≥Λ≥λ n );

[0094] 3) Set the upper limit of the cumulative contribution rate and obtain the standard orthogonal vector u by calculation i , the sample space after dimensionality reduction is U=u1,u2,…,u m ;

[0095] 4), get the principal component feature y=(u1,u2,…,u m ) T x i , and the optimal feature subset is obtained.

[0096] Figure 2 It is a flow chart of combined feature dimensionality reduction of the present invention; Figure 3It is the experimental data information used in the present invention; Figure 4 It is a schematic diagram comparing the experimental results of the combined feature dimensionality reduction method, Filter feature selection, Wrapper feature selection, and PCA feature extraction; Figure 5 It is a schematic diagram comparing the experimental results of the mixed similarity coefficient, the Jaccard similarity coefficient, the cosine similarity and the Dice distance.

Claims

1. A method for constructing a vulnerability attack database based on similarity measurement, characterized in that: include: Step 1, preprocess the data set to obtain a representative data set; Step 2, obtain the optimal feature subset of vulnerability attack by combining feature dimension reduction method; Step 3: Train multiple SVM classifiers to build a classification model to complete the classification and identification of vulnerability attacks; Step 4: Use hybrid similarity measurement to determine whether the classified vulnerability attacks should be stored in the database; Step 5: construct a vulnerability attack database based on the results of step 4; The step 1 specifically includes: Step 1.1, the dirty data in the original data set of network traffic will affect the operating efficiency of the classification algorithm. Therefore, preprocessing the data first can save analysis time. Data preprocessing is divided into two steps: data sampling and data normalization; Step 1.2, use stratified sampling method to sample data, first stratify the original sample according to the rules, then randomly sample each layer, and randomly select a certain number of samples from each layer after stratification, and use equal proportion or unequal proportion to sample in the sampling process; Step 1.3, data normalization is to map the data to a smaller numerical range according to the rules to speed up the data analysis process. The normalization method used is minimum-maximum normalization, and both the training set and the test set must be normalized. The normalization method is used to map the sample dimension to [-1, 1]. The mapping function is: Among them, r is the sample, min represents the minimum value of the sample in the calculated dimension, and max represents the maximum value of the sample in the calculated dimension; The step 2 specifically includes: Step 2.1, filter by ReliefF algorithm, calculate the correlation between features and sample categories, the larger the calculated value, the larger the corresponding weight, which also means that the feature attribute has a greater impact on the recognition result of the sample, when the obtained weight is higher than the preset threshold, the feature is retained, otherwise the feature is deleted, and a feature subset is obtained, and the relevant feature selection CFS algorithm is further filtered on the feature subset. CFS evaluates the value of the feature subset, retains the features with high value, deletes the features with low value, and obtains the first feature subset; Step 2.2, using the heuristic sequence forward search strategy on the first feature subset to further select features for a second time, and obtain an approximately optimal feature subset. First, based on the empty set, select one or several features that can improve the classification accuracy and add them to the candidate feature subset. Repeat this process until the number of features in the candidate feature subset reaches the total number of features, and obtain the second feature subset. Step 2.3, perform principal component analysis feature extraction on the second feature subset, calculate the covariance matrix C of the sample space S, and calculate its orthogonal matrix P and eigenvalues ​​λ1, λ2, …, λ through the covariance matrix C. n , where λ1≥λ2≥…≥λ n , set the upper limit of the cumulative contribution rate, and obtain the standard orthogonal vector u by calculation i , the sample space after dimensionality reduction is U=u1,u2,…,u m Then we get the principal component feature y=(u1,u2,…,u m ) T x i , and the optimal feature subset is obtained.

2. The method according to claim 1, characterized in that: The step 3 specifically includes: Step 3.1, the optimal feature subset obtained after combining feature dimensionality reduction of the training set is used as the input of the multi-classification SVM classification algorithm, and the Gaussian function is selected as the kernel function to complete the training and obtain the vulnerability attack classification model; Step 3.2: classify and identify the test set.

3. The method according to claim 1, characterized in that: The step 4 specifically includes: Step 4.1, similarity measurement is a measurement method for comprehensively evaluating the similarity between two objects. The closer the two objects are, the greater the similarity measurement between them. Conversely, the more distant the two objects are, the smaller the similarity measurement between them. Compare the classified vulnerability attacks with the vulnerability attacks in the database to determine whether they are repeated in the database. The object of comparison is the feature set of the vulnerability attacks. Step 4.2, when the length difference of the compared feature sets is small, the Jaccard similarity coefficient is used for comparative analysis. Assuming there are two sets A and B, the Jaccard similarity coefficient of the two sets is the proportion of the number of elements in the intersection of the two sets in the union of the two sets. The calculation formula is as follows: Step 4.3, when the lengths of the compared feature sets differ greatly, cosine similarity is used. For text type matching calculations, attribute vectors A and B are word frequency vectors in the document. In the comparison process, cosine similarity is regarded as a method to normalize the file length. In the case of information retrieval, word frequency cannot be negative, so the cosine similarity of two samples ranges from 0 to 1. At the same time, the angle between the frequency vectors of two words cannot exceed 90 degrees.

4. The method according to claim 1, characterized in that: The step 5 specifically includes: Step 5.1, check the current vulnerability attack database, if it does not exist, create the database and the corresponding data table; Step 5.2, when the database exists, determine whether the current vulnerability attack should be stored in the database, and if the current vulnerability attack data already exists in the database, delete the data; Step 5.3, when the database exists and the data needs to be stored in the vulnerability attack database, standardize the format of the data and add a new data in the database according to the format in the database.

Images