An unknown attack identification method and device based on spectral clustering and four-quadrant feature division

Abstract

The application provides an unknown attack identification method and device based on spectral clustering and four-quadrant feature division, comprising the following steps: first, collecting attack traffic data in an actual network environment, combining diversified attack samples of a public data set, and performing data analysis and multi-dimensional feature extraction; second, synchronously collecting real-time network traffic and extracting features; then, jointly analyzing historical traffic and real-time traffic data by using a multi-modal spectral clustering algorithm to realize intelligent classification of attack behaviors; finally, deeply analyzing the clustering results based on a four-quadrant feature division model (including four dimensions of key unique features, core common features, potential transition features and noise features), locking potential attack modes through core common features, and constructing accurate attack feature portraits by using key unique features. The application has the characteristics of intelligently identifying unknown attacks, supporting accurate classification of unknown attacks, and providing dynamic protection for network security.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things security, and in particular to an unknown attack identification method and device based on spectral clustering and four-quadrant feature division. BACKGROUND

[0002] With the accelerated digital transformation, network attack means is increasingly complex and concealed, and the traditional detection method based on rules and feature signature has been difficult to cope with unknown security risks such as advanced persistent threat (APT) and zero-day attack. The existing detection technology mainly relies on known attack mode library or static behavior analysis, and when facing the evasive technology (such as obfuscation, polymorphic attack) of the attacker constantly evolving, it often shows a high false positive rate and a high false negative rate. In addition, although the detection scheme based on machine learning improves the automation level, there are still two core bottlenecks: on the one hand, the complex feature engineering and model training process lead to low detection efficiency, which is difficult to meet the real-time protection requirements; on the other hand, most methods lack the adaptive identification ability of unknown attacks, and cannot effectively mine the potential common features and unique features in the attack traffic. SUMMARY

[0003] Therefore, the purpose of the present application is to provide an unknown attack identification method and device based on spectral clustering and four-quadrant feature division, which solves the problems of difficult real-time detection of APT and other concealed unknown attacks in the Internet of Things and low attack identification efficiency. It has the characteristics of high detection efficiency, accurate detection results and support for dynamic data detection.

[0004] To achieve the above purpose, the present application adopts the following technical scheme: an unknown attack identification method based on spectral clustering and four-quadrant feature division, comprising the following steps:

[0005] S1, collecting network attack history traffic data suffered in the actual network environment, or extracting multiple types of network attack history traffic data from a public data set, and analyzing the network attack history traffic data;

[0006] S2, collecting real-time traffic data and analyzing it to extract multi-dimensional features;

[0007] S3, performing multi-modal spectral clustering on the network attack history traffic data and real-time traffic data, and dividing different categories;

[0008] S4, according to the clustering result, performing feature analysis based on the four-quadrant feature division method to identify potential attacks;

[0009] The S4 comprises:

[0010] According to the feature matrix in S3, the inter-class discriminability and intra-class consistency are calculated, and the inter-class discriminability calculation method is as follows:

[0011]

[0012] wherein, denotes the mean of class c1, denotes the mean of class c2, denotes the variance of class c1, denotes the variance of class c2, is the inter-class discriminativeness, =1e -6 , preventing division by zero errors;

[0013] The intra-class consistency is calculated as follows:

[0014]

[0015] wherein, denotes the value of intra-class consistency, is the intra-class variance, is the global variance;

[0016] According to the inter-class discriminativeness and the intra-class consistency C of each feature, the coordinates are constructed, and the feature is divided based on the quadrants; specifically, it is divided into four quadrants: 1) the key unique feature located in the first quadrant, which has high inter-class discriminativeness and high intra-class consistency, and is an important basis for class identification and is used to generate data with class labels; 2) the core common feature located in the second quadrant, which shows low inter-class discriminativeness but high intra-class consistency, and is an important feature for identifying potential attacks; 3) the potential transition feature located in the third quadrant, which has the characteristics of high inter-class discriminativeness but low intra-class consistency, and needs to be analyzed twice; 4) the noise feature located in the fourth quadrant, which is considered as redundant or interference information and is removed.

[0017] In a preferred embodiment, the traffic analysis process comprises:

[0018] The network environment is comprehensively monitored, network traffic is collected, and multi-dimensional fields are analyzed. First, basic fields in a data packet are analyzed, including a source IP address, a destination IP address, a source port number, a destination port number, a protocol type, IP header information, a data packet size, and a timestamp. Then, according to different protocol types, specific protocol fields in the data packet are analyzed. In a TCP / IP protocol data packet, sequence numbers, acknowledgement numbers, TCP flag bits, window sizes, TCP data offsets, and TCP load sizes are analyzed. In a UDP protocol data packet, a UDP load size is analyzed. In an ICMP protocol data packet, types, error codes, identifiers, and sequence numbers are analyzed. In advanced protocols, request methods, request URLs, HTTP headers, and response codes are analyzed. Finally, session fields, including session IDs, connection states, and data flow durations, are analyzed.

[0019] In a preferred embodiment, the process of feature extraction includes:

[0020] For the results parsed in S1, a standard multi-dimensional feature vector input is constructed, as follows:

[0021]

[0022] wherein, represents a feature vector of a data packet d, represents basic features, represents protocol features, represents session features.

[0023] In a preferred embodiment, S3 includes:

[0024] Based on the multi-dimensional features extracted in S2, the multi-dimensional features are taken as inputs of a clustering model, and different methods are used to calculate similarities for different features, to construct a similarity matrix.

[0025] For IP, port, and protocol type fields, 0 or 1 is assigned according to whether they are the same, and Jaccard is used to calculate feature similarities as follows:

[0026]

[0027] wherein, represents a number of dimensions in which both sample values are 1, represents a number of dimensions in which a value in the x sample is 1 but a value in the y sample is 0, represents a number of dimensions in which a value in the y sample is 1 but a value in the x sample is 0.

[0028] For features of discrete data types, Wasserstein Distance is used to calculate similarities as follows:

[0029]

[0030] wherein, is the set of all joint distributions is the Euclidean distance between points is the lower bound, inf represents finding the infimum, represents the transmission plan matrix, i and j represent the ith row and jth column element in the matrix, respectively;

[0031] By eigenvalue decomposition, the similarity matrix is spectrally decomposed to obtain a low-dimensional embedding representation; the characteristic matrix is denoted as , wherein represents the similarity between n data points; based on the characteristic matrix, the Laplacian matrix is calculated as follows:

[0032]

[0033] wherein D is a diagonal matrix containing the degree of each node;

[0034]

[0035] represents the ith diagonal element of the matrix D, represents the element in the ith row and jth column of the adjacency matrix; according to the eigenvalue decomposition of the Laplacian matrix, as follows:

[0036]

[0037] wherein, represents solving the characteristic equation, L is the Laplacian matrix, is the eigenvector corresponding to ; is the eigenvalue, is the corresponding eigenvector;

[0038] According to the eigenvalue decomposition result, the eigenvector corresponding to the eigenvalue is selected to form a low-dimensional embedding;

[0039] Based on the low-dimensional embedding, a clustering algorithm is applied for clustering; in the clustering process, data partitioning is performed based on the clustering algorithm, so as to obtain the class label of the data;

[0040] After clustering is completed, the accuracy and rationality of the clustering result are evaluated according to the external indicators of accuracy and F1 value and the internal indicator of silhouette coefficient; different evaluation indicators are weighted to obtain a comprehensive evaluation result, as follows:

[0041]

[0042] wherein, is the weight of different evaluation indicators; is the point of correct clustering, is the total point of clustering, is the precision, r is the recall, is the average distance of point i and other points in the same cluster, is the average distance of point i and all points in the nearest cluster.

[0043] In a preferred embodiment, according to the feature division result, a potential attack is identified; first, the potential attack category is located based on the core common features, that is, these common features indicate that there is some commonality between the new class and the existing attack, and the potential unknown attack is identified based on such commonality; after the unknown attack is identified, it is further classified and distinguished from other attack types based on the key unique features, and the feature portrait of this type of potential unknown attack is formed.

[0044] The application also provides an unknown attack identification device based on spectral clustering and four-quadrant feature division.

[0045] Compared with the prior art, the application has the following beneficial effects: the unknown attack identification method provided by the application adopts the spectral clustering algorithm and the four-quadrant feature division algorithm, realizes efficient and high-accuracy intelligent identification of unknown attacks, can detect potential threats in time, and improves the security protection capability of the Internet of Things.

[0046] Multi-dimensional data fusion: according to different protocol types, the data packet fields are comprehensively and deeply analyzed, multi-dimensional feature vectors are constructed, and the feature similarity is evaluated differently.

[0047] Intelligent identification of potential unknown attacks: the spectral clustering algorithm and the four-quadrant feature division algorithm are adopted to analyze the feature similarity and intelligently identify potential unknown attacks, thereby improving the security protection capability of the Internet of Things.

[0048] Real-time response capability: compared with the machine learning algorithm, the method of the application is more efficient, can process real-time traffic, discover potential threats in time, and immediately execute security protection measures such as communication disconnection. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the flowchart of the preferred embodiment of the application. DETAILED DESCRIPTION

[0050] The application will be further described below in combination with the drawings and embodiments.

[0051] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application; as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, it will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0053] The basic idea of the application is that, first, by collecting attack traffic data in the actual network environment, and combining the diversified attack samples of the public data set, data analysis and multi-dimensional feature extraction are performed; second, real-time network traffic is synchronously collected and feature extraction is performed; then, a multi-modal spectral clustering algorithm is used to jointly analyze historical traffic and real-time traffic data, to realize intelligent classification of attack behavior; finally, based on a four-quadrant feature division model (including four dimensions of key unique features, core common features, potential transition features and noise features), the clustering results are analyzed in depth, the potential attack mode is locked through the core common features, and the precise attack feature portrait is constructed using the key unique features.

[0054] As shown in Figure 1 The application provides an unknown attack identification method based on spectral clustering and four-quadrant feature division, which comprises the following steps:

[0055] Step (1), comprehensively monitor the network environment, collect network traffic, and analyze multi-dimensional fields. First, analyze the basic fields in the data packet, including source IP address, destination IP address, source port number, destination port number, protocol type, IP header information, packet size, timestamp, etc. Then, according to different protocol types, analyze the protocol fields specific to the data packet, such as TCP / IP protocol data packet, analyze sequence number, acknowledgement number, TCP flag bit, window size, TCP data offset, TCP load size, etc.; UPD protocol data packet, analyze UDP load size, etc.; ICMP protocol data packet, analyze type (echo request, echo response, etc.), error code, identifier, sequence number, etc.; for advanced protocols such as HTTP, analyze request method (GET, POST, DELETE, etc.), request URL, HTTP header (Host, Accept, etc.), response code, etc. Finally, analyze the session fields, such as session ID, connection state, data flow duration, etc.; the details of the traffic analysis fields are shown in Table 1:

[0056] Table 1:

[0057]

[0058] Step (2): For the fields parsed in step (1), construct a standard multidimensional feature vector input, as shown below:

[0059]

[0060] in, This represents the feature vector of data packet d. Indicates basic features, Indicates protocol characteristics, Indicates session characteristics.

[0061] Step (3): Based on the multidimensional features extracted in step (2), these features are used as input to the clustering model. Different methods are used to calculate similarity for different features, and a similarity matrix is ​​constructed. For fields such as IP, port, and protocol type, values ​​of 0 or 1 are assigned based on whether they are the same. Feature similarity is calculated based on Jaccard, as shown below:

[0062]

[0063] in, This represents the number of dimensions where both sample values ​​are 1. This represents the number of dimensions where the value in sample x is 1 but the value in sample y is 0. This represents the number of dimensions where samples y have a value of 1 but samples x have a value of 0. For features of discrete data types, the Wasserstein Distance is used to calculate similarity, as shown below:

[0064]

[0065] in, It is all joint distributions The set, It is a point To improve computational efficiency, this invention uses Euclidean distance to determine the distance between objects.

[0066] Step (4): The similarity matrix is ​​decomposed spectrally using eigenvalue decomposition to obtain a low-dimensional embedding representation. Let the feature matrix be denoted as W, then... represents the similarity between n data points. The Laplacian matrix is ​​calculated based on the feature matrix, as shown below:

[0067]

[0068] Where D is a diagonal matrix containing the degree of each node.

[0069]

[0070] Eigenvalue decomposition is performed according to the Laplacian matrix, as shown below:

[0071]

[0072] wherein, is an eigenvalue, is a corresponding eigenvector.

[0073] According to the eigenvalue decomposition result, the eigenvectors corresponding to the smaller eigenvalues are selected to form a low-dimensional embedding.

[0074] Step (5), based on the low-dimensional embedding in step (4), a clustering algorithm is applied for clustering. The data points in the low-dimensional embedding space have a lower dimension compared to the original space, and the main structural features of the data are preserved. In the clustering process, based on K-means, DBSCAN, etc. Algorithm is used for data division, so as to obtain the class label of the data;

[0075] Step (6), after completing the clustering, the accuracy and rationality of the clustering result are evaluated according to external indicators such as accuracy, F1 value and internal indicators such as silhouette coefficient, and different evaluation indicators are weighted to obtain a comprehensive evaluation result, as shown below:

[0076]

[0077] wherein, is the weight of different evaluation indicators, and is an empirical parameter; is the point clustered correctly, is the total number of clustered points, is the precision, r is the recall, is the average distance between point i and other points in the same cluster, is the average distance between point i and all points in the nearest cluster.

[0078] Step (7), according to the feature matrix in step (3), the inter-class discriminability and intra-class consistency are calculated, and the inter-class discriminability calculation method is as shown below:

[0079]

[0080] wherein, =1e -6 , to prevent division by zero error.

[0081] The intra-class consistency calculation method is as shown below:

[0082]

[0083] wherein, is the intra-class variance, For global variance.

[0084] Step (8), according to the inter-class discriminant of each feature in step (7) and the intra-class consistency C, construct the coordinate f( ,C), and divide the features based on the quadrants. Specifically, divide into four quadrants: 1) key unique features located in the first quadrant, with high inter-class discriminant and high intra-class consistency, which can be used as an important basis for class identification and for generating data with class labels; 2) core common features located in the second quadrant, which exhibit low inter-class discriminant but high intra-class consistency, and are important features for identifying potential attacks; 3) potential transition features located in the third quadrant, which have high inter-class discriminant but low intra-class consistency, such features can reflect the attributes of a specific class, but have limited ability to distinguish between different classes, and need to be analyzed again; 4) noise features located in the fourth quadrant, which perform poorly in both dimensions and should be considered as redundant or interfering information and removed.

[0085] Step (9), according to the feature division result in step (8), identify potential attacks. First, based on the core common features, locate the potential attack classes, i.e. these features indicate that there are some commonalities between the new class and the existing attacks, and based on such commonalities, identify potential unknown attacks. After identifying unknown attacks, based on key unique features, further classify them and distinguish them from other attack types, forming the feature portrait of this type of potential unknown attack. Through this method, not only can potential unknown attacks be identified, but also specific features can be summarized for each identified attack, providing more accurate basis for subsequent security protection.

[0086] The above only describes the embodiments of the present application and does not limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for unknown attack identification based on spectral clustering and four-quadrant feature division, characterized in that, Includes the following steps: S1. Collect historical network attack traffic data suffered in actual network environments, or extract various types of historical network attack traffic data from public datasets, and analyze the historical network attack traffic data. S2. Collect real-time traffic data, analyze it, and extract multi-dimensional features; S3. Perform multimodal spectral clustering on historical and real-time network attack traffic data to classify them into different categories; S4. Based on the clustering results, perform feature analysis using the four-quadrant feature partitioning method to identify potential attacks; S4 includes: Based on the feature matrix in S3, calculate the inter-class discrimination and intra-class consistency. The method for calculating the inter-class discrimination is as follows: wherein denotes the mean of the class c1, denotes the mean of the class c2, denotes the variance of the class c1, denotes the variance of the class c2, is the interclass variance, = 1e -6 , prevents division by zero error; The method for calculating intra-class consistency is as follows: wherein, represents a value of intra-class consistency, is the intra-class variance, is the global variance; inter-class discriminativeness of each feature and intra-class consistency C, construct coordinates Based on the quadrant, the feature is divided; Specifically, it is divided into four quadrants: 1) key unique features located in the first quadrant, with high inter-class discriminativeness and high intra-class consistency, as an important basis for class identification, and used to generate data with class labels; 2) core common features located in the second quadrant, with low inter-class discriminativeness but high intra-class consistency, which are important features for identifying potential attacks; 3) potential transition features located in the third quadrant, with high inter-class discriminativeness but low intra-class consistency, which need to be analyzed twice; 4) noise features located in the fourth quadrant, which are considered as redundant or interference information and are removed.

2. The unknown attack identification method based on spectral clustering and four-quadrant feature partitioning according to claim 1, characterized in that, The traffic parsing process includes: This system comprehensively monitors the network environment, collects network traffic, and analyzes multi-dimensional fields. First, it parses basic fields in data packets, including source IP address, destination IP address, source port number, destination port number, protocol type, IP header information, packet size, and timestamp. Then, it parses protocol-specific fields in data packets based on different protocol types. For example, in TCP / IP packets, it parses sequence number, acknowledgment number, TCP flags, window size, TCP data offset, and TCP payload size; in UDP packets, it parses UDP payload size; in ICMP packets, it parses type, error code, identifier, and sequence number; in higher-level protocols, it parses request method, request URL, HTTP headers, and response codes. Finally, it parses session fields, including session ID, connection state, and data stream duration.

3. The method for identifying unknown attacks based on spectral clustering and four-quadrant feature partitioning according to claim 1, characterized in that, The feature extraction process includes: Based on the results parsed from S1, a standard multidimensional feature vector input is constructed as follows: in, This represents the feature vector of data packet d. Indicates basic features, Indicates protocol characteristics, Indicates session characteristics.

4. The unknown attack identification method based on spectral clustering and four-quadrant feature partitioning according to claim 1, characterized in that, S3 includes: Based on the multidimensional features extracted from S2, these multidimensional features are used as input to the clustering model, and different methods are used to calculate the similarity for different features to construct a similarity matrix. For the IP, port, and protocol type fields, assign a value of 0 or 1 based on whether they are the same, and calculate the feature similarity based on Jaccard. As shown below: in, This represents the number of dimensions where both sample values ​​are 1. This represents the number of dimensions where the value in sample x is 1 but the value in sample y is 0. This represents the number of dimensions where the value in sample y is 1 but the value in sample x is 0; For the characteristics of discrete data types, Wasserstein Distance is used to calculate similarity. As shown below: in, It is all joint distributions The set, It is a point ( Euclidean distance between () and (); inf represents finding the infimum. Let i represent the transmission plan matrix, where i and j represent the element in the i-th row and j-th column of the matrix, respectively. By performing spectral decomposition on the similarity matrix through eigenvalue decomposition, a low-dimensional embedding representation is obtained; the feature matrix is ​​denoted as... ,but Let represent the similarity between n data points; the Laplacian matrix is ​​calculated based on the feature matrix, as shown below: Where D is a diagonal matrix containing the degree of each node; Let i represent the i-th diagonal element of matrix D. Let represent the element in the i-th row and j-th column of the adjacency matrix; perform eigenvalue decomposition based on the Laplace matrix as follows: in, This represents solving the characteristic equation, where L is the Laplace matrix. , For eigenvalues, The corresponding feature vector; Based on the eigenvalue decomposition results, select the eigenvectors corresponding to the eigenvalues ​​to form a low-dimensional embedding; Clustering is performed based on low-dimensional embedding application clustering algorithms. During the clustering process, data is partitioned based on the clustering algorithm to obtain the category labels of the data. After clustering is completed, the accuracy and reasonableness of the clustering results are evaluated based on external indicators such as accuracy and F1 score, and internal indicators such as silhouette coefficient. The different evaluation indicators are weighted to obtain a comprehensive evaluation result, as shown below: in, Weights for different evaluation indicators; For the correctly clustered points, As the overall cluster point, Let r represent precision and r represent recall. Let i be the average distance between point i and other points in the same cluster. Let be the average distance between point i and all points in the nearest cluster.

5. The unknown attack identification method based on spectral clustering and four-quadrant feature partitioning according to claim 1, characterized in that, Based on the feature segmentation results, potential attacks are identified. First, potential attack categories are located based on core common features. These common features indicate that there are certain commonalities between the new category and existing attacks. Potential unknown attacks are identified based on such commonalities. After identifying unknown attacks, they are further classified and distinguished from other attack types based on key unique features, forming a feature profile of this type of potential unknown attack.

Images