A network attack sample automatic clustering identification method based on multi-feature fusion

Abstract

The application relates to the field of network information security and discloses a network attack sample automatic clustering identification method based on multi-feature fusion, which comprises the following steps: extracting sample multi-dimensional features within a set time window and splicing the features into original vectors; calculating feature fusion weights by combining the distribution probability of feature discrete states with information entropy and generating weighted feature vectors; performing feature value decomposition and linear transformation on the weighted feature vectors to reduce dimensions; adaptively calculating the core point threshold and the neighborhood radius parameter of density clustering by combining the total number of dimensions and the second derivative of the smooth distance curve, and performing clustering operation; identifying known variants and new unknown attacks based on the similarity between the clustering clusters and the feature library, and utilizing the closed-loop iteration of the new unknown attack trigger parameters. The application eliminates the asynchronous deviation of feature collection, gets rid of the dependence of clustering parameters on artificial experience, and effectively improves the detection accuracy of complex variants and unknown attacks.

Description

Technical Field

[0001] This invention relates to the field of network information security technology, and in particular to an automatic clustering and identification method for network attack samples based on multi-feature fusion. Background Technology

[0002] As cyberattack methods continue to evolve, utilizing machine learning and unsupervised clustering algorithms to automate the analysis and homology identification of massive amounts of cyberattack samples has become an important approach to threat detection. In conventional detection processes, systems need to extract multi-dimensional features from cyberattack samples to build analytical models.

[0003] Existing multi-feature acquisition methods often separate static file parsing from dynamic sandbox monitoring. This asynchronous acquisition process causes time skew in the lifecycles of features of different dimensions, and directly concatenating heterogeneous data easily introduces feature misalignment and combinatorial noise. In the feature fusion stage, traditional methods often use indiscriminate data concatenation without considering the actual distribution patterns of continuous and discrete features for weight allocation. This results in highly discriminative feature attributes being diluted by a large number of redundant fields, reducing the effectiveness of feature vectors in spatial measurement.

[0004] After feature construction, existing technologies often employ density clustering algorithms to classify sample sets. Directly calculating in the high-dimensional feature space can easily lead to homogenization of Euclidean distances between samples, causing the distance metric to lose its discriminative power. Furthermore, traditional density clustering algorithms generally rely on pre-set, fixed values ​​based on human experience when setting parameters such as core point thresholds and neighborhood radii. Attack samples in a network environment often exhibit a non-uniform density manifold distribution within the feature space, and fixed empirical parameters cannot adaptively match the local density patterns of different attack families. This makes the boundaries between dense and discrete regions in the clustering model blurry, making it difficult to accurately isolate anomalous noise, resulting in a high false negative rate for niche attack variants and a high false positive rate for overall classification.

[0005] Furthermore, most existing attack sample identification systems lack automated self-evolution mechanisms. When new, unknown attacks emerge that deviate from the known feature library distribution space, the system cannot automatically extract the distribution structure of the new attack to trigger updates to underlying features and model operation parameters. This one-way identification mode causes the evolution of the detection model to lag behind changes in attack methods, easily leading to model aging and a decline in defense capabilities over long-term operation. Summary of the Invention

[0006] The technical problem solved by this invention is that existing network attack identification technologies suffer from time-series asynchronous bias when collecting multidimensional heterogeneous data, and direct clustering of high-dimensional features is prone to Euclidean distance failure; moreover, the parameters of traditional density clustering algorithms heavily rely on human experience and cannot be adaptively adjusted according to the manifold shape of data distribution in different network environments. When facing unknown attack variants, they generally suffer from inaccurate identification of cluster density boundaries and low false alarm and false alarm rates.

[0007] To address the above problems, the present invention provides the following technical solution:

[0008] The first aspect of this invention provides an automatic clustering and identification method for network attack samples based on multi-feature fusion, comprising the following steps:

[0009] Extract multi-dimensional features of network attack samples within a set time window and concatenate them into an original feature vector;

[0010] The probability distribution of discrete states of each feature dimension of the original feature vector in the sample set is statistically analyzed and the feature fusion weights are calculated. The weighted feature vector is then generated using the feature fusion weights.

[0011] Eigenvalue decomposition and linear transformation are performed on the weighted eigenvectors to generate dimensionality-reduced eigenvectors.

[0012] The core parameters of the adaptive density clustering algorithm are the threshold parameter and the neighborhood radius parameter. It performs clustering operations on the dimensionality-reduced feature vector and outputs effective clusters and isolated samples.

[0013] Based on the similarity between effective clusters and a known attack feature library, known attack variants and novel unknown attacks are identified, and the novel unknown attacks are used to trigger closed-loop iteration of model parameters.

[0014] Furthermore, within a set time window, multi-dimensional features of network attack samples are extracted and concatenated into an original feature vector, specifically including:

[0015] Initialize and start the synchronous listening time window, and acquire the static feature set, dynamic feature set, traffic feature set and environment-related feature set of network attack samples in parallel within the synchronous listening time window;

[0016] At the end of the synchronous listening time window, the static feature set, dynamic feature set, traffic feature set, and environment-related feature set are concatenated in a fixed dimensional order to output a raw feature vector of uniform length.

[0017] Furthermore, the probability distribution of discrete states of each feature dimension of the original feature vector in the sample set is statistically analyzed, and the feature fusion weights are calculated. Specifically, this includes:

[0018] The equal-frequency binning algorithm is used to map the continuously fluctuating values ​​contained in the original feature vector into a finite number of discrete state intervals. For feature dimensions belonging to discrete types, the category state of the discrete type remains unchanged.

[0019] Calculate the empirical probability of each discrete state of each feature dimension occurring in the entire sample set;

[0020] Based on empirical probabilities, the information entropy values ​​carried by each feature dimension in the sample space are calculated using the information entropy measurement formula.

[0021] Furthermore, after the information entropy is calculated, it also includes:

[0022] Construct an exponential smoothing mapping function that introduces a minimal smoothing constant;

[0023] Substitute the information entropy value into the exponential smoothing mapping function for transformation to obtain the feature fusion weights of attributes that are negatively correlated with the information entropy value.

[0024] Extract the feature fusion weights corresponding to each feature dimension, perform weighted multiplication on the corresponding dimension values ​​in the original feature vector, and generate a weighted feature vector.

[0025] Furthermore, eigenvalue decomposition and linear transformation are performed on the weighted eigenvectors to generate dimensionality-reduced eigenvectors, specifically including:

[0026] The weighted feature vectors corresponding to all samples are arranged in rows and centered to generate a global feature data matrix, and the corresponding covariance matrix is ​​calculated.

[0027] Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues ​​and the corresponding eigenvectors of each eigenvalue;

[0028] Extract the largest eigenvalue that ranks first, and construct a projection matrix by extracting the eigenvector corresponding to the largest eigenvalue. Use the projection matrix to perform a linear transformation on the weighted eigenvector to generate a dimensionality-reduced eigenvector in the low-dimensional feature space.

[0029] Furthermore, the core point threshold parameters and neighborhood radius parameters of the adaptive density clustering algorithm specifically include:

[0030] Read the feature dimension values ​​of the reduced feature vector and the total number of network attack samples to be detected;

[0031] Compare twice the value of the feature dimension with the rounded-down value of the natural logarithm of the total number, and extract the maximum value between the twice value and the rounded-down value as the core threshold parameter of the density clustering algorithm.

[0032] Furthermore, after the core point threshold parameters are set, the following also applies:

[0033] Calculate the Euclidean distance between the dimensionality-reduced feature vectors of all samples to construct a global sample distance matrix. For each network attack sample, extract the nearest neighbor distance parameter with the sorting position as the core point threshold parameter.

[0034] The nearest neighbor distance parameters of all samples are sorted in descending order to construct a discrete initial distance function, and a Gaussian kernel function is called to perform smoothing filtering to generate a smooth distance curve function.

[0035] The central difference method is used to obtain the approximate second derivative of the smooth distance curve function. The target index number that makes the second derivative value reach the global maximum is searched, and the corresponding smooth distance curve function value is extracted and set as the neighborhood radius parameter.

[0036] Furthermore, clustering operations are performed on the dimensionality-reduced feature vectors to output effective clusters and isolated samples, specifically including:

[0037] The core point threshold parameter and the neighborhood radius parameter are input into the density clustering algorithm model to complete the parameter initialization;

[0038] Perform unsupervised density connectivity determination calculations on all dimensionality-reduced feature vectors;

[0039] The dimensionality-reduced feature vectors that meet the density reachability condition are grouped into independent effective clusters, and the discrete dimensionality-reduced feature vectors that do not meet the density connectivity condition of any core point are uniformly assigned to the isolated sample set representing abnormal noise.

[0040] Furthermore, based on the similarity between effective clusters and a known attack feature library, known attack variants and novel unknown attacks are identified, specifically including:

[0041] The arithmetic mean of the dimensionality values ​​of each dimensionality-reduced feature vector within the effective cluster is calculated to generate the cluster centroid vector representing the overall data distribution characteristics.

[0042] The cosine similarity algorithm is used to calculate the cosine value of the spatial angle between the cluster centroid vector and each benchmark feature vector in the known attack feature library;

[0043] Valid clusters with a maximum cosine similarity value greater than or equal to a preset similarity threshold are identified as variants of known attacks, while low-similarity clusters with a maximum cosine similarity value strictly less than a preset similarity threshold and all isolated samples are identified as novel unknown attacks.

[0044] Furthermore, the closed-loop iteration of model parameters is triggered by novel, unknown attacks, specifically including:

[0045] Extract the centroid vectors of effective clusters with low similarity and the dimensionality reduction feature vectors corresponding to each isolated sample, and add them as new benchmark feature vectors to the known attack feature library.

[0046] The newly added raw feature data of the underlying samples is captured and combined into the current global sample dataset;

[0047] Based on the expanded global sample dataset, the binning probability statistics and distance matrix differentiation are re-executed to complete the closed-loop iterative update operation of the core parameters of the feature fusion weight and density clustering algorithm.

[0048] A second aspect of the present invention provides an automatic clustering and identification system for network attack samples based on multi-feature fusion, used to implement the automatic clustering and identification method for network attack samples based on multi-feature fusion of any one of the first aspects described above. The system includes:

[0049] The feature acquisition module is used to extract multi-dimensional features of network attack samples within a set time window and concatenate them into an original feature vector.

[0050] The feature processing module, connected to the feature acquisition module, is used to statistically analyze the discrete state distribution probability of each feature dimension of the original feature vector in the sample set and calculate the feature fusion weights, and generate a weighted feature vector using the feature fusion weights.

[0051] The feature dimensionality reduction module, connected to the feature processing module, is used to perform eigenvalue decomposition and linear transformation on the weighted feature vector to generate a dimensionality-reduced feature vector.

[0052] The clustering derivation module, connected to the feature dimensionality reduction module, is used to adaptively calculate the core point threshold parameters and neighborhood radius parameters of the density clustering algorithm, and to perform clustering operations on the dimensionality-reduced feature vectors to output effective clusters and isolated samples.

[0053] The identification and matching module, connected to the clustering inference module, is used to identify known attack variants and new unknown attacks based on the similarity between effective clusters and a known attack feature library.

[0054] The model iteration module, connected to the identification and matching module, is used to trigger closed-loop iteration of model parameters using novel unknown attacks.

[0055] In summary, the present invention has at least one of the following beneficial technical effects:

[0056] 1. This invention aligns the asynchronous deviations in time periods between static file parsing and sandbox dynamic monitoring by setting a synchronous monitoring time window to collect multi-dimensional features of network attack samples in parallel, thus avoiding feature misalignment caused by splicing multi-source heterogeneous data. Simultaneously, by utilizing equal-frequency binning to statistically analyze sample distribution probabilities and calculating feature fusion weights based on information entropy, the proportions of each dimension can be objectively adjusted according to data distribution patterns, suppressing interference from redundant features and improving the accuracy of the weighted feature vector in subsequent spatial measurements.

[0057] 2. This invention constructs a global nearest neighbor distance descending function, introduces Gaussian kernel filtering and the central difference method to solve for the global maximum of the second derivative, and combines this with adaptive calculation of the core point threshold and neighborhood radius parameters of the density clustering algorithm based on sample dimension and size. This mechanism eliminates the reliance on human experience and presets, and can objectively map the geometric boundaries of dense and discrete regions in the sample manifold distribution, solving the problems of inaccurate boundary identification and high false negative rate of traditional clustering algorithms when facing network attack feature spaces with uneven density.

[0058] 3. This invention utilizes the cluster centroid vector and a known attack feature library to calculate the cosine similarity of the spatial angle between them, distinguishing between known attack variants and novel unknown attacks. Clusters identified as novel attacks, along with isolated samples, are used as baseline features to supplement the feature library, triggering the aggregation of underlying features and the recalculation of clustering parameters. This closed-loop iterative mechanism enables the identification system to automatically update feature weights and clustering model parameters according to the evolution of attack methods, maintaining the defense model's continuous detection capability against unknown malicious behaviors and avoiding the lag and aging of detection rules. Attached Figure Description

[0059] Figure 1 This is a system architecture diagram of the present invention;

[0060] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0061] Figure 3 This is a comparison chart of the feature dimension information entropy and fusion weight calculation results of the present invention;

[0062] Figure 4 This is a comparison chart of the performance evaluation of different models of the present invention on a hybrid attack dataset;

[0063] Figure 5 This is a diagram illustrating the distance curve and second derivative optimization process of the present invention. Detailed Implementation

[0064] The following is in conjunction with the appendix Figure 1 -Appendix Figure 5 The present invention will be further described in detail below.

[0065] like Figure 1 As shown, this invention provides an automatic clustering and identification system for network attack samples based on multi-feature fusion, which may include:

[0066] The feature acquisition module is used to perform multi-dimensional feature extraction on the input network attack samples.

[0067] The feature processing module, connected to the feature acquisition module, is used to perform data cleaning, discretization transformation, and calculation and allocation of feature fusion weights on the extracted features.

[0068] The feature dimensionality reduction module, connected to the feature processing module, is used to perform spatial transformation on the weighted feature vectors to reduce the feature dimension value.

[0069] The clustering derivation module, connected to the feature dimensionality reduction module, is used to calculate the running parameters of the clustering algorithm based on the feature distribution data after dimensionality reduction, and to perform unsupervised clustering operations on the samples.

[0070] The identification and matching module, connected to the clustering inference module, is used to compare the clustering results with a pre-set library of known attack features and output the identification results.

[0071] The model iteration module, connected to the identification and matching module, is used to update the features of newly added attack samples to the known attack feature library and trigger the update operation of calculation parameters.

[0072] like Figure 2 As shown, this invention provides an automatic clustering and identification method for network attack samples based on multi-feature fusion, comprising the following steps:

[0073] S100, the feature acquisition module receives the network attack sample to be detected, and extracts the static features, dynamic features, traffic features and environmental correlation features of the network attack sample in parallel within the set time window. At the end of the time window, the extracted features are spliced ​​together to form the original feature vector.

[0074] S200, the feature processing module performs binning on the continuous numerical features contained in the original feature vector, converts the continuous numerical features into discrete states, calculates the distribution probability of the discrete states of each feature dimension in the sample set, calculates the information entropy of each feature dimension based on the distribution probability, substitutes the calculated information entropy into the exponential smoothing mapping function, obtains the feature fusion weights corresponding to each feature dimension, and uses the feature fusion weights to perform weighted multiplication on the original feature vector to generate a weighted feature vector.

[0075] S300, the feature dimensionality reduction module extracts the weighted feature vectors corresponding to all samples, calculates the covariance matrix composed of the weighted feature vectors, performs eigenvalue decomposition on the covariance matrix, extracts the feature vector corresponding to the largest eigenvalue to construct the projection matrix, and uses the projection matrix to perform a linear transformation on the weighted feature vectors, mapping the weighted feature vectors in the high-dimensional space to the dimensionality-reduced feature vectors in the low-dimensional space.

[0076] S400: The clustering derivation module obtains the feature dimension values ​​and total number of samples of the dimensionality-reduced feature vectors, calculates the core point threshold parameters of the density clustering algorithm, calculates the Euclidean distance between each dimensionality-reduced feature vector, extracts the nearest neighbor distance corresponding to the core point threshold parameters, constructs a mapping function between the sample index and the nearest neighbor distance, calculates the second derivative of the mapping function, extracts the distance value corresponding to the global maximum value of the second derivative, sets this distance value as the neighborhood radius parameter of the density clustering algorithm, initializes the density clustering algorithm with the calculated core point threshold parameters and neighborhood radius parameters, performs clustering operations on the dimensionality-reduced feature vectors, and outputs the cluster and isolated sample partitioning results.

[0077] S500, the identification and matching module calculates the mean of the dimensionality-reduced feature vectors contained in each cluster, generates the cluster centroid vector, calculates the cosine similarity between the cluster centroid vector and each benchmark feature vector in the known attack feature library, determines the clusters whose cosine similarity meets the preset threshold as a variant set of known attacks, and determines the clusters whose cosine similarity is lower than the preset threshold and isolated samples as new unknown attacks. The model iteration module adds the cluster centroid vectors and isolated sample feature vectors of the new unknown attacks to the known attack feature library, and controls the system to re-execute steps S200 to S400 based on the updated known attack feature library to complete the update of the system feature fusion weights and clustering parameters.

[0078] The technical solution provided by the present invention will be described in detail below with reference to specific embodiments.

[0079] To address the issue of varying time consumption for different types of features during the acquisition process in the multi-dimensional feature extraction operation of network attack samples involved in step S100 above, the feature extraction and alignment concatenation are specifically performed through the following sub-steps:

[0080] In step S101, the feature acquisition module obtains a set containing multiple network attack samples to be detected. For any network attack sample in the set, the feature acquisition module initializes and starts the synchronous monitoring time window parameter. The purpose of setting this synchronous monitoring time window parameter is to align the asynchronous deviation between the dynamic running time of the sandbox and the static parsing time of the file, ensuring that all dimensional feature data collected for a single network attack sample are in the same lifecycle stage.

[0081] In step S102, during the duration of the synchronous listening time window, the feature acquisition module performs feature acquisition operations in four dimensions in parallel.

[0082] The feature acquisition module calls its internal static file parsing engine to obtain the static feature set of the network attack sample. The extracted static attributes include malicious code hash values, file format type values, and code instruction opcode sequences.

[0083] For the operation of obtaining code instruction opcode sequences by decompiling binary files, those skilled in the art can use existing standard disassembly analysis tools. The specific underlying decompilation and parsing principles are well-known technologies in this field and will not be elaborated here.

[0084] The feature acquisition module schedules the network attack sample to run within a pre-built isolated sandbox environment, capturing the dynamic feature set generated within the synchronous monitoring time window. The intercepted dynamic behavioral data covers process creation behavior sequences, memory loading behavior sequences, and registry operation records.

[0085] The feature acquisition module controls the bypass traffic monitoring component to intercept network data packets sent out by the network attack sample while it is running in the sandbox, and parses and generates a set of traffic features. The extracted transmission features include data packet size distribution, transmission protocol type encoding, and external network access frequency.

[0086] The feature acquisition module calls the system configuration interface of the isolated sandbox environment to read the set of environment-related features. The extracted related data includes the target system version parameters of the current virtual machine and the execution permission level parameters of the current process.

[0087] In step S103, when the timing value of the synchronous monitoring time window reaches the preset end threshold, the feature acquisition module forcibly terminates the sandbox operation and traffic monitoring operation. The feature acquisition module concatenates the acquired static feature set, dynamic feature set, traffic feature set, and environment-related feature set according to a fixed dimensional order, and outputs them as a raw feature vector of uniform length.

[0088] The mathematical calculation logic of this feature vector concatenation process is as follows:

[0089] ;

[0090] In the formula, For the first The original feature vector assembled from several network attack samples; For the first A row vector consisting of the static feature set of a network attack sample; For the first A row vector consisting of the dynamic feature set of a network attack sample; For the first A row vector consisting of the set of traffic features of a network attack sample; For the first A row vector consisting of the set of environmental features associated with each network attack sample; This is the identifier for the matrix transpose operation.

[0091] After concatenation, the feature acquisition module constructs a comprehensive vector structure containing all dimensional attributes. The expanded original feature vector is represented as follows:

[0092] ;

[0093] In the formula, For the first The original feature vector assembled from several network attack samples; For the first The network attack sample was the first one after splicing and alignment. Each dimension of the feature can take values; This represents the total dimension value after splicing and accumulating multi-source heterogeneous features.

[0094] Through the unified vectorized representation described above, the feature acquisition module transforms the heterogeneous raw acquisition data into a standardized input format suitable for subsequent numerical calculations.

[0095] To address the feature processing and weight calculation operations involved in step S200 above, and in order to unify the mathematical expression format of continuous numerical features and discrete categorical features and to achieve automated weight allocation, the following sub-steps are used to perform data transformation and calculation:

[0096] In step S201, the feature processing module receives the concatenated original feature vector. The feature processing module iterates through all feature dimensions contained in the original feature vector and performs targeted format conversion.

[0097] For feature dimensions belonging to the continuous numerical type, the feature processing module uses an equal-frequency binning algorithm to map continuously fluctuating values ​​into a finite number of discrete state intervals.

[0098] For routine data cleaning operations such as filling missing values ​​and removing outliers in the original feature data, those skilled in the art can use existing mean imputation or isolated forest algorithms. The specific data cleaning principles are well-known technologies in this field and will not be elaborated here.

[0099] For feature dimensions belonging to discrete types, the feature processing module retains their original category states unchanged. After discretization and unified transformation, the feature processing module obtains the set of discrete states corresponding to each feature dimension.

[0100] Step S202: The feature processing module counts the frequency of each discrete state for each feature dimension in the entire sample set. Based on this frequency, the feature processing module calculates the empirical probability corresponding to each discrete state. The mathematical formula for calculating the probability distribution is:

[0101] ;

[0102] In the formula, For the first The first feature dimension The probability distribution of a discrete state; For the th sample in the entire sample set The value of the i-th feature dimension is the i-th feature dimension. The number of samples in each discrete state; This represents the total number of all network attack samples to be detected.

[0103] Step S203: The feature processing module extracts the probability distribution values ​​of each discrete state and calculates the information uncertainty value carried by each feature dimension in the sample space using the information entropy measurement formula. The mathematical formula for calculating information entropy is:

[0104] ;

[0105] In the formula, For the first The information entropy value of each feature dimension; For the first The total number of discrete states contained in each feature dimension after binning; For the first The first feature dimension The probability distribution of a discrete state; This is a base-2 logarithmic function.

[0106] Step S204: The more concentrated the distribution of a specific feature dimension in the sample population, the lower its calculated information entropy value, objectively indicating that the dimension has a more significant discriminative attribute. The feature processing module constructs an exponential smoothing mapping function to convert the calculated information entropy value into feature fusion weights for negatively correlated attributes.

[0107] To avoid zero-value weights in the calculation results, which could lead to the loss of underlying feature attributes, the feature processing module introduces a minimal smoothing constant within the mapping function for numerical adjustment. The derivation formula for the feature fusion weights is as follows:

[0108] ;

[0109] In the formula, To be assigned to the Feature fusion weights for each feature dimension; For the first The information entropy value of each feature dimension; It is a natural constant; This is a preset minimum smoothing constant; This represents the total dimension of the original feature vector; For the first The information entropy value of each feature dimension.

[0110] Step S205: The feature processing module extracts the feature fusion weights corresponding to each feature dimension, and uses these weights to perform weighted multiplication operations on the corresponding dimension values ​​in the original feature vector. After completing the traversal calculations for all dimensions, the feature processing module outputs a weighted feature vector that eliminates feature redundancy interference. The weighted calculation logic is expressed as follows:

[0111] ;

[0112] In the formula, For the first A weighted feature vector generated by weighting a network attack sample; These are the feature fusion weights corresponding to the 1st to Mth feature dimensions, respectively; The first The feature values ​​of the first to Mth dimensions in the original feature vector of a network attack sample; This is the identifier for the matrix transpose operation.

[0113] To address the high-dimensional manifold feature dimensionality reduction operation involved in step S300 above, and to overcome the curse of dimensionality caused by the convergence of Euclidean distances between samples in the high-dimensional feature space, the orthogonal spatial projection and dimensionality reduction transformation of the features are specifically performed through the following sub-steps:

[0114] Step S301: The feature dimensionality reduction module obtains the weighted feature vectors corresponding to the entire network attack sample set, arranges all weighted feature vectors by row, and constructs a global feature data matrix with dimension N×M.

[0115] The feature dimensionality reduction module calculates the mean of each column of the global feature data matrix and subtracts the mean of its corresponding column from each element in the matrix, thus completing the centering process of the feature data.

[0116] For the standard mathematical operations of centering the feature matrix, those skilled in the art can use existing linear algebra operation libraries to implement it. The specific mean subtraction operation is a well-known technique in the field and will not be described in detail here.

[0117] In step S302, the feature dimensionality reduction module calculates the covariance matrix between feature dimensions based on the centralized global feature data matrix. This covariance matrix objectively represents the correlation distribution pattern of data in different feature dimensions. The feature dimensionality reduction module performs eigenvalue decomposition on this covariance matrix to solve for all eigenvalues ​​of the covariance matrix and the eigenvectors corresponding to each eigenvalue.

[0118] Step S303: The feature dimensionality reduction module extracts all the solved feature values ​​and sorts them in descending order of numerical value. The feature dimensionality reduction module truncates the values ​​ranked first. The largest eigenvalue of each, and extract this The feature vectors are the eigenvectors corresponding to the largest eigenvalues. The feature reduction module combines the extracted feature vectors column-wise to construct the projection matrix required for the dimensionality reduction transformation. The extracted feature vectors retain the main information components with the largest variance in the original high-dimensional dataset.

[0119] Step S304: The feature dimensionality reduction module uses the constructed projection matrix to perform a linear transformation calculation on the weighted feature vectors in the high-dimensional space, generating dimensionality-reduced feature vectors in the low-dimensional feature space. The linear transformation calculation logic of this spatial projection is expressed as follows:

[0120] ;

[0121] In the formula, For the first Low-dimensional reduced-dimensional feature vectors generated by mapping network attack samples; For the reason before The projection matrix constructed by combining the eigenvectors corresponding to the largest eigenvalues; This is the transpose of the projection matrix; For the first The high-dimensional weighted feature vector corresponding to each network attack sample before dimensionality reduction.

[0122] Through the above linear transformation operation, the feature dimensionality reduction module compresses the high-dimensional sparse feature vector of length M into a vector of length M. Low-dimensional dense feature vectors eliminate redundant and minimally variable dimensional components in the original data, restoring the effectiveness of Euclidean distance in measuring dissimilarity between samples in geometric space, and providing a numerical basis that meets the distance metric requirements for subsequent density clustering calculations.

[0123] To address the clustering calculation operation involved in step S400 above, and to overcome the shortcomings of conventional density clustering algorithms that rely on pre-set parameters based on human experience, adaptive parameter calculation based on the data distribution manifold morphology is achieved. Specifically, the calculation is performed through the following sub-steps:

[0124] Step S401: The clustering derivation module reads the feature dimension values ​​of the dimensionality-reduced feature vectors and the total number of network attack samples to be detected. Based on the correspondence between dimension values ​​and sample size, the clustering derivation module automatically calculates the core point threshold parameter of the density clustering algorithm using heuristic rules. This parameter limits the minimum number of samples required to form a locally dense cluster. The calculation formula is:

[0125] ;

[0126] In the formula, The threshold parameter is the core point parameter of the density clustering algorithm; A function to extract the maximum value; The numerical value of the feature dimension of the dimensionality-reduced feature vector; This is the floor function; It is a logarithmic function with the natural constant as its base; This represents the total number of network attack samples to be detected.

[0127] In step S402, the clustering derivation module calculates the pairwise Euclidean distances between the dimensionality-reduced feature vectors of all network attack samples, and constructs a global sample distance matrix.

[0128] For each network attack sample, the clustering derivation module extracts the distance values ​​from that sample to all other samples from the global sample distance matrix and sorts them in ascending order. The clustering derivation module truncates the sorting position to the [number]th [value]. The distance value of the bit is used as the nearest neighbor distance parameter for the sample.

[0129] The clustering derivation module summarizes the nearest neighbor distance parameters for all samples and sorts them in descending order of their values. Using the sorted sample index as the independent variable and the corresponding nearest neighbor distance parameter as the dependent variable, the module constructs a discrete initial distance function.

[0130] In step S403, to eliminate the interference of function abrupt changes caused by local data noise, the clustering derivation module calls the Gaussian kernel function to perform smoothing filtering on the initial distance function, generating a smooth distance curve function.

[0131] The inflection point where the dependent variable value in the smooth distance curve function drops sharply objectively maps the distribution boundary between dense and discrete regions in the cluster space.

[0132] The clustering derivation module uses the central difference method to obtain an approximate second derivative of the smooth distance curve function. The calculation formula is:

[0133] ;

[0134] In the formula, To smooth the distance curve function with independent variable as The value of the second derivative at that point; The independent variable is The function value of the smooth distance curve function corresponding to the time; The independent variable is The function value of the smooth distance curve function corresponding to the time; The independent variable is The function value of the smooth distance curve function corresponding to the time; The step size parameter for the difference calculation is preset to a value of 1; This is the sample index number after sorting in descending order.

[0135] The clustering derivation module iterates through all sample index numbers and searches for the target index number in the calculated derivative set that makes the second derivative value reach the global maximum. The clustering derivation module extracts the smooth distance curve function value corresponding to the target index number and directly sets it as the optimal neighborhood radius parameter of the density clustering algorithm.

[0136] In step S404, the clustering derivation module inputs the core point threshold parameters and optimal neighborhood radius parameters obtained from the adaptive solution into the density clustering algorithm model to complete parameter initialization. Based on these set parameters, the clustering derivation module performs unsupervised density connectivity determination calculations on all dimensionality-reduced feature vectors.

[0137] Dimensionally reduced feature vectors that meet the density reachability condition are grouped into independent effective clusters. Discrete dimensionality-reduced feature vectors that do not meet any core point density connectivity condition are uniformly assigned to an isolated sample set representing anomalous noise.

[0138] For the specific core point determination, density direct access, and density connection path expansion operations in density clustering algorithms, those skilled in the art can directly adopt the underlying execution logic of existing density-based clustering algorithms. The specific cluster expansion principle is a well-known technology in this field and will not be elaborated here.

[0139] Regarding the matching identification and system update operations involved in step S500 above, in order to achieve accurate classification of known attack variants and self-evolution of the defense model, the calculations are specifically performed through the following sub-steps:

[0140] Step S501: For each valid cluster output by the clustering derivation module, the identification and matching module extracts the dimensionality-reduced feature vectors corresponding to all network attack samples contained within the cluster.

[0141] The identification and matching module calculates the arithmetic mean of the values ​​of each dimension-reduced feature vector within the cluster in each dimension, and generates a cluster centroid vector that represents the overall data distribution characteristics of the effective cluster.

[0142] Step S502: The identification and matching module retrieves a pre-set known attack feature database. This database stores a set of baseline feature vectors corresponding to various known network attack families. The identification and matching module traverses the known attack feature database and uses a cosine similarity algorithm to calculate the cosine value of the spatial angle between the cluster centroid vector and each baseline feature vector in the database.

[0143] The formula for calculating the cosine similarity score is:

[0144] ;

[0145] In the formula, Let the centroid vector of the target cluster and the first The cosine similarity measure of the baseline feature vectors; For the first The cluster centroid vectors of each effective cluster are calculated by arithmetic mean; The first one stored in the known attack signature database The baseline feature vectors corresponding to a known attack family; This is the result of the dot product of two vectors; Let be the norm modulus of the centroid vector of the target cluster; For the first The norm modulus of the baseline eigenvectors.

[0146] In step S503, the identification and matching module compares and filters all calculated cosine similarity metrics. When the maximum cosine similarity metric between the centroid vector of a valid cluster and a specific baseline feature vector in the database is greater than or equal to a preset similarity threshold, the identification and matching module determines that the valid cluster is highly homologous to the known attack family in the feature manifold space. Based on this, the identification and matching module classifies and outputs all network attack samples contained within the valid cluster as a variant set of the corresponding known attack.

[0147] When the maximum cosine similarity metric between the centroid vector of a valid cluster and all benchmark feature vectors in the database is strictly less than the preset similarity threshold, it objectively indicates that the valid cluster has deviated from the distribution space of the existing feature database.

[0148] Meanwhile, for isolated samples that were not included in any cluster in step S400, since they lack local density characteristics and cannot match known patterns, the identification and matching module uniformly judges low similarity clusters and all isolated samples as new unknown attacks.

[0149] In step S504, the model iteration module acquires the dataset identified as a novel unknown attack. The model iteration module extracts the centroid vectors of effective low-similarity clusters and the dimensionality-reduced feature vectors corresponding to each isolated sample. The model iteration module uses the extracted vector data as new baseline feature vectors and adds them to the known attack feature library, thereby expanding the feature library's recording capacity.

[0150] After the model iteration module completes the writing operation to the known attack feature database, it intercepts and summarizes the newly added original feature data of the underlying samples and combines it into the current global sample dataset. Based on the updated and expanded global sample dataset, the model iteration module triggers control commands to schedule the feature processing module to re-execute the operation flow of steps S200 to S400. The system completes the closed-loop iterative update operation of the feature fusion weights and the core parameters of the density clustering algorithm by re-executing the binning probability statistics and distance matrix differentiation.

[0151] Specific application examples:

[0152] To further illustrate the feasibility of the technical solution of this invention and the actual calculation process, the following explanation is based on numerical deductions of specific application scenarios and related experimental comparison data.

[0153] like Figure 3 In a specific enterprise network traffic monitoring scenario, the system intercepted a batch of samples to be detected, setting a total number of samples. Taking the first network attack sample as an example, the feature acquisition module captures and concatenates the data through a time window, and the output original feature vector covers four basic feature dimensions (simplified example), with specific values ​​of... .

[0154] The feature processing module iterates through all 2000 samples, performing discretization and frequency statistics. Taking the first dimension (e.g., packet size) as an example, it is found that its distribution concentration is relatively high. The corresponding empirical probability is substituted into the information entropy formula to calculate the information entropy value for that dimension. Similarly, the information entropy values ​​for the other three dimensions were calculated as follows: .

[0155] The feature processing module sets a minimum smoothing constant. The information entropy is substituted into the exponential smoothing mapping function to calculate the feature fusion weights. Taking the first dimension as an example, the calculation logic for its feature fusion weights is as follows:

[0156] ;

[0157] In the formula, The feature fusion weights for the first dimension; This is a natural constant. Similarly, calculations yield... , , .

[0158] The system uses the above weights to Perform weighting to generate weighted feature vectors This numerical result indicates that the lower the information entropy (such as the 4th dimension), the better. The higher the weight coefficient assigned, the more it highlights the influence of distance on the weight of high-discrimination features in space.

[0159] After the feature reduction module extracts the first two principal components to form the projection matrix, Mapping to a dimensionality-reduced feature vector in two-dimensional space In the clustering derivation module, the system is based on dimensionality reduction. and sample size Automatically calculates the core point threshold parameters .

[0160] The clustering derivation module extracts the 7th nearest neighbor distance of all samples to construct a smooth distance curve function. By taking the central difference derivative, it detects that the maximum value (inflection point) of the second derivative occurs at a distance of 4.35, and automatically sets the neighborhood radius parameter to 4.35 accordingly.

[0161] After clustering is completed, the identification and matching module calculates the centroid vector of a valid cluster and its cosine similarity with the baseline feature vector of a known botnet family. If the calculated cosine similarity value is 0.94, which is greater than the set threshold of 0.90, the identification and matching module directly classifies the samples contained in that cluster as a variant of that botnet.

[0162] like Figure 4 To verify the practical effect of the embodiments of the present invention, a test set was constructed by extracting 15,000 normal traffic data points and 5,000 mixed attack feature data points from a publicly available network intrusion dataset. The test control group included the traditional K-Means clustering model (empirically set cluster number K=15) and the standard DBSCAN algorithm model (empirically set...). ) and the basic joint model that removes the feature weighting module.

[0163] The experiment performed the same calculations on all models. Based on the confusion matrix output by the models, the true positive rate (i.e., detection rate) and false positive rate (i.e., false alarm rate) were statistically calculated. The experimental data statistics show that the traditional K-Means clustering model is limited by its inability to handle non-convex spherical cluster data, with a detection rate of only 82.4% and a false alarm rate as high as 11.2%.

[0164] The standard DBSCAN algorithm model, due to its use of fixed empirical values ​​for parameters, cannot adapt to the local density unevenness of the high-dimensional feature space, resulting in the erroneous omission of a large number of outlier variant samples, with a detection rate of 85.6%. The basic joint model, which removes the feature weighting module, fails to suppress the redundant variance interference caused by continuous numerical values, resulting in an identification and detection rate of 91.3%.

[0165] In contrast, the automatic clustering and identification method for network attack samples based on multi-feature fusion provided in this invention eliminates redundancy by adaptively adjusting dimensional weights through information entropy and relies entirely on the second derivative to optimize and solve clustering parameters, enabling the model to accurately fit the true high-dimensional data manifold boundary of the attack samples. Statistical analysis shows that the overall detection rate of this invention reaches 97.8%, and the corresponding false positive rate is reduced to 1.5%.

[0166] The above numerical comparisons objectively demonstrate the system's accurate identification capability and low false positive rate when facing complex network attack variants. By supplementing the knowledge base with the identified new and unknown attack features, subsequent test data verified that the model iteration module can achieve a dynamic gain in the detection rate of 0.5% to 0.8%, confirming the substantial technical effectiveness of the closed-loop iteration mechanism in preventing model aging.

[0167] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An automatic clustering and identification method for network attack samples based on multi-feature fusion, characterized in that, Includes the following steps: Extract multi-dimensional features of network attack samples within a set time window and concatenate them into an original feature vector; The probability distribution of the discrete states of each feature dimension of the original feature vector in the sample set is statistically analyzed and the feature fusion weights are calculated. The weighted feature vector is then generated using the feature fusion weights. The weighted feature vector is subjected to eigenvalue decomposition and linear transformation to generate a dimension-reduced feature vector; The core point threshold parameter and neighborhood radius parameter of the adaptive density clustering algorithm are used to perform clustering operations on the dimensionality-reduced feature vector and output effective clusters and isolated samples. Based on the similarity between the effective clusters and the known attack feature library, known attack variants and new unknown attacks are identified, and the new unknown attacks are used to trigger closed-loop iteration of model parameters.

2. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 1, characterized in that, The step of extracting multi-dimensional features of network attack samples within a set time window and concatenating them into an original feature vector specifically includes: Initialize and start a synchronous listening time window, and within the synchronous listening time window, acquire in parallel the static feature set, dynamic feature set, traffic feature set, and environment-related feature set of the network attack sample; At the end of the synchronous listening time window, the static feature set, dynamic feature set, traffic feature set, and environment-related feature set are concatenated in a fixed dimensional order to output the original feature vector of uniform length.

3. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 1, characterized in that, The process of statistically analyzing the discrete state distribution probability of each feature dimension of the original feature vector in the sample set and calculating the feature fusion weights specifically includes: The equal-frequency binning algorithm is used to map the continuously fluctuating values ​​contained in the original feature vector into a finite number of discrete state intervals. For feature dimensions belonging to discrete types, the category state of the discrete type remains unchanged. Calculate the empirical probability of each discrete state of each feature dimension occurring in the entire sample set; Based on the empirical probability, the information entropy value carried by each feature dimension in the sample space is calculated using the information entropy measurement formula.

4. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 3, characterized in that, After the information entropy value is calculated, the following is also included: Construct an exponential smoothing mapping function that introduces a minimal smoothing constant; Substitute the information entropy value into the exponential smoothing mapping function for transformation to obtain the feature fusion weights of attributes that are negatively correlated with the information entropy value; Extract the feature fusion weights corresponding to each feature dimension, perform a weighted multiplication operation on the corresponding dimension values ​​in the original feature vector, and generate the weighted feature vector.

5. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 1, characterized in that, The step of performing eigenvalue decomposition and linear transformation on the weighted feature vector to generate a dimension-reduced feature vector specifically includes: The weighted feature vectors corresponding to all samples are arranged in rows and centered to generate a global feature data matrix, and the corresponding covariance matrix is ​​calculated. Perform eigenvalue decomposition on the covariance matrix to solve for the eigenvalues ​​and the eigenvectors corresponding to each eigenvalue; Extract the largest eigenvalue that ranks first, and construct a projection matrix by extracting the eigenvector corresponding to the largest eigenvalue. Use the projection matrix to perform a linear transformation on the weighted eigenvector to generate the dimensionality-reduced eigenvector in the low-dimensional feature space.

6. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 1, characterized in that, The core point threshold parameter and neighborhood radius parameter of the adaptive density clustering algorithm specifically include: Read the feature dimension values ​​of the reduced feature vector and the total number of network attack samples to be detected; Compare twice the value of the feature dimension with the rounded-down value of the natural logarithm of the total number, and extract the maximum value between the twice value and the rounded-down value as the core point threshold parameter of the density clustering algorithm.

7. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 6, characterized in that, After the core point threshold parameter is set, the following is also included: Calculate the Euclidean distance between the dimensionality-reduced feature vectors of all samples to construct a global sample distance matrix. For each network attack sample, extract the nearest neighbor distance parameter whose sorting position is the threshold parameter of the core point. The nearest neighbor distance parameters of all samples are sorted in descending order to construct a discrete initial distance function, and a Gaussian kernel function is called to perform smoothing filtering to generate a smooth distance curve function. The approximate second derivative of the smooth distance curve function is obtained using the central difference method. The target index number that makes the second derivative value reach the global maximum is searched, and the corresponding smooth distance curve function value is extracted and set as the neighborhood radius parameter.

8. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 1, characterized in that, The step of performing clustering operations on the reduced-dimensional feature vectors to output effective clusters and isolated samples specifically includes: The core point threshold parameter and the neighborhood radius parameter are input into the density clustering algorithm model to complete parameter initialization; Perform unsupervised density connectivity determination calculations on all dimensionality-reduced feature vectors; The dimensionality-reduced feature vectors that meet the density reachability condition are grouped into independent effective clusters, and the discrete dimensionality-reduced feature vectors that do not meet the density connectivity condition of any core point are uniformly assigned to the isolated sample set representing abnormal noise.

9. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 1, characterized in that, The identification of known attack variants and novel unknown attacks based on the similarity between the effective clusters and the known attack feature library specifically includes: The arithmetic mean of the dimension values ​​of each dimension-reduced feature vector within the effective cluster is calculated to generate a cluster centroid vector representing the overall data distribution characteristics. The cosine similarity algorithm is used to calculate the cosine value of the spatial angle between the cluster centroid vector and each benchmark feature vector in the known attack feature library; Valid clusters with a maximum cosine similarity value greater than or equal to a preset similarity threshold are identified as variants of known attacks, while low-similarity clusters with a maximum cosine similarity value strictly less than a preset similarity threshold and all isolated samples are identified as the novel unknown attack.

10. The automatic clustering and identification method for network attack samples based on multi-feature fusion according to claim 9, characterized in that, The closed-loop iteration of the model parameters triggered by the novel unknown attack specifically includes: Extract the centroid vectors of the effective clusters with low similarity and the dimensionality reduction feature vectors corresponding to each isolated sample, and add them as new benchmark feature vectors to the known attack feature library; The newly added raw feature data of the underlying samples is captured and combined into the current global sample dataset; Based on the global sample dataset with expanded sample size, the binning probability statistics and distance matrix differentiation are re-executed to complete the closed-loop iterative update operation of the core parameters of the feature fusion weight and density clustering algorithm.

Images