Network intrusion detection method, system and device based on Hash bucket integration and storage medium
Through the integration of hash buckets and dynamic weighting, the high computational complexity and sample imbalance caused by high-dimensional traffic characteristics are solved, efficient network intrusion detection is achieved, and detection performance and computing efficiency are improved.
Patent Information
- Application Number
- CN202510703588.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-09-05
AI Technical Summary
The high computational complexity, large storage overhead caused by high dimensional traffic characteristics, and extremely unbalanced distribution of normal samples and attack samples, which are difficult to effectively deal with existing methods, resulting in low computational efficiency and insufficient detection accuracy of network intrusion detection.
The undersampling strategy based on hash buckets is adopted to divide the feature space into non-overlapping buckets, and the feature distribution is retained through the hash method, and combined with dynamic weighting integration, the classification accuracy and robustness of the classifier are improved.
It achieves efficient data distribution and balance, improves the classification accuracy and robustness of network intrusion detection, reduces computational complexity and storage requirements, and supports real-time processing of edge devices.
Smart Images
Figure CN120602134A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security technology, and in particular to a network intrusion detection method, system, device and storage medium based on hash bucket integration. Background Art
[0002] High-dimensional traffic features lead to high computational complexity and high storage overhead. Network intrusion detection requires processing high-dimensional traffic features. Traditional methods that directly use raw features result in exponentially increased computational complexity, making real-time processing difficult when processing millions of data points. Furthermore, high-dimensional features require a large amount of storage space, imposing hardware limitations on edge device deployment.
[0003] Furthermore, the extremely unbalanced distribution of normal and attack samples in network traffic can cause traditional machine learning models to be biased toward the majority class (i.e., normal samples) when training, reducing the accuracy of identifying attack samples. Existing methods for handling imbalanced classification have drawbacks. For example, random undersampling, which directly removes normal samples, disrupts traffic statistics, causing information loss and misleading model training. Clustering-based undersampling methods are too complex and unsuitable for large-scale network traffic scenarios. Interpolation-based oversampling methods, which generate synthetic samples in a high-dimensional sparse space, easily introduce noise, amplifying differences in attack signatures. Furthermore, increasing the number of samples increases the data size, further increasing the computational burden and leading to inefficient training.
[0004] To address the information loss and training inefficiencies inherent in traditional resampling methods, this paper proposes an undersampling strategy based on hash bucketing. This strategy efficiently divides the feature space into non-overlapping buckets using a hashing method, preserving feature distribution and avoiding information loss. Furthermore, undersampling based on hash bucketing achieves efficient data distribution balancing. Furthermore, through dynamic weighted ensemble, the classifier's accuracy and robustness for unbalanced traffic data are further improved. Summary of the Invention
[0005] To solve the above technical problems, a network intrusion detection method based on hash bucket integration is proposed, which includes feature extraction and normalization of network traffic data to obtain a standardized feature vector;
[0006] Processing the standardized feature vector based on a feature conversion strategy of nonlinear compression and quantization mapping to obtain a low-dimensional feature representation;
[0007] Perform principal component analysis on the low-dimensional feature representation to generate the corresponding binary hash code;
[0008] Divide the samples into multiple hash buckets according to the binary hash code, and perform undersampling on the majority class samples in each hash bucket in proportion to construct a class-balanced sample subset;
[0009] Repeat the hash bucketing and sampling operations to obtain several balanced sample subsets and train the base classifier;
[0010] Evaluate the performance indicators of each base classifier on the validation set, and calculate the integration weight of each base classifier based on the evaluation results;
[0011] When predicting the test sample, the final prediction category is output based on the weighted voting results of each base classifier.
[0012] As a preferred solution of the network intrusion detection method based on hash bucket integration described in the present invention, the feature extraction and standardization processing includes constructing feature vectors of the statistical features and time series features of network traffic data, and standardizing the features of each dimension in the feature vector based on the normalization calculation formula to eliminate the difference in feature dimensions.
[0013] As a preferred solution of the network intrusion detection method based on hash bucket integration described in the present invention, the feature conversion strategy includes performing nonlinear compression on the standardized feature vector through a neural network model to obtain a low-dimensional feature representation, and performing feature mapping conversion on the low-dimensional feature representation based on a feature space transformation method.
[0014] As a preferred solution of the network intrusion detection method based on hash bucket integration described in the present invention, wherein: the dividing of samples into multiple hash buckets according to binary hash codes and the proportional undersampling of majority class samples in each hash bucket include:
[0015] By calculating the sample quantity distribution in each hash bucket, the sampling quantity of the majority class samples is determined based on the preset class ratio relationship, and the corresponding number of majority class samples is selected from each hash bucket to construct a class-balanced sample subset.
[0016] As a preferred solution of the network intrusion detection method based on hash bucket integration described in the present invention, wherein: the training base classifier includes:
[0017] Each base classifier is trained independently using a balanced sample subset with the same model structure and parameter configuration.
[0018] As a preferred solution of the network intrusion detection method based on hash bucket integration described in the present invention, wherein: the performance indicators of each base classifier are evaluated on the validation set, and the integration weight of each base classifier is calculated according to the evaluation results, including:
[0019] According to the performance index of each base classifier on the validation set, the integration weight of each base classifier is determined by calculating the ratio of the corresponding performance index value of each base classifier to the sum of the performance index values of all base classifiers;
[0020] The performance indicators include balanced accuracy and receiver operating characteristic curve area.
[0021] As a preferred solution of the network intrusion detection method based on hash bucket integration described in the present invention, when predicting the sample to be tested, the final prediction category output based on the weighted voting results of each base classifier includes:
[0022] The samples to be tested are input into each base classifier respectively, the classification results are obtained and based on the integrated weights of each base classifier, all classification results are accumulated and counted according to the weights, and the category with the largest accumulated value is selected as the final prediction result.
[0023] Another object of the present invention is to provide a network intrusion detection system based on hash bucket integration. The present invention solves the problems in the prior art of high computational complexity, large storage overhead, and insufficient detection accuracy caused by extremely unbalanced distribution of normal samples and attack samples due to high-dimensional traffic features. Through specific technical solutions such as feature compression, hash coding, bucket sampling, and classifier integration, effective processing of network traffic data and efficient training of classification models are achieved to support the actual application needs of network intrusion detection.
[0024] As a preferred solution of the network intrusion detection system based on hash bucket integration described in the present invention, it is characterized by including a feature processing module, a feature conversion module, a data sampling module and a classification prediction module
[0025] The feature processing module extracts statistical features and time series features from the network traffic data, and performs standardization processing on the extracted feature vectors to eliminate the dimensional differences between the features and form a standardized feature vector;
[0026] The feature conversion module compresses the normalized feature vector into a low-dimensional feature representation using a nonlinear compression method, and encodes the low-dimensional feature representation based on a feature space mapping method to generate a hash code for hash bucketing;
[0027] The data sampling module divides the samples into multiple hash buckets according to the hash code, and performs undersampling on the majority class samples according to a preset ratio based on the sample category distribution in each hash bucket to generate a class-balanced sample subset;
[0028] The classification prediction module trains multiple base classifiers based on the sample subset, and determines the integration weight according to the performance index of each base classifier on the validation set. When predicting the test sample, the integration weight is used for weighted calculation to output the final prediction category.
[0029] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the network intrusion detection method based on hash bucket integration when executing the computer program.
[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the network intrusion detection method based on hash bucket integration.
[0031] Beneficial effects of the present invention: The present invention achieves balanced category distribution by sampling majority class samples through undersampling methods, but this can easily cause information loss and ignore important attack sample information. Compared with the prior art, the present invention uses efficient iterative quantized hashing to spatially partition data into hash buckets. Sampling from the buckets can effectively retain the distribution information of the data and avoid the information loss that is easily caused by existing undersampling methods. At the same time, through diversified random sampling and dynamic weighted integration, the present invention can effectively enhance the classification effect of the classifier on network traffic data with imbalanced categories and improve the detection performance of network intrusions. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0033] Figure 1 The overall flow chart of a network intrusion detection method based on hash bucket integration provided by one embodiment of the present invention. DETAILED DESCRIPTION
[0034] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.
[0035] Example 1, reference Figure 1 , which is the first embodiment of the present invention, provides a network intrusion detection method based on hash bucket integration, including:
[0036] S1. Perform feature extraction and standardization on network traffic data to obtain a standardized feature vector.
[0037] Network data includes normal traffic and attack traffic.
[0038] Extract features from network traffic data, including:
[0039] 1) Basic connection characteristics: connection duration (seconds), protocol type, target host service type, and connection status;
[0040] 2) Traffic characteristics: the number of data bytes from the source host to the destination host, the number of data bytes from the destination host to the source host, and whether it is a local connection (1 = yes, 0 = no);
[0041] 3) Error and anomaly characteristics: number of incorrect fragments, number of emergency data packets, number of sensitive files or directories accessed, number of login failures, whether the login was successful (1=yes, 0=no), number of times the target host was attacked, whether a root shell was obtained (1=yes, 0=no), whether the su command was attempted (1=yes, 0=no), number of root user operations, number of file creations, number of shell startups, number of control file accesses, number of outbound commands, whether it was a hosts.equiv login (1=yes, 0=no), whether it was a guest login (1=yes, 0=no)
[0042] 4) Time window statistical features: number of connections identical to the current connection in the past 2 seconds, number of connections identical to the current service in the past 2 seconds, SYN error ratio in connections in the past 2 seconds, SYN error ratio for the same service, REJ error ratio in connections in the past 2 seconds, REJ error ratio for the same service, connection ratio for the same service in the past 2 seconds, connection ratio for different services in the past 2 seconds, and connection ratio for the same service but different destination hosts
[0043] 5) Target host statistical characteristics: number of connections with the same target host in the top 100 connections, number of connections with the same service as the target host in the top 100 connections, connection ratio of the target host with the same service, connection ratio of the target host with different services, connection ratio of the target host with the same source port, connection ratio of the target host with the same service but different source hosts, SYN error ratio of the target host, SYN error ratio of the target host with the same service, REJ error ratio of the target host, and REJ error ratio of the target host with the same service.
[0044] The above 41 features are combined with the actual connection type (normal or attack type such as denial of service attack) corresponding to each traffic data to obtain the initial training data set.
[0045] Standardization: The z-score method is used to standardize continuous features to eliminate dimensional differences. The formula is as follows:
[0046]
[0047] Where x is the original eigenvalue, μ and σ are the mean and standard deviation respectively, and x norm is to standardize the feature values, where N is the number of samples in the dataset. This step unifies the numerical ranges of different features and eliminates dimensional differences.
[0048] S2. Process the standardized feature vector based on a feature conversion strategy of nonlinear compression and quantization mapping to obtain a low-dimensional feature representation.
[0049] A preferred embodiment of the neural network model in the present invention uses an autoencoder to compress and represent high-dimensional features, thereby improving the feature expression capability. The autoencoder is constructed using a multi-layer perceptron (MLP) as a prototype, including an encoder and a decoder, which can be expressed as follows.
[0050] Encoder architecture: The input features are compressed through MLP, which consists of two hidden layers. The first hidden layer takes the normalized feature values as input and obtains the dimensionally compressed output through affine transformation and nonlinear activation function, which can be expressed as follows:
[0051] h1=ReLU(W1·x norm +b1)
[0052] The second hidden layer takes the output of the first layer as input, and undergoes another affine transformation and nonlinear activation function to further compress the output dimension, which can be expressed as follows:
[0053] z=ReLU(W2·h1+b2)
[0054] In the above formula, W2 and W1 are weight matrices, b2 and b1 are bias terms, ReLU is a nonlinear activation function, h1 is the output of the first hidden layer of the encoder, and z is the output of the second hidden layer of the encoder.
[0055] The decoder adopts a symmetrical structure and optimizes the encoder's representational capabilities by reconstructing the input. It consists of a reconstruction layer and an output layer. The reconstruction layer takes the encoder output as input and expands the feature dimension through affine transformation and nonlinear activation function. It can be expressed as follows:
[0056] h2=ReLU(W′1·z+b′1)
[0057] The output layer takes the output of the reconstruction layer as input, and after another affine transformation and nonlinear activation function, expands the feature dimension to the input dimension of the encoder, which can be expressed as follows:
[0058]
[0059] Loss function: Minimize reconstruction error (mean square error):
[0060]
[0061] Autoencoders extract essential data features through nonlinear transformations while filtering out noise (such as sudden traffic interference), providing a robust low-dimensional representation for subsequent hash coding. Autoencoders can be optimized using modern optimizers, such as stochastic gradient descent, with a learning rate of 1e-3, a batch size of 128, and a training cycle of 200 epochs. An early stopping strategy is used, stopping optimization when the validation set loss decreases by less than 1e-4 for 10 consecutive epochs.
[0062] An optional embodiment of the neural network model in the present invention is a convolutional neural network;
[0063] When computing resources are limited in a specific network environment or device, and traffic characteristics exhibit localized variations, a convolutional neural network can be used instead of an autoencoder for feature compression. The normalized feature vector is fed into the convolutional neural network, where the convolutional layer extracts local features. The pooling layer reduces the feature dimension and outputs a low-dimensional feature representation.
[0064] This method is suitable for scenarios where there are strong local changes or periodic characteristics in the traffic characteristics, such as the detection of sudden network attack traffic.
[0065] During the quantization mapping phase, a mean-shift clustering algorithm can be used to discretize low-dimensional features. Classification boundaries are determined by calculating the peak density of the feature distribution. Each sample is assigned to a corresponding class center, and a binary hash code is generated based on the class center number. This method can be used in certain specific deployment scenarios of the present invention, such as environments with low real-time computing requirements, moderate data scale, and the need for rapid deployment, but it is not a preferred technical path.
[0066] It should be further explained that:
[0067] While CNNs can extract local feature correlations, their core design is only suitable for data with clear local spatial relationships, such as images and time series. For high-dimensional, sparse, and globally correlated data, such as network traffic features, the convolution kernel may overlook important cross-dimensional feature interactions during its sliding process. The quantization mapping process of mean-shift clustering fails to minimize the quantization error of hash codes like ITQ does. Its clustering results are significantly affected by sample distribution, which can easily lead to uneven sample distribution after hash bucketing. Furthermore, the mean-shift algorithm has high computational complexity, making it unsuitable for real-time processing of large-scale, high-dimensional data, increasing computational burden and response latency.
[0068] By employing a feature compression scheme based on a multi-layer perceptron autoencoder, this method can better preserve the global relevance of network traffic features, effectively reduce feature dimensionality, and reduce computational and storage resource consumption compared to local feature extraction methods based on convolutional neural networks. Furthermore, by optimizing the hash encoding process through the ITQ iterative quantized hashing algorithm, a more balanced and efficient sample bucket distribution is achieved, improving the training efficiency and classification performance of subsequent classifiers while avoiding the computational complexity and uneven bucketing issues associated with mean-shift clustering methods.
[0069] S3. Perform principal component analysis on the low-dimensional feature representation to generate the corresponding binary hash code.
[0070] Random undersampling loses important data distribution information. To preserve this information, common clustering methods such as k-means clustering can be used to cluster the majority class, and then sample within the clusters. This preserves the data distribution information while balancing the number of samples in the two classes. However, clustering methods are generally computationally complex and are not suitable for large-scale network traffic data. Hashing methods can quickly divide the feature space into multiple hash subspaces (also called hash buckets), each of which corresponds to a unique hash code. Samples in the same subspace have the same hash code and are more similar to each other, thus achieving the goal of rapid clustering.
[0071] A preferred embodiment of this solution in the present invention utilizes a hashing method to partition the majority class samples into multiple hash subspaces to preserve the distribution information of the majority class. Locality-sensitive hashing (LSH) can quickly partition the hash subspaces, but with poor accuracy. To balance computational efficiency and information preservation, the solution of the present invention utilizes iterative quantized hashing (ITQ) to map low-dimensional features into binary hash subspaces.
[0072] ITQ can convert high-dimensional data into compact binary hash codes while retaining the Euclidean distance structure of the original data. ITQ (Iterative Quantization) first applies principal component analysis (PCA) to all majority class samples to find the orthogonal projection of the samples in the feature space along the direction of high variance. Subsequently, a unit hypercube is formed at the origin of the PCA projection space. Each vertex of the hypercube represents a hash code, and the samples located in the nearest neighbor area of the vertex will be assigned the corresponding hash code. Subsequently, the quantization loss between the hash code and the original feature vector of the sample is minimized through a rotation operation. It mainly includes three steps: 1. PCA dimensionality reduction; 2. Construction of a unit hypercube; 3. Iterative optimization of the rotation matrix. The three steps are described in detail below.
[0073] PCA dimensionality reduction: For the majority class of the self-encoder output, the low-dimensional feature Z is compressed and its covariance matrix C = Z is calculated. TZ, where T represents the matrix transpose. Take the eigenvectors corresponding to the first k largest eigenvalues to form the projection matrix W, and project the data V = ZW. PCA can retain the directions with the highest data variance, thereby removing redundant dimensions.
[0074] Constructing a unit hypercube: In the PCA projection space (k-dimensional), a hypercube with a side length of 2 and centered at the origin can be constructed, where each vertex represents a possible binary hash code.
[0075] Iteratively optimize the rotation matrix: First randomly initialize an orthogonal rotation matrix Perform the following alternating optimization process:
[0076] Fixed R, optimized hash code B: for each sample v i , calculate the coordinate v after rotation i R; then assign hash code b i =sign(v i R), where sign(v i R) is the sign function, v i The result is 1 if R is greater than 0, otherwise the result is -1.
[0077] Fix B and optimize the rotation matrix R: solve the orthogonal Procrustes problem: in is the square of the Frobenius norm. This problem has a closed-form solution. First, V T B performs singular value decomposition (SVD), then R = UV T (where UΣV T =SVD(V T B), U is the left singular vector matrix, V is the right singular vector matrix, and Σ is a diagonal matrix whose diagonal elements are singular values)
[0078] Convergence condition: The alternating optimization process stops when the predefined maximum number of iterations is reached
[0079] After the optimization is completed, the final hash code can be output: B = sign(VR)∈{-1,1} n×k , where sign is the sign function, n is the number of majority class samples, and k is the predefined hash code length.
[0080] A preferred embodiment of this solution in the present invention is to use an autoencoder or other feature compression model to perform nonlinear dimensionality reduction on the normalized feature vector to obtain a low-dimensional feature representation. The low-dimensional features are binarized using the locality sensitive hashing (LSH) method;
[0081] (1) Randomly generate multiple sets of hyperplane vectors, where the number of hyperplanes in each set corresponds to the length m of the hash code;
[0082] (2) For each low-dimensional feature vector, calculate its projection value on each hyperplane;
[0083] (3) If the projection value is greater than 0, the corresponding hash bit is recorded as 1, otherwise it is recorded as 0, and finally an m-bit binary hash code is generated;
[0084] (4) Divide samples with the same hash code into the same hash bucket.
[0085] Sample sampling and balancing are performed based on the generated hash buckets, and the subsequent classifier training and integration steps remain unchanged.
[0086] LSH partitions data using random hyperplanes and relies solely on random projections. This method fails to optimize for the actual distribution of low-dimensional features and struggles to preserve the Euclidean distance relationships in the original feature space, resulting in a lack of discriminability in the hash code. Because the hash hyperplane is randomly generated, samples tend to be concentrated in certain hash buckets, causing most hash buckets to be insufficient or empty, impacting the effectiveness of subsequent undersampling and easily causing an imbalance in the distribution of training samples. Randomly projected hash partitioning ignores the statistical characteristics of feature distribution and fails to specifically optimize the partitioning effect of the hash code. This is especially true when dealing with complex, multimodal network traffic features, resulting in blurred classification boundaries and hindering the effective learning of subsequent classifiers.
[0087] It should be further explained that:
[0088] By combining PCA dimensionality reduction with ITQ hash coding, this invention effectively preserves the structural information of the feature space and optimizes the distribution balance of the hash code. This overcomes the problems of bucket imbalance, feature information loss, and low classification accuracy found in LSH-based random hashing, thereby improving the training effectiveness and classification performance of intrusion detection models. In particular, in large-scale, high-dimensional, unbalanced data environments, this invention enhances the ability to identify minority attack samples while maintaining computational efficiency, thereby improving the overall security protection capabilities of the system.
[0089] S4. Divide the samples into multiple hash buckets according to the binary hash code, and perform undersampling on the majority class samples in each hash bucket in proportion to construct a class-balanced sample subset.
[0090] By calculating the sample quantity distribution in each hash bucket, the sampling quantity of the majority class samples is determined based on the preset class ratio relationship, and the corresponding number of majority class samples is selected from each hash bucket to construct a class-balanced sample subset.
[0091] To solve the problem of category imbalance, hash bucketing is used to balance data distribution, and ensemble learning is combined to improve robustness.
[0092] Bucket strategy: assign the majority class samples to 2 according to the hash codek In the hash buckets:
[0093] bucket i ={x|h(x)=b i},b i ∈{-1,1} k
[0094] Where h(x) is the hash encoding function.
[0095] Bucket-based random sampling: For the majority class sample, the following formula is used to sample from all buckets:
[0096]
[0097] Among them, N i is the number of majority class samples (i.e., normal traffic samples) of the i-th hash bucket, N attack is the number of attack traffic samples in the entire dataset, N normal is the number of normal traffic samples in the entire data set, is the floor operator.
[0098] For minority class samples, randomly sample N from all minority class samples attack samples, and combined with the majority class samples above to form a balanced data subset.
[0099] S5. Repeat the hash bucketing and sampling operations to obtain several balanced sample subsets and train the base classifier.
[0100] Base classifier training: Repeat the above bucket-based random sampling process S times to obtain S balanced data subsets, and use the balanced data subsets to train S base classifiers Classification and Regression Tree (CART) respectively.
[0101] S6. Evaluate the performance indicators of each base classifier on the validation set, and calculate the integration weight of each base classifier based on the evaluation results.
[0102] According to the performance index of each base classifier on the validation set, the integration weight of each base classifier is determined by calculating the ratio of the corresponding performance index value of each base classifier to the sum of the performance index values of all base classifiers;
[0103] The performance indicators include balanced accuracy and receiver operating characteristic curve area.
[0104] S7. When predicting the sample to be tested, the final predicted category is output based on the weighted voting results of each base classifier.
[0105] Performance evaluation: Calculate the evaluation index balanced accuracy e of each base classifier on the reserved validation dataset:
[0106]
[0107] Where TP is the number of correct predictions for the minority class, TN is the number of correct predictions for the majority class, FP is the number of predictions for the majority class that were predicted as the minority class, and FN is the number of predictions for the minority class that were predicted as the majority class. Balancing precision by equally weighting the recall of each class can prevent the majority class from dominating the evaluation results.
[0108] Weight calculation: Calculate the weight of each base classifier according to the evaluation index value:
[0109]
[0110] where e s is the evaluation index value of the s-th base classifier, Σ s e s Indicates the sum of the evaluation indicators of all base classifiers, w s is the assigned weight of the s-th base classifier.
[0111] Final prediction: assign weight w based on validation set performance s , for a given unknown traffic sample, its final prediction is the weighted voting result:
[0112]
[0113] Among them, C s (x) is the prediction result of the s-th base classifier for sample x, δ(·) is the indicator function, which takes the value of 1 when the condition in the brackets is met and takes the value of 0 otherwise, and y is the true class, which takes the value of 1 to indicate the minority class (normal traffic) and takes the value of -1 to indicate the majority class (attack traffic).
[0114] Example 2 is the second embodiment of the present invention, which provides a network intrusion detection system based on hash bucket integration, including a feature processing module, a feature conversion module, a data sampling module and a classification prediction module;
[0115] The feature processing module extracts statistical features and time series features from the network traffic data, and performs standardization processing on the extracted feature vectors to eliminate the dimensional differences between the features and form a standardized feature vector;
[0116] The feature conversion module compresses the normalized feature vector into a low-dimensional feature representation using a nonlinear compression method, and encodes the low-dimensional feature representation based on a feature space mapping method to generate a hash code for hash bucketing;
[0117] The data sampling module divides the samples into multiple hash buckets according to the hash code, and performs undersampling on the majority class samples according to a preset ratio based on the sample category distribution in each hash bucket to generate a class-balanced sample subset;
[0118] The classification prediction module trains multiple base classifiers based on the sample subset, and determines the integration weight according to the performance index of each base classifier on the validation set. When predicting the test sample, the integration weight is used for weighted calculation to output the final prediction category.
[0119] Example 3 is the third embodiment of the present invention, which differs from the first two embodiments in that:
[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0121] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0122] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.
[0123] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or combination of the following technologies known in the art can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A network intrusion detection method based on hash bucket integration, characterized by: include, Perform feature extraction and standardization on network traffic data to obtain standardized feature vectors; Processing the standardized feature vector based on a feature conversion strategy of nonlinear compression and quantization mapping to obtain a low-dimensional feature representation; Perform principal component analysis on the low-dimensional feature representation to generate the corresponding binary hash code; Divide the samples into multiple hash buckets according to the binary hash code, and perform undersampling on the majority class samples in each hash bucket in proportion to construct a class-balanced sample subset; Repeat the hash bucketing and sampling operations to obtain several balanced sample subsets and train the base classifier; Evaluate the performance indicators of each base classifier on the validation set, and calculate the integration weight of each base classifier based on the evaluation results; When predicting the test sample, the final prediction category is output based on the weighted voting results of each base classifier.
2. The network intrusion detection method based on hash bucket integration according to claim 1, characterized in that: The feature extraction and standardization processing includes constructing feature vectors based on the statistical features and time series features of the network traffic data, and standardizing the features of each dimension in the feature vector based on a normalization calculation formula to eliminate feature dimension differences.
3. The network intrusion detection method based on hash bucket integration according to claim 2, characterized in that: The feature conversion strategy includes performing nonlinear compression on the standardized feature vector through a neural network model to obtain a low-dimensional feature representation, and performing feature mapping conversion on the low-dimensional feature representation based on a feature space transformation method.
4. The network intrusion detection method based on hash bucket integration according to claim 3, characterized in that: The method of dividing the samples into a plurality of hash buckets according to the binary hash code and performing undersampling proportionally on the majority class samples in each hash bucket includes: By calculating the sample quantity distribution in each hash bucket, the sampling quantity of the majority class samples is determined based on the preset class ratio relationship, and the corresponding number of majority class samples is selected from each hash bucket to construct a class-balanced sample subset.
5. The network intrusion detection method based on hash bucket integration according to claim 4, characterized in that: The training base classifier includes: Each base classifier is trained independently using a balanced sample subset with the same model structure and parameter configuration.
6. The network intrusion detection method based on hash bucket integration according to claim 4, characterized in that: The performance indicators of each base classifier are evaluated on the validation set, and the ensemble weights of each base classifier are calculated based on the evaluation results. According to the performance index of each base classifier on the validation set, the integration weight of each base classifier is determined by calculating the ratio of the corresponding performance index value of each base classifier to the sum of the performance index values of all base classifiers; The performance indicators include balanced accuracy and receiver operating characteristic curve area.
7. The network intrusion detection method based on hash bucket integration according to claim 4, characterized in that: When predicting the sample to be tested, the final prediction category output based on the weighted voting results of each base classifier includes: The samples to be tested are input into each base classifier respectively, the classification results are obtained and based on the integrated weights of each base classifier, all classification results are accumulated and counted according to the weights, and the category with the largest accumulated value is selected as the final prediction result.
8. A network intrusion detection system based on hash bucket integration, applying the network intrusion detection method based on hash bucket integration according to any one of claims 1 to 7, characterized in that: Includes: feature processing module, feature conversion module, data sampling module and classification prediction module The feature processing module extracts statistical features and time series features from the network traffic data, and performs standardization processing on the extracted feature vectors to eliminate the dimensional differences between the features and form a standardized feature vector; The feature conversion module compresses the normalized feature vector into a low-dimensional feature representation using a nonlinear compression method, and encodes the low-dimensional feature representation based on a feature space mapping method to generate a hash code for hash bucketing; The data sampling module divides the samples into multiple hash buckets according to the hash code, and performs undersampling on the majority class samples according to a preset ratio based on the sample category distribution in each hash bucket to generate a class-balanced sample subset; The classification prediction module trains multiple base classifiers based on the sample subset, and determines the integration weight according to the performance index of each base classifier on the validation set. When predicting the test sample, the integration weight is used for weighted calculation to output the final prediction category.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the network intrusion detection method based on hash bucket integration as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the network intrusion detection method based on hash bucket integration according to any one of claims 1 to 7 are implemented.