Network intrusion detection method and device based on N-gram frequency and fusion improved algorithm

Abstract

The invention belongs to the technical field of network intrusion detection, and aims to solve the problems of insufficient information utilization and incomplete feature dimension in network intrusion detection. The invention provides a network intrusion detection method based on an N-gram frequency and a fusion improved algorithm, which comprises the following steps of: acquiring an original network data flow packet, and dividing the original network data flow packet into a plurality of independent initial sessions; respectively extracting packet header information, an effective load and a timestamp in the data packet; forming a data packet set by the packet header information extracted from the same data packet, the effective load and the timestamp; all the data packet sets in the same session form a complete session; extracting packet header features, payload features and time features of the complete session to obtain a feature sequence; performing deep feature extraction on the feature sequence to obtain deep session features; and carrying out network data flow detection result classification based on the deep session features. The feature set extracted by the method is complete and has higher classification precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of network intrusion detection, and in particular relates to a network intrusion detection method and device based on N-gram frequency and fusion improved algorithm. Background Art

[0002] The extensive interconnection between people and things has promoted the development of the Internet, but it has also made system vulnerabilities in network facilities increasingly prominent, and the risk of network intrusion and attack has continued to rise.

[0003] Traditional intrusion detection methods based on known attack patterns are powerless in the face of complex and ever-changing network attacks, especially in identifying and defending against unknown attacks.

[0004] In recent years, although the rise of technologies such as machine learning has brought a certain degree of progress in the anomaly detection of network attacks, it still faces many challenges: for example, feature extraction is highly dependent on manual operations, the detection difficulty of unknown attacks remains high, the detection model is complex, and the detection accuracy needs to be improved. These problems have seriously restricted the promotion and development of network intrusion detection systems in practical applications. Summary of the invention

[0005] In order to solve at least one of the above technical problems existing in the prior art, the present invention provides a network intrusion detection method and device based on an N-gram frequency and fusion improved algorithm.

[0006] The present invention is implemented by the following technical scheme: a network intrusion detection method of N-gram frequency and fusion improved algorithm, comprising the following steps: obtaining original network data flow packets, and dividing them into a number of independent initial sessions; sorting the data packets in the initial session by timestamps, and deleting redundant information in the data packets; extracting the header information, effective load and timestamp in the data packets respectively; forming a data packet set with the header information, effective load and timestamp extracted from the same data packet; forming a complete session with all data packet sets in the same session; extracting the header features, effective load features and time features of the complete session to obtain a feature sequence; and performing deep-level feature extraction on the feature sequence to obtain deep-level session features; and classifying network data flow detection results based on the deep-level session features.

[0007] Preferably, before intrusion detection, it also includes the construction of a network intrusion detection model based on N-gram frequency and fusion improved algorithm. The network intrusion detection model based on N-gram frequency and fusion improved algorithm includes a traffic packet preprocessing module, a traffic packet session feature extraction module, a deep session feature extraction module and a classification module; the traffic packet preprocessing module segments and extracts information from the original network data traffic packet to obtain several complete sessions; the traffic packet session feature extraction module extracts features from the complete session to obtain a feature sequence including packet header features, payload features and time features; the deep session feature extraction module performs deep-level feature extraction on the feature sequence to obtain deep-level session features; the classification module classifies the network data traffic detection results based on the deep-level session features.

[0008] Preferably, the i-th data packet set is expressed as:

[0009]

[0010] In the formula, Represents the header information of the i-th data packet, Represents the payload information of the i-th data packet, Indicates the timestamp information of the ith data packet; m indicates the number of data packets in a complete session;

[0011] A complete session s is represented as:

[0012] s=[p1,p2,…,p m ],m∈Z.

[0013] Preferably, a one-dimensional convolutional attention neural network module is used to extract packet header features, and the extracted packet header features of the i-th data packet are expressed as:

[0014]

[0015] In the formula, D1(·) represents the extraction of packet header features;

[0016] N-gram frequency analysis and one-dimensional convolutional attention neural network module are used to extract payload features. The extracted payload features of the i-th data packet are expressed as:

[0017]

[0018] Where D2(·) represents payload feature extraction, NG(·) represents N-gram frequency processing;

[0019] The log function is used to smooth the timestamp and extract the time feature;

[0020] The feature sequence s' of a complete session is expressed as:

[0021] s'=[p' i ,p'2,…,p' m ]

[0022] In the formula, p' i Represents the feature sequence extracted from the i-th data packet.

[0023] Preferably, the steps of performing deep feature extraction on the feature sequence to obtain deep session features are:

[0024] Each feature sequence s' is temporally encoded, and then the encoding results are passed to the multi-head deformable attention module, the add&norm module, and the feed forward module in sequence. After repeating this process N times, a new deep-level session feature s is obtained. p .

[0025] Preferably, deleting redundant information in the data packet includes:

[0026] Remove IP address and MAC address information from each data packet; delete protocol version, option field, padding field, and differentiated service field; filter other types of traffic except TCP protocol traffic.

[0027] The present invention also provides a network intrusion detection device based on N-gram frequency and fusion improved algorithm, including a data acquisition module, which is responsible for collecting the flow data of the monitored network in real time, storing the data to a disk, and transmitting it to a data analysis and standardization module for processing; a data analysis and standardization module, which receives the data transmitted by the data acquisition module, completes data decoding and standardization processing, and extracts features to generate a data set suitable for a network intrusion detection model; an intrusion traffic detection module, which receives the standardized data, performs feature subset selection and model training, classifies the network traffic, identifies the intrusion traffic, and outputs prediction results, confusion matrices and evaluation indicators for display by the detection result display and interaction module; and a result visualization and interaction module, which displays and interactively feedbacks the detection results.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention adopts a feature extraction model based on N-gram frequency, which effectively shortens the training time in the model training stage and reduces resource and time costs. Compared with traditional feature dimensionality reduction methods such as principal component analysis, the feature extraction model based on mutual information theory has more advantages in extracting strongly correlated features of network traffic. The model training results show that under the same conditions, the feature set extracted by this method shows higher classification accuracy.

[0030] The present invention utilizes the fusion improvement algorithm to obtain a faster optimization speed, and on this basis, the intrusion detection model obtains a better classification accuracy for network traffic and a faster convergence speed.

[0031] The present invention uses graphical visualization to display the classification results, which can better display the changes of network traffic over time in a linear manner in the form of time series and locate the process of network intrusion and attack traffic discovery, detection, change and disposal, which can help users better understand and deal with network attacks and provide fast feedback for iterative maintenance of the model.

[0032] The present invention extracts the traffic header and payload features respectively through (1D-Convolutional Attention Network, 1D-CAN), in which the N-gram frequency is innovatively used to represent the payload context information. In addition, (1D-Deformable Attention Transformer, 1D-DAT) is introduced to construct correlation features and extract deep-level session features. The model can be exported and deployed on network devices for detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 This is an example of N-gram frequency processing;

[0035] Figure 2 It is a flow chart of the network intrusion detection model based on N-gram frequency and fusion improved algorithm;

[0036] Figure 3 This is the improved 1D-CAN module structure diagram;

[0037] Figure 4 It is the structure diagram of the bias learning network module;

[0038] Figure 5 This is the improved 1D-DAT module structure diagram. DETAILED DESCRIPTION

[0039] In conjunction with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention. It should be noted that in this specification, relational terms such as first and second are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0041] Embodiment 1:

[0042] A network intrusion detection method based on an improved N-gram frequency and fusion algorithm comprises the following steps: obtaining an original network data flow packet and dividing it into a number of independent initial sessions; sorting the data packets in the initial session by timestamps and deleting redundant information in the data packets; extracting the header information, effective load and timestamp in the data packets respectively; forming a data packet set with the header information, effective load and timestamp extracted from the same data packet; forming a complete session with all the data packet sets in the same session; extracting the header features, effective load features and time features of the complete session to obtain a feature sequence; and performing deep-level feature extraction on the feature sequence to obtain deep-level session features; and classifying the network data flow detection results based on the deep-level session features.

[0043] Specifically, the steps include:

[0044] 1. Split sessions: Use the split-cap NETRESEC (2011) tool to split the original network traffic data packets according to the five-tuple information of the traffic (i.e. source IP address, destination IP address, source port number, destination port number and protocol type) to form multiple independent sessions.

[0045] 2. Session sorting: After segmentation, each session is sorted according to the timestamps of the data packets it contains to ensure that the data packets in the session are arranged in chronological order, laying the foundation for the subsequent extraction of time-related features.

[0046] 3. Remove redundant information: Remove IP address and MAC address information from each data packet to prevent potential ARP spoofing attacks. At the same time, considering that the importance of information such as protocol version, option field, padding field, and differentiated service field is relatively low in network intrusion detection, they are also deleted. Focus on TCP protocol traffic and filter other types of traffic to reduce the complexity of analysis.

[0047] 4. Construct a set: For each data packet processed above, extract the header information, payload, and timestamp according to its structural characteristics. These extracted information together form a data packet set p i , where i represents the sequence number or unique identifier of the data packet. i It is further combined into a complete session s, which will be used as the basic unit of intrusion detection model processing for subsequent analysis and detection work.

[0048] The i-th data packet set is expressed as:

[0049]

[0050] In the formula, Represents the header information of the i-th data packet, Represents the payload information of the i-th data packet, Indicates the timestamp information of the ith data packet; m indicates the number of data packets in a complete session;

[0051] A complete session s is represented as:

[0052] s=[p1,p2,…,p m ],m∈Z.

[0053] 5. Traffic packet feature extraction: In the process of traffic packet processing, the packet header and the payload have completely different responsibilities. First, in terms of information content, the packet header is mainly responsible for encapsulating control information to ensure that the data packet can be correctly routed and transmitted; while the payload carries the actual data content to be transmitted. Secondly, in terms of length, the packet header is usually kept in a relatively fixed range of about 60 bytes. In contrast, the length of the payload varies greatly, which may range from 0 bytes to 1000 bytes. In view of these differences, it is obviously not reasonable to use a single model to process both the packet header and the payload at the same time, because doing so may ignore the differences in characteristics and importance between the two. In addition, as an important part of the data packet, the timestamp contains rich information, but it is often ignored or not fully utilized in previous processing. In order to make more effective use of this information, a more sophisticated and targeted strategy is adopted to perform feature processing on the packet header, payload and timestamp respectively. This method aims to deeply explore the unique value of each part of the data, so as to more accurately extract session features and provide strong support for subsequent intrusion detection work. For each p i , after feature extraction using different methods, they are transformed into packet header features Payload characteristics Time characteristics The final output extracted feature sequence s′ is shown in the following formula:

[0054]

[0055] A one-dimensional convolutional attention neural network module is used to extract packet header features. The extracted packet header features of the i-th data packet are expressed as:

[0056]

[0057] Where D1(·) represents the extraction of packet header features.

[0058] The byte length of the payload is uncertain, and this length variation directly hinders its direct input into the feature extraction module. Currently, common means of dealing with this problem include truncation and padding, but both methods have their own limitations: truncation may lead to the loss of key information, while padding may introduce unnecessary noise, thus affecting the accuracy of feature extraction. In order to more effectively cope with the challenges brought by the change in payload length and accurately extract its features, the N-gram frequency analysis and one-dimensional convolutional attention neural network (1D-CAN) modules are innovatively adopted. These two methods can flexibly handle payloads of different lengths, avoiding information loss and reducing noise interference, thereby achieving accurate extraction of payload features.

[0059] N-gram frequency processing: In view of the fact that the payload of the traffic packet is composed of a hexadecimal string, the N-gram frequency statistics method is introduced. This method calculates the frequency of occurrence of each N character combination in the hexadecimal string in the original traffic data, and then integrates these frequency statistics results into a one-dimensional vector with a fixed length. It is worth noting that the size of the N value directly determines the richness of the context information that can be considered. After multiple experimental verifications, the N value was finally set to 2, so the vector length obtained was 256. Figure 1 An example of how to generate an N-gram frequency vector when N is 2 is shown.

[0060] N-gram frequency analysis and one-dimensional convolutional attention neural network module are used to extract payload features. The extracted payload features of the i-th data packet are expressed as:

[0061]

[0062] Where D2(·) represents payload feature extraction and NG(·) represents N-gram frequency processing.

[0063] The timestamp carries the time attribute of the traffic packet. In order to make it adaptable to the time feature extraction module, the log function is used to smooth the timestamp to extract the effective time feature.

[0064] The feature sequence s' of a complete session is expressed as:

[0065] s'=[p' i ,p'2,…,p' m ]

[0066] In the formula, p' i Represents the feature sequence extracted from the i-th data packet.

[0067] The steps for extracting deep features from the above feature sequence to obtain deep conversation features are as follows:

[0068] Each feature sequence s' is temporally encoded, and then the encoding results are passed to the multi-head deformable attention module, the add&norm module, and the feed forward module in sequence. After repeating this process N times, a new deep-level session feature s is obtained. p .

[0069] Before intrusion detection, it also includes the construction of a network intrusion detection model based on N-gram frequency and fusion improved algorithm. The network intrusion detection model based on N-gram frequency and fusion improved algorithm includes a traffic packet preprocessing module, a traffic packet session feature extraction module, a deep session feature extraction module and a classification module; the traffic packet preprocessing module segments and extracts information from the original network data traffic packet to obtain several complete sessions; the traffic packet session feature extraction module extracts features from the complete session to obtain a feature sequence including packet header features, payload features and time features; the deep session feature extraction module performs deep-level feature extraction on the feature sequence to obtain deep-level session features; the classification module classifies the network data traffic detection results based on the deep-level session features.

[0070] Among them, the data flow in the network intrusion detection model based on N-gram frequency and fusion improved algorithm includes:

[0071] Data collection: Use the data collector to collect the full flow data of the specified network device, realize the collection of the original data set, and store it in the file in pcap format; Data labeling: According to the characteristics of network traffic attacks, manually label the labels of attack samples, where the label values ​​of normal samples and attack samples (including DoS, Probe, U2R and R2L) are 0 and 1 respectively; Data preprocessing: Step 3.1: N-gram frequency conversion: such as Figure 1 The distribution of the downsampled data is shown in Table 1.

[0072] Table 1

[0073]

[0074]

[0075] The work of integrating and improving the algorithm includes:

[0076] 1. The preprocessed traffic data is divided into three components, but even within the same part, the traffic features still show significant differences. These highly differentiated features play a key role in the model's identification of intrusion behavior. In order to make full use of the diversity and spatial characteristics of traffic features, an attention mechanism is introduced to optimize the one-dimensional convolutional neural network, and on this basis, a one-dimensional convolutional attention network (1D-CAN) module is proposed. In the 1D-CAN module, the attention mechanism calculates the maximum and average values ​​on the channel dimension, and then applies a one-dimensional convolution operation to the newly generated feature sequence, and finally outputs a single-channel feature sequence. This mechanism can assign different weights to the input traffic features, thereby emphasizing more important features in the feature extraction stage, effectively improving the accuracy of detection. Figure 3 The specific structure of this module is shown in detail.

[0077] 2. Improve the traditional attention encoder. Compared with the traditional self-attention, the deformable attention adds a bias learning network. The bias learning network structure is as follows Figure 4 As shown, the deformable attention structure is as follows Figure 5 As shown. Assuming that the input vector x is given, first perform a one-dimensional convolution on the input x to generate a query vector Q, and at the same time generate a reference point based on the dimension of the input x. The query vector Q is input into the bias learning network Offset(·) to generate an offset Δp for each reference point (in order to prevent the offset from being too large, the preset parameter s and control offset are used). Secondly, the deformation point is obtained by adding the reference point and the offset Δp. According to the coordinate position corresponding to the deformation point, bilinear interpolation sampling is used on the input to obtain the sampled feature vector x, and then a one-dimensional convolution is performed to generate a key-value vector K and a value vector V. Finally, Q, K, V and the relative position offset R are used to calculate the output result. The deformable attention calculation is shown as follows:

[0078]

[0079] Δp=s·tanh(Offset(Q))

[0080]

[0081]

[0082] DMHA(Z)=Concat(Z (1) ,Z (2) ,…,Z (m )W0

[0083] Where: BI(·) represents bilinear interpolation, Z (i) represents the output of the i-th head, m is the total number of module heads, Represents the bias table, represents the feature dimension after scaling, Softmax is the normalization function, p is the reference point, is the output vector, tanh is the hyperbolic tangent function, DMHA is the multi-head variability attention mechanism, Concat is concatenation, W q , W k , W v , W0 is the projection matrix, W q , W k , W v Q, K, and V are linearly mapped respectively, and W0 linearly maps the outputs of multiple heads.

[0084] It also includes model training: inputting the training set into the detection model for training. Parameter optimization: adjusting the hyperparameters of the model. Setting the initial parameters is shown in Table 2;

[0085] Table 2

[0086]

[0087] The steps of model evaluation and optimization include: Performance testing: setting the optimal parameters and feeding them into the model for prediction. Performance evaluation: statistically analyzing the confusion matrix composed of the predicted results and the actual results, and using formula (11-13) to calculate indicators such as accuracy, precision and detection rate.

[0088]

[0089]

[0090]

[0091] Where: TP represents the true positive rate, FN represents the false negative rate, FP represents the false positive rate, TN represents the true negative rate, ACC represents the accuracy of the model; precision represents the precision of the model; recall represents the recall rate of the model, sensitivity represents the sensitivity, and DR represents the detection rate; its range is [0, 1], 1 represents the best model performance, and 0 represents the worst model performance.

[0092] This implementation case is divided into a training phase and a testing phase. In the training phase, the key features in the training set are extracted through the N-Gram frequency, and then these features are loaded as input into the fusion improvement algorithm model for training, and the 1D-CAN algorithm is used to extract network traffic features. The 1D-DAT model learns deep-level key features. In the testing phase, the features of the test data are extracted and input into the trained fusion classification model. The correctness of the classification result is judged according to the category probability distribution output by the model, and then the model is evaluated.

[0093] Embodiment 2:

[0094] The present invention also provides a network intrusion detection device based on N-gram frequency and fusion improved algorithm, including a data acquisition module, which is responsible for collecting the flow data of the monitored network in real time, storing the data to a disk, and transmitting it to a data analysis and standardization module for processing; a data analysis and standardization module, which receives the data transmitted by the data acquisition module, completes data decoding and standardization processing, and extracts features to generate a data set suitable for a network intrusion detection model; an intrusion traffic detection module, which receives the standardized data, performs feature subset selection and model training, classifies the network traffic, identifies the intrusion traffic, and outputs prediction results, confusion matrices and evaluation indicators for display by the detection result display and interaction module; and a result visualization and interaction module, which displays and interactively feedbacks the detection results.

[0095] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A network intrusion detection method based on N-gram frequency and fusion improved algorithm, characterized in that: The following steps are involved: Obtain the original network data traffic packet and divide it into several independent initial sessions; Sort the data packets in the initial session by timestamp and delete the redundant information in the data packets; Extract the header information, payload and timestamp from the data packet respectively; The packet header information, payload and timestamp extracted from the same data packet are combined into a data packet set; all data packet sets in the same session are combined into a complete session; Extracting the packet header features, payload features and time features of the complete conversation to obtain a feature sequence; and performing deep feature extraction on the feature sequence to obtain deep conversation features; Classify network data traffic detection results based on deep session features.

2. A network intrusion detection method based on N-gram frequency and fusion improved algorithm according to claim 1, characterized in that: Before intrusion detection, it also includes the construction of a network intrusion detection model based on N-gram frequency and fusion improved algorithm. The network intrusion detection model based on N-gram frequency and fusion improved algorithm includes a traffic packet preprocessing module, a traffic packet session feature extraction module, a deep session feature extraction module and a classification module; The traffic packet preprocessing module segments and extracts information from the original network data traffic packets to obtain several complete sessions; The traffic packet session feature extraction module extracts features from the complete session and obtains a feature sequence including packet header features, payload features, and time features; The deep conversation feature extraction module performs deep feature extraction on the feature sequence to obtain deep conversation features; The classification module classifies network data traffic detection results based on deep-level session features.

3. A network intrusion detection method based on N-gram frequency and fusion improved algorithm according to claim 2, characterized in that: The i-th data packet set is expressed as: In the formula, Represents the header information of the i-th data packet, Represents the payload information of the i-th data packet, Indicates the timestamp information of the ith data packet; m indicates the number of data packets in a complete session; A complete session s is represented as: s=[p1,p2,…,p m ],m∈Z。 4. The network intrusion detection method according to claim 3, characterized in that: A one-dimensional convolutional attention neural network module is used to extract packet header features. The extracted packet header features of the i-th data packet are expressed as: In the formula, D1(·) represents the extraction of packet header features; N-gram frequency analysis and one-dimensional convolutional attention neural network module are used to extract payload features. The extracted payload features of the i-th data packet are expressed as: Where D2(·) represents payload feature extraction, NG(·) represents N-gram frequency processing; The Iog function is used to smooth the timestamp and extract the time feature; The feature sequence s′ of a complete session is expressed as: s′=[p′ i ,p′2,…,p′ m ] In the formula, p′ i Represents the feature sequence extracted from the i-th data packet.

5. The network intrusion detection method according to claim 4, characterized in that: The steps for extracting deep features from feature sequences and obtaining deep conversation features are as follows: Each feature sequence s′ is temporally encoded, and then the encoding results are passed to the multi-head deformable attention module, the add&norm module, and the feed forward module in sequence. After repeating this process N times, a new deep-level session feature s is obtained. p .

6. The network intrusion detection method according to claim 1, characterized in that: Deleting redundant information in the data packet includes: Remove IP address and MAC address information from each data packet; delete protocol version, option field, padding field, and differentiated service field; filter other types of traffic except TCP protocol traffic.

7. A network intrusion detection device based on N-gram frequency and fusion improved algorithm, used to implement the network intrusion detection method based on N-gram frequency and fusion improved algorithm as claimed in any one of claims 1 to 6, characterized in that: It includes a data acquisition module, which is responsible for collecting the flow data of the monitored network in real time, storing the data to disk, and transmitting it to the data analysis and standardization module for processing; The data parsing and standardization module receives the data transmitted by the data acquisition module, completes data decoding and standardization, and extracts features to generate a data set suitable for the network intrusion detection model; The intrusion traffic detection module receives standardized data, performs feature subset selection and model training, classifies network traffic, identifies intrusion traffic, and outputs prediction results, confusion matrix, and evaluation indicators for display in the detection result display and interactive module; The result visualization and interaction module displays the test results and provides interactive feedback.

Images