Network intrusion detection method and device based on n-gram frequency and fusion improved algorithm

Abstract

The application belongs to the technical field of network intrusion detection, and aims to solve the problems of insufficient information utilization and incomplete feature dimension in network intrusion detection. A network intrusion detection method based on N-gram frequency and fusion improved algorithm is provided, comprising the following steps: obtaining original network data flow packets and dividing them into several independent initial sessions; extracting the packet header information, payload and timestamp in the data packet respectively; forming a data packet set by the extracted packet header information, payload and timestamp in the same data packet; forming a complete session by all data packet sets in the same session; extracting the packet header feature, payload feature and time feature of the complete session to obtain a feature sequence; performing deep feature extraction on the feature sequence to obtain deep session features; and classifying the network data flow detection results based on the deep session features. The feature set extracted by the application is complete and has higher classification accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network intrusion detection, and particularly relates to a network intrusion detection method and device based on N-gram frequency and fusion improved algorithm. BACKGROUND

[0002] The extensive interconnection of people and things promotes the development of the Internet, but at the same time, the problem of system vulnerabilities in network facilities is increasingly prominent, and the risk of network intrusion and attack is rising.

[0003] Traditional intrusion detection methods based on known attack patterns are not capable enough when facing complex and variable network attacks, especially in identifying and defending unknown attacks.

[0004] In recent years, although the rise of machine learning and other technologies has brought some progress to network attack anomaly detection, there are still many challenges: such as high dependence on manual operation for feature extraction, high difficulty in detecting unknown attacks, complex detection model construction, and detection accuracy to be improved, etc. These problems seriously restrict the promotion and development of network intrusion detection systems in practical applications. SUMMARY

[0005] The application provides a network intrusion detection method and device based on N-gram frequency and fusion improved algorithm to solve at least one of the above technical problems in the prior art.

[0006] The application adopts the following technical solution: a network intrusion detection method based on N-gram frequency and fusion improved algorithm, comprising the following steps: obtaining original network data flow packets and dividing them into a plurality of independent initial sessions; sorting the data packets in the initial sessions according to the time stamp and deleting the redundant information in the data packets; extracting the packet header information, payload and time stamp in the data packets; forming a data packet set by the extracted packet header information, payload and time stamp in the same data packet; forming a complete session by all data packet sets in the same session; extracting the packet header features, payload features and time features of the complete session to obtain a feature sequence; performing deep feature extraction on the feature sequence to obtain deep session features; and classifying the network data flow detection results based on the deep session features.

[0007] Preferably, before intrusion detection, the method further includes constructing a network intrusion detection model based on N-gram frequency and a fusion-improved algorithm. This model includes a traffic packet preprocessing module, a traffic packet session feature extraction module, a deep session feature extraction module, and a classification module. The traffic packet preprocessing module segments and extracts information from the original network data traffic packets to obtain several complete sessions. The traffic packet session feature extraction module extracts features from the complete sessions to obtain a feature sequence containing header features, payload features, and time features. The deep session feature extraction module performs deep-level feature extraction on the feature sequence to obtain deep-level session features. The classification module classifies the network data traffic detection results based on the deep-level session features.

[0008] Preferably, the first The set of data packets is represented as:

[0009]

[0010] In the formula, Indicates the first The header information of each data packet. Indicates the first The payload information of each data packet Indicates the first The timestamp information of each data packet; This indicates the number of packets in a complete session;

[0011] Full Session Represented as:

[0012] .

[0013] Preferably, a one-dimensional convolutional attention neural network module is used to extract the packet header features, and the extracted first... The header characteristics of each data packet are represented as follows:

[0014]

[0015] In the formula, This indicates the extraction of header features;

[0016] The payload features were extracted using N-gram frequency analysis and a one-dimensional convolutional attention neural network module. The payload characteristics of each data packet are represented as follows:

[0017]

[0018] In the formula, This indicates payload feature extraction. This indicates N-gram frequency processing;

[0019] adopt The function is used for smoothing the timestamp, and time characteristics are extracted;

[0020] A complete session feature sequence is expressed as:

[0021]

[0022] In the formula, , represents the feature sequence of the extracted th data packet, The value range of 1- .

[0023] Preferably, the step of performing deep feature extraction on the feature sequence to obtain deep session features is:

[0024] Each feature sequence is time encoded, and then the encoding result is sequentially transmitted to the multi-head deformable attention module, the add&norm module and the feed forward module, and after the process is repeated N times, a brand new deep session feature .

[0025] Preferably, the step of deleting redundant information in the data packet comprises:

[0026] IP address and MAC address information are removed in each data packet; protocol version, option field, padding field and differentiated service field are deleted; and other types of traffic except TCP protocol traffic are filtered.

[0027] The application also provides a network intrusion detection device based on N-gram frequency and fusion improved algorithm, comprising a data acquisition module, which is responsible for collecting traffic data of a monitored network in real time, and storing the data to a disk, and transmitting the data to a data analysis and standardization module for processing; the data analysis and standardization module receives the data transmitted by the data acquisition module, completes data decoding and standardization processing, and extracts features to generate a data set suitable for a network intrusion detection model; an intrusion flow detection module receives the standardized data, performs feature subset selection and model training, classifies network traffic, identifies intrusion flow, and outputs prediction results, confusion matrix and evaluation indexes for display by a detection result display and interaction module; and a result visualization and interaction module displays and interacts with the detection results.

[0028] Compared with the prior art, the application has the beneficial effects that:

[0029] The application adopts a feature extraction model based on N-gram frequency, effectively shortens the training time in the model training stage, and reduces the resource and time cost.

[0030] The application utilizes a fusion improved algorithm for improvement to obtain a faster optimization speed, and on this basis, the intrusion detection model obtains better classification accuracy for network traffic and faster convergence speed.

[0031] The application adopts a graphical visualization form to display the classification result, can better linearly display the network traffic change condition with time in the form of time series, and locate the network intrusion and attack traffic discovery, detection, change and disposal process, can help users better understand and dispose network attacks, and provides quick feedback for iterative maintenance of the model.

[0032] The application extracts traffic packet header and payload features through (1D-Convolutional Attention Network, 1D-CAN), wherein the N-gram frequency is innovatively used to represent the payload context information. In addition, (1D-Deformable Attention Transformer, 1D-DAT) is introduced to construct associated features and extract deep session features, and the model can be derived and deployed on a network device for detection. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0034] Figure 1 is an N-gram frequency processing diagram;

[0035] Figure 2 is a network intrusion detection model flowchart based on N-gram frequency and fusion improved algorithm;

[0036] Figure 3 is a structure diagram of an improved 1D-CAN module;

[0037] Figure 4 is a structure diagram of a bias learning network module;

[0038] Figure 5 is a diagram of an improved 1D-DAT module structure. DETAILED DESCRIPTION

[0039] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0040] It should be noted that the structures, proportions, sizes, etc. shown in the drawings of the specification are merely used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and do not have technical substantive significance. Any modification of the structure, change of the proportion relationship, or adjustment of the size, without affecting the effects and purposes that can be achieved by the present application, should fall within the scope of the technical content disclosed by the present application. It should be noted that in the specification, relationship terms such as first and second are merely used to distinguish one entity from another entity, and do not necessarily require or imply any actual relationship or order between the entities.

[0041] Embodiment 1:

[0042] A network intrusion detection method of N-gram frequency and fusion improved algorithm, comprising the following steps: obtaining original network data flow packets, and dividing them into a plurality of independent initial sessions; sorting the data packets in the initial sessions according to time stamps, and deleting redundant information in the data packets; extracting the packet header information, payload and time stamp in the data packets respectively; forming a data packet set by the extracted packet header information, payload and time stamp in the same data packet; forming a complete session by all data packet sets in the same session; extracting the packet header feature, payload feature and time feature of the complete session to obtain a feature sequence; and performing deep feature extraction on the feature sequence to obtain deep session features; and classifying the network data flow detection results based on the deep session features.

[0043] Specifically, the following steps are included:

[0044] 1. Split session: using split-Cap NETRESEC (2011) tool, according to the five-tuple information of the flow (i.e. source IP address, destination IP address, source port number, destination port number and protocol type), the original network flow data packets are divided to form a plurality of independent sessions.

[0045] 2. Session ordering: For each session, the data packets are sorted according to their timestamps, ensuring that the packets in a session are in chronological order, which lays the foundation for extracting time-related features later.

[0046] 3. Redundancy removal: IP and MAC addresses are removed from each data packet to prevent potential ARP spoofing attacks. Meanwhile, considering that the importance of protocol version, option field, padding field, and differentiated services field is relatively low in network intrusion detection, these fields are also removed. Focus on TCP protocol traffic, and filter other types of traffic to reduce analysis complexity.

[0047] 4. Building a set: For each data packet processed by the above, the header information, payload, and timestamp are extracted according to its structural characteristics. These extracted information together form a data packet set , where represents the sequence number or unique identifier of the data packet. Multiple data packet sets are further combined into a complete session s, which will be the basic unit for intrusion detection model processing, used for subsequent analysis and detection work.

[0048] The th data packet set is represented as:

[0049]

[0050] In the formula, represents the header information of the th data packet, represents the payload information of the th data packet, represents the timestamp information of the th data packet; represents the number of data packets in a complete session;

[0051] The complete session is represented as:

[0052] .

[0053] 5. Flow packet feature extraction: In the process of flow packet processing, the header and the payload bear completely different responsibilities. First, in terms of information content, the header is mainly responsible for encapsulating control information to ensure that the data packet can be correctly routed and transmitted; while the payload carries the actual data content to be transmitted. Second, in terms of length, the header usually remains in a relatively fixed range, about 60 bytes, while the length of the payload varies greatly, which can be from 0 bytes to 1000 bytes. Given these differences, it is obviously unreasonable to use a single model to process the header and the payload at the same time, because doing so may ignore the differences between the two in terms of characteristics and importance. In addition, the timestamp, as an important part of the data packet, contains a wealth of information, but it has often been ignored or not fully utilized in the past. In order to make more effective use of this information, a more fine-grained and targeted strategy is adopted to process the features of the header, the payload, and the timestamp respectively. This method aims to delve into the unique value of each part of the data, so as to more accurately extract session features and provide strong support for subsequent intrusion detection work. For each , after feature extraction by different methods, the header feature , the payload feature , and the time feature are converted into the extracted feature sequence , as shown below:

[0054] , , The value range of is 1-

[0055] Among them, the one-dimensional convolution attention neural network module is used to extract the header feature, and the header feature of the th data packet is represented as:

[0056]

[0057] In the formula, represents the extraction of the header feature.

[0058] The byte length of the payload has uncertainty, and the change in this length directly hinders its direct input into the feature extraction module. Currently, the common means to handle this problem include truncation and padding, but both methods have their own limitations: truncation operation may cause the loss of key information, while padding operation may introduce unnecessary noise, thereby affecting the accuracy of feature extraction. In order to more effectively cope with the challenge of payload length variation and accurately extract its features, N-gram frequency analysis and one-dimensional convolution attention neural network (1D-CAN) module are innovatively adopted. These two methods can flexibly handle payloads of different lengths, avoiding both information loss and noise interference, thereby achieving accurate extraction of payload features.

[0059] N-gram frequency processing: Given the characteristic that the payload of the traffic packet is composed of hexadecimal strings, the N-gram frequency statistical method is introduced. This method calculates the frequency of every N character combination in the original traffic data, and then integrates these frequency statistics into a one-dimensional vector with a fixed length. It is worth noting that the size of N directly determines the richness of the context information that can be considered. After several experiments, the value of N is finally set to 2, so the length of the vector obtained is 256. Figure 1 An example of how to generate an N-gram frequency vector when N is 2 is shown.

[0060] The N-gram frequency analysis and one-dimensional convolution attention neural network module are used to extract payload features, and the extracted payload feature of the first data packet is represented as:

[0061]

[0062] In the formula, represents payload feature extraction, represents N-gram frequency processing.

[0063] The timestamp carries the time attribute of the traffic packet. In order to make it adapt to the time feature extraction module, the function is used to smooth the timestamp, thereby extracting the effective time feature.

[0064] The feature sequence of a complete session is represented as:

[0065]

[0066] In the formula, , represents the feature sequence of the first data packet.​

[0067] The step of deep-level feature extraction on the feature sequence is as follows:

[0068] Each feature sequence is time-encoded, and then the encoding result is sequentially transmitted to the multi-head deformable attention module, the add&norm module and the feed forward module. After the process is repeated N times, a brand new deep-level conversation feature is obtained .

[0069] Before the intrusion detection, the network intrusion detection model based on the N-gram frequency and the fusion improved algorithm is constructed, which includes a traffic packet preprocessing module, a traffic packet conversation feature extraction module, a deep-level conversation feature extraction module and a classification module. The traffic packet preprocessing module segments and extracts information from the original network data traffic packet to obtain several complete conversations. The traffic packet conversation feature extraction module extracts features from the complete conversations to obtain a feature sequence containing packet header features, payload features and time features. The deep-level conversation feature extraction module extracts deep-level features from the feature sequence to obtain deep-level conversation features. The classification module classifies the network data traffic detection results based on the deep-level conversation features.

[0070] The data flow in the network intrusion detection model based on the N-gram frequency and the fusion improved algorithm includes:

[0071] Data collection: Use a data collector to collect full-flow data of specified network equipment to realize collection of the original data set and store it in a file in the pcap format; data labeling: manually label the labels of attack samples according to the characteristics of network traffic attacks, wherein the labeling values of normal samples, attack samples (including DoS, Probe, U2R and R2L) are 0 and 1 respectively; data preprocessing: step 3.1: N-gram frequency conversion: as shown in the table, the data distribution after downsampling is shown in Table 1. Figure 1

[0072] Table 1

[0073]

[0074] The work of the fusion improved algorithm includes:

[0075] ​​1. The preprocessed traffic data is divided into three parts, but even within the same part, there are significant differences between traffic features. These large differences in features play a key role in identifying intrusion behavior for the model. In order to make full use of the difference and spatial characteristics of traffic features, an attention mechanism is introduced to optimize the one-dimensional convolutional neural network, and on this basis, a one-dimensional convolutional attention network (1D-CAN) module is proposed. In the 1D-CAN module, the attention mechanism calculates the maximum and average values in the channel dimension, and then applies a one-dimensional convolution operation to the newly generated feature sequence, finally outputting a single-channel feature sequence. This mechanism can assign different weights to input traffic features, emphasizing more important features during feature extraction and effectively improving detection accuracy. Figure 3 The specific structure of the module is shown in detail.

[0076] 2. The traditional attention encoder is improved. Compared with the traditional self-attention, the deformable attention adds a bias learning network. The bias learning network structure is shown in Figure 4 , and the deformable attention structure is shown in Figure 5 . Assuming that the given input vector , first, one-dimensional convolution is performed on the input to generate a query vector , and reference points are generated according to the dimensions of the input . The query vector is input into the bias learning network to generate the offset of each reference point (to prevent the offset from being too large, a preset parameter s is used to control the offset), second, the deformation point is obtained by adding the reference point and the offset , and the bilinear interpolation sampling is performed on the input according to the coordinate position corresponding to the deformation point to obtain the sampled feature vector, and then one-dimensional convolution is performed to generate the key-value vector and the value vector . Finally, , , and the relative position offset are calculated to obtain the output result. The deformable attention calculation is as follows:

[0077] , ,

[0078]

[0079]

[0080]

[0081]

[0082] In the formula: represents a bilinear interpolation, represents the output of the head, is the total number of module heads, represents a bias table, represents the scaled feature dimension, Softmax is a normalization function, is a reference point, is an output vector, tanh is a hyperbolic tangent function, DMHA is a multi-head variability attention mechanism, and Concat is splicing, , , , is a projection matrix, , , linearly map , , , linearly map the outputs of multiple heads.

[0083] It also includes model training: inputting the training set into the detection model for training. Parameter optimization: adjusting the hyperparameters of the model. Set the initial parameters as shown in Table 2;

[0084] Table 2

[0085]

[0086] The steps of model evaluation and optimization include: performance test: set the optimal parameters into the model for prediction. Performance evaluation: statistics of the confusion matrix composed of prediction results and true results, and calculation of accuracy, precision and detection rate and other indicators by using formula (11-13).

[0087]

[0088]

[0089]

[0090] In the formula: represents the true positive rate, represents the false negative rate, represents the false positive rate, represents the true negative rate, represents the accuracy of the model; represents the precision of the model; represents the recall rate of the model, sensitivity, The detection rate, which ranges from [0, 1], 1 indicates that the model performance is the best, and 0 indicates that the model performance is the worst.

[0091] In the embodiment, the training phase and the test phase are divided, the key features in the training set are extracted through N-Gram frequency, then the features are loaded into the fusion improved algorithm model as input for training, and the network traffic features are extracted by using 1D-CAN algorithm, and the 1D-DAT model learns the deep key features. In the test phase, the features of the test data are input into the trained fusion classification model, and whether the classification result is correct or not is judged according to the category probability distribution output by the model, and then the model evaluation is carried out.

[0092] Example 2:

[0093] The application also provides a network intrusion detection device based on N-gram frequency and fusion improved algorithm, which comprises a data acquisition module, which is responsible for collecting traffic data of a monitored network in real time and storing the data to a disk, and transmitting the data to a data analysis and standardization module for processing; the data analysis and standardization module receives the data transmitted by the data acquisition module, completes data decoding and standardization processing, and extracts features to generate a data set suitable for a network intrusion detection model; an intrusion flow detection module receives the standardized data, selects a feature subset and trains a model, classifies network traffic, identifies intrusion flow, and outputs a prediction result, a confusion matrix and evaluation indexes for a detection result display and interaction module to display; a result visualization and interaction module displays and interacts with the detection result.

[0094] The above is only the preferred specific embodiment of the application, but the protection scope of the application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A network intrusion detection method based on an improved N-gram frequency and fusion algorithm, characterized in that, Includes the following steps: The process involves acquiring raw network data traffic packets and segmenting them into several independent initial sessions. Prior to intrusion detection, it also includes constructing a network intrusion detection model based on N-gram frequency and a fusion-improved algorithm. This model comprises a traffic packet preprocessing module, a traffic packet session feature extraction module, a deep session feature extraction module, and a classification module. The traffic packet preprocessing module segments and extracts information from the raw network data traffic packets to obtain several complete sessions. The traffic packet session feature extraction module extracts features from these complete sessions, resulting in a feature sequence containing header features, payload features, and time features. The deep session feature extraction module performs deep feature extraction on the feature sequence to obtain deep session features. The classification module classifies network data traffic detection results based on deep session features; Sort the data packets in the initial session by timestamp and remove redundant information from the data packets; Extract the packet header information, payload, and timestamp from the data packet separately; The header information, payload, and timestamp extracted from the same data packet are combined into a data packet set; all data packet sets in the same session constitute a complete session; the first The set of data packets is represented as: In the formula, Indicates the first The header information of each data packet. Indicates the first The payload information of each data packet Indicates the first The timestamp information of each data packet; Indicates the number of packets in a complete session; a complete session Represented as: ; The packet header features, payload features, and time features of the complete session are extracted to obtain a feature sequence; then, deep feature extraction is performed on the feature sequence to obtain deep session features; a one-dimensional convolutional attention neural network module is used to extract the packet header features, and the extracted header features are... The header characteristics of each data packet are represented as follows: In the formula, This represents the extraction of payload features; N-gram frequency analysis and a one-dimensional convolutional attention neural network module are used to extract payload features, and the extracted features are... The payload characteristics of each data packet are represented as follows: In the formula, This indicates payload feature extraction. Indicates N-gram frequency processing; employs The function smooths the timestamp and extracts the time features. A complete session's characteristic sequence Represented as: In the formula, , Indicates the extracted first The characteristic sequence of each data packet The value range is 1- ; The classification of network data traffic detection results based on deep session features involves extracting deep features from the feature sequences to obtain deep session features. The steps are as follows: For each feature sequence... Temporal encoding is performed, and the encoded results are then sequentially passed to the multi-head deformable attention module, the add&norm module, and the feed-forward module. This process is repeated N times to obtain novel deep conversation features. .

2. The network intrusion detection method according to claim 1, characterized in that: Removing redundant information from data packets includes: Remove IP address and MAC address information from each packet; delete protocol version, options field, padding field, and differentiated services field; filter traffic types other than TCP protocol traffic.

3. A network intrusion detection device based on an N-gram frequency and fusion improved algorithm, used to implement the network intrusion detection method of the N-gram frequency and fusion improved algorithm as described in any one of claims 1 or 2, characterized in that: It includes a data acquisition module, which is responsible for collecting traffic data of the monitored network in real time, storing the data on the disk, and transmitting it to the data parsing and standardization module for processing; The data parsing and standardization module receives data from the data acquisition module, performs data decoding and standardization, and extracts features to generate a dataset suitable for the network intrusion detection model. The intrusion traffic detection module receives standardized data, performs feature subset selection and model training, classifies network traffic, identifies intrusion traffic, and outputs prediction results, confusion matrix, and evaluation metrics for display in the detection result display and interactive module. The results visualization and interaction module displays and provides interactive feedback on the test results.

Images