Robustness evaluation method for intrusion detection system

Abstract

The invention discloses a robustness evaluation method for an intrusion detection system. The method comprises the following steps: S1, extracting network traffic data features and converting the network traffic data features into a pcap file; s2, preprocessing the data in the step S1, and dividing the data into benign traffic and attack traffic; s3, performing distance optimization-based adversarial attack iterative training on the attack traffic to increase tiny disturbance so as to generate an optimal adversarial sample; and S4, mixing the optimal adversarial sample traffic and the benign traffic into a robustness evaluation set, inputting the robustness evaluation set into a CBTC intrusion detection system for evaluation, and checking classification results of the test set and the robustness evaluation set. The method can help design a deep learning model with higher robustness. According to the innovative robustness evaluation method, more targeted optimization suggestions can be provided for the system, developers can be helped to recognize and repair the potential poor robustness problem in the model design stage, and therefore the overall defense capacity of the intrusion detection system is improved.

Description

Technical Field

[0001] The invention relates to the technical field of network traffic analysis, and in particular to a robustness evaluation method for an intrusion detection system. Background Art

[0002] Intrusion detection system is one of the key technologies for network security defense, which aims to detect and defend against malicious activities in the network. Traditional IDS is mainly based on misuse detection and anomaly detection. Misuse detection can effectively deal with known threats by identifying known attack patterns, but it is powerless against unknown attacks and new threats. Anomaly detection can detect unknown attacks by establishing a baseline of normal traffic behavior and detecting activities that deviate from the baseline, but it is prone to a high false alarm rate.

[0003] In recent years, with the development of machine learning and deep learning technologies, intrusion detection systems based on these technologies have gradually emerged. Deep learning models are widely used in network traffic analysis, improving detection accuracy and real-time response capabilities. However, these models are usually black box models, lack of interpretability, and are very sensitive to small perturbations of the input data.

[0004] Adversarial attacks are a type of attack that has been widely studied in the field of machine learning in recent years. By introducing small perturbations in the input data, attackers can cause significant deviations in the output of deep learning models. Adversarial attacks are particularly threatening in intrusion detection systems because these perturbations are difficult for humans to detect, but can effectively evade detection by intrusion detection systems, causing the system to make incorrect decisions. Existing intrusion detection systems generally show insufficient robustness when dealing with adversarial attacks and are easily affected by malicious inputs.

[0005] The current prior art mainly has the following defects:

[0006] 1) In the design and application of current intrusion detection systems, most of them focus on optimizing detection accuracy and response speed, while ignoring the robustness of the system in the face of adversarial attacks. The lack of specially designed robustness evaluation methods for intrusion detection systems makes it difficult to effectively evaluate their performance in various complex and potential threat environments. This limitation means that even if the model performs well in ordinary test environments, it may not be able to resist advanced and hidden adversarial attacks in actual applications, thus exposing the entire system to serious security risks.

[0007] 2) Although existing robustness evaluation methods can reveal the performance of models under traditional attacks to a certain extent, these methods do not dig deep enough into the vulnerabilities of deep learning models. Due to the development of adversarial attack technology, existing evaluation models based on traditional attack methods are difficult to fully reveal the performance degradation of deep learning models when facing strong adversarial attacks.

[0008] 3) In the field of computer vision research, adversarial attack is a technical method that can add tiny disturbances to samples to mislead deep learning models into classification errors, and then evaluate the robustness of the system. Intrusion detection systems must have strong enough robustness, because when an intrusion detection system is attacked, it will be misled to make wrong decisions, which will directly lead to serious network security problems and cause great harm. Summary of the invention

[0009] The purpose of the present invention is to provide a robustness evaluation method for an intrusion detection system in view of the above problems. The anti-attack technology in the field of computer vision is modified and introduced into the intrusion detection system to make up for the problem that the intrusion detection system has no robustness evaluation method.

[0010] The present invention optimizes based on distance and can generate more targeted and hidden adversarial samples, thereby launching more challenging attacks on deep learning models. This method can effectively expose the weaknesses of the model under adversarial attacks, making the robustness evaluation of the intrusion detection system more accurate and comprehensive.

[0011] In order to achieve the above object, the technical solution of the present invention is:

[0012] A robustness evaluation method for an intrusion detection system, comprising:

[0013] S1, extract network traffic data features and convert them into pcap files;

[0014] S2, pre-processing the data in step S1, dividing it into benign traffic and attack traffic;

[0015] S3, use distance optimization-based adversarial attack iterative training to add tiny perturbations to the attack traffic and generate the best adversarial samples;

[0016] S4. Mix the best adversarial sample traffic and benign traffic into a robustness evaluation set, input it into the CBTC intrusion detection system for evaluation, and check the classification results of the test set and the robustness evaluation set.

[0017] As an improvement to the above technical solution, the network flow data is obtained from a flow probe introduced by the wireless router of the intrusion detection system, and features are extracted using CICFlowMeter.

[0018] As an improvement to the above technical solution, in step S3, the step of the distance optimization-based adversarial attack iterative training is:

[0019] S31, taking the attack traffic as the original input sample x, setting the optimization parameters c, κ, max_iter, and performing iterative training on the input sample x;

[0020] w = arctanh(x);

[0021] Adversarial example x′=(tanh(w)+1) / 2;

[0022] Minimize the objective function f(x′)=min(max(Z(x′) i -Z(x′) t -κ));

[0023] The total loss, that is, the total loss function, is expressed as

[0024] Where w is the inverse hyperbolic tangent function of the original input sample x, f(x′) is an auxiliary loss function used to measure the attack effect of the adversarial sample x′; Z(x′) is the logical output of the model, i is the number of iterations, ranging from 1 to max_iter, max_iter is the maximum number of iterations; ||x′-x|| p represents the p-norm of the perturbation; t is the target category; c is a constant that balances the perturbation amount and the attack effect; κ is an adjustable parameter used to control the confidence of the adversarial sample;

[0025] S32. If the maximum number of iterations is reached or the optimization target converges, that is, J(x′)<loss, the optimization process is stopped and loss=J(x′), that is, x′ is obtained. best = x′;

[0026] S33. If the maximum number of iterations is not reached or the optimization target converges, continue to optimize and repeat steps S31 and S32 until the maximum number of iterations is reached or the optimization target converges, and finally the best adversarial sample x′ is obtained. best =x′.

[0027] As an improvement to the above technical solution, the p norm is the l0 norm, the l2 norm or the l ∞ norm, the l0 norm refers to the number of non-zero elements in the perturbed elements, and the l2 norm is the sum of the squares of the perturbed elements and then the square root. ∞ The norm is the maximum value of each element of the perturbation, where l2 represents the adversarial sample with the smallest possible overall perturbation generated by optimizing the Euclidean distance.

[0028] As an improvement to the above technical solution, the input data of the CBTC intrusion detection system is the CICIDS-2017 dataset, and the CICIDS-2017 dataset is divided into a training set and a test set after preprocessing.

[0029] As an improvement to the above technical solution, the strength evaluation of the robustness is measured using the accuracy ACC, recall rate Recall, F1 value, and attack success rate ASR;

[0030] Among them, ACC is used to evaluate the classification ability of the model before and after mixed sampling, and the formula is expressed as:

[0031]

[0032] Recall is used to measure the model's ability to identify positive samples, and the formula is expressed as;

[0033]

[0034] The F1 value provides a balanced evaluation indicator when considering both the accuracy and recall of the model; the formula is:

[0035]

[0036] ASR is the pre-attack ACC before and post-attack ACC after The difference is used to measure the classification ability of the attack misleading model, and the formula is: ASR = ACC after -ACC before ; The higher the ASR, the stronger the attack effect on the model and the weaker the model robustness.

[0037] As an improvement to the above technical solution, the CBTC intrusion detection system is trained by convolutional neural networks, long short-term memory networks, and two residual neural networks of different depths, ResNet18 and ResNet34. The model training uses Adam optimizer, Relu activation function, and cross entropy loss for gradient descent. The batch size is 128, the learning rate is set to 0.0001, and 50 epochs are trained.

[0038] Compared with the prior art, the present invention has the following advantages and positive effects:

[0039] The robustness evaluation method of the intrusion detection system of the present invention introduces the adversarial attack technology in the field of computer vision into the intrusion detection system after modification, thereby making up for the problem that the intrusion detection system has no robustness evaluation method. The method can generate more targeted and more concealed adversarial samples, thereby carrying out more challenging attacks on deep learning models. It can effectively reveal the weaknesses of the model under adversarial attacks, making the robustness evaluation of the intrusion detection system more accurate and comprehensive.

[0040] The present invention can help design more robust deep learning models. This innovative robustness evaluation method can provide more targeted optimization suggestions for the system, helping developers to identify and fix potential poor robustness problems during the model design phase, thereby improving the overall defense capabilities of the intrusion detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 This is a block diagram of the intrusion detection system robustness evaluation method of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work, any modifications, equivalent substitutions, improvements, etc., should be included in the protection scope of the present invention.

[0044] like Figure 1 As shown, this embodiment discloses a robustness evaluation method for an intrusion detection system. First, network traffic data is obtained from a flow probe introduced by the wireless router of the intrusion detection system, and features are extracted and converted into a usable pcap file using CICFlowMeter. After preprocessing the data, benign traffic and attack traffic are distinguished. The attack traffic is subjected to adversarial attack iterative training based on distance optimization to add small perturbations to generate adversarial samples. Then, the adversarial sample traffic and benign traffic are mixed into a robustness evaluation set, which is input into the CBTC intrusion detection system of each deep learning model pre-trained using a public data set. Finally, the classification results of the test set and the robustness evaluation set are checked, and the robustness performance of the model is evaluated by the decrease in the evaluation index.

[0045] 1 Adversarial Attack Based on Distance Optimization

[0046] Adversarial attacks aim to trick deep learning models into outputting incorrect predictions by making subtle modifications to input data. Such attacks reveal the model's vulnerability to malicious inputs, thereby evaluating the model's security and robustness.

[0047] Adversarial attacks based on distance optimization transform the task of generating adversarial samples into a mathematical optimization problem by constructing an optimization problem, so that the attack samples can successfully deceive the model while maintaining similarity with the original samples. The goal of the attack is to find an adversarial sample x′ that is misclassified in the model and the perturbation amount ||x′-x|| p As small as possible. Specifically, it can be described as the optimization problem of the following formula 1:

[0048]

[0049] J(x′) can be considered as the total loss. Among them, ||x′-x|| p The p-norm represents the disturbance, usually the l0-norm, l2-norm or l ∞ norm. The l2 norm is to generate adversarial samples with the smallest overall perturbation possible by optimizing the Euclidean distance. c is a constant that balances the perturbation amount and the attack effect; f(x′) is an auxiliary loss function used to measure the attack effect of the adversarial sample x′. The loss function f(x′) is defined as the following formula 2:

[0050] f(x′)=min(max(Z(x′) i -Z(x′) t -κ)) (2)

[0051] Z(x′) is the model’s logical output, t is the target category, and κ is an adjustable parameter used to control the confidence of the adversarial sample. By minimizing f(x′), we can ensure that the adversarial sample x′ is classified as the target category t with a higher confidence than the non-target category. The specific process is shown in Algorithm 1.

[0052] Algorithm 1: Adversarial attack based on distance optimization

[0053]

[0054] As shown in Algorithm 1, the calculation process is as follows: First, the original sample x is used as the initial value of the adversarial sample x′, and the optimization parameters c, κ and the maximum number of iterations are set. Secondly, through the iterative optimization method, the objective function J(x′) is minimized to obtain the optimal perturbation ||x′-x|| p . Finally, according to the optimization results, the final adversarial sample x′ is generated. If the maximum number of iterations is reached or the optimization target converges, the optimization process is stopped; otherwise, the optimization continues. The purpose of the intrusion detection system is to detect attacks and defend against them. Benign samples can pass through, so in order to deceive the model classification, the algorithm will perturb the attack class samples to the benign class samples.

[0055] 2 Experimental design

[0056] 2.1 Experimental Preparation

[0057] To evaluate the robustness of the model, the CICIDS-2017 dataset is the input data of the intrusion detection system, and the accuracy (ACC), recall (Recall) and F1 value (F1score), as well as the attack success rate (ASR) are used to measure the strength of the robustness. Among them, ACC is used to evaluate the classification ability of the model before and after the attack, which can be expressed as shown in the following formula 3.

[0058]

[0059] Recall is used to measure the model's ability to identify positive samples, as shown in Formula 4 below.

[0060]

[0061] F1 provides a balanced evaluation metric by considering both the precision and recall of the model, as shown in Formula 5 below.

[0062]

[0063] ASR is the pre-attack ACC before and post-attack ACC after The difference is used to measure the classification ability of the attack misleading model, and the formula is: ASR = ACC after -ACC before ; The higher the ASR, the stronger the attack effect on the model and the weaker the model robustness.

[0064] ASR=ACC after -ACC before (6)

[0065] The deep learning models used in the experiment are convolutional neural network (CNN), long short-term memory network (LSTM) and two residual neural networks (ResNet) of different depths, ResNet18 and ResNet34. The model training uses Adam optimizer, Relu activation function, cross entropy loss for gradient descent, batch size is 128, learning rate is set to 0.0001, and training is done for 50 epochs.

[0066] 2.2 Experimental Results and Analysis

[0067] Model evaluation and robustness testing

[0068] First, controlled experiments using C&W attack and distance optimization-based adversarial attack on the CNN model demonstrate the effectiveness of the method. The results are shown in Table 1 below.

[0069] Table 1: C&W and distance optimization-based adversarial attack experiments

[0070]

[0071] Experiments show that on the CNN model, ASR increased by 2.26%, recall decreased by 3.93%, and F1 value decreased by 2.46%. Compared with C&W, the distance-optimized adversarial attack with mixed sampling and optimal feature sets with strong correlation and low redundancy is obviously more aggressive. Ablation experiments show that compared with C&W attacks, the distance-optimized adversarial attack has better effects and more significant evaluation of the model's robustness.

[0072] Then, adversarial attacks based on distance optimization are used on CNN, LSTM, ResNet18 and ResNet34 to evaluate the robustness of the models by the decrease in their evaluation indicators. The results are shown in Table 2.

[0073] Table 2: Evaluation indicators of each model before and after the adversarial attack based on distance optimization

[0074]

[0075] The performance of CNN, LSTM and ResNet models dropped significantly after being attacked by distance optimization, proving that these deep learning models all have poor robustness. Due to its simple structure, CNN has a model performance of 35.44% ASR, 75.95% Recall, and 79.97% F1. LSTM is a model suitable for time series data such as network traffic, and its structure is relatively complex. Therefore, the decline in robustness is relatively small when facing attacks, but ASR is also 23.69%, Recall is reduced by 51.3%, and F1 is reduced by 56.91. Although ResNet18 with a deeper depth is more robust than CNN in the face of adversarial attacks. ResNet34 has a deeper depth and a more complex model than ResNer18, but it does not show better robustness in adversarial attacks. ASR is 5.78% higher, and the decline in Recll and F1 has also increased by 7.23% and 5.76%. This shows that the relationship between model depth and robustness is not that the deeper the depth, the stronger the robustness.

[0076] The experimental results show that CNN, LSTM, ResNet18 and ResNet34 all have poor robustness problems, which also proves the effectiveness of adversarial attack robustness evaluation based on distance optimization.

Claims

1. A robustness evaluation method for an intrusion detection system, characterized in that: The steps include: S1, extract network traffic data features and convert them into pcap files; S2, pre-processing the data in step S1, dividing it into benign traffic and attack traffic; S3, use distance optimization-based adversarial attack iterative training to add tiny perturbations to the attack traffic and generate the best adversarial samples; S4. Mix the best adversarial sample traffic and benign traffic into a robustness evaluation set, input it into the CBTC intrusion detection system for evaluation, and check the classification results of the test set and the robustness evaluation set.

2. The method for evaluating the robustness of an intrusion detection system according to claim 1, wherein: The network traffic data is obtained from the traffic probe introduced by the wireless router of the intrusion detection system, and features are extracted using CICFlowMeter.

3. The method for evaluating the robustness of an intrusion detection system according to claim 1, wherein: In step S3, the steps of the distance optimization-based adversarial attack iterative training are: S31, taking the attack traffic as the original input sample x, setting the optimization parameters c, κ, max_iter, and performing iterative training on the input sample x; w = arctanh(x); Adversarial example x′=(tanh(w)+1) / 2; Minimize the objective function f(x′)=min(max(Z(x′) i -Z(x′) t -κ)); The total loss, that is, the total loss function, is expressed as Where w is the inverse hyperbolic tangent function of the original input sample x, f(x′) is an auxiliary loss function used to measure the attack effect of the adversarial sample x′; Z(x′) is the logical output of the model, i is the number of iterations, ranging from 1 to max_iter, max_iter is the maximum number of iterations; ||x′-x|| p represents the p-norm of the perturbation; t is the target category; c is a constant that balances the perturbation amount and the attack effect; κ is an adjustable parameter used to control the confidence of the adversarial sample; S32. If the maximum number of iterations is reached or the optimization target converges, that is, J(x′)<loss, the optimization process is stopped and loss=J(x′), that is, x′ is obtained. best = x′; S33. If the maximum number of iterations is not reached or the optimization target converges, continue to optimize and repeat steps S31 and S32 until the maximum number of iterations is reached or the optimization target converges, and finally the best adversarial sample x′ is obtained. best =x′.

4. The method for evaluating the robustness of an intrusion detection system as claimed in claim 3, characterized in that: The p-norm is the l0-norm, l2-norm, or l ∞ norm, the l0 norm refers to the number of non-zero elements in the perturbed elements, and the l2 norm is the sum of the squares of the perturbed elements and then the square root. ∞ The norm is the maximum value of each element of the perturbation, where l2 represents the adversarial sample with the smallest possible overall perturbation generated by optimizing the Euclidean distance.

5. The method for evaluating the robustness of an intrusion detection system as claimed in claim 1, characterized in that: The input data of the CBTC intrusion detection system is the CICIDS-2017 dataset, which is divided into a training set and a test set after preprocessing.

6. The method for evaluating the robustness of an intrusion detection system as claimed in claim 5, characterized in that: The strength of the robustness evaluation is measured using the accuracy ACC, recall Recall, F1 value, and attack success rate ASR; Among them, ACC is used to evaluate the classification ability of the model before and after mixed sampling, and the formula is expressed as: Recall is used to measure the model's ability to identify positive samples, and the formula is expressed as; The F1 value provides a balanced evaluation indicator when considering both the precision and recall of the model; the formula is: ASR is the pre-attack ACC before and post-attack ACC after The difference is used to measure the classification ability of the attack misleading model, and the formula is: ASR = ACC after -ACC before ; The higher the ASR, the stronger the attack effect on the model and the weaker the model robustness.

7. The method for evaluating the robustness of an intrusion detection system as claimed in claim 6, characterized in that: The CBTC intrusion detection system is trained by convolutional neural networks, long short-term memory networks, and two residual neural networks of different depths, ResNet18 and ResNet34. The model training uses the Adam optimizer, Relu activation function, and cross entropy loss for gradient descent. The batch size is 128, the learning rate is set to 0.0001, and the training is 50 epochs.

Images