Network intrusion detection method based on hierarchical Bloom filtering and large language model

Abstract

The invention relates to a network intrusion detection method based on hierarchical Bloom filtering and a large language model, and belongs to the technical field of network security. The method comprises the following steps: carrying out preliminary rapid screening on input network communication data by using a hierarchical Bloom filter, training a network intrusion detection model, carrying out anomaly detection through a final network intrusion detection model obtained after training, and generating a detailed report of detected anomaly by using a large language model. According to the method, time and computing resources are saved, the overall efficiency of detection is also improved, real-time anomaly detection and response can be realized, the interpretability of a detection result is enhanced through a report generated by a large language model, and the specific condition and possible influence of an abnormal behavior can be understood conveniently.

Description

Technical Field

[0001] The invention belongs to the technical field of network security, and in particular relates to a network intrusion detection method based on hierarchical Bloom filtering and a large language model. Background Art

[0002] In recent years, with the rapid development of network technology and intelligent systems, network security is no longer just a means of protecting information, but has become a comprehensive intelligent system that integrates advanced technologies such as advanced intelligent sensors, data analysis tools, and security protection measures. In this system, special attention is paid to network traffic based on the Controller Area Network (CAN), because CAN is widely used in the fields of automobiles and industrial control, and its data communication characteristics have posed new challenges to network security technology. In this context, network traffic anomaly intrusion detection technology is particularly critical, which involves how to identify potential security threats and abnormal behaviors by analyzing CAN network traffic, thereby protecting the network from attacks.

[0003] CAN is a serial communication protocol that allows multiple nodes (such as sensors, actuators, control units, etc.) to exchange information through a communication line. It is an indispensable part of modern automobiles and industrial automation. However, CAN was originally designed for efficient communication and did not put security first, which makes CAN-based network systems vulnerable to network attacks such as message injection and node impersonation. Therefore, traffic anomaly intrusion detection for CAN networks has become an important research direction.

[0004] In view of the characteristics of CAN networks, researchers have developed a variety of intrusion detection system (IDS) solutions based on data analysis and machine learning. These solutions identify abnormal communication patterns by analyzing the message time interval, message type, frequency and other characteristics in the CAN network, thereby discovering potential security threats at an early stage. For example, IDS based on time interval analysis can identify message sending behaviors that do not conform to normal communication patterns; IDS based on deep learning uses complex algorithm models to learn the differences between normal and abnormal messages to improve detection accuracy and efficiency; real-time IDS based on random forests analyzes message features by building multiple decision trees to achieve fast and effective anomaly detection.

[0005] Although these methods have improved the security of CAN networks to a certain extent, they still face many challenges, especially when dealing with complex and diverse attack patterns. In recent years, with the development of deep learning and natural language processing technology, large language models (LLM) have gradually become a new research hotspot. Large language models can generate high-quality texts through training on massive data and perform well in multiple natural language processing tasks. Introducing large language models into network intrusion detection systems can further enhance the intelligence level and response capabilities of the system.

[0006] Specifically, the large language model can be used to generate detailed anomaly detection reports, including descriptions of detected anomalies, potential threat analysis, and countermeasure recommendations. These reports can help the Security Operations Center (SOC) quickly understand and respond to security incidents, thereby improving overall security protection capabilities. In addition, the large language model can also provide smarter intrusion detection strategies and model update recommendations by analyzing historical and real-time data.

[0007] For traffic anomaly intrusion detection based on CAN network, future research will focus more on the innovation of algorithms and technologies, as well as the integrated application of emerging technologies, to cope with the escalating means of network attacks and protect the security of critical infrastructure. In this process, hierarchical Bloom filter, as an efficient and low error rate filtering tool, will play an important role in improving the real-time and accuracy of the detection system. Through a multi-level filtering mechanism, hierarchical Bloom filter can more effectively process massive data, identify abnormal traffic, and ensure safe application in key areas. At the same time, considering the particularity of CAN network, these studies also need to pay more attention to real-time and system stability while ensuring high efficiency and accuracy, to ensure safe application in key areas. Hierarchical Bloom filter can provide higher flexibility and stability through its hierarchical structure and dynamic adjustment capabilities, meeting the development needs of future network intrusion detection technology.

[0008] The existing technology has the following disadvantages:

[0009] 1) High system complexity: Due to the complexity of design and implementation, the existing system faces many difficulties in actual deployment and maintenance, resulting in high maintenance costs and prone to errors.

[0010] 2) Insufficient processing speed: The processing speed of existing systems cannot meet the needs of real-time detection, resulting in a significant increase in detection delay when facing a high-traffic network environment.

[0011] 3) Insufficient recognition capabilities for new unknown attacks: Especially for zero-day attacks, existing systems lack sufficient recognition capabilities and are unable to effectively detect and respond to emerging attack patterns.

[0012] 4) Poor adaptability to diversity: Different vehicles and devices may use different communication protocols and data formats. Existing intrusion detection solutions are difficult to flexibly adapt to these diversities, resulting in limited application in heterogeneous environments.

[0013] 5) High computational complexity: Existing network traffic anomaly intrusion detection solutions have high computational complexity when processing large amounts of data, which slows down the detection speed and cannot meet the real-time requirements. This makes it difficult for the system to maintain efficient operation under high load conditions. Summary of the invention

[0014] The purpose of the present invention is to solve the limitations of existing network intrusion detection solutions, and to provide a network intrusion detection method based on hierarchical Bloom filtering and a large language model, which uses hierarchical Bloom filters for feature extraction, shortens the detection time with its fast search capability, meets the real-time detection requirements, and improves the recognition capability of unknown attacks through automated feature learning. Furthermore, a large language model is used to generate a detailed report of the detected anomalies, enhancing the interpretability and response speed of security events.

[0015] To achieve the above-mentioned purpose, the technical solution of the present invention is: a network intrusion detection method based on hierarchical Bloom filtering and a large language model, which uses a hierarchical Bloom filter to perform preliminary and rapid screening of input network communication data, trains a network intrusion detection model, performs anomaly detection through the final network intrusion detection model obtained after training, and uses a large language model to generate a detailed report of the detected anomalies.

[0016] In one embodiment of the present invention, the method includes:

[0017] Collect network communication data;

[0018] Preprocess the collected network communication data, including data cleaning, category balancing, and feature dimension reduction;

[0019] Use hierarchical Bloom filters to perform preliminary and rapid screening of preprocessed data to quickly determine whether the data may belong to a known set of normal communication patterns;

[0020] Input the data filtered by the hierarchical Bloom filter into the network intrusion detection model, train the network intrusion detection model, and use the final network intrusion detection model obtained by training to identify and classify normal and abnormal communication data in the vehicle network;

[0021] Generates detailed reports of detected anomalies using large language models. The reports include detailed descriptions of detected anomalies, potential threat analysis, and recommended countermeasures.

[0022] In one embodiment of the present invention, the specific implementation method of preprocessing the collected network communication data is as follows:

[0023] Data cleaning: remove invalid data of empty records and undefined records, and handle missing values ​​and outliers;

[0024] Feature selection: Use the Pearson correlation coefficient method to select features, calculate the correlation coefficient between each feature and the label, and select features with higher correlation coefficients; the Pearson correlation coefficient calculation formula is as follows:

[0025]

[0026] Among them, x i and i Represent the feature value and label value of the i-th sample respectively, and Represent the mean of features and labels respectively, and n represents the number of samples;

[0027] Category balance: Use the synthetic minority oversampling technique SMOTE for category balance. For a minority class sample s, randomly select its k nearest neighbor samples, and then randomly interpolate between s and the nearest neighbor samples to generate new samples; the interpolation formula is:

[0028] s new =s+rand(0,1)*(s nm -s)

[0029] Where rand(0,1) represents a random number between 0 and 1. Repeat the class balancing process until the number of minority class samples is balanced with the number of majority class samples.

[0030] Data standardization: Perform min-max standardization on the features and scale the feature values ​​to the interval [0,1]. The standardization formula is:

[0031]

[0032] Among them, x is the original eigenvalue, x′ is the standardized eigenvalue, min and max are the minimum and maximum values ​​of the feature respectively.

[0033] In one embodiment of the present invention, the specific implementation method of using the layered Bloom filter to perform preliminary and rapid screening on the preprocessed data is as follows:

[0034] Hierarchical Bloom filter initialization: The Bloom filter is divided into multiple layers, each layer has an independent bit array and hash function.

[0035] According to the amount of data n and the expected false positive rate p, calculate the optimal bit array size m and number of hash functions k:

[0036]

[0037]

[0038]

[0039] Among them, L is the number of layers, P i is the false alarm rate of each layer. m is rounded up and k is rounded down;

[0040] Construct a hierarchical Bloom filter: For each sample of normal traffic data, use k hash functions to calculate its hash value and set the corresponding position in the bit array to 1;

[0041] Filter data: For each sample of the detection data, use the same k hash functions to calculate its hash value, and check whether the bits in the corresponding bit array are all 1. If so, it is judged as normal traffic, otherwise it is judged as abnormal traffic.

[0042] In one embodiment of the present invention, the specific implementation method of training the network intrusion detection model is as follows:

[0043] The samples judged as abnormal traffic by the hierarchical Bloom filter are divided into training set and test set;

[0044] Perform K-fold cross validation on the training set, adjust the network intrusion detection model hyperparameters, and use methods including grid search to find the optimal hyperparameter combination;

[0045] Retrain the network intrusion detection model on the entire training set using the optimal hyperparameters;

[0046] The performance of the network intrusion detection model is evaluated on the test set, and the final network intrusion detection model is selected based on the performance indicators.

[0047] In one embodiment of the present invention, the network intrusion detection model adopts XGBoost, random forest, decision tree, ExtraTrees, support vector machine, naive Bayes, AdaBoost and MLP models.

[0048] In one embodiment of the present invention, the performance indicators for evaluating the performance of the network intrusion detection model include:

[0049] Accuracy:

[0050] Accuracy:

[0051] Recall:

[0052] F1 score:

[0053] Among them, TP, TN, FP, and FN represent the number of true positive, true negative, false positive, and false negative samples, respectively.

[0054] In one embodiment of the present invention, the specific implementation method of identifying and classifying normal and abnormal communication data in the vehicle network through the final network intrusion detection model is as follows:

[0055] Build an online detection system, that is, cascade the hierarchical Bloom filter and the final network intrusion detection model to perform network intrusion detection on real-time network traffic; the hierarchical Bloom filter initially filters out normal traffic, and the final network intrusion detection model further identifies samples that the Bloom filter determines to be abnormal traffic and outputs the final detection results;

[0056] Collect detection data during the operation of the online detection system, regularly update the hierarchical Bloom filter and the final network intrusion detection model, perform incremental learning of the model, and introduce new data sets to prevent concept drift.

[0057] In one embodiment of the present invention, the final detection result output by the final network intrusion detection model is input into the large language model, and the large language model generates a detailed report of the detected anomaly, and the report content includes a detailed description of the detected anomaly, potential threat analysis and recommended countermeasures.

[0058] In one embodiment of the present invention, the method is applied to intelligent transportation system scenarios and smart vehicle sharing scenarios.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) Efficient detection process: By quickly filtering data through layered Bloom filters, it is possible to efficiently determine whether the data belongs to the normal traffic set, thereby pre-screening a large amount of normal traffic and reducing the amount of data to be processed later. This not only saves time and computing resources, but also improves the overall efficiency of detection.

[0061] (2) Real-time anomaly detection and response: Experimental results show that the detection time of this scheme is only 17.17 microseconds, which is much smaller than the upper limit of 38 microseconds for malicious attack prevention detection, and can achieve real-time anomaly detection and response.

[0062] (3) Introduction of large language models: Large language models are used to generate detailed anomaly detection reports, including anomaly descriptions, threat analysis, and countermeasures. The reports generated by large language models enhance the interpretability of detection results and facilitate understanding of the specific circumstances and possible impacts of abnormal behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1The present invention is a flow chart of a network intrusion detection method based on hierarchical Bloom filtering and large language model.

[0064] Figure 2 The present invention is a framework diagram of a network intrusion detection method based on hierarchical Bloom filtering and a large language model. DETAILED DESCRIPTION

[0065] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0067] In addition, the terms "first" and "second" etc. are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0068] The present invention provides a network intrusion detection method based on hierarchical Bloom filtering and a large language model, which uses a hierarchical Bloom filter to perform preliminary and rapid screening on input network communication data, trains a network intrusion detection model, performs anomaly detection through the final network intrusion detection model obtained after training, and generates a detailed report of the detected anomaly using a large language model.

[0069] The following is the specific implementation process of the present invention.

[0070] Please refer to Figure 1 , Figure 1 : is a flowchart of a network intrusion detection method based on hierarchical Bloom filtering and a large language model provided in an embodiment of the present application, comprising the following steps:

[0071] S101. Data collection and preprocessing. This method does not rely on a specific data set, but adopts diversified data sources to cover various types of network attacks and normal communication behaviors to ensure the breadth and practicality of the research. This process involves integrating network communication data from multiple sources; preprocessing the collected CAN message data, including data cleaning, feature selection, and category balancing. Data cleaning ensures that the data set is purified before model training; by implementing feature selection, it aims to avoid overfitting and underfitting, optimize model performance, and shorten model training and response time, thereby improving overall efficiency;

[0072] 1) Data Collection: Network communication data are collected from multiple sources, including public datasets such as KDD Cup 99, NSL-KDD, UNSW-NB15, and CIC-IDS2017, as well as raw network traffic data captured in actual network environments.

[0073] 2) Data cleaning: remove invalid data such as empty records and undefined records, and handle missing values, outliers, etc.

[0074] 3) Feature selection: Use the Pearson correlation coefficient method to select features, calculate the correlation coefficient between each feature and the label, and select the feature with a higher correlation coefficient. The Pearson correlation coefficient calculation formula is as follows:

[0075]

[0076] Among them, x i and i Represent the feature value and label value of the i-th sample respectively, and Represent the mean of features and labels respectively, and n represents the number of samples;

[0077] 4) Category balance: Use the synthetic minority oversampling technique SMOTE for category balance. For a minority class sample s, randomly select its k nearest neighbor samples, and then randomly interpolate between s and the nearest neighbor samples to generate new samples; the interpolation formula is:

[0078] s new =s+rand(0,1)*(s nm -s)

[0079] Where rand(0,1) represents a random number between 0 and 1. Repeat the class balancing process until the number of minority class samples is balanced with the number of majority class samples.

[0080] 5) Data standardization: Perform min-max standardization on the features and scale the feature values ​​to the interval [0,1]. The standardization formula is:

[0081]

[0082] Among them, x is the original eigenvalue, x′ is the standardized eigenvalue, min and max are the minimum and maximum values ​​of the feature respectively.

[0083] S102. Hierarchical Bloom filter: Bloom filter is an efficient data structure that aims to quickly determine whether an element may belong to a set. Data is filtered through Bloom filter to quickly determine whether the data exists in the set. By introducing a multi-layer structure, each layer has a different error rate, which is suitable for frequently changing data sets. Thus, a large amount of normal traffic can be pre-screened, saving time and computing resources for subsequent classification models.

[0084] 1) Hierarchical Bloom filter initialization: According to the amount of data n and the expected false positive rate p, calculate the optimal bit array size m and number of hash functions k:

[0085]

[0086]

[0087] Among them, m is rounded up and k is rounded down;

[0088] 2) Construct a hierarchical Bloom filter: For each sample of normal traffic data, use k hash functions to calculate its hash value and set the corresponding position in the bit array to 1. The hash function can be MurmurHash, FNV, etc.

[0089] 3) Filter data: For each sample of the detection data, use the same k hash functions to calculate its hash value, and check whether the bits in the corresponding bit array are all 1. If so, it is judged as normal traffic, otherwise it is judged as abnormal traffic.

[0090] S103, anomaly detection, the hierarchical Bloom filter may produce false positives, that is, misjudging normal traffic as abnormal traffic. In order to solve this problem, the method of the present invention further introduces a machine learning model to accurately classify samples that the hierarchical Bloom filter judges as abnormal traffic;

[0091] 1) Feature extraction: Extract the features of samples that are judged as abnormal traffic by the hierarchical Bloom filter as input to the machine learning model;

[0092] 2) Select machine learning models: Use XGBoost, Random Forest, Decision Tree, Extra Trees, Support Vector Machine, Naive Bayes, AdaBoost and MLP models for training and performance evaluation.

[0093] a) XGBoost and AdaBoost use boosting algorithms to perform weighted combinations of multiple classifiers, with weights updated based on the classification error rate of the previous round. The model output is the weighted average of each classifier. The objective function includes a loss function and a regularization term:

[0094]

[0095] Among them, l is the loss function, Ω is the regularization term, and f k is the kth tree.

[0096] b) Random forest, decision tree and ExtraTrees use decision tree structure to divide samples layer by layer. The model output is the voting result of each decision tree. When splitting a node, select the feature and threshold that will reduce the impurity the most. The commonly used Gini impurity measure is:

[0097]

[0098] Among them, p k is the proportion of samples belonging to the kth class. Extreme random trees randomly select feature subsets when splitting, which increases randomness.

[0099] c) The support vector machine maps the sample to a high-dimensional feature space and finds the optimal classification hyperplane. The model output is the signed distance from the sample to the hyperplane. Introduce the slack variable x i , the optimization goal is:

[0100]

[0101] sty i (w T x i +b)≥1-ξ i ,ξ i ≥0,1=1,2,...,n

[0102] Among them, C is the penalty coefficient, x i is the slack variable of the i-th sample.

[0103] d) Naive Bayesian method is based on the assumption of independence of feature conditional probabilities and calculates the posterior probability of each category according to Bayesian theorem. The model output is the category with the largest posterior probability.

[0104] e) The MLP model optimizes the hidden layer weights through forward propagation and back propagation to construct a complex nonlinear decision boundary. The model output is the matching score of each category.

[0105] 3) Model training: For samples that are judged as suspicious abnormal by the Bloom filter, the above machine learning model is used for training:

[0106] a) Divide the data into training set and test set.

[0107] b) Perform K-fold cross validation on the training set and adjust model hyperparameters, such as the number and depth of trees, the kernel function type of SVM, the number of hidden layers of MLP, etc. Use grid search and other methods to find the optimal hyperparameter combination.

[0108] c) Retrain the model on the entire training set using the optimal hyperparameters.

[0109] d) Evaluate model performance on the test set, such as accuracy, precision, recall, F1 score, AUC, etc. Select the final model based on the performance indicators.

[0110] S104, Evaluation and Application

[0111] 1) Comprehensively evaluate the performance of different machine learning models on the test set and select the best performing model as the final intrusion detection model. The evaluation indicators include:

[0112] Accuracy:

[0113] Accuracy:

[0114] Recall:

[0115] F1 score:

[0116] Among them, TP, TN, FP, and FN represent the number of true positive, true negative, false positive, and false negative samples, respectively.

[0117] 2) Build an online detection system, cascade the layered Bloom filter and the machine learning model, and perform intrusion detection on real-time network traffic. The layered Bloom filter initially filters out most of the normal traffic, and the machine learning model further identifies the traffic that the Bloom filter determines to be suspicious, and outputs the final detection results.

[0118] 3) Collect detection data during system operation, regularly update hierarchical bloom filters and machine learning models, and continuously improve system performance. Perform incremental learning of the model and introduce new data sets to prevent concept drift.

[0119] S105: The large language model generates a detection report. The detected abnormal data is input into the large language model. The large language model generates a report including abnormal description, threat analysis and corresponding countermeasures.

[0120] The present invention is an intrusion detection model integrating hierarchical bloom filtering, machine learning algorithm and large language model. Figure 2 , Figure 2It is a framework diagram of a network intrusion detection method based on layered bloom filtering and large language model. First, the data is preprocessed by data cleaning, category balancing, feature dimension reduction, etc.; the preprocessed data is filtered by Layered BloomFilter to quickly determine whether the data exists in the collection, and pre-screen a large amount of normal traffic, thereby saving time and computing resources for subsequent classification models. After filtering, the data is divided into two parts: one part is the data that passes the Layered Bloom Filter, and the other part is the data that fails. For the data that passes, they will undergo further normal processing. The data that fails is considered redundant or abnormal, and then the specific anomaly is detected through the machine learning model.

[0121] In the model training stage, the present invention adopts an innovative combined model training method. First, Layered Bloom Filter is used for anomaly detection. Layered Bloom Filter has high judgment efficiency, and its reasoning time is one order of magnitude less than that of machine learning model, which makes it have significant advantages in preliminary screening of abnormal data. However, Layered Bloom Filter may produce false positives, that is, normal traffic is misjudged as abnormal traffic. In order to solve this problem, this scheme further introduces a variety of machine learning models, including XGBoost, random forest, decision tree, Extra Trees, support vector machine, naive Bayes, AdaBoost and MLP, etc., to accurately classify the traffic judged as abnormal by Layered Bloom Filter. Although these machine learning models have high judgment accuracy, they are time-consuming. Therefore, this two-level screening method is adopted to ensure both detection speed and classification accuracy. This combined use method can effectively improve the accuracy of prediction and also ensure the efficiency of the system. At the same time, for the detected anomaly, it will be used as input to generate the corresponding detection report using a large language model.

[0122] The specific implementation is as follows:

[0123] 1. Data preprocessing:

[0124] 1) Clean the network data packets and remove empty and undefined records.

[0125] 2) Perform feature selection and remove non-critical features, such as removing features such as ‘CAN_ID’ and ‘DLC’ that contribute little to the model.

[0126] 3) Create artificial instances by class balancing through synthetic minority oversampling technique (SMOTE).

[0127] 4) Standardize the processed data to ensure data quality.

[0128] 2. Hierarchical Bloom filter parameter selection:

[0129] 1) The number of bits m of the hierarchical Bloom filter is selected according to the scale of network traffic data, for example, m=100000.

[0130] 2) The number of hash functions k is selected according to the empirical formula k=ln(2)*m / data volume, for example k=3.

[0131] 3. Feature extraction:

[0132] 1) Extract the message ID, data length, data content and other features, such as extracting fields such as 'DATA0' to 'DATA7'.

[0133] 2) Normalize the extracted features using min-max normalization.

[0134] 4. Machine Learning Model Selection:

[0135] 1) Select ExtraTrees (ET) as the classification model.

[0136] 2) Model training uses preprocessed network data packets.

[0137] 3) Use methods such as cross-validation to evaluate model performance and select the optimal model.

[0138] 5. Large language model generates detection report:

[0139] 1) Input the detected abnormal data into the large language model to generate an anomaly detection report containing the following content.

[0140] 2) Exception description: A detailed description of the detected exception, including time, source, type, and other information.

[0141] 3) Threat analysis: Analyze the potential threats that the anomaly may bring, including the possible impact on the system.

[0142] 4) Countermeasures: Provide specific countermeasure suggestions, such as isolating abnormal nodes, patching vulnerabilities, and improving monitoring levels.

[0143] The performance of the method of the present invention on the car hacker dataset was evaluated by F1 score, recall rate, precision and accuracy, which are usually used to measure the effect of machine learning algorithms. In addition, the performance of the present invention was compared with seven other commonly used network intrusion detection methods, including support vector machine, random forest, naive Bayes, decision tree, XGBoost, AdaBoost and multi-layer perceptron, and the results are shown in Table 1. In addition, a comparative ablation experiment was also conducted, in which the dataset filtered by the hierarchical Bloom algorithm was directly used for anomaly detection using the above model, and the results are shown in Table 2.

[0144] Table 1

[0145]

[0146]

[0147] Table 2

[0148]

[0149] Tables 1 and 2 show the experimental results without and with hierarchical Bloom filters, respectively. As can be seen from Table 1, without the hierarchical Bloom filter algorithm, the average accuracy of the eight mainstream machine learning algorithms in detecting anomalies on the selected data set is 98.6165%, among which the Extra Trees algorithm performs best; however, in terms of detection time, the performance of these algorithms did not meet the expected goal, which was higher than the upper limit of malicious attack prevention detection time of 38 microseconds.

[0150] As can be seen from the table, after the hierarchical Bloom filter algorithm initially filtered normal and abnormal traffic, the accuracy, precision, recall and F1 scores of the eight algorithms increased by 0.1%, 0.5%, 0.13% and 0.4% respectively. In particular, the detection time was significantly shortened from an average of 639.05 microseconds to 90.25 microseconds, which shows that the hierarchical Bloom filter greatly improves the speed of data processing.

[0151] Among them, the network intrusion detection algorithm of hierarchical bloom filtering and extreme random tree proposed in the present invention ensures more than 99% accuracy and recall rate, and its detection time is only 17.17 microseconds, which is less than the upper limit of 38 microseconds of prevention detection time, so that the effect of stopping the attack in advance before the malicious attack frame is processed by the node can be achieved. These results show that the method proposed in the present invention can effectively distinguish normal samples from attack samples in the test data set while ensuring time efficiency.

[0152] In addition, the present invention is also compared with five advanced CAN-IDS models in terms of detection time: GIDS, DCNN, LSTM-based IDS and MTH-IDS. GIDS uses GAN to implement unsupervised IDS, and DCNN uses supervised learning methods, mainly for the detection of known attack types. LSTM-based IDS also uses supervised learning as a baseline for evaluation. MTH-IDS combines supervised and unsupervised learning techniques. From the comparison of detection time in Table 3, it can be observed that the present invention has the best detection efficiency.

[0153] Table 3

[0154]

[0155]

[0156] Taking into account the accuracy, false alarm rate inference and detection time comparison, the extreme random tree model combined with hierarchical bloom filter (LBF+Extra Trees) performs best among all compared models. It not only achieves the highest performance in various performance indicators, but also leads by a large margin in processing speed, significantly improving the detection efficiency. Therefore, the network intrusion detection algorithm based on hierarchical bloom filter and extreme random tree was selected as the final solution. Its efficient detection performance and fast response time make it an ideal choice for network intrusion detection systems, and it can effectively deal with complex security threats, including zero-day attacks and unknown attacks. Furthermore, by introducing the large language model (LLM), this solution can automatically generate anomaly detection reports. These reports cover the description of the detected anomalies, potential threat analysis, and recommended countermeasures, which greatly improves the efficiency and response speed of security incident processing.

[0157] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A network intrusion detection method based on hierarchical Bloom filtering and large language model, characterized in that: A hierarchical Bloom filter is used to perform preliminary and rapid screening of input network communication data, train a network intrusion detection model, perform anomaly detection through the final network intrusion detection model obtained after training, and use a large language model to generate a detailed report of the detected anomalies.

2. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 1, characterized in that: include: Collect network communication data; Preprocess the collected network communication data, including data cleaning, category balancing, and feature dimension reduction; Use hierarchical Bloom filters to perform preliminary and rapid screening of preprocessed data to quickly determine whether the data may belong to a known set of normal communication patterns; Input the data filtered by the hierarchical Bloom filter into the network intrusion detection model, train the network intrusion detection model, and use the final network intrusion detection model obtained by training to identify and classify normal and abnormal communication data in the vehicle network; Generates detailed reports of detected anomalies using large language models. The reports include detailed descriptions of detected anomalies, potential threat analysis, and recommended countermeasures.

3. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 2, characterized in that: The specific implementation method of preprocessing the collected network communication data is as follows: Data cleaning: remove invalid data of empty records and undefined records, and handle missing values ​​and outliers; Feature selection: Use the Pearson correlation coefficient method to select features, calculate the correlation coefficient between each feature and the label, and select features with higher correlation coefficients; the Pearson correlation coefficient calculation formula is as follows: Among them, x i and i Represent the feature value and label value of the i-th sample respectively, and Represent the mean of features and labels respectively, and n represents the number of samples; Category balance: Use the synthetic minority oversampling technique SMOTE for category balance. For the minority class sample s, randomly select its k nearest neighbor samples, and then randomly interpolate between s and the nearest neighbor samples to generate new samples; the interpolation formula is: s new =s+rand(0,1)*(s nm -s) Where rand(0,1) represents a random number between 0 and 1. Repeat the class balancing process until the number of minority class samples is balanced with the number of majority class samples. Data standardization: Perform min-max standardization on the features and scale the feature values ​​to the interval [0,1]. The standardization formula is: Among them, x is the original eigenvalue, x′ is the standardized eigenvalue, min and max are the minimum and maximum values ​​of the feature respectively.

4. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 2, characterized in that: The specific implementation method of using hierarchical Bloom filters to perform preliminary and rapid screening of preprocessed data is as follows: Hierarchical Bloom filter initialization: According to the amount of data n and the expected false positive rate p, calculate the optimal bit array size m and number of hash functions k: Among them, m is rounded up and k is rounded down; Construct a hierarchical Bloom filter: For each sample of normal traffic data, use k hash functions to calculate its hash value and set the corresponding position in the bit array to 1; Filter data: For each sample of the detection data, use the same k hash functions to calculate its hash value, and check whether the bits in the corresponding bit array are all 1. If so, it is judged as normal traffic, otherwise it is judged as abnormal traffic.

5. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 2, characterized in that: The specific implementation method of training the network intrusion detection model is as follows: The samples judged as abnormal traffic by the hierarchical Bloom filter are divided into training set and test set; Perform K-fold cross validation on the training set, adjust the network intrusion detection model hyperparameters, and use methods including grid search to find the optimal hyperparameter combination; Retrain the network intrusion detection model on the entire training set using the optimal hyperparameters; The performance of the network intrusion detection model is evaluated on the test set, and the final network intrusion detection model is selected based on the performance indicators.

6. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 5, characterized in that: The network intrusion detection model uses XGBoost, Random Forest, Decision Tree, Extra Trees, Support Vector Machine, Naive Bayes, AdaBoost and MLP models.

7. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 5, characterized in that: The performance indicators for evaluating the performance of network intrusion detection models include: Accuracy: Accuracy: Recall: F1 score: Among them, TP, TN, FP, and FN represent the number of true positive, true negative, false positive, and false negative samples, respectively.

8. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 2, characterized in that: The specific implementation method of identifying and classifying normal and abnormal communication data in the vehicle network through the final network intrusion detection model is as follows: Build an online detection system, that is, cascade the hierarchical Bloom filter and the final network intrusion detection model to perform network intrusion detection on real-time network traffic; the hierarchical Bloom filter initially filters out normal traffic, and the final network intrusion detection model further identifies samples that the Bloom filter determines to be abnormal traffic and outputs the final detection results; Collect detection data during the operation of the online detection system, regularly update the hierarchical Bloom filter and the final network intrusion detection model, perform incremental learning of the model, and introduce new data sets to prevent concept drift.

9. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 8, characterized in that: The final detection results output by the final network intrusion detection model are input into the large language model, and the large language model generates a detailed report of the detected anomalies. The report content includes a detailed description of the detected anomalies, potential threat analysis, and recommended countermeasures.

10. A network intrusion detection method based on hierarchical Bloom filtering and large language model according to claim 1, characterized in that: This method is applied to intelligent transportation system scenarios and smart vehicle sharing scenarios.

Images