Network intrusion detection method based on pre-training language model federal segmentation learning

Abstract

The invention provides a network intrusion detection method based on pre-training language model federal segmentation learning, aiming at dual requirements of privacy protection and attack recognition in a network intrusion detection system. For the challenges of difficult traffic feature extraction, limited edge device computing power, data privacy barriers among distributed nodes and the like in attack detection, the method fuses the privacy protection mechanism of federated learning and the computing unloading advantages of segmentation learning, and constructs a pre-training language model learning architecture oriented to traffic feature analysis. According to the method, a pre-training language model is decoupled into an edge side lightweight module and a cloud deep reasoning module through a hierarchical segmentation strategy, and balance optimization of global model stability and local personalization is realized through federal aggregation based on a multi-head attention mechanism. An adversarial enhancement mechanism is introduced to the edge side, the feature extraction ability of a pre-training language model is combined with the dynamic game of a generative adversarial network, and the generalization ability of the model to complex network attacks is improved through adversarial sample reconstruction.

Description

Technical Field

[0001] The present invention is an application technology of artificial intelligence technology in network security, specifically a network intrusion detection method based on federated segmentation learning of a pre-trained language model. Background Art

[0002] As a breakthrough in the field of artificial intelligence, large language models[1] have achieved a leap forward in deep semantic parsing and generation capabilities through self-supervised pre-training mechanisms. This type of model can not only autonomously construct a distributed representation system of language symbols from a massive corpus, but also demonstrate multimodal transfer learning capabilities. Its feature fusion architecture based on the attention mechanism can effectively capture potential cross-domain correlation patterns. While continuously refreshing performance benchmarks in natural language processing tasks such as machine translation and text summarization, it also shows great potential for application in other fields[2]. For example, in the field of network intrusion detection, by learning and analyzing large amounts of historical data, language models can identify complex patterns and trends, predict potential system vulnerabilities, and detect abnormal behavior at an early stage[3].

[0003] In the context of increasing privacy protection, most sensitive data is isolated and cannot be directly shared. Traditional centralized training models are difficult to build high-quality language models due to data limitations, especially when the training samples are insufficient, the model performance is significantly limited [4]. In order to make full use of a large amount of distributed and diverse private data [5], Federated Learning (FL) proposes a privacy-safe collaboration paradigm that can train global models across multiple datasets without sharing the original data [6,7]. Under this framework, edge computing nodes periodically train models based on local private data, and only upload model parameter gradients to cloud servers for secure aggregation, completing the iterative update of the global model under the premise of strictly ensuring that the original data does not leave the domain. It is worth noting that although FL alleviates the conflict between data privacy and value release, the significant contradiction between the limited computing resources of edge nodes (such as CPU processing power, memory capacity, etc.) and the demand for large model training still constitutes a key bottleneck for actual deployment [8].

[0004] The challenges / problems and related work in the existing technology mainly include:

[0005] 1) Take into account both global individuality and local individuality.

[0006] Due to privacy issues, most private data is stored on edge devices. In order to solve the data silos formed by sensitive data stored locally on edge nodes, FL implements global model training through cloud server coordination. After edge nodes train models based on local data sets, they only need to upload parameter gradients to participate in federated global aggregation, ensuring that the original data always resides locally. However, data from different edge nodes usually differ significantly. How to effectively maintain the stability and generalization ability of the global model while achieving personalized modeling through local fine-tuning, thereby building a collaborative optimization framework of "global consensus-local adaptation", remains an important issue. [9] Consider introducing a meta-network with batch norm and learning rate parameters for each client given local statistics, and learn these meta-networks through FL, allowing the entire FL network to collaboratively learn personalized strategies customized for each client.

[10] In the FL scenario, Kullback–Leibler divergence estimation and adaptive weight calculation are used to improve the accuracy of the model and the robustness to adversarial attacks. A convolutional neural network based on adaptive maximum pooling and an improved bidirectional long short-term memory network with attention and residual connections are implemented, which can effectively detect intrusions, thereby enhancing the security and privacy of the network.

[0007] 2) Dual optimization of detection accuracy and training cost.

[0008] In the deployment of edge systems, the optimization of model accuracy and training overhead constitutes a core challenge. Limited by the computing bottleneck of edge nodes (such as CPU-GPU computing power imbalance, memory bandwidth constraints, etc.), lightweight computing units are difficult to carry the tensor parallel computing requirements required for large model training. In addition, the periodic full-model synchronization mechanism in the federated architecture has negative effects: on the one hand, the cross-layer transmission of massive parameters aggravates the communication bandwidth pressure between edge nodes and cloud servers; on the other hand, the full-model gradient exchange may expose the model structure characteristics and break the privacy and security boundaries of federated learning [11,12]. In order to overcome the challenge of limited device computing resources,

[13] believes that deep neural networks can perform split learning (SL).

[14] split the BERT model into three main layers: Embedding layer, Transformer layer and Head layer, decoupling the potential dependencies. Based on the complexity analysis and computational analysis of deep neural networks, the Transformer layer that requires high computing resources is allocated to the cloud server, and the Embedding layer and Head layer with limited resources are deployed on the client. Each client can privately adjust a specific NLP task and incrementally merge datasets from multiple devices through federated learning and split training to improve the robustness and generalization of pre-trained models. In order to reduce the computational burden of resource-constrained edge nodes and better protect model privacy,

[15] proposed a novel split federated learning (SFL) framework MergeSFL, which combines feature merging and batch size adjustment in SFL and is used to improve model accuracy and training efficiency.

[0009] 3) Integrate feature extraction and traffic generation process.

[0010] Pre-trained language models based on the Transformer architecture have shown innovative value in the field of network security situation awareness. The multi-granularity semantic representation space constructed by its bidirectional self-attention mechanism can realize context-aware modeling of network traffic metadata. Pre-trained language models represented by BERT

[16] can simultaneously consider the context information of the data through hierarchical feature fusion, thereby improving the ability to recognize complex network traffic patterns. Given BERT's powerful feature extraction and generalization capabilities, researchers have tried to apply BERT to vulnerability exploitability prediction and system log anomaly detection.

[17] proposed ExBERT, which uses the collected vulnerability description corpus to fine-tune the pre-trained BERT model, extract the semantic information of the vulnerability description, and thus predict network security vulnerabilities.

[18] is a log anomaly detection model based on BERT that can detect log parsing idleness. BERT can effectively extract semantic information from serialized log data, capture detailed features, and improve the accuracy of log anomaly detection. It is worth further exploring to effectively integrate pre-trained language models with existing deep learning models or frameworks to improve network intrusion detection performance. Summary of the invention

[0011] In response to the dual needs of privacy protection and attack identification in network intrusion detection systems, the present invention proposes a network intrusion detection method based on federated segmentation learning of a pre-trained language model. In order to address the challenges faced in attack detection, such as the difficulty in extracting traffic features, the limited computing power of edge devices, and the data privacy barriers between distributed nodes, the present invention integrates the privacy protection mechanism of federated learning with the computational offloading advantages of segmentation learning to construct a pre-trained language model learning architecture for traffic feature analysis. The present invention is specifically as follows.

[0012] A network intrusion detection method based on federated segmentation learning of a pre-trained language model. In an edge computing system with a cloud server and N edge nodes, the edge nodes use edge models to identify and classify local network traffic, and the cloud server is responsible for segmentation model training and federated parameter aggregation.

[0013] The edge model includes a discriminator D, a generator G, and a classification network C, where the discriminator D and the generator G are the discriminator D and the generator G of the conditional generative adversarial network CGAN; the pre-trained language model BERT is embedded in the discriminator D for feature extraction of local network traffic, and the classification network C is used to classify network traffic based on the extracted features;

[0014] The edge model is divided into the discriminator D, the generator G and the classification network C, where the BERT model is further divided into the Embedding layer, the Transformer layer and the Head layer;

[0015] The Transformer layer and the Head layer constitute the top-level model; the generator G, the discriminator D, and the Embedding layer constitute the bottom-level model; each edge node uses its local data to train the bottom-level model, and the cloud server maintains the top-level model;

[0016] In federated segmentation learning: each edge node uses its local data to train the edge model and sends the updated parameters to the cloud server; the cloud server trains the segmentation model and aggregates the segmentation model updates of each edge node through a multi-head attention mechanism to form the global model parameters and distribute them back to the edge nodes; each edge node then makes further fine-tuning based on the local data, retaining the basic structure of the global model but making personalized adjustments on the local data.

[0017] The present invention decouples the pre-trained language model into a lightweight module on the edge (bottom model) and a deep inference module on the cloud (top model) through a hierarchical segmentation strategy, and achieves a balanced optimization of global model stability and local personalization through federated aggregation based on a multi-head attention mechanism. The edge side introduces an adversarial enhancement mechanism, combining the feature extraction capability of the pre-trained language model with the dynamic game of the generative adversarial network, and improves the model's generalization ability against complex network attacks through adversarial sample reconstruction.

[0018] Experiments on the CSE-CIC-IDS2018 dataset show that compared with the comparison method, the weighted F1-score of this method is improved by 1.248% to 14.95%, reaching 97.875%. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A federated segmentation learning framework for cloud-edge collaboration;

[0020] Figure 2 The discriminator model architecture representing the edge nodes;

[0021] Figure 3 Represents the layered architecture of BERT;

[0022] Figure 4 Represents the iterative process of generative adversarial segmentation of the edge model;

[0023] Figure 5 Indicates the precision of different methods;

[0024] Figure 6 Recall representing different methods;

[0025] Figure 7 Indicates the F1-score of different methods;

[0026] Figure 8 Represents the change in detection performance with and without segmentation of the model. DETAILED DESCRIPTION

[0027] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0028] 1 Overview

[0029] This paper proposes a new federated segmentation learning framework based on the pre-trained language model. This framework integrates the distributed collaboration advantages of federated learning and the computational offloading characteristics of segmentation learning to achieve safe and efficient deployment of large language models. The main contributions include:

[0030] A federated segmentation training mechanism based on pre-trained language models in edge environments is proposed, which takes into account both model personalization and training cost optimization. The model is decoupled and layered, and the edge nodes and cloud servers stably update global parameters during training and communication. Each edge node can perform personalized fine-tuning based on local data.

[0031] Construct an adversarial learning edge model enhanced by a pre-trained language model to optimize instance feature extraction. BERT can enhance the feature extraction capability of network traffic and forms a complementary relationship with the generative adversarial model. The powerful feature extraction capability of BERT is deeply integrated with the generative adversarial network, so that the generator generates adversarial samples through traffic feature reconstruction, forms a dynamic game with the discriminator, and effectively improves the model's learning ability for network traffic.

[0032] Experimental results show that the weighted accuracy, precision, recall and F1-score of the proposed model on the CSE-CIC-IDS2018 dataset are better than those of the comparative detection framework. Compared with other methods, the F1-score of the proposed model is improved by 1.248% to 14.95%.

[0033] The rest of the paper is organized as follows.

[0034] Section 2 introduces the federated segmentation learning architecture for edge models.

[0035] Section 3 introduces the model’s training strategy and cost analysis.

[0036] Section IV introduces the dataset selection and experimental methods.

[0037] Section 5 presents the experimental results and performance evaluation.

[0038] Section 6 concludes the paper.

[0039] 2 Cloud-edge collaborative federated segmentation learning architecture

[0040] like Figure 1As shown in Figure 1, consider an edge computing system with a cloud server and N edge nodes. The Intrusion Detection System (IDS) is deployed at the edge of the network, and the cloud platform is responsible for segmentation model training and federated parameter aggregation. Federated segmentation learning performs deep learning tasks through a set of nodes coordinated by a cloud server.

[0041] In this architecture, each edge node trains a model based on local data and sends edge updates to the cloud server. The cloud server trains the segmentation model and aggregates the segmentation model updates of each node through a multi-head attention mechanism to form global model parameters and distribute them back to the edge nodes. Each edge node then makes further fine-tuning based on local data, retaining the basic structure of the global model but performing personalized adjustments on local data. In a data heterogeneous environment, efficient personalization is achieved.

[0042] 2.1 Edge Model Enhanced by Pre-trained Language Model

[0043] The pre-trained language model in this embodiment adopts BERT, which inherits the self-attention-based architecture of Transformer and can learn more information about global input and output dependencies.

[0044] like Figure 1 As shown in the figure, in the edge nodes, D and G represent the discriminator and generator of the Conditional Generative Adversarial Networks (CGAN)

[19] respectively, and C represents the classification network layer of the model. The BERT model is embedded in the discriminator D, and C enhanced by the pre-trained language model is constructed to solve the classification problem of network traffic.

[0045] like Figure 2 As shown in Figure 1, the first token in the BERT input sequence is a special classification token ([CLS]), and its final hidden state is used as the aggregate sequence representation for the classification task. BERT is initialized using the parameters of the pre-trained language model, and then fine-tuned using the pre-processed network intrusion dataset.

[0046] In the edge model, BERT is not only used to enhance the feature extraction capability of network traffic. More importantly, BERT and G form a complementary relationship. Specifically, in order to learn the appropriate probability distribution traffic features, G generates a high-dimensional feature representation of network traffic that changes with the fine-tuning of BERT parameters. G continuously learns the high-dimensional feature representation of network traffic, and D also continuously updates its parameters.

[0047] 2.2 Segmentation and Transfer of Pre-trained Language Model

[0048] The basic idea of ​​federated segmentation learning is to divide the model into two sub-models at the segmentation layer, namely the bottom model and the top model, denoted as w b 、w p Split learning (SL) enables edge nodes with low computational overhead to train language models in sufficient training time. To train efficiently on data, BERT should be split in a way that preserves the underlying dependencies between modules.

[0049] like Figure 3 As shown, the BERT model is divided into three main layers: Embedding, Transformer, and Head, which can decouple potential dependencies. Based on the complexity analysis and computational analysis of deep neural networks, Transformer requires high computing resources, which can be allocated on powerful cloud servers. After training the model, each edge node can adjust specific tasks. In addition, by using federated learning, split training can incrementally merge data sets from multiple edge nodes to improve the robustness and generalization of pre-trained models.

[0050] In this framework, the edge model enhanced by the pre-trained language model is divided into the Embedding layer, Transformer layer, and Head layer of G, C, and BERT. Considering the communication cost and computing time, the top model contains the Transformer layer and the Head layer, which are represented as The underlying model contains the Embedding layers of G, C, and BERT, which are represented as In federated segmentation learning, server maintenance Each edge node uses its local data to train In addition, the nodes complete the entire training process by interacting with the top-level model on the cloud server.

[0051] 3. Training strategies that take into account both global and local personalization

[0052] 3.1 Generative Adversarial Segmentation Iterations of Edge Model

[0053] In this architecture, the cloud server maintains the top-level model Each edge node uses its local data to train the underlying model The iterative goal of the segmentation model is to find the optimal model w = [w b ,w p ], so as to minimize the loss function L(w). Let w p,i 、w b,i They represent the top model and bottom model corresponding to edge node i, respectively, L(wp,i ) indicates w p,i The loss function, L(w b,i ) indicates w b,i The loss function of .

[0054]

[0055] like Figure 4 As shown in the figure, the basic training process mainly includes the forward / backward propagation of the bottom model and the forward / backward propagation of the top model. First, each edge node uses a batch of local data samples for forward propagation and passes the features of the split layer to the cloud server, which then performs forward / backward propagation to update the top model. Then, the cloud server sends the back-propagated gradient back to the edge node and updates the bottom model through back-propagation. Such a complete forward / backward propagation process can be regarded as a local iteration.

[0056] The real traffic data x is used as sentences processed in the BERT model. Special characters such as [CLS] and [SEP] are added before and after each traffic sequence so that Recognize input. In the cloud server The attention scores between traffic sequences are calculated through multi-head attention, and each attention head performs a self-attention function by inputting the embedding sequence of x to obtain a new vector. The vector of hidden state [CLS] positions is used as the semantic representation of the traffic sequence. Process h and g, where h is the flow data through w p The probability distribution of the encoding mapped to a high-dimensional space, g is A high-dimensional feature representation generated from the class label y and the random noise vector z. The output probability represents the traffic type. During the training process, Trying to generate high-dimensional feature representations of network traffic to confuse classifiers, Try to correctly distinguish which type of traffic data.

[0057] set up represents the underlying model of edge node i at the kth iteration, S i represents a small batch of data samples, It means that given x and After one iteration, the underlying model is updated to:

[0058]

[0059] set up represents the top model at the kth iteration, For a given Output and The stochastic gradient of . Then the top model after one iteration update is:

[0060]

[0061] During the iteration process, D is expressed as w D (x) represents the category of x. ) indicates distinction The generated samples. In the marginal model, and w D The optimization objective function is expressed as

[0062]

[0063] The objective function is expressed as, Try to generate something that can deceive w D The flow rate, w D Trying to distinguish between real traffic and generated traffic. In the adversarial optimization process, BERT promotes w D The classification ability is continuously improved. Continuously improve your generation capabilities.

[0064] Algorithm 1 Generative Adversarial Segmentation of Edge Model Iteration Requirements: Cloud Server S; Edge Node Set I; Edge Node Training Round E

[0065] 01: Initialization

[0066] ServerTrmFordprop

[0067] 02: ←Calculation The forward propagation

[0068] 03: h←calculation The forward propagation

[0069] 04: Send h

[0070] ServerTrmBackprop

[0071] 05: ←Calculate according to formula (3) Back propagation

[0072] 06: ←Calculate according to formula (3) Back propagation

[0073] 07: Send

[0074] Edge node i:

[0075] EdgeUpdate():

[0076] 08:for each local epoch e=1...E do09: ←Calculation The forward propagation

[0077] 10: h←ServerTrmFordprop

[0078] 11: g←

[0079] 12: ←

[0080] 13: ←Calculated according to formula (2) Back propagation

[0081] 14: ←

[0082] 15: ←Calculated according to formula (2) Back propagation

[0083] 16: ←Calculated according to formula (2) Back propagation

[0084] 17: end for

[0085] 3.2 Parameter Aggregation Based on Multi-Head Attention Mechanism

[0086] The cloud server maintains independent Transformer layer and Head layer for edge nodes, which are represented as In the cloud, a multi-head attention mechanism is used to aggregate the information from edge nodes. And the aggregated Send back to the edge node.

[0087] After multiple layers of stacking, Q, K, and V are generated, where Q, K, and V are query matrix, key matrix, and value matrix respectively. and are input into the multi-head attention mechanism as Q and K respectively. For each attention head, a self-attention function is executed to obtain a new vector. Through the convergence of multiple attention heads, the edge nodes that contribute more to the global model update are focused on, thereby achieving more accurate model aggregation. The attention weights of the aggregated model parameters are expressed as [γ1,γ2,...,γ i ],

[0088]

[0089] After completing several rounds of local iterative updates, Updated to

[0090]

[0091] Algorithm 2 Parameter aggregation based on multi-head attention mechanism

[0092] Requirements: Cloud server S; edge node set I

[0093] Cloud Server:

[0094]

[0095]

[0096] 3.3 Cost Analysis of Joint Training

[0097] This section analyzes the communication cost and computational time of federated segmentation learning. The communication cost specifically refers to the amount of data exchanged between the cloud server and the edge node i in federated segmentation learning. Given I edge nodes, p is the number of training batches under a global epoch, |w| is the number of parameters of the complete model, and L is the number of parameters propagated between segmentation layers. In the traditional federated learning FL mode, all |w| needs to be uploaded to the cloud server. FL involves bidirectional transmission of the model, so the data volume is 2|w|. In federated segmentation learning, only the parameters between segmentation layers need to be transmitted. Since the segmentation model contains forward and backward propagation, the data volume is 4pL. Therefore, when 4pL<2|w|, the communication cost of this method is less than that of traditional FL.

[0098] The computation time reflects the time cost of model training. is the computation time of a global epoch, is the aggregation time of w. For segmentation learning, the computation time of the top model is expressed as The computation time of the underlying model is expressed as The original computational cost of the edge model can be expressed as because Greater than and therefore The edge model calculation time of the method of the present invention is shorter than that of the traditional FL.

[0099] Table 1 Cost analysis of joint training

[0100]

[0101] Figure 1 to Figure 4 Translation of key terms in:

[0102] Multi-head self-attention, Scaled Dot-Product Attention, Split Learning, Gradient dispatching, Forward propagation, Augmented Traffic, Original Traffic, Fully connected layers, Traffic Feature, Traffic data sequence.

[0103] 4 Experimental design

[0104] 4.1 Dataset

[0105] Table 2. Distribution of the number of datasets

[0106]

[0107] 4.2 Environment Configuration

[0108] All models in the experiment are run in Python 3.5, Tensorflow 2.12.0 environment. The computer configuration is Intel Core i9-13900K CPU, RTX 4090 24G and 128GB RAM.

[0109] 4.3 Comparison Method

[0110] In order to conduct a comprehensive comparison and verification, this paper selects and designs a variety of experimental methods. In order to evaluate the impact of different strategies on the overall performance, the proposed schemes are divided into four categories as shown in Table 3. For a comprehensive and objective comparison, the two distributed learning methods listed in Table 4 are selected as benchmark methods (the benchmark methods have no model segmentation).

[0111] Table 3 Classification of the proposed methods

[0112]

[0113] Table 4 Classification of benchmark methods

[0114]

[0115] 4.4 Evaluation Metrics

[0116] Accuracy: Accuracy represents the ratio of the number of correctly predicted network flows to the total number of classification predictions, which measures the ability of the model to correctly predict normal traffic and attack traffic.

[0117]

[0118] Precision: Precision indicates the proportion of correctly classified attacks in a set of attack predictions, and measures the model's ability to accurately detect attack traffic.

[0119]

[0120] Recall: Recall indicates the proportion of correctly classified samples in a given set of attack samples, and measures the model's ability to detect attack traffic. This indicator represents the model's attack detection capability.

[0121]

[0122] F1-Score: F1-Score is defined as the harmonic mean of Precision and Recall, which comprehensively reflects the effect of the model.

[0123]

[0124] TP=True Positives, TN=True Negatives, FP=False Positives, FN=False Negatives.

[0125] 5 Results Analysis

[0126] This section first compares the weighted average classification performance of different methods on the test set, and then analyzes the Precision, Recall, and F1-score of the model to detect specific types of network traffic to evaluate the detection performance of the proposed method for different attack traffic.

[0127] 5.1 Detection Accuracy

[0128] Table 5 shows the Accuracy, Precision, Recall and F1-score of different methods on the CSE-CIC-IDS2018 test set. Proposed-1 outperforms other methods in all evaluation indicators, followed by Proposed-2 and Proposed-4, while Baseline-2 is the lowest. The Accuracy, Precision and F1-score of Proposed-1 reach 97.806%, 98.037% and 97.875% respectively, which are the highest among all methods. Compared with Proposed-2 and Proposed-4, the Accuracy of Proposed-1 is improved by 1.466% and 1.295% respectively, the Precision is improved by 0.701% and 1.186% respectively, and the F1-score is improved by 1.254% and 1.248% respectively. Compared with Baseline-1, the four performance indicators of Proposed-1 are improved by 2.274%, 1.442%, 2.274% and 2.065% respectively. Proposed-1 adopts federated segmentation learning. Each edge node retains a copy of the corresponding Transformer layer on the cloud server, and the edge nodes are trained in parallel. All Transformer layers perform parameter aggregation based on the multi-head attention mechanism on the cloud server. Compared with the Proposed-2 and Proposed-4 models, Proposed-1 integrates the CGAN framework as an edge model and uses a multi-head attention mechanism for federated segmentation aggregation, which can improve Precision, Recall and F1-score. Proposed-1 has the feature extraction capability of BERT and the generalization capability of CGAN, and combines the multi-head attention mechanism for parameter aggregation. These advantages further reduce misclassification and underreporting, and improve F1-score.

[0129] At the same time, the four performance indicators of Proposed-1 are improved by 12.063% to 14.95% over Baseline-2. This is because the self-attention mechanism of the BERT model allows the establishment of a dependency model regardless of the distance of the features in the input or output sequence. Even for the farthest network traffic statistical features in the sequence, BERT can capture the intrinsic connection between network statistical features through the self-attention mechanism. The classifier integrating BERT can obtain more information about the attack category from the captured high-dimensional features, thereby improving the classification accuracy. Compared with BERT, LSTM has a weaker ability to focus on the global dependencies between network statistical features. When faced with a small proportion of attack categories, they will not be able to correctly distinguish the types of attacks.

[0130] Table 5 Weighted average performance of different methods

[0131]

[0132] Figure 5 , Figure 6 and Figure 7 The precision, recall and F1-score of different methods for identifying normal traffic and possible attack types are given respectively. Each network traffic corresponds to multiple statistical features, which cover the characteristic information of network traffic. Each statistical feature corresponds to a word with a specific meaning. The intrinsic relationship between statistical features is equivalent to the contextual relationship in a sentence. Compared with other methods, Proposed-1 has the highest detection accuracy on almost all network flow types. Figure 5 As shown in the figure, for the detection evaluation index Precision of the more concealed attack type Infilteration, Proposed-1 improves by 15.624% and 38.196% compared with Baseline-1 and Baseline-2, respectively, and Proposed-1 also improves by 15.155% and 7.622% compared with Proposed-2 and Proposed-4, reaching 76.994%. This shows that Proposed-1 can significantly improve the problem that some attack types are difficult to identify.

[0133] like Figure 6 As shown, compared with Baseline-2, Proposed-1 has significantly improved Recall in each traffic type. It is worth noting that for most network attack categories, the Recall of each method using BERT as a feature extractor is higher than Precision, but for benign categories, the Recall is lower than Precision. This is due to the emphasis on detecting attack categories. BERT may classify benign categories as attack categories in some cases. Therefore, the Recall of the benign category with the largest proportion is lower than the Recall of some attack categories with a smaller proportion. In terms of the performance indicator Recall for detecting Benigh and Infilteration, Proposed-1 has improved on the basis of the detection results of Proposed-4. Compared with Proposed-2 and Baseline-1, Proposed-1 has a decrease of 2.054% and 2.308% in detecting Infilteration, respectively, and an increase of 3.094% and 4.489 in detecting Benigh, respectively. This shows that Proposed-1 improves the situation of classifying benign categories as attack categories and can reduce the false alarm rate.

[0134] Figure 7The F1-scores of different methods for identifying normal traffic and possible attack types are shown. Proposed-1 outperforms other methods in the comprehensive indicator F1-score of almost all network attack categories. For some highly concealed attack types (such as Infilteration), Proposed-2 and Proposed-4 still have the problems of high false alarm probability and unclear detection effect improvement. Compared with Proposed-2 and Proposed-4, the F1-score of Proposed-1 for detecting Infilteration attack types increased by 9.229% and 7.403% respectively, reaching 83.899%. In Proposed-1, BERT and CGAN promote each other's learning, and at the same time integrate the multi-head attention mechanism for parameter aggregation of federated segmentation, thereby improving the model detection performance, especially for some attack types that are difficult to identify or account for a small proportion.

[0135] 5.2 Impact of Segmentation

[0136] This group of experiments selected Proposed-1 based on language model segmentation and Proposed-3 without language model segmentation to investigate the impact of the proposed model segmentation on the detection accuracy in collaborative training. Figure 8 As shown in the figure, as the number of training rounds increases, the trends in detection performance of the segmented model and the non-segmented model are almost the same.

[0137] 6 Conclusion

[0138] In order to solve the problems of performance optimization and resource constraints in edge network intrusion detection scenarios, this paper proposes a federated segmentation learning framework based on the collaborative optimization of pre-trained language models and conditional generative adversarial networks. The high-order feature extraction layer of the pre-trained language model is deployed in the cloud, while the lightweight adversarial enhancement module on the edge side is retained to achieve dual optimization of detection accuracy and resource efficiency. A hierarchical federated aggregation mechanism is constructed, and the local traffic features of edge nodes are fused using a multi-head self-attention mechanism to achieve global model aggregation. An edge model enhanced by generative adversarial learning is developed to improve the model detection performance through adversarial sample reconstruction and dynamic game between the discriminator.

[0139] References

[0140] [1]Ling C, Zhao

[0141] [2]Yao Y,Duan J,Xu K,et al.A survey on large language model(llm)security and privacy:Thegood,the bad,and the ugly[J].High-ConfidenceComputing,2024:100211.

[0142] [3]G.Lira O,Marroquin A,To M A.Harnessing the advanced capabilitiesof llm for adaptiveintrusion detection systems[C] / / International Conferenceon Advanced Information Networkingand Applications.Cham:Springer NatureSwitzerland,2024:453-464.

[0143] [4]Jin Z,Zhou J,Li B,et al.FL-IIDS:A novel federated learning-basedincremental intrusiondetection system[J].Future Generation Computer Systems,2024,151:57-70.

[0144] [5]Bhavsar M,Bekele Y,Roy K,et al.FL-IDS:Federated Learning-BasedIntrusion DetectionSystem Using Edge Devices for Transportation IoT[J].IEEEAccess,2024.

[0145] [6]Qi P,Chiaro D,Guzzo A,et al.Model aggregation techniques infederated learning:Acomprehensive survey[J].Future Generation ComputerSystems,2024,150:272-293.

[0146] [7]Lyu L,Yu H,Ma X,et al.Privacy and robustness in federatedlearning:Attacks and defenses[J].IEEE transactions on neural networks andlearning systems,2022.

[0147] [8]Park J,Samarakoon S,Elgabli A,et al.Communication-efficient anddistributed learning overwireless networks:Principles and applications[J].Proceedings of the IEEE,2021,109(5):796-819.[9]Lee R,Kim M,Li D,etal.Fedl2p:Federated learning to personalize[J].Advances in NeuralInformationProcessing Systems,2024,36.

[0148]

[10] Begum K,Mozumder M A I,Joo M I,et al.BFLIDS:Blockchain-drivenfederated learning forintrusion detection in IoMT networks[J].Sensors,2024,24(14):4591.

[0149]

[11] Thapa C,Arachchige P C M,Camtepe S,et al.Splitfed:When federatedlearning meets splitlearning[C] / / Proceedings of the AAAI Conference onArtificial Intelligence.2022,36(8):8485-8493.

[0150]

[12] Han D J,Bhatti H I,Lee J,et al.Accelerating federated learningwith split learning on locallygenerated losses[C] / / ICML 2021 workshop onfederated learning for user privacy and dataconfidentiality.ICML Board.2021.

[0151]

[13] Vepakomma P,Gupta O,Swedish T,et al.Split learning for health:Distributed deep learningwithout sharing raw patient data[J].arXiv preprintarXiv:1812.00564,2018.

[0152]

[14] Tian Y,Wan Y,Lyu L,et al.FedBERT:When federated learning meetspre-training[J].ACMTransactions on Intelligent Systems and Technology(TIST),2022,13(4):1-26.

[0153]

[15] Liao Y,Xu Y,Xu H,et al.Mergesfl:Split federated learning withfeature merging and batchsize regulation[C] / / 2024 IEEE 40th InternationalConference on Data Engineering(ICDE).IEEE,2024:2054-2067.

[0154]

[16] Devlin J,Chang M W,Lee K,et al.BERT:Pre-training of DeepBidirectional Transformers forLanguage Understanding.arXiv 1810.04805[preprint]https: / / doi.org / 10.48550 / arXiv.1810.04805.Posted October 11,2018[J].Accessed May,2024.

[0155]

[17] Yin J,Tang M J,Cao J,et al.Apply transfer learning tocybersecurity:Predicting exploitabilityof vulnerabilities by description[J].Knowledge-Based Systems,2020,210:106529.

[0156]

[18] Lee Y,Kim J,Kang P.Lanobert:System log anomaly detection based onbert masked languagemodel[J].Applied Soft Computing,2023,146:110689.

[0157]

[19] Mirza M.Conditional generative adversarial nets[J].arXiv preprintarXiv:1411.1784,2014。

Claims

1. A network intrusion detection method based on federated segmentation learning of a pre-trained language model, characterized by In an edge computing system with a cloud server and N edge nodes, the edge nodes use edge models to identify and classify local network traffic, and the cloud server is responsible for segmentation model training and federated parameter aggregation; The edge model is an architecture that integrates BERT and the conditional generative adversarial network CGAN, including a generator G and a discriminator D composed of BERT and a classification network C. The pre-trained language model BERT embedded in the discriminator D is used to extract features of local network traffic, and the classification network C performs detection and classification based on the extracted traffic features. The edge model is divided into BERT feature extraction layer, classification network C and generator G, where the BERT model is further divided into Embedding layer, Transformer layer and Head layer; The Transformer layer and the Head layer constitute the top-level model, which are represented as and The top model is expressed as The generator G, classification network C and Embedding layer constitute the underlying model, which are represented as and The underlying model is expressed as The marginal model is represented as w = [w b ,w p ]; Each edge node uses its local data to train and Cloud server maintenance and In federated segmentation learning: each edge node uses its local data to train the edge model and sends the updated parameters to the cloud server; the cloud server trains the segmentation model and aggregates the segmentation model updates of each edge node through a multi-head attention mechanism to form the global model parameters and distribute them back to the edge nodes; each edge node then makes further fine-tuning based on the local data, retaining the basic structure of the global model but making personalized adjustments on the local data.

2. The network intrusion detection method based on pre-trained language model federated segmentation learning according to claim 1 is characterized in that the segmentation The iterative goal of the model is to find the optimal model w = [w b ,w p ], to minimize the global model loss function L(w); Assume w p,i and w b,i They represent the top model and bottom model corresponding to edge node i, respectively, L(w p,i ) indicates w p,i The loss function, L(w b,i ) indicates w b,i The loss function is 3. The network intrusion detection method based on pre-trained language model federated segmentation learning according to claim 1 is characterized in that the edge The basic training process of the model includes: first, each edge node performs forward propagation with a batch of local data samples to pass the features of the underlying model to the cloud server; the cloud server performs forward / backward propagation to update the top model; then, the cloud server sends the back-propagated gradient back to the edge node to update the underlying model through back-propagation; such a complete forward / backward propagation process is regarded as a local iteration; In the model training of edge node i: The real traffic data x is used as the sentence processed in the BERT model, and special characters [CLS] and [SEP] are added before and after each traffic data sequence; In the cloud server: The attention scores between the traffic data sequences are calculated by multi-head attention, where each attention head performs a self-attention function by inputting the embedding sequence of x to obtain a new vector; The vector of hidden state [CLS] positions is used as the semantic representation of the traffic sequence; In the edge node: Process h and g, where h represents the flow data through w p The probability distribution of the encoding mapping to the high-dimensional space, g represents High-dimensional feature representation generated based on class label y and random noise vector z; The output probability represents the traffic type; During the training process, Trying to generate high-dimensional feature representations of network traffic to confuse classifiers, Try to correctly distinguish which type of traffic data; set up represents the underlying model of edge node i at the kth iteration, S i represents a small batch of data samples, It means that given x and stochastic gradient; after one iteration, the underlying model is updated to: set up represents the top model at the kth iteration, For a given Output and The stochastic gradient of ; then the top model after one iteration update is: In the iterative process, the discriminator D is expressed as w D (x) represents the category of x. Indicates distinction Generated samples; In the edge model, and w D The optimization objective function is expressed as The meaning of the objective function is, Try to generate something that can deceive w D The flow rate, w D Try to differentiate between real and generated traffic.

4. The network intrusion detection method based on pre-trained language model federated segmentation learning according to claim 1 is characterized by: The parameter aggregation based on the multi-head attention mechanism is: The cloud server maintains independent Transformer layer and Head layer for edge node i, which are represented as In the cloud server, the multi-head attention mechanism is used to aggregate the parameters, and aggregate them The parameters are sent back to the edge node; After multiple layers of stacking, Q, K, and V are generated, where Q, K, and V are query matrix, key matrix, and value matrix respectively; in the aggregation process, and They are input into the multi-head attention mechanism as Q and K respectively; for each attention head, the self-attention function is executed to obtain a new vector. Through the convergence of multiple attention heads, the edge nodes that contribute greatly to the global model update are focused on, thereby achieving model parameter aggregation; The attention weights of the aggregate model parameters are denoted as [γ1,γ2,...,γ i ], After completing multiple rounds of local iterative updates, Updated to

Images