Network intrusion detection method based on pre-training language model federated segmentation learning

Abstract

In view of the dual demands of privacy protection and attack identification in the network intrusion detection system, the application proposes a network intrusion detection method based on pre-training language model federated segmentation learning. In view of the difficulties in extracting traffic features, the limited computing power of edge devices, and the data privacy barriers between distributed nodes in attack detection, the method combines the privacy protection mechanism of federated learning and the computing offloading advantage of segmentation learning to construct a pre-training language model learning architecture for traffic feature analysis. The method decouples the pre-training language model into a lightweight module on the edge side and a deep inference module in the cloud through a hierarchical segmentation strategy, and realizes the balanced optimization of global model stability and local individualization through federated aggregation based on the multi-head attention mechanism. The edge side introduces an adversarial enhancement mechanism, combines the feature extraction capability of the pre-training language model with the dynamic game of the generative adversarial network, and improves the generalization ability of the model to complex network attacks through adversarial sample reconstruction.

Description

TECHNICAL FIELD

[0001] The application is an application of artificial intelligence technology in network security, in particular to a network intrusion detection method based on pre-training language model federated segmentation learning. BACKGROUND

[0002] As a breakthrough in the field of artificial intelligence, large language models [1] have achieved a leap in deep semantic analysis and generation capabilities through self-supervised pre-training mechanisms. Such models not only can autonomously construct a distributed representation system of language symbols from massive amounts of corpus, but also exhibit multi-modal transfer learning capabilities. Their feature fusion architecture based on attention mechanisms can effectively capture potential cross-domain association patterns, continuously breaking performance benchmarks in natural language processing tasks such as machine translation and text summarization, while also showing great potential for application in other fields [2]. In the field of network intrusion detection, through learning and analyzing a large amount of historical data, language models can identify complex patterns and trends, predict potential system vulnerabilities, and detect abnormal behavior at an early stage [3].

[0003] Under the background of increasing privacy protection, most sensitive data are isolated and prohibited from direct sharing. Traditional centralized training mode is difficult to build high-quality language models due to data limitations, especially when the model performance is significantly limited by insufficient training samples [4]. In order to fully utilize a large amount of distributed and diversified private data [5], federated learning (FL) proposes a privacy-safe collaboration paradigm, which can train a global model across multiple datasets without sharing the original data [6, 7]. Under this framework, edge computing nodes periodically train models based on local private data, and only upload model parameter gradients to the cloud server for secure aggregation, completing the iterative update of the global model while strictly ensuring that the original data does not leave the domain. It is worth noting that although FL alleviates the conflict between data privacy and value release, the significant contradiction between the limited computing resources (such as CPU processing power, memory capacity, etc.) of edge nodes and the demand for large model training still constitutes a key bottleneck for actual deployment [8].

[0004] The challenges / problems and related work in the prior art are mainly:

[0005] 1) Balancing global and local individualization.

[0006] Due to privacy concerns, most private data are stored on edge devices. To address the data silo phenomenon formed by sensitive data stored locally on edge nodes, FL coordinates global model training through a cloud server. After the edge nodes train the model based on the local data set, they only need to upload the parameter gradient to participate in the federal global aggregation, ensuring that the original data always resides locally. However, the data of different edge nodes usually differ significantly. How to effectively maintain the stability and generalization ability of the global model while achieving personalized modeling through local fine-tuning, thereby building a collaborative optimization framework of "global consensus-local adaptation", is still an important problem. Considering the introduction of batch norm and learning rate parameters for each client given the local statistical data, the meta-network learns these meta-networks through FL, allowing the entire FL network to collaboratively learn personalized strategies for each client. Kullback-Leibler divergence estimation and adaptive weight calculation are used in the FL scenario to improve the accuracy of the model and the robustness of the adversarial attack, realizing an adaptive max-pooling-based convolutional neural network and an improved bidirectional long short-term memory network with attention and residual connections, which can effectively detect intrusions, thereby enhancing the security and privacy of the network.

[0007] 2) Dual optimization of detection accuracy and training cost.

[0008] In the deployment of edge systems, the optimization of model accuracy and training overhead constitutes a core challenge. Limited by the computational bottleneck of edge nodes (such as CPU-GPU power imbalance, memory bandwidth constraints, etc.), lightweight computing units are difficult to carry out the tensor parallel computing required by large model training. In addition, the periodic full model synchronization mechanism in the federated architecture has a negative effect: on the one hand, the transmission of massive parameters across layers exacerbates the communication bandwidth pressure between edge nodes and cloud servers; on the other hand, the exchange of full model gradients may expose the model structure characteristics, breaking the privacy and security boundaries of federated learning [11, 12]. In order to overcome the challenge of limited device computing resources,

[13] believes that deep neural networks can be split learning (SL).

[14] splits the BERT model into three main layers: Embedding layer, Transformer layer and Head layer, decoupling the potential dependencies. Based on the complexity analysis and computational analysis of deep neural networks, the Transformer layer, which requires high computing resources, is allocated on the cloud server, and the resource-constrained Embedding layer and Head layer are deployed on the client. Each client can privately adjust specific NLP tasks and incrementally combine datasets from multiple devices through federated learning and split training to improve the robustness and generalization of pre-trained models. In order to reduce the computational burden of resource-constrained edge nodes and better protect model privacy,

[15] proposes a novel split federated learning (SFL) framework MergeSFL, which combines feature merging and batch size adjustment, and is used to improve model accuracy and training efficiency.

[0009] 3) Integrated feature extraction and traffic generation process.

[0010] Pre-training language models based on the Transformer architecture have shown innovative value in network security situation awareness. The multi-granularity semantic representation space constructed by its bidirectional self-attention mechanism can realize the context-aware modeling of network traffic metadata. Pre-training language models represented by BERT can consider the context information before and after the data through hierarchical feature fusion, improving the ability to identify complex network traffic patterns. Given the strong feature extraction and generalization capabilities of BERT, researchers have attempted to apply BERT to vulnerability prediction and system log anomaly detection. ExBERT is proposed to fine-tune the pre-trained BERT model using a collected vulnerability description corpus to extract the semantic information of vulnerability descriptions, thereby predicting network security vulnerabilities. BERT-based log anomaly detection model can detect log parsing idleness. BERT can effectively extract semantic information from serialized log data, capture detailed features, and improve the accuracy of log anomaly detection. Integrating pre-training language models with existing deep learning models or frameworks to improve network intrusion detection performance is worth further exploring. SUMMARY

[0011] In view of the dual needs of privacy protection and attack identification in network intrusion detection systems, the present application proposes a network intrusion detection method based on pre-training language model federated segmentation learning. In view of the challenges of traffic feature extraction difficulty, limited computing power of edge devices, and data privacy barriers between distributed nodes in attack detection, the present application combines the privacy protection mechanism of federated learning and the computing offloading advantage of segmentation learning to construct a pre-training language model learning architecture for traffic feature analysis. The present application is as follows.

[0012] A network intrusion detection method based on pre-training language model federated segmentation learning, in an edge computing system with a cloud server and N edge nodes, the edge nodes use edge models to identify and classify local network traffic, and the cloud server is responsible for segmentation model training and federated parameter aggregation;

[0013] The edge model includes a discriminator D, a generator G and a classification network C, wherein the discriminator D and the generator G are the discriminator D and the generator G of the conditional generative adversarial network CGAN; a pre-training language model BERT is embedded in the discriminator D for feature extraction of local network traffic, and the classification network C is used for network traffic classification based on the extracted features;

[0014] The edge model is divided into a discriminator D, a generator G and a classification network C, wherein the BERT model is further divided into an Embedding layer, a Transformer layer and a Head layer;

[0015] The transformer layer and the head layer constitute a top model; the generator G, the discriminator D and the embedding layer constitute a bottom model; each edge node trains the bottom model using its local data, and the cloud server maintains the top model;

[0016] In federated split learning: each edge node trains an edge model using its local data and sends the updated parameters to the cloud server; the cloud server trains a split model and aggregates the split model updates of each edge node through a multi-head attention mechanism to form global model parameters, and distributes them back to the edge nodes; each edge node further fine-tunes according to the local data, retains the basic structure of the global model, but makes individual adjustments on the local data.

[0017] The present application decouples the pre-trained language model into an edge-side lightweight module (bottom model) and a cloud-side deep inference module (top model) through a hierarchical split strategy, and realizes the balanced optimization of global model stability and local individualization through federated aggregation based on a multi-head attention mechanism. The edge side introduces an adversarial enhancement mechanism, combines the feature extraction capability of the pre-trained language model with the dynamic game of the generative adversarial network, and improves the generalization ability of the model to complex network attacks through adversarial sample reconstruction.

[0018] Experiments on the CSE-CIC-IDS2018 dataset show that compared with the comparative method, the weighted F1-score of the present application is improved by 1.248% to 14.95%, reaching 97.875%. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A federated split learning framework representing cloud-edge collaboration;

[0020] Figure 2 A discriminator model architecture representing an edge node;

[0021] Figure 3 A hierarchical architecture of BERT;

[0022] Figure 4 A generative adversarial split iteration process of an edge model;

[0023] Figure 5 Precision of different methods;

[0024] Figure 6 Recall of different methods;

[0025] Figure 7 F1-score of different methods;

[0026] Figure 8 Detection performance changes of model splitting and non-splitting. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0028] 1 Overview

[0029] This invention proposes a novel federated segmentation learning framework based on pre-trained language models. This framework integrates the distributed collaboration advantages of federated learning with the computational offloading characteristics of segmentation learning, enabling the secure and efficient deployment of large language models. Its main contributions include:

[0030] A federated segmentation training mechanism based on a pre-trained language model is proposed for edge environments, balancing model personalization and training cost optimization. The model is decoupled and layered, with edge nodes and the cloud server stably updating global parameters during training and communication. Each edge node can be individually fine-tuned based on local data.

[0031] An adversarial learning edge model enhanced by a pre-trained language model is constructed to optimize instance feature extraction. BERT enhances the feature extraction capability of network traffic and forms a complementary relationship with generative adversarial models. Deeply integrating BERT's powerful feature extraction capabilities with generative adversarial networks allows the generator to reconstruct adversarial examples from traffic features, creating a dynamic game with the discriminator and effectively improving the model's learning ability regarding network traffic.

[0032] Experimental results show that the model proposed in this invention outperforms the contrastive detection framework in weighted accuracy, precision, recall, and F1-score on the CSE-CIC-IDS2018 dataset. Compared with other methods, the F1-score of the proposed model is improved by 1.248% to 14.95%.

[0033] The remainder of the text is organized as follows.

[0034] The second section introduces the federated segmentation learning architecture for edge models.

[0035] The third section introduces the model's training strategy and cost analysis.

[0036] Section 4 introduces the dataset selection and experimental methods.

[0037] Section 5 presents the experimental results and performance evaluation.

[0038] Section 6 summarizes the entire text.

[0039] 2. Cloud-Edge Collaborative Federated Segmentation Learning Architecture

[0040] like Figure 1As shown, consider an edge computing system with a cloud server and N edge nodes, the Intrusion Detection System (IDS) is deployed at the network edge, and the cloud platform is responsible for the segmentation model training and federated parameter aggregation. Federated segmentation learning performs deep learning tasks through a set of nodes coordinated by a cloud server.

[0041] In this architecture, each edge node trains a model based on local data and sends edge updates to the cloud server. The cloud server trains the segmentation model and aggregates the segmentation model updates of each node through the multi-head attention mechanism to form global model parameters and distribute them back to the edge nodes. Each edge node further fine-tunes based on local data, retaining the basic structure of the global model but making individual adjustments on local data. In a data heterogeneous environment, efficient personalization is achieved.

[0042] 2.1 Edge model enhanced by pre-trained language model

[0043] The pre-trained language model in this embodiment adopts BERT, which inherits the self-attention-based architecture of Transformer and can learn more information about global input and output dependencies.

[0044] As shown in Figure 1 In the edge node, D and G represent the discriminator and generator of Conditional Generative Adversarial Networks (CGAN)

[19] , respectively, and C represents the classification network layer of the model. The BERT model is embedded in the discriminator D to build a pre-trained language model enhanced C to solve the classification problem of network traffic.

[0045] As shown in Figure 2 The first token in the input sequence of BERT is a special classification token ([CLS]), and its final hidden state is used as the aggregated sequence representation for the classification task. BERT is initialized using the parameters of the pre-trained language model, and then fine-tuned using the preprocessed network intrusion dataset.

[0046] In the edge model, the role of BERT is not only to enhance the feature extraction capability of network traffic. More importantly, BERT and G form a complementary relationship. Specifically, in order to learn a suitable probability distribution of traffic features, G generates a high-dimensional feature representation of network traffic that changes with the fine-tuning of BERT parameters. G continuously learns the high-dimensional feature representation of network traffic, and D also continuously updates its parameters.

[0047] 2.2 Segmentation and transfer of pre-trained language model

[0048] The basic idea of federated split learning is to split the model into two sub-models, i.e., the bottom model and the top model, on the split layer, denoted as w b , w p . Split learning (SL) enables low-computing edge nodes to train language models within sufficient training time. To effectively train on data, BERT should be split in a way that preserves the potential dependencies between modules.

[0049] As shown in Figure 3 , the BERT model is divided into three main layers: Embedding, Transformer, and Head, which can decouple potential dependencies. Based on the complexity analysis and computational analysis of deep neural networks, the Transformer requires high computing resources and can be allocated on powerful cloud servers. After training the model, each edge node can adjust specific tasks. In addition, by using federated learning, split training can incrementally merge datasets from multiple edge nodes to improve the robustness and generalization of pre-trained models.

[0050] In this framework, the edge model enhanced by the pre-trained language model is split into G, C, the Embedding layer, the Transformer layer, and the Head layer of BERT. Considering the communication cost and computing time, the top model contains the Transformer layer and the Head layer, denoted as The bottom model contains G, C, and the Embedding layer of BERT, denoted as In federated split learning, the server maintains while each edge node trains In addition, the nodes complete the entire training process by interacting with the top model on the cloud server.

[0051] 3 Training strategy that balances global and local individualization

[0052] 3.1 Iterative generation of edge models by generative adversarial networks

[0053] In this architecture, the cloud server maintains the top model while each edge node trains the bottom model The iterative goal of the split model is to find the optimal model w = [w b , w p ] that minimizes the loss function L(w). Let w p,i , w b,i denote the top model and the bottom model corresponding to edge node i, respectively, and L(wp,i ) indicates w p,i The loss function, L(w) b,i ) indicates w b,i The loss function.

[0054]

[0055] like Figure 4 As shown, the basic training process mainly includes forward / backward propagation of the bottom-level model and forward / backward propagation of the top-level model. First, each edge node performs forward propagation using a batch of local data samples, passing the features of the split layer to the cloud server. The cloud server then performs forward / backward propagation to update the top-level model. Next, the cloud server sends the backpropagated gradients back to the edge nodes, updating the bottom-level model through backpropagation. Such a complete forward / backward propagation process can be considered a local iteration.

[0056] The actual traffic data x is treated as sentences processed in the BERT model. Special characters, such as [CLS] and [SEP], are added before and after each traffic sequence to facilitate processing in edge nodes. Input recognition. In the cloud server. Attention scores between traffic sequences are calculated using multi-head attention, where each attention head performs a self-attention function on the embedding sequence of input x to obtain a new vector. The vector of the hidden state [CLS] position is used as a semantic representation of the flow sequence. In edge nodes... Process h and g, where h is the traffic data passing through w p The probability distribution of the encoding mapped to a high-dimensional space, g is... A high-dimensional feature representation generated based on class label y and random noise vector z. The output probability represents the traffic type. During training, An attempt is made to generate high-dimensional feature representations of network traffic to confuse the classifier. Trying to correctly distinguish which type of traffic data.

[0057] set up S represents the underlying model of edge node i in the k-th iteration. i This represents a small batch of data samples. Given x and The stochastic gradient is then applied. After one iteration, the underlying model is updated to:

[0058]

[0059] set up This represents the top-level model at the k-th iteration. For a given Output and The stochastic gradient. Then the top-level model after one iteration update is:

[0060]

[0061] During the iteration process, D is represented as w D (x) represents the category to be determined for x. ) indicates distinction The generated samples. In the edge model, and w D The optimization objective function is expressed as:

[0062]

[0063] The objective function is represented as follows: Attempting to generate something that can deceive w D Traffic, w D It attempts to distinguish between real traffic and generated traffic. During the adversarial optimization process, BERT promotes... D Then continuously improve classification ability, while Continuously improve your generation capabilities.

[0064] Algorithm 1's generative adversarial segmentation iteration requirements for edge models: cloud server S; edge node set I; edge node training rounds E

[0065] 01: Initialization

[0066] ServerTrmFordprop

[0067] 02: ←Calculate Forward propagation

[0068] 03: h ← Calculation Forward propagation

[0069] 04: Send h

[0070] ServerTrmBackprop

[0071] 05: ←Calculate according to formula (3) Backpropagation

[0072] 06: ←Calculate according to formula (3) Backpropagation

[0073] 07: Send

[0074] Edge node i:

[0075] EdgeUpdate():

[0076] 08: for each local epoch e = 1...E do09: ← Compute the forward propagation of

[0077] 10: h ← ServerTrmFordprop

[0078] 11: g ←

[0079] 12: ←

[0080] 13: ← Compute the backward propagation of

[0081] 14: ←

[0082] 15: ← Compute the backward propagation of

[0083] 16: ← Compute the backward propagation of

[0084] 17: end for

[0085] 3.2 Parameter aggregation based on multi-head attention mechanism

[0086] The cloud server maintains independent Transformer layers and Head layers for edge nodes, denoted as In the cloud, the multi-head attention mechanism is used to aggregate from edge nodes and send the aggregated back to edge nodes.

[0087] After multi-layer stacking, Q, K, V are generated, where Q, K, V are query matrix, key matrix, and value matrix respectively. In the aggregation process, and ​​​​are input into the multi-head attention mechanism as Q and K, respectively. For each attention head, a self-attention function is performed to obtain a new vector. Through the convergence of multiple attention heads, the edge nodes that contribute more to the global model update are focused on, thereby achieving more accurate model aggregation. The attention weights of the aggregated model parameters are denoted as [γ1, γ2,..., γ i ],

[0088]

[0089] After several rounds of local iterative updates, are updated to

[0090]

[0091] Algorithm 2 Parameter aggregation based on multi-head attention mechanism

[0092] Requirements: cloud server S; edge node set I

[0093] Cloud server:

[0094]

[0095]

[0096] 3.3 Cost analysis of joint training

[0097] This subsection analyzes the communication cost and computation time of federated split learning. The communication cost specifically refers to the amount of data exchanged between the cloud server and the edge node i in federated split learning. Given I edge nodes, p is the number of training batches in a global epoch, |w| is the parameter size of the complete model, and L is the parameter size propagated between split layers. In the traditional federated learning FL mode, the entire |w| needs to be uploaded to the cloud server. FL involves bidirectional transmission of the model, so the data volume is 2|w|. In federated split learning, only the parameters between the split layers need to be transmitted, and since the split model includes forward and backward propagation, the data volume is 4pL. Therefore, when 4pL < 2|w|, the communication cost of the method in this paper is less than that of traditional FL.

[0098] The computation time reflects the time cost of model training. is the computation time of a global epoch, is the aggregation time of w. For split learning, the computation time of the top-level model is represented as The computation time of the bottom-level model is represented as The original computation cost of the edge model can be represented as Since is greater than and Therefore The edge model calculation time of the method of the application is less than that of the conventional FL.

[0099] Table 1 Cost analysis of joint training

[0100]

[0101] Figures 1 to 4 Translation of main terms in :

[0102] Multi-head self-attention, Scaled Dot-Product Attention, Split Learning, Gradient dispatching, Forwardpropagation, Augmented Traffic, Original Traffic, Fullconnected layers, Traffic Feature, Traffic data sequence.

[0103] 4 Experimental design

[0104] 4.1 Dataset

[0105] Table 2 Number distribution of the dataset

[0106]

[0107] 4.2 Environment configuration

[0108] All models in the experiment run in the environment of Python 3.5, Tensorflow 2.12.0. The computer configuration is Intel Core i9-13900K CPU, RTX 4090 24G and 128GB RAM.

[0109] 4.3 Comparison method

[0110] In order to make a comprehensive comparison and verification, this paper selects and designs multiple experimental methods. In order to evaluate the influence of different strategies on the overall performance, the proposed scheme is divided into four categories as shown in Table 3. In order to compare comprehensively and objectively, two distributed learning methods listed in Table 4 are selected as the benchmark method (the benchmark method has no model splitting).

[0111] Table 3 Classification of the proposed method

[0112]

[0113] Table 4 Classification of Baseline Methods

[0114]

[0115] 4.4 Evaluation Metrics

[0116] Accuracy: Accuracy represents the ratio of correctly predicted network flows to the total number of classification predictions, measuring the model's ability to correctly predict normal traffic and attack traffic.

[0117]

[0118] Precision: Precision represents the proportion of correctly classified attacks in a set of attack predictions, measuring the model's precision in detecting attack traffic. This metric represents the model's ability to accurately identify attack traffic.

[0119]

[0120] Recall: Recall represents the proportion of correctly classified attacks in a given set of attack samples, measuring the model's recall in detecting attack traffic. This metric represents the model's ability to accurately identify attack traffic.

[0121]

[0122] F1-Score: F1-Score is defined as the harmonic mean of Precision and Recall, providing a comprehensive reflection of the model's performance.

[0123]

[0124] TP = True Positives, TN = True Negatives, FP = False Positives, FN = False Negatives.

[0125] 5 Results Analysis

[0126] This section first compares the weighted average classification performance of different methods on the test set, and then analyzes the Precision, Recall, and F1-score of the model in detecting specific types of network traffic to evaluate the detection performance of the proposed method for different attack traffic.

[0127] 5.1 Detection Accuracy

[0128] Table 5 shows the Accuracy, Precision, Recall and F1-score of different methods on the CSE-CIC-IDS2018 test set. Proposed-1 outperforms other methods in all evaluation metrics, followed by Proposed-2 and Proposed-4, while Baseline-2 is the lowest. The Accuracy, Precision and F1-score of Proposed-1 reach 97.806%, 98.037% and 97.875%, respectively, which are the highest among all methods. Compared with Proposed-2 and Proposed-4, the Accuracy of Proposed-1 increases by 1.466% and 1.295%, respectively, the Precision increases by 0.701% and 1.186%, respectively, and the F1-score increases by 1.254% and 1.248%, respectively. Compared with Baseline-1, the four performance indicators of Proposed-1 increase by 2.274%, 1.442%, 2.274% and 2.065%, respectively. Proposed-1 adopts federated split learning, each edge node retains the corresponding Transformer layer copy on the cloud server, and the edge nodes are trained in parallel. All Transformer layers are parameter aggregated based on multi-head attention mechanism on the cloud server. Compared with Proposed-2 and Proposed-4, Proposed-1 integrates the CGAN framework as the edge model and adopts the multi-head attention mechanism for federated split aggregation, which can improve the Precision, Recall and F1-score. Proposed-1 has the feature extraction capability of BERT and the generalization capability of CGAN, and combines the multi-head attention mechanism for parameter aggregation, which further reduces misclassification and false negatives and improves the F1-score.

[0129] Meanwhile, the four performance indicators of Proposed-1 increase by 12.063% to 14.95% compared with Baseline-2. This is because the self-attention mechanism of the BERT model allows the establishment of a dependency model without considering the distance of features in the input or output sequence. Even if the network traffic statistical features in the sequence are farthest apart, BERT can capture the internal relationship between network statistical features through the self-attention mechanism. The classifier integrated with BERT can obtain more information about attack categories from the captured high-dimensional features, thereby improving classification accuracy. Compared with BERT, LSTM has weaker ability to focus on global dependency relationships between network statistical features. When faced with small proportions of attack categories, they will not be able to correctly distinguish attack types.

[0130] Table 5 Weighted average performance of different methods

[0131]

[0132] Figure 5 、 Figure 6 and Figure 7 Precision, Recall and F1-score of different methods for identifying normal traffic and possible attack types are given respectively. Each piece of network traffic corresponds to multiple statistical features, which cover the feature information of network traffic. Each statistical feature corresponds to a word with a specific meaning. The internal relationship between statistical features is equivalent to the context relationship in a sentence. Compared with other methods, Proposed-1 has the highest detection accuracy for almost all network flow types. As shown in Figure 5 Fig. 6, for the detection of the Infilteration attack type with high concealment, the evaluation index Precision of Proposed-1 is improved by 15.624% and 38.196% compared with Baseline-1 and Baseline-2 respectively, and is also improved by 15.155% and 7.622% compared with Proposed-2 and Proposed-4 respectively, reaching 76.994%. This shows that Proposed-1 can significantly improve the problem that some attack types are difficult to identify.

[0133] As shown in Figure 6 , compared with Baseline-2, the Recall of Proposed-1 on each traffic type is significantly improved. It is worth noting that for most network attack categories, the Recall of each method using BERT as a feature extractor is higher than the Precision, but for the benign category, the Recall is lower than the Precision. This is because BERT may classify the benign category as an attack category in some cases due to the emphasis on detecting attack categories. Therefore, the Recall of the benign category with the largest proportion is lower than the Recall of some attack categories with small proportions. In the performance indicators Recall of detecting the Benigh and Infilteration categories, Proposed-1 has improved the detection results of Proposed-4. Compared with Proposed-2 and Baseline-1, Proposed-1 has a 2.054% and 2.308% decrease in detecting Infilteration, and a 3.094% and 4.489% increase in detecting Benigh. This shows that Proposed-1 improves the situation of classifying the benign category as an attack category, and can reduce the false positive rate.

[0134] Figure 7The F1-score of different methods to identify normal traffic and possible attack types is shown. Proposed-1 is superior to other methods in the comprehensive indicator F1-score of almost all network attack categories. For some high concealment attack types (such as Infilteration), Proposed-2 and Proposed-4 still have the problem of high false positive probability and unobvious detection effect improvement. Compared with Proposed-2 and Proposed-4, the F1-score of Proposed-1 in detecting Infilteration attack type is increased by 9.229% and 7.403% respectively, reaching 83.899%. In Proposed-1, BERT and CGAN promote each other in learning, while the multi-head attention mechanism is used for federated segmentation of parameter aggregation, thereby improving the model detection performance, especially for some difficult to identify or small attack types.

[0135] 5.2 Segmentation Impact

[0136] In this experiment, Proposed-1 based on language model segmentation and Proposed-3 without language model segmentation are selected to investigate the impact of model segmentation on detection accuracy in collaborative training. As shown in Figure 8 , with the increase of training rounds, the trend of detection performance of segmented models and non-segmented models is almost consistent.

[0137] 6 Summary

[0138] In view of the performance optimization and resource constraint problems in the edge network intrusion detection scene, the invention proposes a federated segmentation learning framework based on pre-training language model and conditional generative adversarial network collaborative optimization. The high-order feature extraction layer of the pre-training language model is deployed in the cloud, while the lightweight adversarial enhancement module is retained on the edge side, realizing the dual optimization of detection accuracy and resource efficiency. A hierarchical federated aggregation mechanism is constructed, and the local traffic features of edge nodes are fused by using the multi-head self-attention mechanism, so as to realize the aggregation of global model. An edge model enhanced by generative adversarial learning is developed, which improves the model detection performance through adversarial sample reconstruction and discriminator dynamic game.

[0139] REFERENCES

[0140] [1] Ling C, Zhao X, Lu J, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey[J]. arXiv preprint arXiv:2305.18703, 2023.

[0141] [2] Yao Y, Duan J, Xu K, et al. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly [J]. High-Confidence Computing, 2024: 100211.

[0142] [3] G. Lira O, Marroquin A, To M A. Harnessing the advanced capabilities of llm for adaptive intrusion detection systems [C] / / International Conference on Advanced Information Networking and Applications. Cham: Springer Nature Switzerland, 2024: 453-464.

[0143] [4] Jin Z, Zhou J, Li B, et al. FL-IIDS: A novel federated learning-based incremental intrusion detection system [J]. Future Generation Computer Systems, 2024, 151: 57-70.

[0144] [5] Bhavsar M, Bekele Y, Roy K, et al. FL-IDS: Federated Learning-Based Intrusion Detection System Using Edge Devices for Transportation IoT [J]. IEEE Access, 2024.

[0145] [6] Qi P, Chiaro D, Guzzo A, et al. Model aggregation techniques in federated learning: A comprehensive survey [J]. Future Generation Computer Systems, 2024, 150: 272-293.

[0146] [7] Lyu L, Yu H, Ma X, et al. Privacy and robustness in federated learning: Attacks and defenses [J]. IEEE transactions on neural networks and learning systems, 2022.

[0147] [8] Park J, Samarakoon S, Elgabli A, et al. Communication-efficient and distributed learning over wireless networks: Principles and applications [J]. Proceedings of the IEEE, 2021, 109(5): 796-819. [9] Lee R, Kim M, Li D, et al. FedL2P: Federated learning to personalize [J]. Advances in Neural Information Processing Systems, 2024, 36.

[0148]

[10] Begum K, Mozumder M A I, Joo M I, et al. BFLIDS: Blockchain-driven federated learning for intrusion detection in IoMT networks [J]. Sensors, 2024, 24(14): 4591.

[0149]

[11] Thapa C, Arachchige P C M, Camtepe S, et al. SplitFed: When federated learning meets split learning [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8485-8493.

[0150]

[12] Han D J, Bhatti H I, Lee J, et al. Accelerating federated learning with split learning on locally generated losses [C] / / ICML 2021 workshop on federated learning for user privacy and data confidentiality. ICML Board. 2021.

[0151]

[13] Vepakomma P, Gupta O, Swedish T, et al. Split learning for health: Distributed deep learning without sharing raw patient data [J]. arXiv preprint arXiv:1812.00564, 2018.

[0152]

[14] Tian Y, Wan Y, Lyu L, et al. FedBERT: When federated learning meets pre-training [J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2022, 13(4): 1-26.

[0153]

[15] Liao Y, Xu Y, Xu H, et al. Mergesfl: Split federated learning with feature merging and batchsize regulation [C] / / 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024: 2054-2067.

[0154]

[16] Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv 1810.04805 [preprint] https: / / doi.org / 10.48550 / arXiv.1810.04805. Posted October 11, 2018 [J]. Accessed May, 2024.

[0155]

[17] Yin J, Tang M J, Cao J, et al. Apply transfer learning to cybersecurity: Predicting exploitability of vulnerabilities by description [J]. Knowledge-Based Systems, 2020, 210: 106529.

[0156]

[18] Lee Y, Kim J, Kang P. Lanobert: System log anomaly detection based on bert masked language model [J]. Applied Soft Computing, 2023, 146: 110689.

[0157]

[19] Mirza M. Conditional generative adversarial nets [J]. arXiv preprint arXiv:1411.1784, 2014.

Claims

1. A network intrusion detection method based on pre-training language model federated segmentation learning, characterized in that, In an edge computing system with a cloud server and N edge nodes, the edge nodes use an edge model to identify and classify local network traffic, and the cloud server is responsible for splitting model training and federated parameter aggregation; The edge model is an architecture integrating BERT and conditional generative adversarial network CGAN, including a generator G and a discriminator D composed of BERT and a classification network C; the pre-trained language model BERT embedded in the discriminator D is used to extract features of local network traffic, and the classification network C detects and classifies based on the extracted traffic features; The edge model is divided into a BERT feature extraction layer, a classification network C and a generator G, wherein the BERT model is further divided into an Embedding layer, a Transformer layer and a Head layer; The Transformer layer and the Head layer constitute a top model, and are respectively represented as and The top model is represented as ; the generator G, the classification network C and the Embedding layer constitute a bottom model, and are respectively represented as , and The bottom model is represented as ; and the edge model is represented as ; Each edge node trains using its local data , and The cloud server maintains and ; In federated split learning: each edge node trains the edge model using its local data and sends the updated parameters to the cloud server; the cloud server trains the split model and aggregates the split model updates of each edge node through the multi-head attention mechanism to form global model parameters and distributes them back to the edge nodes; each edge node further fine-tunes according to the local data, retains the basic structure of the global model, but makes individual adjustments on the local data; The iterative goal of the segmentation model is to find the optimal model such that the global model loss function is minimized; Let and denote the top-level model and the bottom-level model corresponding to the edge node i, respectively, denote the loss function of denote the loss function of denote the loss function of denote the loss function of ； The parameter aggregation based on the multi-head attention mechanism is: The cloud server maintains independent Transformer layers and Head layers for the edge node i, denoted as , In the cloud server, the multi-head attention mechanism is used to aggregate the parameters from the edge nodes and send the aggregated parameters back to the edge nodes; After multi-layer stacking, Q, K, V are generated, where Q, K and V are query matrix, key matrix and value matrix respectively; in the aggregation process, and are input into the multi-head attention mechanism as Q and K respectively; For each attention head, a self-attention function is executed to obtain a new vector, and through the convergence of multiple attention heads, the edge nodes that contribute greatly to the global model update are focused, thereby realizing model parameter aggregation; The attention weights of the aggregated model parameters are represented as , ； After completing the multiple rounds of local iterative updates, are updated to 。 2. The network intrusion detection method based on pre-trained language model federated segment learning according to claim 1, characterized in that, The basic training process of the edge model includes: first, each edge node performs forward propagation with a batch of local data samples to pass the features of the underlying model to the cloud server; the cloud server performs forward / backward propagation to update the top-level model; then, the cloud server sends the backward propagation gradient back to the edge node to update the underlying model through backward propagation; such a complete forward / backward propagation process is a local iteration; In the model training of edge node i: The real traffic data x is treated as a sentence in the BERT model, and special characters [CLS] and [SEP] are added before and after each traffic data sequence respectively; In the cloud server: The attention score between the flow data sequences is calculated by multi-head attention, each attention head performs a self-attention function through the embedding sequence of the input x to obtain a new vector; the vector of the hidden state [CLS] position is used as the semantic representation of the flow sequence; In the edge node: process h and g, where h represents the flow data passing through encoding a probability distribution that maps to a high-dimensional space, representing a high-dimensional feature representation generated from the class label y and a random noise vector z; the output probability represents the flow type; During training, attempting to generate a high-dimensional feature representation of network traffic to confuse the classifier, attempting to correctly distinguish which class of traffic data; Let denote the base model of edge node i at the kth iteration, S i denote a mini-batch of data samples, denote the stochastic gradient of the loss function L with respect to the base model S ; then after one iteration, the base model is updated as: ； Let represent the top-level model at the kth iteration, be the random gradient of the loss function Output and the random gradient of the loss function The updated top-level model after one iteration is: ； During the iteration process, the discriminator D is represented as , represents the class of x, represents the distinction generated samples; In the edge model, and The optimization objective function is expressed as ； The meaning of the objective function is, attempting to generate traffic that can deceive a flow, attempting to distinguish between real traffic and generated traffic.

Images