Industrial internet network intrusion detection method based on pre-trained large language model

Abstract

The invention provides an industrial internet network intrusion detection method based on a pre-trained large language model, and the method comprises the steps: carrying out the data preprocessing of original network flow data through protocol self-adaptive flow aggregation and session segmentation, feature screening and feature coding; constructing a stream format text data set used for training a generation model GPT-2 and a packet level classification text-label data set used for training a classification model DistilBERT; then fine tuning training is carried out on the generation model GPT-2 and the classification model DistilBERT; then calling a trained generation model GPT-2 to generate a traffic sequence, obtaining a prediction data packet, calling a trained classification model DistilBERT, performing anomaly judgment on sequence data in the prediction data packet one by one, and outputting a classification result of anomaly judgment; the active intrusion detection of first generation and then discrimination is realized. According to the method, prediction can be made before real attack traffic arrives, and network attacks are prevented.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network traffic intrusion detection, and in particular to an industrial internet network intrusion detection method based on a pre-trained large language model. BACKGROUND

[0002] With the rapid development and wide application of industrial internet technology, network security problems are becoming increasingly serious, and network attacks on industrial control systems and critical infrastructure are showing a high incidence trend. Traditional network intrusion detection technologies, such as signature-based detection methods, perform pattern matching by maintaining an attack feature library, but this method is difficult to identify unknown zero-day attacks and new attack variants. While the detection method based on statistical anomaly can detect attacks that deviate from normal behavior patterns, it usually treats network traffic as independent statistical data, ignoring the temporal correlation and contextual information between traffic, resulting in a high false positive rate and being unable to effectively deal with complex, multi-stage coordinated attacks.

[0003] To overcome the limitations of traditional methods, researchers have introduced machine learning and deep learning techniques. These methods can automatically learn and extract attack patterns from high-dimensional complex traffic data, improving the accuracy of detection to some extent. However, these methods usually rely on manually designed expert features, limiting the model's ability to discover potential attack patterns. In addition, their performance is highly dependent on large-scale and accurately labeled training datasets, but in real network environments, obtaining high-quality labeled data is costly, and there is a serious class imbalance problem between attack samples and normal samples, which limits the model's generalization ability and robustness in real-world scenarios. In addition, since these methods are essentially passive analysis of existing traffic, they are difficult to provide timely warnings and early defenses, and their defense lag is particularly prominent in scenarios such as industrial internet that require high real-time performance and security.

[0004] In recent years, large language models (LLM) have shown excellent sequence modeling and context understanding capabilities in natural language processing and other fields, bringing new solutions to the field of network intrusion detection. By analogy, network traffic data can be considered as a kind of "language" with specific grammar and semantics, and large language models (LLM) can be used to capture complex and long-term dependencies in traffic sequences. The "pre-training-fine-tuning" paradigm of large language models (LLM) can learn general traffic pattern rules from massive unlabeled data, and then fine-tune with a small amount of labeled data for specific tasks, significantly reducing the dependence on labeled data and effectively improving the ability to identify unknown attacks.

[0005] In summary, the existing network intrusion detection technology route still focuses on classifying or abnormity distinguishing of the occurred network traffic. Such a technical paradigm is essentially a "post-event" passive defense, and the inherent analysis delay makes it difficult for the system to effectively intervene before the attack causes substantial damage, which is particularly prominent in the industrial internet and other high real-time and high security application scenarios. SUMMARY

[0006] The purpose of the present application is to provide an industrial internet network intrusion detection method based on a pre-trained large language model, to solve the limitations of the existing industrial internet network intrusion detection technology, which relies on passive analysis, response lag, and is difficult to cope with class imbalance data in the industrial internet scenario. The method of the present application constructs an active prediction framework of "first generation, then discrimination", which predicts the future possible network traffic sequence, and analyzes and discriminates the predicted sequence in real time, aiming to move the defense gate forward, change passive response to active warning, and achieve the purpose of preventing network attacks in advance.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows:

[0008] The industrial internet network intrusion detection method based on the pre-trained large language model comprises the following steps:

[0009] Step S1, protocol adaptive flow aggregation and session segmentation are performed on the original network traffic data, five-tuple information including source IP address, destination IP address, source port, destination port and protocol type is constructed and a timestamp is obtained, then the original network traffic data is aggregated and grouped based on the five-tuple information and the timestamp to obtain an aggregated data set; then feature screening and feature coding are performed on the aggregated data set, and a flow format text data set for training the generation model GPT-2 and a packet level classification text-label data set for training the classification model DistilBERT are constructed respectively;

[0010] Step S2, the pre-training weight of the generation model GPT-2 is loaded, the control token is added and the embedding layer dimension of the generation model GPT-2 is adjusted, the generation model GPT-2 is fine-tuned on the flow format text data set obtained in step S1 with the goal of causal language modeling, and a traffic generation model capable of predicting the next timestamp packet based on the historical window packet is obtained;

[0011] Step S3, the pre-training weight of the classification model DistilBERT is loaded, a task architecture for single packet multi-classification is constructed, the classification model DistilBERT is fine-tuned on the packet level classification text-label data set obtained in step S1 using the Focal Loss loss function and the early stopping strategy, and a classification model capable of packet-by-packet abnormality discrimination is obtained;

[0012] Step S4: Using the first L data packets of the real network flow at the current timestamp as the initial historical window, a prediction framework is formed using the generative model GPT-2 trained in step S2 and the classification model DistilBERT trained in step S3. First, the trained generative model GPT-2 is called to generate traffic sequences and obtain predicted data packets. Then, the trained classification model DistilBERT is called to perform anomaly detection on each sequence data in the predicted data packets and output the classification results of the anomaly detection, thus realizing proactive intrusion detection that first generates and then detects.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] Step S11: First, protocol-adaptive flow aggregation and session segmentation are performed on the original network traffic data through protocol inference, IP address normalization, L4 port construction, and timestamp resolution to construct a five-tuple information containing source IP address, destination IP address, source port, destination port, and protocol type. Then, the five-tuple information is sorted to construct a direction-independent session key Key as a session identifier. Based on the session key Key, the data packets of the original network traffic data are aggregated and grouped, with data packets having the same session key Key grouped into the same data group, aggregating them into a network flow containing bidirectional communication content. A protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet is calculated. Based on the difference Perform session segmentation on network flows and calculate the difference. Greater than the preset session timeout threshold for the corresponding protocol type The data packet is determined to be the start of a new session, and a session segment number S is assigned to each session segment. segment The network flow is divided into multiple sessions; finally, each session is assigned a session key (Key) and a session segment number (S). segment A globally unique flow ID is assigned to each shared, unique session, and a network flow defined by the flow ID is constructed. To obtain the aggregated dataset;

[0015] Step S12: Perform feature filtering and feature encoding on the aggregated dataset, remove unstructured load features, single constant value features, and sparse features, and encode discrete features and categorical features numerically, convert all features into numerical format, and obtain the filtered dataset composed of D-dimensional feature columns.

[0016] Step S13: Obtain the set of all stream IDs generated in step S11. , will set After random shuffling, the data is divided into four mutually exclusive ID subsets in a 7:1:1:1 ratio. Based on the flow IDs in the four mutually exclusive ID subsets, the network flows in the dataset filtered in step S12 are... Extract and group the data respectively to construct mutually exclusive training, validation, test, and generation sets at the stream level; then, combine each network stream in each dataset. The data packets are arranged in chronological order, and all feature values ​​in each data packet are concatenated into a text string. The same network flow Text strings of all data packets Connect them using newlines in chronological order, and in the network stream. Add a stream start marker at the beginning of the network stream. Add a stream end marker at the end position of each network stream. Convert it into a long text sequence to obtain a streaming text dataset for training the generative model GPT-2;

[0017] Step S14: Construct classification samples on a per-data-packet basis, concatenating the features from each data packet in the filtered dataset obtained in step S12 into a text string. Simultaneously, the attack category feature columns in the original network traffic data are mapped to multi-classification label IDs; each text string is... Combined with the corresponding tag ID, a package-level classification sample is constructed, containing text strings. The packet-level sample dataset with two features, label ID and label ID, is obtained by stratified sampling of the packet-level sample dataset and dividing it into a packet-level training set, validation set and test set in a ratio of 8:1:1, resulting in a packet-level classification text-label dataset in CSV format for training the DistilBERT classification model.

[0018] Further, in step S11, the session key Key is defined as follows: , In the formula, Proto represents the protocol type. Indicates the source IP address. Indicates the destination IP address. Indicates the source port. Indicates the destination port; This means taking the minimum value between two elements. This means taking the maximum value of two elements.

[0019] Furthermore, in step S11, a protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet is calculated. , according to the difference The specific method for session segmentation of network flows is as follows:

[0020] According to the protocol type Proto, define the protocol adaptive session timeout threshold ; sort the data packets in each network flow in ascending order according to the timestamp ; traverse the sorted data packets, calculate the difference between the timestamp of the first data packet and the timestamp of the second data packet , , ; if the difference is greater than the session timeout threshold corresponding to the protocol type Proto , then determine that the data packet with timestamp corresponds to the start of a new session, and assign a new session segment number S to the data packet with timestamp segment , and divide the network flow into multiple session segments.

[0021] Further, the step S2 specifically comprises the following steps:

[0022] Step S21, load the pre-training weight file of the generation model GPT-2; the generation model GPT-2 includes a BPE tokenizer, a decoder, and an output layer; the decoder includes an embedding layer and N-layer Transformer decoder blocks; the embedding layer includes a token embedding layer and a position embedding layer; the Transformer decoder block is composed of a first layer normalization layer, a multi-head self-attention mechanism, a first residual connection, a second layer normalization layer, a feedforward neural network, and a second residual connection; the output layer is composed of a language modeling head; the BPE tokenizer is used to convert the text data packet string into a digital ID sequence that the model can understand, and to decode the model output ID sequence into text; the generation model GPT-2 is used for text generation tasks;

[0023] Step S22, add control tokens, expand the vocabulary of the BPE tokenizer, and adjust the weight matrix dimension of the embedding layer according to the sequence length; the control tokens include a flow start token, a flow end token, and a padding token, which are used to indicate the start and end of the network flow, and to unify the sequence length in batch training;

[0024] Step S23, use the BPE tokenizer to perform tokenization processing on the long text sequence of each network flow in the flow format text data set constructed in step S13, and convert the text string ​The input token index sequence is generated by decomposing the input sequence into a series of vocabulary units and mapping each vocabulary unit to its corresponding semantic numerical ID in the vocabulary table, and the initial input attention mask sequence corresponding to the input token index sequence is generated; then the length of all sequences is unified by controlling the token, and the input token index sequence and the input attention mask sequence with consistent length are obtained; then the prediction target sequence for calculating the loss function of the generation model GPT-2 is constructed according to the input token index sequence and the input attention mask sequence; the prediction target sequence, the input token index sequence and the input attention mask sequence are input into the generation model GPT-2 as input sequences;

[0025] In step S24, the token embedding layer of the generation model GPT-2 maps each token index in the input token index sequence generated in step S23 to a token embedding vector , and obtains a token embedding vector sequence ; define as the length of the input sequence in the current batch, and the position embedding layer generates a corresponding position index for each token in the input sequence from 0 to , and maps all position indexes to a fixed-dimensional position embedding vector ; finally, the position embedding vector is added to the token embedding vector sequence element by element to obtain an input representation that integrates semantic information and time sequence position information ;

[0026] In step S25, the input representation is input into the N-layer Transformer decoder block of the generation model GPT-2 for calculation to obtain the output representation of the decoder; the dimension of the output representation of the decoder is projected to the dimension size of the extended BPE tokenizer vocabulary through the language modeling head of the output layer of the generation model GPT-2 to obtain a logit vector; the probability distribution of each token in the vocabulary table as the next timestamp token is calculated according to the logit vector;

[0027] In step S26, the cross-entropy loss function is used to optimize the generation model GPT-2, and the generation model GPT-2 is fine-tuned to learn the sequence generation rule of network traffic data packets, and a trained generation model GPT-2 is obtained as a final traffic generation model for generating predicted traffic sequences.

[0028] Further, the step S3 specifically includes the following steps:

[0029] ​Step S31, load the pre-training weight file of the classification model DistilBERT; the classification model DistilBERT comprises a WordPiece tokenizer, an encoder, and an output layer; the encoder of the classification model DistilBERT is composed of an embedding layer and N-layer stacked bidirectional Transformer encoder blocks; the embedding layer comprises a token embedding layer and a position embedding layer; the output layer of the classification model DistilBERT is composed of a fully connected classification head; the output class number of the fully connected classification head is set to the total number of classification labels, and each class is represented by a mapping value;

[0030] Step S32, configure specific control tokens for the WordPiece tokenizer, including classification marks for aggregating global semantic information, separation marks for identifying the end of text, and fillers for uniform sequence length;

[0031] Step S33, convert the text string constructed in step S14 into a classification input tensor using the WordPiece tokenizer, the classification input tensor comprising a classification token index sequence and a classification attention mask sequence, and the lengths of all sequences in the classification input tensor are unified, and the text string in the classification input tensor exceeding the maximum sequence length is truncated, and the text string less than the maximum sequence length is padded with fillers;

[0032] Step S34, send the classification input tensor into the embedding layer of the classification model DistilBERT for processing; the token embedding layer maps the classification token index sequence into a classification token embedding vector, and the position embedding layer generates a classification position embedding vector corresponding to the classification token embedding vector; the classification token embedding vector and the classification position embedding vector are added element by element to obtain a classification input representation that fuses semantic and positional information;

[0033] Step S35, send the classification input representation into the N-layer stacked bidirectional Transformer encoder block for processing, calculate the global context information by using the bidirectional attention mechanism, and obtain a hidden state vector containing the complete semantics of the data packet;

[0034] Step S36, input the hidden state vector output by the last layer of the bidirectional Transformer encoder block into the fully connected classification head of the output layer, project the features to the class space, and output the probability distribution of the data packet belonging to each classification label by using the Softmax function;

[0035] ​​​Step S37: Construct the Focal Loss function suitable for imbalanced samples. By introducing a modulation factor, reduce the weight of easily classified samples and improve the classification performance of the DistilBERT model on imbalanced data.

[0036] Step S38: The AdamW optimizer is used to calculate the gradient of the loss function and update the parameters of the classification model DistilBERT through the backpropagation algorithm. An early stopping strategy is introduced to select the optimal model and obtain the trained classification model DistilBERT, which is used as the final classification model for anomaly detection.

[0037] Furthermore, step S4 specifically includes the following steps:

[0038] Step S41: First, extract the preceding data from the real network stream to be detected. The data packets are used as initial historical data packets; the D-dimensional feature columns obtained after filtering the initial historical data packets in step S12 are converted into text strings according to the method described in step S13, and concatenated in chronological order to form an initial historical window; the initial historical window is used as the initial input and fed into the generative model GPT-2 trained in step S2 to start the process of generating the predicted traffic sequence.

[0039] Step S42: After starting the process of generating the predicted traffic sequence, an iterative loop is performed. The last L data packets of the current existing data packets, including the initial historical data packets and the generated data packets, are always taken as new prompt inputs to form a context sliding window for the iterative loop. Based on the context sliding window, the generation model GPT-2 autoregressively generates the text string of the predicted data packets for the next timestamp.

[0040] Step S43: The text string of the generated single prediction data packet is sent to the DistilBERT classification model trained in step S3 to perform real-time analysis on the prediction data packet, output the classification result of anomaly detection and record it.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention provides an industrial internet network intrusion detection method based on a pre-trained large language model, aiming to solve the problems of passive defense and delayed response in traditional network intrusion detection technologies. This method achieves proactive predictive intrusion detection through a three-level architecture of "data preprocessing, model fine-tuning training, and joint invocation output," enabling "generate first, then discriminate." By predictively generating possible future network traffic sequences and performing real-time analysis and anomaly detection on these predicted sequences, the defense gate is moved forward, transforming passive response into proactive early warning. This method can provide early warning before actual attack traffic arrives, significantly improving the real-time performance and reliability of industrial internet network security protection. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the industrial internet network intrusion detection method based on a pre-trained large language model of the present invention.

[0044] Figure 2 This is a schematic diagram of the model module of the industrial internet network intrusion detection method based on a pre-trained large language model of the present invention;

[0045] Figure 3 This is a loss curve diagram of the training of the generative model GPT-2 in this invention;

[0046] Figure 4 This is a loss curve diagram of the DistilBERT classification model trained in this invention;

[0047] Figure 5 This is a confusion matrix diagram of the multi-class test set of the DistilBERT classification model in this invention. Detailed Implementation

[0048] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0049] Reference Figure 1 and Figure 2 This invention provides an industrial internet network intrusion detection method based on a pre-trained large language model, mainly consisting of four steps: data preprocessing, fine-tuning training of the traffic generation model GPT-2, fine-tuning training of the traffic classification model DistilBERT, and joint invocation of a prediction framework to output prediction results. The data preprocessing step is responsible for processing the raw network traffic dataset into data suitable for input to both the generation model GPT-2 and the classification model DistilBERT. The preprocessed data is used to train the generation model GPT-2 and the classification model DistilBERT in parallel during the fine-tuning training step. Finally, the pre-trained generation model GPT-2 and the classification model DistilBERT are jointly invoked to form a prediction framework. This framework enables the real-time generation of predicted data packets for the next time stamp based on a limited number of real historical data packets captured at the current time stamp in an industrial internet scenario. It also performs anomaly classification on the traffic sequences in the generated predicted data packets and outputs the prediction results. This method aims to construct a proactive prediction framework of "generate first, then judge," transforming passive defense into proactive early warning. The specific steps of the method are as follows:

[0050] Step S1, data preprocessing: protocol adaptive flow aggregation and session segmentation are performed on the original network traffic data, five-tuple information including source IP address, destination IP address, source port, destination port, protocol type and timestamp are constructed, and the original network traffic data is aggregated based on the five-tuple information and the timestamp to obtain an aggregated data set; then, feature screening and feature coding are performed on the aggregated data set, and a flow format text data set for training the generation model GPT-2 and a packet level classification text-label data set for training the classification model DistilBERT are respectively constructed;

[0051] The purpose of the data preprocessing is to extract features from the original network traffic data in PCAP file format (binary format) captured in the industrial internet environment, convert the original network traffic data into network traffic data in CSV format, and clean, aggregate and texturize the network traffic data in CSV format to convert it into a data format suitable for training model input. The data preprocessing method of the present application adopts a double-path processing strategy to construct data sets suitable for two different downstream tasks: one is a flow format text data set for training the generation model GPT-2, and the other is a packet level classification text-label data set for training the classification model DistilBERT. The data preprocessing method is suitable for the complex and heterogeneous traffic background of the industrial internet scenario. The protocol adaptive flow aggregation and session segmentation, feature screening and feature coding steps in the method can efficiently process complex traffic data containing Modbus protocol, HTTP protocol, MQTT protocol, TCP protocol, UDP protocol, ICMP protocol, ARP protocol and other industrial communication protocols, and prepare for the subsequent model to fully learn the pattern of the complex traffic of the industrial internet.

[0052] Specifically, the data preprocessing includes the following sub-steps:

[0053] Step S11, protocol adaptive flow aggregation and session segmentation: This step is oriented to the flow level traffic generation task. Since the original network traffic data of the industrial internet is usually a packet level feature set (a set of features aggregated in units of packets), the present application first infers and constructs the normalized five-tuple information (source IP address, destination IP address, source port, destination port, protocol type) and timestamp from the dispersed feature fields in the original network traffic data through protocol inference, IP address normalization, L4 port construction, and timestamp analysis, and then sorts the five-tuple information to generate a uniform session key Key as a session identifier; based on the session key Key, the packets of the original network traffic data are aggregated and grouped, and the packets with the same session key Key are grouped into the same data group and aggregated into a network flow containing bidirectional communication content; a protocol adaptive session timeout threshold is defined, and the difference between the timestamp of each packet in the network flow and the timestamp of the previous adjacent packet is calculated , according to the difference , the network flow is segmented into sessions, and the packets with a difference greater than the preset session timeout threshold of the corresponding protocol type are determined as the start of a new session, and each session is assigned a session segment number S segment , the network flow is segmented into multiple sessions; finally, each unique session defined by the session key Key and the session segment number S segment is assigned a globally unique flow ID, and a network flow defined by the flow ID is constructed , and an aggregated data set is obtained. The specific construction method steps are as follows:

[0054] Step S111, Protocol Inference: Since the original network traffic data of industrial internet comes from various sensor devices, there is no uniform "protocol" field in the data set, so the present application infers the protocol type Proto of each data packet in the original network traffic data by checking the existence and validity of specific related fields in different protocol types. The inference logic follows the following priority order: 1. If the related fields of TCP protocol, i.e. TCP source port field (tcp.srcport field) or TCP destination port field (tcp.dstport field), exist and have valid values, the protocol type is determined as TCP protocol, and Proto = TCP is assigned; 2. If the TCP condition is not met (i.e. the related fields of TCP protocol do not exist), but the related fields of UDP protocol, i.e. UDP port field (udp.port field), exist and have valid values, the protocol type is determined as UDP protocol, and Proto = UDP is assigned; 3. If the TCP and UDP conditions are not met (i.e. the related fields of TCP protocol and UDP protocol do not exist), but there is a valid value of ICMP protocol related field (such as ICMP check field icmp.checksum) in the data row, the protocol type is determined as ICMP protocol, and Proto = ICMP is assigned; 4. If the above conditions are not met, but there is a valid value of ARP protocol related field (such as ARP operation code field arp.opcode), the protocol type is determined as ARP protocol, and Proto = ARP is assigned; 5. Otherwise, the protocol type is determined as OTHER protocol, and Proto = OTHER is assigned.

[0055] Step S112, IP Address Normalization: Source IP address and destination IP address are mainly extracted from IP source host field (ip.src_host field) and IP destination host field (ip.dst_host field). To ensure correct aggregation when IP information is missing, the present application uses the source IPv4 address field (arp.src.proto_ipv4 field) and the destination IPv4 address field (arp.dst.proto_ipv4 field) of the ARP protocol to backfill the missing values of source IP address and destination IP address . At the same time, all IP addresses are cleaned to remove invalid entries such as 0.0.0.0 or null values.

[0056] Step S113, L4 Port Construction: Source port and destination port The construction depends on the protocol type inferred in step S111. If Proto = TCP, the source port and the destination port are obtained from the TCP source port field (tcp.srcport field) and the TCP destination port field (tcp.dstport field), respectively; if Proto = UDP, since only a single UDP port field (udp.port field) is usually provided in the original network traffic data, the present application uniformly maps the UDP port field to the destination port and sets the source port to 0; if Proto is ICMP, ARP or OTHER, both the source port and the destination port are set to 0.

[0057] In step S114, the timestamp of the data packet is extracted from the timestamp field (frame.time) and uniformly parsed into a high-precision second-level floating-point number format for subsequent sorting and timeout judgment.

[0058] After the above steps are completed, the five-tuple information and the timestamp of all data packets are obtained.

[0059] In step S115, flow aggregation and session segmentation:

[0060] To ensure that the bidirectional data packets of the same session can be aggregated into the same flow, the present application constructs a direction-independent session key. The IP address pair (source IP address, destination IP address) and the port pair (source port, destination port) in each data packet and the protocol type are normalized and sorted to generate a uniform session key Key as a session identifier. The session key Key is defined as: , wherein represents the minimum value of the two elements, represents the maximum value of the two elements;

[0061] Through the above steps, a unique session identifier for bidirectional data flow is established, ensuring that the uplink and downlink data packets in the same session are marked as the same logical entity, thereby providing an indexing basis for subsequent aggregation of discrete and chaotic original data packets into ordered network flows.

[0062] Considering the differences in session characteristics of different protocols (such as TCP long connections and UDP short queries) in the industrial internet, the present application adopts a protocol-adaptive session timeout threshold The session timeout threshold of TCP protocol is set to 120 seconds according to the experience of industrial scene The session timeout threshold of UDP protocol is set to 60 seconds The session timeout threshold of ICMP protocol is set to 30 seconds The session timeout threshold of ARP protocol is set to 10 seconds The session timeout threshold of OTHER protocol is set to 10 seconds .

[0063] Then, the data packets of the original network traffic data are aggregated into groups based on the direction-independent session key Key, and all data packets with the same session key Key are classified into the same group (i.e., aggregated into a network flow containing bidirectional communication contents), and the data packets in each group are sorted in ascending order according to the timestamp The difference between the timestamp of the i-th data packet and the timestamp of the adjacent data packet (i.e., the (i-1)-th data packet) is calculated , , If the difference is greater than the preset session timeout threshold of the corresponding protocol type Proto, it is determined that the data packet with the timestamp corresponds to the start of a new session, and the data packet with the timestamp is assigned a new session segment number S segment The network flow is divided into multiple session segments (i.e., the long network flow is divided into multiple short network flows ), and the reorganization from packet-level data to flow-level data is finally completed.

[0064] Finally, a globally unique flow ID is assigned to each unique session (i.e., each network flow after division ) defined by the session key Key and the session segment number S segment , and a network flow defined by the flow ID is constructed to obtain an aggregated data set for subsequent flow-level data division and modeling. The formula of the flow ID is: , wherein Flow_ID represents the flow ID, MD5 is a hash function used to ensure the uniqueness of the flow ID.

[0065] Step S12, feature screening and feature coding: In order to reduce data noise, reduce model training overhead and improve the effectiveness of features, the flow data in the aggregated data set obtained in step S11 is subjected to feature screening and dimension reduction. The specific steps include:

[0066] Step S121, eliminate unstructured load features: Eliminate the load class fields and unstructured text fields of the network flow data in the data set; Specifically, it includes HTTP file data field (http.file_data), TCP payload field (tcp.payload), MQTT message body field (mqtt.msg), MQTT protocol name field (mqtt.protoname) and HTTP request full URI field (http.request.full_uri). The content of the above fields is usually composed of binary stream without fixed format, text of arbitrary length or random string, which lacks fixed semantic structure and has too large value space. Directly used for model training will cause dimension disaster or overfitting, so it is eliminated.

[0067] Step S122, eliminate single-value constant features: Calculate the number of unique values of each feature column in the data set, and delete the constant value feature column with unique value number of 1 (such as columns with all null values or numerical values completely the same). Such features are the same in all samples and have no discrimination.

[0068] Step S123, eliminate sparse features: Set a threshold of zero value proportion =75%, calculate the proportion of zero value or null value in numerical features. Eliminate numerical features with zero value proportion higher than the threshold .

[0069] Step S124, numerical coding for discrete features and category features: Traverse all feature columns in the data set, identify non-numeric feature columns with data type of string object (Object) or category (Category) (including source IP address, destination IP address, source port, destination port, protocol type), and map them to unique integer indexes, thereby converting all features to numerical format.

[0070] After the above feature screening and feature coding, the remaining feature columns (D dimensions) form the screened data set, which is used for subsequent construction of the flow format text data set.

[0071] Step S13, construct a flow format text data set for training the generated model GPT-2: First, in order to ensure the accuracy of model generalization ability evaluation and prevent the same network stream data packets from crossing the data set to cause information leakage, mutual partitioning based on flow ID is performed. Obtain all flow IDs generated in step S11 to form a set , the collection is divided into four mutually exclusive ID subsets in the ratio of 7:1:1:1 after random shuffling, and the network flow in the data set screened in step S12 is classified according to the flow ID in the four mutually exclusive ID subsets , respectively, and grouped to construct a flow-level mutually exclusive training set, a validation set, a test set and a generation set.

[0072] Subsequently, each network flow in each data set is subjected to text serialization construction. The data packets in the network flow are arranged in chronological order, and all feature values in each data packet are concatenated into a text string , for example , D is the feature dimension of the data set screened in step S12.

[0073] Finally, the text strings of all data packets in the same network flow are connected with line breaks in chronological order, and a flow start marker is added at the start of the network flow <bos>at the end of the network stream adding a stream end marker at the end position of the stream <eos>Each network flow is finally converted into a long text sequence in the format: <bos>[newline] [newline] ... [newline] [newline] <eos>This involves using mutually exclusive training, validation, and test sets, as well as all network streams in the generator set. Converting it into a long text sequence yields the streaming text dataset used to train the generative model GPT-2.

[0074] Step S14: Construct a packet-level classification text-label dataset for training the DistilBERT classification model. This step targets the packet-level anomaly classification task, constructing classification samples on a per-data packet basis, using the same feature selection criteria and numerical encoding method as in Step S12. All feature values ​​from each data packet are... Concatenate into a text string ,For example The text string is constructed in the same way as the network stream defined in step S13. Text string of internal data packet Maintain consistency. Simultaneously, map the "Attack_type" feature column from the original network traffic data to multi-category label IDs. For each text string... Together with the corresponding tag ID, they form a package-level classification sample, ultimately constructing a sample containing text strings. The package-level sample dataset contains two features: the label ID and the label itself. The package-level sample dataset is then stratified and divided into a training set, a validation set, and a test set in an 8:1:1 ratio. This dataset is then saved as a package-level classification text-label dataset in CSV format for training the DistilBERT classification model.

[0075] Step S2: Fine-tune the training of the generative model GPT-2: Load the pre-trained weights of the generative model GPT-2, add special control tokens, and adjust the dimension of the token embedding layer of the generative model GPT-2. Train the model on the streaming text dataset obtained in step S13 with the goal of Causal Language Modeling (CLM) to obtain a traffic generation model that can predict the next time (i.e., the next timestamp) of data packets based on historical window data packets.

[0076] The purpose of fine-tuning the generative model GPT-2 is to use the pre-trained generative model GPT-2 to fine-tune the training on the streaming text dataset constructed in step S1, so that it can predict the next timestamp of a single data packet based on the traffic sequence in the historical window data packets within a network flow, thereby realizing the proactive generation and simulation of potential attack traffic in the industrial Internet scenario.

[0077] In the industrial internet environment, network communication usually follows the strict mode of specific industrial protocols (such as Modbus protocol, MQTT protocol, etc.), including the unique communication rules between devices composed of programmable logic controllers (PLC), supervisory control and data acquisition systems (SCADA), and various sensors. The generated model GPT-2 trained by the present application aims to deeply learn and reproduce these complex and time-dependent communication patterns.

[0078] The generated model GPT-2 used in the present application is a Transformer architecture containing a word segmenter (BPE segmenter), a decoder (Decoder-only), and an output layer (language modeling head). Through causal masking (CausalMasking), it ensures that only historical sequences are relied on during traffic generation. The decoder contains an embedding layer and N-layer Transformer decoder blocks. The embedding layer includes a token embedding layer and a position embedding layer. The internal structure of the Transformer decoder block is mainly composed of a first layer normalization layer, a multi-head self-attention mechanism (MHA), a first residual connection, a second layer normalization layer, a feed-forward neural network (FFN), and a second residual connection. Among them, the multi-head self-attention mechanism (MHA) is responsible for capturing long-distance dependencies in the sequence, and the feed-forward neural network (FFN) is responsible for nonlinear transformation and enhancement of features. This framework is particularly suitable for traffic generation tasks in the industrial internet scenario, because network flows in the industrial internet scenario often have strong time series and context correlation. The multi-head self-attention mechanism can efficiently capture long-distance dependencies, and the autoregressive property of the multi-head self-attention mechanism supports packet-by-packet prediction based on limited historical packets, thereby realizing a model that can effectively predict and generate future possible network traffic sequences. The specific implementation steps of the fine-tuning training of the generated model GPT-2 are as follows:

[0079] Step S21, initialization of the generated model GPT-2 and the BPE segmenter:

[0080] Before fine-tuning the generative model GPT-2, the pre-trained model components need to be loaded first. Specifically, the pre-trained weight file and configuration file of the generative model GPT-2 are downloaded from the public pre-trained model library Hugging Face and saved to a preset local file path. Then the pre-trained weight of the generative model GPT-2 (model GPT-2LMHeadModel) and the corresponding BPE tokenizer GPT-2Tokenizer are loaded from the local file path. The model GPT-2LMHeadModel is a GPT-2 model with a language model head, suitable for text generation tasks; the BPE tokenizer GPT-2Tokenizer is responsible for converting the text data packet string into a numerical ID sequence that the model can understand, and decoding the ID sequence output by the model into text.

[0081] Step S22, special vocabulary expansion and embedding layer adjustment:

[0082] In order to enable the generative model GPT-2 to understand the structure of the network stream, the vocabulary of the tokenizer is expanded in this step. On the basis of the original BPE tokenizer GPT-2Tokenizer vocabulary, three special control tokens are added: stream start token <bos>, stream end marker <eos>and fill indicia <pad>These markers are used to indicate the beginning and end of a network flow, respectively, and to unify the sequence length in batch training. They enable the model to convert continuous network flow data into "sentence" units with clear boundaries, thus better learning the structure and regularity of communication sessions.

[0083] After expanding the vocabulary of the BPE tokenizer, the weight matrix dimension of the Token Embedding Layer of the model needs to be adjusted synchronously to match the vocabulary dimension size after the addition of the control token. In the subsequent fine-tuning process, the weight matrix will be continuously optimized, enabling the GPT-2 model to understand and utilize the newly added control token to extract specific semantic functions in the context of network flow.

[0084] Step S23, data reading and tokenization:

[0085] Step S231, obtain all network flow long text sequences from the flow format text dataset constructed in step S13;

[0086] Step S232, use the BPE tokenizer to tokenize each network flow long text sequence, decompose the text string in the long text sequence into a series of vocabulary units, and map each vocabulary unit to its corresponding semantic numerical ID in the vocabulary table to generate an input token index sequence (input_ids). At the same time, generate the corresponding initial input attention mask sequence (attention_mask) according to the input token index sequence, and set the mask value of the initial input attention mask sequence corresponding to all valid tokens to 1.

[0087] Step S233, for all input token index sequences and initial input attention mask sequences in the current training batch, according to the length of the longest sequence in the batch, fill the end of the shorter sequences: use the padding marker <pad>padding the end of the input token index sequence (i.e. padding tokens <pad>Fill in stream end marker <eos>corresponding tail padding (corresponding padding tokens <pad>the position of the last token in the sequence) to ensure that all sequences in the current batch have the same length.

[0088] At step S234, the prediction target sequence for computing the loss function of the generative model GPT-2 is constructed. Specifically, first, a copy of the padded input token index sequence obtained at step S233 is made as the initial prediction target sequence. Subsequently, the prediction target sequence is filtered using the padded input attention mask sequence obtained at step S233 to remove the padding tokens in the prediction target sequence that correspond to the padding tokens in the padded input token index sequence. <pad>The value of the position of the number (i.e. the value of the mask value is 0) is replaced with a specific ignore identification value (such as -100). The processed prediction target sequence is input into the generation model GPT-2 together with the input attention mask sequence and the input token index sequence filled in step S233 as an input sequence, and the misplacement prediction (i.e. predicting the next token based on the current token) is automatically performed by the internal mechanism of the generation model GPT-2. The specific ignore identification value ensures that the padding part does not participate in gradient calculation.

[0089] Step S24, token embedding and position embedding:

[0090] Step S241, token embedding. Each token index in the input token index sequence generated in step S232 is mapped to a token embedding vector , obtaining a high-dimensional token embedding vector sequence , , . Wherein, is the position index of the vocabulary unit in the input token index sequence, , represents the lookup mapping operation of the model embedding layer, that is, the discrete index is mapped to the corresponding vector by table lookup, is the length of the input sequence in the current batch, is the token embedding vector.

[0091] Step S242, position embedding. In order to enable the generation model GPT-2 to have the ability to identify the time sequence characteristics of the flow sequence data, a learnable position embedding layer needs to be introduced. First, generate the corresponding position index for each token at position 0 to in the input sequence, and map these position indexes to a fixed-dimensional position embedding vector through the position embedding layer. Finally, add the position embedding vector and the token embedding vector sequence obtained in step S241 element by element, to obtain an input representation that integrates semantic information and time sequence position information : ;

[0092] Step S25, the Transformer decoder block of the generation model GPT-2 is calculated. The core of the generation model GPT-2 is composed of a plurality of stacked Transformer decoder blocks. Unlike the traditional Transformer structure, the Transformer decoder block in the generation model GPT-2 adopts the architecture of the pre-layer normalization layer, that is, the layer normalization processing is performed before entering the multi-head self-attention mechanism and the feed-forward neural network. Each Transformer decoder block sequentially includes a first layer normalization layer, a multi-head self-attention mechanism (MHA), a first residual connection, a second layer normalization layer, a feed-forward neural network (FFN), and a second residual connection, wherein the multi-head self-attention mechanism and the feed-forward neural network are both dimensionally mapped and transformed through linear layers. For the first layer Transformer decoder block, the specific calculation method is as follows:

[0093] Step S251, first layer normalization and projection: first, the output representation of the first layer Transformer decoder block is subjected to the first layer normalization to obtain the first layer normalized state representation : , wherein represents the layer normalization operation.

[0094] Subsequently, the first layer normalized state representation is input to the multi-head self-attention mechanism. Through the linear transformation matrix of the linear layer in the multi-head self-attention mechanism, the first layer normalized state representation is projected into the query matrix , the key matrix and the value matrix : ;

[0095] Step S252, masked self-attention calculation: for each head of the multi-head self-attention mechanism, the attention weight is calculated: , wherein represents the weight of single-head attention, represents the normalized exponential function, represents the query matrix, represents the transpose of the key matrix, is the dimension of the key matrix (used to scale the dot product), represents the causal mask matrix, The value matrix is represented. In order to ensure the autoregressive property of the generated model GPT-2, a causal mask matrix is introduced , which ensures that the position can only pay attention to the information at position and before it.

[0096] Step S253, multi-head splicing and linear projection: in the multi-head self-attention mechanism, the self-attention mechanism is applied to multiple "heads" in parallel, each head independently calculates attention, then splices the outputs of all heads, and projects through an output linear layer to obtain the final output representation of the multi-head self-attention mechanism (MHA) The formula of the multi-head self-attention mechanism is: , In the formula, denotes the vector splicing operation, denotes the output result of the first attention head, denotes the total number of attention heads, denotes the output result of the th attention head, denotes the linear projection weight matrix of the output linear layer of the multi-head self-attention mechanism, which is used to map the spliced multi-head features back to the hidden layer dimension of the model. This multi-head self-attention mechanism allows the model to learn information from different representation subspaces and pay attention to the dependencies of different positions in the sequence.

[0097] Step S254, first residual connection and second layer normalization: the output representation obtained after the multi-head self-attention mechanism calculation is connected with the input representation before entering the multi-head self-attention mechanism to obtain the intermediate state representation : ,

[0098] Then the second layer normalization is performed on the intermediate state representation to obtain the second layer normalized state representation , and the calculation formula is as follows: ,

[0099] The second layer normalized state representation is used as the input of the feedforward neural network;

[0100] Step S255, feedforward neural network calculation: the feedforward neural network is a position-shared fully connected network composed of two linear transformation layers, which is used to perform nonlinear transformation on the features, and the specific steps are as follows:

[0101] The first step is to expand the feature dimension through the first linear transformation layer and pass it through the Gaussian Error Linear Unit (GELU) activation: ,

[0102] The second step is to project the feature dimension back to the original dimension through the second linear transformation layer: ,

[0103] where, is the intermediate activation state of the hidden layer (the output of the first linear transformation layer), represents the Gaussian Error Linear Unit, and are the weight matrix and bias of the first linear transformation layer, respectively, and are the weight matrix and bias of the second linear transformation layer, respectively, is the output of the second linear transformation layer, which is the final output of the feedforward neural network.

[0104] Step S256, second residual connection and decoder output: the output of the feedforward neural network is residual connected with the intermediate state representation to obtain the final output representation of the layer Transformer decoder block : ,

[0105] The output representation will be the input of the layer Transformer decoder block until the output representation of the decoder is obtained by passing through the N-layer Transformer decoder block.

[0106] Step S257, language modeling head and logits generation: after processing through the N-layer Transformer decoder block, the final hidden state representation output by the model (i.e., the state representation after the second layer normalization in the last layer of the Transformer decoder block) is passed to the language modeling head. The language modeling head is a linear layer that projects the output representation of the Nth layer of the Transformer decoder block to the dimension size of the augmented BPE tokenizer vocabulary, obtaining the logits. The calculation formula of the language modeling head is as follows: , where Logits represents the logits output by the language modeling head, The weight matrix represents the language modeling head. The bias term representing the language modeling head;

[0107] Step S258, Probability Distribution Calculation: Apply the normalized exponential function (Softmax function) to the log-odds vector (Logits) obtained in step S257 to obtain the probability distribution of each word in the vocabulary as the word of the next timestamp: , In the formula, Indicates in a given historical sequence arrive Under the conditions, the first The meta-index of words is The conditional probability, This represents the log-odds vector corresponding to the output at the end of the historical sequence. This is the normalization exponential function, used to convert numerical values ​​into probability values ​​that sum to 1.

[0108] Step S26, Optimization of loss function and generative model GPT-2:

[0109] For including Network flow The stream-level training set is defined as the first stream obtained after word segmentation in step S232. Network flow The total length of the converted word sequence is Cross-entropy loss function modeled using causal language Optimize the generative model GPT-2 using the cross-entropy loss function. The calculation formula is: , In the formula, For the first Network flow The Middle The actual words at each position, where log represents the logarithmic function. The representation model is based on a given historical lexical sequence. Predict the conditional probability of the word at the current position under the given conditions. This represents all learnable parameters of the generative model GPT-2 (including lexical embedding parameters, positional embedding parameters, decoder block weights, and language modeling head weights).

[0110] During training, the cross-entropy loss function is minimized. It can drive the generation of all parameters of the GPT-2 model. The optimization makes the model more accurate for subsequent data packets that actually occur (corresponding to real words). ) can give higher prediction condition probability , so that the model learns the deep time sequence rules and patterns contained in the industrial internet traffic data.

[0111] The calculated loss is propagated back to the model through the back propagation algorithm, and the weights of the model are updated. The optimizer AdamW combines the cosine scheduler to adjust the learning rate, so as to realize the effective update of the model parameters.

[0112] After training, the weight parameters of the fine-tuned generation model GPT-2 and the updated BPE tokenizer configuration are saved to the specified path, and the trained generation model GPT-2 is obtained as the final traffic generation model used to generate the predicted traffic sequence.

[0113] Step S3, fine-tuning training of the classification model DistilBERT:

[0114] Load the pre-training weights of the classification model DistilBERT, construct a task architecture for single packet multi-classification, train on packet-level classification samples using the Focal Loss loss function and early stopping strategy, and obtain a classification model capable of packet-by-packet anomaly discrimination.

[0115] The classification model DistilBERT includes a WordPiece tokenizer, an encoder, and an output layer. The encoder of the classification model DistilBERT includes an embedding layer and N-layer bidirectional Transformer encoder blocks. The purpose of fine-tuning the classification model DistilBERT is to train an efficient and accurate multi-classification model for anomaly discrimination of single network traffic data packets. The classification model needs to be able to distinguish between normal traffic packets and potential attack traffic packets. To meet the requirements of real-time performance and resource efficiency in the industrial internet environment, the present application selects a packet-level anomaly classification model DistilBERT. The packet-level anomaly classification model DistilBERT is a distilled and compressed version of the BERT model, which has fewer parameters and faster inference speed, while maintaining comparable performance to the original BERT model in various downstream tasks, making it effective for deployment on edge computing devices or resource-constrained industrial control hosts. The specific implementation steps are as follows:

[0116] Step S31, initialization of the classification model DistilBERT and the WordPiece tokenizer:

[0117] First, download the pre-trained weight file and configuration file of the DistilBERT classification model from the publicly available pre-trained model library Hugging Face, and save them to a preset local file path. Then, load the pre-trained classification model DistilBERT and its corresponding WordPiece word segmenter from the local file path. To adapt to the multi-class classification task of this invention, when loading the model, the number of output categories of the fully connected classification head of the model is set to the total number of categories in the classification labels, and each category is represented by a mapped value, thus completing the model initialization.

[0118] Step S32, Lexical Expansion and Embedding Layer Adaptation:

[0119] To meet the requirements of classification tasks, specific control units are configured in the WordPiece word segmenter. These control units mainly include the classification marker [CLS], the separator marker [SEP], and the padding character [PAD]. The classification marker [CLS] is added at the beginning of each sequence to aggregate global semantic information of the entire data packet for subsequent classification; the separator marker [SEP] is added at the end of the sequence to indicate the end of the text; and the padding character [PAD] is used to standardize the sequence length.

[0120] Step S33, Construction of the classification dataset:

[0121] This step uses the packet-level text-label dataset in CSV format constructed in step S14. For each packet sample, the text string constructed in step S14 is segmented using the WordPiece tokenizer. Convert the classification input tensor to the model, including the classification term index sequence and the classification attention mask sequence. Set the maximum sequence length in the classification input tensor. For text strings exceeding the specified length Truncate text strings that are shorter than the specified length. Fill with the filler symbol [PAD] as described in step S32.

[0122] Step S34, Embedding layer calculation:

[0123] The embedding layers of the DistilBERT classification model mainly consist of a word embedding layer and a positional embedding layer. The computational principles of the word embedding layer and the positional embedding layer are consistent with those of the embedding layer in the generative model GPT-2 described in step S24. They are used to map the discrete word indices in the classification input tensor into vector representations and to capture the positional information of the sequence, respectively. After fusing semantic and positional information, the classification input representation is obtained. It is worth noting that, to achieve lightweight design and efficient inference, the DistilBERT classification model removes the segmentation embedding layer from the original BERT model.

[0124] Step S35, bidirectional Transformer encoder block:

[0125] Each bidirectional Transformer encoder block contains multi-head bidirectional self-attention mechanism, feedforward neural network, layer normalization layer and residual connection. The calculation logic of the layer normalization layer, feedforward neural network and residual connection inside is consistent with the Transformer decoder block of the generation model GPT-2 described in step S25. The key difference between the bidirectional Transformer encoder block and the Transformer decoder block in step S25 is the visible domain of the attention mechanism. The generation model GPT-2 uses causal mask and can only focus on historical information. While the classification model DistilBERT uses bidirectional attention mechanism and does not use causal mask matrix when calculating attention weight, so that any position of the word in the sequence can focus on all word in the context. This global context awareness is crucial for classification tasks, because it allows the model to fully understand the semantics of the entire data packet text to make accurate judgments, so that the model can obtain a hidden state vector containing the complete semantics of the data packet.

[0126] Step S36, fully connected classification head output:

[0127] This is the essential difference between the classification model DistilBERT and the generation model GPT-2 at the output end. The generation model GPT-2 uses a language modeling head to project the output to the vocabulary space to predict the next word. While the classification model DistilBERT uses a sequence fully connected classification head to project the features to a specific dimension of the class space. Specifically, the final hidden state vector output by the last bidirectional Transformer encoder block of the classification model DistilBERT is taken as the semantic aggregation representation of the entire data packet sequence, which is input to a linear classification layer. The vector dimension is mapped from the model hidden layer dimension to the target classification number (i.e. the total number of classification labels), and then the Softmax function is used to output the probability distribution of the data packet belonging to each classification label.

[0128] Step S37, Focal Loss loss function:

[0129] In order to effectively solve the class imbalance problem commonly existing in industrial internet traffic data, the Focal Loss loss function is used to calculate the classification loss. The Focal Loss loss function is an improvement of the standard cross-entropy loss function. It introduces a modulation factor , dynamically reduces the contribution weight of a large number of easily classified samples to the loss calculation. This enables the classification model to pay more attention to the few-class samples that are difficult to classify during the training process, which is very effective for handling class-imbalance datasets. The specific calculation formula of the Focal Loss loss function is as follows: , wherein, represents the value of the Focal Loss loss function, is the prediction probability of the model for the true class, is the modulation factor (set to in this embodiment). When the prediction probability tends to 1 (representing that the sample belongs to an easily classified sample), the weight tends to 0, thereby significantly reducing the contribution of the sample to the total loss, so that the model focuses on difficult-to-classify samples during optimization, improving the classification performance of the model on imbalanced data.

[0130] Step S38, model training and early stopping strategy:

[0131] The AdamW optimizer is used to calculate the loss function gradient and update the parameters of the classification model DistilBERT through the backpropagation algorithm . During the training process, an early stopping strategy is introduced to select the optimal model. Specifically, after each training round, the performance of the model is evaluated on an independent validation set. If the loss of the validation set fails to decrease for a number of consecutive periods (in this embodiment, the threshold is set to 2 consecutive periods), the training process is terminated in advance. This effectively prevents overfitting of the model on the training set and ensures that the selected model has the best generalization performance. Finally, the model weight parameters of the period with the lowest validation set loss are saved, obtaining the trained classification model DistilBERT, which is used as the final classification model for anomaly discrimination.

[0132] Step S4, joint calling of the prediction framework to output the prediction result: taking the first L packets (L=3 in this example) of the real network flow as the initial history window (i.e., the sliding window), using the trained generation model GPT-2 in step S2 and the trained classification model DistilBERT in step S3 to form a prediction framework, first calling the trained generation model GPT-2 to generate a traffic sequence, obtaining a predicted packet, then calling the trained classification model DistilBERT to perform anomaly discrimination on the sequence data in the predicted packet one by one, outputting the classification result of the anomaly discrimination, and realizing the active intrusion detection of first generation and then discrimination.

[0133] The joint calling prediction framework outputs the prediction results, which is the core of the active intrusion detection of the present application. The trained generation model GPT-2 in step S2 and the trained classification model DistilBERT in step S3 are combined to form a closed-loop workflow of "first generation and then discrimination". This step realizes the transition from traditional passive and lagging analysis to active and predictive defense. The traditional intrusion detection system analyzes the network traffic that has occurred, while the present application can identify the attack intention at the initial stage of the attack sequence or even before the malicious data packet is actually transmitted to the target network through high-probability prediction of the next time traffic. This "pre-cognitive" defense capability is the core of the present application that distinguishes it from the prior art, and provides timely and effective security response for critical infrastructure such as industrial internet. The joint calling prediction framework and outputting the prediction results specifically include the following sub-steps:

[0134] Step S41, construction and generation of real-time prompt: first, extract the first L data packets from the real network stream to be detected as initial historical data packets. The D-dimensional feature column of the L data packets filtered in step S12 is converted into a text string according to the method in step S13, and is spliced in time sequence to form an initial historical window. The initial historical window is taken as the initial input and is sent to the trained generation model GPT-2 in step S2 to start the generation process of the predicted traffic sequence.

[0135] Step S42, iterative generation: after the generation is started, an iterative loop is performed, and the last L data packets of the current existing data packets (including the initial historical data packets and the generated data packets) are always taken as the new prompt input to form the context sliding window of the iterative loop. The generation model GPT-2 generates the text string of the predicted data packet of the next timestamp based on the context sliding window of the iterative loop.

[0136] Step S43, real-time anomaly discrimination: the text string of the newly generated single predicted data packet is sent to the trained classification model DistilBERT in step S3 for real-time analysis of the predicted data packet, and the classification result is output and recorded.

[0137] Step S44, termination condition of the generation process: to ensure the effectiveness and finiteness of the generation process, the model stops the generation of the current predicted traffic sequence when one of the following conditions is met: (1) natural termination. The generation model GPT-2 outputs a special flow end marker <eos>This means that, based on the traffic patterns it has learned, the model predicts that the network session (i.e., the network flow) has ended naturally. (2) Forced termination. The number of generated packets reaches the preset maximum network flow length threshold. This preset maximum network flow length threshold is used to prevent unlimited generation due to model prediction errors or abnormal patterns, ensuring the effective utilization of system resources.

[0138] Step S45, Quality Assessment of the Generated Traffic Sequence: After the generation process is completed, in order to verify the accuracy of the model prediction, this invention also assesses the quality of the generated traffic sequence for offline performance verification of the model. Evaluation metrics include perplexity (PPL) and cosine similarity.

[0139] Step S451: Perplexity is used to measure the reasonableness of the generated traffic sequence content. The lower the perplexity (PPL) value, the higher the certainty of the model regarding the generated traffic sequence, and the more the traffic sequence conforms to the traffic pattern it has learned. This embodiment uses "average per packet perplexity" as the evaluation index, that is, the perplexity of each generated data packet is calculated separately, and then the arithmetic mean of all perplexities is calculated to obtain the average perplexity. Specifically, the formula for calculating the average perplexity is: , in, R represents the average perplexity of the generated traffic sequence, and R is the total number of packets actually generated in the current network flow. For the first The generated data packets, NLL being the negative log-likelihood, correspond to the cross-entropy loss of the model output ( ), This is for exponential function operations.

[0140] Step S452: Cosine similarity is used in the model validation phase to measure the similarity of semantic features between the generated predicted data packets and the real data packets. It is obtained by calculating the cosine similarity between the hidden state vectors output by the last layer of the model. The formula for calculating cosine similarity is as follows: , in, Represents cosine similarity. and Generate data packets respectively and real data packets After alignment, the first The hidden state vector of each word. The length of the aligned flow sequence.

[0141] To verify the effectiveness of the method, the network intrusion detection public data set Edge-IIoTset is selected for experiment. The Edge-IIoTset data set simulates a typical industrial scene under the edge computing architecture, collects data combined with various sensors, control devices and communication protocols, covers network traffic and system logs from normal working conditions to multiple types of malicious behaviors, and contains 14 attack categories, including normal traffic (Normal), ICMP distributed denial of service attack (DDoS_ICMP), SQL injection attack (SQL_injection), vulnerability scanning (Vulnerability_scanner), HTTP distributed denial of service attack (DDoS_HTTP), password brute force cracking (Password), TCP distributed denial of service attack (DDoS_TCP), malicious upload (Uploading), backdoor attack (Backdoor), port scanning (Port_Scanning), cross-site scripting attack (XSS), UDP distributed denial of service attack (DDoS_UDP), ransomware (Ransomware) and fingerprinting (Fingerprinting). Compared with traditional Internet of Things data sets, the Edge-IIoTset data set is more representative and valuable in terms of attack diversity, industrial protocol coverage and edge node behavior modeling.

[0142] In this embodiment, 20% of the data in the Edge-IIoTset data set is selected for experiment. The statistical results of the sample number distribution of each attack category in the Edge-IIoTset data set are shown in Table 1, and the data set is divided according to the data preprocessing steps described in step S1.

[0143] Table 1: Attack category sample number distribution

[0144] According to the method, the experiment mainly includes two tasks: task one is the fine-tuning of the traffic generation model corresponding to step S2, which aims to train the generation model GPT-2 to learn the sequence generation rules of industrial internet traffic; task two is the fine-tuning of the anomaly classification model corresponding to step S3, which aims to train the classification model DistilBERT to accurately determine whether the data packet is abnormal.

[0145] For the fine-tuning training of the generation model GPT-2 of task one, the fine-tuning training epoch is set to 5, the batch size is set to 8, and the gradient accumulation step is set to 1. The initial learning rate is set to , and cosine learning rate scheduling is used. During the training process, the step is saved according to the log, and the training and validation loss curves are output, as shown in Figure 3 .

[0146] By Figure 3 It can be seen that the training loss and validation loss of the generated model GPT-2 are steadily decreased with the increase of the training step number, and finally converge, and there is no overfitting phenomenon of significant rebound of the validation loss, indicating that the model effectively learns the sequence generation rule of the industrial internet traffic data. This result verifies the feasibility of the "generation" link of the traffic sequence of the present application.

[0147] For the fine-tuning training of the classification model DistilBERT of task two, the training epoch is set to 4, the batch size is set to 8, and the gradient accumulation is set to 4. The initial learning rate is set to , and the cosine learning rate scheduling is also used. In order to solve the problem of data imbalance existing in the industrial scene and enhance the learning ability of the model to difficult samples, the Focal Loss loss function is used for the loss function, and the modulation factor is set to 2.0. The early stopping strategy is also used in the training, and the precision, recall, F1 and confusion matrix are used as the main evaluation indexes. The experimental results of this task are shown in Figure 4 、 Figure 5 and Table 2.

[0148] Table 2: Evaluation indexes of the classification model DistilBERT

[0149] The experimental results of the classification model are shown in Figure 4 、 Figure 5 and Table 2. According to the results of Figure 4 , the training loss and validation loss of the classification model DistilBERT are rapidly decreased and converged after the second round; at the same time, the accuracy is stably above 0.98. The model evaluation indexes of Table 2 further verify the performance of the model, and the accuracy, precision, recall and F1 score are all above 0.983. In addition, Figure 5 the confusion matrix graph of the multi-classification test set shows that most of the samples are correctly classified, and the misclassified samples on the non-diagonal line are very few (the values on the diagonal line represent the number of correctly classified samples, and the values on the non-diagonal line represent the number of misclassified samples), which indicates that the classification model DistilBERT trained by the present application has high accuracy and excellent F1 score, and can provide reliable and efficient technical support for the abnormal "determination" link.

[0150] In conclusion, the experimental results of the method verify that the generation model GPT-2 and the classification model DistilBERT can achieve the expected performance indicators on the target data set through fine-tuning training, thereby providing key support for the overall feasibility of the "first generation and then discrimination" active intrusion detection framework.< / eos> ​< / pad> < / pad> < / eos> < / pad> < / pad> < / pad> < / eos> < / bos> < / eos> < / bos> ​< / eos> < / bos>

Claims

1. An industrial internet network intrusion detection method based on a pre-trained large language model, characterized in that, The method comprises the following steps: Step S1, protocol adaptive flow aggregation and session segmentation are performed on original network flow data, five-tuple information including source IP address, destination IP address, source port, destination port and protocol type is constructed, and a timestamp is obtained, and then the original network flow data is aggregated and grouped based on the five-tuple information and the timestamp to obtain an aggregated data set; Then, feature screening and feature coding are performed on the aggregated data set, and a flow format text data set for training a generation model GPT-2 and a packet level classification text-label data set for training a classification model DistilBERT are constructed respectively; Step S2, the pre-training weight of the generation model GPT-2 is loaded, the control token is added, and the embedding layer dimension of the generation model GPT-2 is adjusted, the generation model GPT-2 is fine-tuned on the flow format text data set obtained in step S1 with the goal of causal language modeling, and a traffic generation model capable of predicting the next timestamp packet based on the historical window packet is obtained; Step S3, the pre-training weight of the classification model DistilBERT is loaded, a task architecture facing single packet multi-classification is constructed, the classification model DistilBERT is fine-tuned on the packet level classification text-label data set obtained in step S1 by using a Focal Loss loss function and an early stopping strategy, and a classification model capable of packet-by-packet anomaly discrimination is obtained; Step S4, the first L packets of a real network stream at a current timestamp are taken as an initial historical window, a prediction framework is formed by using the trained generation model GPT-2 in step S2 and the trained classification model DistilBERT in step S3, the trained generation model GPT-2 is called to generate a traffic sequence, a predicted packet is obtained, the trained classification model DistilBERT is called to perform anomaly discrimination on the sequence data in the predicted packet, a classification result of the anomaly discrimination is output, and active intrusion detection of generating first and then discriminating is realized.

2. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 1, characterized in that, The step S1 specifically comprises the following steps: Step S11, first, the original network traffic data is protocol-adaptive flow aggregation and session segmentation by protocol inference, IP address normalization, L4 port construction, timestamp analysis, to build a five-tuple information containing source IP address, destination IP address, source port, destination port, protocol type; then sort the five-tuple information, build a direction-independent session key Key as a session identifier; based on the session key Key, the data packets of the original network traffic data are aggregated and grouped, the data packets with the same session key Key are grouped into the same data group, and are aggregated into a network flow containing bidirectional communication content; define a protocol-adaptive session timeout threshold, and calculate the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet ; according to the difference , the network flow is session segmented, and the data packet with a difference greater than the preset session timeout threshold of the corresponding protocol type is determined as the start of a new session , each session is assigned a session segment number S segment , the network flow is segmented into multiple sessions; finally, each unique session defined by the session key Key and the session segment number S segment is assigned a globally unique flow ID, and a network flow defined by the flow ID is constructed , to obtain an aggregated data set; Step S12, feature screening and feature coding are performed on the aggregated data set, unstructured load features, single value constant features and sparse features are removed, discrete features and category features are numerically coded, all features are converted into numerical format, and a screened data set composed of D-dimensional feature columns is obtained; Step S13: Obtain the set of all stream IDs generated in step S11. , will set After random shuffling, the data is divided into four mutually exclusive ID subsets in a 7:1:1:1 ratio. Based on the flow IDs in the four mutually exclusive ID subsets, the network flows in the dataset filtered in step S12 are... Extract and group the data respectively to construct mutually exclusive training, validation, test, and generation sets at the stream level; then, combine each network stream in each dataset. The data packets are arranged in chronological order, and all feature values ​​in each data packet are concatenated into a text string. The same network flow The text string of all data packets Connect them using newlines in chronological order, and in the network stream. Add a stream start marker at the beginning of the network stream. Add a stream end marker at the end position of each network stream. Convert it into a long text sequence to obtain a streaming text dataset for training the generative model GPT-2; Step S14, build a classification sample in a single data packet unit, splice the features in each data packet in the filtered data set obtained in step S12 into a text string ; at the same time, map the attack category features in the original network traffic data into multi-classification label IDs; each text string and the corresponding label ID form a packet-level classification sample, and a packet-level sample data set containing two columns of features of text string and label ID is constructed, and the packet-level sample data set is stratified sampled, and the packet-level training set, validation set and test set are divided in a ratio of 8:1:1 to obtain a CSV format packet-level classification text-label data set for training the classification model DistilBERT.

3. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 2, characterized in that, In the step S11, the session key Key is defined as: ， In the formula, Proto represents a protocol type, SrcIP represents a source IP address, DstIP represents a destination IP address, SrcPort represents a source port, DstPort represents a destination port; Min represents taking a minimum value of two elements, Max represents taking a maximum value of two elements.

4. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 3, characterized in that, In the step S11, a session timeout threshold of protocol adaptation is defined, and a difference between a timestamp of each data packet in the network flow and a timestamp of a previous adjacent data packet is calculated According to the difference A specific method for session segmentation of the network flow is: According to the protocol type Proto, define the protocol adaptive session timeout threshold ; Data packets within each network flow are timestamped Sort the data packets in ascending order; traverse the sorted data packets and calculate the... timestamp of each data packet With the timestamp of each data packet The difference between , If the difference Session timeout threshold greater than the corresponding protocol type Proto Then determine the timestamp The corresponding data packet marks the start of a new session and is a timestamp. The corresponding data packet is assigned a new session segment number S segment This divides the network flow into multiple sessions.

5. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 4, characterized in that, The step S2 specifically comprises the following steps: Step S21, load the pre-training weight file of the generation model GPT-2; the generation model GPT-2 comprises a BPE tokenizer, a decoder, and an output layer; the decoder comprises an embedding layer and N layers of Transformer decoder blocks; the embedding layer comprises a token embedding layer and a position embedding layer; each Transformer decoder block is composed of a first layer normalization layer, a multi-head self-attention mechanism, a first residual connection, a second layer normalization layer, a feed-forward neural network, and a second residual connection; the output layer is composed of a language modeling head; the BPE tokenizer is used to convert a text data packet string into a digital ID sequence that can be understood by the model, and to decode the ID sequence output by the model into text; and the generation model GPT-2 is used for a text generation task; Step S22, add control tokens, expand the vocabulary of the BPE tokenizer, and adjust the weight matrix dimension of the embedding layer according to the sequence length; the control tokens include a flow start token, a flow end token, and a padding token, which are used to indicate the start and end of a network flow, and to unify the sequence length in batch training, respectively; Step S23: Use the BPE tokenizer to analyze each network stream in the streaming text dataset constructed in step S13. The long text sequence is segmented into words to separate the text strings in the long text sequence. The process is decomposed into a series of lexical units, and each lexical unit is mapped to its corresponding semantically meaningful numerical ID in the vocabulary to generate an input lexical index sequence. Based on this sequence, an initial input attention mask sequence is generated. Then, by controlling the length of all sequences, a consistent input lexical index sequence and input attention mask sequence are obtained. Next, a prediction target sequence is constructed based on the input lexical index sequence and input attention mask sequence to calculate the loss function of the generative model GPT-2. Finally, the prediction target sequence, the input lexical index sequence, and the input attention mask sequence are fed into the generative model GPT-2 as input sequences. Step S24, the word embedding layer of the generated model GPT-2 maps each word index in the input word index sequence generated in step S23 to a word embedding vector , obtaining a sequence of word embedding vectors ; definition , where L is the length of the input sequence in the current batch, and the position embedding layer generates a corresponding position index for each word in the input sequence from 0 to , and maps all the position indexes to a fixed-dimensional position embedding vector ; finally, the position embedding vector is added element-wise to the sequence of word embedding vectors , obtaining an input representation that fuses semantic information and time sequence position information ; Step S25, input the representation into the N-layer Transformer decoder block of the generation model GPT-2 for calculation to obtain an output representation of the decoder; project the dimension of the output representation of the decoder to the dimension size of the vocabulary of the extended BPE tokenizer through the language modeling head of the output layer in the generation model GPT-2 to obtain a logit vector; calculate the probability distribution of each token in the vocabulary as the token of the next time stamp according to the logit vector; Step S26, optimize the generation model GPT-2 using a cross-entropy loss function, fine-tune the generation model GPT-2, make the generation model GPT-2 learn the sequence generation rule of network traffic data packets, obtain a trained generation model GPT-2, and use the trained generation model GPT-2 as a traffic generation model for generating a predicted traffic sequence.

6. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 5, characterized in that, The step S3 specifically comprises the following steps: Step S31, load the pre-training weight file of the classification model DistilBERT; the classification model DistilBERT comprises a WordPiece tokenizer, an encoder, and an output layer; the encoder of the classification model DistilBERT is composed of an embedding layer and N layers of stacked bidirectional Transformer encoder blocks; the embedding layer comprises a token embedding layer and a position embedding layer; the output layer of the classification model DistilBERT is composed of a fully connected classification head; the output class number of the fully connected classification head is set to the total number of classification labels, and each class is represented by a mapping value; Step S32, configure specific control tokens for the WordPiece tokenizer; the specific control tokens include a classification token for aggregating global semantic information, a separation token for identifying the end of text, and a padding symbol for unifying the sequence length; Step S33: Use the WordPiece tokenizer to segment the text string constructed in step S14. The input is converted into a classification input tensor, which includes a classification term index sequence and a classification attention mask sequence. The lengths of all sequences in the classification input tensor are standardized, and text strings exceeding the maximum sequence length are excluded. Truncate text strings that are shorter than the maximum sequence length. Use fill characters to fill; Step S34, send the classification input tensor into the embedding layer of the classification model DistilBERT for processing; the token embedding layer maps the classification token index sequence into a classification token embedding vector, and the position embedding layer generates a classification position embedding vector corresponding to the classification token embedding vector; the classification token embedding vector and the classification position embedding vector are added element by element to obtain a classification input representation that fuses semantic and position information; Step S35, send the classification input representation into the N layers of stacked bidirectional Transformer encoder blocks for processing; the bidirectional attention mechanism is used to calculate global context information to obtain a hidden state vector containing the complete semantics of the data packet; Step S36, the hidden state vector output by the last bidirectional Transformer encoder block is input to the fully connected classification head of the output layer, the features are projected to the category space, and the probability distribution of the data packet belonging to each classification label is output through the Softmax function; Step S37, a Focal Loss loss function suitable for class-imbalance samples is constructed, the weight of easy-to-classify samples is reduced by introducing a modulation factor, and the classification performance of the classification model DistilBERT on imbalanced data is improved; Step S38, an AdamW optimizer is used to calculate the loss function gradient and update the parameters of the classification model DistilBERT through the back propagation algorithm, an early stopping strategy is introduced to select the optimal model, and a trained classification model DistilBERT is obtained as the final classification model for anomaly discrimination.

7. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 6, characterized in that, The step S4 specifically comprises the following steps: Step S41: First, extract the preceding data from the real network stream to be detected. The data packets are used as initial historical data packets; the D-dimensional feature columns obtained after filtering the initial historical data packets in step S12 are converted into text strings according to the method described in step S13, and concatenated in chronological order to form an initial historical window; the initial historical window is used as the initial input and fed into the generative model GPT-2 trained in step S2 to start the process of generating the predicted traffic sequence. Step S42, after starting the generation process of the predicted traffic sequence, an iterative loop is performed, always taking the last L data packets of the current existing data packets including the initial historical data packets and the generated data packets as the new prompt input, forming a context sliding window for the iterative loop; the generated model GPT-2 generates the text string of the predicted data packet of the next timestamp based on the context sliding window in an autoregressive manner; Step S43, the generated text string of the single predicted data packet is sent to the classification model DistilBERT trained in step S3, the predicted data packet is analyzed in real time, and the classification result of anomaly discrimination is output and recorded.

Images