Industrial internet network intrusion detection method based on pre-trained large language model

Abstract

The application provides an industrial internet network intrusion detection method based on a pre-trained large language model, wherein first, original network flow data is preprocessed through protocol adaptive flow aggregation and session segmentation, feature screening and feature coding to construct a flow format text data set for training a generation model GPT-2 and a packet level classification text-label data set for training a classification model DistilBERT; then, the generation model GPT-2 and the classification model DistilBERT are fine-tuned; then, the trained generation model GPT-2 is called to generate a flow sequence and obtain predicted data packets, the trained classification model DistilBERT is called to perform abnormality discrimination on the sequence data in the predicted data packets one by one, and a classification result of the abnormality discrimination is output; and active intrusion detection of first generation and then discrimination is realized. The method can make a prediction before real attack flow arrives and prevent network attacks.

Description

Technical Field

[0001] This invention relates to the field of network traffic intrusion detection, specifically to an industrial internet network intrusion detection method based on a pre-trained large language model. Background Technology

[0002] With the rapid development and widespread application of Industrial Internet technology, cybersecurity issues are becoming increasingly serious, and cyberattacks targeting industrial control systems and critical infrastructure are on the rise. Traditional network intrusion detection technologies, such as signature-based detection methods, rely on pattern matching by maintaining attack signature databases, but this method struggles to identify unknown zero-day attacks and new attack variants. While statistical anomaly-based detection methods can detect attacks that deviate from normal behavior patterns, they typically treat network traffic as independent statistical data, ignoring the temporal correlation and contextual information between traffic flows, resulting in a high false positive rate and an inability to effectively address complex, multi-stage, coordinated attacks.

[0003] To overcome the limitations of traditional methods, researchers have introduced machine learning and deep learning techniques. These methods can automatically learn and extract attack patterns from high-dimensional and complex traffic data, improving detection accuracy to some extent. However, these methods typically rely on manually designed expert features, limiting the model's ability to discover potential attack patterns. Furthermore, their performance is highly dependent on large-scale training datasets with precise labels, but in real-world network environments, obtaining high-quality labeled data is costly, and there is a widespread and severe class imbalance between attack and normal samples, limiting the model's generalization ability and robustness in real-world scenarios. Moreover, since these methods are essentially passive analyses of existing traffic, they struggle with timely warnings and early defenses, making their defensive lag particularly pronounced in scenarios with extremely high real-time and security requirements, such as the Industrial Internet.

[0004] In recent years, Large Language Models (LLMs) have demonstrated outstanding sequence modeling and contextual understanding capabilities in fields such as natural language processing, bringing new solutions to the field of network intrusion detection. By analogying network traffic data to a "language" with specific syntax and semantics, LLMs can capture complex and long-term dependencies in traffic sequences. The "pre-training-fine-tuning" paradigm of LLMs can learn general traffic patterns from massive amounts of unlabeled data, and then fine-tune them for specific tasks using a small amount of labeled data, thereby significantly reducing dependence on labeled data and effectively improving the ability to identify unknown attacks.

[0005] In summary, existing network intrusion detection technologies still focus on classifying or identifying anomalies in network traffic that has already occurred. This technological paradigm is essentially a reactive, "post-incident" defense. Its inherent analysis delay makes it difficult for the system to effectively intervene before an attack causes substantial damage. This limitation is particularly prominent in high-real-time and high-security application scenarios such as the Industrial Internet. Summary of the Invention

[0006] The purpose of this invention is to provide an industrial internet network intrusion detection method based on a pre-trained large language model, addressing the limitations of existing industrial internet network intrusion detection technologies that rely on passive analysis, have delayed responses, and struggle to handle imbalanced data in industrial internet scenarios. This invention constructs a proactive prediction framework of "generate first, then discriminate," predictively generating possible future network traffic sequences and performing real-time analysis and anomaly detection on these sequences. This aims to move the defense line forward, transforming passive response into proactive early warning, thereby achieving the goal of preventing network attacks in advance.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] The industrial internet network intrusion detection method based on pre-trained large language models includes the following steps:

[0009] Step S1: Perform protocol-adaptive flow aggregation and session segmentation on the original network traffic data, construct a five-tuple information including source IP address, destination IP address, source port, destination port, and protocol type, and obtain the timestamp. Then, aggregate and group the original network traffic data based on the five-tuple information and timestamp to obtain the aggregated dataset. Next, perform feature filtering and feature encoding on the aggregated dataset, and construct a streaming text dataset for training the generative model GPT-2 and a packet-level classification text-label dataset for training the classification model DistilBERT, respectively.

[0010] Step S2: Load the pre-trained weights of the generative model GPT-2, add control lexical units and adjust the embedding layer dimension of the generative model GPT-2. With the goal of causal language modeling, fine-tune and train the generative model GPT-2 on the streaming text dataset obtained in step S1 to obtain a traffic generation model that can predict the next timestamp data packet based on historical window data packets.

[0011] Step S3: Load the pre-trained weights of the classification model DistilBERT, construct a task architecture for single-packet multi-classification, and fine-tune the classification model DistilBERT on the packet-level classification text-label dataset obtained in step S1 using the Focal Loss loss function and early stopping strategy to obtain a classification model capable of packet-by-packet anomaly detection.

[0012] Step S4: Using the first L data packets of the real network flow at the current timestamp as the initial historical window, a prediction framework is formed using the generative model GPT-2 trained in step S2 and the classification model DistilBERT trained in step S3. First, the trained generative model GPT-2 is called to generate traffic sequences and obtain predicted data packets. Then, the trained classification model DistilBERT is called to perform anomaly detection on each sequence data in the predicted data packets and output the classification results of the anomaly detection, thus realizing proactive intrusion detection that first generates and then detects.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] Step S11: First, protocol-adaptive flow aggregation and session segmentation are performed on the original network traffic data through protocol inference, IP address normalization, L4 port construction, and timestamp resolution to construct a five-tuple information containing source IP address, destination IP address, source port, destination port, and protocol type. Then, the five-tuple information is sorted to construct a direction-independent session key Key as a session identifier. Based on the session key Key, the data packets of the original network traffic data are aggregated and grouped, with data packets having the same session key Key grouped into the same data group, aggregating them into a network flow containing bidirectional communication content. A protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet is calculated. Based on the difference Perform session segmentation on the network flow and calculate the difference. Greater than the preset session timeout threshold for the corresponding protocol type The data packet is determined to be the start of a new session, and a session segment number S is assigned to each session segment. segment The network flow is divided into multiple sessions; finally, each session is assigned a session key (Key) and a session segment number (S). segment A globally unique flow ID is assigned to each shared, unique session, and a network flow defined by the flow ID is constructed. To obtain the aggregated dataset;

[0015] Step S12: Perform feature filtering and feature encoding on the aggregated dataset, remove unstructured load features, single constant value features, and sparse features, and encode discrete features and categorical features numerically, convert all features into numerical format, and obtain the filtered dataset composed of D-dimensional feature columns.

[0016] Step S13: Obtain the set of all stream IDs generated in step S11. , will set After random shuffling, the data is divided into four mutually exclusive ID subsets in a 7:1:1:1 ratio. Based on the flow IDs in the four mutually exclusive ID subsets, the network flows in the dataset filtered in step S12 are... Extract and group the data respectively to construct mutually exclusive training, validation, test, and generation sets at the stream level; then, combine each network stream in each dataset. The data packets are arranged in chronological order, and all feature values ​​in each data packet are concatenated into a text string. The same network flow Text strings of all data packets Connect them using newlines in chronological order, and in the network stream. Add a stream start marker at the beginning of the network stream. Add a stream end marker at the end position of each network stream. Convert it into a long text sequence to obtain a streaming text dataset for training the generative model GPT-2;

[0017] Step S14: Construct classification samples on a per-data-packet basis, concatenating the features from each data packet in the filtered dataset obtained in step S12 into a text string. Simultaneously, the attack category feature columns in the original network traffic data are mapped to multi-classification label IDs; each text string is... Combined with the corresponding tag ID, a package-level classification sample is constructed, containing text strings. The package-level sample dataset with two features, label ID and label ID, is obtained by stratified sampling of the package-level sample dataset and dividing it into a package-level training set, validation set and test set in a ratio of 8:1:1, resulting in a package-level classification text-label dataset in CSV format for training the DistilBERT classification model.

[0018] Further, in step S11, the session key Key is defined as follows:

[0019] ,

[0020] In the formula, Proto represents the protocol type. Indicates the source IP address. Indicates the destination IP address. Indicates the source port. Indicates the destination port; This means taking the minimum value between two elements. This means taking the maximum value of two elements.

[0021] Furthermore, in step S11, a protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet is calculated. According to the difference The specific method for session segmentation of network flows is as follows:

[0022] Based on the protocol type (Proto), define a protocol-adaptive session timeout threshold. ; For each data packet within a network flow, timestamp Sort the data packets in ascending order; traverse the sorted data packets and calculate the... timestamp of each data packet With the timestamp of each data packet The difference between , If the difference Session timeout threshold greater than the corresponding protocol type Proto Then determine the timestamp The corresponding data packet marks the start of a new session and is a timestamp. The corresponding data packet is assigned a new session segment number S segment This divides the network flow into multiple sessions.

[0023] Furthermore, step S2 specifically includes the following steps:

[0024] Step S21: Load the pre-trained weight file of the generative model GPT-2; the generative model GPT-2 includes a BPE tokenizer, a decoder, and an output layer. The decoder includes an embedding layer and an N-layer Transformer decoder block. The embedding layer includes a word embedding layer and a position embedding layer. The Transformer decoder block consists of a stacked first-layer normalization layer, a multi-head self-attention mechanism, a first-layer residual connection, a second-layer normalization layer, a feedforward neural network, and a second-layer residual connection; the output layer consists of a language modeling head; the BPE tokenizer is used to convert textual data packet strings into a sequence of numeric IDs that the model can understand, and to decode the ID sequence output by the model back into text; the generative model GPT-2 is used for text generation tasks;

[0025] Step S22: Add control terms to expand the vocabulary of the BPE segmenter and adjust the dimension of the weight matrix of the embedding layer according to the sequence length; the control terms include flow start marker, flow end marker and padding marker, which are used to indicate the start and end of the network flow and to unify the sequence length in batch training.

[0026] Step S23: Use the BPE tokenizer to analyze each network stream in the streaming text dataset constructed in step S13. The long text sequence is segmented into words to separate the text strings in the long text sequence. The process is decomposed into a series of lexical units, and each lexical unit is mapped to its corresponding semantically meaningful numerical ID in the vocabulary to generate an input lexical index sequence. Based on this sequence, an initial input attention mask sequence is generated. Then, by controlling the length of all sequences, a consistent input lexical index sequence and input attention mask sequence are obtained. Next, a prediction target sequence is constructed based on the input lexical index sequence and input attention mask sequence to calculate the loss function of the generative model GPT-2. Finally, the prediction target sequence, the input lexical index sequence, and the input attention mask sequence are fed into the generative model GPT-2 as input sequences.

[0027] Step S24, generate the lexical embedding layer of model GPT-2 by taking each lexical index from the input lexical index sequence generated in step S23. Mapped to word embedding vectors Obtain word embedding vector sequence ;definition The length of the input sequence in the current batch, and the position embedding layer is the position from 0 to 1 in the input sequence. Each lexical at a given position generates a corresponding position index, and all position indices are mapped to a fixed-dimensional position embedding vector. Finally, the position embedding vector is used. With word embedding vector sequence By adding elements together, we obtain the input representation that integrates semantic information and temporal location information. ;

[0028] Step S25, represent the input The output representation of the decoder is obtained by feeding it into the N-layer Transformer decoder block of the generative model GPT-2. The dimension of the decoder's output representation is projected onto the dimension of the vocabulary of the expanded BPE word segmenter through the language modeling head of the output layer of the generative model GPT-2 to obtain the log odds vector. The probability distribution of each word in the vocabulary as the word of the next timestamp is calculated based on the log odds vector.

[0029] Step S26: The cross-entropy loss function is used to optimize the generator model GPT-2. The generator model GPT-2 is fine-tuned and trained so that it learns the sequence generation rules of network traffic data packets. The trained generator model GPT-2 is obtained as the final traffic generation model used to generate predicted traffic sequences.

[0030] Furthermore, step S3 specifically includes the following steps:

[0031] Step S31: Load the pre-trained weight file of the classification model DistilBERT; the classification model DistilBERT includes a WordPiece word segmenter, an encoder, and an output layer. The encoder of the classification model DistilBERT consists of an embedding layer and N stacked bidirectional Transformer encoder blocks. The embedding layer includes a word embedding layer and a position embedding layer; the output layer of the classification model DistilBERT consists of a fully connected classification head; set the number of output categories of the fully connected classification head to the total number of categories of the classification label, and use the mapped values ​​to represent each category.

[0032] Step S32: Configure specific control units for the WordPiece word segmenter. The specific control units include classification tags for aggregating global semantic information, delimiters for identifying the end of text, and padding characters for unifying sequence length.

[0033] Step S33: Use the WordPiece tokenizer to segment the text string constructed in step S14. The input is converted into a classification input tensor, which includes a classification term index sequence and a classification attention mask sequence. The lengths of all sequences in the classification input tensor are standardized, and text strings exceeding the maximum sequence length are excluded. Truncate text strings that are shorter than the maximum sequence length. Use fill characters to fill;

[0034] Step S34: The classification input tensor is fed into the embedding layer of the classification model DistilBERT for processing. The word embedding layer maps the classification word index sequence to the classification word embedding vector, and the position embedding layer generates the classification position embedding vector corresponding to the classification word embedding vector. The classification word embedding vector and the classification position embedding vector are added element by element to obtain the classification input representation that integrates semantic and position information.

[0035] Step S35: The classification input representation is fed into an N-layer stacked bidirectional Transformer encoder block for processing. The bidirectional attention mechanism is used to calculate global context information and obtain a hidden state vector containing the complete semantics of the data packet.

[0036] Step S36: Input the hidden state vector output by the last bidirectional Transformer encoder block into the fully connected classification head of the output layer, project the features into the class space, and output the probability distribution of the data packet belonging to each class label through the Softmax function;

[0037] Step S37: Construct the Focal Loss function suitable for imbalanced samples. By introducing a modulation factor, reduce the weight of easily classified samples and improve the classification performance of the DistilBERT model on imbalanced data.

[0038] Step S38: The AdamW optimizer is used to calculate the gradient of the loss function and update the parameters of the classification model DistilBERT through the backpropagation algorithm. An early stopping strategy is introduced to select the optimal model and obtain the trained classification model DistilBERT, which is used as the final classification model for anomaly detection.

[0039] Furthermore, step S4 specifically includes the following steps:

[0040] Step S41: First, extract the preceding data from the real network stream to be detected. The data packets are used as initial historical data packets; the D-dimensional feature columns obtained after filtering the initial historical data packets in step S12 are converted into text strings according to the method described in step S13, and concatenated in chronological order to form an initial historical window; the initial historical window is used as the initial input and fed into the generative model GPT-2 trained in step S2 to start the process of generating the predicted traffic sequence.

[0041] Step S42: After starting the process of generating the predicted traffic sequence, an iterative loop is performed. The last L data packets of the current existing data packets, including the initial historical data packets and the generated data packets, are always taken as new prompt inputs to form a context sliding window for the iterative loop. Based on the context sliding window, the generation model GPT-2 autoregressively generates the text string of the predicted data packets for the next timestamp.

[0042] Step S43: The text string of the generated single prediction data packet is sent to the DistilBERT classification model trained in step S3 to perform real-time analysis on the prediction data packet, output the classification result of anomaly detection and record it.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention provides an industrial internet network intrusion detection method based on a pre-trained large language model, aiming to solve the problems of passive defense and delayed response in traditional network intrusion detection technologies. This method achieves proactive predictive intrusion detection through a three-level architecture of "data preprocessing, model fine-tuning training, and joint invocation output," enabling "generate first, then discriminate." By predictively generating possible future network traffic sequences and performing real-time analysis and anomaly detection on these predicted sequences, the defense gate is moved forward, transforming passive response into proactive early warning. This method can provide early warning before actual attack traffic arrives, significantly improving the real-time performance and reliability of industrial internet network security protection. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the industrial internet network intrusion detection method based on a pre-trained large language model of the present invention.

[0046] Figure 2 This is a schematic diagram of the model module of the industrial internet network intrusion detection method based on a pre-trained large language model of the present invention;

[0047] Figure 3 This is a loss curve diagram of the training of the generative model GPT-2 in this invention;

[0048] Figure 4 This is a loss curve diagram of the DistilBERT classification model trained in this invention;

[0049] Figure 5 This is a confusion matrix diagram of the multi-class test set of the DistilBERT classification model in this invention. Detailed Implementation

[0050] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0051] Reference Figure 1 and Figure 2 This invention provides an industrial internet network intrusion detection method based on a pre-trained large language model, mainly consisting of four steps: data preprocessing, fine-tuning training of the traffic generation model GPT-2, fine-tuning training of the traffic classification model DistilBERT, and joint invocation of a prediction framework to output prediction results. The data preprocessing step is responsible for processing the raw network traffic dataset into data suitable for input to both the generation model GPT-2 and the classification model DistilBERT. The preprocessed data is used to train the generation model GPT-2 and the classification model DistilBERT in parallel during the fine-tuning training step. Finally, the pre-trained generation model GPT-2 and the classification model DistilBERT are jointly invoked to form a prediction framework. This framework enables the real-time generation of predicted data packets for the next time stamp based on a limited number of real historical data packets captured at the current time stamp in an industrial internet scenario. It also performs anomaly classification on the traffic sequences in the generated predicted data packets and outputs the prediction results. This method aims to construct a proactive prediction framework of "generate first, then judge," transforming passive defense into proactive early warning. The specific steps of the method are as follows:

[0052] Step S1, Data Preprocessing: The original network traffic data is subjected to protocol-adaptive flow aggregation and session segmentation to construct a five-tuple information including source IP address, destination IP address, source port, destination port, protocol type, and timestamp. Then, the original network traffic data is aggregated and grouped based on the five-tuple information and timestamp to obtain the aggregated dataset. Then, feature filtering and feature encoding are performed on the aggregated dataset, and a streaming text dataset for training the generative model GPT-2 and a packet-level classification text-label dataset for training the classification model DistilBERT are constructed respectively.

[0053] The purpose of the data preprocessing is to extract features from raw network traffic data in PCAP file format (binary format) captured in the Industrial Internet environment, convert the raw network traffic data into CSV format network traffic data, and then clean, aggregate, and text-ize the CSV format network traffic data to transform it into a data format suitable for training model input. The data preprocessing method of this invention adopts a dual-path processing strategy, constructing datasets suitable for two different downstream tasks: one is a streaming text dataset for training the GPT-2 generative model, and the other is a packet-level classification text-label dataset for training the DistilBERT classification model. This data preprocessing method is adaptable to the complex and heterogeneous traffic background of the Industrial Internet scenario. Its protocol-adaptive stream aggregation and session segmentation, feature filtering, and feature encoding steps can efficiently process complex traffic data containing multiple industrial communication protocols such as Modbus, HTTP, MQTT, TCP, UDP, ICMP, and ARP, preparing the subsequent model to fully learn the patterns of complex Industrial Internet traffic.

[0054] Specifically, the data preprocessing includes the following sub-steps:

[0055] Step S11, Protocol-Adaptive Flow Aggregation and Session Segmentation: This step addresses the flow-level traffic generation task. Since the raw network traffic data of the Industrial Internet is typically a packet-level feature set (a feature set summarized in data packets), this invention first infers and constructs standardized five-tuple information (source IP address, destination IP address, source port, destination port, protocol type) and timestamps from the scattered feature fields in the raw network traffic data through protocol inference, IP address normalization, L4 port construction, and timestamp resolution. Then, the five-tuple information is sorted to generate a unified session key as a session identifier. Based on the session key, the data packets of the raw network traffic data are aggregated and grouped, with data packets having the same session key grouped into the same data group, aggregating them into a network flow containing bidirectional communication content. A protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the previous adjacent data packet is calculated. According to the difference Perform session segmentation on the network flow and calculate the difference. Greater than the preset session timeout threshold for the corresponding protocol type The data packet is determined to be the start of a new session, and a session segment number S is assigned to each session segment. segment The network flow is divided into multiple sessions; finally, each session is assigned a session key (Key) and a session segment number (S). segment A globally unique flow ID is assigned to each shared, unique session, and a network flow defined by the flow ID is constructed. The aggregated dataset is obtained. The specific construction steps are as follows:

[0056] Step S111, Protocol Inference: Since the raw network traffic data of the Industrial Internet comes from various sensor devices and there is no unified "protocol" field in the dataset, this invention infers the protocol type Proto of each data packet in the raw network traffic data by checking the existence and validity of specific related fields in different protocol types. The inference logic follows this priority order: 1. If the TCP protocol-related fields, i.e., the TCP source port field (tcp.srcport field) or the TCP destination port field (tcp.dstport field), exist and have valid values, the protocol type is determined to be TCP, and Proto=TCP is assigned; 2. If the TCP condition is not met (i.e., the TCP protocol-related fields do not exist), but the UDP protocol-related fields, i.e., the UDP port field (udp.port field), exist and have valid values, the protocol type is determined to be UDP, and Proto=UDP is assigned; 3. If both the TCP and UDP conditions are not met (i.e., the TCP and UDP protocol-related fields do not exist), but the data row contains valid values ​​for ICMP protocol-related fields (such as the ICMP checksum field icmp.checksum), the protocol type is determined to be ICMP, and Proto=ICMP is assigned; 4. If the above conditions are not met, but the ARP protocol-related fields (such as the ARP opcode field arp.opcode) contain valid values, the protocol type is determined to be ARP, and Proto=ARP is assigned; 5. Otherwise, the protocol type is determined to be OTHER, and Proto=OTHER is assigned.

[0057] Step S112, IP address normalization: Source IP address and destination IP address The IP address is primarily extracted from the source host field (ip.src_host field) and the destination host field (ip.dst_host field). To ensure correct aggregation even when IP information is missing, this invention utilizes the source IPv4 address field (arp.src.proto_ipv4 field) and destination IPv4 address field (arp.dst.proto_ipv4 field) of the ARP protocol to separately process the source IP address. and destination IP address Missing values ​​are filled in. At the same time, all IP addresses are cleaned to remove invalid entries such as 0.0.0.0 or null values.

[0058] Step S113, L4 Port Construction: Source Port and destination port The construction depends on the protocol type inferred in step S111. If Proto=TCP, then the source port... and destination port The ports are obtained from the TCP source port field (tcp.srcport field) and the TCP destination port field (tcp.dstport field), respectively. If Proto=UDP, since the original network traffic data usually only provides a single UDP port field (udp.port field), this invention maps the UDP port field to the destination port uniformly. and the source port Set to 0; if Proto is ICMP, ARP, or OTHER, then the source port... and destination port Set all values ​​to 0.

[0059] Step S114, Timestamp Parsing: The timestamp of the data packet Extracted from the timestamp field (frame.time) and uniformly parsed into a high-precision second-level floating-point number format for subsequent sorting and timeout judgment.

[0060] After completing the above steps, obtain the 5-tuple information of all data packets. and timestamp .

[0061] Step S115, Stream aggregation and session segmentation:

[0062] To ensure that bidirectional data packets within the same session can be aggregated into the same stream, this invention constructs a direction-independent session key. The IP address pairs (source IP address, destination IP address), port pairs (source port, destination port), and protocol type in each data packet are normalized and sorted to generate a unified session key Key as the session identifier. The session key Key is defined as:

[0063] ,

[0064] In the formula, This means taking the minimum value between two elements. This indicates taking the maximum value of two elements;

[0065] By establishing a unique session identifier for bidirectional data flow through the above steps, it is ensured that uplink and downlink data packets in the same session are marked as the same logical entity, thereby providing an index basis for subsequently aggregating discrete and chaotic raw data packets into an ordered network flow.

[0066] Considering the differences in session characteristics among different protocols (such as TCP long connections and UDP short queries) in the Industrial Internet, this invention adopts a protocol-adaptive session timeout threshold. Based on experience in industrial scenarios, session timeout thresholds for the TCP protocol are set accordingly. UDP protocol session timeout threshold Seconds, the session timeout threshold for the ICMP protocol. Seconds, the session timeout threshold of the ARP protocol. Seconds, and session timeout thresholds for protocols other than the four mentioned above (OTHER protocols). Second.

[0067] Next, based on the direction-independent session key, the data packets of the original network traffic data are aggregated and grouped. All data packets with the same session key are grouped into the same data group (i.e., aggregated into a network flow containing bidirectional communication content), and the data packets within each group are timestamped. Sort the data packets in ascending order. Iterate through the sorted data packets and calculate the... timestamp of each data packet With its preceding adjacent data packet (i.e., the first) (timestamp of data packet) The difference between , If the difference The timeout exceeds the preset session timeout threshold for the corresponding protocol type (Proto). Then determine the timestamp The corresponding data packet marks the start of a new session and is a timestamp. The corresponding data packet is assigned a new session segment number S segment This involves segmenting a network flow into multiple sessions (i.e., breaking a long network flow into multiple shorter network flows). This process ultimately completes the reorganization from packet-level data to stream-level data.

[0068] Finally, for each session segment consisting of a session key (Key) and a session segment number (S) segment A unique session defined in common (i.e., each segmented network flow) Assign a globally unique flow ID and construct a network flow defined by the flow ID. The aggregated dataset is then obtained for subsequent stream-level data partitioning and modeling. The formula for the stream ID is as follows:

[0069] ,

[0070] Where Flow_ID represents the flow ID, It is an MD5 hash function used to ensure the uniqueness of stream IDs.

[0071] Step S12, Feature Selection and Encoding: To reduce data noise, decrease model training overhead, and improve feature effectiveness, feature selection and dimensionality reduction are performed on the streaming data in the aggregated dataset obtained in Step S11. Specific steps include:

[0072] Step S121, Remove Unstructured Payload Features: Payload-related fields and unstructured text fields in the network traffic data of the dataset are removed. Specifically, these include the HTTP file data field (http.file_data), the TCP payload field (tcp.payload), the MQTT message body field (mqtt.msg), the MQTT protocol name field (mqtt.protoname), and the HTTP request full URI field (http.request.full_uri). The content of these fields typically consists of unformatted binary streams, text of arbitrary length, or random strings. They lack a fixed semantic structure and have an excessively large value space. Directly using them for model training can lead to the curse of dimensionality or overfitting, therefore they are removed.

[0073] Step S122, Remove features with single constant values: Calculate the number of unique values ​​in each feature column of the dataset, and delete the feature columns with a single constant value (such as columns with all empty values ​​or completely identical values). Such features are the same in all samples and have no distinguishing effect.

[0074] Step S123, Remove sparse features: Set a threshold for the proportion of zero values. =75%, calculates the proportion of zero or null values ​​in numerical features. Removes values ​​where the proportion of zero values ​​exceeds a certain threshold. Numerical features.

[0075] Step S124: Numerical encoding of discrete and categorical features: Traverse all feature columns in the dataset, identify non-numerical feature columns (including source IP address, destination IP address, source port, destination port, and protocol type) whose data type is string object (Object) or category (Category), and map them to unique integer indices, thereby converting all features into numerical format.

[0076] After the above feature selection and feature encoding, the retained feature columns (D dimensions) form the selected dataset, which is used for the subsequent construction of the streaming text dataset.

[0077] Step S13, construct a streaming text dataset for training the generative model GPT-2: First, to ensure the accuracy of the model's generalization ability evaluation, prevent the same network stream from being used. Data packets crossing datasets can lead to information leakage, so a mutual exclusion partitioning based on flow IDs is performed. A set of all flow IDs generated in step S11 is obtained. , will set After random shuffling, the data is divided into four mutually exclusive ID subsets in a 7:1:1:1 ratio. Based on the flow IDs in the four mutually exclusive ID subsets, the network flows in the dataset filtered in step S12 are... The corresponding data are extracted and grouped to construct a stream-level mutually exclusive training set, validation set, test set, and generation set.

[0078] Subsequently, for each network flow in each dataset Perform text serialization and construction. Transform the network stream. The data packets are arranged in chronological order, and all feature values ​​in each data packet are stored in the data packet. Concatenate into a text string ,For example D is the feature dimension of the dataset after filtering in step S12.

[0079] Finally, the same network flow Text strings of all data packets Connect them using newlines in chronological order, and in the network stream. Add a stream start marker at the beginning position <bos>In network flow Add a stream end marker at the end position <eos>Ultimately, each network flow Convert it into a long text sequence, in the following format: <bos>[newline character] [newline character] ...[line break] [newline character] <eos>This involves using mutually exclusive training, validation, and test sets, as well as all network streams in the generator set. Converting it into a long text sequence yields the streaming text dataset used to train the generative model GPT-2.

[0080] Step S14: Construct a packet-level classification text-label dataset for training the DistilBERT classification model. This step targets the packet-level anomaly classification task, constructing classification samples on a per-data packet basis, using the same feature selection criteria and numerical encoding method as in Step S12. All feature values ​​from each data packet are... Concatenate into a text string ,For example The text string is constructed in the same way as the network stream defined in step S13. Text string of internal data packet Maintain consistency. Simultaneously, map the "Attack_type" feature column from the original network traffic data to multi-category label IDs. For each text string... Together with the corresponding tag ID, they form a package-level classification sample, ultimately constructing a sample containing text strings. The package-level sample dataset contains two features: the label ID and the label itself. The package-level sample dataset is then stratified and divided into a training set, a validation set, and a test set in an 8:1:1 ratio. This dataset is then saved as a package-level classification text-label dataset in CSV format for training the DistilBERT classification model.

[0081] Step S2: Fine-tune the training of the generative model GPT-2: Load the pre-trained weights of the generative model GPT-2, add special control tokens, and adjust the dimension of the token embedding layer of the generative model GPT-2. Train the model on the streaming text dataset obtained in step S13 with the goal of Causal Language Modeling (CLM) to obtain a traffic generation model that can predict the next time (i.e., the next timestamp) of data packets based on historical window data packets.

[0082] The purpose of fine-tuning the generative model GPT-2 is to use the pre-trained generative model GPT-2 to fine-tune the training on the streaming text dataset constructed in step S1, so that it can predict the next timestamp of a single data packet based on the traffic sequence in the historical window data packets within a network flow, thereby realizing the proactive generation and simulation of potential attack traffic in the industrial Internet scenario.

[0083] In the Industrial Internet environment, network communication typically follows strict patterns based on specific industrial protocols (such as Modbus and MQTT), encompassing unique communication patterns between devices comprised of programmable logic controllers (PLCs), supervisory and data acquisition systems (SCADA) and various sensors. The generative model GPT-2 trained in this invention aims to deeply learn and reproduce these complex, time-dependent communication patterns.

[0084] The generative model GPT-2 used in this invention is a Transformer architecture that includes a word segmenter (BPE segmenter), a decoder-only module, and an output layer (language modeling head). It ensures that the traffic generation process relies solely on historical sequences through causal masking. The decoder consists of an embedding layer and N Transformer decoder blocks. The embedding layer includes a word embedding layer and a position embedding layer. The internal structure of the Transformer decoder block mainly consists of a stacked first-layer normalization layer, a multi-head self-attention (MHA) mechanism, a first residual connection, a second-layer normalization layer, a feed-forward neural network (FFN), and a second residual connection. The MHA mechanism is responsible for capturing long-range dependencies in the sequence, while the FFN performs non-linear transformations and enhancements on the features. This framework is particularly suitable for traffic generation tasks in industrial internet scenarios because network flows in these scenarios often exhibit strong temporal and contextual relevance. Multi-head self-attention mechanisms can efficiently capture long-distance dependencies, while their autoregressive nature supports packet-by-packet prediction based on a limited historical set of packets. This enables models that effectively predict and generate potential future network traffic sequences. The specific implementation steps for fine-tuning the GPT-2 generative model are as follows:

[0085] Step S21, Initialization of the generation model GPT-2 and the BPE word segmenter:

[0086] Before fine-tuning the generative model GPT-2, the pre-trained model components need to be loaded first. Specifically, the pre-trained weights and configuration files for the generative model GPT-2 are downloaded from the publicly available pre-trained model library Hugging Face and saved to a preset local file path. Then, the pre-trained weights of the generative model GPT-2 (model GPT-2LMHeadModel) and the corresponding BPE tokenizer GPT-2Tokenizer are loaded from the local file path. Model GPT-2LMHeadModel is a GPT-2 model with a language model head, suitable for text generation tasks; the BPE tokenizer GPT-2Tokenizer is responsible for converting the textualized data packet string into a sequence of numeric IDs that the model can understand, and for decoding the ID sequence output by the model back into text.

[0087] Step S22, Special vocabulary expansion and embedding layer adjustment:

[0088] In order for the generative model GPT-2 to understand the structure of network flows, this step expands the vocabulary of the tokenizer, adding three special control terms to the original BPE tokenizer GPT-2Tokenizer vocabulary: flow start token. <bos>Stream end marker <eos>and fill mark <pad>These markers are used to indicate the start and end of network streams, and to standardize sequence lengths during batch training. They enable the model to transform continuous network stream data into "sentence" units with well-defined boundaries, thereby better learning the structure and patterns of communication sessions.

[0089] After expanding the vocabulary of the BPE tokenizer, the dimensions of the weight matrix of the model's token embedding layer need to be adjusted accordingly to match the size of the vocabulary after the addition of control tokens. During subsequent fine-tuning, the weight matrix will be continuously optimized, enabling the generative model GPT-2 to understand and utilize the newly added control tokens to extract specific semantic functions within the context of the network stream.

[0090] Step S23, Data Reading and Word Segmentation:

[0091] Step S231: Obtain all network streams from the streaming text dataset constructed in step S13. A long text sequence;

[0092] Step S232: Use the BPE tokenizer to segment each network stream. The long text sequence is segmented into words to separate the text strings in the long text sequence. The input is decomposed into a series of lexical units, and each lexical unit is mapped to its corresponding semantically meaningful numerical ID in the vocabulary, generating an input lexical index sequence (input_ids). At the same time, an initial input attention mask sequence (attention_mask) is generated based on the input lexical index sequence, and the mask values ​​corresponding to all currently valid lexical units in the initial input attention mask sequence are set to 1.

[0093] Step S233: For all input word index sequences and initial input attention mask sequences in the current training batch, pad the shorter sequences at the end according to the length of the longest sequence in the batch: use padding markers. <pad>End padding (i.e., padding markers) is applied to the input lexical index sequence. <pad>Fill in the end-of-flow marker <eos>(Previous), and use the value 0 to pad the end of the initial input attention mask sequence accordingly (corresponding to the padding marker). <pad>(position), to ensure that all sequences in the current batch have the same length.

[0094] Step S234: Construct the prediction target sequence for calculating the loss function of the generative model GPT-2. Specifically, first, copy the padded input word index sequence obtained in step S233 as the initial prediction target sequence. Then, use the padded input attention mask sequence from step S233 to filter the prediction target sequence, identifying the elements in the prediction target sequence corresponding to the padded markers. <pad>The numerical values ​​at the positions (i.e., the values ​​with a mask value of 0) are replaced with specific ignore flag values ​​(such as -100). The processed predicted target sequence, along with the padded input word index sequence and the input attention mask sequence from step S233, are fed into the generative model GPT-2 as input sequences. The internal mechanism of the generative model GPT-2 automatically performs misaligned prediction (i.e., predicting the next word based on the word at the current time step). The specific ignore flag value ensures that the padded part does not participate in gradient calculation.

[0095] Step S24, Lexical embedding and positional embedding:

[0096] Step S241, Token Embedding. Each token index in the input token index sequence generated in step S232 is... Mapped to word embedding vectors Obtain high-dimensional word embedding vector sequence , .in, This is the position index of a lexical unit within the input lexical index sequence. , This represents the lookup mapping operation in the model embedding layer, which maps discrete indices to corresponding vectors using a table lookup method. The length of the input sequence in the current batch. For the first Word embedding vectors.

[0097] Step S242, Positional Embedding. To enable the generative model GPT-2 to recognize the temporal features of streaming sequence data, a learnable positional embedding layer needs to be introduced. First, a positional embedding layer is created for the input sequence from 0 to... Each lexical at a given position generates a corresponding position index, and these position indices are mapped to a fixed-dimensional position embedding vector through a position embedding layer. Finally, the position embedding vector is... With the word embedding vector sequence obtained in step S241 By adding elements together, we obtain the input representation that integrates semantic information and temporal location information. :

[0098] ;

[0099] Step S25: Calculate the Transformer decoder block for the generative model GPT-2. The core of the generative model GPT-2 consists of multiple stacked Transformer decoder blocks. Unlike the traditional Transformer structure, the Transformer decoder blocks in the generative model GPT-2 adopt a pre-layer normalization layer architecture, that is, layer normalization is performed before entering the multi-head self-attention mechanism and feedforward neural network. Each Transformer decoder block sequentially includes a first layer normalization layer, a multi-head self-attention mechanism (MHA), a first residual connection, a second layer normalization layer, a feedforward neural network (FFN), and a second residual connection. Both the multi-head self-attention mechanism and the feedforward neural network perform dimensionality mapping and transformation through linear layers. For the... The specific calculation method for the layer Transformer decoder block is as follows:

[0100] Step S251, First layer normalization and projection: First, for the first layer... The output representation of the layer Transformer decoder block Perform the first layer normalization to obtain the state representation after the first layer normalization. :

[0101] ,

[0102] in, Presentation layer normalization operation.

[0103] The state representation after the first level normalization is then... Input is fed into a multi-head self-attention mechanism. The linear transformation matrix of the linear layer in the multi-head self-attention mechanism is then applied. The state representation after the first level normalization Projection as query matrix Key matrix Sum matrix :

[0104] ;

[0105] Step S252, Masked Self-Attention Calculation: For each head of the multi-head self-attention mechanism, calculate the attention weights:

[0106] ,

[0107] in, The weights represent the single-head attention. Represents the normalized exponential function, Represents the query matrix. This represents the transpose of the key matrix. is the dimension of the key matrix (used for scaling the dot product). Represents the causal mask matrix. The value matrix is ​​used to represent the values. To ensure the autoregressive properties of the generative model GPT-2, a causal mask matrix is ​​introduced. Ensure location Only the location can be focused on Information from previous moments.

[0108] Step S253, Multi-head Concatenation and Linear Projection: In the multi-head self-attention mechanism, the self-attention mechanism is applied in parallel to multiple "heads". Each head independently calculates the attention, and then the outputs of all heads are concatenated and projected through the output linear layer to obtain the final output representation of the multi-head self-attention mechanism (MHA). The formula for the multi-head self-attention mechanism is expressed as:

[0109] ,

[0110] In the formula, This represents a vector concatenation operation. This represents the output of the first attention head. This indicates the total number of attention heads. Indicates the first The output of each attention head The linear projection weight matrix represents the output linear layer of the multi-head self-attention mechanism, used to map the concatenated multi-head features back to the hidden layer dimension of the model. This multi-head self-attention mechanism allows the model to learn information from different representation subspaces and focus on dependencies at different positions in the sequence.

[0111] Step S254, First residual connection and second layer normalization: Output representation obtained after the multi-head self-attention mechanism is calculated. It will be residually connected to the input representation before entering the multi-head self-attention mechanism to obtain the intermediate state representation. :

[0112] ,

[0113] Then represent the intermediate states. Perform a second layer normalization to obtain the state representation after the second layer normalization. The calculation formula is as follows:

[0114] ,

[0115] State representation after second layer normalization As input to the feedforward neural network;

[0116] Step S255, Feedforward Neural Network Calculation: The feedforward neural network is a location-shared fully connected network consisting of two linear transformation layers, used to perform non-linear transformations on the features. The specific steps are as follows:

[0117] The first step is to expand the feature dimension through the first linear transformation layer, followed by activation by a Gaussian error linear unit (GELU):

[0118] ,

[0119] The second step involves projecting the feature dimensions back to the original dimensions using a second linear transformation layer:

[0120] ,

[0121] in, It is the intermediate activation state of the hidden layer (the output of the first linear transformation layer). Represents a linear unit with Gaussian error. and These are the weight matrix and bias of the first linear transformation layer, respectively. and These are the weight matrix and bias of the second linear transformation layer, respectively. It is the output of the second linear transformation layer, which is the final output of the feedforward neural network.

[0122] Step S256, Second residual connection with decoder output: The output of the feedforward neural network is connected... Intermediate state representation Perform residual join to obtain the first The final output representation of the layer Transformer decoder block :

[0123] ,

[0124] Output representation Will be the first The input to the layer Transformer decoder block is passed through N layers of Transformer decoder blocks to obtain the output representation of the decoder.

[0125] Step S257, Language Modeling Head and Logits Generation: After processing by N layers of Transformer decoder blocks, the final hidden state representation output by the model. (i.e., the state representation after the second layer normalization in the last Transformer decoder block) The final normalized result is passed to the language modeling head. The language modeling head is a linear layer that projects the dimension of the output representation of the Nth layer Transformer decoder block onto the dimension of the vocabulary of the augmented BPE tokenizer, resulting in a log-odds vector (Logits). The calculation formula for the language modeling head is as follows:

[0126] ,

[0127] In the formula, Logits represents the log-odds vector output by the language modeling head. This represents the weight matrix of the language modeling head. The bias term representing the language modeling head;

[0128] Step S258, Probability Distribution Calculation: Apply the normalized exponential function (Softmax function) to the log-odds vector (Logits) obtained in step S257 to obtain the probability distribution of each word in the vocabulary as the word of the next timestamp:

[0129] ,

[0130] In the formula, Indicates in a given historical sequence arrive Under the conditions, the first The meta-index of words is The conditional probability, This represents the log-odds vector corresponding to the output at the end of the historical sequence. This is the normalization exponential function, used to convert numerical values ​​into probability values ​​that sum to 1.

[0131] Step S26, Optimization of loss function and generative model GPT-2:

[0132] For including Network flow The stream-level training set is defined as the first stream obtained after word segmentation in step S232. Network flow The total length of the converted word sequence is Cross-entropy loss function modeled using causal language Optimize the generative model GPT-2 using the cross-entropy loss function. The calculation formula is:

[0133] ,

[0134] In the formula, For the first Network flow The Middle The actual words at each position, where log represents the logarithmic function. The representation model is based on a given historical lexical sequence. Predict the conditional probability of the word at the current position under the given conditions. This represents all learnable parameters of the generative model GPT-2 (including lexical embedding parameters, positional embedding parameters, decoder block weights, and language modeling head weights).

[0135] During training, the cross-entropy loss function is minimized. It can drive the generation of all parameters of the GPT-2 model. The optimization makes the model more accurate for subsequent data packets that actually occur (corresponding to real words). It can provide higher predictive conditional probabilities. This enables the model to learn the deep temporal patterns and dynamics contained in industrial internet traffic data.

[0136] The calculated loss is propagated back to the model via backpropagation to update the model's weights. The AdamW optimizer, combined with a cosine scheduler, adjusts the learning rate to achieve efficient updates to the model parameters.

[0137] After training, the fine-tuned weight parameters of the generative model GPT-2 and the updated BPE segmenter configuration are saved to the specified path to obtain the trained generative model GPT-2, which is used as the final traffic generation model for generating predicted traffic sequences.

[0138] Step S3: Fine-tune the DistilBERT classification model:

[0139] Load the pre-trained weights of the DistilBERT classification model, construct a task architecture for single-packet multi-class classification, and train it on packet-level classification samples using the Focal Loss loss function and early stopping strategy to obtain a classification model capable of packet-by-packet anomaly detection.

[0140] The DistilBERT classification model comprises a WordPiece segmenter, an encoder, and an output layer. The encoder of DistilBERT includes an embedding layer and an N-layer bidirectional Transformer encoder block. The purpose of fine-tuning the DistilBERT classification model is to train an efficient and accurate multi-class classification model for anomaly detection of individual network traffic packets. The classification model must be able to distinguish between normal traffic packets and potentially attack packets. To meet the real-time and resource efficiency requirements of the Industrial Internet environment, this invention selects the packet-level anomaly classification model DistilBERT. DistilBERT is a distilled and compressed version of the BERT model, with fewer parameters and faster inference speed, while maintaining performance comparable to the original BERT model in various downstream tasks, enabling effective deployment on edge computing devices or resource-constrained industrial control hosts. The specific implementation steps are as follows:

[0141] Step S31, Initialization of the DistilBERT classification model and the WordPiece word segmenter:

[0142] First, download the pre-trained weight file and configuration file of the DistilBERT classification model from the publicly available pre-trained model library Hugging Face, and save them to a preset local file path. Then, load the pre-trained classification model DistilBERT and its corresponding WordPiece word segmenter from the local file path. To adapt to the multi-class classification task of this invention, when loading the model, the number of output categories of the fully connected classification head of the model is set to the total number of categories in the classification labels, and each category is represented by a mapped value, thus completing the model initialization.

[0143] Step S32, Lexical Expansion and Embedding Layer Adaptation:

[0144] To meet the requirements of classification tasks, specific control units are configured in the WordPiece word segmenter. These control units mainly include the classification marker [CLS], the separator marker [SEP], and the padding character [PAD]. The classification marker [CLS] is added at the beginning of each sequence to aggregate global semantic information of the entire data packet for subsequent classification; the separator marker [SEP] is added at the end of the sequence to indicate the end of the text; and the padding character [PAD] is used to standardize the sequence length.

[0145] Step S33, Construction of the classification dataset:

[0146] This step uses the packet-level text-label dataset in CSV format constructed in step S14. For each packet sample, the text string constructed in step S14 is segmented using the WordPiece tokenizer. Convert the classification input tensor to the model, including the classification term index sequence and the classification attention mask sequence. Set the maximum sequence length in the classification input tensor. For text strings exceeding the specified length Truncate text strings that are shorter than the specified length. Fill with the filler symbol [PAD] as described in step S32.

[0147] Step S34, Embedding layer calculation:

[0148] The embedding layers of the DistilBERT classification model mainly consist of a word embedding layer and a positional embedding layer. The computational principles of the word embedding layer and the positional embedding layer are consistent with those of the embedding layer in the generative model GPT-2 described in step S24. They are used to map the discrete word indices in the classification input tensor into vector representations and to capture the positional information of the sequence, respectively. After fusing semantic and positional information, the classification input representation is obtained. It is worth noting that, to achieve lightweight design and efficient inference, the DistilBERT classification model removes the segmentation embedding layer from the original BERT model.

[0149] Step S35, Bidirectional Transformer Encoder Block:

[0150] Each bidirectional Transformer encoder block contains a multi-head bidirectional self-attention mechanism, a feedforward neural network, a layer normalization layer, and residual connections. The computational logic of its internal layer normalization layer, feedforward neural network, and residual connections is consistent with the Transformer decoder block of the generative model GPT-2 described in step S25. The key difference between the bidirectional Transformer encoder block and the Transformer decoder block in step S25 lies in the visibility domain of the attention mechanism. The generative model GPT-2 uses a causal mask, which can only focus on historical information; while the classification model DistilBERT uses a bidirectional attention mechanism, which does not use a causal mask matrix when calculating attention weights, allowing any word in the sequence to pay attention to all words in the context. This global context awareness capability is crucial for classification tasks because it allows the model to fully understand the semantics of the entire data packet text for accurate judgment, enabling the model to obtain a hidden state vector containing the complete semantics of the data packet.

[0151] Step S36, Output the fully connected classification header:

[0152] This is the fundamental difference between the classification model DistilBERT and the generative model GPT-2 at the output end. The generative model GPT-2 uses a language modeling head to project the output onto the vocabulary space to predict the next word; while the classification model DistilBERT uses a fully connected sequence classification head to project features onto a category space with a specific dimension. Specifically, the final hidden state vector output from the last bidirectional Transformer encoder block of the classification model DistilBERT is taken as the semantic aggregation representation of the entire data packet sequence. This vector is then input into a linear classification layer, where the vector dimension is mapped from the model's hidden layer dimension to the number of target categories (i.e., the total number of categories). Subsequently, the Softmax function is used to output the probability distribution of the data packet belonging to each category label.

[0153] Step S37, Focal Loss function:

[0154] To effectively address the prevalent class imbalance problem in industrial internet traffic data, this invention employs the Focal Loss function when calculating classification loss. The Focal Loss function is an improvement upon the standard cross-entropy loss function. It introduces a modulation factor... This dynamically reduces the weight of easily classifiable samples in the loss calculation. This allows the classification model to focus more on the difficult-to-classify minority class samples during training, which is very effective for handling imbalanced datasets. The specific formula for the Focal Loss function is expressed as:

[0155] ,

[0156] in, This represents the value of the Focal Loss function. This represents the model's predicted probability for the true class. Modulation factor (set in this embodiment) When predicting probability When the value approaches 1 (representing that the sample belongs to an easily classifiable sample), the weight... The value approaches 0, thus significantly reducing the contribution of this sample to the total loss, allowing the model to focus on difficult-to-classify samples during optimization and improving the model's classification performance on imbalanced data.

[0157] Step S38, Model Training and Early Stopping Strategy:

[0158] The AdamW optimizer is used, and the gradient of the loss function is calculated through backpropagation to update the parameters of the DistilBERT classification model. During training, an early stopping strategy is introduced to select the optimal model. Specifically, after each training epoch, the model's performance is evaluated on an independent validation set. If the loss on the validation set fails to decrease for several consecutive epochs (in this embodiment, the threshold is set to two consecutive epochs), the training process is terminated early. This effectively prevents the model from overfitting on the training set and ensures that the selected model has the best generalization performance. Finally, the model weight parameters of the epoch with the lowest validation set loss are saved to obtain the trained classification model DistilBERT, which serves as the final classification model for anomaly detection.

[0159] Step S4: Jointly call the prediction framework to output the prediction results: Using the first L data packets of the real network flow (L=3 in this example) as the initial historical window (i.e., sliding window), the prediction framework is composed of the generative model GPT-2 trained in step S2 and the classification model DistilBERT trained in step S3. First, the trained generative model GPT-2 is called to generate traffic sequences and obtain predicted data packets. Then, the trained classification model DistilBERT is called to perform anomaly detection on each sequence data in the predicted data packets and output the classification results of the anomaly detection, thus realizing proactive intrusion detection that first generates and then detects.

[0160] The joint invocation of the prediction framework and the output of the prediction results are the core of this invention's proactive intrusion detection. By jointly invoking the generative model GPT-2 trained in step S2 and the classification model DistilBERT trained in step S3, a closed-loop workflow of "generate first, then discriminate" is constructed. This step realizes the transformation from traditional passive, lagging analysis to proactive, predictive defense. Traditional intrusion detection systems analyze network traffic that has already occurred, while this invention, through high-probability prediction of traffic at the next moment, can identify attack intent at the initial stage of the attack sequence, even before malicious data packets are actually transmitted to the target network. This "pre-cognition" defense capability is the core difference between this invention and existing technologies, providing timely and effective security responses for critical infrastructure such as the Industrial Internet. The joint invocation of the prediction framework and the output of the prediction results specifically include the following sub-steps:

[0161] Step S41, Construction and generation of real-time prompts: First, extract the preceding words from the real network stream to be detected. The L data packets are used as initial historical data packets. The D-dimensional feature columns of the L data packets, after being filtered in step S12, are converted into text strings according to the method described in step S13, and then concatenated in chronological order to form an initial historical window. The initial historical window is used as the initial input and fed into the generative model GPT-2 trained in step S2 to start the process of generating the predicted traffic sequence.

[0162] Step S42, Iterative Generation: After generation begins, an iterative loop is performed, always taking the last L data packets of the currently existing data packets (including initial historical data packets and generated data packets) as new prompt inputs, forming a context sliding window for the iterative loop. Based on this context sliding window of the iterative loop, the generative model GPT-2 autoregressively generates the most likely (i.e., most probable) text string for the predicted data packet of the next timestamp.

[0163] Step S43, Real-time anomaly detection: The text string of the newly generated single prediction data packet is sent to the DistilBERT classification model trained in step S3 to perform real-time analysis on the prediction data packet, output its classification result and record it.

[0164] Step S44, Termination Condition of Generation Process: To ensure the effectiveness and finiteness of the generation process, the model stops generating the current predicted flow sequence if one of the following conditions is met: (1) Natural termination. The generation model GPT-2 outputs a special flow end marker. <eos>This means that, based on the traffic patterns it has learned, the model predicts that the network session (i.e., the network flow) has ended naturally. (2) Forced termination. The number of generated packets reaches the preset maximum network flow length threshold. This preset maximum network flow length threshold is used to prevent unlimited generation due to model prediction errors or abnormal patterns, ensuring the effective utilization of system resources.

[0165] Step S45, Quality Assessment of the Generated Traffic Sequence: After the generation process is completed, in order to verify the accuracy of the model prediction, this invention also assesses the quality of the generated traffic sequence for offline performance verification of the model. Evaluation metrics include perplexity (PPL) and cosine similarity.

[0166] Step S451: Perplexity is used to measure the reasonableness of the generated traffic sequence content. The lower the perplexity (PPL) value, the higher the certainty of the model regarding the generated traffic sequence, and the more the traffic sequence conforms to the traffic pattern it has learned. This embodiment uses "average per packet perplexity" as the evaluation index, that is, the perplexity of each generated data packet is calculated separately, and then the arithmetic mean of all perplexities is calculated to obtain the average perplexity. Specifically, the formula for calculating the average perplexity is:

[0167] ,

[0168] in, R represents the average perplexity of the generated traffic sequence, and R is the total number of packets actually generated in the current network flow. For the first The generated data packets, NLL being the negative log-likelihood, correspond to the cross-entropy loss of the model output ( ), This is for exponential function operations.

[0169] Step S452: Cosine similarity, used in the model validation phase, measures the semantic similarity between the generated predicted data packets and the real data packets. It is obtained by calculating the cosine similarity between the hidden state vectors output by the last layer of the model. The formula for calculating cosine similarity is as follows:

[0170] ,

[0171] in, Represents cosine similarity. and Generate data packets respectively and real data packets After alignment, the first The hidden state vector of each word. The length of the aligned flow sequence.

[0172] To verify the effectiveness of the method proposed in this invention, this embodiment uses the publicly available network intrusion detection dataset Edge-IIoTset for experiments. The Edge-IIoTset dataset simulates typical industrial scenarios under an edge computing architecture, combining data collected from various sensors, control devices, and communication protocols. It covers network traffic and system logs from normal operating conditions to various types of malicious behavior, including 14 attack categories, such as normal traffic, ICMP distributed denial-of-service attack (DDoS_ICMP), SQL injection attack (SQL_injection), vulnerability scanning (Vulnerability_scanner), HTTP distributed denial-of-service attack (DDoS_HTTP), password brute-force attack (Password), TCP distributed denial-of-service attack (DDoS_TCP), malicious upload (Uploading), backdoor attack (Backdoor), port scanning (Port_Scanning), cross-site scripting attack (XSS), UDP distributed denial-of-service attack (DDoS_UDP), ransomware, and fingerprinting. Compared to traditional IoT datasets, the Edge-IIoTset dataset is more representative and valuable for research in terms of attack diversity, industrial protocol coverage, and edge node behavior modeling.

[0173] In this embodiment, 20% of the data in the Edge-IIoTset dataset was selected for the experiment. The statistical results of the sample quantity distribution of various attack categories in the Edge-IIoTset dataset are shown in Table 1, and the dataset was divided according to the data preprocessing steps described in step S1.

[0174] Table 1: Distribution of Sample Quantity for Attack Categories

[0175]

[0176] According to the method proposed in this invention, the experiment mainly includes two tasks: the first task is to fine-tune the traffic generation model corresponding to step S2, which aims to train the generation model GPT-2 to learn the sequence generation rules of industrial Internet traffic; the second task is to fine-tune the anomaly classification model corresponding to step S3, which aims to train the classification model DistilBERT to accurately determine whether data packets are abnormal.

[0177] For fine-tuning the generative model GPT-2 in Task 1, the experiment was set with 5 epochs, a batch size of 8, and a gradient accumulation step of 1. The initial learning rate was set to... This is combined with cosine learning rate scheduling. Logs are saved step-by-step during training, and training and validation loss curves are output, such as... Figure 3 As shown.

[0178] Depend on Figure 3 It can be seen that both the training loss and validation loss of the generative model GPT-2 steadily decrease with the increase of training steps and eventually converge, without the overfitting phenomenon of a significant rebound in validation loss. This indicates that the model effectively learns the sequence generation rules of industrial internet traffic data. This result verifies the feasibility of the "generation" stage of the traffic sequence in this invention.

[0179] For fine-tuning the DistilBERT classification model for Task 2, the experiment was set with 4 training epochs, a batch size of 8, and gradient accumulation of 4. The initial learning rate was set to... Similarly, a cosine learning rate scheduling is used. To address the common data imbalance problem in industrial scenarios and enhance the model's learning ability on difficult samples, the Focal Loss loss function is adopted, where the modulation factor... The system was set to version 2.0. An early stopping strategy was also implemented during training, and precision, recall, F1 score, and confusion matrix were used as the primary evaluation metrics. Experimental results for this task can be found in [link to relevant documentation]. Figure 4 , Figure 5 And Table 2.

[0180] Table 2: Evaluation Metrics for DistilBERT Classification Model

[0181]

[0182] The experimental results of the classification model are as follows Figure 4 , Figure 5 And as shown in Table 2. According to Figure 4 The results show that the training and validation losses of the DistilBERT classification model rapidly decrease and converge after the second round; meanwhile, the accuracy stabilizes above 0.98. The model evaluation metrics in Table 2 further confirm the model's performance, with accuracy, precision, recall, and F1 score all exceeding 0.983. Furthermore, Figure 5 The confusion matrix of the multi-class test set shows that the vast majority of samples are correctly classified, and there are very few misclassified samples off the diagonal (the values ​​on the diagonal represent the number of correctly classified samples, and the values ​​off the diagonal represent the number of misclassified samples). This indicates that the classification model DistilBERT trained in this invention has high accuracy and excellent F1 score, and can provide reliable and efficient technical support for the anomaly "discrimination" process.

[0183] In summary, the experimental results of the method of this invention verify that both the generative model GPT-2 and the classification model DistilBERT can achieve the expected performance indicators through fine-tuning training on the target dataset, providing key support for the overall feasibility of the "generate first, then discriminate" proactive intrusion detection framework proposed in this invention.< / eos> < / pad> < / pad> < / eos> < / pad> < / pad> < / pad> < / eos> < / bos> < / eos> < / bos> < / eos> < / bos>

Claims

1. An industrial internet network intrusion detection method based on a pre-trained large language model, characterized in that, Includes the following steps: Step S1: Perform protocol-adaptive flow aggregation and session segmentation on the original network traffic data, construct a five-tuple information including source IP address, destination IP address, source port, destination port, and protocol type, and obtain the timestamp. Then, aggregate and group the original network traffic data based on the five-tuple information and timestamp to obtain the aggregated dataset. Then, feature filtering and feature encoding are performed on the aggregated dataset, and a streaming text dataset for training the generative model GPT-2 and a bag-level classification text-label dataset for training the classification model DistilBERT are constructed respectively. Step S2: Load the pre-trained weights of the generative model GPT-2, add control lexical units and adjust the embedding layer dimension of the generative model GPT-2. With the goal of causal language modeling, fine-tune and train the generative model GPT-2 on the streaming text dataset obtained in step S1 to obtain a traffic generation model that can predict the next timestamp data packet based on historical window data packets. Step S3: Load the pre-trained weights of the classification model DistilBERT, construct a task architecture for single-packet multi-classification, and fine-tune the classification model DistilBERT on the packet-level classification text-label dataset obtained in step S1 using the Focal Loss loss function and early stopping strategy to obtain a classification model capable of packet-by-packet anomaly detection. Step S4: Using the first L data packets of the real network flow at the current timestamp as the initial historical window, a prediction framework is formed using the generative model GPT-2 trained in step S2 and the classification model DistilBERT trained in step S3. First, the trained generative model GPT-2 is called to generate traffic sequences and obtain predicted data packets. Then, the trained classification model DistilBERT is called to perform anomaly detection on each sequence data in the predicted data packets and output the classification results of the anomaly detection, thus realizing proactive intrusion detection that first generates and then detects.

2. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S11: First, protocol-adaptive flow aggregation and session segmentation are performed on the original network traffic data through protocol inference, IP address normalization, L4 port construction, and timestamp resolution to construct a five-tuple information containing source IP address, destination IP address, source port, destination port, and protocol type. Then, the five-tuple information is sorted to construct a direction-independent session key Key as a session identifier. Based on the session key Key, the data packets of the original network traffic data are aggregated and grouped, with data packets having the same session key Key grouped into the same data group, aggregating them into a network flow containing bidirectional communication content. A protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet is calculated. Based on the difference Perform session segmentation on the network flow and calculate the difference. Greater than the preset session timeout threshold for the corresponding protocol type The data packet is determined to be the start of a new session, and a session segment number S is assigned to each session segment. segment The network flow is divided into multiple sessions; finally, each session is assigned a session key (Key) and a session segment number (S). segment A globally unique flow ID is assigned to each shared, unique session, and a network flow defined by the flow ID is constructed. To obtain the aggregated dataset; Step S12: Perform feature filtering and feature encoding on the aggregated dataset, remove unstructured load features, single constant value features, and sparse features, and encode discrete features and categorical features numerically, convert all features into numerical format, and obtain the filtered dataset composed of D-dimensional feature columns. Step S13: Obtain the set of all stream IDs generated in step S11. , will set After random shuffling, the data is divided into four mutually exclusive ID subsets in a 7:1:1:1 ratio. Based on the flow IDs in the four mutually exclusive ID subsets, the network flows in the dataset filtered in step S12 are... Extract and group the data respectively to construct mutually exclusive training, validation, test, and generation sets at the stream level; then, combine each network stream in each dataset. The data packets are arranged in chronological order, and all feature values ​​in each data packet are concatenated into a text string. The same network flow Text strings of all data packets Connect them using newlines in chronological order, and in the network stream. Add a stream start marker at the beginning of the network stream. Add a stream end marker at the end position of each network stream. Convert it into a long text sequence to obtain a streaming text dataset for training the generative model GPT-2; Step S14: Construct classification samples on a per-data-packet basis, concatenating the features from each data packet in the filtered dataset obtained in step S12 into a text string. Simultaneously, the attack category feature columns in the original network traffic data are mapped to multi-classification label IDs; each text string is... Combined with the corresponding tag ID, a package-level classification sample is constructed, containing text strings. The package-level sample dataset with two features, label ID and label ID, is obtained by stratified sampling of the package-level sample dataset and dividing it into a package-level training set, validation set and test set in a ratio of 8:1:1, resulting in a package-level classification text-label dataset in CSV format for training the DistilBERT classification model.

3. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 2, characterized in that, In step S11, the session key Key is defined as follows: ， In the formula, Proto represents the protocol type. Indicates the source IP address. Indicates the destination IP address. Indicates the source port. Indicates the destination port; This means taking the minimum value between two elements. This means taking the maximum value of two elements.

4. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 3, characterized in that, In step S11, a protocol-adaptive session timeout threshold is defined, and the difference between the timestamp of each data packet in the network flow and the timestamp of the previous adjacent data packet is calculated. According to the difference The specific method for session segmentation of network flows is as follows: Based on the protocol type (Proto), define a protocol-adaptive session timeout threshold. ; Data packets within each network flow are timestamped Sort the data packets in ascending order; traverse the sorted data packets and calculate the... timestamp of each data packet With the timestamp of each data packet The difference between , If the difference Session timeout threshold greater than the corresponding protocol type Proto Then determine the timestamp The corresponding data packet marks the start of a new session and is a timestamp. The corresponding data packet is assigned a new session segment number S segment This divides the network flow into multiple sessions.

5. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 4, characterized in that, Step S2 specifically includes the following steps: Step S21: Load the pre-trained weight file of the generative model GPT-2; the generative model GPT-2 includes a BPE tokenizer, a decoder, and an output layer. The decoder includes an embedding layer and an N-layer Transformer decoder block. The embedding layer includes a word embedding layer and a position embedding layer. The Transformer decoder block consists of a stacked first-layer normalization layer, a multi-head self-attention mechanism, a first-layer residual connection, a second-layer normalization layer, a feedforward neural network, and a second-layer residual connection; the output layer consists of a language modeling head; the BPE tokenizer is used to convert textual data packet strings into a sequence of numeric IDs that the model can understand, and to decode the ID sequence output by the model back into text; the generative model GPT-2 is used for text generation tasks; Step S22: Add control terms to expand the vocabulary of the BPE segmenter and adjust the dimension of the weight matrix of the embedding layer according to the sequence length; the control terms include flow start marker, flow end marker and padding marker, which are used to indicate the start and end of the network flow and to unify the sequence length in batch training. Step S23: Use the BPE tokenizer to analyze each network stream in the streaming text dataset constructed in step S13. The long text sequence is segmented into words to separate the text strings in the long text sequence. The process is decomposed into a series of lexical units, and each lexical unit is mapped to its corresponding semantically meaningful numerical ID in the vocabulary to generate an input lexical index sequence. Based on this sequence, an initial input attention mask sequence is generated. Then, by controlling the length of all sequences, a consistent input lexical index sequence and input attention mask sequence are obtained. Next, a prediction target sequence is constructed based on the input lexical index sequence and input attention mask sequence to calculate the loss function of the generative model GPT-2. Finally, the prediction target sequence, the input lexical index sequence, and the input attention mask sequence are fed into the generative model GPT-2 as input sequences. Step S24, generate the lexical embedding layer of model GPT-2 by taking each lexical index from the input lexical index sequence generated in step S23. Mapped to word embedding vectors Obtain word embedding vector sequence ;definition The length of the input sequence in the current batch, and the position embedding layer is the position from 0 to 1 in the input sequence. Each lexical at a given position generates a corresponding position index, and all position indices are mapped to a fixed-dimensional position embedding vector. Finally, the position embedding vector is used. With word embedding vector sequence By adding elements together, we obtain the input representation that integrates semantic information and temporal location information. ; Step S25, represent the input The output representation of the decoder is obtained by feeding it into the N-layer Transformer decoder block of the generative model GPT-2. The dimension of the decoder's output representation is projected onto the dimension of the vocabulary of the expanded BPE word segmenter through the language modeling head of the output layer of the generative model GPT-2 to obtain the log odds vector. The probability distribution of each word in the vocabulary as the word of the next timestamp is calculated based on the log odds vector. Step S26: The cross-entropy loss function is used to optimize the generator model GPT-2. The generator model GPT-2 is fine-tuned and trained so that it learns the sequence generation rules of network traffic data packets. The trained generator model GPT-2 is obtained as the final traffic generation model used to generate predicted traffic sequences.

6. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 5, characterized in that, Step S3 specifically includes the following steps: Step S31: Load the pre-trained weight file of the classification model DistilBERT; the classification model DistilBERT includes a WordPiece word segmenter, an encoder, and an output layer. The encoder of the classification model DistilBERT consists of an embedding layer and N stacked bidirectional Transformer encoder blocks. The embedding layer includes a word embedding layer and a position embedding layer; the output layer of the classification model DistilBERT consists of a fully connected classification head; set the number of output categories of the fully connected classification head to the total number of categories of the classification label, and use the mapped values ​​to represent each category. Step S32: Configure specific control units for the WordPiece word segmenter. The specific control units include classification tags for aggregating global semantic information, delimiters for identifying the end of text, and padding characters for unifying sequence length. Step S33: Use the WordPiece tokenizer to segment the text string constructed in step S14. The input is converted into a classification input tensor, which includes a classification term index sequence and a classification attention mask sequence. The lengths of all sequences in the classification input tensor are standardized, and text strings exceeding the maximum sequence length are excluded. Truncate text strings that are shorter than the maximum sequence length. Use fill characters to fill; Step S34: The classification input tensor is fed into the embedding layer of the classification model DistilBERT for processing. The word embedding layer maps the classification word index sequence to the classification word embedding vector, and the position embedding layer generates the classification position embedding vector corresponding to the classification word embedding vector. The classification word embedding vector and the classification position embedding vector are added element by element to obtain the classification input representation that integrates semantic and position information. Step S35: The classification input representation is fed into an N-layer stacked bidirectional Transformer encoder block for processing. The bidirectional attention mechanism is used to calculate global context information and obtain a hidden state vector containing the complete semantics of the data packet. Step S36: Input the hidden state vector output by the last bidirectional Transformer encoder block into the fully connected classification head of the output layer, project the features into the class space, and output the probability distribution of the data packet belonging to each class label through the Softmax function; Step S37: Construct the Focal Loss function suitable for imbalanced samples. By introducing a modulation factor, reduce the weight of easily classified samples and improve the classification performance of the DistilBERT model on imbalanced data. Step S38: The AdamW optimizer is used to calculate the gradient of the loss function and update the parameters of the classification model DistilBERT through the backpropagation algorithm. An early stopping strategy is introduced to select the optimal model and obtain the trained classification model DistilBERT, which is used as the final classification model for anomaly detection.

7. The industrial internet network intrusion detection method based on a pre-trained large language model according to claim 6, characterized in that, Step S4 specifically includes the following steps: Step S41: First, extract the preceding data from the real network stream to be detected. The data packets are used as initial historical data packets; the D-dimensional feature columns obtained after filtering the initial historical data packets in step S12 are converted into text strings according to the method described in step S13, and concatenated in chronological order to form an initial historical window; the initial historical window is used as the initial input and fed into the generative model GPT-2 trained in step S2 to start the process of generating the predicted traffic sequence. Step S42: After starting the process of generating the predicted traffic sequence, an iterative loop is performed. The last L data packets of the current existing data packets, including the initial historical data packets and the generated data packets, are always taken as new prompt inputs to form a context sliding window for the iterative loop. Based on the context sliding window, the generation model GPT-2 autoregressively generates the text string of the predicted data packets for the next timestamp. Step S43: The text string of the generated single prediction data packet is sent to the DistilBERT classification model trained in step S3 to perform real-time analysis on the prediction data packet, output the classification result of anomaly detection and record it.

Images