Encryption traffic classification method and system based on generative pre-training language model and low-rank adaptation
Patent Information
- Application Number
- CN202611253236.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-03
AI Technical Summary
Existing technologies for encrypted traffic classification suffer from several problems: full-scale fine-tuning schemes are difficult to deploy on consumer-grade hardware; efficient parameter fine-tuning strategies have not been systematically studied; preprocessing pipelines lack multi-scenario compatibility; engineering systems have not been formed; and generative models have not fully leveraged the advantages of sequence modeling.
An encrypted traffic classification method based on generative pre-trained language models and low-rank adaptation is adopted. By adaptively recognizing directory layout, reading streaming data packets, correcting low-rank matrices, and loosely coupling modules, an end-to-end system is constructed to achieve efficient parameter training and real-time classification.
It enables the running of large language models on consumer-grade hardware, reduces storage and GPU memory overhead, provides multi-scenario compatibility and engineering design, improves classification accuracy and efficiency, and shortens the optimization cycle.
Smart Images
Figure CN123035006A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of traffic encryption technology, specifically to an encrypted traffic classification method and system based on generative pre-trained language models and low-rank adaptation. Background Technology
[0002] Encrypted traffic classification refers to the process of fine-grained identification and classification of network data streams encrypted with protocols such as SSL / TLS, utilizing technologies such as machine learning, deep learning, and fingerprinting. As network traffic encryption becomes commonplace, traditional Deep Packet Inspection (DPI) technology has become ineffective due to its inability to parse ciphertext payloads. This technology instead analyzes side-channel metadata such as characteristic fields from the handshake phase, packet length sequences, arrival time intervals, and directional statistical characteristics. By constructing multi-dimensional feature models and intelligent classification algorithms, it can accurately distinguish various application services such as video streams, web browsing, and P2P downloads without decrypting the content, and further identify malicious software communications and abnormal traffic. This technology aims to balance user privacy protection and network management needs, and is a key means to ensure network bandwidth quality, achieve refined operation, and enhance network security situational awareness.
[0003] However, existing technologies have the following technical shortcomings: full fine-tuning schemes rely on data center-grade GPUs, making them difficult to deploy on consumer-grade hardware; the optimal configuration of parameter-efficient fine-tuning strategies in the field of traffic classification has not been systematically studied; the preprocessing pipeline lacks multi-scenario compatibility and often causes memory overflow due to full loading of pcap files; existing academic solutions are mostly scattered code prototypes and have not formed end-to-end engineering systems; and existing GPT-2-based solutions have only initially introduced generative models and have not been deeply integrated with parameter-efficient adaptation strategies to fully leverage their sequence modeling advantages. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for classifying encrypted traffic based on a generative pre-trained language model and low-rank adaptation.
[0005] The objective of this invention is achieved through the following technical solution:
[0006] In the first aspect, this application discloses a method for classifying encrypted traffic based on a generative pre-trained language model and low-rank adaptation, comprising the following steps:
[0007] S1. Adaptively identify the directory layout and construct the tag mapping from the original packet capture file pcap. Aggregate packets bidirectionally according to the communication quintuple, extract the first M bytes of payload of the first N packets of each stream, and output the stream-level byte sequence.
[0008] S2. Receive the stream-level byte sequence and convert it into a hexadecimal space-separated string. Then, use the BPE tokenizer to convert the hexadecimal space-separated string into a fixed-length token ID sequence and attention mask. The BPE tokenizer is pre-trained using GPT-2.
[0009] S3. Load GPT-2-small pre-trained weights and freeze them throughout. Calculate the hidden states corresponding to each token batch through 12 Transformer layers and pass the hidden state sequence of the last layer to the adaptation layer.
[0010] S4. Inject low-rank matrix pairs into the backbone linear layer of the GPT-2-small pre-trained weights, correct the forward calculation, train the low-rank parameters and the classification head, and keep the backbone weights unchanged.
[0011] S5. Based on the attention mask, extract the hidden vector of the last valid token of each sample from the forward-computed corrected last hidden state as the global feature of the entire stream.
[0012] S6. Map the global features to class probabilities using a linear classification head and output the classification results; simultaneously calculate the classification and efficiency metrics and save the optimal checkpoint.
[0013] S7. Treat steps S1-S6 as an atomic process, automatically traverse hyperparameter combinations, summarize the performance of each configuration, and output the optimal configuration reference.
[0014] S8. Loosely couple steps S1-S6 into an end-to-end system in a modular manner, and start an independent resource monitoring thread. During the inference process, use the optimal checkpoint to perform the preprocessing and forward link of steps S1-S6 on the new packet capture file pcap, and finally output the real-time classification result.
[0015] Based on the first aspect, step S1 specifically includes the following steps:
[0016] S11. Automatically detect three types of layouts from the pcap directory of the raw data packet capture file, including single-level subdirectories, nested directories, and CSV annotation files, and build a category label mapping table based on the three types of layouts;
[0017] S12. A streaming data packet reader, PcapReader, is used to read data packets file by file. That is, after each file is processed, the streaming buffer is refreshed to control memory peaks.
[0018] S13. For each data packet, extract its communication quintuple and perform bidirectional normalization to ensure that bidirectional communication data packets are classified into the same flow FF. The communication quintuple includes source IP, destination IP, source port, destination port and transport layer protocol. The bidirectional normalization includes sorting the source IP and destination IP in lexicographical order and sorting the source port and destination port in numerical order.
[0019] S14. Concatenate the first N data packets within the same stream, and the first M bytes of payload of each packet, into a stream-level byte sequence. .
[0020] Based on the first aspect, step S2 specifically includes:
[0021] S21. Using encoding functions The stream-level byte sequence Convert each byte to a 2-digit hexadecimal character, separating bytes with spaces, to generate a Hex space-separated string. ;
[0022] S22. Using the vocabulary and merging rules of the original GPT-2 pre-trained word segmenter, call... function pairs Perform word-level encoding; generate a standardized token ID sequence by truncating or padding to a fixed sequence length L. and its corresponding attention mask, where, Let L represent an integer vector space of length L.
[0023] Based on the first aspect, step S3 specifically includes:
[0024] Load GPT-2-small pre-trained weights from the local path; receive the generated batched token ID sequence. With attention masks, causal self-attention computation is performed layer by layer, finally outputting the sequence of hidden states in the last layer. Among them, during the entire training process, all the original weight matrices of GPT-2-small Completely frozen, not participating in gradient updates, and used only as a fixed feature extractor.
[0025] Based on the first aspect, step S4 specifically includes:
[0026] Inject low-rank decomposition matrix pairs onto the specified weight matrix of the backbone linear layer of the GPT-2-small pre-trained weights. The specified weight matrix includes the attention projection matrix, output projection matrix, and feedforward network matrix of all Transformer layers.
[0027] For the original weight matrix Its forward computation is corrected to: ,in, The scaling factor is represented by r, which represents the low-rank adaptive LoRA rank. and Indicates trainable parameters;
[0028] The corrected forward computation is embedded into each layer of the Transformer in step S3, and the original path is... With adaptation path Perform summation and output the last hidden state after low-rank adaptive LoRA correction. .
[0029] Based on the first aspect, step S5 specifically includes:
[0030] S51, Receive the last hidden state after the low-rank adaptive LoRA correction. Based on the attention mask generated in step S2, the index of the last valid token in each sample is located.
[0031] S52. Using extraction strategies, from Extract the 768-dimensional hidden state vector corresponding to the position of the last valid token. This serves as the global feature of the entire network flow FF; the extraction strategy utilizes the autoregressive properties of generative models, aggregating all preceding context information through the token at the end of the sequence; and the 768-dimensional hidden state vector... It serves as the input to the linear classification head in step S6, and also provides a computational path for the gradient backpropagation of the hidden state vector during training.
[0032] Based on the first aspect, step S6 specifically includes:
[0033] S61, using the linear classification head weight matrix The 768-dimensional hidden state vector The original score vector is mapped to C dimensions, and after softmax normalization, the class with the highest probability is taken as the classification decision.
[0034] ;in, and Indicates category index, Represents the linear classifier head weight matrix The row vectors Represents the linear classifier head weight matrix The row vectors The predicted category output by the model. Indicates transpose;
[0035] S62. Jointly optimize the linear classifier head with the low-rank adaptive LoRA parameters from step S4. After each epoch, calculate the accuracy, macro-average / F1, precision, and recall on the test set according to the class label mapping table, and record the training time and inference latency. If the current validation F1 is better than the historical best, save the corresponding low-rank adaptive LoRA parameter weights and linear classifier head weights to the local checkpoint.
[0036] Based on the first aspect, step S7 specifically includes:
[0037] Based on steps S1-S6, an atomic task including a training-validation process is constructed, and a unified ablation experiment scheduler is built to automatically traverse four hyperparameter dimensions, specifically including:
[0038] Low-rank adaptive LoRA rank Injection location combination Sequence length and number of packets in the stream ,in, This indicates that only the attention projection matrix is injected at the injection location. The injection location represents the injection attention and output projection matrix. This indicates that the injection location is all linear layers;
[0039] Each configuration triggers a complete S1→S6 link and uses a fixed random seed 42; before running, the backbone network is reset to the initial pre-training state through the loading interface of step S3;
[0040] Finally, the classification and efficiency metrics of each configuration are summarized, and the impact trend of each hyperparameter dimension on performance and the optimal configuration reference are output.
[0041] Based on the first aspect, step S8 specifically includes:
[0042] Steps S1-S6 are loosely coupled into an end-to-end system as the main process, with modules connected through standard PyTorch tensors and DataLoader interfaces.
[0043] The independent resource monitoring thread includes parallel collection of CPU utilization, system memory usage, GPU utilization, and video memory usage at preset time intervals;
[0044] The independent resource monitoring thread runs asynchronously with the main process, and automatically generates a resource time-series line chart after the task is completed.
[0045] The end-to-end system loads the optimal checkpoint, reuses the GPT-2-small pre-trained weights in step S3, performs preprocessing and forward linking steps S1-S6 on the new packet capture file pcap, and finally outputs the real-time classification result.
[0046] Secondly, this application discloses an encrypted traffic classification system based on a generative pre-trained language model and low-rank adaptation, which utilizes the aforementioned encrypted traffic classification method based on a generative pre-trained language model and low-rank adaptation, including:
[0047] The stream-level payload extractor adaptively identifies three directory layouts and constructs a label mapping through the pcap input port of the packet capture file. Its stream-level byte sequence output after bidirectional normalization of the 5-tuple is connected to the input of the Hex encoder, while the label mapping table output is connected to the label verification input of the classification index evaluator.
[0048] The Hex encoder converter is used to convert the received byte sequence into a Hex space-separated string, and its output is connected to the input of the BPE tokenizer.
[0049] The BPE segmenter is used to convert Hex space-separated strings into a fixed-length token ID sequence and attention mask by a GPT-2 pre-trained BPE segmenter. Its token output and mask output are connected to the standardized input interface of the GPT-2-small backbone network.
[0050] The GPT-2-small backbone network is loaded with 124M parameters of frozen pre-trained weights, and the output of the hidden state sequence of the last layer is connected to the input of the first LoRA low-rank adapter.
[0051] The first LoRA low-rank adapter injects low-rank decomposition matrix pairs into the attention projection matrix, output projection matrix, and feedforward network matrix linear layers of the backbone network for corrected forward computation. Its corrected last-layer hidden state output is connected to the input of the last-position token feature extractor.
[0052] The last token feature extractor extracts the hidden state vector of the last non-padded token position of each sample based on the attention mask, and its stream-level feature vector output is connected to the input of the linear classification head.
[0053] The linear classification head maps feature vectors to the original scores logits of each category and outputs classification decisions after softmax normalization. Its prediction output is connected to the prediction input interface of the classification index evaluator, and its optimal checkpoint storage path is connected to the loading interface of the system inference deployment stage.
[0054] The classification metric evaluator and the efficiency metric evaluator receive the prediction results and true labels from the linear classification head in parallel. The classification metric evaluator outputs accuracy, macro-average F1, weighted F1, precision, and recall to the system evaluation report interface, while the efficiency metric evaluator outputs training time, inference latency, number of trainable parameters, and peak memory usage to the same evaluation report interface.
[0055] The real-time resource monitor runs in an independent thread. Its CPU sampling end, memory sampling end, GPU utilization sampling end and video memory sampling end collect system resource usage in parallel. The time-series data output end is connected to the monitoring chart generation end of the evaluation report interface.
[0056] The second LoRA low-rank adapter has its trainable parameter output and the trainable parameter output of the linear classification head connected together to the parameter update input of the AdamW optimizer during the training phase. After the adapter weights and classification head weights updated by the AdamW optimizer are evaluated by the validation set F1, they are selectively back-connected to the checkpoint saving interface of the first LoRA low-rank adapter and the checkpoint saving interface of the linear classification head for seamless loading during the inference phase.
[0057] The beneficial effects of this invention are:
[0058] 1) This application introduces the standard LoRA low-rank adaptation technique, compressing the number of trainable parameters of the GPT-2-small backbone network from a scale requiring full fine-tuning to an extremely low level. This allows large language model training tasks, which originally relied on data center-grade GPUs, to run completely on consumer-grade hardware. Simultaneously, the original weights of the backbone network remain frozen throughout training, requiring only the storage of a small number of parameter copies of the low-rank matrix and classification head. This significantly reduces storage and GPU memory overhead during model deployment. This frees the encryption traffic classification scheme based on large language models from dependence on expensive computing resources, making it practically feasible to extend to edge devices and terminal environments.
[0059] 2) This application constructs a four-dimensional hyperparameter ablation experimental framework to examine the interactive effects of different configuration dimensions on classification performance in low-rank adaptation, revealing the independent roles and coupling rules of each hyperparameter in encrypted traffic classification tasks. This eliminates the need for subsequent similar solutions to blindly try various parameter combinations from scratch, allowing them to directly refer to the optimal configuration range provided by this method for engineering selection. The ablation analysis results provide the first efficient parameter adaptation benchmark for generative pre-trained language models in the field of traffic classification, transforming hyperparameter selection from an experience-driven trial-and-error process into an evidence-based systematic decision-making process, significantly shortening the optimization cycle before solution implementation and reducing human resource costs.
[0060] 3) The preprocessing pipeline in this application has the ability to automatically identify three different pcap directory layouts, and can adaptively construct category label mappings without requiring users to manually specify the annotation method, covering most file organization forms in actual network acquisition scenarios. Meanwhile, the preprocessing stage adopts a streaming reading mechanism, processing file by file and releasing used cache immediately, fundamentally avoiding the memory overflow problem caused by the traditional full-load method. This allows this method to easily handle the processing needs of large-scale traffic datasets. Even on ordinary workstations with limited single-machine memory resources, it can complete the preprocessing tasks of tens of thousands or even hundreds of thousands of pcap files, and data scale is no longer a barrier to the application of this solution.
[0061] 4) This application integrates modules such as streaming preprocessing, adaptive data loading, efficient parameter training and inference, multi-dimensional performance evaluation, and real-time resource monitoring into a loosely coupled complete system. During training and inference, it automatically records classification metrics and resource consumption data, and generates a complete evaluation report and resource time-series chart after the task is completed, providing a fully traceable basis for solution verification and performance tuning. This engineering design significantly reduces the conversion cost from prototype verification to production deployment.
[0062] 5) This application employs the GPT-2-small generative pre-trained language model as its backbone network. Its Transformer Decoder architecture, through large-scale autoregressive pre-training, achieves deep modeling capabilities for long-range dependencies in sequence data. This architectural design enables the model to fully utilize the semantic relationships between consecutive positions in the byte sequence when analyzing encrypted traffic payloads, capturing implicit temporal patterns that are difficult to extract using traditional encoder architectures or LSTM and CNN schemes. Especially for encrypted streams with long payloads and complex structures, the generative model, with its unidirectional causal modeling characteristics, can more accurately grasp the statistical regularities gradually accumulated in the complete communication context, thereby providing a more reliable feature basis for classification decisions and improving classification accuracy in complex scenarios. Attached Figure Description
[0063] Figure 1 This is a flowchart illustrating the encrypted traffic classification method based on a generative pre-trained language model and low-rank adaptation, according to an embodiment of the present invention.
[0064] Figure 2 This is a schematic diagram of the data preprocessing process for the encrypted traffic classification method based on generative pre-trained language models and low-rank adaptation, according to an embodiment of the present invention.
[0065] Figure 3 This is a schematic diagram of the model training process for the encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation, according to an embodiment of the present invention.
[0066] Figure 4 This is a schematic diagram of the inference process of the encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation in an embodiment of the present invention. Detailed Implementation
[0067] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0068] See Figure 1-4 This application discloses an encrypted traffic classification method based on generative pre-trained language models and low-rank adaptation. By introducing standard LoRA low-rank adaptation technology, the number of trainable parameters of the GPT-2-small backbone network is compressed from a scale of full fine-tuning to an extremely low level, enabling large language model training tasks that originally relied on data center-grade GPUs to run completely on consumer-grade hardware. Simultaneously, the original weights of the backbone network remain frozen throughout training, requiring only the storage of a small number of parameter copies of the low-rank matrix and classification head, which significantly reduces storage and GPU memory overhead during model deployment. This improvement frees encrypted traffic classification schemes based on large language models from dependence on expensive computing resources, making them practically feasible for promotion to edge devices and terminal environments, truly bringing the semantic modeling capabilities of large models from research scenarios to practical engineering applications. Specifically, it includes the following steps:
[0069] S1. Adaptively identify the directory layout and construct the tag mapping from the original packet capture file pcap, aggregate the packets bidirectionally according to the communication quintuple, extract the first M bytes of payload of the first N packets of each stream, and output the stream-level byte sequence (for subsequent encoding).
[0070] S2. Receive the stream-level byte sequence and convert it into a hexadecimal space-separated string. Then, use the BPE tokenizer to convert the hexadecimal space-separated string into a fixed-length token ID sequence and attention mask. The BPE tokenizer is pre-trained using GPT-2.
[0071] S3. Load GPT-2-small pre-trained weights and freeze them throughout. Calculate the hidden states corresponding to each token batch through 12 Transformer layers and pass the hidden state sequence of the last layer to the adaptation layer.
[0072] S4, in the backbone linear layer of the GPT-2-small pre-trained weights (e.g., attention projection matrix c), attn Output projection matrix c projFeedforward network matrix c fc Inject low-rank matrix pairs into the classifier, correct the forward computation, and train the low-rank parameters and the classifier head while keeping the backbone weights unchanged.
[0073] S5. Based on the attention mask, extract the hidden vector of the last valid token of each sample from the forward-computed corrected last hidden state as the global feature of the entire stream.
[0074] S6. Map the global features to class probabilities using a linear classification head and output the classification results; simultaneously calculate the classification and efficiency metrics and save the optimal checkpoint.
[0075] S7. Treat steps S1-S6 as an atomic process, automatically traverse hyperparameter combinations (rank, injection position, sequence length, number of packets), summarize the performance of each configuration, and output the optimal configuration reference.
[0076] S8. Loosely couple steps S1-S6 into an end-to-end system in a modular manner, and start an independent resource monitoring thread. During the inference process, use the optimal checkpoint to perform the preprocessing and forward link of steps S1-S6 on the new packet capture file pcap, and finally output the real-time classification result.
[0077] For example, step S1 specifically includes the following steps:
[0078] S11. Automatically detect three types of layouts from the pcap directory of the raw data packet capture file, including single-level subdirectories, nested directories, and CSV annotation files, and build a category label mapping table based on the three types of layouts;
[0079] S12. A streaming data packet reader, PcapReader, is used to read data packets file by file. That is, after each file is processed, the streaming buffer is refreshed to control memory peaks.
[0080] S13. For each data packet, extract its communication quintuple and perform bidirectional normalization to ensure that bidirectional communication data packets are classified into the same flow FF. The communication quintuple includes source IP, destination IP, source port, destination port and transport layer protocol. The bidirectional normalization includes sorting the source IP and destination IP in lexicographical order and sorting the source port and destination port in numerical order.
[0081] S14. Concatenate the first N (N=10 in this embodiment) data packets within the same stream, along with the first M (M=256 in this embodiment) bytes of payload from each packet, into a stream-level byte sequence. This stream-level byte sequence As input to step S2, the label mapping table is used for classification evaluation in step S6.
[0082] For example, step S2 specifically includes:
[0083] S21. Using encoding functions The stream-level byte sequence Convert each byte to a 2-digit hexadecimal character, separating bytes with spaces, to generate a Hex space-separated string. ;
[0084] S22. Using the vocabulary and merging rules of the original GPT-2 pre-trained word segmenter, call... function pairs Perform sub-word-level encoding; generate a standardized tokenID sequence by truncating or padding to a fixed sequence length L (L=1024 in this embodiment). and its corresponding attention mask (and the token ID sequence) (and their corresponding attention masks are divided into training and testing sets according to a preset ratio), where, Let L represent an integer vector space of length L.
[0085] In this step, the token ID sequence The attention mask is passed to the data loader (DataLoader) in step S3 in tensor form, and can be cached as a .pt file to avoid repeated preprocessing.
[0086] In steps S1-S2, the automatic identification capability for three different pcap directory layouts can adaptively construct category label mappings without requiring users to manually specify the annotation method, covering most file organization forms in actual network data collection scenarios. Simultaneously, a streaming reading mechanism is adopted, processing file-by-file and releasing used cache immediately, fundamentally avoiding the memory overflow problem caused by traditional full-load methods. This allows this method to easily handle the processing needs of large-scale traffic datasets. Even on ordinary workstations with limited single-machine memory resources, it can complete preprocessing tasks for tens of thousands or even hundreds of thousands of pcap files, eliminating data scale as a barrier to application.
[0087] For example, step S3 specifically includes:
[0088] Load GPT-2-small pre-trained weights from the local path; receive the generated batched token ID sequence. With attention masks, causal self-attention computation is performed layer by layer, finally outputting the sequence of hidden states in the last layer. (its dimensions are) ), where, throughout the entire training process, all the original weight matrices of GPT-2-small ( express OK The column's real matrix space is completely frozen and does not participate in gradient updates; it is only used as a fixed feature extractor.
[0089] The freezing strategy for this step is established before the low-rank adaptive LoRA injection in step S4.
[0090] In this step, the GPT-2-small generative pre-trained language model is used as the backbone network. Its Transformer Decoder architecture, through large-scale autoregressive pre-training, acquires the ability to deeply model long-range dependencies in sequence data. This architectural design enables the model to fully utilize the semantic relationships between consecutive positions in the byte sequence when analyzing encrypted traffic payloads, capturing implicit temporal patterns that are difficult to extract using traditional encoder architectures or LSTM and CNN schemes. Especially for encrypted streams with long payloads and complex structures, the generative model, with its unidirectional causal modeling characteristics, can more accurately grasp the statistical regularities gradually accumulated in the complete communication context, thus providing a more reliable feature basis for classification decisions and effectively improving classification accuracy in complex scenarios.
[0091] For example, step S4 specifically includes:
[0092] Inject low-rank decomposition matrix pairs onto the specified weight matrix of the backbone linear layer of the GPT-2-small pre-trained weights. The specified weight matrix includes the attention projection matrix, output projection matrix, and feedforward network matrix of all Transformer layers.
[0093] For the original weight matrix Its forward computation is corrected to: ,in, Indicates the scaling factor (in this embodiment) ), r represents the LoRA rank (in this embodiment r=4, i.e. Meanwhile, dropout is set to 0.1. and ( and They represent OK column sum OK The column space of real matrices represents the trainable parameters;
[0094] The corrected forward computation is embedded into each layer of the Transformer in step S3, and the original path is... With adaptation path Perform summation and output the last hidden state after low-rank adaptive LoRA correction. .
[0095] In this step, the number of trainable parameters is 0.3M, which are tracked by the optimizer along with the classification head from step S6.
[0096] For example, step S5 specifically includes:
[0097] S51, Receive the last hidden state after the low-rank adaptive LoRA correction. Based on the attention mask generated in step S2, the index of the last valid token in each sample is located.
[0098] S52. Using extraction strategies, from Extract the 768-dimensional hidden state vector corresponding to the position of the last valid token. This serves as the global feature of the entire network flow FF; the extraction strategy utilizes the autoregressive properties of generative models, aggregating all preceding context information through the token at the end of the sequence; and the 768-dimensional hidden state vector... It serves as the input to the linear classification head in step S6, and also provides a computational path for the gradient backpropagation of the hidden state vector during training.
[0099] For example, step S6 specifically includes:
[0100] S61, using the linear classification head weight matrix The 768-dimensional hidden state vector The original scores (logits) are mapped to a C-dimensional vector, then normalized using softmax, and the class with the highest probability is chosen as the classification decision.
[0101] ;in, and Indicates category index, Represents the linear classifier head weight matrix The row vectors Represents the linear classifier head weight matrix The row vectors The predicted category output by the model. Indicates transpose;
[0102] S62. Jointly optimize the linear classification head with the LoRA parameters from step S4 (in this embodiment, the AdamW optimizer is used, and the learning rate is set to...). (Weight decay is set to 0.01); after each epoch, the accuracy, macro average / F1, precision and recall are calculated on the test set according to the class label mapping table, and the training time and inference latency are recorded at the same time; if the current validation F1 is better than the historical best, the corresponding low-rank adaptive LoRA parameter weights and linear classifier head weights are saved to the local checkpoint (for inference in step S8).
[0103] For example, step S7 specifically includes:
[0104] Based on steps S1-S6, an atomic task including a training-validation process is constructed, and a unified ablation experiment scheduler is built to automatically traverse four hyperparameter dimensions, specifically including:
[0105] Low-rank adaptive LoRA rank Injection location combination Sequence length and number of packets in the stream ,in, This indicates that only the attention projection matrix is injected at the injection location. The injection location represents the injection attention and output projection matrix. This indicates that the injection location is all linear layers (attention projection matrix plus output projection matrix plus feedforward network);
[0106] Each configuration triggers a complete S1→S6 link using a fixed random seed of 42 (to ensure reproducibility); before running, the backbone network is reset to its initial pre-trained state via the loading interface in step S3 (to avoid parameter contamination).
[0107] Finally, the classification metrics (accuracy, macro average / weighted F1, precision, recall) and efficiency metrics (training time, inference latency, number of trainable parameters, peak memory usage) of each configuration are summarized, and the impact trend of each hyperparameter dimension on performance and the optimal configuration reference are output.
[0108] Experiments have found that, , , , It represents the optimal balance between accuracy and efficiency.
[0109] This step, through the construction of a four-dimensional hyperparameter ablation experimental framework, examined the interactive impact of different configuration dimensions on classification performance in low-rank adaptation, revealing the independent roles and coupling patterns of each hyperparameter in encrypted traffic classification tasks. This eliminates the need for subsequent similar solutions to blindly try various hyperparameter combinations from scratch, allowing for direct engineering selection based on the optimal configuration range provided by this method. The ablation analysis results provide the first efficient parameter adaptation benchmark for generative pre-trained language models in the field of traffic classification, transforming hyperparameter selection from an experience-driven trial-and-error process into an evidence-based systematic decision-making process, significantly shortening the optimization cycle before solution implementation and reducing human resource costs.
[0110] For example, step S8 specifically includes:
[0111] Steps S1-S6 are loosely coupled into an end-to-end system as the main process, with modules connected through standard PyTorch tensors and DataLoader interfaces.
[0112] The independent resource monitoring thread includes parallel collection of CPU utilization, system memory usage, GPU utilization, and video memory usage at preset time intervals (set to 1 second in this embodiment);
[0113] The independent resource monitoring thread runs asynchronously with the main process, and automatically generates a resource time-series line chart after the task is completed.
[0114] The end-to-end system loads the optimal checkpoint, reuses the GPT-2-small pre-trained weights in step S3, performs preprocessing and forward linking steps S1-S6 on the new packet capture file pcap, and finally outputs the real-time classification result to verify the feasibility of the solution on consumer-grade GPUs.
[0115] This step integrates modules such as streaming preprocessing, adaptive data loading, efficient parameter training and inference, multi-dimensional performance evaluation, and real-time resource monitoring into a loosely coupled complete system. These modules communicate through standardized data interfaces, ensuring end-to-end automation of the entire process while allowing developers to independently replace or upgrade any module according to actual needs. During training and inference, classification metrics and resource consumption data are automatically recorded, and a complete evaluation report and resource time-series chart are generated after the task is completed, providing a fully traceable basis for solution verification and performance tuning. This engineering design significantly reduces the conversion cost from prototype verification to production deployment, making this method not only an algorithmic solution but also a directly deployable software system.
[0116] For example, this application discloses an encrypted traffic classification system based on generative pre-trained language models and low-rank adaptation, which utilizes the aforementioned encrypted traffic classification method based on generative pre-trained language models and low-rank adaptation, including:
[0117] The stream-level payload extractor adaptively identifies three directory layouts and constructs a label mapping through the pcap input port of the packet capture file. Its stream-level byte sequence output after bidirectional normalization of the 5-tuple is connected to the input of the Hex encoder, while the label mapping table output is connected to the label verification input of the classification index evaluator.
[0118] The Hex encoder converter is used to convert the received byte sequence into a Hex space-separated string, and its output is connected to the input of the BPE tokenizer.
[0119] The BPE segmenter is used to convert Hex space-separated strings into a fixed-length token ID sequence and attention mask by a GPT-2 pre-trained BPE segmenter. Its token output and mask output are connected to the standardized input interface of the GPT-2-small backbone network.
[0120] The GPT-2-small backbone network is loaded with 124M parameters of frozen pre-trained weights, and the output of the hidden state sequence of the last layer is connected to the input of the first LoRA low-rank adapter.
[0121] The first LoRA low-rank adapter injects low-rank decomposition matrix pairs into the attention projection matrix, output projection matrix, and feedforward network matrix linear layers of the backbone network for corrected forward computation. Its corrected last-layer hidden state output is connected to the input of the last-position token feature extractor.
[0122] The last token feature extractor extracts the hidden state vector of the last non-padded token position of each sample based on the attention mask, and its stream-level feature vector output is connected to the input of the linear classification head.
[0123] The linear classification head maps feature vectors to the original scores logits of each category and outputs classification decisions after softmax normalization. Its prediction output is connected to the prediction input interface of the classification index evaluator, and its optimal checkpoint storage path is connected to the loading interface of the system inference deployment stage.
[0124] The classification metric evaluator and the efficiency metric evaluator receive the prediction results and true labels from the linear classification head in parallel. The classification metric evaluator outputs accuracy, macro-average F1, weighted F1, precision, and recall to the system evaluation report interface, while the efficiency metric evaluator outputs training time, inference latency, number of trainable parameters, and peak memory usage to the same evaluation report interface.
[0125] The real-time resource monitor runs in an independent thread. Its CPU sampling end, memory sampling end, GPU utilization sampling end and video memory sampling end collect system resource usage in parallel. The time-series data output end is connected to the monitoring chart generation end of the evaluation report interface.
[0126] The second LoRA low-rank adapter has its trainable parameter output and the trainable parameter output of the linear classification head connected together to the parameter update input of the AdamW optimizer during the training phase. After the adapter weights and classification head weights updated by the AdamW optimizer are evaluated by the validation set F1, they are selectively back-connected to the checkpoint saving interface of the first LoRA low-rank adapter and the checkpoint saving interface of the linear classification head for seamless loading during the inference phase.
[0127] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.
Claims
1. A method for classifying encrypted traffic based on generative pre-trained language models and low-rank adaptation, characterized in that, Includes the following steps: S1. Adaptively identify the directory layout and construct the tag mapping from the original packet capture file pcap. Aggregate packets bidirectionally according to the communication quintuple, extract the first M bytes of payload of the first N packets of each stream, and output the stream-level byte sequence. S2. Receive the stream-level byte sequence and convert it into a hexadecimal space-separated string. Then, use the BPE tokenizer to convert the hexadecimal space-separated string into a fixed-length token ID sequence and attention mask. The BPE tokenizer is pre-trained using GPT-2. S3. Load GPT-2-small pre-trained weights and freeze them throughout. Calculate the hidden states corresponding to each token batch through 12 Transformer layers and pass the hidden state sequence of the last layer to the adaptation layer. S4. Inject low-rank matrix pairs into the backbone linear layer of the GPT-2-small pre-trained weights, correct the forward calculation, train the low-rank parameters and the classification head, and keep the backbone weights unchanged. S5. Based on the attention mask, extract the hidden vector of the last valid token of each sample from the forward-computed corrected last hidden state as the global feature of the entire stream. S6. Map the global features to class probabilities using a linear classification head and output the classification results; simultaneously calculate the classification and efficiency metrics and save the optimal checkpoint. S7. Treat steps S1-S6 as an atomic process, automatically traverse hyperparameter combinations, summarize the performance of each configuration, and output the optimal configuration reference. S8. Loosely couple steps S1-S6 into an end-to-end system in a modular manner, and start an independent resource monitoring thread. During the inference process, use the optimal checkpoint to perform the preprocessing and forward link of steps S1-S6 on the new packet capture file pcap, and finally output the real-time classification result.
2. The encrypted traffic classification method based on generative pre-trained language models and low-rank adaptation as described in claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Automatically detect three types of layouts from the pcap directory of the raw data packet capture file, including single-level subdirectories, nested directories, and CSV annotation files, and build a category label mapping table based on the three types of layouts; S12. A streaming data packet reader, PcapReader, is used to read data packets file by file. That is, after each file is processed, the streaming buffer is refreshed to control memory peaks. S13. For each data packet, extract its communication quintuple and perform bidirectional normalization to ensure that bidirectional communication data packets are classified into the same flow FF. The communication quintuple includes source IP, destination IP, source port, destination port and transport layer protocol. The bidirectional normalization includes sorting the source IP and destination IP in lexicographical order and sorting the source port and destination port in numerical order. S14. Concatenate the first N data packets within the same stream, and the first M bytes of payload of each packet, into a stream-level byte sequence. .
3. The encrypted traffic classification method based on generative pre-trained language models and low-rank adaptation as described in claim 2, characterized in that, Step S2 specifically includes: S21. Using encoding functions The stream-level byte sequence Convert each byte to a 2-digit hexadecimal character, separating bytes with spaces, to generate a Hex space-separated string. ; S22. Using the vocabulary and merging rules of the original GPT-2 pre-trained word segmenter, call... function pairs Perform word-level encoding; generate a standardized token ID sequence by truncating or padding to a fixed sequence length L. and its corresponding attention mask, where, Let L represent an integer vector space of length L.
4. The encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation according to claim 3, characterized in that, Step S3 specifically includes: Load GPT-2-small pre-trained weights from the local path; receive the generated batched token ID sequence. With attention masks, causal self-attention computation is performed layer by layer, finally outputting the sequence of hidden states in the last layer. Among them, during the entire training process, all the original weight matrices of GPT-2-small Completely frozen, not participating in gradient updates, and only used as a fixed feature extractor.
5. The encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation according to claim 4, characterized in that, Step S4 specifically includes: Inject low-rank decomposition matrix pairs onto the specified weight matrix of the backbone linear layer of the GPT-2-small pre-trained weights. The specified weight matrix includes the attention projection matrix, output projection matrix, and feedforward network matrix of all Transformer layers. For the original weight matrix Its forward computation is corrected to: ,in, The scaling factor is represented by r, which represents the low-rank adaptive LoRA rank. and Indicates trainable parameters; The corrected forward computation is embedded into each layer of the Transformer in step S3, and the original path is... With adaptation path Perform summation and output the last hidden state after low-rank adaptive LoRA correction. .
6. The encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation according to claim 5, characterized in that, Step S5 specifically includes: S51, Receive the last hidden state after the low-rank adaptive LoRA correction. Based on the attention mask generated in step S2, the index of the last valid token in each sample is located. S52. Using extraction strategies, from Extract the 768-dimensional hidden state vector corresponding to the position of the last valid token. This serves as the global feature of the entire network flow FF; the extraction strategy utilizes the autoregressive properties of generative models, aggregating all preceding context information through the token at the end of the sequence; and the 768-dimensional hidden state vector... It serves as the input to the linear classification head in step S6, and also provides a computational path for the gradient backpropagation of the hidden state vector during training.
7. The encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation according to claim 1, characterized in that, Step S6 specifically includes: S61, using the linear classification head weight matrix The 768-dimensional hidden state vector The original score vector is mapped to C dimensions, and after softmax normalization, the class with the highest probability is taken as the classification decision. ;in, and Indicates category index, Represents the linear classifier head weight matrix The row vectors Represents the linear classifier head weight matrix The row vectors The predicted category output by the model. Indicates transpose; S62. Jointly optimize the linear classifier head with the low-rank adaptive LoRA parameters from step S4. After each epoch, calculate the accuracy, macro-average / F1, precision, and recall on the test set according to the class label mapping table, and record the training time and inference latency. If the current validation F1 is better than the historical best, save the corresponding low-rank adaptive LoRA parameter weights and linear classifier head weights to the local checkpoint.
8. The encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation according to claim 7, characterized in that, Step S7 specifically includes: Based on steps S1-S6, an atomic task including a training-validation process is constructed, and a unified ablation experiment scheduler is built to automatically traverse four hyperparameter dimensions, specifically including: Low-rank adaptive LoRA rank Injection location combination Sequence length and number of packets in the stream ,in, This indicates that only the attention projection matrix is injected at the injection location. The injection location represents the injection attention and output projection matrix. This indicates that the injection location is all linear layers; Each configuration triggers a complete S1→S6 link and uses a fixed random seed 42; before running, the backbone network is reset to the initial pre-training state through the loading interface of step S3; Finally, the classification and efficiency metrics of each configuration are summarized, and the impact trend of each hyperparameter dimension on performance and the optimal configuration reference are output.
9. The encrypted traffic classification method based on generative pre-trained language model and low-rank adaptation according to claim 8, characterized in that, Step S8 specifically includes: Steps S1-S6 are loosely coupled into an end-to-end system as the main process, with modules connected through standard PyTorch tensors and DataLoader interfaces. The independent resource monitoring thread includes parallel collection of CPU utilization, system memory usage, GPU utilization, and video memory usage at preset time intervals; The independent resource monitoring thread runs asynchronously with the main process, and automatically generates a resource time-series line chart after the task is completed. The end-to-end system loads the optimal checkpoint, reuses the GPT-2-small pre-trained weights in step S3, performs preprocessing and forward linking steps S1-S6 on the new packet capture file pcap, and finally outputs the real-time classification result.
10. A cryptographic traffic classification system based on generative pre-trained language models and low-rank adaptation, employing the cryptographic traffic classification method based on generative pre-trained language models and low-rank adaptation as described in any one of claims 1-9, characterized in that, include: The stream-level payload extractor adaptively identifies three directory layouts and constructs a label mapping through the pcap input port of the packet capture file. Its stream-level byte sequence output after bidirectional normalization of the 5-tuple is connected to the input of the Hex encoder, while the label mapping table output is connected to the label verification input of the classification index evaluator. The Hex encoder converter is used to convert the received byte sequence into a Hex space-separated string, and its output is connected to the input of the BPE tokenizer. The BPE segmenter is used to convert Hex space-separated strings into a fixed-length token ID sequence and attention mask by a GPT-2 pre-trained BPE segmenter. Its token output and mask output are connected to the standardized input interface of the GPT-2-small backbone network. The GPT-2-small backbone network is loaded with 124M parameters of frozen pre-trained weights, and the output of the hidden state sequence of the last layer is connected to the input of the first LoRA low-rank adapter. The first LoRA low-rank adapter injects low-rank decomposition matrix pairs into the attention projection matrix, output projection matrix, and feedforward network matrix linear layers of the backbone network for corrected forward computation. Its corrected last-layer hidden state output is connected to the input of the last-position token feature extractor. The last token feature extractor extracts the hidden state vector of the last non-padded token position of each sample based on the attention mask, and its flow-level feature vector output is connected to the input of the linear classification head. The linear classification head maps feature vectors to the original scores logits of each category and outputs classification decisions after softmax normalization. Its prediction output is connected to the prediction input interface of the classification index evaluator, and its optimal checkpoint storage path is connected to the loading interface of the system inference deployment stage. The classification metric evaluator and the efficiency metric evaluator receive the prediction results and true labels from the linear classification head in parallel. The classification metric evaluator outputs accuracy, macro-average F1, weighted F1, precision, and recall to the system evaluation report interface, while the efficiency metric evaluator outputs training time, inference latency, number of trainable parameters, and peak memory usage to the same evaluation report interface. The real-time resource monitor runs in an independent thread. Its CPU sampling end, memory sampling end, GPU utilization sampling end and video memory sampling end collect system resource usage in parallel. The time-series data output end is connected to the monitoring chart generation end of the evaluation report interface. The second LoRA low-rank adapter has its trainable parameter output and the trainable parameter output of the linear classification head connected together to the parameter update input of the AdamW optimizer during the training phase. After the adapter weights and classification head weights updated by the AdamW optimizer are evaluated by the validation set F1, they are selectively back-connected to the checkpoint saving interface of the first LoRA low-rank adapter and the checkpoint saving interface of the linear classification head for seamless loading during the inference phase.