A unified network traffic representation extraction method and system based on BERT-GGNN

Abstract

This invention discloses a method and system for extracting a unified network traffic representation based on BERT-GGNN, belonging to the field of network traffic recognition technology. The method includes: dividing network traffic data packets into tokens and constructing an input sequence; performing semantic embedding through token embedding, position embedding, and segment embedding to construct token representation units; pre-training a BERT model to extract semantic features of network traffic; designing a graph structure representation based on network traffic packets to generate node features and edge information of the graph; inputting this graph into a gated graph neural network (GGNN) to extract network traffic relationship feature representations; introducing bidirectional mutual attention computation and feature connection operations through a co-attention layer to fuse the network traffic semantic features with the network traffic relationship feature representations, and outputting the fused unified network traffic representation. This invention improves the feature representation capability of network traffic and increases the generalization ability for downstream classification tasks.

Description

Technical Field

[0001] This invention relates to the field of network traffic identification technology, specifically to a method and system for extracting a unified representation of network traffic based on BERT-GGNN. Background Technology

[0002] Network traffic classification technology plays a crucial role in network management and network security. It not only helps identify traffic categories from different applications or network services, but also is significant for protecting the privacy and anonymity of internet users, as well as monitoring and preventing cybercrime. With the widespread application of traffic encryption technologies, such as Tor and VPNs, traditional methods based on Deep Packet Inspection (DPI) are no longer effective for encrypted traffic, posing new challenges to network traffic classification.

[0003] To address this challenge, researchers have begun employing machine learning methods to classify encrypted traffic by analyzing its statistical characteristics. While these methods have improved classification accuracy to some extent, they suffer from limitations in feature selection, relying on expert experience and struggling to uncover deep-seated features, thus restricting the accuracy and comprehensiveness of classification. Furthermore, these methods have limited cross-class generalization capabilities, making them difficult to apply flexibly across different categories. In recent years, deep learning methods have gained attention for their ability to automatically learn complex feature patterns; however, they are highly dependent on the scale and distribution of labeled data. Insufficient or improperly distributed labeled data can easily lead to overfitting, affecting the model's generalization ability on new datasets.

[0004] Pre-trained models have achieved significant results in fields such as natural language processing and computer vision. They learn unbiased data representations from large amounts of unlabeled data and then easily transfer to downstream tasks by fine-tuning on a limited amount of labeled data. In the field of encrypted traffic classification, PERT innovatively introduced ALBERT and achieved excellent classification performance. However, this model lacks a specific design for the characteristics of encrypted traffic and fails to fully consider the unique features of encrypted traffic. ET-BERT introduced BERT and designed two self-supervised pre-training tasks. Although it achieved high accuracy on multiple traffic classification tasks, it only considered the semantic features of a single network flow and lacked the mining of global network packet traffic transmission structure and message correlation.

[0005] Existing traffic classification methods are insufficient in representing traffic features. Overly simplistic feature representations coupled with simple model structures result in poor generalization ability of traditional deep learning traffic classification models. These methods often only focus on the statistical features of individual network flows, while ignoring the semantic information of the network flows themselves and the structural information between flows, thus limiting the model's ability to learn deeply about encrypted traffic patterns. Summary of the Invention

[0006] The technical problem this invention aims to solve is to address the insufficient accuracy and generalization ability of network traffic classification in existing technologies. This is achieved by designing a comprehensive network traffic processing framework, including a network traffic semantic extraction module, a network traffic relationship extraction module, and a common attention layer module, to realize in-depth analysis and feature learning of network traffic. To solve the above problem, this invention adopts the following technical solution:

[0007] First, this invention proposes a unified network traffic representation extraction method based on BERT-GGNN, comprising the following steps:

[0008] S1. Design a network traffic semantic extractor to extract semantics from network traffic data, divide network traffic data packets into tokens and construct input sequences;

[0009] S2. The word is semantically embedded using three methods: word embedding, position embedding, and segment embedding. The three embedding results are added together to construct a complete word representation unit.

[0010] S3. Use the BERT model to pre-train the word representation units and extract semantic features of network traffic;

[0011] S4. Design a graph structure representation based on network traffic packets, realize the modeling of traffic context relationships, and generate the node features and edge information of the graph;

[0012] S5. Input the node features and edge information of the graph into the gated graph neural network GGNN to extract the relational graph features between network traffic, mine the global relationship between traffic, and generate a graph structure feature representation.

[0013] S6. Introduce a common attention layer, and fuse the semantic features of network traffic and the feature representation of graph structure through bidirectional mutual attention calculation and feature connection operation, and output the unified representation of network traffic after fusion.

[0014] Preferably, step S1 specifically includes the following steps:

[0015] The network traffic data bit stream is divided into packet bytes consisting of two 8-bit bit streams, and each packet byte is likened to a word in natural language; a network packet consisting of multiple packet bytes is likened to a sentence in natural language; network streams and sessions consisting of network packets with the same 5-tuple are likened to a paragraph in natural language; and network streams are divided into request streams and response streams.

[0016] Each network packet is converted into a token format, where each token consists of two adjacent packet bytes with values ​​ranging from 0 to 65535. A dictionary containing 65536 tokens is constructed, and special tokens [CLS], [SEP], [PAD], and [MASK] are added to the dictionary. [CLS] is used to indicate the start position of the input sequence, [SEP] is used to separate the request stream and the response stream, [PAD] is used to pad to meet the length requirements of the input sequence, and [MASK] is used to implement random masking to enhance the model's ability to learn contextual information.

[0017] The tokens in the request and response streams are combined into an input sequence, and then the sequence is divided into segments A and B, which belong to the segment embedding region.

[0018] Preferably, in step S2, after constructing a complete word representation unit, the word representation unit is used as input and processed by multiple Transformer encoding layers to obtain an embedding representation with a preset dimension for each word; wherein each encoding layer of the Transformer model consists of a self-attention mechanism and a feedforward neural network.

[0019] Preferably, in step S3, the BERT model is used to pre-train the word representation unit to extract semantic features of network traffic, specifically including network traffic packet byte masking task and network traffic flow order prediction task;

[0020] The network traffic packet byte masking task includes: randomly masking 15% of the network traffic packet bytes in the input semantic embedding, of which 80% are replaced with [MASK], 10% remain unchanged, and 10% are replaced with other non-special terms, and predicting the label of the masked position according to the context, and adjusting the weights of the BERT model using the cross-entropy loss function;

[0021] The network traffic flow order prediction task includes: randomly replacing the input in the request and response flow pairs with a 50% probability, so that the model performs binary prediction to determine whether the two sub-flows correspond.

[0022] Preferably, step S4, which designs a graph structure representation based on network traffic packets, specifically includes the following:

[0023] Represent the packet characteristics of each data packet in the same network flow as a node in a graph, denoted as . ,in , The number of network packets contained in a flow;

[0024] The order and timing information between data packets are represented as edges of a graph, denoted as . ,in and These represent the indices of the sending and receiving nodes, respectively; the edge weights are calculated by the time interval between data packets. To determine, denoted as ;

[0025] timestamp of data packet The direction of the edges is determined by traversing the data packets to connect the edges to nodes that arrived earlier and nodes that arrived later, in order to represent the temporal order of the data packets.

[0026] Each network flow also contains a global attribute that represents the overall characteristics of the flow.

[0027] Preferably, step S6 specifically includes the following steps:

[0028] Representing the graph structure features extracted by the gated graph neural network (GGNN) semantic feature representation of network traffic extracted by the BERT model As input to the common attention layer;

[0029] Through bidirectional mutual attention calculation, respectively and Generate attention representations for the interaction based on conditions;

[0030] The attention representations after interaction are fused through connection operations to obtain a unified representation of network traffic;

[0031] The fused network traffic is uniformly represented and input into the fully connected layer for classification prediction. The classification model is trained using the cross-entropy loss function, and the pre-trained BERT model parameters remain fixed during the classification stage.

[0032] In step 5, the gated graph neural network enables nodes to receive information from their neighbors through node state updates and message passing mechanisms, thereby achieving in-depth mining of traffic relationships in the graph structure. After multiple layers of message passing, the final graph representation is obtained, which can effectively reflect the global relationship characteristics between network traffic and provide important relational information for subsequent feature fusion.

[0033] Preferably, step S6 involves bidirectional mutual attention calculation, respectively using... and To generate attention representations after conditional interactions, specifically including:

[0034] BERT attention conditioned on GGNN features is represented as follows Calculate using the following formula:

[0035] ,

[0036] in, , and It is a trainable parameter matrix;

[0037] The GGNN attention representation conditioned on BERT features is as follows: Calculate using the following formula:

[0038] ,

[0039] in, , and It is a trainable parameter matrix;

[0040] Feature fusion is calculated using the following formula:

[0041] ,

[0042] in, Indicates the first A unified representation obtained by feature fusion of network flows; Indicates the first The graph structure feature representation obtained by the network flowing through the GGNN; Indicates the first The semantic feature representation of network traffic obtained by the BERT model; This represents a feature connection function used to concatenate graph structure feature representations with network traffic semantic feature representations along the feature dimension.

[0043] After feature fusion, a fully connected layer is used for classification prediction, and the classification model is trained using the cross-entropy loss function. To facilitate the representation of classification prediction, let... ,in, Indicates the first A unified representation obtained by fusing network flows through a common attention layer. (Classification prediction) With loss Calculate using the following formula:

[0044] ,

[0045] ,

[0046] in , These are trainable parameters. and These are the predicted tags and the actual tags for traffic classification; This represents the normalized exponential function, used to convert the classification score output by the fully connected layer into probability values ​​for each traffic category; Indicates the first A unified representation obtained by feature fusion of network flows; This represents the classification cross-entropy loss function;

[0047] Meanwhile, this invention proposes a unified network traffic representation extraction system based on BERT-GGNN, comprising:

[0048] The network traffic semantic extraction module is configured to perform the following process: designing a network traffic semantic extractor to extract semantics from network traffic data, dividing network traffic data packets into tokens and constructing an input sequence; semantically embedding the tokens using three methods: token embedding, position embedding, and segment embedding, adding the three embedding results to construct a complete token representation unit; and pre-training the token representation unit using the BERT model to extract network traffic semantic features.

[0049] The network traffic relationship extraction module is configured to perform the following process: design a graph structure representation based on network traffic packets, realize the modeling of traffic context relationships, and generate the node features and edge information of the graph; input the node features and edge information of the graph into a gated graph neural network (GGNN) to extract the relationship graph features between network traffic, mine the global relationships between traffic, and generate a graph structure feature representation.

[0050] The common attention layer module is configured to perform the following process: introduce a common attention layer, fuse the network traffic semantic features and graph structure feature representations through bidirectional mutual attention calculation and feature connection operations, and output a unified representation of the fused network traffic.

[0051] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the present invention.

[0052] Finally, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it implements the method described in the present invention.

[0053] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0054] (1) This invention achieves semantic embedding of network traffic packets by designing a network traffic semantic extractor; it designs two pre-training tasks for network traffic, namely the network traffic packet byte mask task and the network traffic packet sequence prediction task, to fully mine the semantics of network traffic.

[0055] (2) This invention realizes the modeling of traffic context relationship by designing a relationship graph representation between network traffic; and uses gated graph neural network (GGNN) to extract the relationship graph features between network traffic and mine the global relationship between traffic.

[0056] (3) This invention designs a feature fusion module based on a multi-head attention structure to achieve the fusion of network traffic semantic features and global relation features. From the model level, it improves the feature representation capability of network traffic and increases the generalization of downstream classification tasks. Attached Figure Description

[0057] Figure 1 This invention relates to an architecture diagram of a unified network traffic representation extraction method based on BERT-GGNN. Detailed Implementation

[0058] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0059] Example 1: Reference Figure 1 This embodiment provides a unified network traffic representation extraction method based on BERT-GGNN. It designs a comprehensive network traffic processing framework, including a network traffic semantic extraction module, a network traffic relationship extraction module, and a common attention layer module, to achieve in-depth analysis and feature learning of network traffic. In the network traffic semantic extraction module, this embodiment designs a network traffic semantic extractor and constructs complete lexical representation units through lexical embedding, positional embedding, and segment embedding. BERT is used for pre-training to extract network traffic semantic features. In the network traffic relationship extraction module, this invention designs a graph structure representation based on network traffic packets and uses a gated graph neural network (GGNN) to extract relationship graph features between network traffic, mining global relationships between traffic. The common attention layer module introduces a common attention mechanism to fuse the network traffic semantic representation extracted by BERT and the network traffic relationship representation extracted by GGNN, thereby achieving an effective combination of network traffic semantic features and global relationship features.

[0060] Specifically, this embodiment includes the following steps:

[0061] In this embodiment, raw traffic files in PCAP format are used as input. These files record data packets during network communication in chronological order. Each data packet includes fields such as timestamp, source address, destination address, source port, destination port, transport layer protocol type, and message byte content. During processing, the raw data packets are aggregated based on the five-tuple consisting of the source address, destination address, source port, destination port, and transport layer protocol to form session-level network flow samples, which are then divided into request flows and response flows according to the communication direction.

[0062] Building upon this, the message byte content in the request and response streams is converted into lexical sequences, which are then used as input to the BERT model to extract semantic features of network traffic. Simultaneously, data packets within the same network flow are represented as graph nodes, and the temporal order between data packets is represented as directed edges, with the arrival time interval of the data packets used as edge weights. This forms the graph structure input to a GGNN model, used to extract network traffic relationship features. Through these processes, the original PCAP format traffic file is converted into both semantic and relational representations of network traffic. These two types of features are then fused through a shared attention layer to obtain a unified representation vector for the corresponding network flow samples.

[0063] Step 1: Analogize network traffic data to natural language and design a network traffic semantic extractor based on BERT.

[0064] Specifically, the network traffic semantic extractor includes three steps:

[0065] S101: Network traffic is divided into sessions according to the five-tuple. Network packets in the network flow are divided into request flow and response flow according to the source and destination IPs.

[0066]

[0067] in, This represents a session derived from the partitioning of the same quintuple network flow; This represents the request flow from the source address to the destination address; This represents the response flow from the destination address to the source address; Represents the request stream The Middle One network packet; Represents response stream The Middle One network packet; and These represent the sequence numbers of network packets in the request stream and response stream, respectively. Represents the set of positive integers; superscript Indicates the direction from the source address to the destination address, superscript This indicates the direction from the destination address to the source address.

[0068] S102: Convert each network packet in the request and response streams into token format. The steps are as follows:

[0069] S1021: Split the hexadecimal value in each packet in the request stream and response stream into a unit consisting of two adjacent packet bytes.

[0070] S1022: Each unit is considered as a word element, with a value range of 0 to 65535 (hexadecimal FFFF), and the dictionary size is 65536.

[0071] S1023: Add special words to the dictionary: [CLS], [SEP], [PAD] and [MASK].

[0072] Wherein, [CLS]: used to mark the start position of the input sequence, and its last hidden layer state is used as a comprehensive representation of the entire input sequence. [SEP]: used to separate the request stream and the response stream. [PAD]: used to pad the input sequence to meet the model's minimum length requirement. [MASK]: used to randomly mask some words to help the model learn contextual information.

[0073] S103: The request stream in S containing the extracted word units. and response stream They are paired for flow order prediction during the pre-training phase. The model uses special lexical units [SEP] and segment embeddings to distinguish between sub-flows belonging to segments A and B, where segment A is denoted as sub-FlowA and segment B as sub-FlowB.

[0074] Step 2, network traffic semantic embedding, involves constructing complete lexical representation units through three embedding methods—lexical embedding, positional embedding, and segment embedding. The following is a summary of this step:

[0075] S201: Token Embeddings: Representations are constructed based on the position information of tokens in the vocabulary, including special tokens such as [CLS], [SEP], [MASK], and [PAD], as well as actual traffic data tokens. The embedding dimension is set to 768.

[0076] S202: Position Embeddings: Representations are constructed based on the positional information of words in the input sequence to preserve the order information of words in the sequence. The embedding dimension is also 768.

[0077] S203: Segment Embeddings: Representations are constructed based on the sub-flow (sub-FlowA and sub-FlowB) information to which the lexical belongs, to distinguish between request and response flows. The embedding dimension is 768.

[0078] S204: Combination Embedding: Add the lexical embedding, positional embedding, and segment embedding to obtain the complete lexical representation, which is used for subsequent Transformer model processing.

[0079] S205: Transformer processing: Through multiple Transformer encoding layers, the complete lexical representation is converted into an embedding representation with a dimension of 768 for each group, so as to realize an abstract representation of the lexical units in the original network packet.

[0080] Step 3, BERT pre-training, includes two tasks: (1) Network traffic packet bytemasking task and (2) Network traffic flow order prediction task.

[0081] S301: The network traffic packet bytemasking task is a pre-training task on network traffic data, similar to the MLM (Masked Language Model) task in the BERT model. The following are the detailed steps of this task:

[0082] S3011: Semantic embedding preparation, which combines the three types of embedding—lexical embedding, positional embedding, and segment embedding—to form the final semantic embedding.

[0083] S3012: Random masking. During pre-training, 15% of the tokens in the semantic embedding of the input model are randomly selected for masking. Of these, 80% of the selected tokens are replaced with the special token [MASK], 10% remain unchanged, and 10% are replaced with other non-special tokens.

[0084] S3013: Masking position prediction, predicting the token at the masked position based on context. The model needs to predict the original word to be masked based on context information.

[0085] S3014: Loss calculation, using cross-entropy loss to adjust the weights of the BERT model. Loss function. The calculation formula is:

[0086]

[0087] in, This represents the total number of samples participating in the training. For sample index; For each sample, there are indexes of positions randomly masked; each sample has a total of One mask position; For word indexes in the dictionary; As an indicator variable, when the first The original word corresponding to the mask position is exactly the first word in the dictionary. When there are 1 word elements, ,otherwise ; Indicates the first in the dictionary Each word element; Indicates the first The special word [MASK] inserted at each mask position; For the model to predict the first The actual word at each mask position is The probability of; It is the natural logarithm; This is the cross-entropy loss function for the network traffic packet bytemasking task.

[0088] S302: The network traffic packet order prediction task is similar to the Next Sentence Prediction (NSP) task in the BERT model. The following are the breakdown steps of this task:

[0089] S3021: Performs a random replacement operation on the input in the request and response stream pair with a 50% probability.

[0090] S3022: The pre-trained model needs to perform binary input prediction to determine whether two substreams correspond. The cross-entropy loss function here... It is represented as follows:

[0091]

[0092] in Indicates the first Individual flow pair, Indicate whether it belongs to a consecutive pair. = 0 indicates that the two sub-streams are not corresponding request and response streams. = 1 indicates a correspondence; Indicates the first Each sentence is predicted to be of class The probability, As an indicator variable, when the first The true category of each sub-flow pair is hour ,otherwise .

[0093] S303: Calculate the final loss function of the BERT model during the pre-training phase: During the pre-training phase, the two loss functions proposed above are added together to obtain the final loss function of the BERT model. .

[0094]

[0095] S304: The BERT model consists of an encoder from the Transformer module, which includes a self-attention layer and a feedforward layer. The self-attention layer can be represented as:

[0096]

[0097] in These are matrices representing queries, keys, and values. It is the dimension of the matrix.

[0098] S305: Define multi-head attention: .

[0099] The calculation formula for each head is as follows: ; This indicates the number of attention heads in a multi-head attention mechanism; Indicates the first The output of each attention head; , and They represent the first The trainable parameter matrix used to generate the query vector, key vector, and value vector in each attention head; This represents a trainable parameter matrix that maps the result of concatenating multiple attention heads to the output space. This represents a feature concatenation function, used to concatenate the outputs of multiple attention heads along the feature dimension; This represents the attention function, which is used to calculate attention weights based on the correlation between the query matrix and the key matrix, and to perform a weighted summation of the value matrix using the attention weights.

[0100] S306: After passing through multiple Transformer model modules, the final word feature representation is obtained. .

[0101] Step 4: Network Traffic Relationship Graph Representation. Design a graph structure representation based on network traffic packets. Use quintuples to divide network traffic into flows, and then convert each network flow into a graph structure. The graph consists of a set of nodes and edges, and each flow also contains a global attribute.

[0102] S401: Graph Nodes: In this invention, graph nodes are represented as packet characteristics of each data packet within the same network flow. Represented as... ,in . The number of network packets contained in a flow.

[0103] S402: Graph Edges: Graph edges are assigned to display relationships between nodes. In this invention, edges are used to indicate the sequential relationships and timing information between data packets.

[0104] S403: The temporal order relationship between data packets is treated as a directed edge, represented as follows: The timestamp of the data packet is represented as... ,in , It is the total number of data packets contained in a network stream.

[0105] S404: Calculate the time interval between arrival times of data packets. To label the weight of each edge, denoted as: .in, It is the sending node index. It is the index of the receiving node.

[0106] Step 5: Use a gated graph neural network for graph feature propagation and extraction:

[0107] S501: Model Initialization and Node State Update. GGNN (Gated Graph Sequence Neural Networks) is chosen, a spatial domain message-passing model based on GRU (Gated Recurrent Unit), suitable for handling sequence problems. Nodes in... The embedding of a time moment is determined by its own embedding, the embedding of its neighboring nodes, and the edge information of interactions between nodes.

[0108]

[0109]

[0110]

[0111] in It is an adjacency matrix. This represents the result of a node interacting with its neighboring nodes through edges. Controlling forgotten information, Controls newly generated information. σ is the sigmoid function. , and It is a trainable weight matrix.

[0112] S502: Calculation and updating of candidate hidden states for node states, using the following formula:

[0113]

[0114]

[0115] in, , and It is a trainable weight matrix. Controlling forgotten information, Controlling newly generated information, It is the first The final update node state of the layer, It is a candidate hidden state, which is an intermediate result generated during the calculation of the hidden state at the current moment.

[0116] S503: Multi-layer message passing and final representation retrieval: By stacking layers T times, a node can receive information from its T-hop neighbors, achieving multi-layer message passing. The message passing process is as follows: After message passing at layer M, the final graph representation C is obtained.

[0117] in Let be the parameter set of the t-th layer gated graph neural network.

[0118] Step 6: Introduce a common attention layer to fuse the semantic representation of network traffic extracted by BERT and the relationship representation between network traffic extracted by GGNN:

[0119] S601: Calculate the query, key, and value matrix based on the traffic feature representations extracted by GGNN and BERT. The keys and values ​​of the traffic feature representations from both networks are passed as input to each other's multi-head attention blocks. and Calculate the query, key, and value matrices respectively.

[0120] For GGNN feature representation, calculate ;

[0121] For BERT feature representation, calculate .

[0122] S602: Calculate mutual attention features: using the formula: Calculate the attention representation of the GGNN conditioned on the BERT output using the formula: Calculate the attention representation of BERT conditioned on the output of GGNN.

[0123] S603: Feature Fusion: This method uses concatenation to fuse the mutual attention features of two representations. The formula is as follows:

[0124]

[0125] in, Indicates the first A unified representation obtained by feature fusion of network flows; Indicates the first The graph structure feature representation obtained by the network flowing through the GGNN; Indicates the first The semantic feature representation of network traffic obtained by the BERT model; This represents a feature connection function used to concatenate graph structure feature representations with network traffic semantic feature representations along the feature dimension.

[0126] S604: After feature fusion, a fully connected layer is used for classification. The classification model is trained using the minimized cross-entropy loss. A pre-trained BERT model is used in this process, and the parameters of the BERT model are not adjusted.

[0127]

[0128]

[0129] , These are trainable parameters. and These are the predicted and actual labels for traffic classification.

[0130] Example 2: This example proposes a unified network traffic representation extraction system based on BERT-GGNN, including:

[0131] The network traffic semantic extraction module is configured to perform the following process: designing a network traffic semantic extractor to extract semantics from network traffic data, dividing network traffic data packets into tokens and constructing an input sequence; semantically embedding the tokens using three methods: token embedding, position embedding, and segment embedding, adding the three embedding results to construct a complete token representation unit; and pre-training the token representation unit using the BERT model to extract network traffic semantic features.

[0132] The network traffic relationship extraction module is configured to perform the following process: design a graph structure representation based on network traffic packets, realize the modeling of traffic context relationships, and generate the node features and edge information of the graph; input the node features and edge information of the graph into a gated graph neural network (GGNN) to extract the relationship graph features between network traffic, mine the global relationships between traffic, and generate a graph structure feature representation.

[0133] The common attention layer module is configured to perform the following process: introduce a common attention layer, fuse the network traffic semantic features and graph structure feature representations through bidirectional mutual attention calculation and feature connection operations, and output a unified representation of the fused network traffic.

[0134] Example 3: This example proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in this invention.

[0135] Example 4: This example proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed, it implements the method described in this invention.

[0136] It should be noted that the processing flow of embodiments 2-4 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.

[0137] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for extracting a unified network traffic representation based on BERT-GGNN, characterized in that, Includes the following steps: S1. Design a network traffic semantic extractor to extract semantics from network traffic data, divide network traffic data packets into tokens and construct input sequences; S2. The word is semantically embedded using three methods: word embedding, position embedding, and segment embedding. The three embedding results are added together to construct a complete word representation unit. S3. Use the BERT model to pre-train the word representation units and extract semantic features of network traffic; S4. Design a graph structure representation based on network traffic packets, realize the modeling of traffic context relationships, and generate the node features and edge information of the graph; S5. Input the node features and edge information of the graph into the gated graph neural network GGNN to extract the relational graph features between network traffic, mine the global relationship between traffic, and generate a graph structure feature representation. S6. Introduce a common attention layer, and fuse the semantic features of network traffic and the feature representation of graph structure through bidirectional mutual attention calculation and feature connection operation, and output the unified representation of network traffic after fusion.

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: The network traffic data bit stream is divided into packet bytes consisting of two 8-bit bit streams, and each packet byte is likened to a word in natural language; a network packet consisting of multiple packet bytes is likened to a sentence in natural language; network streams and sessions consisting of network packets with the same 5-tuple are likened to a paragraph in natural language; and network streams are divided into request streams and response streams. Each network packet is converted into a token format, where each token consists of two adjacent packet bytes with values ​​ranging from 0 to 65535. A dictionary containing 65536 tokens is constructed, and special tokens [CLS], [SEP], [PAD], and [MASK] are added to the dictionary. [CLS] is used to indicate the start position of the input sequence, [SEP] is used to separate the request stream and the response stream, [PAD] is used to pad to meet the length requirements of the input sequence, and [MASK] is used to implement random masking to enhance the model's ability to learn contextual information. The tokens in the request and response streams are combined into an input sequence, and then the sequence is divided into segments A and B, which belong to the segment embedding region.

3. The method according to claim 1, characterized in that, In step S2, after constructing a complete lexical representation unit, the lexical representation unit is used as input and processed by multiple Transformer encoding layers to obtain an embedding representation of each lexical unit with a preset dimension; wherein each encoding layer of the Transformer model consists of a self-attention mechanism and a feedforward neural network.

4. The method according to claim 2, characterized in that, In step S3, the BERT model is used to pre-train the word representation unit to extract semantic features of network traffic, specifically including network traffic packet byte masking task and network traffic flow order prediction task. The network traffic packet byte masking task includes: randomly masking 15% of the network traffic packet bytes in the input semantic embedding, of which 80% are replaced with [MASK], 10% remain unchanged, and 10% are replaced with other non-special terms, and predicting the label of the masked position according to the context, and adjusting the weights of the BERT model using the cross-entropy loss function; The network traffic flow order prediction task includes: randomly replacing the input in the request and response flow pairs with a 50% probability, so that the model performs binary prediction to determine whether the two sub-flows correspond.

5. The method according to claim 1, characterized in that, Step S4 involves designing a graph structure representation based on network traffic packets, specifically including the following: Represent the packet characteristics of each data packet in the same network flow as a node in a graph, denoted as . ,in , The number of network packets contained in a network flow; The order and timing information between data packets are represented as edges of a graph, denoted as . ,in and These represent the indices of the sending and receiving nodes, respectively; the edge weights are calculated by the time interval between data packets. To determine, denoted as ; timestamp of data packet The direction of the edges is determined by traversing the data packets to connect the edges to nodes that arrived earlier and nodes that arrived later, in order to represent the temporal order of the data packets. Each network flow also contains a global attribute that represents the overall characteristics of the flow.

6. The method according to claim 1, characterized in that, Step S6 specifically includes the following steps: Representing the graph structure features extracted by the gated graph neural network (GGNN) semantic feature representation of network traffic extracted by the BERT model As input to the common attention layer; Through bidirectional mutual attention calculation, respectively with and Generate attention representations for the interaction based on conditions; The attention representations after interaction are fused through connection operations to obtain a unified representation of network traffic; The fused network traffic is uniformly represented and input into the fully connected layer for classification prediction. The classification model is trained using the cross-entropy loss function, and the pre-trained BERT model parameters remain fixed during the classification stage.

7. The method according to claim 6, characterized in that, Step S6 calculates bidirectional mutual attention, respectively using... and To generate attention representations after conditional interactions, specifically including: BERT attention conditioned on GGNN features is represented as follows Calculate using the following formula: ， in, , and It is a trainable parameter matrix; The GGNN attention representation conditioned on BERT features is as follows: Calculate using the following formula: ， in, , and It is a trainable parameter matrix; This represents the scaled dot product attention function, which calculates attention weights based on the correlation between the query matrix and the key matrix, and performs a weighted summation on the value matrix. Feature fusion is calculated using the following formula: ， in, Indicates the first A unified representation of network flows after fusion This represents the graph structure feature representation extracted by the graph neural network model. This represents the semantic features of network traffic extracted by the BERT model. This indicates a concatenation operation that connects elements according to their feature dimensions. Classification prediction With loss Calculate using the following formula: ， ， in , These are trainable parameters. and These are the predicted tags and the actual tags for traffic classification; This is a normalized exponential function used to convert the output of the classification layer into a probability distribution; For the first A unified representation obtained after feature fusion of each network flow; This is the cross-entropy loss function for classification.

8. A unified network traffic representation extraction system based on BERT-GGNN, characterized in that, include: The network traffic semantic extraction module is configured to perform the following process: designing a network traffic semantic extractor to extract semantics from network traffic data, dividing network traffic data packets into tokens and constructing an input sequence; semantically embedding the tokens using three methods: token embedding, position embedding, and segment embedding, adding the three embedding results to construct a complete token representation unit; and pre-training the token representation unit using the BERT model to extract network traffic semantic features. The network traffic relationship extraction module is configured to perform the following process: design a graph structure representation based on network traffic packets, realize the modeling of traffic context relationships, and generate the node features and edge information of the graph; input the node features and edge information of the graph into a gated graph neural network (GGNN) to extract the relationship graph features between network traffic, mine the global relationships between traffic, and generate a graph structure feature representation. The common attention layer module is configured to perform the following process: introduce a common attention layer, fuse the network traffic semantic features and graph structure feature representations through bidirectional mutual attention calculation and feature connection operations, and output a unified representation of the fused network traffic.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed, it implements the method as described in claims 1-7.

Images