Multi-modal traffic classification method and device, equipment and storage medium
By extracting and fusion of the traffic images and traffic maps of network traffic, the problem of performance bottlenecks in the existing technology based on single-modal information is solved, and more efficient and accurate traffic classification is achieved.
Patent Information
- Application Number
- CN202411881417.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-04-15
AI Technical Summary
In the prior art, network traffic classification based on single-modal information is prone to performance bottlenecks.
The multimodal traffic classification method is adopted to extract the traffic image and traffic map of network traffic, and the effective multimodal features are captured from the images and figures respectively, and fused to obtain a comprehensive feature representation of the traffic.
It reduces feature loss, improves the performance and accuracy of traffic classification, and solves the performance bottleneck problem of traffic classification based on single-modal information.
Smart Images
Figure CN119832303A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of network traffic classification, and in particular to a multimodal traffic classification method, apparatus, device and storage medium. Background Art
[0002] Network traffic classification is an important research area to improve service quality and enhance network security. At present, most traffic classification methods only focus on single modal information, such as traffic classification based on statistical features and traffic classification based on deep learning, which easily leads to significant feature loss and performance bottlenecks.
[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0004] The main purpose of this application is to provide a multimodal traffic classification method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that traffic classification based on single-modal information is prone to performance bottlenecks.
[0005] To achieve the above objectives, the present application provides a multi-modal traffic classification method, the method comprising:
[0006] Perform feature extraction on the traffic image of the network traffic to be classified and determine the image features;
[0007] Extracting features from the flow graph of the network traffic to be classified to determine graph features;
[0008] Performing multimodal fusion on the image features and the graph features to obtain multimodal fusion features;
[0009] Based on the multimodal fusion features, the network traffic to be classified is classified to obtain a classification result.
[0010] In one embodiment, the step of extracting features from the traffic image of the network traffic to be classified and determining the image features includes:
[0011] Inputting the traffic image of the network traffic to be classified into a global feature extractor for feature extraction to obtain global features, wherein the global feature extractor at least includes a Transformer model, and the Transformer model performs feature extraction based on a spatial self-attention mechanism;
[0012] The flow image is evenly divided into a first partial image and a second partial image, and the first partial image and the second partial image are respectively input into a local feature extractor for feature extraction to obtain corresponding local features, wherein the local feature extractor at least includes a reversible neural network unit, and the reversible neural network unit performs feature extraction based on a convolutional neural network;
[0013] The global feature and the local feature are input into a feature fusion encoder for feature fusion to obtain the image feature.
[0014] In one embodiment, the step of extracting features from the flow graph of the network traffic to be classified and determining graph features comprises:
[0015] The flow map is enhanced to obtain a flow enhancement map;
[0016] Based on the flow enhancement map, determining a sequence of feature vectors of the flow enhancement map;
[0017] The feature vector sequence of the flow enhancement graph is input into a long short-term memory network to determine the temporal dependency of the flow data, and the graph features are extracted based on the temporal dependency of the flow data.
[0018] In one embodiment, the step of enhancing the flow map to obtain a flow enhancement map includes:
[0019] Based on the node drop rate, randomly drop nodes and corresponding connection edges in the flow graph to obtain a node drop flow graph;
[0020] Based on the edge drop rate, edges are randomly dropped in the node drop flow graph to obtain the flow enhancement graph.
[0021] In one embodiment, the step of performing multimodal fusion on the image feature and the graph feature to obtain a multimodal fusion feature further includes:
[0022] Determining a target fusion strategy among preset fusion strategies, wherein the preset fusion strategies include at least a splicing fusion strategy, a full attention fusion strategy, a paired fusion strategy, and a cross attention fusion strategy;
[0023] Based on the target fusion strategy, determining a fusion relationship between the image feature and the graph feature;
[0024] Based on the image features, the graph features and the fusion relationship, obtaining initial multimodal fusion features;
[0025] The initial multimodal fusion features are input into a stream-level classification head for conversion to obtain the multimodal fusion features.
[0026] In one embodiment, the step of extracting features from the traffic image of the network traffic to be classified and determining the image features also includes:
[0027] Selecting continuous data packets from the network traffic to be classified;
[0028] Based on the mask feature regression matrix, converting the continuous data packets into a two-dimensional matrix;
[0029] Performing image transformation on the two-dimensional matrix to obtain an initial flow image;
[0030] Based on a preset mask ratio, the initial traffic image is randomly masked to obtain the traffic image.
[0031] In one embodiment, the step of extracting features from the flow graph of the network traffic to be classified and determining the graph features also includes:
[0032] Obtaining a byte sequence of the network traffic to be classified;
[0033] Determining nodes of the traffic graph based on bytes in the byte sequence;
[0034] Determining edges of the traffic graph based on point mutual information between the bytes;
[0035] Based on the nodes and edges of the traffic graph, a traffic graph of the network traffic to be classified is constructed.
[0036] In addition, to achieve the above purpose, the present application also proposes a multi-modal traffic classification device, the multi-modal traffic classification device comprising:
[0037] An image encoding module is used to extract features from the traffic image of the network traffic to be classified and determine image features;
[0038] A graph encoding module, used to extract features from the flow graph of the network traffic to be classified and determine graph features;
[0039] A multimodal feature fusion module, used for performing multimodal fusion on the image feature and the graph feature to obtain a multimodal fusion feature;
[0040] The traffic classification module is used to classify the network traffic to be classified based on the multimodal fusion features to obtain a classification result.
[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a multimodal traffic classification device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the multimodal traffic classification method as described above.
[0042] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal traffic classification method as described above are implemented.
[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the multimodal traffic classification method as described above are implemented.
[0044] The present application provides a multimodal traffic classification method, which extracts features from the traffic image of the network traffic to be classified and determines the image features; extracts features from the traffic graph of the network traffic to be classified and determines the graph features; performs multimodal fusion on the image features and the graph features to obtain multimodal fusion features; and classifies the network traffic to be classified based on the multimodal fusion features to obtain classification results. Effective multimodal features are captured from the image and the graph respectively, and are fused to obtain a comprehensive feature representation of the traffic, reduce feature loss, perform traffic classification tasks at the same byte-level granularity, utilize the diversity of traffic data, improve classification performance, ensure classification accuracy, and solve the technical problem that performance bottlenecks are prone to occur when traffic classification is based on single-modal information. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0047] Figure 1 This is a flow chart of the first embodiment of the multi-modal traffic classification method of the present application;
[0048] Figure 2 This is a flow chart of Embodiment 2 of the multi-modal traffic classification method of the present application;
[0049] Figure 3 A schematic diagram of the overall architecture of the multimodal traffic classification method provided in Example 2 of the present application;
[0050] Figure 4 This is a schematic diagram of the module structure of the multi-modal traffic classification device according to an embodiment of the present application;
[0051] Figure 5Schematic diagram of the device structure of the hardware operating environment involved in the multimodal traffic classification method in the embodiment of the present application.
[0052] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0055] The main solutions of the embodiments of the present application are: to perform feature extraction on the traffic image of the network traffic to be classified and determine the image features; to perform feature extraction on the traffic graph of the network traffic to be classified and determine the graph features; to perform multimodal fusion on the image features and the graph features to obtain multimodal fusion features; and to classify the network traffic to be classified based on the multimodal fusion features to obtain a classification result.
[0056] Currently, most traffic classification methods only focus on single modal information, which easily leads to significant feature loss and performance bottlenecks.
[0057] The present application provides a solution to capture effective multimodal features from images and graphs respectively, and fuse them to obtain a comprehensive feature representation of the traffic, reduce feature loss, perform traffic classification tasks at the same byte-level granularity, and utilize the diversity of traffic data to improve classification performance, ensure classification accuracy, and solve the technical problem that performance bottlenecks are prone to occur when traffic classification is based on single-modal information.
[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a multimodal traffic classification device, etc., and this embodiment does not specifically limit this. The following takes a multimodal traffic classification device as an example to illustrate this embodiment and the following embodiments.
[0059] The present application embodiment provides a multi-modal traffic classification method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the multi-modal traffic classification method of the present application.
[0060] In this embodiment, the multimodal traffic classification method includes steps S10 to S40:
[0061] Step S10, extracting features from the traffic image of the network traffic to be classified to determine image features;
[0062] It should be noted that the network traffic to be classified refers to the network traffic that needs to be classified at present. The traffic image is the image corresponding to the classified network traffic, and the features extracted from the traffic image are the image features.
[0063] In a feasible implementation, the step of determining the traffic image of the network traffic to be classified includes: selecting continuous data packets from the network traffic to be classified; converting the continuous data packets into a two-dimensional matrix based on a mask feature regression matrix; performing image conversion on the two-dimensional matrix to obtain an initial traffic image; and randomly performing mask processing on the initial traffic image based on a preset mask ratio to obtain the traffic image.
[0064] It should be noted that continuous data packets refer to continuous data packets selected from each flow. The specific number can be determined according to actual needs and is not specifically limited. This embodiment uses a formatted two-dimensional matrix to represent the original flow. In the specific implementation, the mask feature regression (MFR) matrix is used to format the selected continuous data packets into a two-dimensional matrix as a feature representation of the flow. The specific size of the two-dimensional matrix is determined according to the actual situation. This embodiment uses the original byte as the initial feature of the flow, that is, the value of each pixel in the flow representation matrix.
[0065] It is understandable that when the two-dimensional matrix is converted into a corresponding image, the image obtained by the conversion can be considered as a preliminary flow image, that is, an initial flow image, and further processing is required to obtain the final flow image. The preset mask ratio is a pre-set mask ratio, and the specific value can be determined according to actual needs, and no specific limitation is made to this. In this embodiment, the initial flow image is randomly masked according to the pre-set mask ratio to obtain the final flow image I.
[0066] In a feasible implementation, step S10 may include steps S101 to S103:
[0067] Step S101, inputting the traffic image of the network traffic to be classified into a global feature extractor for feature extraction to obtain global features, wherein the global feature extractor at least includes a Transformer model, wherein the Transformer r The model extracts features based on the spatial self-attention mechanism;
[0068] It should be noted that network packets usually exhibit obvious regional features, such as TCP and IP regions, each of which has specific features. In order to capture local features within these regions and global features between different regions, this embodiment designs a global feature extractor (BE) and a local feature extractor (DE).
[0069] In this embodiment, the global feature extractor includes at least a Transformer model, which uses a spatial self-attention mechanism to capture the contextual relationship between different parts of the traffic, extract features, and obtain global features. Considering the balance between computational efficiency and performance, a lightweight Transformer model is used. r The (LT) module is the backbone of the model. The LT module can reduce the number of parameters by reducing the embedding dimension while maintaining the same performance. The calculation relationship of the global feature is as follows:
[0070] BGF=LT(I)
[0071] Where BGF represents the extracted global feature, I represents the input traffic image, and LT represents the LT module of the global feature extractor.
[0072] Step S102, dividing the flow image into a first partial image and a second partial image on average, respectively inputting the first partial image and the second partial image into a local feature extractor for feature extraction to obtain corresponding local features, wherein the local feature extractor at least includes a reversible neural network unit, and the reversible neural network unit performs feature extraction based on a convolutional neural network;
[0073] In this embodiment, a local feature extractor is used to extract detailed local features of a traffic image. The local feature extractor includes at least an invertible neural network unit (INN). The INN module performs feature extraction based on a convolutional neural network (CNN), which can enable input and output features to generate each other, thereby better retaining input information.
[0074] In the specific implementation, the traffic image I is divided into two equal parts, namely the first part image I1 and the second part image I2, as shown below:
[0075] I1, I2=Shuffle(Separate(I))
[0076] In the formula, I1 represents the first part of the image, I2 represents the second part of the image, and I represents the traffic image. This division can focus on specific areas of the traffic image, thereby analyzing local relationship features in more detail and obtaining more accurate local features. The calculation relationship of local features is as follows:
[0077] DLF(I1)=I1*exp(CNN(I2))+CNN(I2)
[0078] DLF(I2)=I2*exp(CNN(I1))+CNN(I1)
[0079] In the formula, I1 represents the first part of the image, I2 represents the second part of the image, I represents the flow image, DLF represents the extracted local features, and CNN represents the processing operation of the convolutional neural network. INN adopts the forward training method for the following reasons: (1) The inherent information retention property of INN can accurately capture the local relationship features; (2) Using only forward training can reduce the computational complexity and meet the feature extraction requirements.
[0080] Step S103: input the global feature and the local feature into a feature fusion encoder for feature fusion to obtain the image feature.
[0081] It should be noted that the image features in this embodiment are features obtained by fusing global features and local features.
[0082] It can be understood that in order to fuse global features and local features, this embodiment designs a feature fusion encoder (FFE), which can be expressed by the following calculation relationship:
[0083] R=PReLU(Pooling(CNN(BGF+DLF))).
[0084] In the formula, R represents the fused image features, PReLU represents the activation function used, BGF represents the global features, DLF represents the local features, Pooling represents the pooling operation, and CNN represents the processing operation of the convolutional neural network. The feature fusion encoder is used to fuse the global features with the local features, and finally the image features of the flow image are obtained.
[0085] Furthermore, in order to restore the masked traffic image, this embodiment designs a corresponding decoder, which shares parameters with the feature fusion encoder. At this time, the cross entropy loss and the absolute error loss (L1 loss) can be used to calculate the decoupling loss, and its expression is as follows:
[0086] R d =Decoder(R)
[0087] L {ICL} =λ*CE(R d ,R)+v*L1(R d , R)
[0088] In the formula, R drepresents the image restored by the decoder, R represents the fused image features, L1(R d , R) represents L1 loss, CE(R d , R) represents the cross entropy loss, L {ICL} Represents the decoupled loss, λ is the coefficient affecting the cross entropy loss, v is the coefficient affecting the L1 loss, λ∈[0,1], v∈[0,1].
[0089] In this implementation, for traffic images, a masked autoencoder (MAE)-based image encoding module (MAEIEM) is designed, which includes a global feature extractor and a local feature extractor to cope with the regional characteristics of network data packets and enhance the model's semantic understanding of the data.
[0090] Step S20, extracting features from the flow graph of the network flow to be classified to determine graph features;
[0091] It should be noted that the traffic graph is a graph corresponding to the classified network traffic, and the features extracted from the traffic graph are graph features.
[0092] In a feasible implementation, the step of determining the traffic image of the network traffic to be classified includes: obtaining a byte sequence of the network traffic to be classified; determining the nodes of the traffic graph based on the bytes in the byte sequence; determining the edges of the traffic graph based on the point mutual information between the bytes; and constructing the traffic graph of the network traffic to be classified based on the nodes and edges of the traffic graph.
[0093] It should be noted that this embodiment converts the byte sequence into a graph G = {V, E, X}, i.e., a flow graph. The bytes in the byte sequence correspond one-to-one to the nodes in the flow graph, and all nodes of the flow graph are determined according to the bytes in the byte sequence.
[0094] It can be understood that this embodiment uses point mutual information (PMI) to model the correlation between two bytes. If the PMI value is positive, the edge is connected, otherwise, no edge is established, thereby determining all edges of the flow graph. According to the determined nodes and edges, the corresponding flow graph is constructed.
[0095] In a feasible implementation, step S20 may include steps S201 to S203:
[0096] Step S201, enhancing the flow map to obtain a flow enhancement map;
[0097] It should be noted that this embodiment enhances the flow graph, and the enhanced flow graph is a flow enhanced graph.
[0098] In a feasible implementation, step S201 may include: based on the node discard rate, randomly discarding nodes and corresponding connection edges in the traffic graph to obtain a node discard traffic graph; based on the edge discard rate, randomly discarding edges in the node discard traffic graph to obtain the traffic enhancement graph.
[0099] It should be noted that the node drop rate is the pre-set probability of a node being dropped, and the connection edge is the edge connecting the nodes. After the node is dropped, the flow graph can obtain the node drop flow graph. The calculation formula of node drop is as follows:
[0100] G′={{v i ⊙p i |v i ∈V}, {e ij ⊙p i |e ij ∈E}}={V′,E′}
[0101] Where G′ represents the node discard flow graph, p i ∈{0, 1} is from the Bernoulli distribution p i =~B(P {ND} ), indicating whether to discard node v i and its connected edges e ij , P ND ∈[0, 1] represents the node drop rate.
[0102] It can be understood that the edge drop rate is the pre-set probability of the edge being dropped. After the edge drop, the node drop flow graph can obtain the final flow enhancement graph. The calculation relationship of edge drop is as follows:
[0103] G AUG = {V′, {e ij ⊙p {ij} |e ij ∈E′}}
[0104] In the formula, G AUG represents the flow enhancement graph, p {ij} ∈{0, 1} from Bernoulli distribution p {ij} ~B(P {ED} ), indicating whether to discard edge e Ij , P {ED} ∈[0, 1] represents the edge drop rate.
[0105] In this embodiment, data enhancement is performed by randomly dropping nodes and edges.
[0106] Step S202, based on the flow enhancement map, determining a feature vector sequence of the flow enhancement map;
[0107] It can be understood that the feature vector sequence is a sequence of feature vectors. After obtaining the enhanced graph, GE is used to calculate the feature vector of the traffic enhancement graph, thereby obtaining the feature vector sequence. The calculation relationship is as follows:
[0108] Z AUG =GE(G AUG )
[0109] In the formula, Z AUG represents the feature vector, G AUG Represents the flow enhancement graph.
[0110] It should be understood that at this time, the original flow graph G can be used as an anchor point. For unsupervised contrast loss, only two different views of the same sample are considered as positive sample pairs, which is far from enough in representation learning. Therefore, this embodiment introduces supervised contrast loss to obtain more positive samples, so that the supervised loss value between positive samples can be calculated. The calculation relationship is as follows:
[0111]
[0112] Where L{PCL} represents the supervised contrast loss. Through supervised contrast loss, we can better utilize positive sample pairs and improve the representation learning ability of the model.
[0113] Step S203: input the feature vector sequence of the flow enhancement graph into a long short-term memory network, determine the temporal dependency of the flow data, and extract the graph features based on the temporal dependency of the flow data.
[0114] It is understandable that this embodiment uses a long short-term memory network (LSTM) to obtain flow-level feature representation. LSTM captures the temporal dependency of flow data by processing feature vector sequences, thereby extracting corresponding flow-level features, which are the final graph features. The calculation relationship is as follows:
[0115] F = LSTM(Z1, Z2, ..., Z n ).
[0116] Where Z1, Z2, ..., Z n represents the feature vector in the feature vector sequence, LSTM represents the processing operation of the long short-term memory network, and F represents the graph feature.
[0117] In this implementation, for the flow graph, a contrastive learning based graph encoding module (CLGEM) is designed, including a flow graph encoder and a fusion encoder, to reveal the semantic relationship between bytes to enhance the representation ability of the graph neural network.
[0118] Step S30, performing multimodal fusion on the image feature and the graph feature to obtain a multimodal fusion feature;
[0119] It should be noted that this embodiment captures effective multimodal features from the traffic image and the traffic graph respectively, and fuses them to obtain a comprehensive feature representation of the traffic.
[0120] In this embodiment, in order to effectively integrate features from different modalities (image features and graph features), a variety of different multimodal feature fusion methods are designed, such as splicing, full attention, pairing, cross attention, etc. The appropriate method can be flexibly selected according to actual needs, and this embodiment does not make specific limitations on this.
[0121] Step S40: classify the network traffic to be classified based on the multimodal fusion features to obtain a classification result.
[0122] It should be noted that the classification result is the final result obtained after classifying the network traffic to be classified. In this embodiment, multi-modal fusion features are used to classify traffic.
[0123] In this embodiment, a multimodal feature fusion encoder (MFFM) is designed to integrate features from image and graph modalities to achieve comprehensive feature representation.
[0124] This embodiment provides a multimodal traffic classification method, which extracts features from the traffic image of the network traffic to be classified and determines the image features; extracts features from the traffic graph of the network traffic to be classified and determines the graph features; performs multimodal fusion on the image features and the graph features to obtain multimodal fusion features; and classifies the network traffic to be classified based on the multimodal fusion features to obtain a classification result. Effective multimodal features are captured from the image and the graph respectively, and are fused to obtain a comprehensive feature representation of the traffic, reduce feature loss, perform traffic classification tasks at the same byte-level granularity, utilize the diversity of traffic data, improve classification performance, and ensure classification accuracy.
[0125] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , step S30 may include steps S301 to S304:
[0126] Step S301, determining a target fusion strategy in preset fusion strategies, wherein the preset fusion strategies at least include a splicing fusion strategy, a full attention fusion strategy, a paired fusion strategy, and a cross attention fusion strategy;
[0127] It should be noted that the preset fusion strategy is the preset multimodal fusion method, which at least includes the splicing fusion strategy, the full attention fusion strategy, the pair fusion strategy and the cross attention fusion strategy. The target fusion strategy is the multimodal fusion method finally selected, which can be selected from the preset fusion strategies according to actual needs, and there is no specific limitation on this.
[0128] It is understandable that since the image modality and graph modality represent byte-level semantic features and byte-level relationships respectively, they are not suitable for similarity-based fusion. Therefore, this embodiment adopts splicing to achieve the fusion of multimodal features, that is, the splicing fusion strategy. On the basis of splicing, the multi-head self-attention (Multihead Self-Attention, MHA) mechanism is used for fusion, that is, the full attention fusion strategy. The dot product is used to fuse image features and graph features, that is, the pair fusion strategy. The image features and graph features are stacked and input into the Transformer model, and the self-attention mechanism is used for fusion, that is, the cross-attention fusion strategy.
[0129] Step S302, determining a fusion relationship between the image feature and the graph feature based on the target fusion strategy;
[0130] If the target fusion strategy is the splicing fusion strategy, the fusion relationship between image features and graph features is as follows:
[0131] F {final} =CONCAT(R, F)
[0132] In the formula, F {final} represents the initial multimodal fusion features obtained by fusion, R represents image features, and F represents graph features.
[0133] If the target fusion strategy is the full attention fusion strategy, the fusion relationship between image features and graph features is as follows:
[0134] O = CONCAT (R, F)
[0135] F {final} =O+MHA(O)
[0136] In the formula, F {final} represents the initial multimodal fusion features obtained by fusion, R represents image features, F represents graph features, and O represents the concatenated features.
[0137] If the target fusion strategy is a pairwise fusion strategy, the fusion relationship between image features and graph features is as follows:
[0138] F {final} =(R·F)
[0139] In the formula, F {final}represents the initial multimodal fusion features obtained by fusion, R represents image features, and F represents graph features.
[0140] If the target fusion strategy is the cross-attention fusion strategy, the image features and graph features are stacked as T = {RF}, and T is input into the Transformer model, where the query vector, key vector, and value vector of each fragment are defined by the following equations:
[0141]
[0142] Next, we apply the scaled dot product attention to obtain the fusion relationship between image features and graph features, as shown below:
[0143]
[0144] In the formula, F {final} represents the initial multimodal fusion features obtained by fusion, R represents image features, F represents graph features, T represents stacked features, Q represents query vector, K represents key vector, and V represents value vector.
[0145] Furthermore, a variety of cross-attention fusion strategies can be set, and different QKV combinations correspond to different feature fusion methods.
[0146] Step S303, obtaining an initial multimodal fusion feature based on the image feature, the graph feature and the fusion relationship;
[0147] It can be understood that by substituting the image features and graph features into the corresponding calculation formulas, preliminary multimodal fusion features, namely initial multimodal fusion features, are determined. The initial multimodal fusion features need to be further transformed to obtain the final multimodal fusion features.
[0148] In specific implementation, the splicing and fusion strategy is the best.
[0149] Step S304: input the initial multimodal fusion features into a stream-level classification head for conversion to obtain the multimodal fusion features.
[0150] It should be noted that this embodiment uses a flow-level classification head to convert the initial multimodal fusion feature into flow-level feature information to obtain a multimodal fusion feature f. The conversion is completed through linear transformation and PReLU activation function. The formula is as follows:
[0151]
[0152] In the formula, f represents the multimodal fusion feature, F {final} represents the initial multimodal fusion feature, w {f1} and w {f2} is the weight of the stream-level classification head, b{f1} and b {f2} is bias.
[0153] It can be understood that at this time, the cross entropy loss of the multimodal fusion feature f can be calculated as follows:
[0154] L {FLCLS} =CE(f,y f )
[0155] Where, L {FLCLS} represents the cross entropy loss of the multimodal fusion feature f, CE() represents the cross entropy loss function, y f is the target variable consistent with the flow label.
[0156] It should be understood that the overall training objective can be formalized as the following computational relationship:
[0157] L=L {FLCLS} +α·L {PCL} +γ·L {ICL}
[0158] Where, L {FLCLS} represents the cross entropy loss of multimodal fusion features, L {ICL} represents the decoupling loss, L {PCL} represents the supervised contrast loss, α represents the coefficient controlling the contribution of the graphics task, and γ represents the coefficient controlling the contribution of the image task, α, γ∈[0, 1].
[0159] The present embodiment provides a multimodal traffic classification method, which determines a target fusion strategy in a preset fusion strategy, wherein the preset fusion strategy includes at least a splicing fusion strategy, a full attention fusion strategy, a pair fusion strategy, and a cross attention fusion strategy; based on the target fusion strategy, determines the fusion relationship between the image feature and the graph feature; based on the image feature, the graph feature, and the fusion relationship, obtains an initial multimodal fusion feature; inputs the initial multimodal fusion feature into a stream-level classification head for conversion, and obtains the multimodal fusion feature, which captures effective multimodal features from the image and the graph respectively, and fuses them to obtain a comprehensive feature representation of the traffic, reduce feature loss, perform traffic classification tasks at the same byte-level granularity, utilize the diversity of traffic data, improve classification performance, and ensure classification accuracy.
[0160] For example, to help understand the implementation process of the multi-modal traffic classification method obtained by combining this embodiment with the above-mentioned embodiment 2, please refer to Figure 3 , Figure 3 A schematic diagram of the overall architecture of a multi-modal traffic classification method is provided, specifically:
[0161] It consists of three main modules: MAE-based image encoding module (MAEIEM), contrastive learning-based graph encoding module (CLGEM) and multimodal feature fusion module (MFFM). MAEIEM includes a Transformer-based global feature extractor and a CNN-based local feature extractor, and a decoder that shares parameters with the encoder is designed to restore the masked image. CLGEM includes a contrastive learning-based flow graph encoder and a feature fusion encoder. MFFM includes a multimodal feature fusion encoder to integrate features from image modality and graph modality to achieve comprehensive feature representation.
[0162] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multimodal traffic classification method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0163] This application also provides a multi-modal flow classification device, please refer to Figure 4 , the multimodal traffic classification device comprises:
[0164] An image encoding module 10 is used to extract features from the traffic image of the network traffic to be classified and determine image features;
[0165] A graph encoding module 20, used to extract features from the flow graph of the network traffic to be classified and determine graph features;
[0166] A multimodal feature fusion module 30, used for performing multimodal fusion on the image feature and the graph feature to obtain a multimodal fusion feature;
[0167] The traffic classification module 40 is used to classify the network traffic to be classified based on the multimodal fusion features to obtain a classification result.
[0168] In a feasible implementation manner, the image encoding module 10 is further used to input the traffic image of the network traffic to be classified into a global feature extractor for feature extraction to obtain global features, wherein the global feature extractor at least includes a Transformer model, and the Transformer model performs feature extraction based on a spatial self-attention mechanism;
[0169] The flow image is evenly divided into a first partial image and a second partial image, and the first partial image and the second partial image are respectively input into a local feature extractor for feature extraction to obtain corresponding local features, wherein the local feature extractor at least includes a reversible neural network unit, and the reversible neural network unit performs feature extraction based on a convolutional neural network;
[0170] The global feature and the local feature are input into a feature fusion encoder for feature fusion to obtain the image feature.
[0171] In a feasible implementation manner, the graph encoding module 20 is further used to enhance the flow graph to obtain a flow enhancement graph;
[0172] Based on the flow enhancement map, determining a sequence of feature vectors of the flow enhancement map;
[0173] The feature vector sequence of the flow enhancement graph is input into a long short-term memory network to determine the temporal dependency of the flow data, and the graph features are extracted based on the temporal dependency of the flow data.
[0174] In a feasible implementation manner, the graph encoding module 20 is further used to randomly discard nodes and corresponding connection edges in the flow graph based on the node discard rate to obtain a node discard flow graph;
[0175] Based on the edge drop rate, edges are randomly dropped in the node drop flow graph to obtain the flow enhancement graph.
[0176] In a feasible implementation manner, the multimodal feature fusion module 30 is further used to determine a target fusion strategy in a preset fusion strategy, wherein the preset fusion strategy at least includes a splicing fusion strategy, a full attention fusion strategy, a pair fusion strategy, and a cross attention fusion strategy;
[0177] Based on the target fusion strategy, determining a fusion relationship between the image feature and the graph feature;
[0178] Based on the image features, the graph features and the fusion relationship, obtaining initial multimodal fusion features;
[0179] The initial multimodal fusion features are input into a stream-level classification head for conversion to obtain the multimodal fusion features.
[0180] In a feasible implementation manner, the image encoding module 10 is further used to select continuous data packets from the network traffic to be classified;
[0181] Based on the mask feature regression matrix, converting the continuous data packets into a two-dimensional matrix;
[0182] Performing image transformation on the two-dimensional matrix to obtain an initial flow image;
[0183] Based on a preset mask ratio, the initial traffic image is randomly masked to obtain the traffic image.
[0184] In a feasible implementation manner, the graph encoding module 20 is further used to obtain the byte sequence of the network traffic to be classified;
[0185] Determining nodes of the traffic graph based on bytes in the byte sequence;
[0186] Determining edges of the traffic graph based on point mutual information between the bytes;
[0187] Based on the nodes and edges of the traffic graph, a traffic graph of the network traffic to be classified is constructed.
[0188] The multimodal traffic classification device provided by the present application adopts the multimodal traffic classification method in the above-mentioned embodiment, which can solve the technical problem that traffic classification based on single-modal information is prone to performance bottlenecks. Compared with the prior art, the beneficial effects of the multimodal traffic classification device provided by the present application are the same as the beneficial effects of the multimodal traffic classification method provided by the above-mentioned embodiment, and other technical features in the multimodal traffic classification device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.
[0189] The present application provides a multimodal traffic classification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal traffic classification method in the above-mentioned embodiment one.
[0190] Reference below Figure 5 , which shows a schematic diagram of the structure of a multimodal traffic classification device suitable for implementing an embodiment of the present application. The multimodal traffic classification device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The multimodal traffic classification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0191] like Figure 5As shown, the multimodal traffic classification device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multimodal traffic classification device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the multimodal traffic classification device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a multimodal traffic classification device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0192] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0193] The multimodal traffic classification device provided by the present application adopts the multimodal traffic classification method in the above embodiment, which can solve the technical problem that traffic classification based on single-modal information is prone to performance bottlenecks. Compared with the prior art, the beneficial effects of the multimodal traffic classification device provided by the present application are the same as the beneficial effects of the multimodal traffic classification method provided by the above embodiment, and other technical features in the multimodal traffic classification device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0194] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0195] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0196] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the multimodal traffic classification method in the above-mentioned embodiment.
[0197] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0198] The computer-readable storage medium may be included in the multimodal traffic classification device; or may exist independently without being assembled into the multimodal traffic classification device.
[0199] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the multimodal traffic classification device, the multimodal traffic classification device: performs feature extraction on the traffic image of the network traffic to be classified to determine the image features; performs feature extraction on the traffic graph of the network traffic to be classified to determine the graph features; performs multimodal fusion on the image features and the graph features to obtain multimodal fusion features; and classifies the network traffic to be classified based on the multimodal fusion features to obtain a classification result.
[0200] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0201] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0202] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0203] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal traffic classification method, and can solve the technical problem that traffic classification based on single-modal information is prone to performance bottlenecks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the multimodal traffic classification method provided by the above-mentioned embodiment, and will not be repeated here.
[0204] The present application also provides a computer program product, including a computer program, which implements the steps of the multi-modal traffic classification method as described above when the computer program is executed by a processor.
[0205] The computer program product provided by the present application can solve the technical problem that traffic classification based on single-modal information is prone to performance bottlenecks. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the multi-modal traffic classification method provided in the above embodiment, which will not be repeated here.
[0206] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A multimodal traffic classification method, characterized in that: The method comprises: Perform feature extraction on the traffic image of the network traffic to be classified and determine the image features; Extracting features from the flow graph of the network traffic to be classified to determine graph features; Performing multimodal fusion on the image features and the graph features to obtain multimodal fusion features; Based on the multimodal fusion features, the network traffic to be classified is classified to obtain a classification result.
2. The method according to claim 1, characterized in that The step of extracting features from the traffic image of the network traffic to be classified and determining the image features comprises: Inputting the traffic image of the network traffic to be classified into a global feature extractor for feature extraction to obtain global features, wherein the global feature extractor at least includes a Transformer model, and the Transformer model performs feature extraction based on a spatial self-attention mechanism; The flow image is evenly divided into a first partial image and a second partial image, and the first partial image and the second partial image are respectively input into a local feature extractor for feature extraction to obtain corresponding local features, wherein the local feature extractor at least includes a reversible neural network unit, and the reversible neural network unit performs feature extraction based on a convolutional neural network; The global feature and the local feature are input into a feature fusion encoder for feature fusion to obtain the image feature.
3. The method according to claim 1, characterized in that The step of extracting features from the flow graph of the network traffic to be classified and determining graph features comprises: The flow map is enhanced to obtain a flow enhancement map; Based on the flow enhancement map, determining a sequence of feature vectors of the flow enhancement map; The feature vector sequence of the flow enhancement graph is input into a long short-term memory network to determine the temporal dependency of the flow data, and the graph features are extracted based on the temporal dependency of the flow data.
4. The method according to claim 3, characterized in that The step of enhancing the flow map to obtain a flow enhancement map comprises: Based on the node drop rate, randomly drop nodes and corresponding connection edges in the flow graph to obtain a node drop flow graph; Based on the edge drop rate, edges are randomly dropped in the node drop flow graph to obtain the flow enhancement graph.
5. The method according to claim 1, characterized in that The step of performing multimodal fusion on the image feature and the graph feature to obtain a multimodal fusion feature also includes: Determining a target fusion strategy among preset fusion strategies, wherein the preset fusion strategies include at least a splicing fusion strategy, a full attention fusion strategy, a paired fusion strategy, and a cross attention fusion strategy; Based on the target fusion strategy, determining a fusion relationship between the image feature and the graph feature; Based on the image features, the graph features and the fusion relationship, obtaining initial multimodal fusion features; The initial multimodal fusion features are input into a stream-level classification head for conversion to obtain the multimodal fusion features.
6. The method according to claim 1, characterized in that Before the step of extracting features from the traffic image of the network traffic to be classified and determining the image features, the following steps are also included: Selecting continuous data packets from the network traffic to be classified; Based on the mask feature regression matrix, converting the continuous data packets into a two-dimensional matrix; Performing image transformation on the two-dimensional matrix to obtain an initial flow image; Based on a preset mask ratio, the initial traffic image is randomly masked to obtain the traffic image.
7. The method according to any one of claims 1 to 6, characterized in that The step of extracting features from the flow graph of the network traffic to be classified and determining the graph features also includes: Obtaining a byte sequence of the network traffic to be classified; Determining nodes of the traffic graph based on bytes in the byte sequence; Determining edges of the traffic graph based on point mutual information between the bytes; Based on the nodes and edges of the traffic graph, a traffic graph of the network traffic to be classified is constructed.
8. A multi-modal traffic classification device, characterized in that: The device comprises: An image encoding module is used to extract features from the traffic image of the network traffic to be classified and determine image features; A graph encoding module, used to extract features from the flow graph of the network traffic to be classified and determine graph features; A multimodal feature fusion module, used for performing multimodal fusion on the image feature and the graph feature to obtain a multimodal fusion feature; The traffic classification module is used to classify the network traffic to be classified based on the multimodal fusion features to obtain a classification result.
9. A multimodal traffic classification device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multimodal traffic classification method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal traffic classification method according to any one of claims 1 to 7 are implemented.